Orienting attention in visual working memory requires central capacity: decreased retro-cue effects under dual-task conditions

Hearing fearful prosody impairs visual working memory maintenance

Interference by distractors has been associated multiple times with diminished visual and auditory working memory (WM) performance. Negative emotional distractors in particular lead to detrimental effects on WM. However, these associations have only been seen when distractors and items to maintain in WM are from the same sensory modality. In this study, we investigate cross-modal interference on WM. We invited 20 participants to complete a visual change-detection task, assessing visual WM (VWM), while hearing emotional (fearful) and neutral auditory distractors. Electrophysiological activity was recorded to measure contralateral delay activity (CDA) and auditory P2 event-related potentials (ERP), indexing WM maintenance and distractor salience respectively. At the behavioral level, fearful prosody didn't decrease significantly working memory accuracy, compared to neutral prosody. Regarding ERPs, fearful distractors evoked a greater P2 amplitude than neutral distractors. Correlations between the two ERP potentials indicated that P2 amplitude difference between the two types of prosody was associated with the difference in CDA amplitude for fearful and neutral trials. This association suggests that cognitive resources required to process fearful prosody detrimentally impact VWM maintenance. That result provides a piece of additional evidence that negative emotional stimuli produce greater interference than neutral stimuli and that the cognitive resources used to process stimuli from different modalities come from a common pool.

The Intensity of Internal and External Attention Assessed with Pupillometry

Not only is visual attention shifted to objects in the external world, attention can also be directed to objects in memory. We have recently shown that pupil size indexes how strongly items are attended externally, which was reflected in more precise encoding into visual working memory. Using a retro-cuing paradigm, we here replicated this finding by showing that stronger pupil constrictions during encoding were reflective of the depth of encoding. Importantly, we extend this previous work by showing that pupil size also revealed the intensity of internal attention toward content stored in visual working memory. Specifically, pupil dilation during the prioritization of one among multiple internally stored representations predicted the precision of the prioritized item. Furthermore, the dynamics of the pupillary responses revealed that the intensity of internal and external attention independently determined the precision of internalized visual representations. Our results show that both internal and external attention are not all-or-none processes, but should rather be thought of as continuous resources that can be deployed at varying intensities. The employed pupillometric approach allows to unravel the intricate interplay between internal and external attention and their effects on visual working memory.

Sustained attention required for effective dimension-based retro-cue benefit in visual working memory

In visual working memory (VWM) tasks, participants' performances can be improved through the use of dimension-based retro-cues, which direct internal attention to prioritize a particular dimension (e.g., color or orientation) of VWM representations even after the stimuli disappear. This phenomenon is known as the dimension-based retro-cue benefit (RCB). The present study investigates whether sustained attention is required for the dimension-based RCB by inserting interference or interruption between the retro-cue and the test array to distract attention. We tested the effects of perceptual interference or cognitive interruption on dimension-based RCB when the interference (Experiments 1 and 2 with masks) or interruption (Experiments 3 and 4 with an odd-even task) occurred concurrently with the stages for the maintenance of prioritized information (long cue-and-interference/interruption interstimulus interval, e.g., Experiments 1 and 3) or the deployment of attention (short cue-and-interference/interruption interstimulus interval, e.g., Experiments 2 and 4). Our results demonstrate that perceptual interference or cognitive interruption attenuates the dimension-based RCB. These findings suggest that sustained attention is necessary for the effective prioritization of a specific dimension of VWM representations.

Turning Attention Inside Out: How Working Memory Serves Behavior

Flexible behavior requires guidance not only by sensations that are available immediately but also by relevant mental contents carried forward through working memory. Therefore, selective-attention functions that modulate the contents of working memory to guide behavior (inside-out) are just as important as those operating on sensory signals to generate internal contents (outside-in). We review the burgeoning literature on selective attention in the inside-out direction and underscore its functional, flexible, and future-focused nature. We discuss in turn the purpose (why), targets (what), sources (when), and mechanisms (how) of selective attention inside working memory, using visual working memory as a model. We show how the study of internal selective attention brings new insights concerning the core cognitive processes of attention and working memory and how considering selective attention and working memory together paves the way for a rich and integrated understanding of how mind serves behavior.

Attentional distraction affects maintenance of information in visual sensory memory

Classical theoretical models suggest that visual short-term memory can be divided in two main memory systems: sensory memory, a short-lasting but high-capacity memory storage and working memory, a long-lasting but low-capacity memory store. Whilst, previous research has systematically shown a strong interplay between attentional mechanisms and working memory, less clear is the role of attention in sensory memory. In the present study we approach this issue by asking whether withdrawing attentional resources by a dual task (Experiment 1) or by presenting task irrelevant information during memory maintenance (Experiment 2 and 3) similarly or differently affect sensory and working memory. Overall, results showed that sensory memory content was undermined not only by a simultaneous high-demanding cognitive task but even when purely task-irrelevant and non-masking visual distractors were presented during maintenance. Our data provide support against theories that consider sensory memories as a case of visual awareness free of attention.

Common Neural Mechanisms Control Attention and Working Memory

Although previous studies point to qualitative similarities between working memory (WM) and attention, the degree to which these two constructs rely on shared neural mechanisms remains unknown. Focusing on one such potentially shared mechanism, we tested the hypothesis that selecting an item within WM utilizes similar neural mechanisms as selecting a visible item via a shift of attention. We used fMRI and machine learning to decode both the selection among items visually available and the selection among items stored in WM in human subjects (both sexes). Patterns of activity in visual, parietal, and to a lesser extent frontal cortex predicted the locations of the selected items. Critically, these patterns were strikingly interchangeable; classifiers trained on data during attentional selection predicted selection from WM, and classifiers trained on data during selection from memory predicted attentional selection. Using models of voxel receptive fields, we visualized topographic population activity that revealed gain enhancements at the locations of the externally and internally selected items. Our results suggest that selecting among perceived items and selecting among items in WM share a common mechanism. This common mechanism, analogous to a shift of spatial attention, controls the relative gains of neural populations that encode behaviorally relevant information.SIGNIFICANCE STATEMENT How we allocate our attention to external stimuli that we see and to internal representations of stimuli stored in memory might rely on a common mechanism. Supporting this hypothesis, we demonstrated that not only could patterns of human brain activity predict which items were selected during perception and memory, but that these patterns were interchangeable during external and internal selection. Additionally, this generalized selection mechanism operates by changes in the gains of the neural populations both encoding attended sensory representations and storing relevant memory representations.

The part-list cuing effect in working memory: The influence of task presentation mode

The modulation of part-list cues on long-term memory has been well-documented, whereas its impact on working memory remains largely unknown. The current study recruited a working memory part-list cuing paradigm to investigate how re-exposing part-list items affected item representation in working memory, and more specifically, whether the cuing effect was modulated by the task presentation mode. Our results showed that when the part-list re-exposure and no-part-list re-exposure trials were presented in separate blocks, using the re-exposed items as retrieval cues (part-list cue condition) significantly impaired recognition speed, accuracy and elevated judgement criteria (Experiment 1a), whereas merely relearning the re-exposed items (part-list relearning condition) has no such effect (Experiment 1b). When the part-list cue trials are randomly interleaved with the no-part-list cue trials, recognition accuracy was significantly lower in the part-list cue condition, whereas the recognition speed and judgement criteria were not significantly different under the two conditions (Experiment 2). These results indicate that re-exposing subsets of previously memorized items as retrieval cues can reduce the strength of other representations in working memory. Moreover, the effect of part-list cues in working memory is affected by task presentation mode. The mechanisms of part-list cuing within working memory were discussed.

Selection in working memory is resource-demanding: Concurrent task effects on the retro-cue effect

In a retro-cue paradigm, after memorizing a set of objects, people are cued to remember only a subset. Improved memory from the retro-cue suggests that selection processes can benefit items stored in working memory. Does selection in working memory require attention? If so, an attention-demanding task should disrupt retro-cue effects. Studies using a dual-task paradigm have found mixed results, with only one study (Janczyk & Berryhill, Attention, Perception, and Psychophysics, 76 (3), 715–724, 2014) showing a decreased retro-cue effect by a secondary task. Here we explore a potential issue in that study – the temporal overlap of the secondary task response with the memory test presentation. This raises questions about whether the secondary task was impairing selection processes in memory or was impacting the memory response. We replicated their paradigm by inserting a tone discrimination task at the retro-cue offset, but we also included a condition in which the tone task and the memory test were temporally separated. In Experiment 1, performing the tone task did not impair the retro-cue effect. In Experiment 2, we added an articulatory suppression task as in Janczyk and Berryhill’s study, and we found that the requirement to execute the tone task impaired retro-cue effects. This impairment was independent of whether the tone and memory tasks overlapped. These findings suggest that internal prioritization can be impaired by dual-task interference, but may only occur when such interference is robust enough, for example, due to switching between multiple tasks.

Object-based attention in retaining binding in working memory: Influence of activation states of working memory

It has been suggested that retaining bindings in working memory (WM) requires more object-based attention than retaining constituent features. Recent studies have found that when memorized stimuli are presented sequentially, the most recent stimulus is in a highly accessible privileged state such that it is retained in a relatively automatic and resource-free manner, whereas the other stimuli are in a non-privileged state. The current study investigated whether the activation states of WM modulate the role of object-based attention in retaining bindings in WM. To address this question, we presented three colored shapes sequentially and added a transparent-motion task (Experiment 1) or a mental rotation task (Experiment 2) into the WM maintenance phase to consume object-based attention. We consistently found that consuming object-based attention led to a larger impairment to bindings relative to constituent features, which is independent of the WM activation states, suggesting that object-based attention is critical in retaining bindings in WM across activation states of WM.

More Than Hitting the Correct Key Quickly

Many studies have documented that multitasking reduces Response Time (RT) indicators of implicit sequence learning as well as the expression of acquired sequence knowledge in RT benefits. In these tasks it is only relevant that the correct key is hit quickly, not where it is hit. We explored how variability in response location is influenced by (a) breaking a repeating sequence of target locations, (b) multitasking demands in the current trial, and (c) presence of multitasking in the block. Participants performed a Serial Reaction Time Task (SRTT) on a touchscreen while shutting down a beep tone by pressing the space bar with their non-dominant hand (throughout Experiment 1 and in the second half of Experiment 2). The first-order sequence of four response locations on the screen was broken by off-sequence deviants in 1/6th of the trials. Our results show a dissociation between RT and response location variability. While the effect of breaking the sequence on RT was larger under single- than under multitasking, breaking the sequence only led to an increase in response location variability under multitasking. Experiment 3 suggested that the impact of sequence knowledge on either aspect of performance in the SRTT is limited by interference from an additional task.

Differential effects of cue-based and sequence knowledge-based predictability on multitasking performance

Everyday multitasking often is characterized by predictable sequences. While such sequential regularities are present in setups using the Serial Reaction Time Task (SRTT), many laboratory studies on dual-tasking performance use random sequences of stimuli in either of the two tasks. In the current study, following single-task training on the SRTT, participants completed trials where they were confronted with an additional visual-manual task with either a random (Experiment 1) or a partially predictable (Experiment 2) stimulus sequence. In the SRTT, we cued participants with respect to which of the four stimulus options were yet to occur (before a new round with all four options would start). We randomly mixed a sequence to be practiced with random sequences of the same length and with the same constraint. Thus, we were able to vary predictability of upcoming stimuli (from chance to 100%) as well as sequence knowledge (practiced vs. random sequence) in order to assess how cueing and sequence knowledge, as two potential bases of prediction, would affect performance in single- and dual-tasking. Results suggest that both cueing and sequence knowledge-based prediction can lead to shorter RTs in dual-tasking. In previous studies, the disruption of sequence learning by adding a task with a random stimulus sequence has been linked to the effects of automatic prediction between events in the two tasks. In line with these studies, dual-task performance did not impede usage of sequence knowledge when a task with a predictable (rather than random) sequence of stimuli was added to the SRTT.

Pruning representations in a distributed model of working memory: a mechanism for refreshing and removal?

Substantial behavioral evidence suggests that attention plays an important role in working memory. Frequently, attention is characterized as enhancing representations by increasing their strength or activation level. Despite the intuitive appeal of this idea, using attention to strengthen representations in computational models can lead to unexpected outcomes. Representational strengthening frequently leads to worse, rather than better, performance, contradicting behavioral results. Here, we propose an alternative to a pure strengthening account, in which attention is used to selectively strengthen useful and weaken less useful components of distributed memory representations, thereby pruning the representations. We use a simple sampling algorithm to implement this pruning mechanism in a computational model of working memory. Our simulations show that pruning representations in this manner leads to improvements in performance compared with a lossless (i.e., decay-free) baseline condition, for both discrete recall (e.g., of a list of words) and continuous reproduction (e.g., of an array of colors). Pruning also offers a potential explanation of why a retro-cue drawing attention to one memory item during the retention interval improves performance. These results indicate that a pruning mechanism could provide a viable alternative to pure strengthening accounts of attention to representations in working memory.

Where to attend next: guiding refreshing of visual, spatial, and verbal representations in working memory

One of the functions that attention may serve in working memory (WM) is boosting information accessibility, a mechanism known as attentional refreshing. Refreshing is assumed to be a domain-general process operating on visual, spatial, and verbal representations alike. So far, few studies have directly manipulated refreshing of individual WM representations to measure the WM benefits of refreshing. Recently, a guided-refreshing method was developed, which consists of presenting cues during the retention interval of a WM task to instruct people to refresh (i.e., attend to) the cued items. Using a continuous-color reconstruction task, previous studies demonstrated that the error in reporting a color varies linearly with the frequency with which it was refreshed. Here, we extend this approach to assess the WM benefits of refreshing different representation types, from colors to spatial locations and words. Across six experiments, we show that refreshing frequency modulates performance in all stimulus domains in accordance with the tenet that refreshing is a domain-general process in WM. The benefits of refreshing were, however, larger for visual-spatial than verbal materials.

Is refreshing in working memory impaired in older age? Evidence from the retro-cue paradigm

Impairments in refreshing have been suggested as one source of working memory (WM) deficits in older age. Retro-cues provide an important method of investigating this question: a retro-cue guides attention to one WM item, thereby arguably refreshing it and increasing its accessibility compared with a no-cue baseline. In contrast to the refreshing deficit hypothesis, intact retro-cue benefits have been found in older adults. Refreshing, however, is assumed to boost not one but several WM representations when sequentially applied to them. Hence, intact refreshing requires the flexible switching of attention among WM items. So far, it remains an open question whether older adults show this flexibility. Here, we investigated whether older adults can use multiple cues to sequentially refresh WM representations. Younger and older adults completed a continuous-color delayed-estimation task, in which the number of retro-cues (0, 1, or 2) presented during the retention interval was manipulated. The results showed a similar retro-cue benefit for younger and older adults, even in the two-cue condition in which participants had to switch attention between items to refresh representations in WM. These findings suggest that the capacity to use cues to refresh information in visual WM may be preserved with age.

Are there multiple ways to direct attention in working memory?

In visual working memory tasks, memory for an item is enhanced if participants are told that the item is relatively more valuable than others presented within the same trial. Experiment 1 explored whether these probe value boosts (termed prioritization effects in previous literature) are affected by probe frequency (i.e., how often the more valuable item is tested). Participants were presented with four colored shapes sequentially and asked to recall the color of one probed item following a delay. They were informed that the first item was more valuable (differential probe value) or as valuable as the other items (equal probe value), and that this item would be tested more frequently (differential probe frequency) or as frequently (equal probe frequency) as the other items. Probe value and probe frequency boosts were observed at the first position, though both were accompanied by costs to other items. Probe value and probe frequency boosts were additive, suggesting the manipulations yield independent effects. Further supporting this, experiment 2 revealed that probe frequency boosts are not reliant on executive resources, directly contrasting with previous findings regarding probe value. Taken together, these outcomes suggest there may be several ways in which attention can be directed in working memory.

In search of the focus of attention in working memory: 13 years of the retro-cue effect

The concept of attention has a prominent place in cognitive psychology. Attention can be directed not only to perceptual information, but also to information in working memory (WM). Evidence for an internal focus of attention has come from the retro-cue effect: Performance in tests of visual WM is improved when attention is guided to the test-relevant contents of WM ahead of testing them. The retro-cue paradigm has served as a test bed to empirically investigate the functions and limits of the focus of attention in WM. In this article, we review the growing body of (behavioral) studies on the retro-cue effect. We evaluate the degrees of experimental support for six hypotheses about what causes the retro-cue effect: (1) Attention protects representations from decay, (2) attention prioritizes the selected WM contents for comparison with a probe display, (3) attended representations are strengthened in WM, (4) not-attended representations are removed from WM, (5) a retro-cue to the retrieval target provides a head start for its retrieval before decision making, and (6) attention protects the selected representation from perceptual interference. The extant evidence provides support for the last four of these hypotheses.

Retro-dimension-cue benefit in visual working memory

In visual working memory (VWM) tasks, participants’ performance can be improved by a retro-object-cue. However, previous studies have not investigated whether participants’ performance can also be improved by a retro-dimension-cue. Three experiments investigated this issue. We used a recall task with a retro-dimension-cue in all experiments. In Experiment 1, we found benefits from retro-dimension-cues compared to neutral cues. This retro-dimension-cue benefit is reflected in an increased probability of reporting the target, but not in the probability of reporting the non-target, as well as increased precision with which this item is remembered. Experiment 2 replicated the retro-dimension-cue benefit and showed that the length of the blank interval after the cue disappeared did not influence recall performance. Experiment 3 replicated the results of Experiment 2 with a lower memory load. Our studies provide evidence that there is a robust retro-dimension-cue benefit in VWM. Participants can use internal attention to flexibly allocate cognitive resources to a particular dimension of memory representations. The results also support the feature-based storing hypothesis.

The (Un)Clear Effects of Invalid Retro-Cues

Studies with the retro-cue paradigm have shown that validly cueing objects in visual working memory long after encoding can still benefit performance on subsequent change detection tasks. With regard to the effects of invalid cues, the literature is less clear. Some studies reported costs, others did not. We here revisit two recent studies that made interesting suggestions concerning invalid retro-cues: One study suggested that costs only occur for larger set sizes, and another study suggested that inclusion of invalid retro-cues diminishes the retro-cue benefit. New data from one experiment and a reanalysis of published data are provided to address these conclusions. The new data clearly show costs (and benefits) that were independent of set size, and the reanalysis suggests no influence of the inclusion of invalid retro-cues on the retro-cue benefit. Thus, previous interpretations may be taken with some caution at present.

Only pre-cueing but no retro-cueing effects emerge with masked arrow cues

The impact of masked stimulation on cognitive control processes is investigated with much interest. In many cases, masked stimulation suffices to initiate and employ control processes. Shifts of attention either happen in the external environment or internally, for example, in working memory. In the former, even masked cues (i.e., cues that are presented for a period too short to allow strategic use) were shown efficient for shifting attention to particular locations in pre-cue paradigms. Internal attention shifting can be investigated using retro-cues: long after encoding, a valid cue indicates the location to-be-tested via change detection, and this improves performance (retro-cue effect). In the present experiment, participants performed in both a pre- and a retro-cue task with masked and normally presented cues. While the masked cues benefitted performance in the pre-cue task, they did not in the retro-cue task. These results inform about limits of masked stimulation.

L. Olivers C. The time course of protecting a visual memory representation from perceptual interference

Cueing a remembered item during the delay of a visual memory task leads to enhanced recall of the cued item compared to when an item is not cued. This cueing benefit has been proposed to reflect attention within visual memory being shifted from a distributed mode to a focused mode, thus protecting the cued item against perceptual interference. Here we investigated the dynamics of building up this mnemonic protection against visual interference by systematically varying the stimulus onset asynchrony (SOA) between cue onset and a subsequent visual mask in an orientation memory task. Experiment 1 showed that a cue counteracted the deteriorating effect of pattern masks. Experiment 2 demonstrated that building up this protection is a continuous process that is completed in approximately half a second after cue onset. The similarities between shifting attention in perceptual and remembered space are discussed.

The role of effect grouping in free-choice response selection

Which motor actions are preferred to replace an initially planned but momentary not executable action? Previous research (Khan, Mourton, Buckolz, Adams, & Hayes, 2010, Acta Psychologica) suggests that anatomical constraints seem to be a major determinant for such choices: For example, participants more frequently chose to respond with the finger homologous to the prepared one. We argue that in this case finger homology is confounded with action effect similarity, and action effects have been ascribed a crucial role in action selection. We report two experiments. Experiment 1 replicated the results obtained by Khan et al. In Experiment 2, we introduced visual action effects in the paradigm. Results from this experiment clearly point to a role of effect similarity in addition to mere finger homology status for the choice frequency effect.