Planning the Potential Future during Multi-item Visual Working Memory

Memory load influences our preparedness to act on visual representations in working memory without affecting their accessibility

It is well established that when we hold more content in working memory, we are slower to act upon part of that content when it becomes relevant for behavior. Here, we asked whether this load-related slowing is due to slower access to the sensory representations held in working memory (as predicted by serial working-memory search), or by a reduced preparedness to act upon those sensory representations once accessed. To address this, we designed a visual-motor working-memory task in which participants memorized the orientation of two or four colored bars, of which one was cued for reproduction. We independently tracked EEG markers associated with the selection of visual (cued item location) and motor (relevant manual action) information from the EEG time-frequency signal, and compared their latencies as a function of memory load. We confirm slower memory-guided behavior with higher working-memory load and show that this is associated with delayed motor selection. In contrast, we find no evidence for a concomitant delay in the latency of visual selection. Moreover, we show that variability in decision times within each memory-load condition is associated with corresponding changes in the latency of motor, but not visual selection. These results reveal how memory load affects our preparedness to act on sensory representations in working memory, while leaving sensory access itself unaffected. This posits action readiness as a key factor that shapes the speed of memory-guided behavior and that underlies delayed responding with higher working-memory load.

Action Planning Renders Objects in Working Memory More Attentionally Salient

A rapidly growing body of work suggests that visual working memory (VWM) is fundamentally action oriented. Consistent with this, we recently showed that attention is more strongly biased by VWM representations of objects when we plan to act on those objects in the future. Using EEG and eye tracking, here, we investigated neurophysiological correlates of the interactions between VWM and action. Participants (n = 36) memorized a shape for a subsequent VWM test. At test, a probe was presented along with a secondary object. In the action condition, participants gripped the actual probe if it matched the memorized shape, whereas in the control condition, they gripped the secondary object. Crucially, during the VWM delay, participants engaged in a visual selection task, in which they located a target as fast as possible. The memorized shape could either encircle the target (congruent trials) or a distractor (incongruent trials). Replicating previous findings, we found that eye gaze was biased toward the VWM-matching shape and, importantly, more so when the shape was directly associated with an action plan. Moreover, the ERP results revealed that during the selection task, future action-relevant VWM-matching shapes elicited (1) a stronger Ppc (posterior positivity contralateral), signaling greater attentional saliency; (2) an earlier PD (distractor positivity) component, suggesting faster suppression; (3) a larger inverse (i.e., positive) sustained posterior contralateral negativity in incongruent trials, consistent with stronger suppression of action-associated distractors; and (4) an enhanced response-locked positivity over left motor regions, possibly indicating enhanced inhibition of the response associated with the memorized item during the interim task. Overall, these results suggest that action planning renders objects in VWM more attentionally salient, supporting the notion of selection-for-action in working memory.

Neural Signatures of Competition between Voluntary and Involuntary Influences over the Focus of Attention in Visual Working Memory

Adaptive behavior relies on the selection and prioritization of relevant sensory inputs from the external environment as well as from among internal sensory representations held in working memory. Recent behavioral evidence suggests that the classic distinction between voluntary (goal-driven) and involuntary (stimulus-driven) influences over attentional allocation also applies to the selection of internal representations held in working memory. In the current EEG study, we set out to investigate the neural dynamics associated with the competition between voluntary and involuntary control over the focus of attention in visual working memory. We show that when voluntary and involuntary factors compete for the internal focus of attention, prioritization of the appropriate item is delayed-as reflected both in delayed gaze biases that track internal selection and in delayed neural beta (15-25 Hz) dynamics that track the planning for the upcoming memory-guided manual action. We further show how this competition is paralleled-possibly resolved-by an increase in frontal midline theta (4-8 Hz) activity that, moreover, predicts the speed of ensuing memory-guided behavior. Finally, because theta increased following retrocues that effectively reduced working-memory load, our data unveil how frontal theta activity during internal attentional focusing tracks demands on cognitive control over and above working-memory load. Together, these data yield new insight into the neural dynamics that govern the focus of attention in visual working memory, and disentangle the contributions of frontal midline theta activity to the processes of control versus retention in working memory.

Mapping visual working memory models to a theoretical framework

The body of research on visual working memory (VWM)-the system often described as a limited memory store of visual information in service of ongoing tasks-is growing rapidly. The discovery of numerous related phenomena, and the many subtly different definitions of working memory, signify a challenge to maintain a coherent theoretical framework to discuss concepts, compare models and design studies. A lack of robust theory development has been a noteworthy concern in the psychological sciences, thought to be a precursor to the reproducibility crisis (Oberauer & Lewandowsky, Psychonomic Bulletin & Review, 26, 1596-1618, 2019). I review the theoretical landscape of the VWM field by examining two prominent debates-whether VWM is object-based or feature-based, and whether discrete-slots or variable-precision best describe VWM limits. I share my concerns about the dualistic nature of these debates and the lack of clear model specification that prevents fully determined empirical tests. In hopes of promoting theory development, I provide a working theory map by using the broadly encompassing memory for latent representations model (Hedayati et al., Nature Human Behaviour, 6, 5, 2022) as a scaffold for relevant phenomena and current theories. I illustrate how opposing viewpoints can be brought into accordance, situating leading models of VWM to better identify their differences and improve their comparison. The hope is that the theory map will help VWM researchers get on the same page-clarifying hidden intuitions and aligning varying definitions-and become a useful device for meaningful discussions, development of models, and definitive empirical tests of theories.

Be prepared for interruptions: EEG correlates of anticipation when dealing with task interruptions and the role of aging

Dealing with task interruptions requires the flexible use of working memory and attentional control mechanisms, which are prone to age-related changes. We investigated effects of age on dealing with task interruptions and potential advantages of anticipating an interruption using EEG and a retrospective cueing (retro-cue) paradigm. Thirty-two young (18-30 years) and 28 older (55-70 years) participants performed a visual working memory task, where they had to report the orientation of a target following a retro-cue. Within blocks of 10 trials, they were always, never, or randomly interrupted with an arithmetic task before the onset of the retro-cue. The interruption-induced decline in primary task performance was more pronounced in older participants, while only these benefited from anticipation. The EEG analysis revealed reduced theta and alpha/beta response to the retro-cue following interruptions, especially for the older participants. In both groups, anticipated interruptions were associated with increased theta and alpha/beta power prior and during the interruption, and stronger beta suppression to the retro-cue. The results indicate that interruptions impede the refocusing of attention on the task-relevant representation of the primary task, especially in older people, while anticipation facilitates preparation for the interruption task and resumption of the primary task.

Prospection of Potential Actions during Visual Working Memory Starts Early, Is Flexible, and Predicts Behavior

For visual working memory to serve upcoming behavior, it is crucial that we prepare for the potential use of working-memory contents ahead of time. Recent studies have demonstrated how the prospection and planning for an upcoming manual action starts early after visual encoding, and occurs alongside visual retention. Here, we address whether such "output planning" in visual working memory flexibly adapts to different visual-motor mappings, and occurs even when an upcoming action will only potentially become relevant for behavior. Human participants (female and male) performed a visual-motor working memory task in which they remembered one or two colored oriented bars for later (potential) use. We linked, and counterbalanced, the tilt of the visual items to specific manual responses. This allowed us to track planning of upcoming behavior through contralateral attenuation of β band activity, a canonical motor-cortical EEG signature of manual-action planning. The results revealed how action encoding and subsequent planning alongside visual working memory (1) reflect anticipated task demands rather than specific visual-motor mappings, (2) occur even for actions that will only potentially become relevant for behavior, and (3) are associated with faster performance for the encoded item, at the expense of performance to other working-memory content. This reveals how the potential prospective use of visual working memory content is flexibly planned early on, with consequences for the speed of memory-guided behavior.SIGNIFICANCE STATEMENT It is increasingly studied how visual working memory helps us to prepare for the future, in addition to how it helps us to hold onto the past. Recent studies have demonstrated that the planning of prospective actions occurs alongside encoding and retention in working memory. We show that such early "output planning" flexibly adapts to varying visual-motor mappings, occurs both for certain and potential actions, and predicts ensuing working-memory guided behavior. These results highlight the flexible and future-oriented nature of visual working memory, and provide insight into the neural basis of the anticipatory dynamics that translate visual representations into adaptive upcoming behavior.

Capacity and selection in immersive visual working memory following naturalistic object disappearance

Visual working memory-holding past visual information in mind for upcoming behavior-is commonly studied following the abrupt removal of visual objects from static two-dimensional (2D) displays. In everyday life, visual objects do not typically vanish from the environment in front of us. Rather, visual objects tend to enter working memory following self or object motion: disappearing from view gradually and changing the spatial relation between memoranda and observer. Here, we used virtual reality (VR) to investigate whether two classic findings from visual working memory research-a capacity of around three objects and the reliance on space for object selection-generalize to more naturalistic modes of object disappearance. Our static reference condition mimicked traditional laboratory tasks whereby visual objects were held static in front of the participant and removed from view abruptly. In our critical flow condition, the same visual objects flowed by participants, disappearing from view gradually and behind the observer. We considered visual working memory performance and capacity, as well as space-based mnemonic selection, indexed by directional biases in gaze. Despite vastly distinct modes of object disappearance and altered spatial relations between memoranda and observer, we found comparable capacity and comparable gaze signatures of space-based mnemonic selection. This finding reveals how classic findings from visual working memory research generalize to immersive situations with more naturalistic modes of object disappearance and with dynamic spatial relations between memoranda and observer.

Genuine beta bursts in human working memory: controlling for the influence of lower-frequency rhythms

Human working memory is associated with significant modulations in oscillatory brain activity. However, the functional role of brain rhythms at different frequencies is still debated. Modulations in the beta frequency range (15-40 Hz) are especially difficult to interpret because they could be artifactually produced by (more prominent) oscillations in lower frequencies that show non-sinusoidal properties. In this study, we investigate beta oscillations during working memory while controlling for the possible influence of lower frequency rhythms. We collected electroencephalography (EEG) data in 31 participants who performed a spatial working-memory task with two levels of cognitive load. In order to rule out the possibility that observed beta activity was affected by non-sinusoidalities of lower frequency rhythms, we developed an algorithm that detects transient beta oscillations that do not coincide with more prominent lower frequency rhythms in time and space. Using this algorithm, we show that the amplitude and duration of beta bursts decrease with memory load and during memory manipulation, while their peak frequency and rate increase. In addition, interindividual differences in performance were significantly associated with beta burst rates. Together, our results show that beta rhythms are functionally modulated during working memory and that these changes cannot be attributed to lower frequency rhythms with non-sinusoidal properties.

Turning Attention Inside Out: How Working Memory Serves Behavior

Flexible behavior requires guidance not only by sensations that are available immediately but also by relevant mental contents carried forward through working memory. Therefore, selective-attention functions that modulate the contents of working memory to guide behavior (inside-out) are just as important as those operating on sensory signals to generate internal contents (outside-in). We review the burgeoning literature on selective attention in the inside-out direction and underscore its functional, flexible, and future-focused nature. We discuss in turn the purpose (why), targets (what), sources (when), and mechanisms (how) of selective attention inside working memory, using visual working memory as a model. We show how the study of internal selective attention brings new insights concerning the core cognitive processes of attention and working memory and how considering selective attention and working memory together paves the way for a rich and integrated understanding of how mind serves behavior.

Preparing for the unknown: How working memory provides a link between perception and anticipated action

What mechanisms underlie the transfer of a working memory representation into a higher-level code for guiding future actions? Electrophysiological correlates of attentional selection and motor preparation processes within working memory were investigated in two retrospective cuing tasks. In the first experiment, participants stored the orientation and location of a grating. Subsequent feature cues (selective vs. neutral) indicated which feature would be the target for later report. The oscillatory response in the mu and beta frequency range with an estimated source in the sensorimotor cortex contralateral to the responding hand was used as correlate of motor preparation. Mu/beta suppression was stronger following the selective feature cues compared to the neutral cue, demonstrating that purely feature-based selection is sufficient to form a prospective motor plan. In the second experiment, another retrospective cue was included to study whether knowledge of the task at hand is necessary to initiate motor preparation. Following the feature cue, participants were cued to either compare the stored feature(s) to a probe stimulus (recognition task) or to adjust the memory probe to match the target feature (continuous report task). An analogous suppression of mu oscillations was observed following a selective feature cue, even ahead of task specification. Further, a subsequent selective task cue again elicited a mu/beta suppression, which was stronger after a continuous report task cue. This indicates that working memory is able to flexibly store different types of information in higher-level mental codes to provide optimal prerequisites for all required action possibilities.