Independent Effects of Eye and Hand Movements on Visual Working Memory

Representation and computation in visual working memory

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

Prospection of Potential Actions during Visual Working Memory Starts Early, Is Flexible, and Predicts Behavior

For visual working memory to serve upcoming behavior, it is crucial that we prepare for the potential use of working-memory contents ahead of time. Recent studies have demonstrated how the prospection and planning for an upcoming manual action starts early after visual encoding, and occurs alongside visual retention. Here, we address whether such "output planning" in visual working memory flexibly adapts to different visual-motor mappings, and occurs even when an upcoming action will only potentially become relevant for behavior. Human participants (female and male) performed a visual-motor working memory task in which they remembered one or two colored oriented bars for later (potential) use. We linked, and counterbalanced, the tilt of the visual items to specific manual responses. This allowed us to track planning of upcoming behavior through contralateral attenuation of β band activity, a canonical motor-cortical EEG signature of manual-action planning. The results revealed how action encoding and subsequent planning alongside visual working memory (1) reflect anticipated task demands rather than specific visual-motor mappings, (2) occur even for actions that will only potentially become relevant for behavior, and (3) are associated with faster performance for the encoded item, at the expense of performance to other working-memory content. This reveals how the potential prospective use of visual working memory content is flexibly planned early on, with consequences for the speed of memory-guided behavior.SIGNIFICANCE STATEMENT It is increasingly studied how visual working memory helps us to prepare for the future, in addition to how it helps us to hold onto the past. Recent studies have demonstrated that the planning of prospective actions occurs alongside encoding and retention in working memory. We show that such early "output planning" flexibly adapts to varying visual-motor mappings, occurs both for certain and potential actions, and predicts ensuing working-memory guided behavior. These results highlight the flexible and future-oriented nature of visual working memory, and provide insight into the neural basis of the anticipatory dynamics that translate visual representations into adaptive upcoming behavior.

Turning Attention Inside Out: How Working Memory Serves Behavior

Flexible behavior requires guidance not only by sensations that are available immediately but also by relevant mental contents carried forward through working memory. Therefore, selective-attention functions that modulate the contents of working memory to guide behavior (inside-out) are just as important as those operating on sensory signals to generate internal contents (outside-in). We review the burgeoning literature on selective attention in the inside-out direction and underscore its functional, flexible, and future-focused nature. We discuss in turn the purpose (why), targets (what), sources (when), and mechanisms (how) of selective attention inside working memory, using visual working memory as a model. We show how the study of internal selective attention brings new insights concerning the core cognitive processes of attention and working memory and how considering selective attention and working memory together paves the way for a rich and integrated understanding of how mind serves behavior.

Visual working memory representations bias attention more when they are the target of an action plan

Attention has frequently been regarded as an emergent property of linking sensory representations to action plans. It has recently been proposed that similar mechanisms may operate within visual working memory (VWM), such that linking an object in VWM to an action plan strengthens its sensory memory representation, which then expresses as an attentional bias. Here we directly tested this hypothesis by comparing attentional biases induced by VWM representations which were the target of a future action, to those induced by VWM representations that were equally task-relevant, but not the direct target of action. We predicted that the first condition would result in a more prioritized memory state and hence stronger attentional biases. Specifically, participants memorized a geometric shape for a subsequent memory test. At test, in case of a match, participants either had to perform a grip movement on the matching object (action condition), or perform the same movement, but on an unrelated object (control condition). To assess any attentional biases, during the delay period between memorandum and test, participants performed a visual selection task in which either the target was surrounded by the memorized shape (congruent trials) or a distractor (incongruent trials). Eye movements were measured as a proxy for attentional priority. We found a significant interaction for saccade latencies between action condition and shape congruency, reflecting more pronounced VWM-based attentional biases in the action condition. Our results are consistent with the idea that action plans prioritize sensory representations in VWM.

Motor engagement enhances incidental memory for task-irrelevant items

Actions shape what we see and memorize. A previous study suggested the interaction between motor and memory systems by showing that memory encoding for task-irrelevant items was enhanced when presented with motor-response cues. However, in the studies on the attentional boost effect, it has been revealed that detection of the target stimulus can lead to memory enhancement without requiring overt action. Thus, the direct link between the action and memory remains unclear. To exclude the effect of the target detection process as a potential confounder, this study assessed the benefit of action for memory by separating items from the response cue in time. In our pre-registered online experiment (N = 142), participants responded to visual Go cues by pressing a key (i.e., motor task) or counting (i.e., motor-neutral cognitive task) while ignoring No-go cues. In each trial, two task-irrelevant images were sequentially presented after the cue disappearance. After encoding the Go/No-go tasks, participants performed a surprise recognition memory test for those images. Importantly, we quantified the impact of overt execution of the action by comparing memories with and without motor response and the impact of covert motor processes (e.g., preparation and planning of action) by comparing memory between the motor and cognitive tasks. The results showed no memory differences between Go and No-go trials in the motor task. This means that the execution itself was not critical for memory enhancement. However, the memory performance in the motor No-go trials was higher than that in the cognitive No-go trials, only for the items presented away from the cues in time. Therefore, engaging the motor task itself could increase incidental memory for the task-irrelevant items compared to a passive viewing situation. We added empirical evidence on the online interaction between action and memory encoding. These memory advantages could be especially brought in action preparation and planning. We believe this fact may expand our present understanding of everyday memory, such as active learning.

Encoding specificity instead of online integration of real-world spatial regularities for objects in working memory

Most objects show high degrees of spatial regularity (e.g. beach umbrellas appear above, not under, beach chairs). The spatial regularities of real-world objects benefit visual working memory (VWM), but the mechanisms behind this spatial regularity effect remain unclear. The "encoding specificity" hypothesis suggests that spatial regularity will enhance the visual encoding process but will not facilitate the integration of information online during VWM maintenance. The "perception-alike" hypothesis suggests that spatial regularity will function in both visual encoding and online integration during VWM maintenance. We investigated whether VWM integrates sequentially presented real-world objects by focusing on the existence of the spatial regularity effect. Throughout five experiments, we manipulated the presentation (simultaneous vs. sequential) and regularity (with vs. without regularity) of memory arrays among pairs of real-world objects. The spatial regularity of memory objects presented simultaneously, but not sequentially, improved VWM performance. We also examined whether memory load, verbal suppression and masking, and memory array duration hindered the spatial regularity effect in sequential presentation. We found a stable absence of the spatial regularity effect, suggesting that the participants were unable to integrate real-world objects based on spatial regularities online. Our results support the encoding specificity hypothesis, wherein the spatial regularity of real-world objects can enhance the efficiency of VWM encoding, but VWM cannot exploit spatial regularity to help organize sampled sequential information into meaningful integrations.

Planning the Potential Future during Multi-item Visual Working Memory

Working memory allows us to retain visual information to guide upcoming future behavior. In line with this future-oriented purpose of working memory, recent studies have shown that action planning occurs during encoding and retention of a single visual item, for which the upcoming action is certain. We asked whether and how this extends to multi-item visual working memory, when visual representations serve the potential future. Human participants performed a visual working-memory task with a memory-load manipulation (one/two/four items) and a delayed orientation-reproduction report (of one item). We measured EEG to track 15- to 25-Hz beta activity in electrodes contralateral to the required response hand-a canonical marker of action planning. We show an attenuation of beta activity, not only in Load 1 (with one certain future action) but also in Load 2 (with two potential future actions), compared with Load 4 (with low prospective-action certainty). Moreover, in Load 2, potential action planning occurs regardless whether both visual items afford similar or dissimilar manual responses, and it predicts the speed of ensuing memory-guided behavior. This shows that potential action planning occurs during multi-item visual working memory and brings the perspective that working memory helps us prepare for the potential future.

Less motor (re-)planning requires fewer working memory resources

In the current study, we asked if less motor re-planning requires fewer resources in working memory (WM). To this end, participants executed a spatial WM task in parallel to different sequential motor tasks: (1) a randomised task with a high amount of motor re-planning and (2) an ordered task with a lower amount of motor re-planning. Recall performance in the spatial WM task was measured as the dependent variable. Hand posture was used to calculate the percentage of motor re-planning and, thus, to validate the experimental manipulation. The percentage of motor re-planning was lower in the ordered task, while spatial WM performance was higher. This indicates that WM resources depleted by the motor task scale with the amount of motor re-planning. Results further showed a significant recency effect (i.e. better recall of late items) in the spatial WM task. As previous studies found that recency effects in a verbal WM task are disrupted by a concurrent motor task, the presence of recency in the current study indicates a differential interference of a concurrent motor task on verbal vs. spatial recall, which has important implications for several current models of WM.

Does the combination of physical activity and attention training affect the motor skills and cognitive activities of individuals with mild intellectual disability?

Individuals with mild intellectual disability (MID) were worse than their peers who typically develop in motor skills and attention-demanding assignments. In this study, effect of a 14-week physical activity and attention training practise on the motor skills, visual retention, perception and attention levels of students with MID were analysed. Twenty-two individuals between 7 and 14 ages participated voluntarily. Activities based on developing attention skills and physical activities enhancing fine-gross motor skills (40 + 60 min./2 days/14 weeks) were given to the experimental group. d2 Test of Attention, Benton Visual Retention and Bruininks-Oseretsky Test of Motor Proficiency-2nd version tests were used as data collection tools. There was significant difference in terms of total number of items processed, commissions, raw score of errors, total number of items minus error scores, concentration performance, Benton visual retention test and perception, fine motor skill precision, fine motor skill integration, manual dexterity and upper-limb coordination values regarding group and time dependant (p< .05). However, there was no significant difference in omissions and fluctuation rate values (p> .05). As a result; it is determined that the combination of physical activity and attention training practises features a positive effect on visual retention, perception, attention and motor skill levels of students with MID.

Shifting attention in visuospatial short-term memory does not require oculomotor planning: Insight from congenital gaze paralysis

Attention allows pieces of information stored in visuospatial short-term memory (VSSTM) to be selectively processed. Previous studies showed that shifts of attention in VSSTM in response to a retro-cue are accompanied by eye movements in the direction of the position of the memorized item although there is nothing left to look at. This finding raises the possibility that shifts of attention in VSSTM are underpinned by mechanisms originally involved in the planning and control of eye movements. To explore this possibility, we investigated the ability of an individual with congenital horizontal gaze paralysis (HGP2) to shift her attention horizontally or vertically toward a memorized item within VSSTM using a retro-cue paradigm. As efficient oculomotor programming is not innate but requires some trial and error learning and adaptation to develop, congenital paralysis prevents this development. Consequently, if shifts of attention in VSSTM rely on the same mechanisms as those supporting the programming of eye movements, then horizontal congenital gaze paralysis should necessarily prevent typical retro-cueing effect in the paralyzed axis. At odds with this prediction, HGP2 showed a typical retro-cueing effect in her paralyzed axis. This original finding indicates that selecting an item within VSSTM does not depend on the ability to program a saccade.

Keeping it real: Looking beyond capacity limits in visual cognition

Research within visual cognition has made tremendous strides in uncovering the basic operating characteristics of the visual system by reducing the complexity of natural vision to artificial but well-controlled experimental tasks and stimuli. This reductionist approach has for example been used to assess the basic limitations of visual attention, visual working memory (VWM) capacity, and the fidelity of visual long-term memory (VLTM). The assessment of these limits is usually made in a pure sense, irrespective of goals, actions, and priors. While it is important to map out the bottlenecks our visual system faces, we focus here on selected examples of how such limitations can be overcome. Recent findings suggest that during more natural tasks, capacity may be higher than reductionist research suggests and that separable systems subserve different actions, such as reaching and looking, which might provide important insights about how pure attentional or memory limitations could be circumvented. We also review evidence suggesting that the closer we get to naturalistic behavior, the more we encounter implicit learning mechanisms that operate "for free" and "on the fly." These mechanisms provide a surprisingly rich visual experience, which can support capacity-limited systems. We speculate whether natural tasks may yield different estimates of the limitations of VWM, VLTM, and attention, and propose that capacity measurements should also pass the real-world test within naturalistic frameworks. Our review highlights various approaches for this and suggests that our understanding of visual cognition will benefit from incorporating the complexities of real-world cognition in experimental approaches.

Simple action planning can affect attentional allocation in subsequent visual search

Previous research has shown that mental representations of actions can influence performance on relevant tasks or dimensions even when there is no overt execution of the action. In this study, we examined whether cognitive processes prior to the physical execution of an action can elicit attentional bias towards irrelevant tasks or dimensions of that action. Participants performed two independent tasks-an action task and a search task-where they were instructed to plan an action and execute the action following the visual search task. We found that the same features of the object were prioritized in the subsequent search task when participants had planned an action response on the object in comparison to when they had not. This effect occurred even when the feature was irrelevant to the tasks or required action. Furthermore, the effect of action planning without physical response was found to be comparable to the effect of physical response. These results suggest that planning of a simple action can induce attentional bias to irrelevant features of objects even without physical action.

Output planning at the input stage in visual working memory

Working memory serves as the buffer between past sensations and future behavior, making it vital to understand not only how we encode and retain sensory information in memory but also how we plan for its upcoming use. We ask when prospective action goals emerge alongside the encoding and retention of visual information in working memory. We show that prospective action plans do not emerge gradually during memory delays but are brought into memory early, in tandem with sensory encoding. This action encoding (i) precedes a second stage of action preparation that adapts to the time of expected memory utilization, (ii) occurs even ahead of an intervening motor task, and (iii) predicts visual memory-guided behavior several seconds later. By bringing prospective action plans into working memory at an early stage, the brain creates a dual (visual-motor) memory code that can make memories more effective and robust for serving ensuing behavior.

Bold moves: Inevitable saccadic selection in visual short-term memory

Selection for visual short-term memory (vstm) provides a basis for many cognitive functions. Saccadic eye movements sway this selection in favor of stimuli previously seen at locations congruent with their target. In three experiments, we provide converging evidence that this saccadic selection is implemented as a fundamental, inevitable selection process, rather than a top-down strategy. In particular, benefits for congruent over incongruent items were largely constant across set sizes ranging from two to eight items (Experiment 1), showing that saccadic selection imposes priorities on vstm irrespective of memory load and is effective even when only few representations need to be maintained. Moreover, a decrement in performance for incongruent items occurred reliably, whether the congruent location contained a task-relevant item or an irrelevant noise patch (Experiment 2). Finally, saccadic selection was immune to a strong manipulation of the observer's attentional priorities (Experiment 3). Given the prevalence of saccades in natural vision, our results demonstrate a fundamental and ecologically relevant selection mechanism for vstm: Saccades systematically eliminate information seen at non-target locations, while information at the saccade target remains available to recall. This simple heuristic is effective in the absence of informative cues and may incapacitate voluntary selection mechanisms that are incongruent with ongoing movement plans.

Visual working memory and action: Functional links and bi-directional influences

Working memory bridges perception to action over extended delays, enabling flexible goal-directed behaviour. To date, studies of visual working memory – concerned with detailed visual representations such as shape and colour – have considered visual memory predominantly in the context of visual task demands, such as visual identification and search. Another key purpose of visual working memory is to directly inform and guide upcoming actions. Taking this as a starting point, I review emerging evidence for the pervasive bi-directional links between visual working memory and (planned) action, and discuss these links from the perspective of their common goal of enabling flexible and precise behaviour.

The interdependence of attention, working memory and gaze control: behavior and neural circuitry

Visual attention, visual working memory, and gaze control are basic functions that all select a subset of visual input to guide immediate or subsequent behavior. In this review, we focus on the relationship between these three functions and describe evidence, both at the behavioral and neural circuit levels that they are heavily interdependent. We start with the demonstration that gaze control - or saccade preparation in particular - leads to spatial attention. Next, we show that spatial attention and working memory interact at the behavioral level and rely on a common set of neural mechanisms. Next, we discuss the evidence that gaze control mechanisms are involved in spatial working memory. Lastly, we highlight the links between gaze control and non-spatial memory.

Task-Irrelevant Features in Visual Working Memory Influence Covert Attention: Evidence from a Partial Report Task

Selecting a target based on a representation in visual working memory (VWM) affords biasing covert attention towards objects with memory-matching features. Recently, we showed that even task-irrelevant features of a VWM template bias attention. Specifically, when participants had to saccade to a cued shape, distractors sharing the cue's search-irrelevant color captured the eyes. While a saccade always aims at one target location, multiple locations can be attended covertly. Here, we investigated whether covert attention is captured similarly as the eyes. In our partial report task, each trial started with a shape-defined search cue, followed by a fixation cross. Next, two colored shapes, each including a letter, appeared left and right from fixation, followed by masks. The letter inside that shape matching the preceding cue had to be reported. In Experiment 1, either target, distractor, both, or no object matched the cue's irrelevant color. Target-letter reports were most frequent in target-match trials and least frequent in distractor-match trials. Irrelevant cue and target color never matched in Experiment 2. Still, participants reported the distractor more often to the target's disadvantage, when cue and distractor color matched. Thus, irrelevant features of a VWM template can influence covert attention in an involuntarily object-based manner when searching for trial-wise varying targets.