Turning Attention Inside Out: How Working Memory Serves Behavior

Memory-based predictions prime perceptual judgments across head turns in immersive, real-world scenes

Each view of our environment captures only a subset of our immersive surroundings. Yet, our visual experience feels seamless. A puzzle for human neuroscience is to determine what cognitive mechanisms enable us to overcome our limited field of view and efficiently anticipate new views as we sample our visual surroundings. Here, we tested whether memory-based predictions of upcoming scene views facilitate efficient perceptual judgments across head turns. We tested this hypothesis using immersive, head-mounted virtual reality (VR). After learning a set of immersive real-world environments, participants (n = 101 across 4 experiments) were briefly primed with a single view from a studied environment and then turned left or right to make a perceptual judgment about an adjacent scene view. We found that participants' perceptual judgments were faster when they were primed with images from the same (vs. neutral or different) environments. Importantly, priming required memory: it only occurred in learned (vs. novel) environments, where the link between adjacent scene views was known. Further, consistent with a role in supporting active vision, priming only occurred in the direction of planned head turns and only benefited judgments for scene views presented in their learned spatiotopic positions. Taken together, we propose that memory-based predictions facilitate rapid perception across large-scale visual actions, such as head and body movements, and may be critical for efficient behavior in complex immersive environments.

Coordinated social interactions are supported by integrated neural representations

Joint actions are defined as coordinated interactions of two or more agents towards a shared goal, often requiring different and complementary individual contributions. However, how humans can successfully act together without the interfering effects of observing incongruent movements is still largely unknown. It has been proposed that interpersonal predictive processes are at play to allow the formation of a Dyadic Motor Plan, encompassing both agents' shares. Yet, direct empirical support for such an integrated motor plan is still limited. In this study, we aimed at testing the properties of these anticipated representations. We collected EEG data while human participants (N = 36; 27 females) drew shapes simultaneously to a virtual partner, in two social contexts: either they had to synchronize and act jointly, or they performed the movements alongside, but independently. We adopted a multivariate approach to show that the social context influenced how the upcoming action of the partner is anticipated during the interval preceding the movement. We found evidence that acting jointly induces an encoding of the partner's action that is strongly intertwined with the participant's action, supporting the hypothesis of an integrative motor plan in joint but not in parallel actions.

Predicting the effectiveness of binaural beats on working memory

Working memory is vital for short-term information processing. Binaural beats can enhance working memory by improving attention and memory consolidation through neural synchronization. However, individual differences in cognitive and neuronal functioning affect effectiveness of binaural beats, necessitating personalized approaches. This study aimed to develop a machine learning model to predict binaural beats's effectiveness on working memory using electroencephalography. Sixty healthy participants underwent a 5-min electroencephalography recording, an initial working memory evaluation, 15 min of binaural beats stimulation, and a subsequent working memory evaluation using digit span tests of increasing difficulty. Recall accuracy and response times were measured. Differential scores from pre-evaluation and post-evaluation labeled participants as active or inactive to binaural beats stimulation. electroencephalography data, recorded using 14 electrodes, provided brain activity estimates across theta, alpha, beta, and gamma frequency bands, resulting in 56 features (14 channels × 4 bands) for the machine learning model. Several classifiers were tested to identify the most effective model. The weighted K-nearest neighbors model achieved the highest accuracy (90.0%) and area under the receiver operating characteristic curve (92.24%). Frontal and parietal electroencephalography channels in theta and alpha bands were crucial for classification. This study's findings offer significant clinical insights, enabling informed interventions and preventing resource inefficiency.

Action similarity warps visual feature space in working memory

Visual working memory (VWM) retains representations of past visual information for future action. Yet to date, most studies have approached VWM as just serving perception beyond the immediate. Whether and how prospective actions shape information in VWM remains largely unknown, in part because typical experimental setups limit behavior to simple button presses. In two experiments (one preregistered), using a novel interactive VWM task, we show that the similarity of the actions that we intend to perform on memory items adaptively distorts their representation. Participants memorized the orientation of two bars, after which they were informed as to which manual actions they should reproduce these orientations with in a memory recall test. We observed that perceptually similar items were remembered as more distinct when paired with different action plans versus the same action plan. A control experiment showed that this action-induced effect reflects a true change in the visual representation rather than a motor bias. These findings demonstrate that VWM representations are flexibly adapted to guide specific overt actions and provide evidence that action plans can retrospectively warp sensory feature space in VWM.

The ventral midline thalamus and long-term memory: What consolidation, what retrieval, what plasticity in rodents?

The ventral midline thalamus, including the reuniens and rhomboid (ReRh) nuclei, connects bidirectionally with the medial prefrontal cortex (mPFC) and hippocampus (Hip), both essential for memory processes. This review compiles and discusses studies on a role for the ReRh nuclei in the system consolidation of memories, also considering their potentially limited participation in memory retrieval or early phases of consolidation. It also examines scientific literature on short- and long-term plasticity in ReRh-mPFC and ReRh-Hip connections, emphasizing plasticity's importance in understanding these nuclei's role in memory. The idea that the two nuclei are at the crossroads of information exchange between the mPFC and the Hip is not new, but the relationship between this status and the plasticity of their connections remains elusive. Since this perspective is relatively recent, our concluding section suggests conceptual and practical avenues for future research, aiming perhaps to bring more order to the apparently multi-functional implication of the ventral midline thalamus in cognition.

A domain-general process for theta-rhythmic sampling of either environmental information or internally stored information

Many everyday tasks, such as shopping for groceries, require the sampling of both environmental information and internally stored information. Selective attention involves the preferential processing and sampling of behaviorally important information from the external environment, while working memory involves the preferential processing and sampling of behaviorally important, internally stored information. These essential cognitive processes share neural resources within a large-scale network that includes frontal, parietal, and sensory cortices, and these shared neural resources can lead to between-domain interactions. Previous research has linked external sampling during selective attention and internal sampling during working memory to theta-rhythmic (3-8 Hz) neural activity in higher-order (e.g., frontal cortices) and sensory regions (e.g., visual cortices). Such theta-rhythmic neural activity might help to resolve the competition for shared neural resources by isolating neural activity associated with different functions over time. Here, we used EEG and a dual-task design (i.e., a task that required both external and internal sampling) to directly compare (i) theta-dependent fluctuations in behavioral performance during external sampling with (ii) theta-dependent fluctuations in behavioral performance during internal sampling. Our findings are consistent with a domain-general, theta-rhythmic process for sampling either external information or internal information. We further demonstrate that interactions between external and internal information-specifically, when to-be-detected information matches to-be-remembered information-are not dependent on theta-band activity (i.e., theta phase). Given that these theta-independent 'match effects' occur during early processing stages (peaking at 75 ms), we propose that theta-rhythmic sampling modulates external and internal information during later processing stages.

Neural dynamics of shifting attention between perception and working-memory contents

In everyday tasks, our focus of attention shifts seamlessly between contents in the sensory environment and internal memory representations. Yet, research has mainly considered external and internal attention in isolation. We used magnetoencephalography to compare the neural dynamics of shifting attention to visual contents within vs. between the external and internal domains. Participants performed a combined perception and working-memory task in which two sequential cues guided attention to upcoming (external) or memorized (internal) sensory information. Critically, the second cue could redirect attention to visual content within the same or alternative domain as the first cue. Multivariate decoding unveiled distinct patterns of human brain activity when shifting attention within vs. between domains. Brain activity distinguishing within- from between-domain shifts was broadly distributed and highly dynamic. Intriguingly, crossing domains did not invoke an additional stage prior to shifting attention. Alpha lateralization, a canonical marker of shifting spatial attention, showed no delay when cues redirected attention to the same vs. alternative domain. Instead, evidence suggested that neural states associated with a given domain linger and influence subsequent shifts of attention within vs. between domains. Our findings provide critical insights into the neural dynamics that govern attentional shifts between perception and working memory.

Two-dimensional neural geometry underpins hierarchical organization of sequence in human working memory

Working memory (WM) is constructive in nature. Instead of passively retaining information, WM reorganizes complex sequences into hierarchically embedded chunks to overcome capacity limits and facilitate flexible behaviour. Here, to investigate the neural mechanisms underlying hierarchical reorganization in WM, we performed two electroencephalography and one magnetoencephalography experiments, wherein humans retain in WM a temporal sequence of items, that is, syllables, which are organized into chunks, that is, multisyllabic words. We demonstrate that the one-dimensional sequence is represented by two-dimensional neural representational geometry in WM arising from left prefrontal and temporoparietal regions, with separate dimensions encoding item position within a chunk and chunk position in the sequence. Critically, this two-dimensional geometry is observed consistently in different experimental settings, even during tasks not encouraging hierarchical reorganization in WM and correlates with WM behaviour. Overall, these findings strongly support that complex sequences are reorganized into factorized multidimensional neural representational geometry in WM, which also speaks to general structure-based organizational principles given WM's involvement in many cognitive functions.

Attentional shifts bias microsaccade direction but do not cause new microsaccades

Brain circuitry that controls where we look also contributes to attentional selection of visual contents outside current fixation, or content within the spatial layout of working memory. A behavioural manifestation of this contribution comes from modulations in microsaccade direction that accompany spatial attention shifts. Here, we address whether such modulations come about because attention shifts trigger new microsaccades or whether, instead, spatial attention only biases the direction of ongoing microsaccades that would have been made whether or not attention was also shifted. We utilised an internal-selective-attention task that has recently been shown to yield robust spatial microsaccade modulations and compared microsaccade rates following colour retrocues that were carefully matched for sensory input, but differed in whether they invited an attention shift or not. If attention shifts trigger new microsaccades then we would expect more microsaccades following attention-directing cues than following neutral cues. In contrast, we found no evidence for an increase in overall microsaccade rate, despite robust modulations in microsaccade direction. This implies that shifting spatial attention biases the direction of ongoing microsaccades without changing the probability of microsaccade occurrence. These findings help to explain why microsaccades and visual-spatial shifts of attention are often correlated but not obligatorily linked.

Search flavor labels in beverages: An electrophysiological investigation of color-flavor congruency and association strength in visual search

Individuals are apt to link various characteristics of an object or event through different sensory experiences. We conducted two electrophysiological experiments to investigate the effects of color-flavor congruency and association strength on visual search efficiency and the in-depth cognitive mechanisms underlying multisensory processes. Participants were prompted with a flavor label and asked to identify the primed flavor from four beverage bottle images. Experiment 1 focused on color-flavor congruency and noted faster searches for congruent targets than incongruent ones. EEG data exhibited smaller N2, larger P3 and LPC, and increased parietal-occipital midline (POM) alpha power for incongruent targets than congruent ones. Experiment 2 manipulated color-flavor association strength within each flavor. Behavioral findings showed that searches for targets with weak association strength took longer than those with strong association strength. Moreover, time-frequency analysis displayed that the former evoked greater frontal midline (FM) theta power and higher alpha power than the latter. Altogether, our research indicated that (1) color expectations based on prior experience can automatically guide people's attentional selection, (2) the color-flavor congruency and association strength impact the visual search efficiency via distinct pathways, and (3) theta and alpha activities make a pivotal role in unraveling multisensory information processing. These findings shed some light on the intricate cognitive processes involved in crossmodal visual search and the underlying neurocognitive dynamics.

Unravelling the object-based nature of visual working memory: insight from pointers

Visual working memory (VWM) plays a crucial role in temporarily storing and processing visual information, but the nature of stored representations and their interaction with new inputs has long been unclear. The pointer system refers to how VWM links new sensory inputs to stored information using specific cues. This study aimed to investigate whether the pointer system is based on spatial, feature-based, or object-based cues by employing the repetition benefit effect, where memory performance improves with repeated memory items. Across three experiments, we manipulated spatial positions, shapes, and colors as pointer cues to determine how these features affect VWM consolidation and updating. The results showed that while spatial location serves as a strong pointer cue, shape and color features can also effectively reestablish object correspondence in VWM. These findings support the view that the pointer system in VWM is flexible and object-based, utilizing various feature cues to maintain memory continuity. This study provides new insights into how VWM connects new inputs with stored information through the pointer system.

Putting the "Return" Back in the Inhibition of Return Effect in Working Memory

The inhibition of return effect in perception refers to the observation that one is slower to re-attend a location that was attended right before, compared to a location that was not attended right before. Johnson et al. (2013, Psych. Sc., 24, 1104-1112, doi:10.1177/0956797612466414) observed a similar inhibitory effect for an attended item in working memory, which the authors referred to as an inhibition-of-return-like effect. However, testing an inhibition of return effect requires attention to be disengaged from the attended item, before testing whether participants are slower to return to said item. This was assumed but not experimentally manipulated in the paradigm by Johnson and colleagues. In the current study, we investigated whether an inhibition of return effect can be observed in working memory when attention is experimentally disengaged from the attended item before measuring whether responses are slower for the item in question. Participants were indeed slower to respond to a memory probe that matched the item that was attended right before, compared to a memory probe that matched the item that was not attended right before. Thus, our test with more experimental control did result in an inhibition of return effect in working memory.

Internal coupling: Eye behavior coupled to visual imagery

Our eyes do not only respond to visual perception but also to internal cognition involving visual imagery, which can be referred to as internal coupling. This review synthesizes evidence on internal coupling across diverse domains including episodic memory and simulation, visuospatial memory, numerical cognition, object movement, body movement, and brightness imagery. In each domain, eye movements consistently reflect distinct aspects of mental imagery typically akin to those seen in corresponding visual experiences. Several findings further suggest that internal coupling may not only coincide with but also supports internal cognition as evidenced by improved cognitive performance. Available theoretical accounts suggest that internal coupling may serve at least two functional roles in visual imagery: facilitating memory reconstruction and indicating shifts in internal attention. Moreover, recent insights into the neurobiology of internal coupling highlight substantially shared neural pathways in externally and internally directed cognition. The review concludes by identifying open questions and promising avenues for future research such as exploring moderating roles of context and individual differences in internal coupling.

Aging Processes of Working Memory in Different Modalities

Background: Working memory (WM) involves temporarily storing and manipulating information. Research on the impact of aging on WM has shown inconsistent results regarding the decline in visual and verbal WM, with a lack of studies on tactile WM. This study aimed to assess the effects of aging on WM across verbal, visuospatial, and tactile modalities using span tasks of forward (storage) and backward (manipulation) stages. Methods: A total of 130 participants, divided into four age groups of 20-29, 60-69, 70-79, and 80-89, completed the Digit, Visuospatial, and Tactual Spans. Performance was analyzed using a 3 (Task) × 4 (Group) × 2 (Stage) mixed design repeated measures ANOVA. Results: The analysis revealed significant main effects for modality (p < 0.001, ηp2 = 0.15), age (p < 0.001, ηp2 = 0.48), and stage (p < 0.001, ηp2 = 0.30). Digit Span outperformed the other modalities, while Tactual Span showed the worst performance. Additionally, task performance declined with age, and the forward stage was superior to the backward stage. Interaction effects indicated that Digit Span was less affected by aging compared to the Visuospatial and Tactual Spans (p = 0.004, ηp2 = 0.07). Post hoc analyses further revealed that the Digit Span consistently outperformed the other modalities in both stages, with more pronounced differences observed in the forward stage. Conclusions: Verbal WM is more resilient to aging compared to the other modalities while tactile WM declines with age in a manner similar to verbal and visuospatial WM, suggesting a modality-specific impact of aging on WM.

Investigating the link between temperamental and motor development: a longitudinal study of infants aged 6-42 months

BACKGROUND

Since the 1920s, motor development has been a strong research theme, focusing on infants' acquisition of motor skills, such as turning over and crawling. In the 1980s, a dynamic systems approach began emphasizing children's own motivation, which helped explain individual differences in the emergence of motor skills. However, few studies have examined factors contributing to individual differences in early motor development. In response, we investigated directional associations between temperament and motor development in children aged 6 months to 3 years.

METHOD

The Japan Environment and Children's Study (JECS-A) recruited mothers between January 2011 and March 2014. 2,639 mothers were sent a questionnaire at 6 months, and responses were received from 1,657 of them, with full data for children aged 6 months, 2 years, and 3 years, including from three mothers of twins, were analyzed through structural equation modeling. Question items regarding fine and gross motor activities at each age were selected by pediatric neurologists specializing in developmental disorders. The Japanese version of the Little Developmental Coordination Disorder Questionnaire was administered at 42 months. Temperament was assessed through the parent-reported Behavior Questionnaire (short version) for infants, toddlers, and children. In all three measures, Surgency and Negative Affectivity were extracted, and Effortful Control, a major form of self-regulation, was found from toddlerhood onward, as in previous studies.

RESULTS

A path diagram reveals that at 6 months, Surgency and Orienting/Regulation interacted positively with the motor function (respectively, r = .57; r = 40, ps < .001). Up to about 3 years, Effortful Control plays a role in facilitating the motor function, resulting in positive effects on Control During Movement (CDM), General Coordination (GC), and Fine Motor Movement (FMM) (β = 14; β = 30; β = 37, ps < .001). Surgency had a positive effect on CDM and GC (β = 18; β = 06, ps < .001), whereas Negative Affect had a negative influence on FMM and GC (β = -.08; β = -.08, ps < .001).

CONCLUSION

While Surgency may be a key reactive factor in early motor development, Effortful Control and Movement develop in an interactive manner.

TRIAL REGISTRATION

UMIN000030786. Scientific Title: The Japan Environment and Children's Study. Date of disclosure of the study: 2018/01/15. Only questionnaires were administered in the study.

Comparing Neural Correlates of Memory Encoding and Maintenance for Foveal and Peripheral Stimuli

Visual working memory is believed to rely on top-down attentional mechanisms that sustain active sensory representations in early visual cortex, a mechanism referred to as sensory recruitment. However, both bottom-up sensory input and top-down attentional modulations thereof appear to prioritize the fovea over the periphery, such that initially peripheral percepts may even be assimilated by foveal processes. This raises the question whether and how visual working memory differs for central and peripheral input. To address this, we conducted a delayed orientation recall task in which an orientation was presented either at the center of the screen or at 15° eccentricity to the left or right. Response accuracy, EEG activity, and gaze position were recorded from 30 participants. Accuracy was slightly but significantly higher for foveal versus peripheral memories. Decoding of EEG recordings revealed a clear dissociation between early sensory and later maintenance signals. Although sensory signals were clearly decodable for foveal stimuli, they were not for peripheral input. In contrast, maintenance signals were equally decodable for both foveal and peripheral memories, suggesting comparable top-down components regardless of eccentricity. Moreover, although memory representations were initially spatially specific and reflected in voltage fluctuations, later during the maintenance period, they generalized across locations, as emerged in alpha oscillations, thus revealing a dynamic transformation within memory from separate sensory traces to what we propose are common output-related codes. Furthermore, the combined absence of reliable decoding of sensory signals and robust presence of maintenance decoding indicates that storage activity patterns as measured by EEG reflect signals beyond primary visual cortex. We discuss the implications for the sensory recruitment hypothesis.

Rhythm Facilitates Auditory Working Memory via Beta-Band Encoding and Theta-Band Maintenance

Rhythm, as a prominent characteristic of auditory experiences such as speech and music, is known to facilitate attention, yet its contribution to working memory (WM) remains unclear. Here, human participants temporarily retained a 12-tone sequence presented rhythmically or arrhythmically in WM and performed a pitch change-detection task. Behaviorally, while having comparable accuracy, rhythmic tone sequences showed a faster response time and lower response boundaries in decision-making. Electroencephalographic recordings revealed that rhythmic sequences elicited enhanced non-phase-locked beta-band (16 Hz-33 Hz) and theta-band (3 Hz-5 Hz) neural oscillations during sensory encoding and WM retention periods, respectively. Importantly, the two-stage neural signatures were correlated with each other and contributed to behavior. As beta-band and theta-band oscillations denote the engagement of motor systems and WM maintenance, respectively, our findings imply that rhythm facilitates auditory WM through intricate oscillation-based interactions between the motor and auditory systems that facilitate predictive attention to auditory sequences.

The Basic Units of Working Memory Manipulation Are Boolean Maps, Not Objects

Determining the manipulation unit of working memory is one of the fundamental questions in understanding how working memory functions. The prevalent object-based theory in cognitive research predicts that memory manipulation is performed on the level of objects. Here we show instead that the basic units of working memory manipulation are Boolean maps, a data structure describing what can be perceived in an instant. We developed four new manipulation tasks (with data from 80 adults) and showed that manipulation times only increased when the number of Boolean maps manipulated increased. Increasing the number of orientations manipulated did not induce longer manipulation times, consistent with a key prediction of the Boolean map theory. Our results show that Boolean maps are the manipulation unit of working memory.

Positive correlation between working memory and face-to-face social interaction in everyday life

Working memory (WM) plays a crucial role in various cognitive tasks from language comprehension to problem-solving. However, its influence on social activities has remained largely unexplored. The current two studies on individual differences, a pilot (N = 329) and a pre-registered direct replication (N = 338) study, investigated the relationship between WM and outside-the-lab social interaction by using a listening span task and three social network questionnaires (e.g., how many people a participant had contacted in the past month). The consistent patterns in the two studies were (a) WM recall was positively correlated with social network size, (b) WM recall remained positively correlated with social network size even when accounting for online interactions on WhatsApp and Facebook, and (c) WM recall was positively correlated with social network size by face-to-face interaction. These novel findings would suggest connections between WM and face-to-face social interaction. It was, however, acknowledged that the obtained effect sizes were small, and that further investigation is indeed necessary. In light of this, we also clarify future directions for understanding the relationship between WM and social interaction.

The impact of cognitive functions, psychological disorders, and coping strategies on quality of life and disease outcomes in patients with inflammatory bowel diseases: A cross-sectional study

Quality of life (QoL) in patients with inflammatory bowel disease (IBD) is influenced by several factors, many of which may also impact cognitive function. However, the extent of the interaction among these factors, QoL, and disease outcomes in IBD patients remains unknown. We thus aim to characterize the relationships among psychological disorders, coping mechanisms, cognitive function, and the overall impact on QoL and disease outcomes in patients with IBD. This cross-sectional observational study was conducted at an academic care center. QoL was evaluated using the Short Inflammatory Bowel Disease Questionnaire (SIBDQ), and disease severity was evaluated using the Harvey-Bradshaw Index (HBI) for Crohn's disease (CD) and the Simple Clinical Colitis Activity Index (SCCAI) for ulcerative colitis (UC). We also used the Hospital Anxiety and Depression scale (HADS). Regression models were used to test the associations among QoL, number of hospitalizations, disease severity, cognitive functioning (working memory [WM] and reaction time), and coping strategies while controlling for anxiety and depressive symptoms, age, and sex. This study included 41 patients (24 patients with CD and 17 with UC) whose mean age was 28.2 (±8.4) years (23 males) and mean SIBDQ score was 51.5 (±10.0). Patients with more WM errors had lower QoL scores (P = .041), whereas patients with higher anxiety levels had lower QoL and more active UC (P = .008 and P = .016, respectively). The use of avoidant coping mechanisms was associated with a significantly higher number of hospitalizations (P = .038), and patients who adopted more emotion-focused coping strategies had a longer illness duration (P = .021). Finally, patients with higher education levels were found to use more active coping mechanisms than others. These results confirm the impact of cognitive, psychological, and coping factors on QoL and disease outcomes in patients with IBD; however, the mechanisms by which these factors interrelate remain unclear. Therapies aimed at improving both cognitive functions and psychological conditions may thus be effective at improving QoL and disease outcomes in IBD patients, and education may play a positive role in promoting the adoption of more effective coping strategies among IBD patients.

Forward versus backward recall: Modality testing

OBJECTIVES

One of the most frequently applied paradigms in the clinical and experimental fields for assessing working memory is the simple span task, composed of forward and backward recall conditions. However, the utility of the simple span measures and their relation to working memory modalities has yet to be elucidated. The current study aimed to address which of the span measures is more sensitive to the differentiation between forward and backward recall in the tactile, verbal, and visuo-spatial modalities, and to test if working memory modalities differ in the disparity between the two conditions.

METHODS

134 healthy young adults performed the Tactual, Digit, and Visuo-spatial Span tasks, producing Total Correct scores and Longest Sequence score measures.

RESULTS

There was an interaction effect for condition and modality type, showing better performance for the forward compared to the backward recall in both measures. However, the effect size of the Longest Sequence score was significantly higher than the Total Correct score in all tasks. Furthermore, the Visuo-spatial Span exhibited a larger difference between forward and backward recall compared to the Digit and Tactual Span, whereas no difference was found between the latter two.

CONCLUSIONS

Forward and backward recall are distinguished in all three modalities, and the Longest Sequence score is more sensitive to differentiate between storage and manipulation components of working memory than the Total Correct score. Additionally, the cognitive demand imposed by manipulation compared to storage is the greatest in the visuo-spatial modality.

Jointly looking to the past and the future in visual working memory

Working memory enables us to bridge past sensory information to upcoming future behaviour. Accordingly, by its very nature, working memory is concerned with two components: the past and the future. Yet, in conventional laboratory tasks, these two components are often conflated, such as when sensory information in working memory is encoded and tested at the same location. We developed a task in which we dissociated the past (encoded location) and future (to-be-tested location) attributes of visual contents in working memory. This enabled us to independently track the utilisation of past and future memory attributes through gaze, as observed during mnemonic selection. Our results reveal the joint consideration of past and future locations. This was prevalent even at the single-trial level of individual saccades that were jointly biased to the past and future. This uncovers the rich nature of working memory representations, whereby both past and future memory attributes are retained and can be accessed together when memory contents become relevant for behaviour.

A Neural Decision Signal during Internal Sampling from Working Memory in Humans

How humans transform sensory information into decisions that steer purposeful behavior is a central question in psychology and neuroscience that is traditionally investigated during the sampling of external environmental signals. The decision-making framework of gradual information sampling toward a decision has also been proposed to apply when sampling internal sensory evidence from working memory. However, neural evidence for this proposal remains scarce. Here we show (using scalp EEG in male and female human volunteers) that sampling internal visual representations from working memory elicits a scalp EEG potential associated with gradual evidence accumulation-the central parietal positivity. Consistent with an evolving decision process, we show how this signal (1) scales with the time participants require to reach a decision about the cued memory content and (2) is amplified when having to decide among multiple contents in working memory. These results bring the electrophysiology of decision-making into the domain of working memory and suggest that variability in memory-guided behavior may be driven (at least in part) by variations in the sampling of our inner mental contents.

Neural Signatures of Competition between Voluntary and Involuntary Influences over the Focus of Attention in Visual Working Memory

Adaptive behavior relies on the selection and prioritization of relevant sensory inputs from the external environment as well as from among internal sensory representations held in working memory. Recent behavioral evidence suggests that the classic distinction between voluntary (goal-driven) and involuntary (stimulus-driven) influences over attentional allocation also applies to the selection of internal representations held in working memory. In the current EEG study, we set out to investigate the neural dynamics associated with the competition between voluntary and involuntary control over the focus of attention in visual working memory. We show that when voluntary and involuntary factors compete for the internal focus of attention, prioritization of the appropriate item is delayed-as reflected both in delayed gaze biases that track internal selection and in delayed neural beta (15-25 Hz) dynamics that track the planning for the upcoming memory-guided manual action. We further show how this competition is paralleled-possibly resolved-by an increase in frontal midline theta (4-8 Hz) activity that, moreover, predicts the speed of ensuing memory-guided behavior. Finally, because theta increased following retrocues that effectively reduced working-memory load, our data unveil how frontal theta activity during internal attentional focusing tracks demands on cognitive control over and above working-memory load. Together, these data yield new insight into the neural dynamics that govern the focus of attention in visual working memory, and disentangle the contributions of frontal midline theta activity to the processes of control versus retention in working memory.

Task demand mediates the interaction of spatial and temporal attention

Psychophysical studies typically test attentional mechanisms in isolation, but in everyday life they interact to optimize human behavior. We investigated whether spatial and temporal attention interact in two orientation discrimination experiments that vary in task demand. We manipulated temporal and spatial attention separately and conjointly with well-established methods for testing each spatial or temporal attention. We assessed sensitivity (d') and reaction time for every combination of spatial and timing cues, each of which was valid, neutral, or invalid. Spatial attention modulated sensitivity (d') and speed (reaction time) across temporal attention conditions. Temporal attention modulated sensitivity and speed under high- but not low- task demands. Furthermore, spatial and temporal attention interacted for the high-demand task. This study reveals that task demand matters; in a simple task spatial attention suffices to improve performance, whereas in a more demanding task both spatial and temporal attention interact to boost performance, albeit in a subadditive fashion.

Shifting attention between perception and working memory

Most everyday tasks require shifting the focus of attention between sensory signals in the external environment and internal contents in working memory. To date, shifts of attention have been investigated within each domain, but shifts between the external and internal domain remain poorly understood. We developed a combined perception and working-memory task to investigate and compare the consequences of shifting spatial attention within and between domains in the service of a common orientation-reproduction task. Participants were sequentially cued to attend to items either in working memory or to an upcoming sensory stimulation. Stay trials provided a baseline condition, while shift trials required participants to shift their attention to another item within the same or different domain. Validating our experimental approach, we found evidence that participants shifted attention effectively in either domain (Experiment 1). In addition, we observed greater costs when transitioning attention between as compared to within domains (Experiments 1, 2). Strikingly, these costs persisted even when participants were given more time to complete the attentional shift (Experiment 2). Biases in fixational gaze behaviour tracked attentional orienting in both domains, but revealed no latency or magnitude difference for within- versus between-domain shifts (Experiment 1). Collectively, the results from Experiments 1 and 2 suggest that shifting between attentional domains might be regulated by a unique control function. Our results break new ground for exploring the ubiquitous act of shifting attention between perception and working memory to guide adaptive behaviour in everyday cognition.

We know what attention is!

Attention is one of the most thoroughly investigated psychological phenomena, yet skepticism about attention is widespread: we do not know what it is, it is too many things, there is no such thing. The deficiencies highlighted are not about experimental work but the adequacy of the scientific theory of attention. Combining common scientific claims about attention into a single theory leads to internal inconsistency. This paper demonstrates that a specific functional conception of attention is incorporated into the tasks used in standard experimental paradigms. In accepting these paradigms as valid probes of attention, we commit to this common conception. The conception unifies work at multiple levels of analysis into a coherent scientific explanation of attention. Thus, we all know what attention is.

Microsaccades reflect attention shifts: a mini review of 20 years of microsaccade research

Microsaccades are small, involuntary eye movements that occur during fixation. Since the 1950s, researchers have conducted extensive research on the role of microsaccades in visual information processing, and found that they also play an important role in human advanced visual cognitive activities. Research over the past 20 years further suggested that there is a close relationship between microsaccades and visual attention, yet lacking a timely review. The current article aims to provide a state-of-the-art review and bring microsaccades studies into the sight of attention research. We firstly introduce basic characteristics about microsaccades, then summarized the empirical evidence supporting the view that microsaccades can reflect both external (perception) and internal (working memory) attention shifts. We finally conclude and highlight three promising avenues for future research.

The Relation Between Attention and Memory

The relation between attention and memory has long been deemed important for understanding cognition, and it was heavily researched even in the first experimental psychology laboratory by Wilhelm Wundt and his colleagues. Since then, the importance of the relation between attention and memory has been explored in myriad subdisciplines of psychology, and we incorporate a wide range of these diverse fields. Here, we examine some of the practical consequences of this relation and summarize work with various methodologies relating attention to memory in the fields of working memory, long-term memory, individual differences, life-span development, typical brain function, and neuropsychological conditions. We point out strengths and unanswered questions for our own embedded processes view of information processing, which is used to organize a large body of evidence. Last, we briefly consider the relation of the evidence to a range of other theoretical views before drawing conclusions about the state of the field.

Attention influences the effects of the previous form orientation on the current motion direction estimation

Recent studies have found that the estimates of motion directions are biased toward the previous form orientations, showing serial dependence, and the serial dependence does not involve cognitive abilities. In the current study, we conducted two experiments to investigate whether and how attention-a cognitive ability-affected the serial dependence. The results showed that serial dependence was present in the current study, reproducing the previous findings. Importantly, when the attentional load reduced the reliability (i.e., estimation accuracy and precision) of previous form orientations (Experiment 1), the serial dependence decreased, meaning that the biases of motion direction estimates toward previous form orientations were reduced; in contrast, when the attentional load reduced the reliability of current motion directions (Experiment 2), the serial dependence increased, meaning that the biases of motion direction estimates toward previous form orientations were increased. These trends were well consistent with the prediction of the Bayesian inference theory. Therefore, the current study revealed the involvement of attention in the serial dependence of current motion direction estimation on the previous form orientation, demonstrating that the serial dependence was cognitive and the attentional effect can be a Bayesian inference process, initially revealing its computational mechanism.

The Intensity of Internal and External Attention Assessed with Pupillometry

Not only is visual attention shifted to objects in the external world, attention can also be directed to objects in memory. We have recently shown that pupil size indexes how strongly items are attended externally, which was reflected in more precise encoding into visual working memory. Using a retro-cuing paradigm, we here replicated this finding by showing that stronger pupil constrictions during encoding were reflective of the depth of encoding. Importantly, we extend this previous work by showing that pupil size also revealed the intensity of internal attention toward content stored in visual working memory. Specifically, pupil dilation during the prioritization of one among multiple internally stored representations predicted the precision of the prioritized item. Furthermore, the dynamics of the pupillary responses revealed that the intensity of internal and external attention independently determined the precision of internalized visual representations. Our results show that both internal and external attention are not all-or-none processes, but should rather be thought of as continuous resources that can be deployed at varying intensities. The employed pupillometric approach allows to unravel the intricate interplay between internal and external attention and their effects on visual working memory.

Microsaccades Track Location-Based Object Rehearsal in Visual Working Memory

Besides controlling eye movements, the brain's oculomotor system has been implicated in the control of covert spatial attention and the rehearsal of spatial information in working memory. We investigated whether the oculomotor system also contributes to rehearsing visual objects in working memory when object location is never asked about. To address this, we tracked the incidental use of locations for mnemonic rehearsal via directional biases in microsaccades while participants maintained two visual objects (colored oriented gratings) in working memory. By varying the stimulus configuration (horizontal, diagonal, and vertical) at encoding, we could quantify whether microsaccades were more aligned with the configurational axis of the memory contents, as opposed to the orthogonal axis. Experiment 1 revealed that microsaccades continued to be biased along the axis of the memory content several seconds into the working memory delay. In Experiment 2, we confirmed that this directional microsaccade bias was specific to memory demands, ruling out lingering effects from passive and attentive encoding of the same visual objects in the same configurations. Thus, by studying microsaccade directions, we uncover oculomotor-driven rehearsal of visual objects in working memory through their associated locations.

No obligatory trade-off between the use of space and time for working memory

Space and time can each act as scaffolds for the individuation and selection of visual objects in working memory. Here we ask whether there is a trade-off between the use of space and time for visual working memory: whether observers will rely less on space, when memoranda can additionally be individuated through time. We tracked the use of space through directional biases in microsaccades after attention was directed to memory contents that had been encoded simultaneously or sequentially to the left and right of fixation. We found that spatial gaze biases were preserved when participants could (Experiment 1) and even when they had to (Experiment 2) additionally rely on time for object individuation. Thus, space remains a profound organizing medium for working memory even when other organizing sources are available and utilized, with no evidence for an obligatory trade-off between the use of space and time.

Prospection of Potential Actions during Visual Working Memory Starts Early, Is Flexible, and Predicts Behavior

For visual working memory to serve upcoming behavior, it is crucial that we prepare for the potential use of working-memory contents ahead of time. Recent studies have demonstrated how the prospection and planning for an upcoming manual action starts early after visual encoding, and occurs alongside visual retention. Here, we address whether such "output planning" in visual working memory flexibly adapts to different visual-motor mappings, and occurs even when an upcoming action will only potentially become relevant for behavior. Human participants (female and male) performed a visual-motor working memory task in which they remembered one or two colored oriented bars for later (potential) use. We linked, and counterbalanced, the tilt of the visual items to specific manual responses. This allowed us to track planning of upcoming behavior through contralateral attenuation of β band activity, a canonical motor-cortical EEG signature of manual-action planning. The results revealed how action encoding and subsequent planning alongside visual working memory (1) reflect anticipated task demands rather than specific visual-motor mappings, (2) occur even for actions that will only potentially become relevant for behavior, and (3) are associated with faster performance for the encoded item, at the expense of performance to other working-memory content. This reveals how the potential prospective use of visual working memory content is flexibly planned early on, with consequences for the speed of memory-guided behavior.SIGNIFICANCE STATEMENT It is increasingly studied how visual working memory helps us to prepare for the future, in addition to how it helps us to hold onto the past. Recent studies have demonstrated that the planning of prospective actions occurs alongside encoding and retention in working memory. We show that such early "output planning" flexibly adapts to varying visual-motor mappings, occurs both for certain and potential actions, and predicts ensuing working-memory guided behavior. These results highlight the flexible and future-oriented nature of visual working memory, and provide insight into the neural basis of the anticipatory dynamics that translate visual representations into adaptive upcoming behavior.

Attention with or without working memory: mnemonic reselection of attended information

Attention has been regarded as the 'gatekeeper' controlling what information gets selected into working memory. However, a new perspective has emerged with the discovery of attribute amnesia, a phenomenon revealing that people are frequently unable to report information they have just attended to moments ago. This report failure is thought to stem from a lack of consolidating the attended information into working memory, indicating a dissociation between attention and working memory. Building on these findings, a new concept called memory reselection is proposed to describe a secondary round of selection among the attended information. These discoveries challenge the conventional view of how attention and working memory are related and shed new light onto modeling attention and memory as dissociable processes.

Temporally Dissociable Mechanisms of Spatial, Feature, and Motor Selection during Working Memory-guided Behavior

Working memory (WM) is a capacity- and duration-limited system that forms a temporal bridge between fleeting sensory phenomena and possible actions. But how are the contents of WM used to guide behavior? A recent high-profile study reported evidence for simultaneous access to WM content and linked motor plans during WM-guided behavior, challenging serial models where task-relevant WM content is first selected and then mapped on to a task-relevant motor response. However, the task used in that study was not optimized to distinguish the selection of spatial versus nonspatial visual information stored in memory, nor to distinguish whether or how the chronometry of selecting nonspatial visual information stored in memory might differ from the selection of linked motor plans. Here, we revisited the chronometry of spatial, feature, and motor selection during WM-guided behavior using a task optimized to disentangle these processes. Concurrent EEG and eye position recordings revealed clear evidence for temporally dissociable spatial, feature, and motor selection during this task. Thus, our data reveal the existence of multiple WM selection mechanisms that belie conceptualizations of WM-guided behavior based on purely serial or parallel visuomotor processing.

Time for Memories

The ability to store information about the past to dynamically predict and prepare for the future is among the most fundamental tasks the brain performs. To date, the problems of understanding how the brain stores and organizes information about the past (memory) and how the brain represents and processes temporal information for adaptive behavior have generally been studied as distinct cognitive functions. This Symposium explores the inherent link between memory and temporal cognition, as well as the potential shared neural mechanisms between them. We suggest that working memory and implicit timing are interconnected and may share overlapping neural mechanisms. Additionally, we explore how temporal structure is encoded in associative and episodic memory and, conversely, the influences of episodic memory on subsequent temporal anticipation and the perception of time. We suggest that neural sequences provide a general computational motif that contributes to timing and working memory, as well as the spatiotemporal coding and recall of episodes.

When the mind's eye prevails: The Internal Dominance over External Attention (IDEA) hypothesis

Throughout the 20th century, the psychological literature has considered attention as being primarily directed at the outside world. More recent theories conceive attention as also operating on internal information, and mounting evidence suggests a single, shared attentional focus between external and internal information. Such sharing implies a cognitive architecture where attention needs to be continuously shifted between prioritizing either external or internal information, but the fundamental principles underlying this attentional balancing act are currently unknown. Here, we propose and evaluate one such principle in the shape of the Internal Dominance over External Attention (IDEA) hypothesis: Contrary to the traditional view of attention as being primarily externally oriented, IDEA asserts that attention is inherently biased toward internal information. We provide a theoretical account for why such an internal attention bias may have evolved and examine findings from a wide range of literatures speaking to the balancing of external versus internal attention, including research on working memory, attention switching, visual search, mind wandering, sustained attention, and meditation. We argue that major findings in these disparate research lines can be coherently understood under IDEA. Finally, we consider tentative neurocognitive mechanisms contributing to IDEA and examine the practical implications of more deliberate control over this bias in the context of psychopathology. It is hoped that this novel hypothesis motivates cross-talk between the reviewed research lines and future empirical studies directly examining the mechanisms that steer attention either inward or outward on a moment-by-moment basis.

Capacity and selection in immersive visual working memory following naturalistic object disappearance

Visual working memory-holding past visual information in mind for upcoming behavior-is commonly studied following the abrupt removal of visual objects from static two-dimensional (2D) displays. In everyday life, visual objects do not typically vanish from the environment in front of us. Rather, visual objects tend to enter working memory following self or object motion: disappearing from view gradually and changing the spatial relation between memoranda and observer. Here, we used virtual reality (VR) to investigate whether two classic findings from visual working memory research-a capacity of around three objects and the reliance on space for object selection-generalize to more naturalistic modes of object disappearance. Our static reference condition mimicked traditional laboratory tasks whereby visual objects were held static in front of the participant and removed from view abruptly. In our critical flow condition, the same visual objects flowed by participants, disappearing from view gradually and behind the observer. We considered visual working memory performance and capacity, as well as space-based mnemonic selection, indexed by directional biases in gaze. Despite vastly distinct modes of object disappearance and altered spatial relations between memoranda and observer, we found comparable capacity and comparable gaze signatures of space-based mnemonic selection. This finding reveals how classic findings from visual working memory research generalize to immersive situations with more naturalistic modes of object disappearance and with dynamic spatial relations between memoranda and observer.

Internal selective attention is delayed by competition between endogenous and exogenous factors

External attention is mediated by competition between endogenous (goal-driven) and exogenous (stimulus-driven) factors, with the balance of competition determining which stimuli are selected. Occasionally, exogenous factors "win" this competition and drive the selection of task-irrelevant stimuli. Endogenous and exogenous selection mechanisms may also compete to control the selection of internal representations (e.g., those stored in working memory), but whether this competition is resolved in the same way as external attention is unknown. Here, we leveraged the high temporal resolution of human EEG to determine how competition between endogenous and exogenous factors influences the selection of internal representations. Unlike external attention, competition did not prompt the selection of task-irrelevant working memory content. Instead, it delayed the endogenous selection of task-relevant working memory content by several hundred milliseconds. Thus, competition between endogenous and exogenous factors influences internal selective attention, but in a different way than external selective attention.

Differential aspects of attention predict the depth of visual working memory encoding: Evidence from pupillometry

What determines how much one encodes into visual working memory? Traditionally, encoding depth is considered to be indexed by spatiotemporal properties of gaze, such as gaze position and dwell time. Although these properties inform about where and how long one looks, they do not necessarily inform about the current arousal state or how strongly attention is deployed to facilitate encoding. Here, we found that two types of pupillary dynamics predict how much information is encoded during a copy task. The task involved encoding a spatial pattern of multiple items for later reproduction. Results showed that smaller baseline pupil sizes preceding and stronger pupil orienting responses during encoding predicted that more information was encoded into visual working memory. Additionally, we show that pupil size reflects not only how much but also how precisely material is encoded. We argue that a smaller pupil size preceding encoding is related to increased exploitation, whereas larger pupil constrictions signal stronger attentional (re)orienting to the to-be-encoded pattern. Our findings support the notion that the depth of visual working memory encoding is the integrative outcome of differential aspects of attention: how alert one is, how much attention one deploys, and how long it is deployed. Together, these factors determine how much information is encoded into visual working memory.

Attention in flux

Selective attention comprises essential infrastructural functions supporting cognition-anticipating, prioritizing, selecting, routing, integrating, and preparing signals to guide adaptive behavior. Most studies have examined its consequences, systems, and mechanisms in a static way, but attention is at the confluence of multiple sources of flux. The world advances, we operate within it, our minds change, and all resulting signals progress through multiple pathways within the dynamic networks of our brains. Our aim in this review is to raise awareness of and interest in three important facets of how timing impacts our understanding of attention. These include the challenges posed to attention by the timing of neural processing and psychological functions, the opportunities conferred to attention by various temporal structures in the environment, and how tracking the time courses of neural and behavioral modulations with continuous measures yields surprising insights into the workings and principles of attention.

Prioritizing flexible working memory representations through retrospective attentional strengthening

Previous work has proposed two potential benefits of retrospective attention on working memory (WM): target strengthening and non-target inhibition. It remains unknown which hypothesis contributes to the improved WM performance, yet the neural mechanisms responsible for this attentional benefit are unclear. Here, we recorded electroencephalography (EEG) signals while 33 participants performed a retrospective-cue WM task. Multivariate pattern classification analysis revealed that only representations of target features were enhanced by valid retrospective attention during retention, supporting the target strengthening hypothesis. Further univariate analysis found that mid-frontal theta inter-trial phase coherence (ITPC) and ERP components were modulated by valid retrospective attention and correlated with individual differences and moment-to-moment fluctuations on behavioral outcomes, suggesting that both trait- and state-level variability in attentional preparatory processes influence goal-directed behavior. Furthermore, task-irrelevant target spatial location could be decoded from EEG signals, indicating that enhanced spatial binding of target representation is vital to high WM precision. Importantly, frontoparietal theta-alpha phase-amplitude coupling was increased by valid retrospective attention and predicted the reduced random guessing rates. This long-range connection supported top-down information flow in the engagement of frontoparietal networks, which might organize attentional states to integrate target features. Altogether, these results provide neurophysiological bases that retrospective attention improves WM precision by enhancing flexible target representation and emphasize the critical role of the frontoparietal attentional network in the control of WM representations.

Attention as a multi-level system of weights and balances

This opinion piece is part of a collection on the topic: "What is attention?" Despite the word's place in the common vernacular, a satisfying definition for "attention" remains elusive. Part of the challenge is there exist many different types of attention, which may or may not share common mechanisms. Here we review this literature and offer an intuitive definition that draws from aspects of prior theories and models of attention but is broad enough to recognize the various types of attention and modalities it acts upon: attention as a multi-level system of weights and balances. While the specific mechanism(s) governing the weighting/balancing may vary across levels, the fundamental role of attention is to dynamically weigh and balance all signals-both externally-generated and internally-generated-such that the highest weighted signals are selected and enhanced. Top-down, bottom-up, and experience-driven factors dynamically impact this balancing, and competition occurs both within and across multiple levels of processing. This idea of a multi-level system of weights and balances is intended to incorporate both external and internal attention and capture their myriad of constantly interacting processes. We review key findings and open questions related to external attention guidance, internal attention and working memory, and broader attentional control (e.g., ongoing competition between external stimuli and internal thoughts) within the framework of this analogy. We also speculate about the implications of failures of attention in terms of weights and balances, ranging from momentary one-off errors to clinical disorders, as well as attentional development and degradation across the lifespan. This article is categorized under: Psychology > Attention Neuroscience > Cognition.

Opening Questions in Visual Working Memory

In this reflective piece on visual working memory, I depart from the laboriously honed skills of writing a review. Instead of integrating approaches, synthesizing evidence, and building a cohesive perspective, I scratch my head and share niggles and puzzlements. I expose where my scholarship and understanding are stumped by findings and standard views in the literature.

Long-term memory and working memory compete and cooperate to guide attention

Multiple types of memory guide attention: Both long-term memory (LTM) and working memory (WM) effectively guide visual search. Furthermore, both types of memories can capture attention automatically, even when detrimental to performance. It is less clear, however, how LTM and WM cooperate or compete to guide attention in the same task. In a series of behavioral experiments, we show that LTM and WM reliably cooperate to guide attention: Visual search is faster when both memories cue attention to the same spatial location (relative to when only one memory can guide attention). LTM and WM competed to guide attention in more limited circumstances: Competition only occurred when these memories were in different dimensions - particularly when participants searched for a shape and held an accessory color in mind. Finally, we found no evidence for asymmetry in either cooperation or competition: There was no evidence that WM helped (or hindered) LTM-guided search more than the other way around. This lack of asymmetry was found despite differences in LTM-guided and WM-guided search overall, and differences in how two LTMs and two WMs compete or cooperate with each other to guide attention. This work suggests that, even if only one memory is currently task-relevant, WM and LTM can cooperate to guide attention; they can also compete when distracting features are salient enough. This work elucidates interactions between WM and LTM during attentional guidance, adding to the literature on costs and benefits to attention from multiple active memories.