Neural and Behavioral Evidence for an Online Resetting Process in Visual Working Memory

Electrophysiology Reveals That Intuitive Physics Guides Visual Tracking and Working Memory

Starting in early infancy, our perception and predictions are rooted in strong expectations about the behavior of everyday objects. These intuitive physics expectations have been demonstrated in numerous behavioral experiments, showing that even pre-verbal infants are surprised when something impossible happens (e.g., when objects magically appear or disappear). However, it remains unclear whether and how physical expectations shape different aspects of moment-by-moment online visual scene processing, unrelated to explicit physical reasoning. In two EEG experiments, people watched short videos like those used in behavioral studies with adults and infants, and more recently in AI benchmarks. Objects moved on a stage, and were briefly hidden behind an occluder, with the scene either unfolding as expected, or violating object permanence (adding or removing an object). We measured the contralateral delay activity, an electrophysiological marker of online processing, to examine participants' working memory (WM) representations, as well as their ability to continuously track the objects in the scene. We found that both types of object permanence violations disrupted tracking, even though violations involved perceptually non-salient events (magical vanishing) or new objects that weren't previously tracked (magical creation). Physical violations caused WM to reset, i.e., to discard the original scene representation before it could recover and represent the updated number of items. Providing a physical explanation for the violations (a hole behind the occluder) restored object tracking, and we found evidence that WM continued to represent items that disappeared 'down the hole'. Our results show how intuitive physical expectations shape online representations, and form the basis of dynamic object tracking.

Unravelling the object-based nature of visual working memory: insight from pointers

Visual working memory (VWM) plays a crucial role in temporarily storing and processing visual information, but the nature of stored representations and their interaction with new inputs has long been unclear. The pointer system refers to how VWM links new sensory inputs to stored information using specific cues. This study aimed to investigate whether the pointer system is based on spatial, feature-based, or object-based cues by employing the repetition benefit effect, where memory performance improves with repeated memory items. Across three experiments, we manipulated spatial positions, shapes, and colors as pointer cues to determine how these features affect VWM consolidation and updating. The results showed that while spatial location serves as a strong pointer cue, shape and color features can also effectively reestablish object correspondence in VWM. These findings support the view that the pointer system in VWM is flexible and object-based, utilizing various feature cues to maintain memory continuity. This study provides new insights into how VWM connects new inputs with stored information through the pointer system.

Visual event boundaries trigger forgetting despite active maintenance in visual working memory

The contents of visual perception are inherently dynamic-just as we experience objects in space, so too events in time. The boundaries between these events have downstream consequences. For example, memory for incidentally encountered items is impaired when walking through a doorway, perhaps because event boundaries serve as cues to clear obsolete information from previous events. Although this kind of "memory flushing" can be adaptive, work on visual working memory (VWM) has focused on the opposite function of active maintenance in the face of distraction. How do these two cognitive operations interact? In this study, observers watched animations in which they walked through three-dimensionally rendered rooms with picture frames on the walls. Within the frames, observers either saw images that they had to remember ("encoding") or recalled images they had seen in the immediately preceding frame ("test"). Half of the time, a doorway was crossed during the delay between encoding and test. Across experiments, there was a consistent memory decrement for the first image encoded in the doorway compared to the no-doorway condition while equating time elapsed, distance traveled, and distractibility of the doorway. This decrement despite top-down VWM efforts highlights the power of event boundaries to structure what and when we forget.

The effect of context on pointer allocation in visual working memory

Visual working memory (VWM) can hold a limited amount of visual information and manipulate it. It encodes this information and forms representations of each one of the relevant objects. When an object changes, VWM can either update or reset its representation to account for this change. To access a specific representation VWM relies on a pointer system associating each representation with the corresponding object in the environment. While previous studies described these processes as reacting to a change in the object status, this study investigated the adaptability of the pointer system to the task context. We measured the contralateral delay activity (CDA; an electrophysiological marker of VWM) as a marker of updating and resetting. In two experiments we used a shape change detection task (similar to Balaban & Luria, 2017) and manipulated the proportion of the resetting and updating trials to create different task contexts. Experiment 1 indicated that VWM can adapt to a resetting mode in which it performs resetting in conditions that triggered updating in previous studies. However, Experiment 2 revealed that the pointer system cannot adapt to an updating mode and perform updating in conditions that trigger resetting. These results suggest that VWM can strategically perform resetting, but once a pointer is lost, it's impossible to update the representation and a resetting process is mandatory triggered regardless of the context.

Modeling the contribution of theta-gamma coupling to sequential memory, imagination, and dreaming

Gamma oscillations nested in a theta rhythm are observed in the hippocampus, where are assumed to play a role in sequential episodic memory, i.e., memorization and retrieval of events that unfold in time. In this work, we present an original neurocomputational model based on neural masses, which simulates the encoding of sequences of events in the hippocampus and subsequent retrieval by exploiting the theta-gamma code. The model is based on a three-layer structure in which individual Units oscillate with a gamma rhythm and code for individual features of an episode. The first layer (working memory in the prefrontal cortex) maintains a cue in memory until a new signal is presented. The second layer (CA3 cells) implements an auto-associative memory, exploiting excitatory and inhibitory plastic synapses to recover an entire episode from a single feature. Units in this layer are disinhibited by a theta rhythm from an external source (septum or Papez circuit). The third layer (CA1 cells) implements a hetero-associative net with the previous layer, able to recover a sequence of episodes from the first one. During an encoding phase, simulating high-acetylcholine levels, the network is trained with Hebbian (synchronizing) and anti-Hebbian (desynchronizing) rules. During retrieval (low-acetylcholine), the network can correctly recover sequences from an initial cue using gamma oscillations nested inside the theta rhythm. Moreover, in high noise, the network isolated from the environment simulates a mind-wandering condition, randomly replicating previous sequences. Interestingly, in a state simulating sleep, with increased noise and reduced synapses, the network can "dream" by creatively combining sequences, exploiting features shared by different episodes. Finally, an irrational behavior (erroneous superimposition of features in various episodes, like "delusion") occurs after pathological-like reduction in fast inhibitory synapses. The model can represent a straightforward and innovative tool to help mechanistically understand the theta-gamma code in different mental states.

Location- and Object-Based Representational Mechanisms Account for Bilateral Field Advantage in Multiple-Object Tracking

Keeping track of multiple visually identical and independently moving objects is a remarkable feature of the human visual system. Theoretical accounts for this ability focus on resource-based models that describe parametric decreases of performance with increasing demands during the task (i.e., more relevant items, closer distances, higher speed). Additionally, the presence of two central tracking resources, one within each hemisphere, has been proposed, allowing for an independent maintenance of moving targets within each visual hemifield. Behavioral evidence in favor of such a model shows that human subjects are able to track almost twice as many targets across both hemifields compared with within one hemifield. A number of recent publications argue for two separate and parallel tracking mechanisms during standard object tracking tasks that allow for the maintenance of the relevant information in a location-based and object-based manner. Unique electrophysiological correlates for each of those processes have been identified. The current study shows that these electrophysiological components are differentially present during tracking within either the left or right hemifield. The present results suggest that targets are mostly maintained as an object-based representation during left hemifield tracking, while location-based resources are preferentially engaged during right hemifield tracking. Interestingly, the manner of representation does not seem to have an impact on behavioral performance within the subjects, while the electrophysiological component indicating object-based tracking does correlate with performance between subjects. We propose that hemifield independence during multiple-object tracking may be an indication of the underlying hemispheric bias for parallel location-based and object-based tracking mechanisms.

Results for Nonlinear Diffusion Equations with Stochastic Resetting

In this study, we investigate a nonlinear diffusion process in which particles stochastically reset to their initial positions at a constant rate. The nonlinear diffusion process is modeled using the porous media equation and its extensions, which are nonlinear diffusion equations. We use analytical and numerical calculations to obtain and interpret the probability distribution of the position of the particles and the mean square displacement. These results are further compared and shown to agree with the results of numerical simulations. Our findings show that a system of this kind exhibits non-Gaussian distributions, transient anomalous diffusion (subdiffusion and superdiffusion), and stationary states that simultaneously depend on the nonlinearity and resetting rate.

Dissociable online integration processes in visual working memory

Visual working memory has severe capacity limits, creating a bottleneck for active processing. A key way of mitigating this limitation is by chunking, i.e. compressing several pieces of information into one visual working memory representation. However, despite decades of research, chunking efficiency remains debated because of mixed evidence. We propose that there are actually 2 integration mechanisms: Grouping combines several objects to one representation, and object-unification merges the parts of a single object. Critically, we argue that the fundamental distinction between the 2 processes is their differential use of the pointer system, the indexing process connecting visual working memory representations with perception. In grouping, the objects that are represented together still maintain independent pointers, making integration costly but highly flexible. Conversely, object-unification fuses the pointers as well as the representations, with the single pointer producing highly efficient integration but blocking direct access to individual parts. We manipulated integration cues via task-irrelevant movement, and monitored visual working memory's online electrophysiological marker. Uniquely colored objects were flexibly grouped and ungrouped via independent pointers (experiment 1). If objects turned uniformly black, object-integration could not be undone (experiment 2), requiring visual working memory to reset before re-individuation. This demonstrates 2 integration levels (representational-merging versus pointer-compression) and establishes the dissociation between visual working memory representations and their underlying pointers.

Dynamic neural reconstructions of attended object location and features using EEG

Attention allows us to select relevant and ignore irrelevant information from our complex environments. What happens when attention shifts from one item to another? To answer this question, it is critical to have tools that accurately recover neural representations of both feature and location information with high temporal resolution. In the present study, we used human electroencephalography (EEG) and machine learning to explore how neural representations of object features and locations update across dynamic shifts of attention. We demonstrate that EEG can be used to create simultaneous time courses of neural representations of attended features (time point-by-time point inverted encoding model reconstructions) and attended location (time point-by-time point decoding) during both stable periods and across dynamic shifts of attention. Each trial presented two oriented gratings that flickered at the same frequency but had different orientations; participants were cued to attend one of them and on half of trials received a shift cue midtrial. We trained models on a stable period from Hold attention trials and then reconstructed/decoded the attended orientation/location at each time point on Shift attention trials. Our results showed that both feature reconstruction and location decoding dynamically track the shift of attention and that there may be time points during the shifting of attention when 1) feature and location representations become uncoupled and 2) both the previously attended and currently attended orientations are represented with roughly equal strength. The results offer insight into our understanding of attentional shifts, and the noninvasive techniques developed in the present study lend themselves well to a wide variety of future applications.NEW & NOTEWORTHY We used human EEG and machine learning to reconstruct neural response profiles during dynamic shifts of attention. Specifically, we demonstrated that we could simultaneously read out both location and feature information from an attended item in a multistimulus display. Moreover, we examined how that readout evolves over time during the dynamic process of attentional shifts. These results provide insight into our understanding of attention, and this technique carries substantial potential for versatile extensions and applications.

Sensory Delay Activity: More than an Electrophysiological Index of Working Memory Load

Sustained contralateral delay activity emerges in the retention period of working memory (WM) tasks and has been commonly interpreted as an electrophysiological index of the number of items held in a discrete-capacity WM resource. More recent findings indicate that these visual and tactile components are sensitive to various cognitive operations beyond the storage of discrete items in WM. In this Perspective, we present recent evidence from unisensory and multisensory visual and tactile WM tasks suggesting that, in addition to memory load, sensory delay activity may also be indicative of attentional and executive processes, as well as reflecting the flexible, rather than discrete, allocation of a continuous WM resource. Together, these findings challenge the traditional model of the functional significance of the contralateral delay activity as a pure measure of item load, and suggest that it may also reflect executive, attentional, and perceptual mechanisms operating in hierarchically organized WM systems.

A model of working memory for encoding multiple items and ordered sequences exploiting the theta-gamma code

Recent experimental evidence suggests that oscillatory activity plays a pivotal role in the maintenance of information in working memory, both in rodents and humans. In particular, cross-frequency coupling between theta and gamma oscillations has been suggested as a core mechanism for multi-item memory. The aim of this work is to present an original neural network model, based on oscillating neural masses, to investigate mechanisms at the basis of working memory in different conditions. We show that this model, with different synapse values, can be used to address different problems, such as the reconstruction of an item from partial information, the maintenance of multiple items simultaneously in memory, without any sequential order, and the reconstruction of an ordered sequence starting from an initial cue. The model consists of four interconnected layers; synapses are trained using Hebbian and anti-Hebbian mechanisms, in order to synchronize features in the same items, and desynchronize features in different items. Simulations show that the trained network is able to desynchronize up to nine items without a fixed order using the gamma rhythm. Moreover, the network can replicate a sequence of items using a gamma rhythm nested inside a theta rhythm. The reduction in some parameters, mainly concerning the strength of GABAergic synapses, induce memory alterations which mimic neurological deficits. Finally, the network, isolated from the external environment (“imagination phase”) and stimulated with high uniform noise, can randomly recover sequences previously learned, and link them together by exploiting the similarity among items.

A direct neural measure of variable precision representations in visual working memory

Visual working memory (VWM) is an active representation enabling the manipulation of item information even in the absence of visual input. A common way to investigate VWM is to analyze the performance at later recall. This approach, however, leaves uncertainties about whether the variation of recall performance is attributable to item encoding and maintenance or to the testing of memorized information. Here, we record the contralateral delay activity (CDA), an established electrophysiological measure of item storage and maintenance, in human subjects performing a delayed orientation precision estimation task. This allows us to link the fluctuation of recall precision directly to the process of item encoding and maintenance. We show that for two sequentially encoded orientation items, the CDA amplitude reflects the precision of orientation recall of both items, with higher precision being associated with a larger amplitude. Furthermore, we show that the CDA amplitudes for the items vary independently from each other, suggesting that the precision of memory representations fluctuates independently.NEW & NOTEWORTHY The present work demonstrates for the first time that the contralateral delay activity (CDA), an online electrophysiological measure of the number of representations maintained in memory, is also a reliable measure of the precision of memory representations. Furthermore, we show that the CDA fluctuates independently for individual items held in memory, thereby providing unambiguous direct neurophysiological support for independently fluctuating memory representations.

Topological change induces an interference effect in visual working memory

The “irrelevant-change distracting effect” refers to the effect of changes in irrelevant features on the performance of the target feature, which has frequently been used to study information processing in visual working memory (VWM). In the current study, we reported a novel interference effect in VWM: the topological-change interference effect (TCIE). In a series of six experiments, we examined the influence of topological and nontopological changes as irrelevant features on VWM using a color change detection paradigm. The results revealed that only topological changes, although task irrelevant, could produce a significant interference effect. In contrast, nontopological changes did not produce any evident interference effect. Moreover, the TCIE was a stable and lasting effect, regardless of changes in locations, reporting methods, particular stimulus figures, the other salient feature dimensions and delay interval times. Therefore, our results support the notion that topological invariance that defines perceptual objects plays an essential role in maintaining representations in VWM.

Controlling the Flow of Distracting Information in Working Memory

Visual working memory (WM) must maintain relevant information, despite the constant influx of both relevant and irrelevant information. Attentional control mechanisms help determine which of this new information gets access to our capacity-limited WM system. Previous work has treated attentional control as a monolithic process-either distractors capture attention or they are suppressed. Here, we provide evidence that attentional capture may instead be broken down into at least two distinct subcomponent processes: (1) Spatial capture, which refers to when spatial attention shifts towards the location of irrelevant stimuli and (2) item-based capture, which refers to when item-based WM representations of irrelevant stimuli are formed. To dissociate these two subcomponent processes of attentional capture, we utilized a series of electroencephalography components that track WM maintenance (contralateral delay activity), suppression (distractor positivity), item individuation (N2pc), and spatial attention (lateralized alpha power). We show that new, relevant information (i.e., a task-relevant distractor) triggers both spatial and item-based capture. Irrelevant distractors, however, only trigger spatial capture from which ongoing WM representations can recover more easily. This fractionation of attentional capture into distinct subcomponent processes provides a refined framework for understanding how distracting stimuli affect attention and WM.

Visual working memory representation as a topological defined perceptual object

The question of what the basic unit is of visual working memory remains one of the most fundamental and controversial issues. In the current study, we proposed a unique perspective based on early topological perception to describe the nature of representation in visual working memory. In a series of updating change-detection tasks, the repetition-benefit effect on color memory was not affected when items in the second memory array underwent massive changes of nontopological features from the first memory array. However, when the topological properties of an item changed, the repetition-benefit effect was destroyed, suggesting that the item was perceived as a new object impairing the original memory. Hence, our results suggest that a perceptual object defined by its topological invariance might be a unique perspective from which to describe representations of visual working memory.

Neural evidence for an object-based pointer system underlying working memory

To accomplish even rudimentary tasks, our cognitive system must update its representation of the changing environment. This process relies on visual working memory (VWM), which can actively modify its representations. We argue that this ability depends on a pointer system, such that each representation is stably and uniquely mapped to a specific stimulus. Without these pointers, VWM representations are inaccessible and therefore unusable. In three Electroencephalogram (EEG) experiments, we examined whether the pointers are allocated in an object-based, featural, or spatial manner: three factors that were confounded in previous studies. We used a feature change-detection task, in which objects moved and could separate into independently-moving parts. Despite the movement and separation being completely task-irrelevant, we found that the separation invalidated the pointers. This happened in a shape task, where the separation changed both the objects and the task-relevant features, but importantly, also in a color task, where the separation destroyed the objects while leaving the task-relevant features intact. Furthermore, even in a color task where all items had identical shapes, object-separation invalidated the pointers. This suggests that objects and not task-relevant features underlie the pointer system. Finally, when each object-part could be individuated already before the separation, the pointers were maintained, suggesting that the pointers are specifically tied to objects rather than locations. These results shed new light on the pointers which underlie VWM performance, demonstrating that the pointer system is object-based regardless of the task requirements.

Dissecting the Neural Focus of Attention Reveals Distinct Processes for Spatial Attention and Object-Based Storage in Visual Working Memory

Complex cognition relies on both on-line representations in working memory (WM), said to reside in the focus of attention, and passive off-line representations of related information. Here, we dissected the focus of attention by showing that distinct neural signals index the on-line storage of objects and sustained spatial attention. We recorded electroencephalogram (EEG) activity during two tasks that employed identical stimulus displays but varied the relative demands for object storage and spatial attention. We found distinct delay-period signatures for an attention task (which required only spatial attention) and a WM task (which invoked both spatial attention and object storage). Although both tasks required active maintenance of spatial information, only the WM task elicited robust contralateral delay activity that was sensitive to mnemonic load. Thus, we argue that the focus of attention is maintained via a collaboration between distinct processes for covert spatial orienting and object-based storage.

For whom is social-network usage associated with anxiety? The moderating role of neural working-memory filtering of Facebook information

Is Facebook usage bad for mental health? Existing studies provide mixed results, and direct evidence for neural underlying moderators is lacking. We suggest that being able to filter social-network information from accessing working memory is essential to preserve limited cognitive resources to pursue relevant goals. Accordingly, among individuals with impaired neural social-network filtering ability, enhanced social-network usage would be associated with negative mental health. Specifically, participants performed a novel electrophysiological paradigm that isolates neural Facebook filtering ability. Participants' actual Facebook behavior and anxious symptomatology were assessed. Confirming evidence showed that enhanced Facebook usage was associated with anxious symptoms among individuals with impaired neural Facebook filtering ability. Although less robust and tentative, additional suggestive evidence indicated that this specific Facebook filtering impairment was not better explained by a general filtering deficit. These results involving a neural social-network filtering moderator, may help understand for whom increased online social-network usage is associated with negative mental health.

Delineating resetting and updating in visual working memory based on the object-to-representation correspondence

When an object we represent in visual working memory (VWM) changes, its representation is modified accordingly. VWM can either access and change the existing representation by an updating process, or it can reset, by encoding the object in its novel status as a new representation. Our goal was to show that the determining factor of updating versus resetting is the availability of a stable correspondence between the object and its VWM representation. Here, we demonstrate that updating relies on the object-to-representation mapping to access and modify the appropriate representation, while losing this mapping triggers a resetting process. We compared very similar situations of object separation that either allowed the mapping to hold, or caused it to be lost. When an object that was mapped to one representation separated, VWM reset, manifested by a sharp drop in the contralateral delay activity (CDA) amplitude (an electrophysiological marker of VWM contents; Experiment 1), and a behavioral cost to detect salient changes that co-occurred with the resetting-triggering event (Experiment 2). When each part was mapped to a different representation, the separation resulted in updating, with a gradual rise in CDA amplitude (Experiment 1), and a reduced behavioral cost (Experiment 2). Thus, while updating and resetting resulted in similar final representations (corresponding to the post-change objects), their dynamics were different, depending on the availability of the mapping. Our results reveal the triggering conditions of resetting and updating, establish methods to study these online processes, and highlight the importance of the object-to-representation correspondence in VWM.

Visual working memory can selectively reset a subset of its representations

The visual working memory (VWM) resetting process is triggered when the mapping between an object in the environment and its corresponding VWM representation becomes irrelevant. Resetting involves discarding the no longer relevant representations, and encoding novel representations and mappings. We examined how resetting operates on VWM's contents. Specifically, we tested whether losing only part of the encoded mappings led to resetting all of the VWM representations. Subjects monitored moving polygons for an abrupt shape-change. Occasionally, a polygon separated into two halves that continued to move independently, making the original single mapping irrelevant. This loss of mapping triggered a resetting process, producing a performance cost: subjects missed shape-changes when they occurred during resetting, but not when the changes occurred before or after resetting. Critically, the cost was (1) specific to the separated item, (2) larger when more mappings were lost, and (3) unaffected by the set-size. This suggests that resetting is a "local" process: VWM removes only the representations whose mappings are lost.