Swap errors in visual working memory are fully explained by cue-feature variability

Correlative comparison of visual working memory paradigms and associated models

When studying the working memory (WM), the 'slot model' and the 'resource model' are two main theories used to describe how information retention occurs. The slot model shows that WM capacity consists of a certain number of predefined slots available for information storage. This theory explains that there is a binary condition during information recall in which information is either wholly maintained within a slot or forgotten. The resource model has a resolution-based approach, suggesting a continuous resource able to be distributed among a number of items in WM capacity. Recently hybrid models have been introduced, suggesting that WM may not strictly conform to only one model. Accordingly, to understand the relationship between two of the most widely used paradigms in WM evaluation, we implemented a correlational assessment in two different psychophysics tasks, an analog recall paradigm with sequential bar presentation and a delayed match-to-sample (DMS) task with checkerboard stimuli. Our study revealed significant correlations between WM performance in the DMS task and recall error, precision, and sources of errors in the sequential paradigm. Overall, the findings emphasize the importance of considering both tasks in understanding WM processes, as they shed light on the debate between the slot and resource models by revealing overlapping elements in both theories and the tasks used to evaluate WM capacity.

The neural basis of swap errors in working memory

When making decisions in a cluttered world, humans and other animals often have to hold multiple items in memory at once-such as the different items on a shopping list. Psychophysical experiments in humans and other animals have shown remembered stimuli can sometimes become confused, with participants reporting chimeric stimuli composed of features from different stimuli. In particular, subjects will often make "swap errors" where they misattribute a feature from one object as belonging to another object. While swap errors have been described behaviorally and theoretical explanations have been proposed, their neural mechanisms are unknown. Here, we elucidate these neural mechanisms by analyzing neural population recordings from monkeys performing two multistimulus working memory tasks. In these tasks, monkeys were cued to report the color of an item that either was previously shown at a corresponding location or will be shown at the corresponding location. Animals made swap errors in both tasks. In the neural data, we find evidence that the neural correlates of swap errors emerged when correctly remembered information is selected from working memory. This led to a representation of the distractor color as if it were the target color, underlying the eventual swap error. We did not find consistent evidence that swap errors arose from misinterpretation of the cue or errors during encoding or storage in working memory. These results provide evidence that swap errors emerge during selection of correctly remembered information from working memory, and highlight this selection as a crucial-yet surprisingly brittle-neural process.

Dissecting the components of error in analogue report tasks

Over the last two decades, the analogue report task has become a standard method for measuring the fidelity of visual representations across research domains including perception, attention, and memory. Despite its widespread use, there has been no methodical investigation of the different task parameters that might contribute to response variability. To address this gap, we conducted two experiments manipulating components of a typical analogue report test of memory for colour hue. We found that human response errors were independently affected by changes in storage and maintenance requirements of the task, demonstrated by a strong effect of set size even in the absence of a memory delay. In contrast, response variability remained unaffected by physical size of the colour wheel, implying negligible contribution of motor noise to task performance, or by its chroma radius, highlighting non-uniformity of the standard colour space. Comparing analogue report to a matched forced-choice task, we found variation in adjustment criterion made a limited contribution to analogue report variability, becoming meaningful only with low representational noise. Our findings validate the analogue report task as a robust measure of representational fidelity for most purposes, while also quantifying non-representational sources of noise that would limit its reliability in specialized settings.

Representation and computation in visual working memory

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

Alpha phase-coding supports feature binding during working memory maintenance

The ability to successfully retain and manipulate information in working memory (WM) requires that objects' individual features are bound into cohesive representations; yet, the mechanisms supporting feature binding remain unclear. Binding (or swap) errors, where memorized features are erroneously associated with the wrong object, can provide a window into the intrinsic limits in capacity of WM that represent a key bottleneck in our cognitive ability. We tested the hypothesis that binding in WM is accomplished via neural phase synchrony and that swap errors result from perturbations in this synchrony. Using magnetoencephalography data collected from human subjects in a task designed to induce swap errors, we showed that swaps are characterized by reduced phase-locked oscillatory activity during memory retention, as predicted by an attractor model of spiking neural networks. Further, we found that this reduction arises from increased phase-coding variability in the alpha-band over a distributed network of sensorimotor areas. Our findings demonstrate that feature binding in WM is accomplished through phase-coding dynamics that emerge from the competition between different memories.

A dynamic neural resource model bridges sensory and working memory

Probing memory of a complex visual image within a few hundred milliseconds after its disappearance reveals significantly greater fidelity of recall than if the probe is delayed by as little as a second. Classically interpreted, the former taps into a detailed but rapidly decaying visual sensory or 'iconic' memory (IM), while the latter relies on capacity-limited but comparatively stable visual working memory (VWM). While iconic decay and VWM capacity have been extensively studied independently, currently no single framework quantitatively accounts for the dynamics of memory fidelity over these time scales. Here, we extend a stationary neural population model of VWM with a temporal dimension, incorporating rapid sensory-driven accumulation of activity encoding each visual feature in memory, and a slower accumulation of internal error that causes memorized features to randomly drift over time. Instead of facilitating read-out from an independent sensory store, an early cue benefits recall by lifting the effective limit on VWM signal strength imposed when multiple items compete for representation, allowing memory for the cued item to be supplemented with information from the decaying sensory trace. Empirical measurements of human recall dynamics validate these predictions while excluding alternative model architectures. A key conclusion is that differences in capacity classically thought to distinguish IM and VWM are in fact contingent upon a single resource-limited WM store.

Stochastic attractor models of visual working memory

This paper investigates models of working memory in which memory traces evolve according to stochastic attractor dynamics. These models have previously been shown to account for response-biases that are manifest across multiple trials of a visual working memory task. Here we adapt this approach by making the stable fixed points correspond to the multiple items to be remembered within a single-trial, in accordance with standard dynamical perspectives of memory, and find evidence that this multi-item model can provide a better account of behavioural data from continuous-report tasks. Additionally, the multi-item model proposes a simple mechanism by which swap-errors arise: memory traces diffuse away from their initial state and are captured by the attractors of other items. Swap-error curves reveal the evolution of this process as a continuous function of time throughout the maintenance interval and can be inferred from experimental data. Consistent with previous findings, we find that empirical memory performance is not well characterised by a purely-diffusive process but rather by a stochastic process that also embodies error-correcting dynamics.

Mapping visual working memory models to a theoretical framework

The body of research on visual working memory (VWM)-the system often described as a limited memory store of visual information in service of ongoing tasks-is growing rapidly. The discovery of numerous related phenomena, and the many subtly different definitions of working memory, signify a challenge to maintain a coherent theoretical framework to discuss concepts, compare models and design studies. A lack of robust theory development has been a noteworthy concern in the psychological sciences, thought to be a precursor to the reproducibility crisis (Oberauer & Lewandowsky, Psychonomic Bulletin & Review, 26, 1596-1618, 2019). I review the theoretical landscape of the VWM field by examining two prominent debates-whether VWM is object-based or feature-based, and whether discrete-slots or variable-precision best describe VWM limits. I share my concerns about the dualistic nature of these debates and the lack of clear model specification that prevents fully determined empirical tests. In hopes of promoting theory development, I provide a working theory map by using the broadly encompassing memory for latent representations model (Hedayati et al., Nature Human Behaviour, 6, 5, 2022) as a scaffold for relevant phenomena and current theories. I illustrate how opposing viewpoints can be brought into accordance, situating leading models of VWM to better identify their differences and improve their comparison. The hope is that the theory map will help VWM researchers get on the same page-clarifying hidden intuitions and aligning varying definitions-and become a useful device for meaningful discussions, development of models, and definitive empirical tests of theories.

Probabilistic and rich individual working memories revealed by a betting game

When asked to remember a color, do people remember a point estimate (e.g., a particular shade of red), a point estimate plus an uncertainty estimate, or are memory representations rich probabilistic distributions over feature space? We asked participants to report the color of a circle held in working memory. Rather than collecting a single report per trial, we had participants place multiple bets to create trialwise uncertainty distributions. Bet dispersion correlated with performance, indicating that internal uncertainty guided bet placement. While the first bet was on average the most precisely placed, the later bets systematically shifted the distribution closer to the target, resulting in asymmetrical distributions about the first bet. This resulted in memory performance improvements when averaging across bets, and overall suggests that memory representations contain more information than can be conveyed by a single response. The later bets contained target information even when the first response would generally be classified as a guess or report of an incorrect item, suggesting that such failures are not all-or-none. This paradigm provides multiple pieces of evidence that memory representations are rich and probabilistic. Crucially, standard discrete response paradigms underestimate the amount of information in memory representations.

A mechanistic insight into sources of error of visual working memory in multiple sclerosis

Working memory (WM) is one of the most affected cognitive domains in multiple sclerosis (MS), which is mainly studied by the previously established binary model for information storage (slot model). However, recent observations based on the continuous reproduction paradigms have shown that assuming dynamic allocation of WM resources (resource model) instead of the binary hypothesis will give more accurate predictions in WM assessment. Moreover, continuous reproduction paradigms allow for assessing the distribution of error in recalling information, providing new insights into the organization of the WM system. Hence, by utilizing two continuous reproduction paradigms, memory-guided localization (MGL) and analog recall task with sequential presentation, we investigated WM dysfunction in MS. Our results demonstrated an overall increase in recall error and decreased recall precision in MS. While sequential paradigms were better in distinguishing healthy control from relapsing-remitting MS, MGL were more accurate in discriminating MS subtypes (relapsing-remitting from secondary progressive), providing evidence about the underlying mechanisms of WM deficit in progressive states of the disease. Furthermore, computational modeling of the results from the sequential paradigm determined that imprecision in decoding information and swap error (mistakenly reporting the feature of other presented items) was responsible for WM dysfunction in MS. Overall, this study offered a sensitive measure for assessing WM deficit and provided new insight into the organization of the WM system in MS population.

The neural basis of swap errors in working memory

When making decisions in a cluttered world, humans and other animals often have to hold multiple items in memory at once - such as the different items on a shopping list. Psychophysical experiments in humans and other animals have shown remembered stimuli can sometimes become confused, with participants reporting chimeric stimuli composed of features from different stimuli. In particular, subjects will often make "swap errors" where they misattribute a feature from one object as belonging to another object. While swap errors have been described behaviorally, their neural mechanisms are unknown. Here, we elucidate these neural mechanisms through trial-by-trial analysis of neural population recordings from posterior and frontal brain regions while monkeys perform two multi-stimulus working memory tasks. In these tasks, monkeys were cued to report the color of an item that either was previously shown at a corresponding location (requiring selection from working memory) or will be shown at the corresponding location (requiring attention to a position). Animals made swap errors in both tasks. In the neural data, we find evidence that the neural correlates of swap errors emerged when correctly remembered information is selected incorrectly from working memory. This led to a representation of the distractor color as if it were the target color, underlying the eventual swap error. We did not find consistent evidence that swap errors arose from misinterpretation of the cue or errors during encoding or storage in working memory. These results suggest an alternative to established views on the neural origins of swap errors, and highlight selection from and manipulation in working memory as crucial - yet surprisingly brittle - neural processes.

Redundant representations are required to disambiguate simultaneously presented complex stimuli

A pedestrian crossing a street during rush hour often looks and listens for potential danger. When they hear several different horns, they localize the cars that are honking and decide whether or not they need to modify their motor plan. How does the pedestrian use this auditory information to pick out the corresponding cars in visual space? The integration of distributed representations like these is called the assignment problem, and it must be solved to integrate distinct representations across but also within sensory modalities. Here, we identify and analyze a solution to the assignment problem: the representation of one or more common stimulus features in pairs of relevant brain regions-for example, estimates of the spatial position of cars are represented in both the visual and auditory systems. We characterize how the reliability of this solution depends on different features of the stimulus set (e.g., the size of the set and the complexity of the stimuli) and the details of the split representations (e.g., the precision of each stimulus representation and the amount of overlapping information). Next, we implement this solution in a biologically plausible receptive field code and show how constraints on the number of neurons and spikes used by the code force the brain to navigate a tradeoff between local and catastrophic errors. We show that, when many spikes and neurons are available, representing stimuli from a single sensory modality can be done more reliably across multiple brain regions, despite the risk of assignment errors. Finally, we show that a feedforward neural network can learn the optimal solution to the assignment problem, even when it receives inputs in two distinct representational formats. We also discuss relevant results on assignment errors from the human working memory literature and show that several key predictions of our theory already have support.

There Is no Theory-Free Measure of "Swaps" in Visual Working Memory Experiments

Visual working memory is highly limited, and its capacity is tied to many indices of cognitive function. For this reason, there is much interest in understanding its architecture and the sources of its limited capacity. As part of this research effort, researchers often attempt to decompose visual working memory errors into different kinds of errors, with different origins. One of the most common kinds of memory error is referred to as a "swap," where people report a value that closely resembles an item that was not probed (e.g., an incorrect, non-target item). This is typically assumed to reflect confusions, like location binding errors, which result in the wrong item being reported. Capturing swap rates reliably and validly is of great importance because it permits researchers to accurately decompose different sources of memory errors and elucidate the processes that give rise to them. Here, we ask whether different visual working memory models yield robust and consistent estimates of swap rates. This is a major gap in the literature because in both empirical and modeling work, researchers measure swaps without motivating their choice of swap model. Therefore, we use extensive parameter recovery simulations with three mainstream swap models to demonstrate how the choice of measurement model can result in very large differences in estimated swap rates. We find that these choices can have major implications for how swap rates are estimated to change across conditions. In particular, each of the three models we consider can lead to differential quantitative and qualitative interpretations of the data. Our work serves as a cautionary note to researchers as well as a guide for model-based measurement of visual working memory processes.