Reassessing the Evidence for Capacity Limits in Neural Signals Related to Working Memory

Representation and computation in visual working memory

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

Partial recall: Implications for the discrete slot limit of working memory capacity

The nature of working memory capacity (WMC) has been a highly contested topic among cognitive scientists. Some advocate for the discrete nature of this construct, fixed to a set number of independent slots, each capable of storing a single chunk of bound information. Others advocate for a continuous limit, guided by a pool of immediately available resources spent across the to-be-remembered items. To understand the nature of WMC, it was first essential to separate capacity from other factors, such as performance consistency, which may impact overall WM performance. Recent work by Schor et al., (2020, Psychonomic Bulletin & Review, 27[5], 1006-1013) has provided a method for separating these constructs within a single visual array task. The present study used this statistical model to extract partial information, defined as accurate recall of a correct color, but not location, at a rate greater than expected through guessing. The successful memory of this information would demonstrate that capacity does not rely on the existence of empty slots, which discrete slot model advocates argue, are necessary for successful storage and recall of items. The present study found that participants were able to successfully recall partial information at a rate significantly greater than chance, but not beyond the individual working memory capacity limit. These findings help provide additional support for the discrete resource slot model, while simultaneously casting doubt on its strong object slot model alternative.

Sensory Delay Activity: More than an Electrophysiological Index of Working Memory Load

Sustained contralateral delay activity emerges in the retention period of working memory (WM) tasks and has been commonly interpreted as an electrophysiological index of the number of items held in a discrete-capacity WM resource. More recent findings indicate that these visual and tactile components are sensitive to various cognitive operations beyond the storage of discrete items in WM. In this Perspective, we present recent evidence from unisensory and multisensory visual and tactile WM tasks suggesting that, in addition to memory load, sensory delay activity may also be indicative of attentional and executive processes, as well as reflecting the flexible, rather than discrete, allocation of a continuous WM resource. Together, these findings challenge the traditional model of the functional significance of the contralateral delay activity as a pure measure of item load, and suggest that it may also reflect executive, attentional, and perceptual mechanisms operating in hierarchically organized WM systems.

Storage in Visual Working Memory Recruits a Content-Independent Pointer System

Past work has shown that storage in working memory elicits stimulus-specific neural activity that tracks the stored content. Here, we present evidence for a distinct class of load-sensitive neural activity that indexes items without representing their contents per se. We recorded electroencephalogram (EEG) activity while adult human subjects stored varying numbers of items in visual working memory. Multivariate analysis of the scalp topography of EEG voltage enabled precise tracking of the number of individuated items stored and robustly predicted individual differences in working memory capacity. Critically, this signature of working memory load generalized across variations in both the type and number of visual features stored about each item, suggesting that it tracked the number of individuated memory representations and not the content of those memories. We hypothesize that these findings reflect the operation of a capacity-limited pointer system that supports on-line storage and attentive tracking.

Working Memory Swap Errors Have Identifiable Neural Representations

Working memory is an essential component of cognition that facilitates goal-directed behavior. Famously, it is severely limited and performance suffers when memory load exceeds an individual's capacity. Modeling of visual working memory responses has identified two likely types of errors: guesses and swaps. Swap errors may arise from a misbinding between the features of different items. Alternatively, these errors could arise from memory noise in the feature dimension used for cueing a to-be-tested memory item, resulting in the wrong item being selected. Finally, it is possible that so-called swap errors actually reflect informed guessing, which could occur at the time of a cue, or alternatively, at the time of the response. Here, we combined behavioral response modeling and fMRI pattern analysis to test the hypothesis that swap errors involve the active maintenance of an incorrect memory item. After the encoding of six spatial locations, a retro-cue indicated which location would be tested after memory retention. On accurate trials, we could reconstruct a memory representation of the cued location in both early visual cortex and intraparietal sulcus. On swap error trials identified with mixture modeling, we were able to reconstruct a representation of the swapped location, but not of the cued location, suggesting the maintenance of the incorrect memory item before response. Moreover, participants subjectively responded with some level of confidence, rather than complete guessing, on a majority of swap error trials. Together, these results suggest that swap errors are not mere response-phase guesses, but instead result from failures of selection in working memory, contextual binding errors, or informed guesses, which produce active maintenance of incorrect memory representations.

Memory Fidelity Reveals Qualitative Changes in Interactions Between Items in Visual Working Memory

Memory for objects in a display sometimes reveals attraction-the objects are remembered as more similar to one another than they actually were-and sometimes reveals repulsion-the objects are remembered as more different from one another. The conditions that lead to these opposing memory biases are poorly understood; there is no theoretical framework that explains these contrasting dynamics. In three experiments (each N = 30 adults), we demonstrate that memory fidelity provides a unifying dimension that accommodates the existence of both types of visual working memory interactions. We show that either attraction or repulsion can arise simply as a function of manipulations of memory fidelity. We also demonstrate that subjective ratings of fidelity predict the presence of attraction or repulsion on a trial-by-trial basis. We discuss how these results bear on computational models of visual working memory and contextualize these results within the literature of attraction and repulsion effects in long-term memory and perception.

The functional roles of neural remapping in cortex

Remapping is a property of some cortical and subcortical neurons that update their responses around the time of an eye movement to account for the shift of stimuli on the retina due to the saccade. Physiologically, remapping is traditionally tested by briefly presenting a single stimulus around the time of the saccade and looking at the onset of the response and the locations in space to which the neuron is responsive. Here we suggest that a better way to understand the functional role of remapping is to look at the time at which the neural signal emerges when saccades are made across a stable scene. Based on data obtained using this approach, we suggest that remapping in the lateral intraparietal area is sufficient to play a role in maintaining visual stability across saccades, whereas in the frontal eye field, remapped activity carries information that affects future saccadic choices and, in a separate subset of neurons, is used to maintain a map of locations in the scene that have been previously fixated.

Multivariate analysis reveals a generalizable human electrophysiological signature of working memory load

Working memory (WM) is an online memory system that is critical for holding information in a rapidly accessible state during ongoing cognitive processing. Thus, there is strong value in methods that provide a temporally resolved index of WM load. While univariate EEG signals have been identified that vary with WM load, recent advances in multivariate analytic approaches suggest that there may be rich sources of information that do not generate reliable univariate signatures. Here, using data from four published studies (n = 286 and >250,000 trials), we demonstrate that multivariate analysis of EEG voltage topography provides a sensitive index of the number of items stored in WM that generalizes to novel human observers. Moreover, multivariate load detection ("mvLoad") can provide robust information at the single-trial level, exceeding the sensitivity of extant univariate approaches. We show that this method tracks WM load in a manner that is (1) independent of the spatial position of the memoranda, (2) precise enough to differentiate item-by-item increments in the number of stored items, (3) generalizable across distinct tasks and stimulus displays, and (4) correlated with individual differences in WM behavior. Thus, this approach provides a powerful complement to univariate analytic approaches, enabling temporally resolved tracking of online memory storage in humans.

Reduced Attentional Control in Older Adults Leads to Deficits in Flexible Prioritization of Visual Working Memory

Visual working memory (VWM) resources have been shown to be flexibly distributed according to item priority. This flexible allocation of resources may depend on attentional control, an executive function known to decline with age. In this study, we sought to determine how age differences in attentional control affect VWM performance when attention is flexibly allocated amongst targets of varying priority. Participants performed a delayed-recall task wherein item priority was varied. Error was modelled using a three-component mixture model to probe different aspects of performance (precision, guess-rate, and non-target errors). The flexible resource model offered a good fit to the data from both age groups, but older adults showed consistently lower precision and higher guess rates. Importantly, when demands on flexible resource allocation were highest, older adults showed more non-target errors, often swapping in the item that had a higher priority at encoding. Taken together, these results suggest that the ability to flexibly allocate attention in VWM is largely maintained with age, but older adults are less precise overall and sometimes swap in salient, but no longer relevant, items possibly due to their lessened ability to inhibit previously attended information.

Theory of neural coding predicts an upper bound on estimates of memory variability

Observers reproducing elementary visual features from memory after a short delay produce errors consistent with the encoding-decoding properties of neural populations. While inspired by electrophysiological observations of sensory neurons in cortex, the population coding account of these errors is based on a mathematical idealization of neural response functions that abstracts away most of the heterogeneity and complexity of real neuronal populations. Here we examine a more physiologically grounded model based on the tuning of a large set of neurons recorded in macaque V1 and show that key predictions of the idealized model are preserved. Both models predict long-tailed distributions of error when memory resources are taxed, as observed empirically in behavioral experiments and commonly approximated with a mixture of normal and uniform error components. Specifically, for an idealized homogeneous neural population, the width of the fitted normal distribution cannot exceed the average tuning width of the component neurons, and this also holds to a good approximation for more biologically realistic populations. Examining eight published studies of orientation recall, we find a consistent pattern of results suggestive of a median tuning width of approximately 20°, which compares well with neurophysiological observations. The finding that estimates of variability obtained by the normal-plus-uniform mixture method are bounded from above leads us to reevaluate previous studies that interpreted a saturation in width of the normal component as evidence for fundamental limits on the precision of perception, working memory, and long-term memory.

Electrophysiological correlates of the flexible allocation of visual working memory resources

Visual working memory is a brief, capacity-limited store of visual information that is involved in a large number of cognitive functions. To guide one’s behavior effectively, one must efficiently allocate these limited memory resources across memory items. Previous research has suggested that items are either stored in memory or completely blocked from memory access. However, recent behavioral work proposes that memory resources can be flexibly split across items based on their level of task importance. Here, we investigated the electrophysiological correlates of flexible resource allocation by manipulating the distribution of resources amongst systematically lateralized memory items. We examined the contralateral delay activity (CDA), a waveform typically associated with the number of items held in memory. Across three experiments, we found that, in addition to memory load, the CDA flexibly tracks memory resource allocation. This allocation occurred as early as attentional selection, as indicated by the N2pc. Additionally, CDA amplitude was better-described when fit with a continuous model predicted by load and resources together than when fit with either alone. Our findings show that electrophysiological markers of attentional selection and memory maintenance not only track memory load, but also the proportion of memory resources those items receive.

Perceptual consciousness and cognitive access from the perspective of capacity-unlimited working memory

Theories of consciousness divide over whether perceptual consciousness is rich or sparse in specific representational content and whether it requires cognitive access. These two issues are often treated in tandem because of a shared assumption that the representational capacity of cognitive access is fairly limited. Recent research on working memory challenges this shared assumption. This paper argues that abandoning the assumption undermines post-cue-based 'overflow' arguments, according to which perceptual consciousness is rich and does not require cognitive access. Abandoning it also dissociates the rich/sparse debate from the access question. The paper then explores attempts to reformulate overflow theses in ways that do not require the assumption of limited capacity. Finally, it discusses the problem of relating seemingly non-probabilistic perceptual consciousness to the probabilistic representations posited by the models that challenge conceptions of cognitive access as capacity-limited.This article is part of the theme issue 'Perceptual consciousness and cognitive access'.

Distinct neural substrates for visual short-term memory of actions

Fundamental theories of human cognition have long posited that the short-term maintenance of actions is supported by one of the "core knowledge" systems of human visual cognition, yet its neural substrates are still not well understood. In particular, it is unclear whether the visual short-term memory (VSTM) of actions has distinct neural substrates or, as proposed by the spatio-object architecture of VSTM, shares them with VSTM of objects and spatial locations. In two experiments, we tested these two competing hypotheses by directly contrasting the neural substrates for VSTM of actions with those for objects and locations. Our results showed that the bilateral middle temporal cortex (MT) was specifically involved in VSTM of actions because its activation and its functional connectivity with the frontal-parietal network (FPN) were only modulated by the memory load of actions, but not by that of objects/agents or locations. Moreover, the brain regions involved in the maintenance of spatial location information (i.e., superior parietal lobule, SPL) was also recruited during the maintenance of actions, consistent with the temporal-spatial nature of actions. Meanwhile, the frontoparietal network (FPN) was commonly involved in all types of VSTM and showed flexible functional connectivity with the domain-specific regions, depending on the current working memory tasks. Together, our results provide clear evidence for a distinct neural system for maintaining actions in VSTM, which supports the core knowledge system theory and the domain-specific and domain-general architectures of VSTM.

Selective attention on representations in working memory: cognitive and neural mechanisms

Selective attention and working memory are inter-dependent core cognitive functions. It is critical to allocate attention on selected targets during the capacity-limited working memory processes to fulfill the goal-directed behavior. The trends of research on both topics are increasing exponentially in recent years, and it is considered that selective attention and working memory share similar underlying neural mechanisms. Different types of attention orientation in working memory are introduced by distinctive cues, and the means using retrospective cues are strengthened currently as it is manipulating the representation in memory, instead of the perceptual representation. The cognitive and neural mechanisms of the retro-cue effects are further reviewed, as well as the potential molecular mechanism. The frontal-parietal network that is involved in both attention and working memory is also the neural candidate for attention orientation during working memory. Neural oscillations in the gamma and alpha/beta oscillations may respectively be employed for the feedforward and feedback information transfer between the sensory cortices and the association cortices. Dopamine and serotonin systems might interact with each other subserving the communication between memory and attention. In conclusion, representations which attention shifts towards are strengthened, while representations which attention moves away from are degraded. Studies on attention orientation during working memory indicates the flexibility of the processes of working memory, and the beneficial way that overcome the limited capacity of working memory.