Fundamental bound on the persistence and capacity of short-term memory stored as graded persistent activity

The neural basis of swap errors in working memory

When making decisions in a cluttered world, humans and other animals often have to hold multiple items in memory at once-such as the different items on a shopping list. Psychophysical experiments in humans and other animals have shown remembered stimuli can sometimes become confused, with participants reporting chimeric stimuli composed of features from different stimuli. In particular, subjects will often make "swap errors" where they misattribute a feature from one object as belonging to another object. While swap errors have been described behaviorally and theoretical explanations have been proposed, their neural mechanisms are unknown. Here, we elucidate these neural mechanisms by analyzing neural population recordings from monkeys performing two multistimulus working memory tasks. In these tasks, monkeys were cued to report the color of an item that either was previously shown at a corresponding location or will be shown at the corresponding location. Animals made swap errors in both tasks. In the neural data, we find evidence that the neural correlates of swap errors emerged when correctly remembered information is selected from working memory. This led to a representation of the distractor color as if it were the target color, underlying the eventual swap error. We did not find consistent evidence that swap errors arose from misinterpretation of the cue or errors during encoding or storage in working memory. These results provide evidence that swap errors emerge during selection of correctly remembered information from working memory, and highlight this selection as a crucial-yet surprisingly brittle-neural process.

Representation and computation in visual working memory

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

A dynamic neural resource model bridges sensory and working memory

Probing memory of a complex visual image within a few hundred milliseconds after its disappearance reveals significantly greater fidelity of recall than if the probe is delayed by as little as a second. Classically interpreted, the former taps into a detailed but rapidly decaying visual sensory or 'iconic' memory (IM), while the latter relies on capacity-limited but comparatively stable visual working memory (VWM). While iconic decay and VWM capacity have been extensively studied independently, currently no single framework quantitatively accounts for the dynamics of memory fidelity over these time scales. Here, we extend a stationary neural population model of VWM with a temporal dimension, incorporating rapid sensory-driven accumulation of activity encoding each visual feature in memory, and a slower accumulation of internal error that causes memorized features to randomly drift over time. Instead of facilitating read-out from an independent sensory store, an early cue benefits recall by lifting the effective limit on VWM signal strength imposed when multiple items compete for representation, allowing memory for the cued item to be supplemented with information from the decaying sensory trace. Empirical measurements of human recall dynamics validate these predictions while excluding alternative model architectures. A key conclusion is that differences in capacity classically thought to distinguish IM and VWM are in fact contingent upon a single resource-limited WM store.

The neural basis of swap errors in working memory

When making decisions in a cluttered world, humans and other animals often have to hold multiple items in memory at once - such as the different items on a shopping list. Psychophysical experiments in humans and other animals have shown remembered stimuli can sometimes become confused, with participants reporting chimeric stimuli composed of features from different stimuli. In particular, subjects will often make "swap errors" where they misattribute a feature from one object as belonging to another object. While swap errors have been described behaviorally, their neural mechanisms are unknown. Here, we elucidate these neural mechanisms through trial-by-trial analysis of neural population recordings from posterior and frontal brain regions while monkeys perform two multi-stimulus working memory tasks. In these tasks, monkeys were cued to report the color of an item that either was previously shown at a corresponding location (requiring selection from working memory) or will be shown at the corresponding location (requiring attention to a position). Animals made swap errors in both tasks. In the neural data, we find evidence that the neural correlates of swap errors emerged when correctly remembered information is selected incorrectly from working memory. This led to a representation of the distractor color as if it were the target color, underlying the eventual swap error. We did not find consistent evidence that swap errors arose from misinterpretation of the cue or errors during encoding or storage in working memory. These results suggest an alternative to established views on the neural origins of swap errors, and highlight selection from and manipulation in working memory as crucial - yet surprisingly brittle - neural processes.

Self-Organization of Nonlinearly Coupled Neural Fluctuations Into Synergistic Population Codes

Neural activity in the brain exhibits correlated fluctuations that may strongly influence the properties of neural population coding. However, how such correlated neural fluctuations may arise from the intrinsic neural circuit dynamics and subsequently affect the computational properties of neural population activity remains poorly understood. The main difficulty lies in resolving the nonlinear coupling between correlated fluctuations with the overall dynamics of the system. In this study, we investigate the emergence of synergistic neural population codes from the intrinsic dynamics of correlated neural fluctuations in a neural circuit model capturing realistic nonlinear noise coupling of spiking neurons. We show that a rich repertoire of spatial correlation patterns naturally emerges in a bump attractor network and further reveals the dynamical regime under which the interplay between differential and noise correlations leads to synergistic codes. Moreover, we find that negative correlations may induce stable bound states between two bumps, a phenomenon previously unobserved in firing rate models. These noise-induced effects of bump attractors lead to a number of computational advantages including enhanced working memory capacity and efficient spatiotemporal multiplexing and can account for a range of cognitive and behavioral phenomena related to working memory. This study offers a dynamical approach to investigating realistic correlated neural fluctuations and insights to their roles in cortical computations.

Considering Readout to Understand Working Memory

While working memory (WM) allows us to store past information, its function is to guide future behavior. Given this role, the tight link between how WMs are maintained and how they are read out to be transformed into context-appropriate actions remains relatively unexplored. Beyond helping us understand memory-guided behavior, focusing on WM readout may also help us better understand the neural basis of memory maintenance.

Cortical feedback loops bind distributed representations of working memory

Working memory—the brain’s ability to internalize information and use it flexibly to guide behaviour—is an essential component of cognition. Although activity related to working memory has been observed in several brain regions^1–3, how neural populations actually represent working memory^4–7 and the mechanisms by which this activity is maintained^8–12 remain unclear^13–15. Here we describe the neural implementation of visual working memory in mice alternating between a delayed non-match-to-sample task and a simple discrimination task that does not require working memory but has identical stimulus, movement and reward statistics. Transient optogenetic inactivations revealed that distributed areas of the neocortex were required selectively for the maintenance of working memory. Population activity in visual area AM and premotor area M2 during the delay period was dominated by orderly low-dimensional dynamics^16,17 that were, however, independent of working memory. Instead, working memory representations were embedded in high-dimensional population activity, present in both cortical areas, persisted throughout the inter-stimulus delay period, and predicted behavioural responses during the working memory task. To test whether the distributed nature of working memory was dependent on reciprocal interactions between cortical regions^18–20, we silenced one cortical area (AM or M2) while recording the feedback it received from the other. Transient inactivation of either area led to the selective disruption of inter-areal communication of working memory. Therefore, reciprocally interconnected cortical areas maintain bound high-dimensional representations of working memory.

Experiments in mice alternating between a visual working memory task and a task that is independent of working memory provide insight into the neural representation of working memory and the distributed nature of its maintenance.

Strategy-dependent effects of working-memory limitations on human perceptual decision-making

Deliberative decisions based on an accumulation of evidence over time depend on working memory, and working memory has limitations, but how these limitations affect deliberative decision-making is not understood. We used human psychophysics to assess the impact of working-memory limitations on the fidelity of a continuous decision variable. Participants decided the average location of multiple visual targets. This computed, continuous decision variable degraded with time and capacity in a manner that depended critically on the strategy used to form the decision variable. This dependence reflected whether the decision variable was computed either: (1) immediately upon observing the evidence, and thus stored as a single value in memory; or (2) at the time of the report, and thus stored as multiple values in memory. These results provide important constraints on how the brain computes and maintains temporally dynamic decision variables.

Working memory, the brain’s ability to temporarily store and recall information, is a critical part of decision making – but it has its limits. The brain can only store so much information, for so long. Since decisions are not often acted on immediately, information held in working memory ‘degrades’ over time. However, it is unknown whether or not this degradation of information over time affects the accuracy of later decisions.

The tactics that people use, knowingly or otherwise, to store information in working memory also remain unclear. Do people store pieces of information such as numbers, objects and particular details? Or do they tend to compute that information, make some preliminary judgement and recall their verdict later? Does the strategy chosen impact people’s decision-making?

To investigate, Schapiro et al. devised a series of experiments to test whether the limitations of working memory, and how people store information, affect the accuracy of decisions they make. First, participants were shown an array of colored discs on a screen. Then, either immediately after seeing the disks or a few seconds later, the participants were asked to recall the position of one of the disks they had seen, or the average position of all the disks. This measured how much information degraded for a decision based on multiple items, and how much for a decision based on a single item. From this, the method of information storage used to make a decision could be inferred.

Schapiro et al. found that the accuracy of people’s responses worsened over time, whether they remembered the position of each individual disk, or computed their average location before responding. The greater the delay between seeing the disks and reporting their location, the less accurate people’s responses tended to be. Similarly, the more disks a participant saw, the less accurate their response became. This suggests that however people store information, if working memory reaches capacity, decision-making suffers and that, over time, stored information decays.

Schapiro et al. also noticed that participants remembered location information in different ways depending on the task and how many disks they were shown at once. This suggests people adopt different strategies to retain information momentarily.

In summary, these findings help to explain how people process and store information to make decisions and how the limitations of working memory impact their decision-making ability. A better understanding of how people use working memory to make decisions may also shed light on situations or brain conditions where decision-making is impaired.

Toroidal topology of population activity in grid cells

The medial entorhinal cortex is part of a neural system for mapping the position of an individual within a physical environment¹. Grid cells, a key component of this system, fire in a characteristic hexagonal pattern of locations², and are organized in modules³ that collectively form a population code for the animal’s allocentric position¹. The invariance of the correlation structure of this population code across environments^4,5 and behavioural states^6,7, independent of specific sensory inputs, has pointed to intrinsic, recurrently connected continuous attractor networks (CANs) as a possible substrate of the grid pattern^1,8–11. However, whether grid cell networks show continuous attractor dynamics, and how they interface with inputs from the environment, has remained unclear owing to the small samples of cells obtained so far. Here, using simultaneous recordings from many hundreds of grid cells and subsequent topological data analysis, we show that the joint activity of grid cells from an individual module resides on a toroidal manifold, as expected in a two-dimensional CAN. Positions on the torus correspond to positions of the moving animal in the environment. Individual cells are preferentially active at singular positions on the torus. Their positions are maintained between environments and from wakefulness to sleep, as predicted by CAN models for grid cells but not by alternative feedforward models¹². This demonstration of network dynamics on a toroidal manifold provides a population-level visualization of CAN dynamics in grid cells.

Simultaneous recordings from hundreds of grid cells in rats, combined with topological data analysis, show that network activity in grid cells resides on a toroidal manifold that is invariant across environments and brain states.

Understanding occipital and parietal contributions to visual working memory: Commentary on Xu (2020)

In her commentary, Xu (2020) admonishes the reader that "To have a full understanding of the cognitive mechanisms underlying VWM [visual working memory], both behavioral and neural evidence needs to be taken into account. This is a must, and not a choice, for any study that attempts to capture the nature of VWM" (p. 11). Although we don't disagree with this statement, our overall assessment of this commentary is that it, itself, fails to satisfy several "musts" and, consequently, does not pose a serious challenge for the sensory recruitment framework for understanding visual working memory. These "musts" include accurately characterizing the framework being critiqued, not favoring verbal models and intuition at the expense of formal quantitative models, and providing even-handed interpretation of the work of others. We'll conclude with a summary of how the sensory recruitment framework can be incorporated into a broader working model of visual working memory.

Entropy estimation within in vitro neural-astrocyte networks as a measure of development instability

The brain demands a significant fraction of the energy budget in an organism; in humans, it accounts for 2% of the body mass, but utilizes 20% of the total energy metabolized. This is due to the large load required for information processing; spiking demands from neurons are high but are a key component to understanding brain functioning. Astrocytic brain cells contribute to the healthy functioning of brain circuits by mediating neuronal network energy and facilitating the formation and stabilization of synaptic connectivity. During development, spontaneous activity influences synaptic formation, shaping brain circuit construction, and adverse astrocyte mutations can lead to pathological processes impacting cognitive impairment due to inefficiencies in network spiking activity. We have developed a measure that quantifies information stability within in vitro networks consisting of mixed neural-astrocyte cells. Brain cells were harvested from mice with mutations to a gene associated with the strongest known genetic risk factor for Alzheimer's disease, APOE. We calculate energy states of the networks and using these states, we present an entropy-based measure to assess changes in information stability over time. We show that during development, stability profiles of spontaneous network activity are modified by exogenous astrocytes and that network stability, in terms of the rate of change of entropy, is allele dependent.

Different states of priority recruit different neural representations in visual working memory

We used functional magnetic resonance imaging (fMRI) to investigate the neural codes for representing stimulus information held in different states of priority in working memory. Human participants (male and female) performed delayed recall for 2 oriented gratings that could appear in any of several locations. Priority status was manipulated by a retrocue, such that one became the prioritized memory item (PMI) and another the unprioritized memory item (UMI). Using inverted encoding models (IEMs), we found that, in early visual cortex, the orientation of the UMI was represented in a neural representation that was rotated relative to the PMI. In intraparietal sulcus (IPS), we observed the analogous effect for the representation of the location of the UMI. Taken together, these results provide evidence for a common remapping mechanism that may be responsible for representing stimulus identity and stimulus context with different levels of priority in working memory.

Nonlinear mixed selectivity supports reliable neural computation

Neuronal activity in the brain is variable, yet both perception and behavior are generally reliable. How does the brain achieve this? Here, we show that the conjunctive coding of multiple stimulus features, commonly known as nonlinear mixed selectivity, may be used by the brain to support reliable information transmission using unreliable neurons. Nonlinearly mixed feature representations have been observed throughout primary sensory, decision-making, and motor brain areas. In these areas, different features are almost always nonlinearly mixed to some degree, rather than represented separately or with only additive (linear) mixing, which we refer to as pure selectivity. Mixed selectivity has been previously shown to support flexible linear decoding for complex behavioral tasks. Here, we show that it has another important benefit: in many cases, it makes orders of magnitude fewer decoding errors than pure selectivity even when both forms of selectivity use the same number of spikes. This benefit holds for sensory, motor, and more abstract, cognitive representations. Further, we show experimental evidence that mixed selectivity exists in the brain even when it does not enable behaviorally useful linear decoding. This suggests that nonlinear mixed selectivity may be a general coding scheme exploited by the brain for reliable and efficient neural computation.

Neurons in the brain are unreliable, while both perception and behavior are generally reliable. In this work, we study how the neural population response to sensory, motor, and cognitive features can produce this reliability. Across the brain, single neurons have been shown to respond to particular conjunctions of multiple features, termed nonlinear mixed selectivity. In this work, we show that populations of these mixed selective neurons lead to many fewer decoding errors than populations without mixed selectivity, even when both neural codes are given the same number of spikes. We show that the reliability benefits from mixed selectivity are quite general, holding under different assumptions about metabolic costs and neural noise as well as for both categorical and sensory errors. Further, previous theoretical work has shown that mixed selectivity enables the learning of complex behaviors with simple decoders. Through the analysis of neural data, we show that the brain implements mixed selectivity even when it would not serve this purpose. Thus, we argue that the brain also implements mixed selectivity to exploit its general benefits for reliable and efficient neural computation.

Scale-invariant Characteristics of Forgetting: Toward a Unifying Account of Hippocampal Forgetting across Short and Long Timescales

After over 100 years of relative silence in the cognitive literature, recent advances in the study of the neural underpinnings of memory-specifically, the hippocampus-have led to a resurgence of interest in the topic of forgetting. This review draws a theoretically driven picture of the effects of time on forgetting of hippocampus-dependent memories. We review evidence indicating that time-dependent forgetting across short and long timescales is reflected in progressive degradation of hippocampal-dependent relational information. This evidence provides an important extension to a growing body of research accumulated in recent years, showing that-in contrast to the once prevailing view that the hippocampus is exclusively involved in memory and forgetting over long timescales-the role of the hippocampus also extends to memory and forgetting over short timescales. Thus, we maintain that similar rules govern not only remembering but also forgetting of hippocampus-dependent information over short and long timescales.

Error-correcting dynamics in visual working memory

Working memory is critical to cognition, decoupling behavior from the immediate world. Yet, it is imperfect; internal noise introduces errors into memory representations. Such errors have been shown to accumulate over time and increase with the number of items simultaneously held in working memory. Here, we show that discrete attractor dynamics mitigate the impact of noise on working memory. These dynamics pull memories towards a few stable representations in mnemonic space, inducing a bias in memory representations but reducing the effect of random diffusion. Model-based and model-free analyses of human and monkey behavior show that discrete attractor dynamics account for the distribution, bias, and precision of working memory reports. Furthermore, attractor dynamics are adaptive. They increase in strength as noise increases with memory load and experiments in humans show these dynamics adapt to the statistics of the environment, such that memories drift towards contextually-predicted values. Together, our results suggest attractor dynamics mitigate errors in working memory by counteracting noise and integrating contextual information into memories.

Neural representations in working memory are susceptible to internal noise, which scales with memory load. Here, the authors show that attractor dynamics mitigate the influence of internal noise by pulling memories towards a few stable representations.

Intrinsic neuronal dynamics predict distinct functional roles during working memory

Working memory (WM) is characterized by the ability to maintain stable representations over time; however, neural activity associated with WM maintenance can be highly dynamic. We explore whether complex population coding dynamics during WM relate to the intrinsic temporal properties of single neurons in lateral prefrontal cortex (lPFC), the frontal eye fields (FEF), and lateral intraparietal cortex (LIP) of two monkeys (Macaca mulatta). We find that cells with short timescales carry memory information relatively early during memory encoding in lPFC; whereas long-timescale cells play a greater role later during processing, dominating coding in the delay period. We also observe a link between functional connectivity at rest and the intrinsic timescale in FEF and LIP. Our results indicate that individual differences in the temporal processing capacity predict complex neuronal dynamics during WM, ranging from rapid dynamic encoding of stimuli to slower, but stable, maintenance of mnemonic information.

Prefrontal neurons exhibit both transient and persistent firing in working memory tasks. Here the authors report that the intrinsic timescale of neuronal firing outside the task is predictive of the temporal dynamics of coding during working memory in three frontoparietal brain areas.

Motivation dynamically increases noise resistance by internal feedback during movement

Motivation improves performance, pushing us beyond our normal limits. One general explanation for this is that the effects of neural noise can be reduced, at a cost. If this were possible, reward would promote investment in resisting noise. But how could the effects of noise be attenuated, and why should this be costly? Negative feedback may be employed to compensate for disturbances in a neural representation. Such feedback would increase the robustness of neural representations to internal signal fluctuations, producing a stable attractor. We propose that encoding this negative feedback in neural signals would incur additional costs proportional to the strength of the feedback signal. We use eye movements to test the hypothesis that motivation by reward improves precision by increasing the strength of internal negative feedback. We find that reward simultaneously increases the amplitude, velocity and endpoint precision of saccades, indicating true improvement in oculomotor performance. Analysis of trajectories demonstrates that variation in the eye position during the course of saccades is predictive of the variation of endpoints, but this relation is reduced by reward. This indicates that motivation permits more aggressive correction of errors during the saccade, so that they no longer affect the endpoint. We suggest that such increases in internal negative feedback allow attractor stability, albeit at a cost, and therefore may explain how motivation improves cognitive as well as motor precision.

•

Motivation can increase speed and reduce behavioural variability.

•

This requires stabilising neural representations so they are robust to noise.

•

Stable representations or attractors in neural systems may come at the cost of stronger negative feedback.

•

Examination of trajectory correlations demonstrates that reward increases negative feedback.

•

We propose that the cost of stabilising signals explain why effort is expensive.

Drift in Neural Population Activity Causes Working Memory to Deteriorate Over Time

Short-term memories are thought to be maintained in the form of sustained spiking activity in neural populations. Decreases in recall precision observed with increasing number of memorized items can be accounted for by a limit on total spiking activity, resulting in fewer spikes contributing to the representation of each individual item. Longer retention intervals likewise reduce recall precision, but it is unknown what changes in population activity produce this effect. One possibility is that spiking activity becomes attenuated over time, such that the same mechanism accounts for both effects of set size and retention duration. Alternatively, reduced performance may be caused by drift in the encoded value over time, without a decrease in overall spiking activity. Human participants of either sex performed a variable-delay cued recall task with a saccadic response, providing a precise measure of recall latency. Based on a spike integration model of decision making, if the effects of set size and retention duration are both caused by decreased spiking activity, we would predict a fixed relationship between recall precision and response latency across conditions. In contrast, the drift hypothesis predicts no systematic changes in latency with increasing delays. Our results show both an increase in latency with set size, and a decrease in response precision with longer delays within each set size, but no systematic increase in latency for increasing delay durations. These results were quantitatively reproduced by a model based on a limited neural resource in which working memories drift rather than decay with time.

SIGNIFICANCE STATEMENT Rapid deterioration over seconds is a defining feature of short-term memory, but what mechanism drives this degradation of internal representations? Here, we extend a successful population coding model of working memory by introducing possible mechanisms of delay effects. We show that a decay in neural signal over time predicts that the time required for memory retrieval will increase with delay, whereas a random drift in the stored value predicts no effect of delay on retrieval time. Testing these predictions in a multi-item memory task with an eye movement response, we identified drift as a key mechanism of memory decline. These results provide evidence for a dynamic spiking basis for working memory, in contrast to recent proposals of activity-silent storage.