A probabilistic palimpsest model of visual short-term memory

The neural basis of swap errors in working memory

When making decisions in a cluttered world, humans and other animals often have to hold multiple items in memory at once-such as the different items on a shopping list. Psychophysical experiments in humans and other animals have shown remembered stimuli can sometimes become confused, with participants reporting chimeric stimuli composed of features from different stimuli. In particular, subjects will often make "swap errors" where they misattribute a feature from one object as belonging to another object. While swap errors have been described behaviorally and theoretical explanations have been proposed, their neural mechanisms are unknown. Here, we elucidate these neural mechanisms by analyzing neural population recordings from monkeys performing two multistimulus working memory tasks. In these tasks, monkeys were cued to report the color of an item that either was previously shown at a corresponding location or will be shown at the corresponding location. Animals made swap errors in both tasks. In the neural data, we find evidence that the neural correlates of swap errors emerged when correctly remembered information is selected from working memory. This led to a representation of the distractor color as if it were the target color, underlying the eventual swap error. We did not find consistent evidence that swap errors arose from misinterpretation of the cue or errors during encoding or storage in working memory. These results provide evidence that swap errors emerge during selection of correctly remembered information from working memory, and highlight this selection as a crucial-yet surprisingly brittle-neural process.

Electrophysiological hallmarks for event relations and event roles in working memory

The ability to maintain events (i.e., interactions between/among objects) in working memory is crucial for our everyday cognition, yet the format of this representation is poorly understood. The current ERP study was designed to answer two questions: How is maintaining events (e.g., the tiger hit the lion) neurally different from maintaining item coordinations (e.g., the tiger and the lion)? That is, how is the event relation (present in events but not coordinations) represented? And how is the agent, or initiator of the event encoded differently from the patient, or receiver of the event during maintenance? We used a novel picture-sentence match-across-delay approach in which the working memory representation was "pinged" during the delay, replicated across two ERP experiments with Chinese and English materials. We found that maintenance of events elicited a long-lasting late sustained difference in posterior-occipital electrodes relative to non-events. This effect resembled the negative slow wave reported in previous studies of working memory, suggesting that the maintenance of events in working memory may impose a higher cost compared to coordinations. Although we did not observe significant ERP differences associated with pinging the agent vs. the patient during the delay, we did find that the ping appeared to dampen the ongoing sustained difference, suggesting a shift from sustained activity to activity silent mechanisms. These results suggest a new method by which ERPs can be used to elucidate the format of neural representation for events in working memory.

Probabilistic and rich individual working memories revealed by a betting game

When asked to remember a color, do people remember a point estimate (e.g., a particular shade of red), a point estimate plus an uncertainty estimate, or are memory representations rich probabilistic distributions over feature space? We asked participants to report the color of a circle held in working memory. Rather than collecting a single report per trial, we had participants place multiple bets to create trialwise uncertainty distributions. Bet dispersion correlated with performance, indicating that internal uncertainty guided bet placement. While the first bet was on average the most precisely placed, the later bets systematically shifted the distribution closer to the target, resulting in asymmetrical distributions about the first bet. This resulted in memory performance improvements when averaging across bets, and overall suggests that memory representations contain more information than can be conveyed by a single response. The later bets contained target information even when the first response would generally be classified as a guess or report of an incorrect item, suggesting that such failures are not all-or-none. This paradigm provides multiple pieces of evidence that memory representations are rich and probabilistic. Crucially, standard discrete response paradigms underestimate the amount of information in memory representations.

The neural basis of swap errors in working memory

When making decisions in a cluttered world, humans and other animals often have to hold multiple items in memory at once - such as the different items on a shopping list. Psychophysical experiments in humans and other animals have shown remembered stimuli can sometimes become confused, with participants reporting chimeric stimuli composed of features from different stimuli. In particular, subjects will often make "swap errors" where they misattribute a feature from one object as belonging to another object. While swap errors have been described behaviorally, their neural mechanisms are unknown. Here, we elucidate these neural mechanisms through trial-by-trial analysis of neural population recordings from posterior and frontal brain regions while monkeys perform two multi-stimulus working memory tasks. In these tasks, monkeys were cued to report the color of an item that either was previously shown at a corresponding location (requiring selection from working memory) or will be shown at the corresponding location (requiring attention to a position). Animals made swap errors in both tasks. In the neural data, we find evidence that the neural correlates of swap errors emerged when correctly remembered information is selected incorrectly from working memory. This led to a representation of the distractor color as if it were the target color, underlying the eventual swap error. We did not find consistent evidence that swap errors arose from misinterpretation of the cue or errors during encoding or storage in working memory. These results suggest an alternative to established views on the neural origins of swap errors, and highlight selection from and manipulation in working memory as crucial - yet surprisingly brittle - neural processes.

Semi-orthogonal subspaces for value mediate a tradeoff between binding and generalization

When choosing between options, we must associate their values with the action needed to select them. We hypothesize that the brain solves this binding problem through neural population subspaces. To test this hypothesis, we examined neuronal responses in five reward-sensitive regions in macaques performing a risky choice task with sequential offers. Surprisingly, in all areas, the neural population encoded the values of offers presented on the left and right in distinct subspaces. We show that the encoding we observe is sufficient to bind the values of the offers to their respective positions in space while preserving abstract value information, which may be important for rapid learning and generalization to novel contexts. Moreover, after both offers have been presented, all areas encode the value of the first and second offers in orthogonal subspaces. In this case as well, the orthogonalization provides binding. Our binding-by-subspace hypothesis makes two novel predictions borne out by the data. First, behavioral errors should correlate with putative spatial (but not temporal) misbinding in the neural representation. Second, the specific representational geometry that we observe across animals also indicates that behavioral errors should increase when offers have low or high values, compared to when they have medium values, even when controlling for value difference. Together, these results support the idea that the brain makes use of semi-orthogonal subspaces to bind features together.

A spatially continuous diffusion model of visual working memory

We present results from five visual working memory (VWM) experiments in which participants were briefly shown between 2 and 6 colored squares. They were then cued to recall the color of one of the squares and they responded by choosing the color on a continuous color wheel. The experiments provided response proportions and response time (RT) measures as a function of angle for the choices. Current VWM models for this task include discrete models that assume an item is either within working memory or not and resource models that assume that memory strength varies as a function of the number of items. Because these models do not include processes that allow them to account for RT data, we implemented them within the spatially continuous diffusion model (SCDM, Ratcliff, 2018) and use the experimental data to evaluate these combined models. In the SCDM, evidence retrieved from memory is represented as a spatially continuous normal distribution and this drives the decision process until a criterion (represented as a 1-D line) is reached, which produces a decision. Noise in the accumulation process is represented by continuous Gaussian process noise over spatial position. The models that fit best from the discrete and resource-based classes converged on a common model that had a guessing component and that allowed the height of the normal memory-strength distribution to vary with number of items. The guessing component was implemented as a regular decision process driven by a flat evidence distribution, a zero-drift process. The combination of choice and RT data allows models that were not identifiable based on choice data alone to be discriminated.

Redundant representations are required to disambiguate simultaneously presented complex stimuli

A pedestrian crossing a street during rush hour often looks and listens for potential danger. When they hear several different horns, they localize the cars that are honking and decide whether or not they need to modify their motor plan. How does the pedestrian use this auditory information to pick out the corresponding cars in visual space? The integration of distributed representations like these is called the assignment problem, and it must be solved to integrate distinct representations across but also within sensory modalities. Here, we identify and analyze a solution to the assignment problem: the representation of one or more common stimulus features in pairs of relevant brain regions-for example, estimates of the spatial position of cars are represented in both the visual and auditory systems. We characterize how the reliability of this solution depends on different features of the stimulus set (e.g., the size of the set and the complexity of the stimuli) and the details of the split representations (e.g., the precision of each stimulus representation and the amount of overlapping information). Next, we implement this solution in a biologically plausible receptive field code and show how constraints on the number of neurons and spikes used by the code force the brain to navigate a tradeoff between local and catastrophic errors. We show that, when many spikes and neurons are available, representing stimuli from a single sensory modality can be done more reliably across multiple brain regions, despite the risk of assignment errors. Finally, we show that a feedforward neural network can learn the optimal solution to the assignment problem, even when it receives inputs in two distinct representational formats. We also discuss relevant results on assignment errors from the human working memory literature and show that several key predictions of our theory already have support.

Precision of phonological errors in aphasia supports resource models of phonological working memory in language production

Working memory (WM) is critical for many cognitive functions including language production. A key feature of WM is its capacity limitation. Two models have been proposed to account for such capacity limitation: slot models and resource models. In recent years, resource models have found support in both visual and auditory perception, but do they also extend to production? We investigate this by analyzing sublexical errors from four individuals with aphasia. Using tools from computational linguistics, we first define the concept of "precision" of sublexical errors. We then demonstrate that such precision decreases with increased working memory load, i.e., word length, as predicted by resource models. Finally, we rule out alternative accounts of this effect, such as articulatory simplification. These data provide the first evidence for the applicability of the resource model to production and further point to the generalizability of this account as a model of resource division in WM.

Swap errors in visual working memory are fully explained by cue-feature variability

In cue-based recall from working memory, incorrectly reporting features of an uncued item may be referred to as a “swap” error. One account of these errors ascribes them to variability in memory for the cue features leading to erroneous selection of a non-target item, especially if it is similar to the target in the cue-feature dimension. However, alternative accounts of swap errors include cue-independent misbinding, and strategic guessing when the cued item is not in memory. Here we investigated the cause of swap errors by manipulating the variability with which either cue or report features (orientations in Exp 1; motion directions in Exp 2) were encoded. We found that swap errors increased with increasing variability in memory for the cue features, and their changing frequency could be quantitatively predicted based on recall variability when the same feature was used for report. These results are inconsistent with the hypothesis that swaps are a strategic response to forgotten items, and suggest that swap errors could be wholly accounted for by confusions due to cue-dimension variability. In a third experiment we examined whether spatial configuration of memory arrays in tasks with spatial cueing has an influence on swap error frequency. We observed a specific tendency to make swap errors to non-targets located precisely opposite to the cued location, suggesting that stimulus positions are partially encoded in a non-metric format.

Palimpsest memories stored in memristive synapses

Biological synapses store multiple memories on top of each other in a palimpsest fashion and at different time scales. Palimpsest consolidation is facilitated by the interaction of hidden biochemical processes governing synaptic efficacy during varying lifetimes. This arrangement allows idle memories to be temporarily overwritten without being forgotten, while previously unseen memories are used in the short term. While embedded artificial intelligence can greatly benefit from this functionality, a practical demonstration in hardware is missing. Here, we show how the intrinsic properties of metal-oxide volatile memristors emulate the processes supporting biological palimpsest consolidation. Our memristive synapses exhibit an expanded doubled capacity and protect a consolidated memory while up to hundreds of uncorrelated short-term memories temporarily overwrite it, without requiring specialized instructions. We further demonstrate this technology in the context of visual working memory. This showcases how emerging memory technologies can efficiently expand the capabilities of artificial intelligence hardware toward more generalized learning memories.

Working Memory Swap Errors Have Identifiable Neural Representations

Working memory is an essential component of cognition that facilitates goal-directed behavior. Famously, it is severely limited and performance suffers when memory load exceeds an individual's capacity. Modeling of visual working memory responses has identified two likely types of errors: guesses and swaps. Swap errors may arise from a misbinding between the features of different items. Alternatively, these errors could arise from memory noise in the feature dimension used for cueing a to-be-tested memory item, resulting in the wrong item being selected. Finally, it is possible that so-called swap errors actually reflect informed guessing, which could occur at the time of a cue, or alternatively, at the time of the response. Here, we combined behavioral response modeling and fMRI pattern analysis to test the hypothesis that swap errors involve the active maintenance of an incorrect memory item. After the encoding of six spatial locations, a retro-cue indicated which location would be tested after memory retention. On accurate trials, we could reconstruct a memory representation of the cued location in both early visual cortex and intraparietal sulcus. On swap error trials identified with mixture modeling, we were able to reconstruct a representation of the swapped location, but not of the cued location, suggesting the maintenance of the incorrect memory item before response. Moreover, participants subjectively responded with some level of confidence, rather than complete guessing, on a majority of swap error trials. Together, these results suggest that swap errors are not mere response-phase guesses, but instead result from failures of selection in working memory, contextual binding errors, or informed guesses, which produce active maintenance of incorrect memory representations.

Mechanisms of feature binding in visual working memory are stable over long delays

The ability to accurately retain the binding between the features of different objects is a critical element of visual working memory. The underlying mechanism can be elucidated by analyzing correlations of response errors in dual-report experiments, in which participants have to report two features of a single item from a previously viewed stimulus array. Results from separate previous studies using different cueing conditions have indicated that location takes a privileged role in mediating binding between other features, in that largely independent response errors have been observed when location was used as a cue, but errors were highly correlated when location was one of the reported features. Earlier results from change detection tasks likewise support such a special role of location, but they also suggest that this role is substantially reduced for longer retention intervals in favor of object-based representation. In the present study, we replicated the findings of previous dual-report tasks with different cueing conditions, using matched stimuli and procedures. Moreover, we show that the observed patterns of error correlations remain qualitatively unchanged with longer retention intervals. Fits with neural population models demonstrate that the behavioral results at long, as well as short, delays are best explained by memory representations in independent feature maps, in which an item's features are bound to each other only via their shared location.

Across-Area Synchronization Supports Feature Integration in a Biophysical Network Model of Working Memory

Working memory function is severely limited. One key limitation that constrains the ability to maintain multiple items in working memory simultaneously is so-called swap errors. These errors occur when an inaccurate response is in fact accurate relative to a non-target stimulus, reflecting the failure to maintain the appropriate association or "binding" between the features that define one object (e.g., color and location). The mechanisms underlying feature binding in working memory remain unknown. Here, we tested the hypothesis that features are bound in memory through synchrony across feature-specific neural assemblies. We built a biophysical neural network model composed of two one-dimensional attractor networks - one for color and one for location - simulating feature storage in different cortical areas. Within each area, gamma oscillations were induced during bump attractor activity through the interplay of fast recurrent excitation and slower feedback inhibition. As a result, different memorized items were held at different phases of the network's oscillation. These two areas were then reciprocally connected via weak cortico-cortical excitation, accomplishing binding between color and location through the synchronization of pairs of bumps across the two areas. Encoding and decoding of color-location associations was accomplished through rate coding, overcoming a long-standing limitation of binding through synchrony. In some simulations, swap errors arose: "color bumps" abruptly changed their phase relationship with "location bumps." This model, which leverages the explanatory power of similar attractor models, specifies a plausible mechanism for feature binding and makes specific predictions about swap errors that are testable at behavioral and neurophysiological levels.

Capturing the objects of vision with neural networks

Human visual perception carves a scene at its physical joints, decomposing the world into objects, which are selectively attended, tracked and predicted as we engage our surroundings. Object representations emancipate perception from the sensory input, enabling us to keep in mind that which is out of sight and to use perceptual content as a basis for action and symbolic cognition. Human behavioural studies have documented how object representations emerge through grouping, amodal completion, proto-objects and object files. By contrast, deep neural network models of visual object recognition remain largely tethered to sensory input, despite achieving human-level performance at labelling objects. Here, we review related work in both fields and examine how these fields can help each other. The cognitive literature provides a starting point for the development of new experimental tasks that reveal mechanisms of human object perception and serve as benchmarks driving the development of deep neural network models that will put the object into object recognition.

Temporal dynamics of implicit memory underlying serial dependence

Serial dependence is the effect in which the immediately preceding trial influences participants' responses to the current stimulus. But for how long does this bias last in the absence of interference from other stimuli? Here, we had 20 healthy young adult participants (12 women) perform a coincident timing task using different inter-trial intervals to characterize the serial dependence effect as the time between trials increases. Our results show that serial dependence abruptly decreases from 0.1 s to 1 s inter-trial interval, but it remains pronounced after that for up to 8 s. In addition, participants' response variability slightly decreases over longer intervals. We discuss these results in light of recent models suggesting that serial dependence might rely on a short-term memory trace kept through changes in synaptic weights, which might explain its long duration and apparent stability over time.

Consequence of stroke for feature recall and binding in visual working memory

Visual memory for objects involves the integration, or binding, of individual features into a coherent representation. We used a novel approach to assess feature binding, using a delayed-reproduction task in combination with computational modeling and lesion analysis. We assessed stroke patients and neurotypical controls on a visual working memory task in which spatial arrays of colored disks were presented. After a brief delay, participants either had to report the color of one disk cued by its location or the location of one disk cued by its color. Our results demonstrate that, in the controls, report imprecision and swap errors (non-target reports) can be explained by a single source of variability. Stroke patients showed an overall decrease in memory precision for both color and location, with only limited evidence for deviations from the predicted relationship between report precision and swap errors. These deviations were primarily deficits in reporting items rather than selecting items based on the cue. Atlas-based lesion-symptom mapping showed that selection and reporting deficits, precision in reporting color, and precision in reporting location were associated with different lesion profiles. Deficits in binding are associated with lesions in the left somatosensory cortex, deficits in the precision of reporting color with bilateral fronto-parietal regions, and no anatomical substrates were identified for precision in reporting location. Our results converge with previous reports that working memory representations are widely distributed in the brain and can be found across sensory, parietal, temporal, and prefrontal cortices. Stroke patients demonstrate mostly subtle impairments in visual working memory, perhaps because representations from different areas in the brain can partly compensate for impaired encoding in lesioned areas. These findings contribute to understanding of the relation between memorizing features and their bound representations.

Stochastic sampling provides a unifying account of visual working memory limits

Research into human working memory limits has been shaped by the competition between different formal models, with a central point of contention being whether internal representations are continuous or discrete. Here we describe a sampling approach derived from principles of neural coding as a framework to understand working memory limits. Reconceptualizing existing models in these terms reveals strong commonalities between seemingly opposing accounts, but also allows us to identify specific points of difference. We show that the discrete versus continuous nature of sampling is not critical to model fits, but that, instead, random variability in sample counts is the key to reproducing human performance in both single- and whole-report tasks. A probabilistic limit on the number of items successfully retrieved is an emergent property of stochastic sampling, requiring no explicit mechanism to enforce it. These findings resolve discrepancies between previous accounts and establish a unified computational framework for working memory that is compatible with neural principles.

Recall of facial expressions and simple orientations reveals competition for resources at multiple levels of the visual hierarchy

Many studies of visual working memory have tested humans' ability to reproduce primary visual features of simple objects, such as the orientation of a grating or the hue of a color patch, following a delay. A consistent finding of such studies is that precision of responses declines as the number of items in memory increases. Here we compared visual working memory for primary features and high-level objects. We presented participants with memory arrays consisting of oriented gratings, facial expressions, or a mixture of both. Precision of reproduction for all facial expressions declined steadily as the memory load was increased from one to five faces. For primary features, this decline and the specific distributions of error observed, have been parsimoniously explained in terms of neural population codes. We adapted the population coding model for circular variables to the non-circular and bounded parameter space used for expression estimation. Total population activity was held constant according to the principle of normalization and the intensity of expression was decoded by drawing samples from the Bayesian posterior distribution. The model fit the data well, showing that principles of population coding can be applied to model memory representations at multiple levels of the visual hierarchy. When both gratings and faces had to be remembered, an asymmetry was observed. Increasing the number of faces decreased precision of orientation recall, but increasing the number of gratings did not affect recall of expression, suggesting that memorizing faces involves the automatic encoding of low-level features, in addition to higher-level expression information.

Neural mechanisms of attending to items in working memory

Working memory, the ability to keep recently accessed information available for immediate manipulation, has been proposed to rely on two mechanisms that appear difficult to reconcile: self-sustained neural firing, or the opposite-activity-silent synaptic traces. Here we review and contrast models of these two mechanisms, and then show that both phenomena can co-exist within a unified system in which neurons hold information in both activity and synapses. Rapid plasticity in flexibly-coding neurons allows features to be bound together into objects, with an important emergent property being the focus of attention. One memory item is held by persistent activity in an attended or "focused" state, and is thus remembered better than other items. Other, previously attended items can remain in memory but in the background, encoded in activity-silent synaptic traces. This dual functional architecture provides a unified common mechanism accounting for a diversity of perplexing attention and memory effects that have been hitherto difficult to explain in a single theoretical framework.

Chunking as a rational strategy for lossy data compression in visual working memory

The nature of capacity limits for visual working memory has been the subject of an intense debate that has relied on models that assume items are encoded independently. Here we propose that instead, similar features are jointly encoded through a "chunking" process to optimize performance on visual working memory tasks. We show that such chunking can: (a) facilitate performance improvements for abstract capacity-limited systems, (b) be optimized through reinforcement, (c) be implemented by center-surround dynamics, and (d) increase effective storage capacity at the expense of recall precision. Human performance on a variant of a canonical working memory task demonstrated performance advantages, precision detriments, interitem dependencies, and trial-to-trial behavioral adjustments diagnostic of performance optimization through center-surround chunking. Models incorporating center-surround chunking provided a better quantitative description of human performance in our study as well as in a meta-analytic dataset, and apparent differences in working memory capacity across individuals were attributable to individual differences in the implementation of chunking. Our results reveal a normative rationale for center-surround connectivity in working memory circuitry, call for reevaluation of memory performance differences that have previously been attributed to differences in capacity, and support a more nuanced view of visual working memory capacity limitations: strategic tradeoff between storage capacity and memory precision through chunking contribute to flexible capacity limitations that include both discrete and continuous aspects. (PsycINFO Database Record

Visual recency bias is explained by a mixture model of internal representations

Human bias towards more recent events is a common and well-studied phenomenon. Recent studies in visual perception have shown that this recency bias persists even when past events contain no information about the future. Reasons for this suboptimal behavior are not well understood and the internal model that leads people to exhibit recency bias is unknown. Here we use a well-known orientation estimation task to frame the human recency bias in terms of incremental Bayesian inference. We show that the only Bayesian model capable of explaining the recency bias relies on a weighted mixture of past states. Furthermore, we suggest that this mixture model is a consequence of participants' failure to infer a model for data in visual short-term memory, and reflects the nature of the internal representations used in the task.

Optimal dynamic coding by mixed-dimensionality neurons in the head-direction system of bats

Ethologically relevant stimuli are often multidimensional. In many brain systems, neurons with “pure” tuning to one stimulus dimension are found along with “conjunctive” neurons that encode several dimensions, forming an apparently redundant representation. Here we show using theoretical analysis that a mixed-dimensionality code can efficiently represent a stimulus in different behavioral regimes: encoding by conjunctive cells is more robust when the stimulus changes quickly, whereas on long timescales pure cells represent the stimulus more efficiently with fewer neurons. We tested our predictions experimentally in the bat head-direction system and found that many head-direction cells switched their tuning dynamically from pure to conjunctive representation as a function of angular velocity—confirming our theoretical prediction. More broadly, our results suggest that optimal dimensionality depends on population size and on the time available for decoding—which might explain why mixed-dimensionality representations are common in sensory, motor, and higher cognitive systems across species.

Multidimensional stimuli are often represented by neurons encoding only a single dimension and those encoding multiple dimensions. Here, the authors present theoretical and experimental analyses to show that mixed representations are optimal to efficiently encode such stimuli under different behavioral modes.

In search of the focus of attention in working memory: 13 years of the retro-cue effect

The concept of attention has a prominent place in cognitive psychology. Attention can be directed not only to perceptual information, but also to information in working memory (WM). Evidence for an internal focus of attention has come from the retro-cue effect: Performance in tests of visual WM is improved when attention is guided to the test-relevant contents of WM ahead of testing them. The retro-cue paradigm has served as a test bed to empirically investigate the functions and limits of the focus of attention in WM. In this article, we review the growing body of (behavioral) studies on the retro-cue effect. We evaluate the degrees of experimental support for six hypotheses about what causes the retro-cue effect: (1) Attention protects representations from decay, (2) attention prioritizes the selected WM contents for comparison with a probe display, (3) attended representations are strengthened in WM, (4) not-attended representations are removed from WM, (5) a retro-cue to the retrieval target provides a head start for its retrieval before decision making, and (6) attention protects the selected representation from perceptual interference. The extant evidence provides support for the last four of these hypotheses.

Fundamental bound on the persistence and capacity of short-term memory stored as graded persistent activity

It is widely believed that persistent neural activity underlies short-term memory. Yet, as we show, the degradation of information stored directly in such networks behaves differently from human short-term memory performance. We build a more general framework where memory is viewed as a problem of passing information through noisy channels whose degradation characteristics resemble those of persistent activity networks. If the brain first encoded the information appropriately before passing the information into such networks, the information can be stored substantially more faithfully. Within this framework, we derive a fundamental lower-bound on recall precision, which declines with storage duration and number of stored items. We show that human performance, though inconsistent with models involving direct (uncoded) storage in persistent activity networks, can be well-fit by the theoretical bound. This finding is consistent with the view that if the brain stores information in patterns of persistent activity, it might use codes that minimize the effects of noise, motivating the search for such codes in the brain.

Restoration of fMRI Decodability Does Not Imply Latent Working Memory States

Recent imaging studies have challenged the prevailing view that working memory is mediated by sustained neural activity. Using machine learning methods to reconstruct memory content, these studies found that previously diminished representations can be restored by retrospective cueing or other forms of stimulation. These findings have been interpreted as evidence for an activity-silent working memory state that can be reactivated dependent on task demands. Here, we test the validity of this conclusion by formulating a neural process model of working memory based on sustained activity and using this model to emulate a spatial recall task with retro-cueing. The simulation reproduces both behavioral and fMRI results previously taken as evidence for latent states, in particular the restoration of spatial reconstruction quality following an informative cue. Our results demonstrate that recovery of the decodability of an imaging signal does not provide compelling evidence for an activity-silent working memory state.

Benefits of flexible prioritization in working memory can arise without costs

Most recent models conceptualize working memory (WM) as a continuous resource, divided up according to task demands. When an increasing number of items need to be remembered, each item receives a smaller chunk of the memory resource. These models predict that the allocation of attention to high-priority WM items during the retention interval should be a zero-sum game: improvements in remembering cued items come at the expense of uncued items because resources are dynamically transferred from uncued to cued representations. The current study provides empirical data challenging this model. Four precision retrocueing WM experiments assessed cued and uncued items on every trial. This permitted a test for trade-off of the memory resource. We found no evidence for trade-offs in memory across trials. Moreover, robust improvements in WM performance for cued items came at little or no cost to uncued items that were probed afterward, thereby increasing the net capacity of WM relative to neutral cueing conditions. An alternative mechanism of prioritization proposes that cued items are transferred into a privileged state within a response-gating bottleneck, in which an item uniquely controls upcoming behavior. We found evidence consistent with this alternative. When an uncued item was probed first, report of its orientation was biased away from the cued orientation to be subsequently reported. We interpret this bias as competition for behavioral control in the output-driving bottleneck. Other items in WM did not bias each other, making this result difficult to explain with a shared resource model.

This study challenges the dominant model for how we remember and prioritize pieces of information over short intervals (working memory). The dominant view is that all items in working memory share a single resource, and that we can prioritize one item by redistributing resources in its favor. This view predicts that nonprioritized memories become lost or impoverished. By testing how well participants remember both prioritized and nonprioritized items, we show that this is not the case. Our findings suggest that memories can be prioritized flexibly without necessarily jeopardizing others that may still become relevant.

Neural Architecture for Feature Binding in Visual Working Memory

Binding refers to the operation that groups different features together into objects. We propose a neural architecture for feature binding in visual working memory that employs populations of neurons with conjunction responses. We tested this model using cued recall tasks, in which subjects had to memorize object arrays composed of simple visual features (color, orientation, and location). After a brief delay, one feature of one item was given as a cue, and the observer had to report, on a continuous scale, one or two other features of the cued item. Binding failure in this task is associated with swap errors, in which observers report an item other than the one indicated by the cue. We observed that the probability of swapping two items strongly correlated with the items' similarity in the cue feature dimension, and found a strong correlation between swap errors occurring in spatial and nonspatial report. The neural model explains both swap errors and response variability as results of decoding noisy neural activity, and can account for the behavioral results in quantitative detail. We then used the model to compare alternative mechanisms for binding nonspatial features. We found the behavioral results fully consistent with a model in which nonspatial features are bound exclusively via their shared location, with no indication of direct binding between color and orientation. These results provide evidence for a special role of location in feature binding, and the model explains how this special role could be realized in the neural system.SIGNIFICANCE STATEMENT The problem of feature binding is of central importance in understanding the mechanisms of working memory. How do we remember not only that we saw a red and a round object, but that these features belong together to a single object rather than to different objects in our environment? Here we present evidence for a neural mechanism for feature binding in working memory, based on encoding of visual information by neurons that respond to the conjunction of features. We find clear evidence that nonspatial features are bound via space: we memorize directly where a color or an orientation appeared, but we memorize which color belonged with which orientation only indirectly by virtue of their shared location.

Retrospective Attention Interacts with Stimulus Strength to Shape Working Memory Performance

Orienting attention retrospectively to selective contents in working memory (WM) influences performance. A separate line of research has shown that stimulus strength shapes perceptual representations. There is little research on how stimulus strength during encoding shapes WM performance, and how effects of retrospective orienting might vary with changes in stimulus strength. We explore these questions in three experiments using a continuous-recall WM task. In Experiment 1 we show that benefits of cueing spatial attention retrospectively during WM maintenance (retrocueing) varies according to stimulus contrast during encoding. Retrocueing effects emerge for supraliminal but not sub-threshold stimuli. However, once stimuli are supraliminal, performance is no longer influenced by stimulus contrast. In Experiments 2 and 3 we used a mixture-model approach to examine how different sources of error in WM are affected by contrast and retrocueing. For high-contrast stimuli (Experiment 2), retrocues increased the precision of successfully remembered items. For low-contrast stimuli (Experiment 3), retrocues decreased the probability of mistaking a target with distracters. These results suggest that the processes by which retrospective attentional orienting shape WM performance are dependent on the quality of WM representations, which in turn depends on stimulus strength during encoding.

The neuroscience of working memory capacity and training

Working memory - the ability to maintain and manipulate information over a period of seconds - is a core component of higher cognitive functions. The storage capacity of working memory is limited but can be expanded by training, and evidence of the neural mechanisms underlying this effect is accumulating. Human imaging studies and neurophysiological recordings in non-human primates, together with computational modelling studies, reveal that training increases the activity of prefrontal neurons and the strength of connectivity in the prefrontal cortex and between the prefrontal and parietal cortex. Dopaminergic transmission could have a facilitatory role. These changes more generally inform us of the plasticity of higher cognitive functions.

Evaluating and excluding swap errors in analogue tests of working memory

When observers retrieve simple visual features from working memory, two kinds of error are typically confounded in their recall. First, responses reflect noise or variability within the feature dimension they were asked to report. Second, responses are corrupted by “swap errors”, in which a different item from the memory set is reported in place of the one that was probed. Independent evaluation of these error sources is vital for understanding the structure of internal representations and their binding. However, previous methods for disentangling these errors have been critically dependent on assumptions about the noise distribution, which is a priori unknown. Here I address this question with novel non-parametric (NP) methods, which estimate swap frequency and feature variability with fewer prior assumptions, and without a fitting procedure. The results suggest that swap errors are considerably more prevalent than previously appreciated (accounting for more than a third of responses at set size 8). These methods also identify which items are swapped in for targets: when the target item is cued by location, the items in closest spatial proximity are most likely to be incorrectly reported, thus implicating noise in the probe feature dimension as a source of swap errors.

A Unifying Model of Orientation Crowding in Peripheral Vision

Peripheral vision is fundamentally limited not by the visibility of features, but by the spacing between them [1]. When too close together, visual features can become "crowded" and perceptually indistinguishable. Crowding interferes with basic tasks such as letter and face identification and thus informs our understanding of object recognition breakdown in peripheral vision [2]. Multiple proposals have attempted to explain crowding [3], and each is supported by compelling psychophysical and neuroimaging data [4-6] that are incompatible with competing proposals. In general, perceptual failures have variously been attributed to the averaging of nearby visual signals [7-10], confusion between target and distractor elements [11, 12], and a limited resolution of visual spatial attention [13]. Here we introduce a psychophysical paradigm that allows systematic study of crowded perception within the orientation domain, and we present a unifying computational model of crowding phenomena that reconciles conflicting explanations. Our results show that our single measure produces a variety of perceptual errors that are reported across the crowding literature. Critically, a simple model of the responses of populations of orientation-selective visual neurons accurately predicts all perceptual errors. We thus provide a unifying mechanistic explanation for orientation crowding in peripheral vision. Our simple model accounts for several perceptual phenomena produced by crowding of orientation and raises the possibility that multiple classes of object recognition failures in peripheral vision can be accounted for by a single mechanism.

Spikes not slots: noise in neural populations limits working memory

This opinion article argues that noise (randomness) in neural activity is the limiting factor in visual working memory (WM), determining how accurately we can maintain stable internal representations of external stimuli. Sharing of a fixed amount of neural activity between items in memory explains why WM can be successfully described as a continuous resource. This contrasts with the popular conception of WM as comprising a limited number of memory slots, each holding a representation of one stimulus - I argue that this view is challenged by computational theory and the latest neurophysiological evidence.