A probabilistic clustering theory of the organization of visual short-term memory

Representation and computation in visual working memory

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

A signal-detection account of item-based and ensemble-based visual change detection: A reply to Harrison, McMaster, and Bays

Growing empirical evidence shows that ensemble information (e.g., the average feature or feature variance of a set of objects) affects visual working memory for individual items. Recently, Harrison, McMaster, and Bays (2021) used a change detection task to test whether observers explicitly rely on ensemble representations to improve their memory for individual objects. They found that sensitivity to simultaneous changes in all memorized items (which also globally changed set summary statistics) rarely exceeded a level predicted by the so-called optimal summation model within the signal-detection framework. This model implies simple integration of evidence for change from all individual items and no additional evidence coming from ensemble. Here, we argue that performance at the level of optimal summation does not rule out the use of ensemble information. First, in two experiments, we show that, even if evidence from only one item is available at test, the statistics of the whole memory set affect performance. Second, we argue that optimal summation itself can be conceptually interpreted as one of the strategies of holistic, ensemble-based decision. We also redefine the reference level for the item-based strategy as the so-called "minimum rule," which predicts performance far below the optimum. We found that that both our and Harrison et al. (2021)'s observers consistently outperformed this level. We conclude that observers can rely on ensemble information when performing visual change detection. Overall, our work clarifies and refines the use of signal-detection analysis in measuring and modeling working memory.

Similarity-based clustering of multifeature objects in visual working memory

This study investigated the similarity-based clustering mechanism of multifeature stimuli, wherein items are separated or grouped based on their similarity in visual working memory (VWM). In particular, we investigated whether clustering occurred at an individual feature level or at an integrated object level when participants encoded objects with multiple features for VWM. To test this, we conducted two experiments in which participants remembered and reconstructed a randomly chosen feature (either color or orientation) from one of five presented stimuli. As a key manipulation, we kept the distributions of the two feature dimensions constant while controlling the conjunction between the two dimensions in two different conditions: congruent conjunction (CC) and incongruent conjunction (IC). With this manipulation, we expected to observe the same number of clusters regardless of the conjunction condition when clustering occurred at the feature level. However, we expected a different number of clusters for CC and IC conditions when clustering occurred at the object level. Across two experiments, we consistently observed evidence that favored feature-level clustering. Nevertheless, we found that the swap error rates increased in the IC condition only when two features had to be encoded in VWM. These results suggest that clustering occurs at the feature level in VWM and that feature-level clustering influences item-level feature binding. Therefore, our study demonstrated the flexibility of representational units in VWM.

A quasi-comprehensive exploration of the mechanisms of spatial working memory

Why are some spatial patterns remembered more easily than others? There are many possible mechanisms underlying spatial working memory function. Here, the author explores different mechanisms simultaneously in a single conceptual model. He conducts a large-scale experiment (35.4 million responses used to measure human observers' spatial working memory across 80,000 patterns) and builds a convolutional neural network as a benchmark for what is expected to be explainable. The author then creates a quasi-comprehensive exploration model of spatial working memory based on classic concepts, as well as new notions, including spatial uncertainty, Bayesian integration, out-of-range responses, averaging, grouping, categorical memory, line detection, gap detection, blurring, lateral inhibition, chunking, multiple spatial-frequency channels, redundancy, response bias and random guess. This model provides a tentative overarching framework for the mechanisms of spatial working memory.

Processing spatial configurations in visuospatial working memory is influenced by shifts of overt visual attention

When memorizing multiple objects, humans process them in relation to each other, proposing a configuration benefit. Shifts in overt visual attention through eye movements might influence the processing of spatial configurations. Whereas some research suggests that overt visual attention aids the processing of spatial representations, other research suggests a snapshot-like processing of spatial configurations, thus likely not relying on eye movements. In the first experiment, we focused on the comparison between an enforced fixation and a free view condition regarding configurational effects. Participants encoded objects' locations and were asked for changes at retrieval. One object was displaced in half of the trials and was either accompanied by a configuration or was displayed alone. In the second experiment, we expanded this idea by enforcing fixation during different task phases, namely encoding, maintenance and retrieval. We investigated if a fixed gaze during one specific phase drives the influence of eye movements when processing spatial configurations. We observed reliable configuration benefits for the free view conditions. Whereas a fixed gaze throughout the whole trial reduced the effect, enforced fixations during the task phases did not break the configuration benefit. Our findings suggest that whereas the processing of spatial configurations in memory is supported by the ability of performing shifts of overt visual attention, configurational processing does not rely on these shifts occurring throughout the task. Our results indicate a reciprocal relationship of visuospatial working memory and eye movements.

Tracking the relation between gist and item memory over the course of long-term memory consolidation

Our experiences in the world support memories not only of specific episodes but also of the generalities (the 'gist') across related experiences. It remains unclear how these two types of memories evolve and influence one another over time. In two experiments, 173 human participants encoded spatial locations from a distribution and reported both item memory (specific locations) and gist memory (center for the locations) across 1-2 months. Experiment 1 demonstrated that after 1 month, gist memory was preserved relative to item memory, despite a persistent positive correlation between them. Critically, item memories were biased toward the gist over time. Experiment 2 showed that a spatial outlier item changed this relationship and that the extraction of gist is sensitive to the regularities of items. Our results suggest that the gist starts to guide item memories over longer durations as their relative strengths change.

Limited memory for ensemble statistics in visual change detection

Accounts of working memory based on independent item representations may overlook a possible contribution of ensemble statistics, higher-order regularities of a scene such as the mean or variance of a visual attribute. Here we used change detection tasks to investigate the hypothesis that observers store ensemble statistics in working memory and use them to detect changes in the visual environment. We controlled changes to the ensemble mean or variance between memory and test displays across six experiments. We made specific predictions of observers' sensitivity using an optimal summation model that integrates evidence across separate items but does not detect changes in ensemble statistics. We found strong evidence that observers outperformed this model, but only when task difficulty was high, and only for changes in stimulus variance. Under these conditions, we estimated that the variance of items contributed to change detection sensitivity more strongly than any individual item in this case. In contrast, however, we found strong evidence against the hypothesis that the average feature value is stored in working memory: when the mean of memoranda changed, sensitivity did not differ from the optimal summation model, which was blind to the ensemble mean, in five out of six experiments. Our results reveal that change detection is primarily limited by uncertainty in the memory of individual features, but that memory for the variance of items can facilitate detection under a limited set of conditions that involve relatively high working memory demands.

Optimal attentional allocation in the presence of capacity constraints in uncued and cued visual search

The vision sciences literature contains a large diversity of experimental and theoretical approaches to the study of visual attention. We argue that this diversity arises, at least in part, from the field's inability to unify differing theoretical perspectives. In particular, the field has been hindered by a lack of a principled formal framework for simultaneously thinking about both optimal attentional processing and capacity-limited attentional processing, where capacity is limited in a general, task-independent manner. Here, we supply such a framework based on rate-distortion theory (RDT) and optimal lossy compression. Our approach defines Bayes-optimal performance when an upper limit on information processing rate is imposed. In this article, we compare Bayesian and RDT accounts in both uncued and cued visual search tasks. We start by highlighting a typical shortcoming of unlimited-capacity Bayesian models that is not shared by RDT models, namely, that they often overestimate task performance when information-processing demands are increased. Next, we reexamine data from two cued-search experiments that have previously been modeled as the result of unlimited-capacity Bayesian inference and demonstrate that they can just as easily be explained as the result of optimal lossy compression. To model cued visual search, we introduce the concept of a "conditional communication channel." This simple extension generalizes the lossy-compression framework such that it can, in principle, predict optimal attentional-shift behavior in any kind of perceptual task, even when inputs to the model are raw sensory data such as image pixels. To demonstrate this idea's viability, we compare our idealized model of cued search, which operates on a simplified abstraction of the stimulus, to a deep neural network version that performs approximately optimal lossy compression on the real (pixel-level) experimental stimuli.

Free-recall benefit, inhomogeneity and between-item interference in working memory

We investigated visual working memory (VWM) with a whole-report task, where participants were asked to sequentially recall all the items in an order either chosen by themselves (free recall) or randomly chosen by the computer (forced recall). Comparisons between free and forced recalls helped us understand important but largely neglected aspects of VWM, such as inhomogeneity (different levels of precision) and between-item interference. One unique part of our task was the introduction of a separate item-selection stage before each recall, during which participants located the next item to recall. Their mouse trajectory was recorded and served as a dynamic measure of between-item interference over time. We found a free-recall benefit: the overall precision of all items is higher in free recall than in forced recall. Meanwhile, during item-selection, free recall is associated with faster localization of the target and less interference from the other items in memory. We also found evidence for inhomogeneity and discuss the connection of inhomogeneity and between-item interference to the free-recall benefit.

A compressibility account of the color-sharing bonus in working memory

It has been established that objects sharing color in a visual display can boost working memory. The capacity to encode singletons particularly benefits from the repetition of colors encoded as perceptual groups. We manipulated the algorithmic complexity of visual displays to test whether compressibility of information could account for the color-sharing bonus. This study used a free recall working memory task in which the participants were shown displays of 2 to 8 color items. We examined the influence of set size, complexity, number of same-color clusters and amount of color redundancy. The results showed that the probability of correct recall of the pattern and the proportion of similarity between the pattern and the response decreased with an increase of each manipulated variable, except for color redundancy in terms of probability of correct recall. The model performance of complexity did not differ from that of clusters, but complexity was found more accurate than either set size or color redundancy. The results also showed that similar items were more often recalled adjacently, and complexity correlated strongly with the number of extra color repetitions in the response, suggesting that more complex patterns encouraged the use of information compression. Moreover, color repetitions were more often recalled first and the probability of correct recall for singletons and sub-patterns could be predicted by the compressibility measure. We discuss the potential advantage of using compressibility measures to capture the effects of regularities in visual patterns, in particular to refine analysis of the color-sharing bonus.

Chunking and data compression in verbal short-term memory

Short-term verbal memory is improved when words can be chunked into larger units. Miller (1956) suggested that the capacity of verbal short-term memory is determined by the number of chunks that can be stored in memory, rather than by the number of items or the amount of information. But how does the improvement due to chunking come about, and is memory really determined by the number of chunks? One possibility is that chunking is a form of data compression. It allows more information to be stored in the available capacity. An alternative is that chunking operates primarily by redintegration. Chunks exist only in long-term memory, and enable the corresponding items in short-term memory to be reconstructed more reliably from a degraded trace. We review the data favoring each of these views and discuss the implications of treating chunking as data compression. Contrary to Miller, we suggest that memory capacity is primarily determined both by the amount of information that can be stored but also by the underlying representational vocabulary of the memory system. Given the limitations on the representations that can be stored in verbal short-term memory, chunking can sometimes allow the information capacity of short-term memory to be exploited more efficiently. (202 words).

Do group ensemble statistics bias visual working memory for individual items? A registered replication of Brady and Alvarez (2011)

We performed a registered and precise replication of Experiment 1 reported in Brady and Alvarez (Psychological Science, 22, 384–392, 2011). The original experiment found that participants, who were asked to memorize the size of differently colored circles, reported the size of a probed circle biased toward the mean size of the same-colored group. Because our previous three unpublished replication attempts failed to find this effect, we powered the present registered replication using a Bayes Factor Design Analysis such that it provided compelling evidence regarding the presence or absence of the reported bias with a high probability, even under the assumption of smaller effect sizes. Thus, we recruited 663 participants through Amazon Mechanical Turk. We observed both a significant bias and strong Bayesian evidence in favor of the existence of a bias over the null hypothesis. Thus, our results can be considered a successful replication of the original findings, although with a considerably smaller effect size. We discuss the role of data quality when recruiting participants with Amazon Mechanical Turk. The present findings corroborate the idea that memory representations of individual objects are influenced by summary statistics.

A single, simple, statistical mechanism explains resource distribution and temporal updating in visual short-term memory

Investigations into the way that information is held and integrated within the visual system provides some basis for understanding how visual information is represented and processed. Just over sixty years ago, Swets, Shipley, McKey, and Green (1959) demonstrated that performance within an auditory detection task increases as a function of the square root of the number of stimulus observation intervals, following the predictions of basic sampling theory, indicating the efficient perceptual integration of stimulus information. This principle of observer performance contingent on a constant rate of stimulus sampling also forms the basis of the sample-size model (Palmer, 1990; Sewell, Lilburn, & Smith, 2014) which seeks to provide an account of how memory resources might be divided among item representations in visual short-term memory (VSTM). In this article, we combine the multiple observations paradigm of Swets and colleagues with the VSTM paradigm of Sewell and colleagues and show that the sample-size relationship accounts for both the increase in performance with the number of presentation intervals and the way that performance changes as a function of the number of items in memory. The model provides an account of both the overall information limit of VSTM and an account of the dynamics of that limit, demonstrating not only that observers can selectively update specific representations in memory but that performance in this task is accounted for by a simple statistical constraint. We discuss the implications for models of VSTM capacity and architecture generally, focusing on the implications for objecthood and the characteristics of encoding to and retrieval from memory.

How Do People Code Information in Working Memory When Items Share Features?

Abstract. A large literature suggests that the way we process information is influenced by the categories that we have learned. We examined whether, when we try to uniquely encode items in working memory, the information encoded depends on the other stimuli being simultaneously learned. Participants were required to memorize unknown aliens, presented one at the time, for immediate recognition of their features. Some aliens, called twins, were organized into pairs that shared every feature (nondiscriminative feature) except one (discriminative feature), while some other aliens, called hermits, did not share feature. We reasoned that if people develop unsupervised categories by creating a category for a pair of aliens, we should observe better feature identification performance for nondiscriminative features compared to hermit features, but not compared to discriminative features. On the contrary, if distinguishing features draw attention, we should observe better performance when a discriminative rather than nondiscriminative feature was probed. Overall, our results suggest that when items share features, people code items in working memory by focusing on similarities between items, establishing clusters of items in an unsupervised fashion not requiring feedback on cluster membership.

Composite Face Effect Predicts Configural Encoding in Visual Short-Term Memory

In natural vision, visual scenes consist of individual items (e.g., trees) and global properties of items as a whole (e.g., forest). These different levels of representations can all contribute to perception, natural scene understanding, sensory memory, working memory, and long-term memory. Despite these various hierarchical representations across perception and cognition, the nature of the global representations has received considerably less attention in empirical research on working memory than item representations. The present study aimed to understand the perceptual root of the configural information retained in Visual Short-term Memory (VSTM). Specifically, we assessed whether configural VSTM was related to holistic face processing across participants using an individual differences approach. Configural versus item encoding in VSTM was assessed using Xie and Zhang’s (2017) dual-trace Signal Detection Theory model in a change detection task for orientation. Configural face processing was assessed using Le Grand composite face effect (CFE). In addition, overall face recognition was assessed using Glasgow Face Matching Test (GFMT). Across participants, holistic face encoding, but not face recognition accuracy, predicted configural information, but not item information, retained in VSTM. Together, these findings suggest that configural encoding in VSTM may have a perceptual root.

Ensemble representations reveal distinct neural coding of visual working memory

We characterized the population-level neural coding of ensemble representations in visual working memory from human electroencephalography. Ensemble representations provide a unique opportunity to investigate structured representations of working memory because the visual system encodes high-order summary statistics as well as noisy sensory inputs in a hierarchical manner. Here, we consistently observe stable coding of simple features as well as the ensemble mean in frontocentral electrodes, which even correlated with behavioral indices of the ensemble across individuals. In occipitoparietal electrodes, however, we find that remembered features are dynamically coded over time, whereas neural coding of the ensemble mean is absent in the old/new judgment task. In contrast, both dynamic and stable coding are found in the continuous estimation task. Our findings suggest that the prefrontal cortex holds behaviorally relevant abstract representations while visual representations in posterior and visual areas are modulated by the task demands.

Distinguishing target biases and strategic guesses in visual working memory

Previous studies have shown that when there are statistical regularities in the items stored in visual working memory, the responses are biased toward the ensemble average. This statistical-regularity-induced bias could happen in two ways: (1) a target bias, where the individual memory representations are pulled toward the ensemble average; or (2) a strategic guess, for items that are not memorized, other information in the ensemble (e.g., another item) is reported as a substitute. Here, these two mechanisms are distinguished on the basis of a three-part model (target responses + swap responses + random guesses; e.g., Bays, Catalao, & Husain, 2009, Journal of Vision, 9, 7). The strategic guess is operationalized as swap responses, whereas the target bias is reflected by a bias parameter in the target responses. This model was applied on 8 data sets (22 observers each). In this model, contributions of target biases and strategic guesses can be clearly distinguished from each other because they lead to distinctive patterns in the distribution of responses. In the present results, strategic guesses always contributed substantially to the statistical-regularity-induced biases, whereas target biases were limited to specific conditions. All in all, the Bayesian inference in visual working memory is much more limited than what is previously advocated.

Reorganization of spatial configurations in visual working memory: A matter of set size?

Humans process single objects in relation to other simultaneously maintained objects in visual working memory. This interdependence is called spatial configuration. Humans are able to reorganize global spatial configurations into relevant partial configurations. We conducted three experiments investigating the process underlying reorganization by manipulating memory set size and the presence of configurations at retrieval. Participants performed a location change detection task for a single object probed at retrieval. At the beginning of each trial, participants memorized the locations of all objects (set size: 4, 8, 12, or 16). During maintenance, a valid retro cue highlighted the side containing the object probed at retrieval, thus enabling participants to reorganize the memorized global spatial configuration to the partial cued configuration. At retrieval, the object probed was shown together with either all objects (complete configuration; Experiment 1a), the cued objects only (congruent configuration; all Experiments), the non-cued objects only (incongruent configuration, all Experiments) or alone (no configuration; Experiment 1b). We observed reorganization of spatial configurations as indicated by a superior location change detection performance with a congruent partial configuration than an incongruent partial configuration across all three experiments. We also observed an overall decrease in accuracy with increasing set size. Most importantly, however, we did not find evidence for a reliable impairment of reorganization with increasing set size. We discuss these findings with regard to the memory representation underlying spatial configurations.

Adaptive allocation of human visual working memory capacity during statistical and categorical learning

Human brains are finite, and thus have bounded capacity. An efficient strategy for a capacity-limited agent is to continuously adapt by dynamically reallocating capacity in a task-dependent manner. Here we study this strategy in the context of visual working memory (VWM). People use their VWM stores to remember visual information over seconds or minutes. However, their memory performances are often error-prone, presumably due to VWM capacity limits. We hypothesize that people attempt to be flexible and robust by strategically reallocating their limited VWM capacity based on two factors: (a) the statistical regularities (e.g., stimulus feature means and variances) of the to-be-remembered items, and (b) the requirements of the task that they are attempting to perform. The latter specifies, for example, which types of errors are costly versus irrelevant for task performance. These hypotheses are formalized within a normative computational modeling framework based on rate-distortion theory, an extension of conventional Bayesian approaches that uses information theory to study rate-limited (or capacity-limited) processes. Using images of plants that are naturalistic and precisely controlled, we carried out two sets of experiments. Experiment 1 found that when a stimulus dimension (the widths of plants' leaves) was assigned a distribution, subjects adapted their VWM performances based on this distribution. Experiment 2 found that when one stimulus dimension (e.g., leaf width) was relevant for distinguishing plant categories but another dimension (leaf angle) was irrelevant, subjects' responses in a memory task became relatively more sensitive to the relevant stimulus dimension. Together, these results illustrate the task-dependent robustness of VWM, thereby highlighting the dependence of memory on learning.

Human representation of multimodal distributions as clusters of samples

Behavioral and neuroimaging evidence shows that human decisions are sensitive to the statistical regularities (mean, variance, skewness, etc.) of reward distributions. However, it is unclear what representations human observers form to approximate reward distributions, or probability distributions in general. When the possible values of a probability distribution are numerous, it is cognitively costly and perhaps unrealistic to maintain in mind the probability of each possible value. Here we propose a Clusters of Samples (CoS) representation model: The samples of the to-be-represented distribution are classified into a small number of clusters and only the centroids and relative weights of the clusters are retained for future use. We tested the behavioral relevance of CoS in four experiments. On each trial, human subjects reported the mean and mode of a sequentially presented multimodal distribution of spatial positions or orientations. By varying the global and local features of the distributions, we observed systematic errors in the reported mean and mode. We found that our CoS representation of probability distributions outperformed alternative models in accounting for subjects’ response patterns. The ostensible influence of positive/negative skewness on the over/under estimation of the reported mean, analogous to the “skewness preference” phenomenon in decisions, could be well explained by models based on CoS.

Life is full of uncertainties: An action may yield multiple possible consequences and a percept may imply multiple possible causes. To survive, humans and animals must compensate for the uncertainty in the environment and in their own perceptual and motor systems. However, how humans represent probability distributions to fulfill probabilistic computations for perception and action remains elusive. The number of possible values in a distribution is vast and grows exponentially with the dimension of the distribution. It would be costly, if not impossible, to maintain the probability of each possible value. Here we propose a sparse representation of probability distributions, which can reduce an arbitrary distribution to a small set of coefficients while still keeping important global and local features of the original distribution. Our experiments provide preliminary evidence for the use of such representations in human cognition.

Chunking as a rational strategy for lossy data compression in visual working memory

The nature of capacity limits for visual working memory has been the subject of an intense debate that has relied on models that assume items are encoded independently. Here we propose that instead, similar features are jointly encoded through a "chunking" process to optimize performance on visual working memory tasks. We show that such chunking can: (a) facilitate performance improvements for abstract capacity-limited systems, (b) be optimized through reinforcement, (c) be implemented by center-surround dynamics, and (d) increase effective storage capacity at the expense of recall precision. Human performance on a variant of a canonical working memory task demonstrated performance advantages, precision detriments, interitem dependencies, and trial-to-trial behavioral adjustments diagnostic of performance optimization through center-surround chunking. Models incorporating center-surround chunking provided a better quantitative description of human performance in our study as well as in a meta-analytic dataset, and apparent differences in working memory capacity across individuals were attributable to individual differences in the implementation of chunking. Our results reveal a normative rationale for center-surround connectivity in working memory circuitry, call for reevaluation of memory performance differences that have previously been attributed to differences in capacity, and support a more nuanced view of visual working memory capacity limitations: strategic tradeoff between storage capacity and memory precision through chunking contribute to flexible capacity limitations that include both discrete and continuous aspects. (PsycINFO Database Record

The separable effects of feature precision and item load in visual short-term memory

Visual short-term memory (VSTM) has been described as being limited by the number of discrete visual objects, the aggregate quantity of information across multiple visual objects, or some combination of the two. Many recent studies examining these capacity limitations have shown that increasing the number of items in VSTM increases the frequency and magnitude of errors in a participant's recall of the stimulus. This increase in response dispersion has been interpreted as a loss of precision in an item's representation as the number of items in memory increases, possibly due to a change in the tuning of the underlying representation. However, increased response dispersion can also be caused by a reduction in the total memory strength available for decision making as a consequence of a reduction in the total amount of a fixed resource representing a stimulus. We investigated the effects of load on the precision of memory representations in a fine orientation discrimination task. Accuracy was well captured by extending a simple sample-size model of VSTM, using a tuning function to account for the effect of orientation precision on performance. The best model of the data was one in which the item strength decreased progressively with memory load at all stimulus exposure durations but in which tuning bandwidth was invariant. Our results imply that memory strength and feature precision are experimentally dissociable attributes of VSTM.

Slot-like capacity and resource-like coding in a neural model of multiple-item working memory

For the past decade, research on the storage limitations of working memory has been dominated by two fundamentally different hypotheses. On the one hand, the contents of working memory may be stored in a limited number of "slots," each with a fixed resolution. On the other hand, any number of items may be stored but with decreasing resolution. These two hypotheses have been invaluable in characterizing the computational structure of working memory, but neither provides a complete account of the available experimental data or speaks to the neural basis of the limitations it characterizes. To address these shortcomings, we simulated a multiple-item working memory task with a cortical network model, the cellular resolution of which allowed us to quantify the coding fidelity of memoranda as a function of memory load, as measured by the discriminability, regularity, and reliability of simulated neural spiking. Our simulations account for a wealth of neural and behavioral data from human and nonhuman primate studies, and they demonstrate that feedback inhibition lowers both capacity and coding fidelity. Because the strength of inhibition scales with the number of items stored by the network, increasing this number progressively lowers fidelity until capacity is reached. Crucially, the model makes specific, testable predictions for neural activity on multiple-item working memory tasks. NEW & NOTEWORTHY Working memory is the ability to keep information in mind and is fundamental to cognition. It is actively debated whether the storage limitations of working memory reflect a small number of storage units (slots) or a decrease in coding resolution as a limited resource is allocated to more items. In a cortical model, we found that slot-like capacity and resource-like neural coding resulted from the same mechanism, offering an integrated explanation for storage limitations.

Interactions between visual working memory representations

We investigated whether the representations of different objects are maintained independently in working memory or interact with each other. Observers were shown two sequentially presented orientations and required to reproduce each orientation after a delay. The sequential presentation minimized perceptual interactions so that we could isolate interactions between memory representations per se. We found that similar orientations were repelled from each other whereas dissimilar orientations were attracted to each other. In addition, when one of the items was given greater attentional priority by means of a cue, the representation of the high-priority item was not influenced very much by the orientation of the low-priority item, but the representation of the low-priority item was strongly influenced by the orientation of the high-priority item. This indicates that attention modulates the interactions between working memory representations. In addition, errors in the reported orientations of the two objects were positively correlated under some conditions, suggesting that representations of distinct objects may become grouped together in memory. Together, these results demonstrate that working-memory representations are not independent but instead interact with each other in a manner that depends on attentional priority.

The Whole Warps the Sum of Its Parts

The efficiency of averaging properties of sets without encoding redundant details is analogous to gestalt proposals that perception is parsimoniously organized as a function of recurrent order in the world. This similarity suggests that grouping and averaging are part of a broader set of strategies allowing the visual system to circumvent capacity limitations. To examine how gestalt grouping affects the manner in which information is averaged and remembered, I compared the error in observers' adjustments of remembered sizes of individual circles in two different mean-size sets defined by similarity, proximity, connectedness, or a common region. Overall, errors were more similar within the same gestalt-defined groups than between different gestalt-defined groups, such that the remembered sizes of individual circles were biased toward the mean size of their respective gestalt-defined groups. These results imply that gestalt grouping facilitates perceptual averaging to minimize the error with which individual items are encoded, thereby optimizing the efficiency of visual short-term memory.

Rate-distortion theory and human perception

The fundamental goal of perception is to aid in the achievement of behavioral objectives. This requires extracting and communicating useful information from noisy and uncertain sensory signals. At the same time, given the complexity of sensory information and the limitations of biological information processing, it is necessary that some information must be lost or discarded in the act of perception. Under these circumstances, what constitutes an 'optimal' perceptual system? This paper describes the mathematical framework of rate-distortion theory as the optimal solution to the problem of minimizing the costs of perceptual error subject to strong constraints on the ability to communicate or transmit information. Rate-distortion theory offers a general and principled theoretical framework for developing computational-level models of human perception (Marr, 1982). Models developed in this framework are capable of producing quantitatively precise explanations for human perceptual performance, while yielding new insights regarding the nature and goals of perception. This paper demonstrates the application of rate-distortion theory to two benchmark domains where capacity limits are especially salient in human perception: discrete categorization of stimuli (also known as absolute identification) and visual working memory. A software package written for the R statistical programming language is described that aids in the development of models based on rate-distortion theory.

Evaluating and excluding swap errors in analogue tests of working memory

When observers retrieve simple visual features from working memory, two kinds of error are typically confounded in their recall. First, responses reflect noise or variability within the feature dimension they were asked to report. Second, responses are corrupted by “swap errors”, in which a different item from the memory set is reported in place of the one that was probed. Independent evaluation of these error sources is vital for understanding the structure of internal representations and their binding. However, previous methods for disentangling these errors have been critically dependent on assumptions about the noise distribution, which is a priori unknown. Here I address this question with novel non-parametric (NP) methods, which estimate swap frequency and feature variability with fewer prior assumptions, and without a fitting procedure. The results suggest that swap errors are considerably more prevalent than previously appreciated (accounting for more than a third of responses at set size 8). These methods also identify which items are swapped in for targets: when the target item is cued by location, the items in closest spatial proximity are most likely to be incorrectly reported, thus implicating noise in the probe feature dimension as a source of swap errors.

Organization principles in visual working memory: Evidence from sequential stimulus display

Although the mechanisms of visual working memory (VWM) have been studied extensively in recent years, the active property of VWM has received less attention. In the current study, we examined how VWM integrates sequentially presented stimuli by focusing on the role of Gestalt principles, which are important organizing principles in perceptual integration. We manipulated the level of Gestalt cues among three or four sequentially presented objects that were memorized. The Gestalt principle could not emerge unless all the objects appeared together. We distinguished two hypotheses: a perception-alike hypothesis and an encoding-specificity hypothesis. The former predicts that the Gestalt cue will play a role in information integration within VWM; the latter predicts that the Gestalt cue will not operate within VWM. In four experiments, we demonstrated that collinearity (Experiment 1) and closure (Experiment 2) cues significantly improved VWM performance, and this facilitation was not affected by the testing manner (Experiment 3) or by adding extra colors to the memorized objects (Experiment 4). Finally, we re-established the Gestalt cue benefit with similarity cues (Experiment 5). These findings together suggest that VWM realizes and uses potential Gestalt principles within the stored representations, supporting a perception-alike hypothesis.

Bayesian hierarchical grouping: Perceptual grouping as mixture estimation

We propose a novel framework for perceptual grouping based on the idea of mixture models, called Bayesian hierarchical grouping (BHG). In BHG, we assume that the configuration of image elements is generated by a mixture of distinct objects, each of which generates image elements according to some generative assumptions. Grouping, in this framework, means estimating the number and the parameters of the mixture components that generated the image, including estimating which image elements are "owned" by which objects. We present a tractable implementation of the framework, based on the hierarchical clustering approach of Heller and Ghahramani (2005). We illustrate it with examples drawn from a number of classical perceptual grouping problems, including dot clustering, contour integration, and part decomposition. Our approach yields an intuitive hierarchical representation of image elements, giving an explicit decomposition of the image into mixture components, along with estimates of the probability of various candidate decompositions. We show that BHG accounts well for a diverse range of empirical data drawn from the literature. Because BHG provides a principled quantification of the plausibility of grouping interpretations over a wide range of grouping problems, we argue that it provides an appealing unifying account of the elusive Gestalt notion of Prägnanz.

Neural population coding of multiple stimuli

In natural scenes, objects generally appear together with other objects. Yet, theoretical studies of neural population coding typically focus on the encoding of single objects in isolation. Experimental studies suggest that neural responses to multiple objects are well described by linear or nonlinear combinations of the responses to constituent objects, a phenomenon we call stimulus mixing. Here, we present a theoretical analysis of the consequences of common forms of stimulus mixing observed in cortical responses. We show that some of these mixing rules can severely compromise the brain's ability to decode the individual objects. This cost is usually greater than the cost incurred by even large reductions in the gain or large increases in neural variability, explaining why the benefits of attention can be understood primarily in terms of a stimulus selection, or demixing, mechanism rather than purely as a gain increase or noise reduction mechanism. The cost of stimulus mixing becomes even higher when the number of encoded objects increases, suggesting a novel mechanism that might contribute to set size effects observed in myriad psychophysical tasks. We further show that a specific form of neural correlation and heterogeneity in stimulus mixing among the neurons can partially alleviate the harmful effects of stimulus mixing. Finally, we derive simple conditions that must be satisfied for unharmful mixing of stimuli.

Toward ecologically realistic theories in visual short-term memory research

Recent evidence from neuroimaging and psychophysics suggests common neural and representational substrates for visual perception and visual short-term memory (VSTM).Visual perception is adapted to a rich set of statistical regularities present in the natural visual environment. Common neural and representational substrates for visual perception and VSTM suggest that VSTM is adapted to these same statistical regularities too. This article discusses how the study of VSTM can be extended to stimuli that are ecologically more realistic than those commonly used in standard VSTM experiments and what the implications of such an extension could be for our current view of VSTM. We advocate for the development of unified models of visual perception and VSTM—probabilistic and hierarchical in nature— incorporating prior knowledge of natural scene statistics.

A probabilistic palimpsest model of visual short-term memory

Working memory plays a key role in cognition, and yet its mechanisms remain much debated. Human performance on memory tasks is severely limited; however, the two major classes of theory explaining the limits leave open questions about key issues such as how multiple simultaneously-represented items can be distinguished. We propose a palimpsest model, with the occurrent activity of a single population of neurons coding for several multi-featured items. Using a probabilistic approach to storage and recall, we show how this model can account for many qualitative aspects of existing experimental data. In our account, the underlying nature of a memory item depends entirely on the characteristics of the population representation, and we provide analytical and numerical insights into critical issues such as multiplicity and binding. We consider representations in which information about individual feature values is partially separate from the information about binding that creates single items out of multiple features. An appropriate balance between these two types of information is required to capture fully the different types of error seen in human experimental data. Our model provides the first principled account of misbinding errors. We also suggest a specific set of stimuli designed to elucidate the representations that subjects actually employ.

Humans can remember several visual items for a few seconds and recall them; however, performance deteriorates surprisingly quickly with the number of items that must be stored. Along with increasingly inaccurate recollection, subjects make association errors, sometimes apparently recalling the wrong item altogether. No current model accounts for these data fully. We discuss a simple model that focuses attention on the population representations that are putatively involved, and thereby on limits to the amount of information that can be stored and recalled. We use theoretical and numerical methods to examine the characteristics and performance of our model.

Factorial comparison of working memory models

Three questions have been prominent in the study of visual working memory limitations: (a) What is the nature of mnemonic precision (e.g., quantized or continuous)? (b) How many items are remembered? (c) To what extent do spatial binding errors account for working memory failures? Modeling studies have typically focused on comparing possible answers to a single one of these questions, even though the result of such a comparison might depend on the assumed answers to both others. Here, we consider every possible combination of previously proposed answers to the individual questions. Each model is then a point in a 3-factor model space containing a total of 32 models, of which only 6 have been tested previously. We compare all models on data from 10 delayed-estimation experiments from 6 laboratories (for a total of 164 subjects and 131,452 trials). Consistently across experiments, we find that (a) mnemonic precision is not quantized but continuous and not equal but variable across items and trials; (b) the number of remembered items is likely to be variable across trials, with a mean of 6.4 in the best model (median across subjects); (c) spatial binding errors occur but explain only a small fraction of responses (16.5% at set size 8 in the best model). We find strong evidence against all 6 documented models. Our results demonstrate the value of factorial model comparison in working memory.

Interaction between categorical knowledge and episodic memory across domains

Categorical knowledge and episodic memory have traditionally been viewed as separate lines of inquiry. Here, we present a perspective on the interrelatedness of categorical knowledge and reconstruction from memory. We address three underlying questions: what knowledge do people bring to the task of remembering? How do people integrate that knowledge with episodic memory? Is this the optimal way for the memory system to work? In the review of five studies spanning four category domains (discrete, continuous, temporal, and linguistic), we evaluate the relative contribution and the structure of influence of categorical knowledge on long-term episodic memory. These studies suggest a robustness of peoples’ knowledge of the statistical regularities of the environment, and provide converging evidence of the quality and influence of category knowledge on reconstructive memory. Lastly, we argue that combining categorical knowledge and episodic memory is an efficient strategy of the memory system.

Oscillatory brain state predicts variability in working memory

Our capacity to remember and manipulate objects in working memory (WM) is severely limited. However, this capacity limitation is unlikely to be fixed because behavioral models indicate variability from trial to trial. We investigated whether fluctuations in neural excitability at stimulus encoding, as indexed by low-frequency oscillations (in the alpha band, 8-14 Hz), contribute to this variability. Specifically, we hypothesized that the spontaneous state of alpha band activity would correlate with trial-by-trial fluctuations in visual WM. Electroencephalography recorded from human observers during a visual WM task revealed that the prestimulus desynchronization of alpha oscillations predicts the accuracy of memory recall on a trial-by-trial basis. A model-based analysis indicated that this effect arises from a modulation in the precision of memorized items, but not the likelihood of remembering them (the recall rate). The phase of posterior alpha oscillations preceding the memorized item also predicted memory accuracy. Based on correlations between prestimulus alpha levels and stimulus-related visual evoked responses, we speculate that the prestimulus state of the visual system prefigures a cascade of state-dependent processes, ultimately affecting WM-guided behavior. Overall, our results indicate that spontaneous changes in cortical excitability can have profound consequences for higher visual cognition.

The Adaptive Nature of Visual Working Memory

A growing body of scientific evidence suggests that visual working memory and statistical learning are intrinsically linked. Although visual working memory is severely resource limited, in many cases, it makes efficient use of its available resources by adapting to statistical regularities in the visual environment. However, experimental evidence also suggests that there are clear limits and biases in statistical learning. This raises the intriguing possibility that performance limitations observed in visual working memory tasks can to some degree be explained in terms of limits and biases in statistical-learning ability, rather than limits in memory capacity.

Changing concepts of working memory

Working memory is widely considered to be limited in capacity, holding a fixed, small number of items, such as Miller's 'magical number' seven or Cowan's four. It has recently been proposed that working memory might better be conceptualized as a limited resource that is distributed flexibly among all items to be maintained in memory. According to this view, the quality rather than the quantity of working memory representations determines performance. Here we consider behavioral and emerging neural evidence for this proposal.