Variability in the quality of visual working memory

Expansion and Contraction Modulate Visual Short-Term Memory

We examined the influence of expansion and contraction on visual short-term memory (VSTM) using a change-detection task. In each trial, several expanding/contracting line segments with various orientations were presented in two successive displays. The orientation of objects in the second display was either identical to, or different from, that of the first display. Observers were asked to judge the presence or absence of an orientation change in successive displays. Results showed that memory performance for expanding objects was higher than for contracting objects (expansion benefit: Experiments 1A and 1B). Further experiments focused on VSTM processing (encoding, storage, and retrieval). Regarding the retrieval stage, an expansion benefit was replicated only when the direction of motion was consistent between two successive displays (Experiment 2A). A cueing stimulus enhanced the memory performance for both expanding and contracting motions and eliminated the expansion benefit (Experiment 2B). Regarding the storage stage, we found the expansion benefit occurred only for shorter blank intervals between the two displays (Experiment 3). Regarding the encoding stage, the expansion benefit was observed regardless of presentation times (Experiment 4). These results indicate the possibility that expanding and contracting motions modulate VSTM.

Humans incorporate trial-to-trial working memory uncertainty into rewarded decisions

Information stored in working memory (WM) is incorporated into many daily decisions and actions, and many complex decisions involve WM; however, there has been little work on investigating what WM information is used in memory decisions. Here we try to draw connections between WM and decision making by manipulating prior beliefs in a standard WM task with rewards. We use this paradigm to show that WM contains a representation of the trial-by-trial uncertainty of visual stimuli. This uncertainty is incorporated into rewarded decisions along with other information, such as expectations about the environment. By studying WM in parallel with decision making, we can gain new insight into how these systems work together.

Working memory (WM) plays an important role in action planning and decision making; however, both the informational content of memory and how that information is used in decisions remain poorly understood. To investigate this, we used a color WM task in which subjects viewed colored stimuli and reported both an estimate of a stimulus color and a measure of memory uncertainty, obtained through a rewarded decision. Reported memory uncertainty is correlated with memory error, showing that people incorporate their trial-to-trial memory quality into rewarded decisions. Moreover, memory uncertainty can be combined with other sources of information; after inducing expectations (prior beliefs) about stimuli probabilities, we found that estimates became shifted toward expected colors, with the shift increasing with reported uncertainty. The data are best fit by models in which people incorporate their trial-to-trial memory uncertainty with potential rewards and prior beliefs. Our results suggest that WM represents uncertainty information, and that this can be combined with prior beliefs. This highlights the potential complexity of WM representations and shows that rewarded decision can be a powerful tool for examining WM and informing and constraining theoretical, computational, and neurobiological models of memory.

Distinguishing target biases and strategic guesses in visual working memory

Previous studies have shown that when there are statistical regularities in the items stored in visual working memory, the responses are biased toward the ensemble average. This statistical-regularity-induced bias could happen in two ways: (1) a target bias, where the individual memory representations are pulled toward the ensemble average; or (2) a strategic guess, for items that are not memorized, other information in the ensemble (e.g., another item) is reported as a substitute. Here, these two mechanisms are distinguished on the basis of a three-part model (target responses + swap responses + random guesses; e.g., Bays, Catalao, & Husain, 2009, Journal of Vision, 9, 7). The strategic guess is operationalized as swap responses, whereas the target bias is reflected by a bias parameter in the target responses. This model was applied on 8 data sets (22 observers each). In this model, contributions of target biases and strategic guesses can be clearly distinguished from each other because they lead to distinctive patterns in the distribution of responses. In the present results, strategic guesses always contributed substantially to the statistical-regularity-induced biases, whereas target biases were limited to specific conditions. All in all, the Bayesian inference in visual working memory is much more limited than what is previously advocated.

Error-correcting dynamics in visual working memory

Working memory is critical to cognition, decoupling behavior from the immediate world. Yet, it is imperfect; internal noise introduces errors into memory representations. Such errors have been shown to accumulate over time and increase with the number of items simultaneously held in working memory. Here, we show that discrete attractor dynamics mitigate the impact of noise on working memory. These dynamics pull memories towards a few stable representations in mnemonic space, inducing a bias in memory representations but reducing the effect of random diffusion. Model-based and model-free analyses of human and monkey behavior show that discrete attractor dynamics account for the distribution, bias, and precision of working memory reports. Furthermore, attractor dynamics are adaptive. They increase in strength as noise increases with memory load and experiments in humans show these dynamics adapt to the statistics of the environment, such that memories drift towards contextually-predicted values. Together, our results suggest attractor dynamics mitigate errors in working memory by counteracting noise and integrating contextual information into memories.

Neural representations in working memory are susceptible to internal noise, which scales with memory load. Here, the authors show that attractor dynamics mitigate the influence of internal noise by pulling memories towards a few stable representations.

Differential trajectories of memory quality and guessing across sequential reports from working memory

Working memory enables the storage of few items for a short period of time. Previous research has shown that items in working memory cannot be accessed equally well, indicating that they are held in at least two different states with different capacity limitations. However, it is unclear whether differences between states are due to limitations of the number of items that can be stored, or the quality with which items are stored. We employed a sequential whole-report procedure where participants reported the remembered orientation of each of two or four encoded Gabor patches. In addition, they rated their memory confidence prior to each report. Participants performed 600 trials per condition, allowing us to obtain reliable subjective ratings and estimates of precision, guessing, and misreport using a mixture model, separately for each sequential report. Different measures of memory quality consistently showed discontinuous trajectories across reports with a steep drop from the first to the second remembered item but only slight decreases thereafter. In contrast, both reported and modeled guessing changed continuously across reports. Our results support the notion of two states in working memory and show that they are distinguished by memory quality rather than quantity.

Visual short-term memory for coherent motion in video game players: evidence from a memory-masking paradigm

In this study, we investigated visual short-term memory for coherent motion in action video game players (AVGPs), non-action video game players (NAVGPs), and non-gamers (control group: CONs). Participants performed a visual memory-masking paradigm previously used with macaque monkeys and humans. In particular, we tested whether video game players form a more robust visual short-term memory trace for coherent moving stimuli during the encoding phase, and whether such memory traces are less affected by an intervening masking stimulus presented 0.2 s after the offset of the to-be-remembered sample. The results showed that task performance of all groups was affected by the masking stimulus, but video game players were affected to a lesser extent than controls. Modelling of performance values and reaction times revealed that video game players have a lower guessing rate than CONs, and higher drift rates than CONs, indicative of more efficient perceptual decisions. These results suggest that video game players exhibit a more robust VSTM trace for moving objects and this trace is less prone to external interference.

Visual short-term memory through the lifespan: Preserved benefits of context and metacognition

Visual short-term memory (VSTM) ability falls throughout the life span in healthy adults. Using a continuous report task, in a large, population-based sample, we first confirmed that this decline affects the quality and quantity of reported memories as well as knowledge of which item went where. Visual and sensorimotor precision also worsened with advancing age, but this did not account for the reduced memory performance. We then considered two strategies that older individuals might be able to adopt, to offset these memory declines: the use of contextual encoding, and metacognitive monitoring of performance. Context and metacognitive awareness were both associated with significantly better performance, however these effects did not interact with age in our sample. This suggests that older adults retain their capacity to boost memory performance through attention to external context and monitoring of their performance. Strategies that focus on taking advantage of these preserved abilities may therefore help to maintain VSTM performance with advancing age. The article reports on analysis of the Cambridge Centre for Ageing and Neuroscience (Cam-CAN) data. (PsycINFO Database Record

Encoding differences affect the number and precision of own-race versus other-race faces stored in visual working memory

Other-race faces are discriminated and recognized less accurately than own-race faces. Despite a wealth of research characterizing this other-race effect (ORE), little is known about the nature of the representations of own-race versus other-race faces. This is because traditional measures of this ORE provide a binary measure of discrimination or recognition (correct/incorrect), failing to capture potential variation in the quality of face representations. We applied a novel continuous-response paradigm to independently measure the number of own-race and other-race face representations stored in visual working memory (VWM) and the precision with which they are stored. Participants reported target own-race or other-race faces on a circular face space that smoothly varied along the dimension of identity. Using probabilistic mixture modeling, we found that following ample encoding time, the ORE is attributable to differences in the probability of a face being maintained in VWM. Reducing encoding time, a manipulation that is more sensitive to encoding limitations, caused a loss of precision or an increase in variability of VWM for other-race but not own-race faces. These results suggest that the ORE is driven by the inefficiency with which other-race faces are rapidly encoded in VWM and provide novel insights about how perceptual experience influences the representation of own-race and other-race faces in VWM.

Chunking as a rational strategy for lossy data compression in visual working memory

The nature of capacity limits for visual working memory has been the subject of an intense debate that has relied on models that assume items are encoded independently. Here we propose that instead, similar features are jointly encoded through a "chunking" process to optimize performance on visual working memory tasks. We show that such chunking can: (a) facilitate performance improvements for abstract capacity-limited systems, (b) be optimized through reinforcement, (c) be implemented by center-surround dynamics, and (d) increase effective storage capacity at the expense of recall precision. Human performance on a variant of a canonical working memory task demonstrated performance advantages, precision detriments, interitem dependencies, and trial-to-trial behavioral adjustments diagnostic of performance optimization through center-surround chunking. Models incorporating center-surround chunking provided a better quantitative description of human performance in our study as well as in a meta-analytic dataset, and apparent differences in working memory capacity across individuals were attributable to individual differences in the implementation of chunking. Our results reveal a normative rationale for center-surround connectivity in working memory circuitry, call for reevaluation of memory performance differences that have previously been attributed to differences in capacity, and support a more nuanced view of visual working memory capacity limitations: strategic tradeoff between storage capacity and memory precision through chunking contribute to flexible capacity limitations that include both discrete and continuous aspects. (PsycINFO Database Record

Current directions in visual working memory research: An introduction and emerging insights

Visual working memory (VWM) is a core construct in the cognitive (neuro-)sciences, assumed to serve as a hub for information exchange and thus supporting a multitude of cognitive functions related to processing visual information. Here, we give an introduction into key terms and paradigms and an overview of ongoing debates in the field, to which the articles collected in this Special Issue on 'Current Directions in Visual Working Memory Research' contribute. Our aim is to extract, from this overview, some 'emerging' theoretical insights concerning questions such as the optimal way to examine VWM, which types of mental representations contribute to performance on VWM tasks, and how VWM keeps features from the same object together and apart from features of concurrently maintained objects (the binding problem).

Integration of facial features under memory load

Simple visual items and complex real-world objects are stored into visual working memory as a collection of independent features, not as whole or integrated objects. Storing faces into memory might differ, however, since previous studies have reported perceptual and memory advantage for whole faces compared to other objects. We investigated whether facial features can be integrated in a statistically optimal fashion and whether memory maintenance disrupts this integration. The observers adjusted a probe – either a whole face or isolated features (eyes or mouth region) – to match the identity of a target while viewing both stimuli simultaneously or after a 1.5 second retention period. Precision was better for the whole face compared to the isolated features. Perceptual precision was higher than memory precision, as expected, and memory precision further declined as the number of memorized items was increased from one to four. Interestingly, the whole-face precision was better predicted by models assuming injection of memory noise followed by integration of features than by models assuming integration of features followed by the memory noise. The results suggest equally weighted or optimal integration of facial features and indicate that feature information is preserved in visual working memory while remembering faces.

Variable precision in visual perception

Given the same sensory stimuli in the same task, human observers do not always make the same response. Well-known sources of behavioral variability are sensory noise and guessing. Visual short-term memory (STM) studies have suggested that the precision of the sensory noise is itself variable. However, it is unknown whether precision is also variable in perceptual tasks without a memory component. We searched for evidence for variable precision in 11 visual perception tasks with a single relevant feature, orientation. We specifically examined the effect of distractor stimuli: distractors were absent, homogeneous and fixed across trials, homogeneous and variable, or heterogeneous and variable. We first considered 4 models: with and without guessing, and with and without variability in precision. We quantified the importance of both factors using 6 metrics: factor knock-in difference, factor knock-out difference, and log factor posterior ratio, each based on AIC or BIC. According to all 6 metrics, we found strong evidence for variable precision in 5 experiments. Next, we extended our model space to include potential confounding factors: the oblique effect and decision noise. This left strong evidence for variable precision in only 1 experiment, in which distractors were homogeneous but variable. Finally, when we considered suboptimal decision rules, the evidence also disappeared in this experiment. Our results provide little evidence for variable precision overall and only a hint when distractors are variable. Methodologically, the results underline the importance of including multiple factors in factorial model comparison: Testing for only 2 factors would have yielded an incorrect conclusion. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Strategic allocation of working memory resource

Visual working memory (VWM), the brief retention of past visual information, supports a range of cognitive functions. One of the defining, and largely studied, characteristics of VWM is how resource-limited it is, raising questions about how this resource is shared or split across memoranda. Since objects are rarely equally important in the real world, we ask how people split this resource in settings where objects have different levels of importance. In a psychophysical experiment, participants remembered the location of four targets with different probabilities of being tested after a delay. We then measured their memory accuracy of one of the targets. We found that participants allocated more resource to memoranda with higher priority, but underallocated resource to high- and overallocated to low-priority targets relative to the true probability of being tested. These results are well explained by a computational model in which resource is allocated to minimize expected estimation error. We replicated this finding in a second experiment in which participants bet on their memory fidelity after making the location estimate. The results of this experiment show that people have access to and utilize the quality of their memory when making decisions. Furthermore, people again allocate resource in a way that minimizes memory errors, even in a context in which an alternative strategy was incentivized. Our study not only shows that people are allocating resource according to behavioral relevance, but suggests that they are doing so with the aim of maximizing memory accuracy.

Decision making on spatially continuous scales

A new diffusion model of decision making in continuous space is presented and tested. The model is a sequential sampling model in which both spatially continuously distributed evidence and noise are accumulated up to a decision criterion (a 1 dimensional [1D] line or a 2 dimensional [2D] plane). There are two major advances represented in this research. The first is to use spatially continuously distributed Gaussian noise in the decision process (Gaussian process or Gaussian random field noise) which allows the model to represent truly spatially continuous processes. The second is a series of experiments that collect data from a variety of tasks and response modes to provide the basis for testing the model. The model accounts for the distributions of responses over position and response time distributions for the choices. The model applies to tasks in which the stimulus and the response coincide (moving eyes or fingers to brightened areas in a field of pixels) and ones in which they do not (color, motion, and direction identification). The model also applies to tasks in which the response is made with eye movements, finger movements, or mouse movements. This modeling offers a wide potential scope of applications including application to any device or scale in which responses are made on a 1D continuous scale or in a 2D spatial field. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Internal but not external noise frees working memory resources

The precision with which visual information can be recalled from working memory declines as the number of items in memory increases. This finding has been explained in terms of the distribution of a limited representational resource between items. Here we investigated how the sensory strength of memoranda affects resource allocation. We manipulated signal strength of an orientation stimulus in two ways: we varied the internal (sensory) noise by adjusting stimulus contrast, and varied the external (stimulus) noise by altering the within-stimulus variability. Both manipulations had similar effects on the precision with which the orientation could be recalled, but differed in their impact on memory for other stimuli. These results indicate that increasing internal noise released resources that could be used to store other stimuli more precisely; increasing external noise had no such effect. We show that these observations can be captured by a simple neural model of working memory encoding, in which spiking activity takes on the role of the limited resource.

Investigations of visual short-term memory typically involve memorising clearly visible objects with elementary features, such as monochromatic disks or oriented bars. Results of such studies indicate that memory is allocated like a limited resource, i.e. shared out between objects. However, in daily life we are often confronted with visual features that are difficult to make out, like when an object is in shadow, or poorly-defined, like the color of a variegated leaf. Here we asked whether these kinds of features occupy as much memory resource as simple highly-visible objects. Our results demonstrate that reducing the sensory strength of a stimulus makes the quality of recall worse, but also takes up less resource so other objects can be remembered more precisely. Increasing the variability within a stimulus worsens recall, but has no effect on how other objects are remembered. These findings can be explained by considering how visual information is stored in populations of neurons: only the manipulation of sensory strength changes the amount of spiking activity dedicated to a stimulus.

A two-level hierarchical framework of visual short-term memory

Over the last couple of decades, a vast amount of research has been dedicated to understanding the nature and the architecture of visual short-term memory (VSTM), the mechanism by which currently relevant visual information is maintained. According to discrete-capacity models, VSTM is constrained by a limited number of discrete representations held simultaneously. In contrast, shared-resource models regard VSTM as limited in resources, which can be distributed flexibly between varying numbers of representations; and a new interference model posits that capacity is limited by interference among items. In this article, we begin by reviewing benchmark findings regarding the debate over VSTM limitations, focusing on whether VSTM storage is all-or-none and on whether object complexity affects capacity. After that, we put forward a hybrid framework of VSTM architecture, arguing that this system is composed of a two-level hierarchy of memory stores, each containing a different set of representations: (1) perceptual memory, a resourcelike level containing analog automatically formed representations of visual stimuli in varying degrees of activation, and (2) visual working memory, in which a subset of three to four items from perceptual memory are bound to conceptual representations and to their locations, thus conveying discrete (digital/symbolic) information which appears quantized. While perceptual memory has a large capacity and is relatively nonselective, visual working memory is restricted in the number of items that can be maintained simultaneously, and its content is regulated by a gating mechanism.

Failure of self-consistency in the discrete resource model of visual working memory

The discrete resource model of working memory proposes that each individual has a fixed upper limit on the number of items they can store at one time, due to division of memory into a few independent "slots". According to this model, responses on short-term memory tasks consist of a mixture of noisy recall (when the tested item is in memory) and random guessing (when the item is not in memory). This provides two opportunities to estimate capacity for each observer: first, based on their frequency of random guesses, and second, based on the set size at which the variability of stored items reaches a plateau. The discrete resource model makes the simple prediction that these two estimates will coincide. Data from eight published visual working memory experiments provide strong evidence against such a correspondence. These results present a challenge for discrete models of working memory that impose a fixed capacity limit.

A resource-rational theory of set size effects in human visual working memory

Encoding precision in visual working memory decreases with the number of encoded items. Here, we propose a normative theory for such set size effects: the brain minimizes a weighted sum of an error-based behavioral cost and a neural encoding cost. We construct a model from this theory and find that it predicts set size effects. Notably, these effects are mediated by probing probability, which aligns with previous empirical findings. The model accounts well for effects of both set size and probing probability on encoding precision in nine delayed-estimation experiments. Moreover, we find support for the prediction that the total amount of invested resource can vary non-monotonically with set size. Finally, we show that it is sometimes optimal to encode only a subset or even none of the relevant items in a task. Our findings raise the possibility that cognitive "limitations" arise from rational cost minimization rather than from constraints.

Efficient Coding in Visual Working Memory Accounts for Stimulus-Specific Variations in Recall

Recall of visual features from working memory varies in both bias and precision depending on stimulus parameters. Whereas a number of models can approximate the average distribution of recall error across target stimuli, attempts to model how error varies with the choice of target have been ad hoc. Here we adapt a neural model of working memory to provide a principled account of these stimulus-specific effects, by allowing each neuron's tuning function to vary according to the principle of efficient coding, which states that neural responses should be optimized with respect to the frequency of stimuli in nature. For orientation, this means incorporating a prior that favors cardinal over oblique orientations. While continuing to capture the changes in error distribution with set size, the resulting model accurately described stimulus-specific variations as well, better than a slot-based competitor. Efficient coding produces a repulsive bias away from cardinal orientations, a bias that ought to be sensitive to changes in the environmental statistics. We subsequently tested whether shifts in the stimulus distribution influenced response bias to uniformly sampled target orientations in human subjects (of either sex). Across adaptation blocks, we manipulated the distribution of nontarget items by sampling from a bimodal congruent (incongruent) distribution with peaks centered on cardinal (oblique) orientations. Preadaptation responses were repulsed away from the cardinal axes. However, exposure to the incongruent distribution produced systematic decreases in repulsion that persisted after adaptation. This result confirms the role of prior expectation in generating stimulus-specific effects and validates the neural framework.

SIGNIFICANCE STATEMENT Theories of neural coding have been used successfully to explain how errors in recall from working memory depend on the number of items stored. However, recall of visual features also shows stimulus-specific variation in bias and precision. Here we unify two previously unconnected theories, the neural resource model of working memory and the efficient coding framework, to provide a principled account of these stimulus-specific effects. Given the importance of working memory limitations to multiple aspects of human and animal behavior, and the recent high-profile advances in theories of efficient coding, our modeling framework provides a richer, yet parsimonious, description of how orientation encoding influences visual working memory performance.

Parallel consolidation into visual working memory results in reduced precision representations

Information can be consolidated into visual working memory in parallel, i.e. two items can be consolidated in the same time required to consolidate one. However, while motion direction items consolidated in parallel are encoded at a reduced precision, no such reduction has been reported for colour. Here we examine two possible explanations for the inconsistency between the phenomena associated with consolidating these features in parallel: i) that reduced precision can only be detected when more than two colour items are consolidated in parallel, or ii) that the exposure duration used in previous studies was too long, allowing observers serially consolidate items. Our results show that (like motion direction) colour items consolidated in parallel are encoded at a reduced precision and the critical feature for detecting this phenomenon is the exposure duration. Furthermore, we demonstrate that this process is limited to two items. These findings indicate a general principle of consolidation into visual working memory, that is, a trade-off between the number of items consolidated in parallel and the precision at which they are encoded.

Unsupervised visual discrimination learning of complex stimuli: Accuracy, bias and generalization

Through same-different judgements, we can discriminate an immense variety of stimuli and consequently, they are critical in our everyday interaction with the environment. The quality of the judgements depends on familiarity with stimuli. A way to improve the discrimination is through learning, but to this day, we lack direct evidence of how learning shapes the same-different judgments with complex stimuli. We studied unsupervised visual discrimination learning in 42 participants, as they performed same-different judgments with two types of unfamiliar complex stimuli in the absence of labeling or individuation. Across nine daily training sessions with equiprobable same and different stimuli pairs, participants increased the sensitivity and the criterion by reducing the errors with both same and different pairs. With practice, there was a superior performance for different pairs and a bias for different response. To evaluate the process underlying this bias, we manipulated the proportion of same and different pairs, which resulted in an additional proportion-induced bias, suggesting that the bias observed with equal proportions was a stimulus processing bias. Overall, these results suggest that unsupervised discrimination learning occurs through changes in the stimulus processing that increase the sensory evidence and/or the precision of the working memory. Finally, the acquired discrimination ability was fully transferred to novel exemplars of the practiced stimuli category, in agreement with the acquisition of a category specific perceptual expertise.

Iconic Memories Die a Sudden Death

Iconic memory is characterized by its large storage capacity and brief storage duration, whereas visual working memory is characterized by its small storage capacity. The limited information stored in working memory is often modeled as an all-or-none process in which studied information is either successfully stored or lost completely. This view raises a simple question: If almost all viewed information is stored in iconic memory, yet one second later most of it is completely absent from working memory, what happened to it? Here, I characterized how the precision and capacity of iconic memory changed over time and observed a clear dissociation: Iconic memory suffered from a complete loss of visual items, while the precision of items retained in memory was only marginally affected by the passage of time. These results provide new evidence for the discrete-capacity view of working memory and a new characterization of iconic memory decay.

Synaptic efficacy shapes resource limitations in working memory

Working memory (WM) is limited in its temporal length and capacity. Classic conceptions of WM capacity assume the system possesses a finite number of slots, but recent evidence suggests WM may be a continuous resource. Resource models typically assume there is no hard upper bound on the number of items that can be stored, but WM fidelity decreases with the number of items. We analyze a neural field model of multi-item WM that associates each item with the location of a bump in a finite spatial domain, considering items that span a one-dimensional continuous feature space. Our analysis relates the neural architecture of the network to accumulated errors and capacity limitations arising during the delay period of a multi-item WM task. Networks with stronger synapses support wider bumps that interact more, whereas networks with weaker synapses support narrower bumps that are more susceptible to noise perturbations. There is an optimal synaptic strength that both limits bump interaction events and the effects of noise perturbations. This optimum shifts to weaker synapses as the number of items stored in the network is increased. Our model not only provides a circuit-based explanation for WM capacity, but also speaks to how capacity relates to the arrangement of stored items in a feature space.

Working Memory and Consciousness: The Current State of Play

Working memory (WM), an important posit in cognitive science, allows one to temporarily store and manipulate information in the service of ongoing tasks. WM has been traditionally classified as an explicit memory system-that is, as operating on and maintaining only consciously perceived information. Recently, however, several studies have questioned this assumption, purporting to provide evidence for unconscious WM. In this article, we focus on visual working memory (VWM) and critically examine these studies as well as studies of unconscious perception that seem to provide indirect evidence for unconscious WM. Our analysis indicates that current evidence does not support an unconscious WM store, though we offer independent reasons to think that WM may operate on unconsciously perceived information.

Evidence of gradual loss of precision for simple features and complex objects in visual working memory

Previous studies have suggested that people can maintain prioritized items in visual working memory for many seconds, with negligible loss of information over time. Such findings imply that working memory representations are robust to the potential contaminating effects of internal noise. However, once visual information is encoded into working memory, one might expect it to inevitably begin degrading over time, as this actively maintained information is no longer tethered to the original perceptual input. Here, we examined this issue by evaluating working memory for single central presentations of an oriented grating, color patch, or face stimulus, across a range of delay periods (1, 3, 6, or 12 s). We applied a mixture-model analysis to distinguish changes in memory precision over time from changes in the frequency of outlier responses that resemble random guesses. For all 3 types of stimuli, participants exhibited a clear and consistent decline in the precision of working memory as a function of temporal delay, as well as a modest increase in guessing-related responses for colored patches and face stimuli. We observed a similar loss of precision over time while controlling for temporal distinctiveness. Our results demonstrate that visual working memory is far from lossless: while basic visual features and complex objects can be maintained in a quite stable manner over time, these representations are still subject to noise accumulation and complete termination. (PsycINFO Database Record

Variability in the Precision of Children's Spatial Working Memory

Cognitive modeling studies in adults have established that visual working memory (WM) capacity depends on the representational precision, as well as its variability from moment to moment. By contrast, visuospatial WM performance in children has been typically indexed by response accuracy—a binary measure that provides less information about precision with which items are stored. Here, we aimed at identifying whether and how children’s WM performance depends on the spatial precision and its variability over time in real-world contexts. Using smartphones, 110 Grade 3 and Grade 4 students performed a spatial WM updating task three times a day in school and at home for four weeks. Measures of spatial precision (i.e., Euclidean distance between presented and reported location) were used for hierarchical modeling to estimate variability of spatial precision across different time scales. Results demonstrated considerable within-person variability in spatial precision across items within trials, from trial to trial and from occasion to occasion within days and from day to day. In particular, item-to-item variability was systematically increased with memory load and lowered with higher grade. Further, children with higher precision variability across items scored lower in measures of fluid intelligence. These findings emphasize the important role of transient changes in spatial precision for the development of WM.

Confident failures: Lapses of working memory reveal a metacognitive blind spot

Working memory performance fluctuates dramatically from trial to trial. On many trials, performance is no better than chance. Here, we assessed participants' awareness of working memory failures. We used a whole-report visual working memory task to quantify both trial-by-trial performance and trial-by-trial subjective ratings of inattention to the task. In Experiment 1 (N = 41), participants were probed for task-unrelated thoughts immediately following 20% of trials. In Experiment 2 (N = 30), participants gave a rating of their attentional state following 25% of trials. Finally, in Experiments 3a (N = 44) and 3b (N = 34), participants reported confidence of every response using a simple mouse-click judgment. Attention-state ratings and off-task thoughts predicted the number of items correctly identified on each trial, replicating previous findings that subjective measures of attention state predict working memory performance. However, participants correctly identified failures on only around 28% of failure trials. Across experiments, participants' metacognitive judgments reliably predicted variation in working memory performance but consistently and severely underestimated the extent of failures. Further, individual differences in metacognitive accuracy correlated with overall working memory performance, suggesting that metacognitive monitoring may be key to working memory success.

A neural model of retrospective attention in visual working memory

An informative cue that directs attention to one of several items in working memory improves subsequent recall of that item. Here we examine the mechanism of this retro-cue effect using a model of short-term memory based on neural population coding. Our model describes recalled feature values as the output of an optimal decoding of spikes generated by a tuned population of neurons. This neural model provides a better account of human recall data than an influential model that assumes errors can be described as a mixture of normally distributed noise and random guesses. The retro-cue benefit is revealed to be consistent with a higher firing rate of the population encoding the cued versus uncued items, with no difference in tuning specificity. Additionally, a retro-cued item is less likely to be swapped with another item in memory, an effect that can also be explained by greater activity of the underlying population. These results provide a parsimonious account of the effects of retrospective attention on recall and demonstrate a principled method for investigating neural representations with behavioral tasks.

Synaptic augmentation in a cortical circuit model reproduces serial dependence in visual working memory

Recent work has established that visual working memory is subject to serial dependence: current information in memory blends with that from the recent past as a function of their similarity. This tuned temporal smoothing likely promotes the stability of memory in the face of noise and occlusion. Serial dependence accumulates over several seconds in memory and deteriorates with increased separation between trials. While this phenomenon has been extensively characterized in behavior, its neural mechanism is unknown. In the present study, we investigate the circuit-level origins of serial dependence in a biophysical model of cortex. We explore two distinct kinds of mechanisms: stable persistent activity during the memory delay period and dynamic “activity-silent” synaptic plasticity. We find that networks endowed with both strong reverberation to support persistent activity and dynamic synapses can closely reproduce behavioral serial dependence. Specifically, elevated activity drives synaptic augmentation, which biases activity on the subsequent trial, giving rise to a spatiotemporally tuned shift in the population response. Our hybrid neural model is a theoretical advance beyond abstract mathematical characterizations, offers testable hypotheses for physiological research, and demonstrates the power of biological insights to provide a quantitative explanation of human behavior.

The effects of delay duration on visual working memory for orientation

We used a delayed-estimation paradigm to characterize the joint effects of set size (one, two, four, or six) and delay duration (1, 2, 3, or 6 s) on visual working memory for orientation. We conducted two experiments: one with delay durations blocked, another with delay durations interleaved. As dependent variables, we examined four model-free metrics of dispersion as well as precision estimates in four simple models. We tested for effects of delay time using analyses of variance, linear regressions, and nested model comparisons. We found significant effects of set size and delay duration on both model-free and model-based measures of dispersion. However, the effect of delay duration was much weaker than that of set size, dependent on the analysis method, and apparent in only a minority of subjects. The highest forgetting slope found in either experiment at any set size was a modest 1.14°/s. As secondary results, we found a low rate of nontarget reports, and significant estimation biases towards oblique orientations (but no dependence of their magnitude on either set size or delay duration). Relative stability of working memory even at higher set sizes is consistent with earlier results for motion direction and spatial frequency. We compare with a recent study that performed a very similar experiment.

Accounting for stimulus-specific variation in precision reveals a discrete capacity limit in visual working memory

If we view a visual scene that contains many objects, then momentarily close our eyes, some details persist while others seem to fade. Discrete models of visual working memory (VWM) assume that only a few items can be actively maintained in memory, beyond which pure guessing will emerge. Alternatively, continuous resource models assume that all items in a visual scene can be stored with some precision. Distinguishing between these competing models is challenging, however, as resource models that allow for stochastically variable precision (across items and trials) can produce error distributions that resemble random guessing behavior. Here, we evaluated the hypothesis that a major source of variability in VWM performance arises from systematic variation in precision across the stimuli themselves; such stimulus-specific variability can be incorporated into both discrete-capacity and variable-precision resource models. Participants viewed multiple oriented gratings, and then reported the orientation of a cued grating from memory. When modeling the overall distribution of VWM errors, we found that the variable-precision resource model outperformed the discrete model. However, VWM errors revealed a pronounced "oblique effect," with larger errors for oblique than cardinal orientations. After this source of variability was incorporated into both models, we found that the discrete model provided a better account of VWM errors. Our results demonstrate that variable precision across the stimulus space can lead to an unwarranted advantage for resource models that assume stochastically variable precision. When these deterministic sources are adequately modeled, human working memory performance reveals evidence of a discrete capacity limit. (PsycINFO Database Record

Serial dependence is absent at the time of perception but increases in visual working memory

Recent experiments have shown that visual cognition blends current input with that from the recent past to guide ongoing decision making. This serial dependence appears to exploit the temporal autocorrelation normally present in visual scenes to promote perceptual stability. While this benefit has been assumed, evidence that serial dependence directly alters stimulus perception has been limited. In the present study, we parametrically vary the delay between stimulus and response in a spatial delayed response task to explore the trajectory of serial dependence from the moment of perception into post-perceptual visual working memory. We find that behavioral responses made immediately after viewing a stimulus show evidence of adaptation, but not attractive serial dependence. Only as the memory period lengthens is a blending of past and present information apparent in behavior, reaching its maximum with a delay of six seconds. These results dovetail with other recent findings to bolster the interpretation that serial dependence is a phenomenon of mnemonic rather than perceptual processes. However, even while this pattern of effects in group-averaged data has now been found consistently, we show that the relative strengths of adaptation and serial dependence vary widely across individuals. Finally, we demonstrate that when leading mathematical models of working memory are adjusted to account for these trial-history effects, their fit to behavioral data is substantially improved.

Using a Large-scale Neural Model of Cortical Object Processing to Investigate the Neural Substrate for Managing Multiple Items in Short-term Memory

Many cognitive and computational models have been proposed to help understand working memory. In this article, we present a simulation study of cortical processing of visual objects during several working memory tasks using an extended version of a previously constructed large-scale neural model [Tagamets, M. A., & Horwitz, B. Integrating electrophysiological and anatomical experimental data to create a large-scale model that simulates a delayed match-to-sample human brain imaging study. Cerebral Cortex, 8, 310-320, 1998]. The original model consisted of arrays of Wilson-Cowan type of neuronal populations representing primary and secondary visual cortices, inferotemporal (IT) cortex, and pFC. We added a module representing entorhinal cortex, which functions as a gating module. We successfully implemented multiple working memory tasks using the same model and produced neuronal patterns in visual cortex, IT cortex, and pFC that match experimental findings. These working memory tasks can include distractor stimuli or can require that multiple items be retained in mind during a delay period (Sternberg's task). Besides electrophysiology data and behavioral data, we also generated fMRI BOLD time series from our simulation. Our results support the involvement of IT cortex in working memory maintenance and suggest the cortical architecture underlying the neural mechanisms mediating particular working memory tasks. Furthermore, we noticed that, during simulations of memorizing a list of objects, the first and last items in the sequence were recalled best, which may implicate the neural mechanism behind this important psychological effect (i.e., the primacy and recency effect).

Interactions between visual working memory representations

We investigated whether the representations of different objects are maintained independently in working memory or interact with each other. Observers were shown two sequentially presented orientations and required to reproduce each orientation after a delay. The sequential presentation minimized perceptual interactions so that we could isolate interactions between memory representations per se. We found that similar orientations were repelled from each other whereas dissimilar orientations were attracted to each other. In addition, when one of the items was given greater attentional priority by means of a cue, the representation of the high-priority item was not influenced very much by the orientation of the low-priority item, but the representation of the low-priority item was strongly influenced by the orientation of the high-priority item. This indicates that attention modulates the interactions between working memory representations. In addition, errors in the reported orientations of the two objects were positively correlated under some conditions, suggesting that representations of distinct objects may become grouped together in memory. Together, these results demonstrate that working-memory representations are not independent but instead interact with each other in a manner that depends on attentional priority.

Monkeys and humans take local uncertainty into account when localizing a change

Since sensory measurements are noisy, an observer is rarely certain about the identity of a stimulus. In visual perception tasks, observers generally take their uncertainty about a stimulus into account when doing so helps task performance. Whether the same holds in visual working memory tasks is largely unknown. Ten human and two monkey subjects localized a single change in orientation between a sample display containing three ellipses and a test display containing two ellipses. To manipulate uncertainty, we varied the reliability of orientation information by making each ellipse more or less elongated (two levels); reliability was independent across the stimuli. In both species, a variable-precision encoding model equipped with an “uncertainty–indifferent” decision rule, which uses only the noisy memories, fitted the data poorly. In both species, a much better fit was provided by a model in which the observer also takes the levels of reliability-driven uncertainty associated with the memories into account. In particular, a measured change in a low-reliability stimulus was given lower weight than the same change in a high-reliability stimulus. We did not find strong evidence that observers took reliability-independent variations in uncertainty into account. Our results illustrate the importance of studying the decision stage in comparison tasks and provide further evidence for evolutionary continuity of working memory systems between monkeys and humans.

Visual short-term memory for oriented, colored objects

A central question in the study of visual short-term memory (VSTM) has been whether its basic units are objects or features. Most studies addressing this question have used change detection tasks in which the feature value before the change is highly discriminable from the feature value after the change. This approach assumes that memory noise is negligible, which recent work has shown not to be the case. Here, we investigate VSTM for orientation and color within a noisy-memory framework, using change localization with a variable magnitude of change. A specific consequence of the noise is that it is necessary to model the inference (decision) stage. We find that (a) orientation and color have independent pools of memory resource (consistent with classic results); (b) an irrelevant feature dimension is either encoded but ignored during decision-making, or encoded with low precision and taken into account during decision-making; and (c) total resource available in a given feature dimension is lower in the presence of task-relevant stimuli that are neutral in that feature dimension. We propose a framework in which feature resource comes both in packaged and in targeted form.

When past is present: Substitutions of long-term memory for sensory evidence in perceptual judgments

When perception is underdetermined by current sensory inputs, memories for related experiences in the past might fill in missing detail. To evaluate this possibility, we measured the likelihood of relying on long-term memory versus sensory evidence when judging the appearance of an object near the threshold of awareness. Specifically, we associated colors with shapes in long-term memory and then presented the shapes again later in unrelated colors and had observers judge the appearance of the new colors. We found that responses were well characterized as a bimodal mixture of original and current-color representations (vs. an integrated unimodal representation). That is, although irrelevant to judgments of the current color, observers occasionally anchored their responses on the original colors in memory. Moreover, the likelihood of such memory substitutions increased when sensory input was degraded. In fact, they occurred even in the absence of sensory input when observers falsely reported having seen something. Thus, although perceptual judgments intuitively seem to reflect the current state of the environment, they can also unknowingly be dictated by past experiences.

Clear evidence for item limits in visual working memory

There is a consensus that visual working memory (WM) resources are sharply limited, but debate persists regarding the simple question of whether there is a limit to the total number of items that can be stored concurrently. Zhang and Luck (2008) advanced this debate with an analytic procedure that provided strong evidence for random guessing responses, but their findings can also be described by models that deny guessing while asserting a high prevalence of low precision memories. Here, we used a whole report memory procedure in which subjects reported all items in each trial and indicated whether they were guessing with each response. Critically, this procedure allowed us to measure memory performance for all items in each trial. When subjects were asked to remember 6 items, the response error distributions for about 3 out of the 6 items were best fit by a parameter-free guessing model (i.e. a uniform distribution). In addition, subjects' self-reports of guessing precisely tracked the guessing rate estimated with a mixture model. Control experiments determined that guessing behavior was not due to output interference, and that there was still a high prevalence of guessing when subjects were instructed not to guess. Our novel approach yielded evidence that guesses, not low-precision representations, best explain limitations in working memory. These guesses also corroborate a capacity-limited working memory system - we found evidence that subjects are able to report non-zero information for only 3-4 items. Thus, WM capacity is constrained by an item limit that precludes the storage of more than 3-4 individuated feature values.

Superior Intraparietal Sulcus Controls the Variability of Visual Working Memory Precision

Limitations of working memory (WM) capacity depend strongly on the cognitive resources that are available for maintaining WM contents in an activated state. Increasing the number of items to be maintained in WM was shown to reduce the precision of WM and to increase the variability of WM precision over time. Although WM precision was recently associated with neural codes particularly in early sensory cortex, we have so far no understanding of the neural bases underlying the variability of WM precision, and how WM precision is preserved under high load. To fill this gap, we combined human fMRI with computational modeling of behavioral performance in a delayed color-estimation WM task. Behavioral results replicate a reduction of WM precision and an increase of precision variability under high loads (5 > 3 > 1 colors). Load-dependent BOLD signals in primary visual cortex (V1) and superior intraparietal sulcus (IPS), measured during the WM task at 2-4 s after sample onset, were modulated by individual differences in load-related changes in the variability of WM precision. Although stronger load-related BOLD increase in superior IPS was related to lower increases in precision variability, thus stabilizing WM performance, the reverse was observed for V1. Finally, the detrimental effect of load on behavioral precision and precision variability was accompanied by a load-related decline in the accuracy of decoding the memory stimuli (colors) from left superior IPS. We suggest that the superior IPS may contribute to stabilizing visual WM performance by reducing the variability of memory precision in the face of higher load.

SIGNIFICANCE STATEMENT

This study investigates the neural bases of capacity limitations in visual working memory by combining fMRI with cognitive modeling of behavioral performance, in human participants. It provides evidence that the superior intraparietal sulcus (IPS) is a critical brain region that influences the variability of visual working memory precision between and within individuals (Fougnie et al., 2012; van den Berg et al., 2012) under increased memory load, possibly in cooperation with perceptual systems of the occipital cortex. These findings substantially extend our understanding of the nature of capacity limitations in visual working memory and their neural bases. Our work underlines the importance of integrating cognitive modeling with univariate and multivariate methods in fMRI research, thus improving our knowledge of brain-behavior relationships.

Unexpected events disrupt visuomotor working memory and increase guessing

When an unexpected event, such as a car horn honking, occurs in daily life, it often disrupts our train of thought. In the lab, this effect was recently modeled with a task in which verbal working memory (WM) was disrupted by unexpected auditory events (Wessel et al. in Nature Communications, 7, 11195, 2016). Here we tested whether this effect extends to a different type of WM-namely, visuomotor. We found that unexpected auditory events similarly decremented visuomotor WM. Moreover, this effect persisted for many more trials than had previously been shown for verbal WM, and the effect occurred for two different types of unexpected auditory events. Furthermore, we found that unexpected events decremented WM by decreasing the quantity, but not necessarily the quality, of items stored. These results showed an impact of unexpected events on visuomotor WM that was statistically robust and endured across time. They also showed that the effect was based on an increase in guessing, consistent with a neuroscience-inspired theory that unexpected events "wipe out" WM by stopping the ongoing maintenance of the trace. This new task paradigm is an excellent vehicle for further explorations of distractibility.

Evolution in Mind: Evolutionary Dynamics, Cognitive Processes, and Bayesian Inference

Evolutionary theory describes the dynamics of population change in settings affected by reproduction, selection, mutation, and drift. In the context of human cognition, evolutionary theory is most often invoked to explain the origins of capacities such as language, metacognition, and spatial reasoning, framing them as functional adaptations to an ancestral environment. However, evolutionary theory is useful for understanding the mind in a second way: as a mathematical framework for describing evolving populations of thoughts, ideas, and memories within a single mind. In fact, deep correspondences exist between the mathematics of evolution and of learning, with perhaps the deepest being an equivalence between certain evolutionary dynamics and Bayesian inference. This equivalence permits reinterpretation of evolutionary processes as algorithms for Bayesian inference and has relevance for understanding diverse cognitive capacities, including memory and creativity.

A signature of neural coding at human perceptual limits

Simple visual features, such as orientation, are thought to be represented in the spiking of visual neurons using population codes. I show that optimal decoding of such activity predicts characteristic deviations from the normal distribution of errors at low gains. Examining human perception of orientation stimuli, I show that these predicted deviations are present at near-threshold levels of contrast. The findings may provide a neural-level explanation for the appearance of a threshold in perceptual awareness whereby stimuli are categorized as seen or unseen. As well as varying in error magnitude, perceptual judgments differ in certainty about what was observed. I demonstrate that variations in the total spiking activity of a neural population can account for the empirical relationship between subjective confidence and precision. These results establish population coding and decoding as the neural basis of perception and perceptual confidence.

The impact of early visual cortex transcranial magnetic stimulation on visual working memory precision and guess rate

Neuroimaging studies have demonstrated that activity patterns in early visual areas predict stimulus properties actively maintained in visual working memory. Yet, the mechanisms by which such information is represented remain largely unknown. In this study, observers remembered the orientations of 4 briefly presented gratings, one in each quadrant of the visual field. A 10Hz Transcranial Magnetic Stimulation (TMS) triplet was applied directly at stimulus offset, or midway through a 2-second delay, targeting early visual cortex corresponding retinotopically to a sample item in the lower hemifield. Memory for one of the four gratings was probed at random, and participants reported this orientation via method of adjustment. Recall errors were smaller when the visual field location targeted by TMS overlapped with that of the cued memory item, compared to errors for stimuli probed diagonally to TMS. This implied topographic storage of orientation information, and a memory-enhancing effect at the targeted location. Furthermore, early pulses impaired performance at all four locations, compared to late pulses. Next, response errors were fit empirically using a mixture model to characterize memory precision and guess rates. Memory was more precise for items proximal to the pulse location, irrespective of pulse timing. Guesses were more probable with early TMS pulses, regardless of stimulus location. Thus, while TMS administered at the offset of the stimulus array might disrupt early-phase consolidation in a non-topographic manner, TMS also boosts the precise representation of an item at its targeted retinotopic location, possibly by increasing attentional resources or by injecting a beneficial amount of noise.

A two-phase model of resource allocation in visual working memory

Two broad theories of visual working memory (VWM) storage have emerged from current research, a discrete slot-based theory and a continuous resource theory. However, neither the discrete slot-based theory or continuous resource theory clearly stipulates how the mental commodity for VWM (discrete slot or continuous resource) is allocated. Allocation may be based on the number of items via stimulus-driven factors, or it may be based on task demands via voluntary control. Previous studies have obtained conflicting results regarding the automaticity versus controllability of such allocation. In the current study, we propose a two-phase allocation model, in which the mental commodity could be allocated only by stimulus-driven factors in the early consolidation phase. However, when there is sufficient time to complete the early phase, allocation can enter the late consolidation phase, where it can be flexibly and voluntarily controlled according to task demands. In an orientation recall task, we instructed participants to store either fewer items at high-precision or more items at low-precision. In 3 experiments, we systematically manipulated memory set size and exposure duration. We did not find an effect of task demands when the set size was high and exposure duration was short. However, when we either decreased the set size or increased the exposure duration, we found a trade-off between the number and precision of VWM representations. These results can be explained by a two-phase model, which can also account for previous conflicting findings in the literature. (PsycINFO Database Record

Fechner's law in metacognition: A quantitative model of visual working memory confidence

Although visual working memory (VWM) has been studied extensively, it is unknown how people form confidence judgments about their memories. Peirce (1878) speculated that Fechner's law-which states that sensation is proportional to the logarithm of stimulus intensity-might apply to confidence reports. Based on this idea, we hypothesize that humans map the precision of their VWM contents to a confidence rating through Fechner's law. We incorporate this hypothesis into the best available model of VWM encoding and fit it to data from a delayed-estimation experiment. The model provides an excellent account of human confidence rating distributions as well as the relation between performance and confidence. Moreover, the best-fitting mapping in a model with a highly flexible mapping closely resembles the logarithmic mapping, suggesting that no alternative mapping exists that accounts better for the data than Fechner's law. We propose a neural implementation of the model and find that this model also fits the behavioral data well. Furthermore, we find that jointly fitting memory errors and confidence ratings boosts the power to distinguish previously proposed VWM encoding models by a factor of 5.99 compared to fitting only memory errors. Finally, we show that Fechner's law also accounts for metacognitive judgments in a word recognition memory task, which is a first indication that it may be a general law in metacognition. Our work presents the first model to jointly account for errors and confidence ratings in VWM and could lay the groundwork for understanding the computational mechanisms of metacognition. (PsycINFO Database Record

Recognition Decisions From Visual Working Memory Are Mediated by Continuous Latent Strengths

Making recognition decisions often requires us to reference the contents of working memory, the information available for ongoing cognitive processing. As such, understanding how recognition decisions are made when based on the contents of working memory is of critical importance. In this work we examine whether recognition decisions based on the contents of visual working memory follow a continuous decision process of graded information about the correct choice or a discrete decision process reflecting only knowing and guessing. We find a clear pattern in favor of a continuous latent strength model of visual working memory-based decision making, supporting the notion that visual recognition decision processes are impacted by the degree of matching between the contents of working memory and the choices given. Relation to relevant findings and the implications for human information processing more generally are discussed.

Categorical working memory representations are used in delayed estimation of continuous colors

In the last decade, major strides have been made in understanding visual working memory through mathematical modeling of color production responses. In the delayed color estimation task (Wilken & Ma, 2004), participants are given a set of colored squares to remember, and a few seconds later asked to reproduce those colors by clicking on a color wheel. The degree of error in these responses is characterized with mathematical models that estimate working memory precision and the proportion of items remembered by participants. A standard mathematical model of color memory assumes that items maintained in memory are remembered through memory for precise details about the particular studied shade of color. We contend that this model is incomplete in its present form because no mechanism is provided for remembering the coarse category of a studied color. In the present work, we remedy this omission and present a model of visual working memory that includes both continuous and categorical memory representations. In 2 experiments, we show that our new model outperforms this standard modeling approach, which demonstrates that categorical representations should be accounted for by mathematical models of visual working memory. (PsycINFO Database Record