Comparing memory capacity across stimuli requires maximally dissimilar foils: Using deep convolutional neural networks to understand visual working memory capacity for real-world objects

The capacity of visual working and visual long-term memory plays a critical role in theories of cognitive architecture and the relationship between memory and other cognitive systems. Here, we argue that before asking the question of how capacity varies across different stimuli or what the upper bound of capacity is for a given memory system, it is necessary to establish a methodology that allows a fair comparison between distinct stimulus sets and conditions. One of the most important factors determining performance in a memory task is target/foil dissimilarity. We argue that only by maximizing the dissimilarity of the target and foil in each stimulus set can we provide a fair basis for memory comparisons between stimuli. In the current work we focus on a way to pick such foils objectively for complex, meaningful real-world objects by using deep convolutional neural networks, and we validate this using both memory tests and similarity metrics. Using this method, we then provide evidence that there is a greater capacity for real-world objects relative to simple colors in visual working memory; critically, we also show that this difference can be reduced or eliminated when non-comparable foils are used, potentially explaining why previous work has not always found such a difference. Our study thus demonstrates that working memory capacity depends on the type of information that is remembered and that assessing capacity depends critically on foil dissimilarity, especially when comparing memory performance and other cognitive systems across different stimulus sets.

Investigating the effects of perceptual complexity versus conceptual meaning on the object benefit in visual working memory

Previous research has demonstrated greater visual working memory (VWM) performance for real-world objects compared with simple features. Greater amplitudes of the contralateral delay activity (CDA)-a sustained event-related potential measured during the delay period of a VWM task-have also been noted for meaningful stimuli, despite being thought of as a neural marker of a fixed working memory capacity. The current study aimed to elucidate the factors underlying improved memory performance for real-world objects by isolating the relative contributions of perceptual complexity (i.e., number of visual features) and conceptual meaning (i.e., availability of semantic, meaningful features). Participants (N = 22) performed a lateralized VWM task to test their memory of intact real-world objects, scrambled real-world objects and colours. The CDA was measured during both encoding and WM retention intervals (600-1000 ms and 1300-1700 ms poststimulus onset, respectively), and behavioural performance was estimated by using d' (memory strength in a two-alternative forced choice task). Behavioural results revealed significantly better performance within-subjects for real-world objects relative to scrambled objects and colours, with no difference between colours and scrambled objects. The amplitude of the CDA was also largest for intact real-world objects, with no difference in magnitude for scrambled objects and colours, during working memory maintenance. However, during memory encoding, both the colours and intact real-world objects had significantly greater amplitudes than scrambled objects and were comparable in magnitude. Overall, findings suggest that conceptual meaning (semantics) supports the memory benefit for real-world objects.

Sequential encoding aids working memory for meaningful objects' identities but not for their colors

Previous studies have found that real-world objects' identities are better remembered than simple features like colored circles, and this effect is particularly pronounced when these stimuli are encoded one by one in a serial, item-based way. Recent work has also demonstrated that memory for simple features like color is improved if these colors are part of real-world objects, suggesting that meaningful objects can serve as a robust memory scaffold for their associated low-level features. However, it is unclear whether the improved color memory that arises from the colors appearing on real-world objects is affected by encoding format, in particular whether items are encoded sequentially or simultaneously. We test this using randomly colored silhouettes of recognizable versus unrecognizable scrambled objects that offer a uniquely controlled set of stimuli to test color working memory of meaningful versus non-meaningful objects. Participants were presented with four stimuli (silhouettes of objects or scrambled shapes) simultaneously or sequentially. After a short delay, they reported either which colors or which shapes they saw in a two-alternative forced-choice task. We replicated previous findings that meaningful stimuli boost working memory performance for colors (Exp. 1). We found that when participants remembered the colors (Exp. 2) there was no difference in performance across the two encoding formats. However, when participants remembered the shapes and thus identity of the objects (Exp. 3), sequential presentation resulted in better performance than simultaneous presentation. Overall, these results show that different encoding formats can flexibly impact visual working memory depending on what the memory-relevant feature is.

Which factor affects the storage of real-world object information in visual working memory: perceptual or conceptual information?

Visual working memory (VWM) is a limited dynamic memory system where people temporarily store and process visual information. Previous research showed that real-world objects do not have a fixed capacity compared to simple ones. In konkle's study, they found that the conceptual information and perception information of real-world objects had different effects on visual long-term memory (VLTM) capacity. VLTM capacity was more dependent on conceptual information than the perceptual distinctiveness of real-world objects. However, we did not know how the intrinsic attribute of real-world objects affects VWM capacity yet. In the current research, we set five experiments to explore the comparative effects of conceptual vs. perceptual information of real-world objects in VWM capacity. Our results suggested that VWM capacity was more dependent on the perceptual distinctiveness of real-world objects than on conceptual structure. These data provide evidence that VWM capacity for real-world objects depends more on perceptual information than on conceptual structure.

Massive visual long-term memory is largely dependent on meaning

Previous research demonstrated a massive capacity of visual long-term memory (VLTM) for meaningful images. However, the capacity and limits of a "pure" VLTM that is independent of conceptual information still need to be determined. In the encoding phase of three experiments, participants viewed hundreds of images depicting real-world objects, along with visually similar images that were stripped of their semantic meaning. VLTM was evaluated using a four-alternative-forced-choice test including old and new images and their counterpart mirror transformations. The results revealed superior memory for meaningful than for meaningless stimuli and importantly, there was no hint of a massive VLTM for the meaningless items. Furthermore, when examining memory recognition of visual properties per-se (i.e., original/mirror state), memory was overall poor, and practically negligible for the meaningless items. Taken together, our findings suggest that meaning is critical for massive VLTM and for the ability to store visual properties.

Remembering a Virtual Museum Tour: Viewing Time, Memory Reactivation, and Memory Distortion

A variety of evidence demonstrates that memory is a reconstructive process prone to errors and distortions. However, the complex relationship between memory encoding, strength of memory reactivation, and the likelihood of reporting true or false memories has yet to be ascertained. We address this issue in a setting that mimics a real-life experience: We asked participants to take a virtual museum tour in which they freely explored artworks included in the exhibit, while we measured the participants’ spontaneous viewing time of each explored artwork. In a following memory reactivation phase, participants were presented again with explored artworks (reactivated targets), followed by novel artworks not belonging to the same exhibit (activated lures). For each of these objects, participants provided a reliving rating that indexed the strength of memory reactivation. In the final memory recognition phase, participants underwent an old/new memory task, involving reactivated vs. baseline (i.e., non-reactivated) targets, and activated and baseline lures. The results showed that those targets that were spontaneously viewed for a longer amount of time were more frequently correctly recognized. This pattern was particularly true for reactivated targets associated with greater memory strength (a higher reliving rating). Paradoxically, however, lures that were presented after targets associated with higher reliving ratings in the reactivation phase were more often erroneously recognized as artworks encountered during the tour. This latter finding indicates that memory intrusions, irrespective of the viewing time, are more likely to take place and be incorporated into true memories when the strength of target memory is higher.

Are all real-world objects created equal? Estimating the "set-size" of the search target in visual working memory

Are all real-world objects created equal? Visual search difficulty increases with the number of targets and as target-related visual working memory (VWM) load increases. Our goal was to investigate the load imposed by individual real-world objects held in VWM in the context of search. Measures of visual clutter attempt to quantify real-world set-size in the context of scenes. We applied one of these measures, the number of proto-objects, to individual real-world objects and used contralateral delay activity (CDA) to measure the resulting VWM load. The current study presented a real-world object as a target cue, followed by a delay where CDA was measured. This was followed by a four-object search array. We compared CDA and later search performance from target cues containing a high or low number of proto-objects. High proto-object target cues resulted in greater CDA, longer search RTs, target dwell times, and reduced search guidance, relative to low proto-object targets. These findings demonstrate that targets with more proto-objects result in a higher VWM load and reduced search performance. This shows that the number of proto-objects contained within individual objects produce set-size like effects in VWM and suggests proto-objects may be a viable unit of measure of real-world VWM load. Importantly, this demonstrates that not all real-world objects are created equal.

Real-world objects are not stored in holistic representations in visual working memory

When storing multiple objects in visual working memory, observers sometimes misattribute perceived features to incorrect locations or objects. These misattributions are called binding errors (or swaps) and have been previously demonstrated mostly in simple objects whose features are easy to encode independently and arbitrarily chosen, like colors and orientations. Here, we tested whether similar swaps can occur with real-world objects, where the connection between features is meaningful rather than arbitrary. In Experiments 1 and 2, observers were simultaneously shown four items from two object categories. Within a category, the two exemplars could be presented in either the same or different states (e.g., open/closed; full/empty). After a delay, both exemplars from one of the categories were probed, and participants had to recognize which exemplar went with which state. We found good memory for state information and exemplar information on their own, but a significant memory decrement for exemplar–state combinations, suggesting that binding was difficult for observers and swap errors occurred even for meaningful real-world objects. In Experiment 3, we used the same task, but in one-half of the trials, the locations of the exemplars were swapped at test. We found that there are more errors in general when the locations of exemplars were swapped. We concluded that the internal features of real-world objects are not perfectly bound in working memory, and location updates impair object and feature representations. Overall, we provide evidence that even real-world objects are not stored in an entirely unitized format in working memory.

Greater Visual Working Memory Capacity for Visually Matched Stimuli When They Are Perceived as Meaningful

Almost all models of visual working memory-the cognitive system that holds visual information in an active state-assume it has a fixed capacity: Some models propose a limit of three to four objects, where others propose there is a fixed pool of resources for each basic visual feature. Recent findings, however, suggest that memory performance is improved for real-world objects. What supports these increases in capacity? Here, we test whether the meaningfulness of a stimulus alone influences working memory capacity while controlling for visual complexity and directly assessing the active component of working memory using EEG. Participants remembered ambiguous stimuli that could either be perceived as a face or as meaningless shapes. Participants had higher performance and increased neural delay activity when the memory display consisted of more meaningful stimuli. Critically, by asking participants whether they perceived the stimuli as a face or not, we also show that these increases in visual working memory capacity and recruitment of additional neural resources are because of the subjective perception of the stimulus and thus cannot be driven by physical properties of the stimulus. Broadly, this suggests that the capacity for active storage in visual working memory is not fixed but that more meaningful stimuli recruit additional working memory resources, allowing them to be better remembered.