Neural limits to representing objects still within view

Dissecting the components of error in analogue report tasks

Over the last two decades, the analogue report task has become a standard method for measuring the fidelity of visual representations across research domains including perception, attention, and memory. Despite its widespread use, there has been no methodical investigation of the different task parameters that might contribute to response variability. To address this gap, we conducted two experiments manipulating components of a typical analogue report test of memory for colour hue. We found that human response errors were independently affected by changes in storage and maintenance requirements of the task, demonstrated by a strong effect of set size even in the absence of a memory delay. In contrast, response variability remained unaffected by physical size of the colour wheel, implying negligible contribution of motor noise to task performance, or by its chroma radius, highlighting non-uniformity of the standard colour space. Comparing analogue report to a matched forced-choice task, we found variation in adjustment criterion made a limited contribution to analogue report variability, becoming meaningful only with low representational noise. Our findings validate the analogue report task as a robust measure of representational fidelity for most purposes, while also quantifying non-representational sources of noise that would limit its reliability in specialized settings.

Electrophysiology Reveals That Intuitive Physics Guides Visual Tracking and Working Memory

Starting in early infancy, our perception and predictions are rooted in strong expectations about the behavior of everyday objects. These intuitive physics expectations have been demonstrated in numerous behavioral experiments, showing that even pre-verbal infants are surprised when something impossible happens (e.g., when objects magically appear or disappear). However, it remains unclear whether and how physical expectations shape different aspects of moment-by-moment online visual scene processing, unrelated to explicit physical reasoning. In two EEG experiments, people watched short videos like those used in behavioral studies with adults and infants, and more recently in AI benchmarks. Objects moved on a stage, and were briefly hidden behind an occluder, with the scene either unfolding as expected, or violating object permanence (adding or removing an object). We measured the contralateral delay activity, an electrophysiological marker of online processing, to examine participants' working memory (WM) representations, as well as their ability to continuously track the objects in the scene. We found that both types of object permanence violations disrupted tracking, even though violations involved perceptually non-salient events (magical vanishing) or new objects that weren't previously tracked (magical creation). Physical violations caused WM to reset, i.e., to discard the original scene representation before it could recover and represent the updated number of items. Providing a physical explanation for the violations (a hole behind the occluder) restored object tracking, and we found evidence that WM continued to represent items that disappeared 'down the hole'. Our results show how intuitive physical expectations shape online representations, and form the basis of dynamic object tracking.

Behavioral signatures of the rapid recruitment of long-term memory to overcome working memory capacity limits

Working- and long-term memory are often studied in isolation. To better understand the specific limitations of working memory, effort is made to reduce the potential influence of long-term memory on performance in working memory tasks (e.g., asking participants to remember artificial, abstract items rather than familiar real-world objects). However, in everyday life we use working- and long-term memory in tandem. Here, our goal was to characterize how long-term memory can be recruited to circumvent capacity limits in a typical visual working memory task (i.e., remembering colored squares). Prior work has shown that incidental repetitions of working memory arrays often do not improve visual working memory performance - even after dozens of incidental repetitions, working memory performance often shows no improvement for repeated arrays. Here, we used a whole-report working memory task with explicit rather than incidental repetitions of arrays. In contrast to prior work with incidental repetitions, in two behavioral experiments we found that explicit repetitions of arrays yielded robust improvement to working memory performance, even after a single repetition. Participants performed above chance at recognizing repeated arrays in a later long-term memory test, consistent with the idea that long-term memory was used to rapidly improve performance across array repetitions. Finally, we analyzed inter-item response times and we found a response time signature of chunk formation that only emerged after the array was repeated (inter-response time slowing after two to three items); thus, inter-item response times may be useful for examining the coordinated interaction of visual working and long-term memory in future work.

Putting the "Return" Back in the Inhibition of Return Effect in Working Memory

The inhibition of return effect in perception refers to the observation that one is slower to re-attend a location that was attended right before, compared to a location that was not attended right before. Johnson et al. (2013, Psych. Sc., 24, 1104-1112, doi:10.1177/0956797612466414) observed a similar inhibitory effect for an attended item in working memory, which the authors referred to as an inhibition-of-return-like effect. However, testing an inhibition of return effect requires attention to be disengaged from the attended item, before testing whether participants are slower to return to said item. This was assumed but not experimentally manipulated in the paradigm by Johnson and colleagues. In the current study, we investigated whether an inhibition of return effect can be observed in working memory when attention is experimentally disengaged from the attended item before measuring whether responses are slower for the item in question. Participants were indeed slower to respond to a memory probe that matched the item that was attended right before, compared to a memory probe that matched the item that was not attended right before. Thus, our test with more experimental control did result in an inhibition of return effect in working memory.

Task Termination Triggers Spontaneous Removal of Information From Visual Working Memory

Working memory (WM) is a goal-directed memory system that actively maintains a limited amount of task-relevant information to serve the current goal. By this definition, WM maintenance should be terminated after the goal is accomplished, spontaneously removing no-longer-relevant information from WM. Past studies have failed to provide direct evidence of spontaneous removal of WM content by allowing participants to engage in a strategic reallocation of WM resources to competing information within WM. By contrast, we provide direct neural and behavioral evidence that visual WM content can be largely removed less than 1 s after it becomes obsolete, in the absence of a strategic allocation of resources (total N = 442 adults). These results demonstrate that visual WM is intrinsically a goal-directed system, and spontaneous removal provides a means for capacity-limited WM to keep up with ever-changing demands in a dynamic environment.

Sustained attention operates via dissociable neural mechanisms across different eccentric locations

In primates, foveal and peripheral vision have distinct neural architectures and functions. However, it has been debated if selective attention operates via the same or different neural mechanisms across eccentricities. We tested these alternative accounts by examining the effects of selective attention on the steady-state visually evoked potential (SSVEP) and the fronto-parietal signal measured via EEG from human subjects performing a sustained visuospatial attention task. With a negligible level of eye movements, both SSVEP and SND exhibited the heterogeneous patterns of attentional modulations across eccentricities. Specifically, the attentional modulations of these signals peaked at the parafoveal locations and such modulations wore off as visual stimuli appeared closer to the fovea or further away towards the periphery. However, with a relatively higher level of eye movements, the heterogeneous patterns of attentional modulations of these neural signals were less robust. These data demonstrate that the top-down influence of covert visuospatial attention on early sensory processing in human cortex depends on eccentricity and the level of saccadic responses. Taken together, the results suggest that sustained visuospatial attention operates differently across different eccentric locations, providing new understanding of how attention augments sensory representations regardless of where the attended stimulus appears.

Systematic differences in visual working memory performance are not caused by differences in working memory storage

People vary in their performance on visual working memory tasks, and these individual differences covary with a wide range of higher-level cognitive processes including fluid intelligence. Performance also varies across study displays, purportedly driven by both low- and higher-level processes. Understanding what causes these sources of systematic variability has been crucial for developing theories of working memory. However, here we find that all such variability in performance on a test of visual working memory can be accounted for by concurrent variability in visual iconic memory: A person with relatively high working memory capacity will have high iconic memory capacity, and a particularly easy working memory display will also be easy under iconic memory conditions. These results are supported by a nonparametric factor analysis and hierarchical Bayesian model comparison. In a second experiment the relationship between iconic and working memory holds even when they are measured with substantially different experimental paradigms, and a third experiment suggests that the relationship between tests of iconic and working memory is driven by mechanisms other than iconic or working memory storage, such as variation in perceptual or attentional processes. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Where and when matter in visual recognition

Our perceptual system processes only a selected subset of an incoming stream of stimuli due to sensory biases and limitations in spatial and temporal attention and working memory capacity. In this study, we investigated perceptual access to sensory information that was temporally predictable or unpredictable and spread across the visual field. In a visual recognition task, participants were presented with an array of different number of alphabetical stimuli that were followed by a probe with a delay. They had to indicate whether the probe was included in the stimulus-set or not. To test the impact of temporal attention, coloured cues that were displayed before the visual stimuli indicated the presentation onset of the stimulus-set. We found that temporal predictability of stimulus onset yields higher performance. In addition, recognition performance was biased across the visual field with higher performance for stimuli that were presented on the upper and right visual quadrants. Our findings demonstrate that recognition accuracy is enhanced by temporal cues and has an inherently asymmetric shape across the visual field.

What is attention? Adverbialist theories

This article presents theories of attention that attempt to derive their answer to the question of what attention is from their answers to the question of what it is for some activity to be done attentively. Such theories provide a distinctive account of the difficulties that are faced by the attempt to locate processes in the brain by which the phenomena of attention can be explained. Their account does not share the pessimism of theories suggesting that the concept of attention is defective. Instead it reconstrues the explanatory relationship between attention and the processes that constitute it, in a way that is illustrated here by considering the relationship between attention and the processes that are identified by the biased competition theory. After considering some of the ways in which an adverbialist approach might be developed, the article concludes by suggesting some possible solutions to a problem concerning distraction, by which prominent adverbialist theories of attention have been dogged. This article is categorized under: Psychology > Attention Philosophy > Metaphysics Philosophy > Foundations of Cognitive Science.

Sensory Delay Activity: More than an Electrophysiological Index of Working Memory Load

Sustained contralateral delay activity emerges in the retention period of working memory (WM) tasks and has been commonly interpreted as an electrophysiological index of the number of items held in a discrete-capacity WM resource. More recent findings indicate that these visual and tactile components are sensitive to various cognitive operations beyond the storage of discrete items in WM. In this Perspective, we present recent evidence from unisensory and multisensory visual and tactile WM tasks suggesting that, in addition to memory load, sensory delay activity may also be indicative of attentional and executive processes, as well as reflecting the flexible, rather than discrete, allocation of a continuous WM resource. Together, these findings challenge the traditional model of the functional significance of the contralateral delay activity as a pure measure of item load, and suggest that it may also reflect executive, attentional, and perceptual mechanisms operating in hierarchically organized WM systems.

Storage in Visual Working Memory Recruits a Content-Independent Pointer System

Past work has shown that storage in working memory elicits stimulus-specific neural activity that tracks the stored content. Here, we present evidence for a distinct class of load-sensitive neural activity that indexes items without representing their contents per se. We recorded electroencephalogram (EEG) activity while adult human subjects stored varying numbers of items in visual working memory. Multivariate analysis of the scalp topography of EEG voltage enabled precise tracking of the number of individuated items stored and robustly predicted individual differences in working memory capacity. Critically, this signature of working memory load generalized across variations in both the type and number of visual features stored about each item, suggesting that it tracked the number of individuated memory representations and not the content of those memories. We hypothesize that these findings reflect the operation of a capacity-limited pointer system that supports on-line storage and attentive tracking.

Are all real-world objects created equal? Estimating the "set-size" of the search target in visual working memory

Are all real-world objects created equal? Visual search difficulty increases with the number of targets and as target-related visual working memory (VWM) load increases. Our goal was to investigate the load imposed by individual real-world objects held in VWM in the context of search. Measures of visual clutter attempt to quantify real-world set-size in the context of scenes. We applied one of these measures, the number of proto-objects, to individual real-world objects and used contralateral delay activity (CDA) to measure the resulting VWM load. The current study presented a real-world object as a target cue, followed by a delay where CDA was measured. This was followed by a four-object search array. We compared CDA and later search performance from target cues containing a high or low number of proto-objects. High proto-object target cues resulted in greater CDA, longer search RTs, target dwell times, and reduced search guidance, relative to low proto-object targets. These findings demonstrate that targets with more proto-objects result in a higher VWM load and reduced search performance. This shows that the number of proto-objects contained within individual objects produce set-size like effects in VWM and suggests proto-objects may be a viable unit of measure of real-world VWM load. Importantly, this demonstrates that not all real-world objects are created equal.

Memory and Proactive Interference for spatially distributed items

Our ability to briefly retain information is often limited. Proactive Interference (PI) might contribute to these limitations (e.g., when items in recognition tests are difficult to reject after having appeared recently). In visual Working Memory (WM), spatial information might protect WM against PI, especially if encoding items together with their spatial locations makes item-location combinations less confusable than simple items without a spatial component. Here, I ask (1) if PI is observed for spatially distributed items, (2) if it arises among simple items or among item-location combinations, and (3) if spatial information affects PI at all. I show that, contrary to views that spatial information protects against PI, PI is reliably observed for spatially distributed items except when it is weak. PI mostly reflects items that appear recently or frequently as memory items, while occurrences as test items play a smaller role, presumably because their temporal context is easier to encode. Through mathematical modeling, I then show that interference occurs among simple items rather than item-location combinations. Finally, to understand the effects of spatial information, I separate the effects of (a) the presence and (b) the predictiveness of spatial information on memory and its susceptibility to PI. Memory is impaired when items are spatially distributed, but, depending on the analysis, unaffected by the predictiveness of spatial information. In contrast, the susceptibility to PI is unaffected by either manipulation. Visual memory is thus impaired by PI for spatially distributed items due to interference from recent memory items (rather than test items or item-location combinations).

Inter-electrode correlations measured with EEG predict individual differences in cognitive ability

Human brains share a broadly similar functional organization with consequential individual variation. This duality in brain function has primarily been observed when using techniques that consider the spatial organization of the brain, such as MRI. Here, we ask whether these common and unique signals of cognition are also present in temporally sensitive but spatially insensitive neural signals. To address this question, we compiled electroencephalogram (EEG) data from individuals of both sexes while they performed multiple working memory tasks at two different data-collection sites (n = 171 and 165). Results revealed that trial-averaged EEG activity exhibited inter-electrode correlations that were stable within individuals and unique across individuals. Furthermore, models based on these inter-electrode correlations generalized across datasets to predict participants' working memory capacity and general fluid intelligence. Thus, inter-electrode correlation patterns measured with EEG provide a signature of working memory and fluid intelligence in humans and a new framework for characterizing individual differences in cognitive abilities.

A relational account of visual short-term memory (VSTM)

Visual short-term memory (VSTM) is an important resource that allows temporarily storing visual information. Current theories posit that elementary features (e.g., red, green) are encoded and stored independently of each other in VSTM. However, they have difficulty explaining the similarity effect, that similar items can be remembered better than dissimilar items. In Experiment 1, we tested (N = 20) whether the similarity effect may be due to storing items in a context-dependent manner in VSTM (e.g., as the reddest/yellowest item). In line with a relational account of VSTM, we found that the similarity effect is not due to feature similarity, but to an enhanced sensitivity for detecting changes when the relative colour of a to-be-memorised item changes (e.g., from reddest to not-reddest item; than when an item underwent the same change but retained its relative colour; e.g., still reddest). Experiment 2 (N = 20) showed that VSTM load, as indexed by the CDA amplitude in the EEG, was smaller when the colours were ordered so that they all had the same relationship than when the same colours were out-of-order, requiring encoding different relative colours. With this, we report two new effects in VSTM - a relational detection advantage that describes an enhanced sensitivity to relative changes in change detection, and a relational CDA effect, which reflects that VSTM load, as indexed by the CDA, scales with the number of (different) relative features between the memory items. These findings support a relational account of VSTM and question the view that VSTM stores features such as colours independently of each other.

From "satisfaction of search" to "subsequent search misses": a review of multiple-target search errors across radiology and cognitive science

For over 50 years, the satisfaction of search effect has been studied within the field of radiology. Defined as a decrease in detection rates for a subsequent target when an initial target is found within the image, these multiple target errors are known to underlie errors of omission (e.g., a radiologist is more likely to miss an abnormality if another abnormality is identified). More recently, they have also been found to underlie lab-based search errors in cognitive science experiments (e.g., an observer is more likely to miss a target 'T' if a different target 'T' was detected). This phenomenon was renamed the subsequent search miss (SSM) effect in cognitive science. Here we review the SSM literature in both radiology and cognitive science and discuss: (1) the current SSM theories (i.e., satisfaction, perceptual set, and resource depletion theories), (2) the eye movement errors that underlie the SSM effect, (3) the existing efforts tested to alleviate SSM errors, and (4) the evolution of methodologies and analyses used when calculating the SSM effect. Finally, we present the attentional template theory, a novel mechanistic explanation for SSM errors, which ties together our current understanding of SSM errors and the attentional template literature.

Frontoparietal theta tACS nonselectively enhances encoding, maintenance, and retrieval stages in visuospatial working memory

Neurobiological and cognitive evidence suggests that working memory is processed through three distinctive and well-characterized phases: encoding, maintenance, and retrieval. Several studies have reported that applying theta transcranial alternating current stimulation (tACS) to the right prefrontal and parietal cortices can significantly improve visual working memory performance. However, it remains unclear whether the facilitative effect of tACS on visual working memory is due to a domain-general or stage-specific process. In this study, we combined pre-task right frontoparietal theta tACS (6 Hz, 15 min) with a stage-specific change detection paradigm that provided retro-cues during various stages of working memory. This stage-specific tagging via the use of retro-cues enabled us to probe whether theta tACS would create a nonspecific/additive effect that is equal in magnitude across all cognitive stages or would create a stage-specific effect that is interactive with the retro-cue in a particular stage (e.g., maintenance, retrieval). We observed significant retro-cue and theta tACS effects on visual working memory performance, but no interaction between them. This finding suggests that the aforementioned two factors can facilitate visual working memory processing independently in an additive manner. Furthermore, low-performers benefited more from tACS, and their VWM deficit seemed to have originated from the second half of the memory retention stage, which possibly suggests faster memory decay as the key to poor VWM performance. Together, we conclude that frontoparietal theta tACS likely creates a domain-general boost in visual attention, which in turn benefits overall visual working memory processes that are not specific to the information maintenance or retrieval stages.

Greater Visual Working Memory Capacity for Visually Matched Stimuli When They Are Perceived as Meaningful

Almost all models of visual working memory-the cognitive system that holds visual information in an active state-assume it has a fixed capacity: Some models propose a limit of three to four objects, where others propose there is a fixed pool of resources for each basic visual feature. Recent findings, however, suggest that memory performance is improved for real-world objects. What supports these increases in capacity? Here, we test whether the meaningfulness of a stimulus alone influences working memory capacity while controlling for visual complexity and directly assessing the active component of working memory using EEG. Participants remembered ambiguous stimuli that could either be perceived as a face or as meaningless shapes. Participants had higher performance and increased neural delay activity when the memory display consisted of more meaningful stimuli. Critically, by asking participants whether they perceived the stimuli as a face or not, we also show that these increases in visual working memory capacity and recruitment of additional neural resources are because of the subjective perception of the stimulus and thus cannot be driven by physical properties of the stimulus. Broadly, this suggests that the capacity for active storage in visual working memory is not fixed but that more meaningful stimuli recruit additional working memory resources, allowing them to be better remembered.

Visual and verbal working memory loads interfere with scene-viewing

Working memory is thought to be divided into distinct visual and verbal subsystems. Studies of visual working memory frequently use verbal working memory tasks as control conditions and/or use articulatory suppression to ensure that visual load is not transferred to verbal working memory. Using these verbal tasks relies on the assumption that the verbal working memory load will not interfere with the same processes as visual working memory. In the present study, participants maintained a visual or verbal working memory load as they simultaneously viewed scenes while their eye movements were recorded. Because eye movements and visual working memory are closely linked, we anticipated the visual load would interfere with scene-viewing (and vice versa), while the verbal load would not. Surprisingly, both visual and verbal memory loads interfered with scene-viewing behavior, while eye movements during scene-viewing did not significantly interfere with performance on either memory task. These results suggest that a verbal working memory load can interfere with eye movements in a visual task.

No Evidence for an Object Working Memory Capacity Benefit with Extended Viewing Time

Visual working memory is the ability to hold visual information temporarily in mind. A key feature of working memory is its starkly limited capacity, such that only a few simple items can be remembered at once. Prior work has shown that this capacity limit cannot be circumvented by providing additional encoding time, whether providing just 200 ms or up to 1300 ms, capacity is still limited to only three to four items. In contrast, Brady et al. (2016) hypothesized that real-world objects, but not simple items used in prior research, benefit from additional encoding time and are not subject to traditional capacity limits. They supported this hypothesis with results from both behavior and the contralateral delay activity (CDA), an EEG marker of working memory storage, and concluded that familiar, complex stimuli are necessary to observe encoding time effects. Here, we conducted three replications of Brady et al.'s key manipulation with a larger number of human participants and more trials per condition. We failed to replicate their primary behavioral result (objects benefit more than colors from additional encoding time) and failed to observe an object-specific increase in the CDA. Instead, we found that encoding time benefitted both simple color items and real-world objects, in contrast to both the findings by Brady et al., and some prior work on this topic. Overall, we observed no support for the hypothesis that real-world objects have a different capacity than colored squares. We discuss the implications of our findings for theories of visual working memory (VWM).

Beyond action observation: Neurobehavioral mechanisms of memory for visually perceived bodies and actions

Examining the processing of others' body-related information in the perceivers' brain (action observation) is a key topic in cognitive neuroscience. However, what happens beyond the perceptual stage, when the body is not within view and it is transformed into an associative form that can be stored, updated, and later recalled, remains poorly understood. Here we examine neurobehavioural evidence on the memory processing of visually perceived bodily stimuli (dynamic actions and images of bodies). The reviewed studies indicate that encoding and maintaining bodily stimuli in memory recruits the sensorimotor system. This process arises when bodily stimuli are either recalled through action recognition or reproduction. Interestingly, the memory capacity for these stimuli is rather limited: only 2 or 3 bodily stimuli can be simultaneously held in memory. Moreover, this process is disrupted by increasing concurrent bodily operations; i.e., moving one's body, seeing or memorising additional bodies. Overall, the evidence suggests that the neural circuitry allowing us to move and feel ourselves supports the encoding, retention, and memory recall of others' visually perceived bodies.

Limitations of concurrently representing objects within view and in visual working memory

Representing visibly present stimuli is as limited in capacity as representing invisible stimuli in visual working memory (WM). In this study, we explored whether concurrently representing stimuli within view affects representing objects in visual WM, and if so, whether this effect is modulated by the storage states (active and silent state) of memory contents? In experiment 1, participants were asked to perform the change-detect task in a simultaneous-representing condition in which WM content and the continuously-visible stimuli in view were simultaneously represented, as well as a baseline condition in which only the representations of visual WM content were maintained. The results showed that the representations in visual WM would be impaired when the continuously-visible stimuli in view were concurrently represented, revealed by the reduced CDA amplitude and the lower behavior performance. In experiment 2, a dual-serial retro-cue paradigm was adopted to guide participants to maintain memory items in two different storage states, and the results revealed that simultaneously representing the continuously-visible stimuli and the WM content would only impair the WM representations in the active state. These evidences demonstrated that only the visual WM representations that were maintained in the active state would definitely share the limited resources with the representations of continuously-visible information, and further supported the dissociation between the active state and silent state of visual WM storage.

Electrophysiological evidence supports the role of sustained visuospatial attention in maintaining visual WM contents

Recent empirical and theoretical work suggests that there is a close relationship between visual working memory (WM) and visuospatial attention. Here, we investigated whether visuospatial attention was involved in maintaining object representations in visual WM. To this end, the alpha lateralization and contralateral delay activity (CDA) were analyzed as neural markers for visuospatial attention and visual WM storage, respectively. In the single-task condition, participants performed a grating change-detection task. To probe the role of visuospatial attention in maintaining WM contents, two color squares were presented above and below the fixation point during the retention interval, which remained visible until the detection display was present. In the dual-task condition, participants were required to maintain lateralized gratings while staring at the center-presented color squares, to detect possible subsequent color change. With this task, sustained visuospatial attention that guided to individual memory representations was disrupted. The behavioral data showed that, the insertion of secondary task significantly deteriorated WM performance. For electrophysiological data, we divided the retention interval into two stages, the early stage and late stage, bounded by the onset of the secondary task. We found that CDA amplitude was lower under the dual-task condition than the single-task condition during the late stage, but not the early stage, and the extent to which CDA reduced tracked the impaired memory performance at the individual level. Also, alpha lateralization only could be observed in the single-task condition of the late stage, and completely disappeared in the dual-task condition, indicating the disruption of visuospatial attention directed to memory representations. Individuals who experienced greater visuospatial attention disruption, as indicated by the alpha lateralization, had lower maintenance-associated neural activity (CDA), and suffered greater impairment of memory performance. These findings confirm that sustained visuospatial attention continues improving visual WM processing after the initial encoding phase, and most likely participates in this process by supporting the maintenance of representations in an active state.

Working Memory and Attention - A Conceptual Analysis and Review

There is broad agreement that working memory is closely related to attention. This article delineates several theoretical options for conceptualizing this link, and evaluates their viability in light of their theoretical implications and the empirical support they received. A first divide exists between the concept of attention as a limited resource, and the concept of attention as selective information processing. Theories conceptualizing attention as a resource assume that this resource is responsible for the limited capacity of working memory. Three versions of this idea have been proposed: Attention as a resource for storage and processing, a shared resource for perceptual attention and memory maintenance, and a resource for the control of attention. The first of these three is empirically well supported, but the other two are not. By contrast, when attention is understood as a selection mechanism, it is usually not invoked to explain the capacity limit of working memory - rather, researchers ask how different forms of attention interact with working memory, in two areas. The first pertains to attentional selection of the contents of working memory, controlled by mechanisms of filtering out irrelevant stimuli, and removing no-longer relevant representations from working memory. Within working memory contents, a single item is often selected into the focus of attention for processing. The second area pertains to the role of working memory in cognitive control. Working memory contributes to controlling perceptual attention - by holding templates for targets of perceptual selection - and controlling action - by holding task sets to implement our current goals.

Functional MRI and EEG Index Complementary Attentional Modulations

Functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) are two noninvasive methods commonly used to study neural mechanisms supporting visual attention in humans. Studies using these tools, which have complementary spatial and temporal resolutions, implicitly assume they index similar underlying neural modulations related to external stimulus and internal attentional manipulations. Accordingly, they are often used interchangeably for constraining understanding about the impact of bottom-up and top-down factors on neural modulations. To test this core assumption, we simultaneously manipulated bottom-up sensory inputs by varying stimulus contrast and top-down cognitive modulations by changing the focus of spatial attention. Each of the male and female subjects participated in both fMRI and EEG sessions performing the same experimental paradigm. We found categorically different patterns of attentional modulation on fMRI activity in early visual cortex and early stimulus-evoked potentials measured via EEG (e.g., the P1 component and steady-state visually-evoked potentials): fMRI activation scaled additively with attention, whereas evoked EEG components scaled multiplicatively with attention. However, across longer time scales, a contralateral negative-going potential and oscillatory EEG signals in the alpha band revealed additive attentional modulation patterns like those observed with fMRI. These results challenge prior assumptions that fMRI and early stimulus-evoked potentials measured with EEG can be interchangeably used to index the same neural mechanisms of attentional modulations at different spatiotemporal scales. Instead, fMRI measures of attentional modulations are more closely linked with later EEG components and alpha-band oscillations. Considered together, hemodynamic and electrophysiological signals can jointly constrain understanding of the neural mechanisms supporting cognition.SIGNIFICANCE STATEMENT fMRI and EEG have been used as tools to measure the location and timing of attentional modulations in visual cortex and are often used interchangeably for constraining computational models under the assumption that they index similar underlying neural processes. However, by varying attentional and stimulus parameters, we found differential patterns of attentional modulations of fMRI activity in early visual cortex and commonly used stimulus-evoked potentials measured via EEG. Instead, across longer time scales, a contralateral negative-going potential and EEG oscillations in the alpha band exhibited attentional modulations similar to those observed with fMRI. Together, these results suggest that different physiological processes assayed by these complementary techniques must be jointly considered when making inferences about the neural underpinnings of cognitive operations.

Effects of Visual Working Memory on Individual Differences in Echolocation Performance in Sighted Participants

Echolocation performance differs widely among individuals. This study examined a possible factor that may explain this variation, namely, visual working memory, which is a subcomponent of spatial working memory. Sighted participants performed an object-detection task consisting of initial testing on 2 separate days (up to 8 days apart) with follow-up testing on a third day (up to 1 month after the second day of testing) while manipulating the target distance from 20 to 50 cm. Participants performed two types of visual spatial working memory tasks, one of which required them to memorize color-location combinations and the other, an imaginary pathway. The participants' performance on the object-detection task generally improved in the first 2 days, but there were substantial individual differences in detection ability. A positive correlation was observed between performance on these tasks and visual working memory capacity, except on the second day, after detection ability had improved. These findings suggest that factors contributing to echolocation skill are related to nonauditory factors in a sighted group.

Reassessing the Evidence for Capacity Limits in Neural Signals Related to Working Memory

In 2004, two landmark studies described the discovery of brain imaging (functional magnetic resonance imaging and electroencephalography) signals that increase with the number of items held in visual working memory (WM). These studies claimed that the signals leveled off (plateaued) once the number of memoranda reached the capacity of WM, as estimated by the prevailing model of the time. However, alternative models were not considered, and changing concepts of WM in the more than a decade since these studies were published necessitate a re-evaluation of their findings; newer models that provide the most accurate account of behavioral data do not incorporate a fixed limit on the number of items stored. Furthermore, an important claim made about the original studies, that signals plateau at each individual's estimated capacity, has never been tested. Here, we pit the plateau model of signal strength against an alternative, saturation model, a biophysically plausible account in which signals increase continuously without plateau. We show that the saturation model provides a better description of the original data, challenging the assumption that imaging results provide evidence for a fixed item limit in WM.

Neural evidence for an object-based pointer system underlying working memory

To accomplish even rudimentary tasks, our cognitive system must update its representation of the changing environment. This process relies on visual working memory (VWM), which can actively modify its representations. We argue that this ability depends on a pointer system, such that each representation is stably and uniquely mapped to a specific stimulus. Without these pointers, VWM representations are inaccessible and therefore unusable. In three Electroencephalogram (EEG) experiments, we examined whether the pointers are allocated in an object-based, featural, or spatial manner: three factors that were confounded in previous studies. We used a feature change-detection task, in which objects moved and could separate into independently-moving parts. Despite the movement and separation being completely task-irrelevant, we found that the separation invalidated the pointers. This happened in a shape task, where the separation changed both the objects and the task-relevant features, but importantly, also in a color task, where the separation destroyed the objects while leaving the task-relevant features intact. Furthermore, even in a color task where all items had identical shapes, object-separation invalidated the pointers. This suggests that objects and not task-relevant features underlie the pointer system. Finally, when each object-part could be individuated already before the separation, the pointers were maintained, suggesting that the pointers are specifically tied to objects rather than locations. These results shed new light on the pointers which underlie VWM performance, demonstrating that the pointer system is object-based regardless of the task requirements.

Multi-target attention and visual short-term memory capacity are closely linked in the intraparietal sulcus

The existing literature suggests a critical role for both the right intraparietal sulcus (IPS) and the right temporo‐parietal junction (TPJ) in our ability to attend to multiple simultaneously‐presented lateralized targets (multi‐target attention), and the failure of this ability in extinction patients. Currently, however, the precise role of each of these areas in multi‐target attention is unclear. In this study, we combined the theory of visual attention (TVA) with functional magnetic resonance imaging (fMRI) guided continuous theta burst stimulation (cTBS) in neurologically healthy subjects to directly investigate the role of the right IPS and TPJ in multi‐target attention. Our results show that cTBS at an area of the right IPS associated with multi‐target attention elicits a reduction of visual short‐term memory capacity. This suggests that the right IPS is associated with a general capacity‐limited encoding mechanism that is engaged regardless of whether targets have to be attended or remembered. Curiously, however, cTBS to the right IPS failed to elicit extinction‐like behavior in our study, supporting previous suggestions that different areas of the right IPS may provide different contributions to multi‐target attention. CTBS to the right TPJ failed to induce a change in either TVA parameters or extinction‐like behavior.

Current directions in visual working memory research: An introduction and emerging insights

Visual working memory (VWM) is a core construct in the cognitive (neuro-)sciences, assumed to serve as a hub for information exchange and thus supporting a multitude of cognitive functions related to processing visual information. Here, we give an introduction into key terms and paradigms and an overview of ongoing debates in the field, to which the articles collected in this Special Issue on 'Current Directions in Visual Working Memory Research' contribute. Our aim is to extract, from this overview, some 'emerging' theoretical insights concerning questions such as the optimal way to examine VWM, which types of mental representations contribute to performance on VWM tasks, and how VWM keeps features from the same object together and apart from features of concurrently maintained objects (the binding problem).

Retroactive Attentional Shifts Predict Performance in a Working Memory Task: Evidence by Lateralized EEG Patterns

Shifts of attention within working memory based on retroactive (retro-) cues were shown to facilitate performance in working memory tasks. Although posterior asymmetries in the EEG, such as the contralateral delay activity (CDA), have been used to study the active storage of lateralized working memory representations, results on the relation of such asymmetric effects to retro-cue benefits remain inconclusive. We recorded EEG in a retro-cue working memory task with lateralized items and a continuous performance response. Following either a selective or neutral retro-cue, participants adjusted the orientation of a central memory probe to the cued item. Selective retro-cues elicited an early posterior contralateral negativity (PCN), anterior directing attention negativity (ADAN) and a later modulation of CDA indicating that active storage was concentrated on the cued information. By dividing all trials into three within-condition performance quantiles, we could further show that high working memory accuracy was associated with a sustained increase of the CDA effect following the retro-cue. These results suggest that focusing resources on the active storage of relevant representations is an important factor regarding retro-cue benefits in working memory tasks.

Having More Choices Changes How Human Observers Weight Stable Sensory Evidence

Decision-making becomes slower when more choices are available. Existing models attribute this slowing to poor sensory processing, to attenuated rates of sensory evidence accumulation, or to increases in the amount of evidence required before committing to a decision (a higher decision threshold). However, studies have not isolated the effects of having more choices on sensory and decision-related processes from changes in task difficulty and divided attention. Here, we controlled task difficulty while independently manipulating the distribution of attention and the number of choices available to male and female human observers. We used EEG to measure steady-state visually evoked potentials (SSVEPs) and a frontal late positive deflection (LPD), EEG markers of sensory and postsensory decision-related processes, respectively. We found that dividing attention decreased SSVEP and LPD amplitudes, consistent with dampened sensory responses and slower rates of evidence accumulation, respectively. In contrast, having more choices did not alter SSVEP amplitude and led to a larger LPD. These results suggest that having more options largely spares early sensory processing and slows down decision-making via a selective increase in decision thresholds.SIGNIFICANCE STATEMENT When more choices are available, decision-making becomes slower. We tested whether this phenomenon is due to poor sensory processing, to reduced rates of evidence accumulation, or to increases in the amount of evidence required before committing to a decision (a higher decision threshold). We measured choice modulations of sensory and decision-related neural responses using EEG. We also minimized potential confounds from changes in the distribution of attention and task difficulty, which often covary with having more choices. Dividing attention reduced the activity levels of both sensory and decision-related responses. However, having more choices did not change sensory processing and led to larger decision-related responses. These results suggest that having more choices spares sensory processing and selectively increases decision thresholds.

Iconic Memories Die a Sudden Death

Iconic memory is characterized by its large storage capacity and brief storage duration, whereas visual working memory is characterized by its small storage capacity. The limited information stored in working memory is often modeled as an all-or-none process in which studied information is either successfully stored or lost completely. This view raises a simple question: If almost all viewed information is stored in iconic memory, yet one second later most of it is completely absent from working memory, what happened to it? Here, I characterized how the precision and capacity of iconic memory changed over time and observed a clear dissociation: Iconic memory suffered from a complete loss of visual items, while the precision of items retained in memory was only marginally affected by the passage of time. These results provide new evidence for the discrete-capacity view of working memory and a new characterization of iconic memory decay.

An unpleasant emotional state reduces working memory capacity: electrophysiological evidence

Emotional states can guide the actions and decisions we make in our everyday life through their influence on cognitive processes such as working memory (WM). We investigated the long-lasting interference that an unpleasant emotional state had on goal-relevant WM representations from an electrophysiological perspective. Participants performed a change detection task that was preceded by the presentation of unpleasant or neutral task-irrelevant pictures in a blocked fashion. We focused on the contralateral delay activity (CDA), an event-related potential that is sensitive to the number of task-relevant items stored in WM. We found that the asymptotic limit for the CDA amplitude was lower during the unpleasant emotional state than during the neutral one; that is, an emotional state was capable of reducing how many task-relevant items the participants could hold in WM. Furthermore, both the individuals who experienced more intrusive thoughts and those who were dispositionally anxious were more susceptible to the influence of the emotional state. We provide evidence that an unpleasant emotional state diminished visual WM for task-relevant items, particularly in susceptible individuals. These results open new avenues to uncover the emotional-cognitive processing that underlies maladaptive WM representations and the role of such processing in the development of mental illness.

Delineating resetting and updating in visual working memory based on the object-to-representation correspondence

When an object we represent in visual working memory (VWM) changes, its representation is modified accordingly. VWM can either access and change the existing representation by an updating process, or it can reset, by encoding the object in its novel status as a new representation. Our goal was to show that the determining factor of updating versus resetting is the availability of a stable correspondence between the object and its VWM representation. Here, we demonstrate that updating relies on the object-to-representation mapping to access and modify the appropriate representation, while losing this mapping triggers a resetting process. We compared very similar situations of object separation that either allowed the mapping to hold, or caused it to be lost. When an object that was mapped to one representation separated, VWM reset, manifested by a sharp drop in the contralateral delay activity (CDA) amplitude (an electrophysiological marker of VWM contents; Experiment 1), and a behavioral cost to detect salient changes that co-occurred with the resetting-triggering event (Experiment 2). When each part was mapped to a different representation, the separation resulted in updating, with a gradual rise in CDA amplitude (Experiment 1), and a reduced behavioral cost (Experiment 2). Thus, while updating and resetting resulted in similar final representations (corresponding to the post-change objects), their dynamics were different, depending on the availability of the mapping. Our results reveal the triggering conditions of resetting and updating, establish methods to study these online processes, and highlight the importance of the object-to-representation correspondence in VWM.

Working Memory and Consciousness: The Current State of Play

Working memory (WM), an important posit in cognitive science, allows one to temporarily store and manipulate information in the service of ongoing tasks. WM has been traditionally classified as an explicit memory system-that is, as operating on and maintaining only consciously perceived information. Recently, however, several studies have questioned this assumption, purporting to provide evidence for unconscious WM. In this article, we focus on visual working memory (VWM) and critically examine these studies as well as studies of unconscious perception that seem to provide indirect evidence for unconscious WM. Our analysis indicates that current evidence does not support an unconscious WM store, though we offer independent reasons to think that WM may operate on unconsciously perceived information.

Visual working memory can selectively reset a subset of its representations

The visual working memory (VWM) resetting process is triggered when the mapping between an object in the environment and its corresponding VWM representation becomes irrelevant. Resetting involves discarding the no longer relevant representations, and encoding novel representations and mappings. We examined how resetting operates on VWM's contents. Specifically, we tested whether losing only part of the encoded mappings led to resetting all of the VWM representations. Subjects monitored moving polygons for an abrupt shape-change. Occasionally, a polygon separated into two halves that continued to move independently, making the original single mapping irrelevant. This loss of mapping triggered a resetting process, producing a performance cost: subjects missed shape-changes when they occurred during resetting, but not when the changes occurred before or after resetting. Critically, the cost was (1) specific to the separated item, (2) larger when more mappings were lost, and (3) unaffected by the set-size. This suggests that resetting is a "local" process: VWM removes only the representations whose mappings are lost.

Restoring Latent Visual Working Memory Representations in Human Cortex

Working memory (WM) enables the storage and manipulation of limited amounts of information over short periods. Prominent models posit that increasing the number of remembered items decreases the spiking activity dedicated to each item via mutual inhibition, which irreparably degrades the fidelity of each item's representation. We tested these models by determining if degraded memory representations could be recovered following a post-cue indicating which of several items in spatial WM would be recalled. Using an fMRI-based image reconstruction technique, we identified impaired behavioral performance and degraded mnemonic representations with elevated memory load. However, in several cortical regions, degraded mnemonic representations recovered substantially following a post-cue, and this recovery tracked behavioral performance. These results challenge pure spike-based models of WM and suggest that remembered items are additionally encoded within latent or hidden neural codes that can help reinvigorate active WM representations.

Elucidating and Modulating the Neural Correlates of Visuospatial Working Memory via Noninvasive Brain Stimulation

Visuospatial working memory refers to the short-term memory mechanism that enables humans to remember visual information across visual blackout periods such as eyeblinks or eye movements. In recent years, neuroscientific studies have made great progress in uncovering the brain regions that support visuospatial working memory. In this review, we focus on the role of the posterior parietal cortex in forming and maintaining visual information, and use it as an example to highlight how noninvasive brain-stimulation techniques, particularly transcranial magnetic, direct current, and alternating current stimulation, can shed light on this topic because of their unique strengths in modulating brain activities.

Neural and Behavioral Evidence for an Online Resetting Process in Visual Working Memory

Visual working memory (VWM) guides behavior by holding a set of active representations and modifying them according to changes in the environment. This updating process relies on a unique mapping between each VWM representation and an actual object in the environment. Here, we destroyed this mapping by either presenting a coherent object but then breaking it into independent parts or presenting an object but then abruptly replacing it with a different object. This allowed us to introduce the neural marker and behavioral consequence of an online resetting process in humans' VWM. Across seven experiments, we demonstrate that this resetting process involves abandoning the old VWM contents because they no longer correspond to the objects in the environment. Then, VWM encodes the novel information and reestablishes the correspondence between the new representations and the objects. The resetting process was marked by a unique neural signature: a sharp drop in the amplitude of the electrophysiological index of VWM contents (the contralateral delay activity), presumably indicating the loss of the existent object-to-representation mappings. This marker was missing when an updating process occurred. Moreover, when tracking moving items, VWM failed to detect salient changes in the object's shape when these changes occurred during the resetting process. This happened despite the object being fully visible, presumably because the mapping between the object and a VWM representation was lost. Importantly, we show that resetting, its neural marker, and the behavioral cost it entails, are specific to situations that involve a destruction of the objects-to-representations correspondence.

SIGNIFICANCE STATEMENT

Visual working memory (VWM) maintains task-relevant information in an online state. Previous studies showed that VWM representations are accessed and modified after changes in the environment. Here, we show that this updating process critically depends on an ongoing mapping between the representations and the objects in the environment. When this mapping breaks, VWM cannot access the old representations and instead resets. The novel resetting process that we introduce removes the existing representations instead of modifying them and this process is accompanied by a unique neural marker. During the resetting process, VWM was blind to salient changes in the object's shape. The resetting process highlights the flexibility of our cognitive system in handling the dynamic environment by abruptly abandoning irrelevant schemas.

Induced and Evoked Human Electrophysiological Correlates of Visual Working Memory Set-Size Effects at Encoding

The ability to encode, store, and retrieve visually presented objects is referred to as visual working memory (VWM). Although crucial for many cognitive processes, previous research reveals that VWM strictly capacity limited. This capacity limitation is behaviorally observable in the set size effect: the ability to successfully report items in VWM asymptotes at a small number of items. Research into the neural correlates of set size effects and VWM capacity limits in general largely focus on the maintenance period of VWM. However, we previously reported that neural resources allocated to individual items during VWM encoding correspond to successful VWM performance. Here we expand on those findings by investigating neural correlates of set size during VWM encoding. We hypothesized that neural signatures of encoding-related VWM capacity limitations should be differentiable as a function of set size. We tested our hypothesis using High Density Electroencephalography (HD-EEG) to analyze frequency components evoked by flickering target items in VWM displays of set size 2 or 4. We found that set size modulated the amplitude of the 1st and 2nd harmonic frequencies evoked during successful VWM encoding across frontal and occipital-parietal electrodes. Frontal sites exhibited the most robust effects for the 2nd harmonic (set size 2 > set size 4). Additionally, we found a set-size effect on the induced power of delta-band (1-4 Hz) activity (set size 2 > set size 4). These results are consistent with a capacity limited VWM resource at encoding that is distributed across to-be-remembered items in a VWM display. This resource may work in conjunction with a task-specific selection process that determines which items are to be encoded and which are to be ignored. These neural set size effects support the view that VWM capacity limitations begin with encoding related processes.

Comparing the Effects of 10-Hz Repetitive TMS on Tasks of Visual STM and Attention

Numerous studies have demonstrated that visual STM (VSTM) and attention are tightly linked processes that share a number of neuroanatomical substrates. Here, we used repetitive TMS (rTMS) along with simultaneous EEG to examine the causal relationship between intraparietal sulcus functioning and performance on tasks of attention and VSTM. Participants performed two tasks in which they were required to attend to or remember colored items over a brief interval, with 10-Hz rTMS applied on some of the trials. Although no overall behavioral changes were observed across either task, rTMS did affect individual performance on both the attention and VSTM tasks in a manner that was predicted by individual differences in baseline performance. Furthermore, rTMS also affected ongoing oscillations in the alpha and beta bands, and these changes were related to the observed change in behavioral performance. The results reveal a causal relationship between intraparietal sulcus activity and tasks measuring both visual attention and VSTM.