New perspectives on binding in visual working memory

Unraveling the binding problem in working memory: insights from the hierarchical binding model

The binding problem is a crucial issue in the study of working memory (WM) and remains a central topic of debate among various WM models. Over the past decade, we have explored feature binding within WM, guided by the Hierarchical Binding Model (HBM). This model suggests that WM binding occurs in two stages: an initial implicit binding involving rapid, coarse feature processing, followed by explicit binding where focused attention refines these features via a reentry process. We found that implicit binding is closely related to the attentional processing of features during the perceptual stage. Basic features that can be rapidly and coarsely processed in parallel through spread attention are involuntarily extracted into WM along with the target features, forming a rough bound representation. For explicit binding, we examined the role of attention in retaining explicit binding in WM, emphasizing the unique role of reentry in the HBM. Our findings indicate that WM binding requires additional object attention through the reentry process. These results demonstrate that both implicit and explicit bindings are integral to WM and that the HBM is effective in elucidating the binding mechanisms within WM.

Alpha phase-coding supports feature binding during working memory maintenance

The ability to successfully retain and manipulate information in working memory (WM) requires that objects' individual features are bound into cohesive representations; yet, the mechanisms supporting feature binding remain unclear. Binding (or swap) errors, where memorized features are erroneously associated with the wrong object, can provide a window into the intrinsic limits in capacity of WM that represent a key bottleneck in our cognitive ability. We tested the hypothesis that binding in WM is accomplished via neural phase synchrony and that swap errors result from perturbations in this synchrony. Using magnetoencephalography data collected from human subjects in a task designed to induce swap errors, we showed that swaps are characterized by reduced phase-locked oscillatory activity during memory retention, as predicted by an attractor model of spiking neural networks. Further, we found that this reduction arises from increased phase-coding variability in the alpha-band over a distributed network of sensorimotor areas. Our findings demonstrate that feature binding in WM is accomplished through phase-coding dynamics that emerge from the competition between different memories.

Maintenance of color memoranda in activity-quiescent working memory states: Evidence from impulse perturbation

In the present study, we used an impulse perturbation method to probe working memory maintenance of colors in neurally active and activity-quiescent states, focusing on a set of pre-registered analyses. We analyzed the electroencephalograph (EEG) data of 30 participants who completed a delayed match-to-sample working memory task, in which one of the two items that were presented was retro-cued as task relevant. The analyses revealed that both cued and uncued colors were decodable from impulse-evoked activity, the latter in contrast to previous reports of working memory for orientation gratings. Decoding of colors from oscillations in the alpha band showed that cued items could be decoded therein whereas uncued items could not. Overall, the outcomes suggest that subtle differences exist between the representation of colors, and that of stimuli with spatial properties, but the present results also demonstrate that regardless of their specific neural state, both are accessible through visual impulse perturbation.

Effective connectivity of working memory performance: a DCM study of MEG data

Visual working memory (WM) engages several nodes of a large-scale network that includes frontal, parietal, and visual regions; however, little is understood about how these regions interact to support WM behavior. In particular, it is unclear whether network dynamics during WM maintenance primarily represent feedforward or feedback connections. This question has important implications for current debates about the relative roles of frontoparietal and visual regions in WM maintenance. In the current study, we investigated the network activity supporting WM using MEG data acquired while healthy subjects performed a multi-item delayed estimation WM task. We used computational modeling of behavior to discriminate correct responses (high accuracy trials) from two different types of incorrect responses (low accuracy and swap trials), and dynamic causal modeling of MEG data to measure effective connectivity. We observed behaviorally dependent changes in effective connectivity in a brain network comprising frontoparietal and early visual areas. In comparison with high accuracy trials, frontoparietal and frontooccipital networks showed disrupted signals depending on type of behavioral error. Low accuracy trials showed disrupted feedback signals during early portions of WM maintenance and disrupted feedforward signals during later portions of maintenance delay, while swap errors showed disrupted feedback signals during the whole delay period. These results support a distributed model of WM that emphasizes the role of visual regions in WM storage and where changes in large scale network configurations can have important consequences for memory-guided behavior.

Electrophysiological hallmarks for event relations and event roles in working memory

The ability to maintain events (i.e., interactions between/among objects) in working memory is crucial for our everyday cognition, yet the format of this representation is poorly understood. The current ERP study was designed to answer two questions: How is maintaining events (e.g., the tiger hit the lion) neurally different from maintaining item coordinations (e.g., the tiger and the lion)? That is, how is the event relation (present in events but not coordinations) represented? And how is the agent, or initiator of the event encoded differently from the patient, or receiver of the event during maintenance? We used a novel picture-sentence match-across-delay approach in which the working memory representation was "pinged" during the delay, replicated across two ERP experiments with Chinese and English materials. We found that maintenance of events elicited a long-lasting late sustained difference in posterior-occipital electrodes relative to non-events. This effect resembled the negative slow wave reported in previous studies of working memory, suggesting that the maintenance of events in working memory may impose a higher cost compared to coordinations. Although we did not observe significant ERP differences associated with pinging the agent vs. the patient during the delay, we did find that the ping appeared to dampen the ongoing sustained difference, suggesting a shift from sustained activity to activity silent mechanisms. These results suggest a new method by which ERPs can be used to elucidate the format of neural representation for events in working memory.

No obligatory trade-off between the use of space and time for working memory

Space and time can each act as scaffolds for the individuation and selection of visual objects in working memory. Here we ask whether there is a trade-off between the use of space and time for visual working memory: whether observers will rely less on space, when memoranda can additionally be individuated through time. We tracked the use of space through directional biases in microsaccades after attention was directed to memory contents that had been encoded simultaneously or sequentially to the left and right of fixation. We found that spatial gaze biases were preserved when participants could (Experiment 1) and even when they had to (Experiment 2) additionally rely on time for object individuation. Thus, space remains a profound organizing medium for working memory even when other organizing sources are available and utilized, with no evidence for an obligatory trade-off between the use of space and time.

Sequential encoding aids working memory for meaningful objects' identities but not for their colors

Previous studies have found that real-world objects' identities are better remembered than simple features like colored circles, and this effect is particularly pronounced when these stimuli are encoded one by one in a serial, item-based way. Recent work has also demonstrated that memory for simple features like color is improved if these colors are part of real-world objects, suggesting that meaningful objects can serve as a robust memory scaffold for their associated low-level features. However, it is unclear whether the improved color memory that arises from the colors appearing on real-world objects is affected by encoding format, in particular whether items are encoded sequentially or simultaneously. We test this using randomly colored silhouettes of recognizable versus unrecognizable scrambled objects that offer a uniquely controlled set of stimuli to test color working memory of meaningful versus non-meaningful objects. Participants were presented with four stimuli (silhouettes of objects or scrambled shapes) simultaneously or sequentially. After a short delay, they reported either which colors or which shapes they saw in a two-alternative forced-choice task. We replicated previous findings that meaningful stimuli boost working memory performance for colors (Exp. 1). We found that when participants remembered the colors (Exp. 2) there was no difference in performance across the two encoding formats. However, when participants remembered the shapes and thus identity of the objects (Exp. 3), sequential presentation resulted in better performance than simultaneous presentation. Overall, these results show that different encoding formats can flexibly impact visual working memory depending on what the memory-relevant feature is.

Age-Specific Effects of Visual Feature Binding

Temporary binding of visual features enables objects to be stored and maintained in the visual working memory as a singular structure, irrespective of its inherent complexity. Although working memory capacity is reduced in aging, previous behavioral studies suggest that binding is preserved. Using event-related brain potentials (ERPs), we tested whether stimulus encoding is different in younger (N = 26, mean age = 28.5) and older (N = 22; mean age = 67.4) participants in a change detection task. The processing costs of binding were defined by the difference between feature-alone (color or shape) and feature-binding (color-shape) conditions. The behavioral data revealed that discrimination ability was reduced in the feature-binding condition, and that this effect was more attenuated in older participants. A corresponding ERP effect was not found in early components related to visual feature detection and processing (posterior N1 and frontal P2). However, the late positive complex (LPC) was more often expressed in the feature-binding condition, and the increase in amplitude was more pronounced in older participants. The LPC can be related to attentional allocation processes which might support the maintenance of the more complex stimulus representation in the binding task. However, the selective neural overactivation in the encoding phase observed in older participants does not prevent swap errors in the subsequent retrieval phase.

A spatially continuous diffusion model of visual working memory

We present results from five visual working memory (VWM) experiments in which participants were briefly shown between 2 and 6 colored squares. They were then cued to recall the color of one of the squares and they responded by choosing the color on a continuous color wheel. The experiments provided response proportions and response time (RT) measures as a function of angle for the choices. Current VWM models for this task include discrete models that assume an item is either within working memory or not and resource models that assume that memory strength varies as a function of the number of items. Because these models do not include processes that allow them to account for RT data, we implemented them within the spatially continuous diffusion model (SCDM, Ratcliff, 2018) and use the experimental data to evaluate these combined models. In the SCDM, evidence retrieved from memory is represented as a spatially continuous normal distribution and this drives the decision process until a criterion (represented as a 1-D line) is reached, which produces a decision. Noise in the accumulation process is represented by continuous Gaussian process noise over spatial position. The models that fit best from the discrete and resource-based classes converged on a common model that had a guessing component and that allowed the height of the normal memory-strength distribution to vary with number of items. The guessing component was implemented as a regular decision process driven by a flat evidence distribution, a zero-drift process. The combination of choice and RT data allows models that were not identifiable based on choice data alone to be discriminated.

Internal selective attention is delayed by competition between endogenous and exogenous factors

External attention is mediated by competition between endogenous (goal-driven) and exogenous (stimulus-driven) factors, with the balance of competition determining which stimuli are selected. Occasionally, exogenous factors "win" this competition and drive the selection of task-irrelevant stimuli. Endogenous and exogenous selection mechanisms may also compete to control the selection of internal representations (e.g., those stored in working memory), but whether this competition is resolved in the same way as external attention is unknown. Here, we leveraged the high temporal resolution of human EEG to determine how competition between endogenous and exogenous factors influences the selection of internal representations. Unlike external attention, competition did not prompt the selection of task-irrelevant working memory content. Instead, it delayed the endogenous selection of task-relevant working memory content by several hundred milliseconds. Thus, competition between endogenous and exogenous factors influences internal selective attention, but in a different way than external selective attention.

Working Memory Constrains Long-Term Memory in Children and Adults: Memory of Objects and Bindings

We explored how individual and age-related differences in working memory (WM) capacity affected subsequent long-term memory (LTM) retrieval. Unlike past studies, we tested WM and LTM not only for items, but also for item-color bindings. Our sample included 82 elementary school children and 42 young adults. The participants performed a WM task with images of unique everyday items presented sequentially at varying set sizes in different colors. Later, we tested LTM for items and item-color bindings from the WM task. The WM load during encoding constrained LTM, and participants with a higher WM capacity retrieved more items in the LTM test. Even when accounting for young children's poor item memory by considering only the items that they did remember, they exhibited an exacerbated difficulty with remembering item-color bindings in WM. Their LTM binding performance, however, as a proportion of remembered objects, was comparable to that of older children and adults. The WM binding performance was better during sub-span encoding loads, but with no clear transfer of this benefit to LTM. Overall, LTM item memory performance was constrained by individual and age-related WM limitations, but with mixed consequences for binding. We discuss the theoretical, practical, and developmental implications of this WM-to-LTM bottleneck.

Behavioral and electrophysiological evidence for the flexible recruitment of feature- and object-based processing in visual working memory comparison

Previous research is inconclusive on when visual working memory (VWM) can be object-based or feature-based. Prior event-related potential (ERP) studies using change detection tasks have found that amplitudes of the N200-an ERP index of VWM comparison- are sensitive to changes in both relevant and irrelevant features, suggesting a bias toward object-based processing. To test whether VWM comparison processing can operate in a feature-based manner, we aimed to create circumstances that would support feature-based processing by: 1) using a strong task-relevance manipulation, and 2) repeating features within a display. Participants completed two blocks of a change detection task for four-item displays in which they were told to respond to color changes (task relevant) but not shape changes (task irrelevant). The first block contained only task-relevant changes to create a strong task-relevance manipulation. In the second block, both relevant and irrelevant changes were present. In both blocks, half of the arrays contained within-display feature repetitions (e.g. two items of the same color or shape). We found that during the second block, N200 amplitudes were sensitive to task-relevant but not irrelevant features regardless of repetition status, consistent with feature-based processing. However, analyses of behavioral data and N200 latencies suggested that object-based processing was occurring at some stages of VWM processing on task-irrelevant feature change trials. In particular, task-irrelevant changes may be processed after no task-relevant feature change is revealed. Overall, the results from the current study suggest that the VWM processing is flexible and can be either object- or feature-based.

Integration or separation? Spatial and temporal representations of whole-body movements in visual working memory

Spatial and temporal information are two major feature dimensions of human movements. How these two types of information are represented in working memory-whether as integrated units or as individual features-influences how much information might be retained and how the retained information might be manipulated. In this study, we investigated how spatial (path/trajectory) and temporal (speed/rhythm) information of complex whole-body movements are represented in working memory under a more ecologically valid condition wherein the spatiotemporal continuity of movement sequences was considered. We found that the spatial and temporal information are not automatically integrated but share the storage capacity and compete for a common pool of cognitive resources. The finding rejects the strong form of object-based representation and supports the partial independence of spatial and temporal processing. Nevertheless, we also found that contextual factors, such as the way movements are organized and displayed, can further modulate the level of object-based representation and spatiotemporal integration.

Pathological Slow-Wave Activity and Impaired Working Memory Binding in Post-Traumatic Amnesia

Associative binding is key to normal memory function and is transiently disrupted during periods of post-traumatic amnesia (PTA) following traumatic brain injury (TBI). Electrophysiological abnormalities, including low-frequency activity, are common following TBI. Here, we investigate associative memory binding during PTA and test the hypothesis that misbinding is caused by pathological slowing of brain activity disrupting cortical communication. Thirty acute moderate to severe TBI patients (25 males; 5 females) and 26 healthy controls (20 males; 6 females) were tested with a precision working memory paradigm requiring the association of object and location information. Electrophysiological effects of TBI were assessed using resting-state EEG in a subsample of 17 patients and 21 controls. PTA patients showed abnormalities in working memory function and made significantly more misbinding errors than patients who were not in PTA and controls. The distribution of localization responses was abnormally biased by the locations of nontarget items for patients in PTA, suggesting a specific impairment of object and location binding. Slow-wave activity was increased following TBI. Increases in the δ-α ratio indicative of an increase in low-frequency power specifically correlated with binding impairment in working memory. Connectivity changes in TBI did not correlate with binding impairment. Working memory and electrophysiological abnormalities normalized at 6 month follow-up. These results show that patients in PTA show high rates of misbinding that are associated with a pathological shift toward lower-frequency oscillations.SIGNIFICANCE STATEMENT How do we remember what was where? The mechanism by which information (e.g., object and location) is integrated in working memory is a central question for cognitive neuroscience. Following significant head injury, many patients will experience a period of post-traumatic amnesia (PTA) during which this associative binding is disrupted. This may be because of electrophysiological changes in the brain. Using a precision working memory test and resting-state EEG, we show that PTA patients demonstrate impaired binding ability, and this is associated with a shift toward slower-frequency activity on EEG. Abnormal EEG connectivity was observed but was not specific to PTA or binding ability. These findings contribute to both our mechanistic understanding of working memory binding and PTA pathophysiology.

Aging and feature binding in visual working memory

Older adults have reduced performance in visual working memory tasks in comparison to young adults, but the precipitators of the age-related impairment are not fully understood. The most common interpretation of this difference is that older adults are incapable of maintaining the same amount of object representations as young adults over short intervals (in line with the fixed-slot model of working memory). However, it has remained largely unexplored whether the age-related decline is only due to the number of representations that older individuals can retain in visual working memory, or whether the content of the representation(s) may have an effect as well (in line with the flexible-resource model of working memory). Feature binding studies represent an interesting research line to examine the content of older adults' representations. In this mini-review, we present the main results across feature binding studies in aging, as well as highlight the importance of manipulating both the representation content and number to have a stress test of the various models of working memory and their contribution to aging. Overall, feature binding studies, together with the simultaneous manipulation of set size, will allow us to better understand the nature of the age-related decline of visual working memory.

Swap errors in visual working memory are fully explained by cue-feature variability

In cue-based recall from working memory, incorrectly reporting features of an uncued item may be referred to as a "swap" error. One account of these errors ascribes them to variability in memory for the cue features leading to erroneous selection of a non-target item, especially if it is similar to the target in the cue-feature dimension. However, alternative accounts of swap errors include cue-independent misbinding, and strategic guessing when the cued item is not in memory. Here we investigated the cause of swap errors by manipulating the variability with which either cue or report features (orientations in Exp 1; motion directions in Exp 2) were encoded. We found that swap errors increased with increasing variability in memory for the cue features, and their changing frequency could be quantitatively predicted based on recall variability when the same feature was used for report. These results are inconsistent with the hypothesis that swaps are a strategic response to forgotten items, and suggest that swap errors could be wholly accounted for by confusions due to cue-dimension variability. In a third experiment we examined whether spatial configuration of memory arrays in tasks with spatial cueing has an influence on swap error frequency. We observed a specific tendency to make swap errors to non-targets located precisely opposite to the cued location, suggesting that stimulus positions are partially encoded in a non-metric format.

Similar mechanisms of temporary bindings for identity and location of objects in healthy ageing: an eye-tracking study with naturalistic scenes

The ability to maintain visual working memory (VWM) associations about the identity and location of objects has at times been found to decrease with age. To date, however, this age-related difficulty was mostly observed in artificial visual contexts (e.g., object arrays), and so it is unclear whether it may manifest in naturalistic contexts, and in which ways. In this eye-tracking study, 26 younger and 24 healthy older adults were asked to detect changes in a critical object situated in a photographic scene (192 in total), about its identity (the object becomes a different object but maintains the same position), location (the object only changes position) or both (the object changes in location and identity). Aging was associated with a lower change detection performance. A change in identity was harder to detect than a location change, and performance was best when both features changed, especially in younger adults. Eye movements displayed minor differences between age groups (e.g., shorter saccades in older adults) but were similarly modulated by the type of change. Latencies to the first fixation were longer and the amplitude of incoming saccades was larger when the critical object changed in location. Once fixated, the target object was inspected for longer when it only changed in identity compared to location. Visually salient objects were fixated earlier, but saliency did not affect any other eye movement measures considered, nor did it interact with the type of change. Our findings suggest that even though aging results in lower performance, it does not selectively disrupt temporary bindings of object identity, location, or their association in VWM, and highlight the importance of using naturalistic contexts to discriminate the cognitive processes that undergo detriment from those that are instead spared by aging.

Working Memory Swap Errors Have Identifiable Neural Representations

Working memory is an essential component of cognition that facilitates goal-directed behavior. Famously, it is severely limited and performance suffers when memory load exceeds an individual's capacity. Modeling of visual working memory responses has identified two likely types of errors: guesses and swaps. Swap errors may arise from a misbinding between the features of different items. Alternatively, these errors could arise from memory noise in the feature dimension used for cueing a to-be-tested memory item, resulting in the wrong item being selected. Finally, it is possible that so-called swap errors actually reflect informed guessing, which could occur at the time of a cue, or alternatively, at the time of the response. Here, we combined behavioral response modeling and fMRI pattern analysis to test the hypothesis that swap errors involve the active maintenance of an incorrect memory item. After the encoding of six spatial locations, a retro-cue indicated which location would be tested after memory retention. On accurate trials, we could reconstruct a memory representation of the cued location in both early visual cortex and intraparietal sulcus. On swap error trials identified with mixture modeling, we were able to reconstruct a representation of the swapped location, but not of the cued location, suggesting the maintenance of the incorrect memory item before response. Moreover, participants subjectively responded with some level of confidence, rather than complete guessing, on a majority of swap error trials. Together, these results suggest that swap errors are not mere response-phase guesses, but instead result from failures of selection in working memory, contextual binding errors, or informed guesses, which produce active maintenance of incorrect memory representations.

A model of working memory for latent representations

We propose a mechanistic explanation of how working memories are built and reconstructed from the latent representations of visual knowledge. The proposed model features a variational autoencoder with an architecture that corresponds broadly to the human visual system and an activation-based binding pool of neurons that links latent space activities to tokenized representations. The simulation results revealed that new pictures of familiar types of items can be encoded and retrieved efficiently from higher levels of the visual hierarchy, whereas truly novel patterns are better stored using only early layers. Moreover, a given stimulus in working memory can have multiple codes, which allows representation of visual detail in addition to categorical information. Finally, we validated our model's assumptions by testing a series of predictions against behavioural results obtained from working memory tasks. The model provides a demonstration of how visual knowledge yields compact visual representation for efficient memory encoding.

A direct comparison of attentional orienting to spatial and temporal positions in visual working memory

Different visual attributes effectively guide attention to specific items in visual working memory (VWM), ensuring that particularly important memory contents are readily available. Predictable temporal structures contribute to this efficient use of VWM: items are prospectively prioritized when they are expected to be needed. Occasionally, however, visual events only gain relevance through their timing after they have passed. We investigated retrospective attentional orienting based on temporal position by directly comparing it with orienting to spatial locations, which is typically considered the most powerful selection mechanism. In a colour-change-detection task, in which items appeared sequentially at different locations, symbolic number cues validly indicated the temporal or spatial location of the upcoming probe item either before encoding (precues; Experiment 1) or during maintenance (retrocues; Experiments 1-3). Temporal and spatial cues were physically identical and only differed in their mapping onto either temporal or spatial positions. Predictive cues yielded cueing benefits (i.e., higher accuracy and shorter reaction times) as compared with neutral cues, with larger benefits for precues than for retrocues. Importantly, spatial and temporal cueing benefits did not differ. Equivalent retrocueing benefits were also observed across different cue-probe intervals and irrespective of whether spatial or temporal position was used as retrieval cue, indicating that items were directly bound to temporal position and not prioritized via a space-based mechanism. These findings show that spatial and temporal properties can be used equally well to flexibly prioritise representations held in VWM and they highlight the functional similarities of space and time in VWM.

Salient Cognitive Paradigms to Assess Preclinical Alzheimer's Disease

Despite the growing emphasis to identify early biological markers that can detect the progressive accumulation of brain pathology in the complex pathophysiologic cascade that occurs in Alzheimer's disease (AD), we continue to employ the same neuropsychological paradigms that were developed to detect dementia or frank cognitive impairment. It has become increasingly clear that we cannot expect to measure clinically meaningful change in relationship to these emerging preclinical biomarkers using these traditional cognitive assessment paradigms, nor will we advance the efforts to identify the earliest cognitive changes that emerge in AD. Over the last decade, a few novel promising cognitive assessment paradigms have emerged that have shown promise in identifying subtle cognitive deficits in AD which aids in early detection and monitoring of meaningful cognitive change over time. Some of these cognitive assessment paradigms are reviewed here, including semantic interference, semantic intrusion errors, memory binding, and binding of face and name associations. These paradigms may be useful for AD clinical trials focused on secondary prevention if there is sufficient rigor to suggest that they correlate with AD biomarkers, having robust sensitivity, specificity, and predictive utility among culturally and linguistically diverse populations at-risk for AD.

Mechanisms of feature binding in visual working memory are stable over long delays

The ability to accurately retain the binding between the features of different objects is a critical element of visual working memory. The underlying mechanism can be elucidated by analyzing correlations of response errors in dual-report experiments, in which participants have to report two features of a single item from a previously viewed stimulus array. Results from separate previous studies using different cueing conditions have indicated that location takes a privileged role in mediating binding between other features, in that largely independent response errors have been observed when location was used as a cue, but errors were highly correlated when location was one of the reported features. Earlier results from change detection tasks likewise support such a special role of location, but they also suggest that this role is substantially reduced for longer retention intervals in favor of object-based representation. In the present study, we replicated the findings of previous dual-report tasks with different cueing conditions, using matched stimuli and procedures. Moreover, we show that the observed patterns of error correlations remain qualitatively unchanged with longer retention intervals. Fits with neural population models demonstrate that the behavioral results at long, as well as short, delays are best explained by memory representations in independent feature maps, in which an item's features are bound to each other only via their shared location.

How to refocus attention on working memory representations following interruptions-Evidence from frontal theta and posterior alpha oscillations

Interruptions lead to a deterioration of primary task performance. Applied research usually describes a delay in primary task resumption as an essential component of this performance deficit. Here, we investigate this approach using electrophysiological correlates of the focusing of attention within working memory, a process that is fundamental to switching between different tasks. A lateralized working memory task was frequently interrupted by either a high- or low-demanding arithmetic task and a subsequent retrospective cue indicated the working memory item required for later report. The detrimental effect of interruptions on primary task performance was most pronounced for high-demanding interruptions. After retro-cue presentation, fronto-central theta power (4-7 Hz) was lowest following high-demanding interruptions and posterior alpha power (8-14 Hz) was less suppressed in the two interruption conditions. These effects might be related to a deficit in attentional control processes following the retrospective cue. Furthermore, we introduce the suppression of posterior alpha power contralateral to the remembered primary task stimuli during the interruption phase as a temporal marker for primary task resumption. Especially for cognitively demanding interruption tasks, this effect seems to overlap in time with the processing of the interruption, which should contribute to the primary task performance deficit.

Across-Area Synchronization Supports Feature Integration in a Biophysical Network Model of Working Memory

Working memory function is severely limited. One key limitation that constrains the ability to maintain multiple items in working memory simultaneously is so-called swap errors. These errors occur when an inaccurate response is in fact accurate relative to a non-target stimulus, reflecting the failure to maintain the appropriate association or "binding" between the features that define one object (e.g., color and location). The mechanisms underlying feature binding in working memory remain unknown. Here, we tested the hypothesis that features are bound in memory through synchrony across feature-specific neural assemblies. We built a biophysical neural network model composed of two one-dimensional attractor networks - one for color and one for location - simulating feature storage in different cortical areas. Within each area, gamma oscillations were induced during bump attractor activity through the interplay of fast recurrent excitation and slower feedback inhibition. As a result, different memorized items were held at different phases of the network's oscillation. These two areas were then reciprocally connected via weak cortico-cortical excitation, accomplishing binding between color and location through the synchronization of pairs of bumps across the two areas. Encoding and decoding of color-location associations was accomplished through rate coding, overcoming a long-standing limitation of binding through synchrony. In some simulations, swap errors arose: "color bumps" abruptly changed their phase relationship with "location bumps." This model, which leverages the explanatory power of similar attractor models, specifies a plausible mechanism for feature binding and makes specific predictions about swap errors that are testable at behavioral and neurophysiological levels.

Categorical judgments do not modify sensory representations in working memory

Categorical judgments can systematically bias the perceptual interpretation of stimulus features. However, it remained unclear whether categorical judgments directly modify working memory representations or, alternatively, generate these biases via an inference process down-stream from working memory. To address this question we ran two novel psychophysical experiments in which human subjects had to reverse their categorical judgments about a stimulus feature, if incorrect, before providing an estimate of the feature. If categorical judgments indeed directly altered sensory representations in working memory, subjects' estimates should reflect some aspects of their initial (incorrect) categorical judgment in those trials. We found no traces of the initial categorical judgment. Rather, subjects seemed to be able to flexibly switch their categorical judgment if needed and use the correct corresponding categorical prior to properly perform feature inference. A cross-validated model comparison also revealed that feedback may lead to selective memory recall such that only memory samples that are consistent with the categorical judgment are accepted for the inference process. Our results suggest that categorical judgments do not modify sensory information in working memory but rather act as top-down expectations in the subsequent sensory recall and inference process.

Does the presence of more features in a bound representation in working memory require extra object-based attention?

Recent studies have examined the role of attention in retaining bound representations in working memory (WM) and found that object-based attention plays a pivotal role. However, no study has investigated whether maintaining bound representations with more features in WM requires extra object-based attention. We investigated this by examining whether a secondary task consuming object-based attention was more disruptive to the maintenance of bindings in WM when more features were stored per object. We instructed participants to memorize three bound representations in a WM task while manipulating the number of features (two vs. three features) contained in each representation. Moreover, we manipulated whether a secondary task consuming object-based attention was interpolated into the maintenance phase of WM. If extra object-based attention was required after the addition of an extra feature in the bound representation, the secondary task would result in a greater disruption of the three- rather than two-featured binding. In two experiments, we found that the added secondary task significantly impaired the binding performance, but the performance of the two- and three-featured bindings was disrupted to the same extent. These results suggest that the presence of more features in a bound representation in WM does not require extra object-based attention.

Location-independent feature binding in visual working memory for sequentially presented objects

Spatial location is believed to have a privileged role in binding features held in visual working memory. Supporting this view, Pertzov and Husain (Attention, Perception, & Psychophysics, 76(7), 1914–1924, 2014) reported that recall of bindings between visual features was selectively impaired when items were presented sequentially at the same location compared to sequentially at different locations. We replicated their experiment, but additionally tested whether the observed impairment could be explained by perceptual interference during encoding. Participants viewed four oriented bars in highly discriminable colors presented sequentially either at the same or different locations, and after a brief delay were cued with one color to reproduce the associated orientation. When we used the same timing as the original study, we reproduced its key finding of impaired binding memory in the same-location condition. Critically, however, this effect was significantly modulated by the duration of the inter-stimulus interval, and disappeared if memoranda were presented with longer delays between them. In a second experiment, we tested whether the effect generalized to other visual features, namely reporting of colors cued by stimulus shape. While we found performance deficits in the same-location condition, these did not selectively affect binding memory. We argue that the observed effects are best explained by encoding interference, and that memory for feature binding is not necessarily impaired when memoranda share the same location.

Consequence of stroke for feature recall and binding in visual working memory

Visual memory for objects involves the integration, or binding, of individual features into a coherent representation. We used a novel approach to assess feature binding, using a delayed-reproduction task in combination with computational modeling and lesion analysis. We assessed stroke patients and neurotypical controls on a visual working memory task in which spatial arrays of colored disks were presented. After a brief delay, participants either had to report the color of one disk cued by its location or the location of one disk cued by its color. Our results demonstrate that, in the controls, report imprecision and swap errors (non-target reports) can be explained by a single source of variability. Stroke patients showed an overall decrease in memory precision for both color and location, with only limited evidence for deviations from the predicted relationship between report precision and swap errors. These deviations were primarily deficits in reporting items rather than selecting items based on the cue. Atlas-based lesion-symptom mapping showed that selection and reporting deficits, precision in reporting color, and precision in reporting location were associated with different lesion profiles. Deficits in binding are associated with lesions in the left somatosensory cortex, deficits in the precision of reporting color with bilateral fronto-parietal regions, and no anatomical substrates were identified for precision in reporting location. Our results converge with previous reports that working memory representations are widely distributed in the brain and can be found across sensory, parietal, temporal, and prefrontal cortices. Stroke patients demonstrate mostly subtle impairments in visual working memory, perhaps because representations from different areas in the brain can partly compensate for impaired encoding in lesioned areas. These findings contribute to understanding of the relation between memorizing features and their bound representations.

Incidental encoding of visual information in temporal reference frames in working memory

Visual events are structured in space and time, yet models of visual working memory (VWM) have largely relied on tasks emphasizing spatial aspects. Here, we show that temporal properties of visual events are incidentally encoded along with spatial properties. In five experiments, participants performed change-detection tasks, in which items had unique spatial and temporal coordinates at encoding. Crucially, neither space nor time was task-relevant. The key manipulation concerned the retrieval context: The test array was identical to the memory array either in its entire spatiotemporal structure, or only its spatial or temporal structure. Removing spatial or temporal information at retrieval resulted in costs, indicating that memory relied on both spatial and temporal context in which items were initially perceived. Encoding of spatiotemporal structure occurred incidentally, not strategically, as it was robust even when the retrieval context was perfectly predictable. However, spatial and temporal inter-item spacings influenced the weighting of spatial and temporal information: It favoured the domain in which items were more widely spaced, facilitating their individuation and, likely, access to representations. Across individuals, the weighting of spatial and temporal information varied substantially, but it remained consistent across sessions, suggesting stable preferences for coding in the spatial or temporal domain. No comparable incidental encoding occurred for other task-irrelevant feature dimensions (size or colour). We propose that temporal structure serves as fundamental a function in VWM as spatial structure, scaffolding events that unfold over time.

Insights into visual working memory precision at the feature- and object-level from a hemispheric encoding manipulation

Mnemonic precision is an important aspect of visual working memory (WM). Here, we probed mechanisms that affect precision for spatial (size) and non-spatial (colour) features of an object, and whether these features are encoded and/or stored separately in WM. We probed precision at the feature-level—that is, whether different features of a single object are represented separately or together in WM—and the object-level—that is, whether different features across a set of sequentially presented objects are represented in the same or different WM stores. By manipulating whether stimuli were encoded by the left and/or right hemisphere, we gained further insights into how objects are represented in WM. At the feature-level, we tested whether recall fidelity for the two features of an object fluctuated in tandem from trial to trial. We observed no significant coupling under either central or lateralized encoding, supporting the claim of parallel feature channels at encoding. At the level of WM storage of a set of objects, we found asymmetric feature interference under central encoding, whereby an increase in colour load led to a decrease in size precision. When objects were encoded by a single hemisphere, however, we found largely independent feature stores. Precision for size was more resistant to interference from the size of another object under right-hemisphere encoding; by contrast, precision for colour did not differ across hemispheres, suggesting a more distributed WM store. These findings suggest that distinct features of a single object are represented separately but are then partially integrated during maintenance of a set of sequentially presented objects.

Working Memory in Unilateral Spatial Neglect: Evidence for Impaired Binding of Object Identity and Object Location

Working memory (WM) is known to be impaired in patients with stroke experiencing unilateral spatial neglect (USN). Here, we examined in a systematic manner three WM components: memory of object identity, memory of object location, and binding between object identity and location. Moreover, we used two different retention intervals to isolate maintenance from other mnemonic and perceptual processes. Fourteen USN first-event stroke patients with right-hemisphere damage were tested in two different WM experiments using long and short retention intervals and an analog response scale. Patients exhibited more identification errors for items displayed on the contralesional side. Localization errors were also more prominent in the contralesional side, especially after a long retention interval. These localization errors were often a result of swap errors, that is, erroneous localizations of correctly identified contralesional objects in correctly memorized locations of ipsilesional objects. We conclude that a key WM deficit in USN is a lateralized impairment in binding between the identity of an object and its spatial tag.

The role of attention in retaining the binding of integral features in working memory

Previous studies have suggested that retaining bindings in working memory (WM) requires more object-based attention than retaining constituent features. However, we still need to address the object-based attention hypothesis to determine both the generality (Does the object-based attention hypothesis of binding apply to feature bindings other than those tested?) and the reality (Was the observed effect in previous studies an artifact of the testing process?). We addressed these two issues by focusing on the binding of integral features, which was ignored in previous studies. Integral features can be manipulated independently but cannot be attended to or processed independently of each other, and they are primarily perceived in a more unitary fashion. Consequently, integral-feature bindings should be processed as integrated units without the help of extra object-based attention. We examined whether or not the object-based attention hypothesis applied to integral-feature bindings (generality), and these results enabled us to check the reality of the hypothesis. In line with our prediction, we found that a secondary task consuming object-based attention did not selectively impair the binding performance (Experiments 1, 2, 3, 5, and 7). The absence of selective binding impairment was not attributable to the use of an invalid secondary task (Experiment 4), failure to memorize the binding between length and width (Experiment 6), tapping the incorrect type of attention (Experiment 6), the feasibility of feature categorization (Experiment 7), or poor task performance (Experiment 7). Overall, these results suggest that the object-based attention hypothesis does not fit for the integral-feature bindings, and that the pivotal role of object-based attention reported by previous studies was reliable.

The Binding Problem after an eye movement

Spatial attention is thought to be the "glue" that binds features together (e.g., Treisman & Gelade, 1980, Psychology, 12[1], 97-136)-but attention is dynamic, constantly moving across multiple goals and locations. For example, when a person moves her eyes, visual inputs that are coded relative to the eyes (retinotopic) must be rapidly updated to maintain stable world-centered (spatiotopic) representations. Here, we examined how dynamic updating of spatial attention after a saccadic eye movement affects object-feature binding. Immediately after a saccade, participants were simultaneously presented with four colored and oriented bars (one at a precued spatiotopic target location) and instructed to reproduce both the color and orientation of the target item. Object-feature binding was assessed by applying probabilistic mixture models to the joint distribution of feature errors: feature reports for the target item could be correlated (and thus bound together) or independent. We found that compared with holding attention without an eye movement, attentional updating after an eye movement produced more independent errors, including illusory conjunctions, in which one feature of the item at the spatiotopic target location was misbound with the other feature of the item at the initial retinotopic location. These findings suggest that even when only one spatiotopic location is task relevant, spatial attention-and thus object-feature binding-is malleable across and after eye movements, heightening the challenge that eye movements pose for the binding problem and for visual stability.

Object-based attention in retaining binding in working memory: Influence of activation states of working memory

It has been suggested that retaining bindings in working memory (WM) requires more object-based attention than retaining constituent features. Recent studies have found that when memorized stimuli are presented sequentially, the most recent stimulus is in a highly accessible privileged state such that it is retained in a relatively automatic and resource-free manner, whereas the other stimuli are in a non-privileged state. The current study investigated whether the activation states of WM modulate the role of object-based attention in retaining bindings in WM. To address this question, we presented three colored shapes sequentially and added a transparent-motion task (Experiment 1) or a mental rotation task (Experiment 2) into the WM maintenance phase to consume object-based attention. We consistently found that consuming object-based attention led to a larger impairment to bindings relative to constituent features, which is independent of the WM activation states, suggesting that object-based attention is critical in retaining bindings in WM across activation states of WM.

Context information supports serial dependence of multiple visual objects across memory episodes

Serial dependence is thought to promote perceptual stability by compensating for small changes of an object’s appearance across memory episodes. So far, it has been studied in situations that comprised only a single object. The question of how we selectively create temporal stability of several objects remains unsolved. In a memory task, objects can be differentiated by their to-be-memorized feature (content) as well as accompanying discriminative features (context). We test whether congruent context features, in addition to content similarity, support serial dependence. In four experiments, we observe a stronger serial dependence between objects that share the same context features across trials. Apparently, the binding of content and context features is not erased but rather carried over to the subsequent memory episode. As this reflects temporal dependencies in natural settings, our findings reveal a mechanism that integrates corresponding content and context features to support stable representations of individualized objects over time.

Visual cognition compensates for small changes in an object’s appearance to ensure its perceived continuity. We show that in situations with multiple objects, context features like color, temporal or spatial position are used as anchors to selectively integrate corresponding objects over time.

Theory of neural coding predicts an upper bound on estimates of memory variability

Observers reproducing elementary visual features from memory after a short delay produce errors consistent with the encoding-decoding properties of neural populations. While inspired by electrophysiological observations of sensory neurons in cortex, the population coding account of these errors is based on a mathematical idealization of neural response functions that abstracts away most of the heterogeneity and complexity of real neuronal populations. Here we examine a more physiologically grounded model based on the tuning of a large set of neurons recorded in macaque V1 and show that key predictions of the idealized model are preserved. Both models predict long-tailed distributions of error when memory resources are taxed, as observed empirically in behavioral experiments and commonly approximated with a mixture of normal and uniform error components. Specifically, for an idealized homogeneous neural population, the width of the fitted normal distribution cannot exceed the average tuning width of the component neurons, and this also holds to a good approximation for more biologically realistic populations. Examining eight published studies of orientation recall, we find a consistent pattern of results suggestive of a median tuning width of approximately 20°, which compares well with neurophysiological observations. The finding that estimates of variability obtained by the normal-plus-uniform mixture method are bounded from above leads us to reevaluate previous studies that interpreted a saturation in width of the normal component as evidence for fundamental limits on the precision of perception, working memory, and long-term memory.

Modulation of the pupillary response by the content of visual working memory

Studies of selective attention during perception have revealed modulation of the pupillary response according to the brightness of task-relevant (attended) vs. -irrelevant (unattended) stimuli within a visual display. As a strong test of top-down modulation of the pupil response by selective attention, we asked whether changes in pupil diameter follow internal shifts of attention to memoranda of visual stimuli of different brightness maintained in working memory, in the absence of any visual stimulation. Across 3 studies, we reveal dilation of the pupil when participants orient attention to the memorandum of a dark grating relative to that of a bright grating. The effect occurs even when the attention-orienting cue is independent of stimulus brightness, and even when stimulus brightness is merely incidental and not required for the working-memory task of judging stimulus orientation. Furthermore, relative dilation and constriction of the pupil occurred dynamically and followed the changing temporal expectation that 1 or the other stimulus would be probed across the retention delay. The results provide surprising and consistent evidence that pupil responses are under top-down control by cognitive factors, even when there is no direct adaptive gain for such modulation, since no visual stimuli were presented or anticipated. The results also strengthen the view of sensory recruitment during working memory, suggesting even activation of sensory receptors. The thought-provoking corollary to our findings is that the pupils provide a reliable measure of what is in the focus of mind, thus giving a different meaning to old proverbs about the eyes being a window to the mind.

Current directions in visual working memory research: An introduction and emerging insights

Visual working memory (VWM) is a core construct in the cognitive (neuro-)sciences, assumed to serve as a hub for information exchange and thus supporting a multitude of cognitive functions related to processing visual information. Here, we give an introduction into key terms and paradigms and an overview of ongoing debates in the field, to which the articles collected in this Special Issue on 'Current Directions in Visual Working Memory Research' contribute. Our aim is to extract, from this overview, some 'emerging' theoretical insights concerning questions such as the optimal way to examine VWM, which types of mental representations contribute to performance on VWM tasks, and how VWM keeps features from the same object together and apart from features of concurrently maintained objects (the binding problem).

Two good reasons to say 'change!' - ensemble representations as well as item representations impact standard measures of VWM capacity

Visual working memory (VWM) is a central bottleneck in human information processing. Its capacity is most often measured in terms of how many individual-item representations VWM can hold (k). In the standard task employed to estimate k, an array of highly discriminable colour patches is maintained and, after a short retention interval, compared to a test display (change detection). Recent research has shown that with more complex, structured displays, change-detection performance is, in addition to individual-item representations, supported by ensemble representations formed as a result of spatial subgroupings. Here, by asking participants to additionally localize the change, we reveal indication for an influence of ensemble representations even in the very simple, unstructured displays of the colour-patch change-detection task. Critically, pure-item models from which standard formulae of k are derived do not consider ensemble representations and, therefore, potentially overestimate k. To gauge this overestimation, we develop an item-plus-ensemble model of change detection and change localization. Estimates of k from this new model are about 1 item (~30%) lower than the estimates from traditional pure-item models, even if derived from the same data sets.