Functions of Memory Across Saccadic Eye Movements

Representation and computation in visual working memory

The ability to sustain internal representations of the sensory environment beyond immediate perception is a fundamental requirement of cognitive processing. In recent years, debates regarding the capacity and fidelity of the working memory (WM) system have advanced our understanding of the nature of these representations. In particular, there is growing recognition that WM representations are not merely imperfect copies of a perceived object or event. New experimental tools have revealed that observers possess richer information about the uncertainty in their memories and take advantage of environmental regularities to use limited memory resources optimally. Meanwhile, computational models of visuospatial WM formulated at different levels of implementation have converged on common principles relating capacity to variability and uncertainty. Here we review recent research on human WM from a computational perspective, including the neural mechanisms that support it.

A dynamic neural resource model bridges sensory and working memory

Probing memory of a complex visual image within a few hundred milliseconds after its disappearance reveals significantly greater fidelity of recall than if the probe is delayed by as little as a second. Classically interpreted, the former taps into a detailed but rapidly decaying visual sensory or 'iconic' memory (IM), while the latter relies on capacity-limited but comparatively stable visual working memory (VWM). While iconic decay and VWM capacity have been extensively studied independently, currently no single framework quantitatively accounts for the dynamics of memory fidelity over these time scales. Here, we extend a stationary neural population model of VWM with a temporal dimension, incorporating rapid sensory-driven accumulation of activity encoding each visual feature in memory, and a slower accumulation of internal error that causes memorized features to randomly drift over time. Instead of facilitating read-out from an independent sensory store, an early cue benefits recall by lifting the effective limit on VWM signal strength imposed when multiple items compete for representation, allowing memory for the cued item to be supplemented with information from the decaying sensory trace. Empirical measurements of human recall dynamics validate these predictions while excluding alternative model architectures. A key conclusion is that differences in capacity classically thought to distinguish IM and VWM are in fact contingent upon a single resource-limited WM store.

Working memory in action: Transsaccadic working memory deficits in the left visual field and after transcallosal remapping

Every waking second, we make three saccadic eye movements that move our retinal images. Thus, to attain a coherent image of the world we need to remember visuo-spatial information across saccades. But transsaccadic working memory (tWM) remains poorly understood. Crucially, there has been a debate whether there are any differences in tWM for the left vs. right visual field and depending on saccade direction. However, previous studies have probed tWM with minimal loads whereas spatial differences might arise with higher loads. Here we employed a task that probed higher memory load for spatial information in the left and right visual field and with horizontal as well as vertical saccades. We captured several measures of precision and accuracy of performance that, when submitted to principal component analysis, produced two components. Component 1, mainly associated with precision, yielded greater error for the left than the right visual field. Component 2 was associated with performance accuracy and unexpectedly produced a disadvantage after rightward saccades. Both components showed that performance was worse when rightward or leftward saccades afforded a shift of memory representations between visual fields compared to remapping within the same field. Our study offers several novel findings. It is the first to show that tWM involves at least two components likely reflecting working memory capacity and strategic aspects of working memory, respectively. Reduced capacity for the left, rather than the right visual field is consistent with how the left and right visual fields are known to be represented in the two hemispheres. Remapping difficulties between visual fields is consistent with the limited information transfer across the corpus callosum. Finally, the impact of rightward saccades on working memory might be due to greater interference of the accompanying shifts of attention. Our results highlight the dynamic nature of transsaccadic working memory.

A proposed attention-based model for spatial memory formation and retrieval

Animals use sensory information and memory to build internal representations of space. It has been shown that such representations extend beyond the geometry of an environment and also encode rich sensory experiences usually referred to as context. In mammals, contextual inputs from sensory cortices appear to be converging on the hippocampus as a key area for spatial representations and memory. How metric and external sensory inputs (e.g., visual context) are combined into a coherent and stable place representation is not fully understood. Here, I review the evidence of attentional effects along the ventral visual pathway and in the medial temporal lobe and propose an attention-based model for the integration of visual context in spatial representations. I further suggest that attention-based retrieval of spatial memories supports a feedback mechanism that allows consolidation of old memories and new sensory experiences related to the same place, thereby contributing to the stability of spatial representations. The resulting model has the potential to generate new hypotheses to explain complex responses of spatial cells such as place cells in the hippocampus.

Can working memory account for EMDR efficacy in PTSD?

BACKGROUND

Although eye movement desensitization and reprocessing (EMDR) has been shown to be effective in the treatment of PTSD for years, it remains controversial due to the lack of understanding of its mechanisms of action. We examined whether the working memory (WM) hypothesis -the competition for limited WM resources induced by the dual task attenuates the vividness and emotionality of the traumatic memory - would provide an explanation for the beneficial effect induced by bilateral stimulation.

METHODS

We followed the Prisma guidelines and identified 11 articles categorized in two types of designs: studies involving participants with current PTSD symptoms and participants without PTSD diagnosis.

RESULTS

Regardless of the types of studies, the results showed a reduction of vividness and emotionality in the recall of traumatic stimuli under a dual-task condition compared to a control condition, such as recall alone. However, two studies used a follow-up test to show that this effect does not seem to last long.

CONCLUSION

Our results provide evidence for the WM hypothesis and suggest that recalling a traumatic memory while performing a secondary task would shift the individual's attention away from the retrieval process and result in a reduction in vividness and emotionality, also associated with the reduction of symptoms.

Long-term memory and working memory compete and cooperate to guide attention

Multiple types of memory guide attention: Both long-term memory (LTM) and working memory (WM) effectively guide visual search. Furthermore, both types of memories can capture attention automatically, even when detrimental to performance. It is less clear, however, how LTM and WM cooperate or compete to guide attention in the same task. In a series of behavioral experiments, we show that LTM and WM reliably cooperate to guide attention: Visual search is faster when both memories cue attention to the same spatial location (relative to when only one memory can guide attention). LTM and WM competed to guide attention in more limited circumstances: Competition only occurred when these memories were in different dimensions - particularly when participants searched for a shape and held an accessory color in mind. Finally, we found no evidence for asymmetry in either cooperation or competition: There was no evidence that WM helped (or hindered) LTM-guided search more than the other way around. This lack of asymmetry was found despite differences in LTM-guided and WM-guided search overall, and differences in how two LTMs and two WMs compete or cooperate with each other to guide attention. This work suggests that, even if only one memory is currently task-relevant, WM and LTM can cooperate to guide attention; they can also compete when distracting features are salient enough. This work elucidates interactions between WM and LTM during attentional guidance, adding to the literature on costs and benefits to attention from multiple active memories.

Multiple spatial frames for immersive working memory

As we move around, relevant information that disappears from sight can still be held in working memory to serve upcoming behaviour. How we maintain and select visual information as we move through the environment remains poorly understood because most laboratory tasks of working memory rely on removing visual material while participants remain still. We used virtual reality to study visual working memory following self-movement in immersive environments. Directional biases in gaze revealed the recruitment of more than one spatial frame for maintaining and selecting memoranda following self-movement. The findings bring the important realization that multiple spatial frames support working memory in natural behaviour. The results also illustrate how virtual reality can be a critical experimental tool to characterize this core memory system.

Transsaccadic integration operates independently in different feature dimensions

Our knowledge about objects in our environment reflects an integration of current visual input with information from preceding gaze fixations. Such a mechanism may reduce uncertainty but requires the visual system to determine which information obtained in different fixations should be combined or kept separate. To investigate the basis of this decision, we conducted three experiments. Participants viewed a stimulus in their peripheral vision and then made a saccade that shifted the object into the opposite hemifield. During the saccade, the object underwent changes of varying magnitude in two feature dimensions (Experiment 1, color and location; Experiments 2 and 3, color and orientation). Participants reported whether they detected any change and estimated one of the postsaccadic features. Integration of presaccadic with postsaccadic input was observed as a bias in estimates toward the presaccadic feature value. In all experiments, presaccadic bias weakened as the magnitude of the transsaccadic change in the estimated feature increased. Changes in the other feature, despite having a similar probability of detection, had no effect on integration. Results were quantitatively captured by an observer model where the decision whether to integrate information from sequential fixations is made independently for each feature and coupled to awareness of a feature change.

Intrasaccadic motion streaks jump-start gaze correction

Rapid eye movements (saccades) incessantly shift objects across the retina. To establish object correspondence, the visual system is thought to match surface features of objects across saccades. Here, we show that an object's intrasaccadic retinal trace-a signal previously considered unavailable to visual processing-facilitates this match making. Human observers made saccades to a cued target in a circular stimulus array. Using high-speed visual projection, we swiftly rotated this array during the eyes' flight, displaying continuous intrasaccadic target motion. Observers' saccades landed between the target and a distractor, prompting secondary saccades. Independently of the availability of object features, which we controlled tightly, target motion increased the rate and reduced the latency of gaze-correcting saccades to the initial presaccadic target, in particular when the target's stimulus features incidentally gave rise to efficient motion streaks. These results suggest that intrasaccadic visual information informs the establishment of object correspondence and jump-starts gaze correction.

Object-mediated overwriting across saccades

How are visual sensory representations that are acquired peripherally from a saccade target related to sensory representations generated foveally after the saccade? We tested the hypothesis that, when the two representations are perceived to belong to the same object, the post-saccadic value tends to overwrite the pre-saccadic value. Participants executed a saccade to a colored target object, which sometimes changed during the saccade by ±15°, 30°, or 45° in color space. They were post-cued to report either the pre-saccadic or post-saccadic color in a continuous report procedure. Substantial overwriting of the pre-saccadic color by the post-saccadic color was observed. Moreover, the introduction of a brief post-saccadic blank interval (which disrupted the perception of object correspondence) led to a substantial reduction in overwriting. The results provide the first direct evidence for an object-mediated overwriting mechanism across saccades, in which post-saccadic values automatically replace pre-saccadic values.

Transsaccadic integration relies on a limited memory resource

Saccadic eye movements cause large-scale transformations of the image falling on the retina. Rather than starting visual processing anew after each saccade, the visual system combines post-saccadic information with visual input from before the saccade. Crucially, the relative contribution of each source of information is weighted according to its precision, consistent with principles of optimal integration. We reasoned that, if pre-saccadic input is maintained in a resource-limited store, such as visual working memory, its precision will depend on the number of items stored, as well as their attentional priority. Observers estimated the color of stimuli that changed imperceptibly during a saccade, and we examined where reports fell on the continuum between pre- and post-saccadic values. Bias toward the post-saccadic color increased with the set size of the pre-saccadic display, consistent with an increased weighting of the post-saccadic input as precision of the pre-saccadic representation declined. In a second experiment, we investigated if transsaccadic memory resources are preferentially allocated to attentionally prioritized items. An arrow cue indicated one pre-saccadic item as more likely to be chosen for report. As predicted, valid cues increased response precision and biased responses toward the pre-saccadic color. We conclude that transsaccadic integration relies on a limited memory resource that is flexibly distributed between pre-saccadic stimuli.

A review of interactions between peripheral and foveal vision

Visual processing varies dramatically across the visual field. These differences start in the retina and continue all the way to the visual cortex. Despite these differences in processing, the perceptual experience of humans is remarkably stable and continuous across the visual field. Research in the last decade has shown that processing in peripheral and foveal vision is not independent, but is more directly connected than previously thought. We address three core questions on how peripheral and foveal vision interact, and review recent findings on potentially related phenomena that could provide answers to these questions. First, how is the processing of peripheral and foveal signals related during fixation? Peripheral signals seem to be processed in foveal retinotopic areas to facilitate peripheral object recognition, and foveal information seems to be extrapolated toward the periphery to generate a homogeneous representation of the environment. Second, how are peripheral and foveal signals re-calibrated? Transsaccadic changes in object features lead to a reduction in the discrepancy between peripheral and foveal appearance. Third, how is peripheral and foveal information stitched together across saccades? Peripheral and foveal signals are integrated across saccadic eye movements to average percepts and to reduce uncertainty. Together, these findings illustrate that peripheral and foveal processing are closely connected, mastering the compromise between a large peripheral visual field and high resolution at the fovea.

Stimulus blanking reveals contrast-dependent transsaccadic feature transfer

Across saccadic eye movements, the visual system receives two successive static images corresponding to the pre- and the postsaccadic projections of the visual field on the retina. The existence of a mechanism integrating the content of these images is today still a matter of debate. Here, we studied the transfer of a visual feature across saccades using a blanking paradigm. Participants moved their eyes to a peripheral grating and discriminated a change in its orientation occurring during the eye movement. The grating was either constantly on the screen or briefly blanked during and after the saccade. Moreover, it either was of the same luminance as the background (i.e., isoluminant) or anisoluminant with respect to it. We found that for anisoluminant gratings, the orientation discrimination across saccades was improved when a blank followed the onset of the eye movement. Such effect was however abolished with isoluminant gratings. Additionally, performance was also improved when an anisoluminant grating presented before the saccade was followed by an isoluminant one. These results demonstrate that a detailed representation of the presaccadic image was transferred across saccades allowing participants to perform better on the transsaccadic orientation task. While such a transfer of visual orientation across saccade is masked in real-life anisoluminant conditions, the use of a blank and of an isoluminant postsaccadic grating allowed to reveal its existence.

Visual working memory and action: Functional links and bi-directional influences

Working memory bridges perception to action over extended delays, enabling flexible goal-directed behaviour. To date, studies of visual working memory – concerned with detailed visual representations such as shape and colour – have considered visual memory predominantly in the context of visual task demands, such as visual identification and search. Another key purpose of visual working memory is to directly inform and guide upcoming actions. Taking this as a starting point, I review emerging evidence for the pervasive bi-directional links between visual working memory and (planned) action, and discuss these links from the perspective of their common goal of enabling flexible and precise behaviour.

Feature-based guidance of attention during post-saccadic selection

Current models of trans-saccadic perception propose that, after a saccade, the saccade target object must be localized among objects near the landing position. However, the nature of the attentional mechanisms supporting this process is currently under debate. In the present study, we tested whether surface properties of the saccade target object automatically bias post-saccadic selection using a variant of the visual search task. Participants executed a saccade to a shape-singleton target in a circular array. During this primary saccade, the array sometimes rotated so that the eyes landed between the target and an adjacent distractor, requiring gaze correction. In addition, each object in the array had an incidental color value. On Switch trials, the target and adjacent distractor switched colors. The accuracy and latency of gaze correction to the target (measures that provide a direct index of target localization) were compared with a control condition in which no color switch occurred (No-switch trials). Gaze correction to the target was substantially impaired in the Switch condition. This result was obtained even when participants had substantial incentive to avoid encoding the color of the saccade target. In addition, similar effects were observed when the roles of the two feature dimensions (color and shape) were reversed. The results indicate that saccade target features are automatically encoded before a saccade, are retained in visual working memory across the saccade, and instantiate a feature-based selection operation when the eyes land, biasing attention toward objects that match target features.

Transsaccadic integration benefits are not limited to the saccade target

Across saccades, humans can integrate the low-resolution presaccadic information of an upcoming saccade target with the high-resolution postsaccadic information. There is converging evidence to suggest that transsaccadic integration occurs at the saccade target. However, given divergent evidence on the spatial specificity of related mechanisms such as attention, visual working memory, and remapping, it is unclear whether integration is also possible at locations other than the saccade target. We tested the spatial profile of transsaccadic integration, by testing perceptual performance at six locations around the saccade target and between the saccade target and initial fixation. Results show that integration benefits do not differ between the saccade target and surrounding locations. Transsaccadic integration benefits are not specific to the saccade target and can occur at other locations when they are behaviorally relevant, although there is a trend for worse performance for the location above initial fixation compared with those in the direction of the saccade. This suggests that transsaccadic integration may be a more general mechanism used to reconcile task-relevant pre- and postsaccadic information at attended locations other than the saccade target.

NEW & NOTEWORTHY This study shows that integration of pre- and postsaccadic information across saccades is not restricted to the saccade target. We found performance benefits of transsaccadic integration at attended locations other than the saccade target, and these benefits did not differ from those found at the saccade target. This suggests that transsaccadic integration may be a more general mechanism used to reconcile pre- and postsaccadic information at task-relevant locations.