Short-term visual memory for location in depth: A U-shaped function of time

Visual short-term memory for crossed and uncrossed binocular disparities

Previous work on visual short-term memory (VSTM) has encompassed various stimulus attributes including spatial frequency, color, and contrast, revealing specific time courses and a dependence on stimulus parameters. This study investigates visual short-term memory for binocular depth, using dynamic random dot stereograms (DRDS) featuring disparity planes in front of or behind the plane of fixation. In a delayed match-to-sample paradigm, we employed four distinct reference disparities (17.5', 28.8' either crossed or uncrossed) at two contrast levels (20%, 80%), spanning interstimulus intervals (ISI) of up to 4 s. Test stimuli represented a range of equally spaced values centered around the reference disparity of the ongoing trial. In addition, the impact of a memory masking stimulus was also tested in a separate experiment. Accuracy and point of subjective equality (PSE) served as performance markers. The performance, indicated by the accuracy of responses, was better for smaller reference disparities (±17.5') compared to larger ones (±28'), but both deteriorated as a function of ISI. The PSE demonstrated a consistent shift with increasing ISIs, irrespective of the magnitude of the initial disparity, converging gradually toward the range of 20-22' and deviating from the reference disparity. Notably, the influence of masking stimuli on the PSE was more marked when the mask disparity diverged from the reference value. The findings from our study indicate that the retention of absolute disparity in memory is imprecise, it deteriorates with retention time or due to perturbation by dissimilar masking stimuli. As a result, the memory trace is gradually replaced by a default depth value. This value could potentially signify an optimal point within low-level perceptual memory, however, our results are better explained by perceptual averaging whereby the visual system computationally derives a statistical summary of the presented disparities over time. The latter mechanism would aid in the computation of relative disparity in a dynamically changing environment.

The influence of depth on object selection and manipulation in visual working memory within a 3D context

Recent studies have examined whether the internal selection mechanism functions similarly for perception and visual working memory (VWM). However, the process of how we access and manipulate object representations distributed in a 3D space remains unclear. In this study, we utilized a memory search task to investigate the effect of depth on object selection and manipulation within VWM. The memory display consisted of colored items half positioned at the near depth plane and the other half at the far plane. During memory maintenance, the participants were instructed to search for a target representation and update its color. The results showed that under object-based attention (Experiments 1, 3, and 5), the update time was faster for targets at the near plane than for those at the far plane. This effect was absent in VWM when deploying spatial attention (Experiment 2) and in visual search regardless of the type of attention deployed (Experiment 4). The differential effects of depth on spatial and object-based attention in VWM suggest that spatial attention primarily relied on 2D location information irrespective of depth, whereas object-based attention seemed to prioritize memory representations at the front plane before shifting to the back. Our findings shed light on the interaction between depth perception and the selection mechanisms within VWM in a 3D context, emphasizing the importance of ordinal, rather than metric, spatial information in guiding object-based attention in VWM.

Spatial attention based on 2D location and relative depth order modulates visual working memory in a 3D environment

The attentional effect on visual working memory (VWM) has been a heated research topic in the past two decades. Studies show that VWM performance for an attended memory item can be improved by cueing its two-dimensional (2D) spatial location during retention. However, few studies have investigated the effect of attentional selection on VWM in a three-dimensional setting, and it remains unknown whether depth information can produce beneficial attentional effects on 2D visual representations similar to 2D spatial information. Here we conducted four experiments, displaying memory items at various stereoscopic depth planes, and examined the retro-cue effects of four types of cues - a cue would either indicate the 2D or depth location of a memory item, and either in the form of physical (directly pointing to a location) or symbolic (numerically mapping onto a location) cues. We found that retro-cue benefits were only observed for cues directly pointing to a 2D location, whereas a null effect was observed for cues directly pointing to a depth location. However, there was a retro-cue effect when cueing the relative depth order, though the effect was weaker than that for cueing the 2D location. The selective effect on VWM based on 2D spatial attention is different from depth-based attention, and the divergence suggests that an object representation is primarily bound with its 2D spatial location, weakly bound with its depth order but not with its metric depth location. This indicates that attentional selection based on memory for depth, particularly metric depth, is ineffective.

The role of ensemble average differs in working memory for depth and planar information

The representation of individual planar locations and features stored in working memory can be affected by the average representation. However, less is known about how the average representation affects the short-term storage of depth information. To evaluate the possible different roles of the ensemble average in working memory for planar and depth information, we used mathematical models to fit the data collected from one study on working memory for depth and 12 studies on working memory for planar information. The pattern of recalled depth was well captured by models assuming that there was a probability of reporting the average depth instead of the individual depth, compressing the recalled front-back distance of the stimulus ensemble compared to the perceived distance. However, when modeling the recalled planar information, we found that participants tended to report individual nontarget features when the target was not memorized, and the assumption of reporting average information improves the data fitting only in very few studies. These results provide evidence for our hypothesis that average depth information can be used as a substitution for individual depth information stored in working memory, but for planar visual features, the substitution of target with the average works under a constraint that the average of to-be-remembered features is readily accessible.

The Short-Term Retention of Depth

We review research on the visual working memory for information portrayed by items arranged in depth (i.e., distance to the observer) within peri-personal space. Most items lose their metric depths within half a second, even though their identities and spatial positions are retained. The paradoxical loss of depth information may arise because visual working memory retains the depth of a single object for the purpose of actions such as pointing or grasping which usually apply to only one thing at a time.

Increasing perceptual separateness affects working memory for depth - re-allocation of attention from boundaries to the fixated center

For decades, working memory (WM) has been a heated research topic in the field of cognitive psychology. However, most studies on WM presented visual stimuli on a two-dimensional plane, rarely involving depth perception. Several previous studies have investigated how depth information is stored in WM, and found that WM for depth is even more limited in capacity and the memory performance is poor compared to visual WM. In the present study, we used a change detection task to investigate whether dissociating memory items by different visual features, thereby to increase their perceptual separateness, can improve WM performance for depth. Memory items presented at various depth planes were bound with different colors (Experiments 1 and 3) or sizes (Experiment 2). The memory performance for depth locations of visual stimuli with homogeneous and heterogeneous appearances were tested and compared. The results showed a consistent pattern that although separating items with various feature values did not affect the overall memory performance, the manipulation significantly improved memory performance for the middle depth locations but impaired the performance for the boundary locations when observers fixated at the center of the whole depth volume. The memory benefits of feature separation can be attributed to enhanced individuation of memory items, therefore facilitating a more balanced allocation of attention and memory resources.

Effect of attentional selection on working memory for depth in a retro-cueing paradigm

Recent studies have shown that the temporary storage and manipulation of depth information (working memory for depth; WMd) is largely different from that of visual information in a 2D context (visual working memory; VWM). Although there has been abundant evidence on VWM showing that cueing a memory item during retention could bias attention to its internal representation and thus improves its memory performance (a retro-cue effect), it is unknown whether such an effect differs for WMd that is nested in a 3D context compared with that in a conventional 2D context. Here, we used a change detection task to investigate the effect of attentional selection on WMd by testing several types of retro-cue. The memory array consisted of items positioned at various stereoscopic depth planes, and a cue was presented during retention. Participants needed to make judgments on whether the depth position of target (one memory item) had changed. Our study showed reliable valid retro-cue benefits but no invalid retro-cue cost, indicating that the relational information may be registered in WMd to prevent a strategical removal of the unattended item. There was also a slight improvement in memory performance for cueing depth order compared with that for cueing other feature dimensions or 2D locations. The attentional effect on memory representation in a 3D context is different from that in a 2D context, and the divergence may suggest the distinctive nature of working memory for depth.

Relation matters: relative depth order is stored in working memory for depth

Working memory is considered as a cognitive memory buffer for temporarily holding, processing, and manipulating information. Although working memory for verbal and visual information has been studied extensively in the past literature, few studies have systematically investigated how depth information is stored in working memory. Here, we show that the memory performance for detecting changes in stereoscopic depth is low when there is no change in relative depth order, and the performance is reliably better when depth order is changed. Increasing the magnitude of change only improves memory performance when depth order is kept constant. However, if depth order is changed, the performance remains high, even with a small change magnitude. Our findings suggest that relative depth order is a better indicator for working memory performance than absolute metric depth. The memory representation for individual depth is not independent, but inherently relational, revealing a fundamental organizing principle for depth information in the visual system.

Overestimation and contraction biases of depth information stored in working memory depend on spatial configuration

Depth perception is essential for effective interaction with the environment. Although the accuracy of depth perception has been studied extensively, it is unclear how accurate the depth information is stored in working memory. In this study, we investigated the accuracy and systematic biases of depth representation by a delayed estimation task. The memory array consisted of items presented at various stereoscopic depth positions, and the participants were instructed to estimate the depth position of one target item after a retention interval. We examined the effect of spatial configuration by comparing the memory performance in the whole-display condition where non-target memory items were present during retrieval with that in the single-display condition where non-target memory items were absent. In the single-display condition, we found an overestimation bias that the depth estimates were farther than the corresponding depth positions defined by disparity, and a contraction bias that the stored depth positions near the observer were overestimated and those far from the observer were underestimated. The magnitude of these biases increased with the number of to-be-stored items. However, in the whole-display condition, the overestimation bias was corrected and the contraction bias did not increase with the number of to-be-stored items. Our findings suggested that the number of to-be-stored items could affect the accuracy of depth working memory, and its effect depended crucially on whether the information of spatial configuration of memory display was available at the retrieval stage.

Working memory for stereoscopic depth is limited and imprecise-evidence from a change detection task

Most studies on visual working memory (VWM) and spatial working memory (SWM) have employed visual stimuli presented at the fronto-parallel plane and few have involved depth perception. VWM is often considered as a memory buffer for temporarily holding and manipulating visual information that relates to visual features of an object, and SWM for holding and manipulating spatial information that concerns the spatial location of an object. Although previous research has investigated the effect of stereoscopic depth on VWM, the question of how depth positions are stored in working memory has not been systematically investigated, leaving gaps in the existing literature on working memory. Here, we explore working memory for depth by using a change detection task. The memory items were presented at various stereoscopic depth planes perpendicular to the line of sight, with one item per depth plane. Participants were asked to make judgments on whether the depth position of the target (one of the memory items) had changed. The results showed a conservative response bias that observers tended to make 'no change' responses when detecting changes in depth. In addition, we found that similar to VWM, the change detection accuracy degraded with the number of memory items presented, but the accuracy was much lower than that reported for VWM, suggesting that the storage for depth information is severely limited and less precise than that for visual information. The detection sensitivity was higher for the nearest and farthest depths and was better when the probe was presented along with the other items originally in the memory array, indicating that how well the to-be-stored depth can be stored in working memory depends on its relation with the other depth positions.

Saturation and brightness modulate the effect of depth on visual working memory

Although previous studies show inconsistent results regarding the effect of depth perception on visual working memory (VWM), a recent finding shows that perceptually closer-in-depth items are better remembered than farther items when combining the congruent disparity and relative size cues. In this study, we employed a similar change detection paradigm to investigate the effects of saturation and brightness, alone or in combination with binocular disparity, on VWM. By varying the appearance of the memory items, we aimed to manipulate the visual salience as well as to simulate the aerial perspective cue that induces depth perception. We found that the change detection accuracy was significantly improved for brighter and more saturated items, but not for items solely with higher saturation. Additionally, combining saturation with the congruent disparity cue significantly improved memory performance for perceptually closer items over farther items. Conflicting the disparity cue with saturation eliminated the memory benefit for the closer items. These results indicate that saturation and brightness could modulate the effect of depth on VWM, and both visual salience and depth perception affect VWM possibly through a common underlying mechanism of setting priority for attentional selection.