Is visual short-term memory depthful?

Three-dimensional (3D) stimuli are always better than two-dimensional (2D) multi-tasking? A high cognitive load in 3D-MATB-II

OBJECTIVE

The objective of this study is to investigate the whether multi-tasking performance in (three-dimensional) 3D aid or impede cognition compare to (two-dimensional) 2D environments, as reflected by cognitive load. Specifically, we aim to examine the mechanism of multi-tasking under 3D (virtual reality [VR]) and 2D (PC monitor) conditions using the widely used Multi-Attribute Task Battery (MATB) II paradigm.

METHODOLOGY

The MATB-II sub-tasks, namely "Tracking" and "System Monitoring," were conducted with varying task demands in both 3D conditions (Tracking Far - System Monitoring Near [TF-SN], Tracking Near - System Monitoring Far [TN-SF]) and a 2D condition with no depth perception (No Depth [ND]). Participants' cognitive load was assessed using subjective reporting (NASA-TLX) and physiological measure (root mean square of successive difference (RMSSD) of heart rate variability (HRV)).

RESULTS

The study found that performance was significantly better in the ND condition compared to the TF-SN and TN-SF conditions. Furthermore, higher NASA-TLX scores and lower RMSSD values were observed in the TF-SN and TN-SN conditions compared to the ND condition, providing additional support for the overall findings of the MATB-II paradigm.

CONCLUSION

These findings suggest that processing multiple tasks in different depth planes may lead to poorer performance and increased subjective and physiological cognitive load.

Visual short-term memory for crossed and uncrossed binocular disparities

Previous work on visual short-term memory (VSTM) has encompassed various stimulus attributes including spatial frequency, color, and contrast, revealing specific time courses and a dependence on stimulus parameters. This study investigates visual short-term memory for binocular depth, using dynamic random dot stereograms (DRDS) featuring disparity planes in front of or behind the plane of fixation. In a delayed match-to-sample paradigm, we employed four distinct reference disparities (17.5', 28.8' either crossed or uncrossed) at two contrast levels (20%, 80%), spanning interstimulus intervals (ISI) of up to 4 s. Test stimuli represented a range of equally spaced values centered around the reference disparity of the ongoing trial. In addition, the impact of a memory masking stimulus was also tested in a separate experiment. Accuracy and point of subjective equality (PSE) served as performance markers. The performance, indicated by the accuracy of responses, was better for smaller reference disparities (±17.5') compared to larger ones (±28'), but both deteriorated as a function of ISI. The PSE demonstrated a consistent shift with increasing ISIs, irrespective of the magnitude of the initial disparity, converging gradually toward the range of 20-22' and deviating from the reference disparity. Notably, the influence of masking stimuli on the PSE was more marked when the mask disparity diverged from the reference value. The findings from our study indicate that the retention of absolute disparity in memory is imprecise, it deteriorates with retention time or due to perturbation by dissimilar masking stimuli. As a result, the memory trace is gradually replaced by a default depth value. This value could potentially signify an optimal point within low-level perceptual memory, however, our results are better explained by perceptual averaging whereby the visual system computationally derives a statistical summary of the presented disparities over time. The latter mechanism would aid in the computation of relative disparity in a dynamically changing environment.

The depth of executive function: Depth information aids executive function under challenging task conditions

The present studies investigated how three core aspects of executive functioning may be influenced by the presence of depth information. Specifically, participants were assigned to one of three executive functioning tasks: working memory (i.e., a change detection task), selective attention (i.e., a visual search task), or inhibitory control (i.e., a flanker task). For all three tasks, participants completed trials where the items in the display were presented either all in one depth plane or the target item was isolated in depth. For the working memory and selective attention tasks, there was an additional condition where items were evenly distributed across two depth planes. Each task also had multiple levels of difficulty to explore if task conditions influence the effect of depth information. Results indicated that although depth information can improve both working memory and selective attention performance, this benefit is specific to the task difficulty and depth information can even hinder performance under certain circumstances. Depth information did not appear to influence inhibitory control performance. Future work is required to investigate if depth can improve inhibitory control performance, and how/what task conditions influence the benefit of depth information. Until further research is completed, researchers and designers should be cautious when implementing multidimensional (3D) displays, as it remains unclear if the performance benefits of including depth information outweigh the present costs.

The Short-Term Retention of Depth

We review research on the visual working memory for information portrayed by items arranged in depth (i.e., distance to the observer) within peri-personal space. Most items lose their metric depths within half a second, even though their identities and spatial positions are retained. The paradoxical loss of depth information may arise because visual working memory retains the depth of a single object for the purpose of actions such as pointing or grasping which usually apply to only one thing at a time.

Effect of attentional selection on working memory for depth in a retro-cueing paradigm

Recent studies have shown that the temporary storage and manipulation of depth information (working memory for depth; WMd) is largely different from that of visual information in a 2D context (visual working memory; VWM). Although there has been abundant evidence on VWM showing that cueing a memory item during retention could bias attention to its internal representation and thus improves its memory performance (a retro-cue effect), it is unknown whether such an effect differs for WMd that is nested in a 3D context compared with that in a conventional 2D context. Here, we used a change detection task to investigate the effect of attentional selection on WMd by testing several types of retro-cue. The memory array consisted of items positioned at various stereoscopic depth planes, and a cue was presented during retention. Participants needed to make judgments on whether the depth position of target (one memory item) had changed. Our study showed reliable valid retro-cue benefits but no invalid retro-cue cost, indicating that the relational information may be registered in WMd to prevent a strategical removal of the unattended item. There was also a slight improvement in memory performance for cueing depth order compared with that for cueing other feature dimensions or 2D locations. The attentional effect on memory representation in a 3D context is different from that in a 2D context, and the divergence may suggest the distinctive nature of working memory for depth.

Working memory for stereoscopic depth is limited and imprecise-evidence from a change detection task

Most studies on visual working memory (VWM) and spatial working memory (SWM) have employed visual stimuli presented at the fronto-parallel plane and few have involved depth perception. VWM is often considered as a memory buffer for temporarily holding and manipulating visual information that relates to visual features of an object, and SWM for holding and manipulating spatial information that concerns the spatial location of an object. Although previous research has investigated the effect of stereoscopic depth on VWM, the question of how depth positions are stored in working memory has not been systematically investigated, leaving gaps in the existing literature on working memory. Here, we explore working memory for depth by using a change detection task. The memory items were presented at various stereoscopic depth planes perpendicular to the line of sight, with one item per depth plane. Participants were asked to make judgments on whether the depth position of the target (one of the memory items) had changed. The results showed a conservative response bias that observers tended to make 'no change' responses when detecting changes in depth. In addition, we found that similar to VWM, the change detection accuracy degraded with the number of memory items presented, but the accuracy was much lower than that reported for VWM, suggesting that the storage for depth information is severely limited and less precise than that for visual information. The detection sensitivity was higher for the nearest and farthest depths and was better when the probe was presented along with the other items originally in the memory array, indicating that how well the to-be-stored depth can be stored in working memory depends on its relation with the other depth positions.

Saturation and brightness modulate the effect of depth on visual working memory

Although previous studies show inconsistent results regarding the effect of depth perception on visual working memory (VWM), a recent finding shows that perceptually closer-in-depth items are better remembered than farther items when combining the congruent disparity and relative size cues. In this study, we employed a similar change detection paradigm to investigate the effects of saturation and brightness, alone or in combination with binocular disparity, on VWM. By varying the appearance of the memory items, we aimed to manipulate the visual salience as well as to simulate the aerial perspective cue that induces depth perception. We found that the change detection accuracy was significantly improved for brighter and more saturated items, but not for items solely with higher saturation. Additionally, combining saturation with the congruent disparity cue significantly improved memory performance for perceptually closer items over farther items. Conflicting the disparity cue with saturation eliminated the memory benefit for the closer items. These results indicate that saturation and brightness could modulate the effect of depth on VWM, and both visual salience and depth perception affect VWM possibly through a common underlying mechanism of setting priority for attentional selection.

Depth benefits now loading: Visual working memory capacity and benefits in 3-D

The present studies explored how performance in multidimensional displays varies as a function of visual working memory load, item distribution across depths, and individual capacity differences. In Experiment 1, the benefit of depth information (one depth vs. two depths) was examined across seven set sizes within a change-detection paradigm. Multiple depth planes engendered performance benefits with five items, but elicited performance decrements with three items. These effects were associated with working memory capacity, such that benefits were only observed when the working memory load exceeded an individual's max capacity. Experiment 2 evaluated how the distribution of items in depth aids working memory performance. Equal distribution of items across depths produced higher accuracy compared with when the target was isolated in depth. Lastly, Experiment 3 explored how differences in working memory capacity affect an individual's ability to use depth information to improve their performance. The results indicate that both low-capacity and high-capacity individuals can benefit from depth information, but this may vary as a function of working memory load. Overall, the results indicate that multidimensional displays can improve performance with sufficient working memory load, possibly through some sort of depth tag.

Separating memoranda in depth increases visual working memory performance

Visual working memory is the mechanism supporting the continued maintenance of information after sensory inputs are removed. Although the capacity of visual working memory is limited, memoranda that are spaced farther apart on a 2-D display are easier to remember, potentially because neural representations are more distinct within retinotopically organized areas of visual cortex during memory encoding, maintenance, or retrieval. The impact on memory of spatial separability in depth is less clear, even though depth information is essential to guiding interactions with objects in the environment. On one account, separating memoranda in depth may facilitate performance if interference between items is reduced. However, depth information must be inferred indirectly from the 2-D retinal image, and less is known about how visual cortex represents depth. Thus, an alternative possibility is that separation in depth does not attenuate between-items interference; it may even impair performance, as attention must be distributed across a larger volume of 3-D space. We tested these alternatives using a stereo display while participants remembered the colors of stimuli presented either near or far in the 2-D plane or in depth. Increasing separation in-plane and in depth both enhanced performance. Furthermore, participants who were better able to utilize stereo depth cues showed larger benefits when memoranda were separated in depth, particularly for large memory arrays. The observation that spatial separation in the inferred 3-D structure of the environment improves memory performance, as is the case in 2-D environments, suggests that separating memoranda in depth might reduce neural competition by utilizing cortically separable resources.

Short-term visual memory for location in depth: A U-shaped function of time

Short-term visual memory was studied by displaying arrays of four or five numerals, each numeral in its own depth plane, followed after various delays by an arrow cue shown in one of the depth planes. Subjects reported the numeral at the depth cued by the arrow. Accuracy fell with increasing cue delay for the first 500 ms or so, and then recovered almost fully. This dipping pattern contrasts with the usual iconic decay observed for memory traces. The dip occurred with or without a verbal or color-shape retention load on working memory. In contrast, accuracy did not change with delay when a tonal cue replaced the arrow cue. We hypothesized that information concerning the depths of the numerals decays over time in sensory memory, but that cued recall is aided later on by transfer to a visual memory specialized for depth. This transfer is sufficiently rapid with a tonal cue to compensate for the sensory decay, but it is slowed by the need to tag the arrow cue's depth relative to the depths of the numerals, exposing a dip when sensation has decayed and transfer is not yet complete. A model with a fixed rate of sensory decay and varied transfer rates across individuals captures the dip as well as the cue modality effect.

Evidence for the effect of depth on visual working memory

Visual working memory (VWM) is a cognitive memory buffer for temporarily holding, processing, and manipulating visual information. Previous studies have demonstrated mixed results of the effect of depth perception on VWM, with some showing a beneficial effect while others not. In this study, we employed an adapted change detection paradigm to investigate the effects of two depth cues, binocular disparity and relative size. The memory array consisted of a set of pseudo-randomly positioned colored items, and the task was to judge whether the test item was changed compared to the memory item after a retention interval. We found that presenting the items in stereoscopic depth alone hardly affected VWM performance. When combining the two coherent depth cues, a significant larger VWM capacity of the perceptually closer-in-depth items was observed than that of the farther items, but the capacity for the two-depth-planes condition was not significantly different from that for the one-plane condition. Conflicting the two depth cues resulted in cancelling the beneficial effect of presenting items at a closer depth plane. The results indicate that depth perception could affect VWM, and the visual system may have an advantage in maintaining closer-in-depth objects in working memory.