Spatial and temporal properties of stereoscopic surface interpolation

The Veiled Virgin illustrates visual segmentation of shape by cause

How the brain reconstructs three-dimensional object shape from two-dimensional retinal light patterns remains a mystery. Most research has investigated how cues—such as shading, texture, or perspective—help us estimate visible surface points on the outside of objects. However, our findings show the brain achieves much more than this. Observers not only infer the visible outer surface but also the hidden internal structure of objects—seeing “beneath the skin.” Our findings suggest the brain parses shapes’ features according to their physical causes, potentially allowing us to separate a single continuous surface into multiple superimposed depth layers. This ability likely aids our interactions with objects, by indicating which surface locations are firmly supported from the inside and thus suitable for grasping.

Three-dimensional (3D) shape perception is one of the most important functions of vision. It is crucial for many tasks, from object recognition to tool use, and yet how the brain represents shape remains poorly understood. Most theories focus on purely geometrical computations (e.g., estimating depths, curvatures, symmetries). Here, however, we find that shape perception also involves sophisticated inferences that parse shapes into features with distinct causal origins. Inspired by marble sculptures such as Strazza’s The Veiled Virgin (1850), which vividly depict figures swathed in cloth, we created composite shapes by wrapping unfamiliar forms in textile, so that the observable surface relief was the result of complex interactions between the underlying object and overlying fabric. Making sense of such structures requires segmenting the shape based on their causes, to distinguish whether lumps and ridges are due to the shrouded object or to the ripples and folds of the overlying cloth. Three-dimensional scans of the objects with and without the textile provided ground-truth measures of the true physical surface reliefs, against which observers’ judgments could be compared. In a virtual painting task, participants indicated which surface ridges appeared to be caused by the hidden object and which were due to the drapery. In another experiment, participants indicated the perceived depth profile of both surface layers. Their responses reveal that they can robustly distinguish features belonging to the textile from those due to the underlying object. Together, these findings reveal the operation of visual shape-segmentation processes that parse shapes based on their causal origin.

Depth perception in disparity-defined objects: finding the balance between averaging and segregation

Deciding what constitutes an object, and what background, is an essential task for the visual system. This presents a conundrum: averaging over the visual scene is required to obtain a precise signal for object segregation, but segregation is required to define the region over which averaging should take place. Depth, obtained via binocular disparity (the differences between two eyes’ views), could help with segregation by enabling identification of object and background via differences in depth. Here, we explore depth perception in disparity-defined objects. We show that a simple object segregation rule, followed by averaging over that segregated area, can account for depth estimation errors. To do this, we compared objects with smoothly varying depth edges to those with sharp depth edges, and found that perceived peak depth was reduced for the former. A computational model used a rule based on object shape to segregate and average over a central portion of the object, and was able to emulate the reduction in perceived depth. We also demonstrated that the segregated area is not predefined but is dependent on the object shape. We discuss how this segregation strategy could be employed by animals seeking to deter binocular predators.

This article is part of the themed issue ‘Vision in our three-dimensional world’.

Variation of stereothreshold with random-dot stereogram density

PURPOSE

Reports of dissimilar stereothresholds for contour and random-dot (RD) targets may reflect differences in stimulus properties or differences between local and global stereoscopic processing mechanisms. In this study, we evaluated whether the stereothresholds obtained using low- and high-density RD stimuli are consistent with a distinction between local and global disparity processing.

METHODS

Stereothresholds were measured in eight normal subjects for a small disparate line segment superimposed on RD surrounds with densities that ranged between 0.07 and 28.3%.

RESULTS

Stereothresholds averaged 0.23 arc min for an RD density of 0.39% and approximately doubled for lower and higher densities. The increase in stereothresholds at low densities is likely because of the increased spacing between elements, which reduces their usefulness as a reference for relative disparity judgments. The increase in stereothresholds at high densities is attributed to a crowding effect.

CONCLUSIONS

Because the stereothresholds measured with RD stimuli of low and high density are limited by different constraints, they can be considered to be different types of stereotargets. However, because the stereothresholds measured for RD targets of varying densities are similar to those determined previously for a local, two-rod stereotarget, it is likely that all these stimuli are processed by a single disparity-processing mechanism.

Perceptual asymmetry reveals neural substrates underlying stereoscopic transparency

We describe a perceptual asymmetry found in stereoscopic perception of overlaid random-dot surfaces. Specifically, the minimum separation in depth needed to perceptually segregate two overlaid surfaces depended on the distribution of dots across the surfaces. With the total dot density fixed, significantly larger inter-plane disparities were required for perceptual segregation of the surfaces when the front surface had fewer dots than the back surface compared to when the back surface was the one with fewer dots. We propose that our results reflect an asymmetry in the signal strength of the front and back surfaces due to the assignment of the spaces between the dots to the back surface by disparity interpolation. This hypothesis was supported by the results of two experiments designed to reduce the imbalance in the neuronal response to the two surfaces. We modeled the psychophysical data with a network of inter-neural connections: excitatory within-disparity and inhibitory across disparity, where the spread of disparity was modulated according to figure-ground assignment. These psychophysical and computational findings suggest that stereoscopic transparency depends on both inter-neural interactions of disparity-tuned cells and higher-level processes governing figure ground segregation.

An investigation of surface reconstruction from binocular disparity based on standard regularization theory: comparison between "membrane" and "thin-plate" potential energy models

Visible surfaces of three-dimensional objects are reconstructed from two-dimensional retinal images in the early stages of human visual processing. In the computational model of surface reconstruction based on the standard regularization theory, an energy function is minimized. Two types of model have been proposed, called "membrane" and "thin-plate" after their function formulas, in which the first or the second derivative of depth information is used. In this study, the threshold of surface reconstruction from binocular disparity was investigated using a sparse random dot stereogram, and the predictive accuracy of these models was evaluated. It was found that the thin-plate model reconstructed surfaces more accurately than the membrane model and showed good agreement with experimental results. The likelihood that these models imitate human processing of visual information is discussed in terms of the size of receptive fields in the visual pathways of the human cortex.

Depth propagation and surface construction in 3-D vision

In stereo vision, regions with ambiguous or unspecified disparity can acquire perceived depth from unambiguous regions. This has been called stereo capture, depth interpolation or surface completion. We studied some striking induced depth effects suggesting that depth interpolation and surface completion are distinct stages of visual processing. An inducing texture (2-D Gaussian noise) had sinusoidal modulation of disparity, creating a smooth horizontal corrugation. The central region of this surface was replaced by various test patterns whose perceived corrugation was measured. When the test image was horizontal 1-D noise, shown to one eye or to both eyes without disparity, it appeared corrugated in much the same way as the disparity-modulated (DM) flanking regions. But when the test image was 2-D noise, or vertical 1-D noise, little or no depth was induced. This suggests that horizontal orientation was a key factor. For a horizontal sine-wave luminance grating, strong depth was induced, but for a square-wave grating, depth was induced only when its edges were aligned with the peaks and troughs of the DM flanking surface. These and related results suggest that disparity (or local depth) propagates along horizontal 1-D features, and then a 3-D surface is constructed from the depth samples acquired. The shape of the constructed surface can be different from the inducer, and so surface construction appears to operate on the results of a more local depth propagation process.