Spatial scale interactions in stereo sensitivity and the neural representation of binocular disparity

Coarse-to-fine interaction on perceived depth in compound grating

To encode binocular disparity, the visual system uses a pair of left eye and right eye bandpass filters with either a position or a phase offset between them. Such pairs are considered to exit at multiple scales to encode a wide range of disparity. However, local disparity measurements by bandpass mechanisms can be ambiguous, particularly when the actual disparity is larger than a half-cycle of the preferred spatial frequency of the filter, which often occurs in fine scales. In this study, we investigated whether the visual system uses a coarse-to-fine interaction to resolve this ambiguity at finer scales for depth estimation from disparity. The stimuli were stereo grating patches composed of a target and comparison patterns. The target patterns contained spatial frequencies of 1 and 4 cycles per degree (cpd). The phase disparity of the low-frequency component was 0° (at the horopter), -90° (uncrossed), or 90° (crossed), and that of the high-frequency components was changed independent of the low-frequency disparity, in the range between -90° (uncrossed) and 90° (crossed). The observers' task was to indicate whether the target appeared closer to the comparison pattern, which always shared the disparity with the low-frequency component of the target. Regardless of whether the comparison pattern was a 1-cpd + 4-cpd compound or a 1-cpd simple grating, the perceived depth order of the target and the comparison varied in accordance with the phase disparity of the high-frequency component of the target. This effect occurred not only when the low-frequency component was at the horopter, but also when it contained a large disparity corresponding to one cycle of the high-frequency component (±90°). Our findings suggest a coarse-to-fine interaction in multiscale disparity processing, in which the depth interpretation of the high-frequency changes based on the disparity of the low-frequency component.

ASTEROID: A New Clinical Stereotest on an Autostereo 3D Tablet

Purpose

To describe a new stereotest in the form of a game on an autostereoscopic tablet computer designed to be suitable for use in the eye clinic and present data on its reliability and the distribution of stereo thresholds in adults.

Methods

Test stimuli were four dynamic random-dot stereograms, one of which contained a disparate target. Feedback was given after each trial presentation. A Bayesian adaptive staircase adjusted target disparity. Threshold was estimated from the mean of the posterior distribution after 20 responses. Viewing distance was monitored via a forehead sticker viewed by the tablet's front camera, and screen parallax was adjusted dynamically so as to achieve the desired retinal disparity.

Results

The tablet must be viewed at a distance of greater than ∼35 cm to produce a good depth percept. Log thresholds were roughly normally distributed with a mean of 1.75 log₁₀ arcsec = 56 arcsec and SD of 0.34 log₁₀ arcsec = a factor of 2.2. The standard deviation agrees with previous studies, but ASTEROID thresholds are approximately 1.5 times higher than a similar stereotest on stereoscopic 3D TV or on Randot Preschool stereotests. Pearson correlation between successive tests in same observer was 0.80. Bland-Altman 95% limits of reliability were ±0.64 log₁₀ arcsec = a factor of 4.3, corresponding to an SD of 0.32 log₁₀ arcsec on individual threshold estimates. This is similar to other stereotests and close to the statistical limit for 20 responses.

Conclusions

ASTEROID is reliable, easy, and portable and thus well-suited for clinical stereoacuity measurements.

Translational Relevance

New 3D digital technology means that research-quality psychophysical measurement of stereoacuity is now feasible in the clinic.

Magnitude, precision, and realism of depth perception in stereoscopic vision

Our perception of depth is substantially enhanced by the fact that we have binocular vision. This provides us with more precise and accurate estimates of depth and an improved qualitative appreciation of the three-dimensional (3D) shapes and positions of objects. We assessed the link between these quantitative and qualitative aspects of 3D vision. Specifically, we wished to determine whether the realism of apparent depth from binocular cues is associated with the magnitude or precision of perceived depth and the degree of binocular fusion. We presented participants with stereograms containing randomly positioned circles and measured how the magnitude, realism, and precision of depth perception varied with the size of the disparities presented. We found that as the size of the disparity increased, the magnitude of perceived depth increased, while the precision with which observers could make depth discrimination judgments decreased. Beyond an initial increase, depth realism decreased with increasing disparity magnitude. This decrease occurred well below the disparity limit required to ensure comfortable viewing.

Stereo-Curvature Aftereffect Is Due to More Than Shape Curvature Adaptation

For stereo-curvature aftereffect (sCAE), there is no agreement on whether adaptation occurs at the disparity-specified stage, the percept-specified stage, or both. Additionally, it remains uncertain whether retinal-position-dependent sCAE can be induced by possible adaptation to disparity-specified sources. Our study aimed to investigate the dependency and processing levels of adaptation underlying sCAE using dynamic spherical adaptation stimuli with static fixation. Experiment I examined the dependency by dynamically altering the location or size of adaptation stimuli. Experiment 2 investigated the adaptation levels via three sub-experiments: Experiment 2.1 examined how eccentricity influenced adaptation strength using static adaptation stimuli with different eccentricities, Experiment 2.2 tested a hypothesis about adaptation to a percept-specified primitive shape index (PSI) using dynamic size change of adaptation stimuli, and Experiment 2.3 tested another hypothesis on adapting disparity-specified average disparity information (ADI) using dynamic PSI change of adaptation stimuli. The results showed retinal-position-dependent and scale-independent sCAE. In addition to a possible eccentricity effect, the retinal-position dependence can result from ADI adaptation while the scale independence can be attributed to PSI adaptation. Therefore, sCAE is caused by adaptation at both the disparity-specified and percept-specified stages. Additionally, sCAE endows two coexisting adaptation processes with one dependent on retinal position and one independent of retinal position.

Visual discomfort and depth-of-field

Visual discomfort has been reported for certain visual stimuli and under particular viewing conditions, such as stereoscopic viewing. In stereoscopic viewing, visual discomfort can be caused by a conflict between accommodation and convergence cues that may specify different distances in depth. Earlier research has shown that depth-of-field, which is the distance range in depth in the scene that is perceived to be sharp, influences both the perception of egocentric distance to the focal plane, and the distance range in depth between objects in the scene. Because depth-of-field may also be in conflict with convergence and the accommodative state of the eyes, we raised the question of whether depth-of-field affects discomfort when viewing stereoscopic photographs. The first experiment assessed whether discomfort increases when depth-of-field is in conflict with coherent accommodation-convergence cues to distance in depth. The second experiment assessed whether depth-of-field influences discomfort from a pre-existing accommodation-convergence conflict. Results showed no effect of depth-of-field on visual discomfort. These results suggest therefore that depth-of-field can be used as a cue to depth without inducing discomfort in the viewer, even when cue conflicts are large.

Blur and disparity are complementary cues to depth

Estimating depth from binocular disparity is extremely precise, and the cue does not depend on statistical regularities in the environment. Thus, disparity is commonly regarded as the best visual cue for determining 3D layout. But depth from disparity is only precise near where one is looking; it is quite imprecise elsewhere. Away from fixation, vision resorts to using other depth cues-e.g., linear perspective, familiar size, aerial perspective. But those cues depend on statistical regularities in the environment and are therefore not always reliable. Depth from defocus blur relies on fewer assumptions and has the same geometric constraints as disparity but different physiological constraints. Blur could in principle fill in the parts of visual space where disparity is imprecise. We tested this possibility with a depth-discrimination experiment. Disparity was more precise near fixation and blur was indeed more precise away from fixation. When both cues were available, observers relied on the more informative one. Blur appears to play an important, previously unrecognized role in depth perception. Our findings lead to a new hypothesis about the evolution of slit-shaped pupils and have implications for the design and implementation of stereo 3D displays.

Relationship between threshold and suprathreshold perception of position and stereoscopic depth

We seek to determine the relationship between threshold and suprathreshold perception for position offset and stereoscopic depth perception under conditions that elevate their respective thresholds. Two threshold-elevating conditions were used: (1) increasing the interline gap and (2) dioptric blur. Although increasing the interline gap increases position (Vernier) offset and stereoscopic disparity thresholds substantially, the perception of suprathreshold position offset and stereoscopic depth remains unchanged. Perception of suprathreshold position offset also remains unchanged when the Vernier threshold is elevated by dioptric blur. We show that such normalization of suprathreshold position offset can be attributed to the topographical-map-based encoding of position. On the other hand, dioptric blur increases the stereoscopic disparity thresholds and reduces the perceived suprathreshold stereoscopic depth, which can be accounted for by a disparity-computation model in which the activities of absolute disparity encoders are multiplied by a Gaussian weighting function that is centered on the horopter. Overall, the statement "equal suprathreshold perception occurs in threshold-elevated and unelevated conditions when the stimuli are equally above their corresponding thresholds" describes the results better than the statement "suprathreshold stimuli are perceived as equal when they are equal multiples of their respective threshold values."

Sensors for impossible stimuli may solve the stereo correspondence problem

One of the fundamental challenges of binocular vision is that objects project to different positions on the two retinas (binocular disparity). Neurons in visual cortex show two distinct types of tuning to disparity, position and phase disparity, which are the results of differences in receptive field location and profile, respectively. Here, we point out that phase disparity does not occur in natural images. Why, then, should the brain encode it? We propose that phase-disparity detectors help to work out which feature in the left eye corresponds to a given feature in the right. This correspondence problem is plagued by false matches: regions of the image that look similar, but do not correspond to the same object. We show that phase-disparity neurons tend to be more strongly activated by false matches. Thus, they may act as 'lie detectors', enabling the true correspondence to be deduced by a process of elimination.

The role of binocular stereopsis in monoptic depth perception

In his study of depth from monocular elements, Kaye (1978) [Kaye, M. (1978). Stereopsis without binocular correlation. Vision Research, 18(8), 1013-1022] reported that monocular stimuli, briefly presented to one eye in a stereoscopic display, generated reliable depth percepts. Here we replicate and extend Kaye's findings in an effort to identify the mechanism underlying the phenomenon. Our experiments show that the perception of depth is not a simple result of monocular local sign, for the percept of depth disappears when one eye is patched. In subsequent experiments we assess the possibility that the percept results from a very coarse stereoscopic match to either the centroid of the luminance distribution in the unstimulated eye or a simple match to the line of sight in the unstimulated eye. Our results consistently support the match-to-fovea account, and lead us to conclude that monoptic depth is a stereoscopic phenomenon.

Stereoscopic depth discrimination with contrast windowed stimuli

Depth discrimination with a shifted contrast window was compared to that with a fixed contrast window. Stereoscopic performance with the fixed window was limited to small disparities and varied with spatial frequency. Performance with the shifted window extended to larger disparities and was more similar for low and high spatial frequencies. The results depended upon window shape, indicating that edge blur is an important factor. Stereoscopic performance with shifted patterns was supported at disparities larger than a phase disparity model might predict, suggesting that a combination of position and phase disparity computations are used for the perception of stereoscopic depth.

Stereo sensitivity depends on stereo matching

Stereoacuity thresholds, measured with bar targets, rise as the absolute disparity of the bars is increased. One explanation for this rise is that, as the bars are moved away from the fixation plane, the stereo system uses coarser mechanisms to encode the bars' disparity; coarse mechanisms are insensitive to small changes in target disparity, resulting in higher thresholds. To test this explanation, we measured stereoacuity with a 6 degrees wide 3 cpd grating presented in a rectangular envelope. We varied the disparity of the grating and its edges (envelope) parametrically from 0 to 20 arcmin (i.e., through one full period). To force observers to make judgments based on carrier disparity, we then varied the interocular phase incrementally from trial-to-trial while keeping edge disparity fixed for a given block of trials. The pedestal phase disparity of the grating necessarily cycles through 360 degrees, back to zero disparity, as the edge disparity increases monotonically from 0 to 20 arcmin. Unlike mechanisms that respond to bars, the mechanism that responds to the interocular phase disparity of the grating should have the same sensitivity at 20 arcmin disparity (360 degrees of phase) as it has at zero disparity. So, if stereoacuity were determined by the most sensitive mechanism, thresholds should oscillate with the pedestal phase disparity. However, these gratings are perceived in depth at the disparity of their edges. If stereoacuity were instead determined by the stereo matching operations that generate perceived depth, thresholds should rise monotonically with increasing edge disparity. We found that the rise in grating thresholds with increasing edge disparity was monotonic and virtually identical to the rise in thresholds observed for bars. Stereoacuity is contingent on stereo matching.

Early computational processing in binocular vision and depth perception

Stereoscopic depth perception is a fascinating ability in its own right and also a useful model of perception. In recent years, considerable progress has been made in understanding the early cortical circuitry underlying this ability. Inputs from left and right eyes are first combined in primary visual cortex (V1), where many cells are tuned for binocular disparity. Although the observation of disparity tuning in V1, combined with psychophysical evidence that stereopsis must occur early in visual processing, led to initial suggestions that V1 was the neural correlate of stereoscopic depth perception, more recent work indicates that this must occur in higher visual areas. The firing of cells in V1 appears to depend relatively simply on the visual stimuli within local receptive fields in each retina, whereas the perception of depth reflects global properties of the stimulus. However, V1 neurons appear to be specialized in a number of respects to encode ecologically relevant binocular disparities. This suggests that they carry out essential pre-processing underlying stereoscopic depth perception in higher areas. This article reviews recent progress in developing accurate models of the computations carried out by these neurons. We seem close to achieving a mathematical description of the initial stages of the brain's stereo algorithm. This is important in itself--for instance, it may enable improved stereopsis in computer vision--and paves the way for a full understanding of how depth perception arises.

Temporal Dynamics of Binocular Disparity Processing in the Central Visual Pathway

To solve the stereo correspondence problem (i.e., find the matching features of a visual scene in both eyes), it is advantageous to combine information across spatial scales. The details of how this is accomplished are not clear. Psychophysical studies and mathematical models have suggested various types of interactions across spatial scale, including coarse to fine, fine to coarse, averaging, and population coding. In this study, we investigate dynamic changes in disparity tuning of simple and complex cells in the cat's striate cortex over a short time span. We find that disparity frequency increases and disparity ranges decrease while optimal disparity remains constant, and this conforms to a coarse-to-fine mechanism. We explore the origin of this mechanism by examining the frequency and size dynamics exhibited by binocular simple cells and neurons in the lateral geniculate nucleus (LGN). The results suggest a strong role for a feed-forward mechanism, which could originate in the retina. However, we find that the dynamic changes seen in the disparity range of simple cells cannot be predicted from their left and right eye monocular receptive field (RF) size changes. This discrepancy suggests the possibility of a dynamic nonlinearity or disparity specific feedback that alters tuning or a combination of both mechanisms.

Spatial-scale interaction in human stereoscopic vision in response to sustained and transient stimuli

We investigated temporal properties of stereopsis at different spatial scales in dynamic random-dot stereograms (DRS) consisting of (i) the repeated presentation of two image pairs (i.e. sustained presentation) and (ii) single presentations of two image pairs (transient presentation). In dense stereograms perception of depth is possible if the levels of interocular correlation are above a certain threshold. Therefore, detection of correlation is an important step in stereopsis. Across different experiments, we varied the level of interocular correlation as a function of time by alternating correlated and un/anticorrelated random-dot stereograms. Also, we varied their spatial scale (coarse, fine and alternating between coarse and fine). The presence of stereopsis was examined using a forced-choice depth detection task. For both sustained and transient presentations of the stimuli, the results show that (i) stereopsis has similar temporal properties at coarse and fine spatial scales; (ii) interaction between spatial scales depends on their relative sizes. The results indicate a strong inhibitory influence of rivalry at a coarse scale on stereopsis at a fine scale, and just a weak inhibitory influence of rivalry at a fine scale on stereopsis at a coarse scale. This study provides experimental evidence for a hierarchical organisation of spatial scales in human stereoscopic vision based on neural interaction instead of vergence eye movements.

Functional connectivity of disparity-tuned neurons in the visual cortex

Different mechanisms have been proposed concerning how disparity-tuned neurons might be connected to produce the signals for depth perception. Here we present neurophysiological evidence providing insight on this issue. We have recorded simultaneously from pairs of disparity-tuned neurons in the cat's striate cortex. The purpose was to determine the relationships between disparity tuning and functional connectivity revealed through neural cross-correlograms. Monosynaptic connections tend to be stronger between pairs of cells with similar disparity tuning. Pairs of complex cells tend to have either similar tuning or nearly opposite tuning with an absence of quadrature relations. Pairs with at least one simple cell do have some nearly quadrature relationships when they are recorded from the same electrode. Coarse-to-fine connections (i.e., the presynaptic cell has lower disparity frequency and larger disparity range) tend to be stronger but less frequent than those of a fine-to-coarse nature. Our results are consistent with a system that produces weighted averaging across cells that are tuned to similar disparities but different disparity scales to reduce false matches.

Detecting disparity in two-dimensional patterns

One can measure the disparities between two retinal images in several different ways. Experiments were conducted to identify the measure that is invariant at the threshold for detecting the disparity of two-dimensional patterns. The patterns used were stereo plaids, which permit a partial dissociation between the disparity of the pattern and the disparities of its one-dimensional components. For plaids with near-horizontal disparities, thresholds are limited by a disparity phase shift equal to the threshold phase shift for single gratings. For non-horizontal disparities, thresholds are elevated, yet are still phase-limited. In no disparity direction are thresholds for detecting disparity determined by the spatial extent of the plaids' disparity. Effects of the number and the orientation of components with task-relevant disparities indicate that plaid thresholds are limited by the disparity of the plaid's one-dimensional components. No evidence was found that these components form any higher-order pattern that can be used in detecting disparity. Oblique and near-vertical disparities generate elevated thresholds at a stage beyond component disparity detection. This second stage combines component disparities, which are ambiguous about depth, into pattern disparities capable of supporting veridical depth perception.

Reading a population code: a multi-scale neural model for representing binocular disparity

Although binocular neurons in the primary visual cortex are sensitive to retinal disparity, their activity does not constitute an unambiguous disparity signal. A multi-spatial-scale neural model for disparity computation is developed to examine how population activity might be interpreted to overcome ambiguities at the single neuron level. The model incorporates a front end that encodes disparity by a family of complex cell-like energy units and a second stage that reads the population activity. Disparity is recovered by matching the population response to a set of canonical templates, derived from the mean response to white noise stimuli at a range of disparities. Model predictions are qualitatively consistent with a variety of psychophysical results in the literature, including the effects of spatial frequency on stereoacuity and bias in perceived depths, and the effect of standing disparity on increment thresholds. Model predictions are also consistent with data on qualitative appearance of complex stimuli, including depth averaging, transparency, and corrugation. The model also accounts for the non-linear interaction of disparities in compound grating stimuli. These results show that a template-match approach reduces ambiguities in individual and pooled neuronal responses, and allows for a broader range of percepts, consistent with psychophysics, than other models. Thus, the pattern of neural population activity across spatial scales is a better candidate for the neural correlate of depth perception than the activity of single neurons or the pooled activity of multiple neurons.

Stereoscopic depth processing in the visual cortex: a coarse-to-fine mechanism

For binocular animals viewing a three-dimensional scene, the left and right eyes receive slightly different information, and the brain uses this 'binocular disparity' to interpret stereoscopic depth. An important theoretical conjecture in this mechanism is that coarse processing precedes and constrains finely detailed processing. We present three types of neurophysiological data from the cat's visual cortex that are consistent with a temporal coarse-to-fine tuning of disparity information. First, the disparity tuning of cortical cells generally sharpened during the time course of response. Second, cells responsive to large and small spatial scale had relatively shorter and longer temporal latencies, respectively. Third, cross-correlation analysis between simultaneously recorded pairs of cortical cells showed that connections between disparity-tuned neurons were generally stronger for coarse-to-fine processing than for fine-to-coarse processing. These results are consistent with theoretical and behavioral studies and suggest that rapid, coarse percepts are refined over time in stereoscopic depth perception.

Luminance spatial scale and local stereo-sensitivity

Using filtered, broad band, fractal noise images we measured the dependence of D(min) and D(max) for stereo on luminance spatial frequency. D(min) was found to exhibit a simple dependence on the highest spatial frequency contained in the stimulus. D(max) depended on both image size and spatial frequency in a way that suggests an informational limit. Different rules govern D(min) and D(max) even for first order stereopsis, arguing against a common neural explanation based on independent access to the most pertinent spatial filter.

'Coarse-to-fine' cyclopean processing

Previously (Popple et al, 1998 Vision Research 38 319-326) we found, using random-dot stereograms, that initial vergence increases with the size of a cyclopean disc. A corresponding improvement in stereoacuity within the disc was predicted, because disparities in the disc would be brought closer to the plane of current fixation. In the present experiment, we looked at the effect of the spatial extent of a briefly presented (< or = 500 ms) cyclopean depth pedestal on stereoacuity thresholds. Observers were required to judge the depth of a small, 1.7 deg, central disc relative to a larger surrounding disc in a random-pattern stereogram. The larger disc was set, initially, at a pedestal disparity of +/- 24 min of arc against a fixation-plane surround. The size of the larger disc was varied from 2.6 to 8.0 deg. As predicted, stereoacuity thresholds fell significantly with increasing pedestal disc size. Next, the disparity of the pedestal disc was varied. When pedestal disparity was reduced to +/- 2.4 min of arc, a disparity too small to demand vergence, the size effect disappeared except when the pedestal boundary was within 30 min of arc of the test disc boundary. We argue from this result that the effect was largely due to vergence and not cyclopean integration alone. However, the effect of pedestal size was found to persist with stimuli too brief to permit vergence (< or = 100 ms) suggesting that factors other than vergence may also play a role.

Evolving concepts of spatial channels in vision: from independence to nonlinear interactions

By the 1960s it was evident from neuroanatomy that there were extensive recurrent interactions, both excitatory and inhibitory, among visual cortical neurons. Nevertheless, the psychophysical discovery of 'spatial-frequency channels' gave rise to a decade in which parallel, independent channels were thought to subserve early spatial vision. Recent work, however, has clearly demonstrated that early visual channels do not perform a Fourier or wavelet decomposition of the image. Instead, they interact through a variety of nonlinear pooling mechanisms. Such nonlinear interactions perform important computations in texture perception, stereopsis, and motion and form vision.

The area of spatial integration for initial horizontal disparity vergence

We investigated over what central area disparity in a random dot stereogram is integrated to stimulate an initial vergence response. Vergence was measured subjectively, with a forced choice dichoptic nonius vernier task following a brief (230 msec) stimulus presentation. Stimuli were random-dot stereograms showing a central circular disc of 12.5 min arc crossed retinal disparity in front of, and occluding, a same density fixation plane surround. The size of the disc was varied. All ten observers responded to the brief stimulus. Initial vergence increased with increasing disc diameter and, for nine out of ten subjects, reached a maximum with the disc ca 6 deg, suggesting this is the extent of the spatial integration region. Below 6 deg diameter, surround and target disparities were averaged together. Initial horizontal vergence responds automatically to a cyclopean target presented in the centre of gaze by pooling disparities within a limited but surprisingly large area.