Stereo correspondence in one-dimensional Gabor stimuli

The Statistics of Eye Movements and Binocular Disparities during VR Gaming: Implications for Headset Design

The human visual system evolved in environments with statistical regularities. Binocular vision is adapted to these such that depth perception and eye movements are more precise, faster, and performed comfortably in environments consistent with the regularities. We measured the statistics of eye movements and binocular disparities in virtual-reality (VR) - gaming environments and found that they are quite different from those in the natural environment. Fixation distance and direction are more restricted in VR, and fixation distance is farther. The pattern of disparity across the visual field is less regular in VR and does not conform to a prominent property of naturally occurring disparities. From this we predict that double vision is more likely in VR than in the natural environment. We also determined the optimal screen distance to minimize discomfort due to the vergence-accommodation conflict, and the optimal nasal-temporal positioning of head-mounted display (HMD) screens to maximize binocular field of view. Finally, in a user study we investigated how VR content affects comfort and performance. Content that is more consistent with the statistics of the natural world yields less discomfort than content that is not. Furthermore, consistent content yields slightly better performance than inconsistent content.

Monocular blur alters the tuning characteristics of stereopsis for spatial frequency and size

Our sense of depth perception is mediated by spatial filters at different scales in the visual brain; low spatial frequency channels provide the basis for coarse stereopsis, whereas high spatial frequency channels provide for fine stereopsis. It is well established that monocular blurring of vision results in decreased stereoacuity. However, previous studies have used tests that are broadband in their spatial frequency content. It is not yet entirely clear how the processing of stereopsis in different spatial frequency channels is altered in response to binocular input imbalance. Here, we applied a new stereoacuity test based on narrow-band Gabor stimuli. By manipulating the carrier spatial frequency, we were able to reveal the spatial frequency tuning of stereopsis, spanning from coarse to fine, under blurred conditions. Our findings show that increasing monocular blur elevates stereoacuity thresholds 'selectively' at high spatial frequencies, gradually shifting the optimum frequency to lower spatial frequencies. Surprisingly, stereopsis for low frequency targets was only mildly affected even with an acuity difference of eight lines on a standard letter chart. Furthermore, we examined the effect of monocular blur on the size tuning function of stereopsis. The clinical implications of these findings are discussed.

Stereopsis is adaptive for the natural environment

Humans and many animals have forward-facing eyes providing different views of the environment. Precise depth estimates can be derived from the resulting binocular disparities, but determining which parts of the two retinal images correspond to one another is computationally challenging. To aid the computation, the visual system focuses the search on a small range of disparities. We asked whether the disparities encountered in the natural environment match that range. We did this by simultaneously measuring binocular eye position and three-dimensional scene geometry during natural tasks. The natural distribution of disparities is indeed matched to the smaller range of correspondence search. Furthermore, the distribution explains the perception of some ambiguous stereograms. Finally, disparity preferences of macaque cortical neurons are consistent with the natural distribution.

Coarse-fine dichotomies in human stereopsis

There is a long history of research into depth percepts from very large disparities, beyond the fusion limit. Such diplopic stimuli have repeatedly been shown to provide reliable depth percepts. A number of researchers have pointed to differences between the processing of small and large disparities, arguing that they are subserved by distinct neural mechanisms. Other studies have pointed to a dichotomy between the processing of 1st- and 2nd-order stimuli. Here we review literature on the full range of disparity processing to determine how well different proposed dichotomies map onto one another, and to identify unresolved issues.

Limits of stereopsis explained by local cross-correlation

Human stereopsis has two well-known constraints: the disparity-gradient limit, which is the inability to perceive depth when the change in disparity within a region is too large, and the limit of stereoresolution, which is the inability to perceive spatial variations in disparity that occur at too fine a spatial scale. We propose that both limitations can be understood as byproducts of estimating disparity by cross-correlating the two eyes' images, the fundamental computation underlying the disparity-energy model. To test this proposal, we constructed a local cross-correlation model with biologically motivated properties. We then compared model and human behaviors in the same psychophysical tasks. The model and humans behaved quite similarly: they both exhibited a disparity-gradient limit and had similar stereoresolution thresholds. Performance was affected similarly by changes in a variety of stimulus parameters. By modeling the effects of stimulus blur and of using different sizes of image patches, we found evidence that the smallest neural mechanism humans use to estimate disparity is 3-6 arcmin in diameter. We conclude that the disparity-gradient limit and stereoresolution are indeed byproducts of using local cross-correlation to estimate disparity.

Upper disparity limit after LASIK

We evaluate the effect of the emmetropization technique LASIK (laser-assisted in situ keratomileusis) on stereoscopic vision. For this, we used a mirror stereoscope to measure the upper disparity limit D(max) before (with best correction) and after LASIK for 30 patients. The results show that the upper disparity limit declines from 41.1 min of arc on average to 31.3 min of arc after LASIK, being significant in 83% of the patients. This deterioration is significantly correlated with an increase in the postsurgical interocular differences in higher-order aberrations, corneal asphericity, and presurgical anisometropia. New ablation algorithms should minimize interocular differences in order to improve binocular visual performance.

The wallpaper illusion explained

In the wallpaper illusion, a repetitive pattern appears to shift from one depth plane to the plane nearest fixation. We measured the timing of this shift for a 6 degrees wide, 3-cpd sinusoidal grating presented in a rectangular envelope; the edges (envelope) of the grating were presented at 20 arcmin of disparity (one period) behind the fixation plane. We asked observers to signal when the segment appeared to move from the edge plane forward to the fixation plane. Initially, the shift from the edge plane took 4-6 s, but after many trials, the shift became faster. Additional experiments demonstrated that the envelope was adapting, thereby permitting the alternative match. Our measurements for a range of spatial frequencies and disparities showed that these shifts to the fixation plane occurred only if the envelope disparity was more than one-half period of the carrier; that is, phase disparity >180 degrees. We also found that stereoacuity for the initial envelope-based match was poor, as might be expected for a target presented far off the fixation plane. However, once the perceived shift in depth occurred, stereoacuity improved fivefold without any change in the physical stimulus. We speculate that access to the most sensitive V1 neurons depends on the extrastriate processes that determine perceived depth--in this case, second-order envelope mechanisms.

Stereoscopic depth discrimination with contrast windowed stimuli

Depth discrimination with a shifted contrast window was compared to that with a fixed contrast window. Stereoscopic performance with the fixed window was limited to small disparities and varied with spatial frequency. Performance with the shifted window extended to larger disparities and was more similar for low and high spatial frequencies. The results depended upon window shape, indicating that edge blur is an important factor. Stereoscopic performance with shifted patterns was supported at disparities larger than a phase disparity model might predict, suggesting that a combination of position and phase disparity computations are used for the perception of stereoscopic depth.

Evidence for elongated receptive field structure for mechanisms subserving stereopsis

To study the spatial extent and shape of the binocular disparity mechanisms subserving depth perception, we employ the spatial summation paradigm of contrast threshold for front/back depth discrimination at a fixed binocular disparity. The stimuli were Gabor patches with disparity set at either 4 or 8 arcmin and spatial frequency set at an optimal value of 4 cy/deg. Contrast threshold was measured as a function of length and width of the Gabor patches to determine the aspect ratio of greatest efficiency. The space constant of the Gaussian envelope varied between 0.0375 degrees and 0.9 degrees in either vertical or horizontal directions, or both simultaneously. For vertical elongation of the Gabor patches, discrimination sensitivity improved by 4-6 dB for a doubling of the length of the Gabor patches, then reduced more slowly as the length further increased. However, extending the Gabor patches horizontally across cycles produced little or no sensitivity improvement. Instead, discrimination performance collapsed in a fashion that is incompatible with many models of disparity processing. The results imply that the main mechanisms subserving stereoscopic depth discrimination are vertically elongated for vertical-bar Gabors and encounter special difficulties integrating horizontal disparity information. Disparity discrimination sensitivity for very small targets was also much greater than predicted by the single-mechanism fit, implying the presence of a second, independent mechanism with a very small summation field, which may underlie the fine stereoscopic processing system.

Binocular Visual Performance After LASIK

PURPOSE

To analyze binocular visual function after LASIK.

METHODS

Eye aberrometry and corneal topography was obtained for both eyes in 68 patients (136 eyes). To evaluate visual performance, monocular and binocular contrast sensitivity function and disturbance index for quantifying halos were measured. Tests were performed under mesopic conditions.

RESULTS

Binocular summation and disturbance index diminished significantly (P<.0001) after LASIK with increasing interocular differences in corneal and eye aberrations. Binocular visual deterioration was greater than monocular deterioration for contrast sensitivity function and disturbance index.

CONCLUSIONS

Binocular function deteriorates more than monocular function after LASIK. This deterioration increases as the interocular differences in aberrations and corneal shape increase. Improvements in ablation algorithms should minimize these interocular differences.

Stereo dynamics are not scale-dependent

The experiments reported here focus on the temporal dynamics of stereopsis in an effort to shed light on how low level mechanisms might contribute to the execution of coarse-to-fine processing in the human stereo system. Because previous studies have used a variety of stimuli and configurations, we assess the effect of exposure duration on stereo thresholds using band-limited Gabor patches for a range of stimulus configurations. In preliminary studies, we found that the best stereo sensitivity-spatial frequency relationship was obtained when using configurations in which the size and target-reference spacing were consistent with spatially scaled stimuli. Sub-optimal stereo sensitivity as a function of spatial frequency was observed when the size and separation were fixed. Further, we found that the temporal properties of stereopsis were consistently sustained in nature irrespective of the stimulus spatial frequency content. This latter finding suggests that if coarse-to-fine stereo processing does occur it does not follow as a consequence of the dynamics of low-level disparity transduction.

Stereo sensitivity depends on stereo matching

Stereoacuity thresholds, measured with bar targets, rise as the absolute disparity of the bars is increased. One explanation for this rise is that, as the bars are moved away from the fixation plane, the stereo system uses coarser mechanisms to encode the bars' disparity; coarse mechanisms are insensitive to small changes in target disparity, resulting in higher thresholds. To test this explanation, we measured stereoacuity with a 6 degrees wide 3 cpd grating presented in a rectangular envelope. We varied the disparity of the grating and its edges (envelope) parametrically from 0 to 20 arcmin (i.e., through one full period). To force observers to make judgments based on carrier disparity, we then varied the interocular phase incrementally from trial-to-trial while keeping edge disparity fixed for a given block of trials. The pedestal phase disparity of the grating necessarily cycles through 360 degrees, back to zero disparity, as the edge disparity increases monotonically from 0 to 20 arcmin. Unlike mechanisms that respond to bars, the mechanism that responds to the interocular phase disparity of the grating should have the same sensitivity at 20 arcmin disparity (360 degrees of phase) as it has at zero disparity. So, if stereoacuity were determined by the most sensitive mechanism, thresholds should oscillate with the pedestal phase disparity. However, these gratings are perceived in depth at the disparity of their edges. If stereoacuity were instead determined by the stereo matching operations that generate perceived depth, thresholds should rise monotonically with increasing edge disparity. We found that the rise in grating thresholds with increasing edge disparity was monotonic and virtually identical to the rise in thresholds observed for bars. Stereoacuity is contingent on stereo matching.

Induced Aniseikonia Diminishes Binocular Contrast Sensitivity and Binocular Summation

PURPOSE

To study the effect on binocular contrast sensitivity and binocular summation of aniseikonia induced by size (afocal) lenses.

METHODS

In 18 young emmetropic observers, the monocular and binocular contrast sensitivity function was measured under normal conditions and after inducing different degrees of aniseikonia (2%, 3%, and 5% and for 6 observers, 8%) in the right eye. The spatial frequencies tested were 2.4, 3.7, 6.0, 9.2, 12, 15, 20, and 24 cpd.

RESULTS

The results reveal a significant decline in binocular contrast sensitivity and binocular summation for 5% aniseikonia, this decline being more pronounced for intermediate and high spatial frequencies.

CONCLUSIONS

These results, together with others on different binocular functions, show the importance of aniseikonia in binocular vision. On this basis, we recommend that aniseikonia be considered in surgical processes such as cataract and refractive surgery, whereby aniseikonia could be induced and binocular performance subsequently diminished.

A Bayesian approach to the stereo correspondence problem

I present a probabilistic approach to the stereo correspondence problem. Rather than trying to find a single solution in which each point in the left retina is assigned a partner in the right retina, all possible matches are considered simultaneously and assigned a probability of being correct. This approach is particularly suitable for stimuli where it is inappropriate to seek a unique partner for each retinal position--for instance, where objects occlude each other, as in Panum's limiting case. The probability assigned to each match is based on a Bayesian analysis previously developed to explain psychophysical data (Read, 2002). This provides a convenient way to incorporate constraints that enable the ill-posed correspondence problem to be solved. The resulting model behaves plausibly for a variety of different stimuli.

Spatial interactions minimize relative disparity between adjacent surfaces

Computational models of stereopsis employ a number of algorithms that constrain stereo matches to produce the smallest absolute disparity and to minimize the relative disparity between nearby features. In some natural scenes, such as large slanted textured surfaces, these two constraints lead to different matching solutions. The current study utilized a stimulus in which there was a large discrepancy in both the magnitude and direction of matches that solved for minimum absolute and minimum relative disparity. This discrepancy revealed a dominance for the minimum relative disparity over the minimum absolute disparity matching solution that increased with spatial proximity, spatial frequency and width of adjacent features. The likelihood of a minimum-relative-disparity matching solution also increased when the difference between the amplitudes of the alternative relative disparities was large. When alternative relative disparity matching solutions had similar amplitudes but opposite signs (crossed vs. uncrossed), an idiosyncratic depth bias served as a tie-breaker. The present results show that absolute disparity matches are constrained to minimize relative disparity between adjacent features.

The physiology of stereopsis

Binocular disparity provides the visual system with information concerning the three-dimensional layout of the environment. Recent physiological studies in the primary visual cortex provide a successful account of the mechanisms by which single neurons are able to signal disparity. This work also reveals that additional processing is required to make explicit the types of signal required for depth perception (such as the ability to match features correctly between the two monocular images). Some of these signals, such as those encoding relative disparity, are found in extrastriate cortex. Several other lines of evidence also suggest that the link between perception and neuronal activity is stronger in extrastriate cortex (especially MT) than in the primary visual cortex.

Reversed stereo depth and motion direction with anti-correlated stimuli

We used anti-correlated stimuli to compare the correspondence problem in stereo and motion. Subjects performed a two-interval forced-choice disparity/motion direction discrimination task for different displacements. For anti-correlated 1d band-pass noise, we found weak reversed depth and motion. With 2d anti-correlated stimuli, stereo performance was impaired, but the perception of reversed motion was enhanced. We can explain the main features of our data in terms of channels tuned to different spatial frequencies and orientation. We suggest that a key difference between the solution of the correspondence problem by the motion and stereo systems concerns the integration of information at different orientations.

Weighted directional energy model of human stereo correspondence

Previous work [Prince, S. J. D, & Eagle, R. A. (1999). Size-disparity correlation in human binocular depth perception. Proceedings of the Royal Society: Biological Sciences, 266, 1361-1365] has demonstrated that disparity sign discrimination performance in isolated bandpass patterns is supported at disparities much larger than a phase disparity model might predict. One possibility is that this extended performance relies on a separate second-order system [Hess, R. F., & Wilcox, L. M. (1994). Linear and non-linear filtering in stereopsis. Vision Research, 34, 2431-2438]. Here, a 'weighted directional energy' model is developed which explains a large body of crossed versus uncrossed disparity discrimination data with a single mechanism. This model assumes a population of binocular complex cells at every image point with a range of position disparity shifts. These cells sample a local energy function which is weighted so that energy at large disparities is relatively attenuated. Disparity sign is determined by summing and comparing energy at crossed and uncrossed disparities in the presence of noise. The model qualitatively predicts matching data for one-dimensional Gabor stimuli. This scheme also predicts DMax in Gabor stimuli and filtered noise. Moreover, a range of 'non-linear' phenomena, in which disparity is perceived from contrast envelope information alone, can be explained. The weighted directional energy model presents a biologically plausible, parsimonious explanation of matching behaviour in bandpass stimuli for both 'first-order' and 'second-order' stimuli which obviates the need for multiple mechanisms in stereo correspondence.

Local disparity not perceived depth is signaled by binocular neurons in cortical area V1 of the Macaque

Binocular neurons that are closely related to depth perception should respond selectively for stimuli eliciting an appropriate depth sensation. To separate perceived depth from local disparity within the receptive field, sinusoidal luminance gratings were presented within a circular aperture. The disparity of the aperture was coupled to that of the grating, thereby rendering unambiguous the psychophysical matching between repeating cycles of the grating. In cases in which the stimulus disparity differs by one horizontal period of the grating, the portion of the grating that locally covers a receptive field is binocularly identical, but the depth sensation is very different because of the aperture. For 117 disparity-selective V1 neurons tested in two monkeys, the overwhelming majority responded equally well to configurations that were locally identical but led to different perceptions of depth. Because the psychophysical sensation is not reflected in the firing rate of V1 neurons, the signals that make stereo matches explicit are most likely elaborated in extrastriate cortex.