The Active Side of Stereopsis: Fixation Strategy and Adaptation to Natural Environments

Topical Review: Methodological Variables in Clinical and Laboratory Measurements of Fixation Disparity

SIGNIFICANCE

Fixation disparity is a small vergence error that does not disrupt fusion. Fixation disparity measures correlate with binocular symptoms. This article covers methodological differences between clinical fixation disparity measurement devices, findings when objective and subjective fixation disparities are compared, and the potential impact of binocular capture on fixation disparity measurements. Fixation disparity is a small vergence error that occurs in nonstrabismic individuals and does not disrupt fusion. This article reviews clinical fixation disparity variables and their clinical diagnostic value. Clinical devices that are used to measure these variables are described, as are studies in which the output from these devices has been compared. Methodological differences between the devices such as the location of the fusional stimulus, the rate at which judgments of dichoptic alignment are made, and the strength of the accommodative stimulus are all considered. In addition, the article covers theories of the neural origins of fixation disparity and control system models incorporating fixation disparity. Studies in which objective fixation disparities (oculomotor portion of fixation disparity assessed with an eye tracker) and subjective fixation disparities (sensory portion of fixation disparity assessed psychophysically with dichoptic Nonius lines) have been compared are also examined, and consideration is given to why some investigators find differences in these measures, whereas other investigators do not. The conclusion thus far is that there are likely complex interactions between vergence adaptation, accommodation, and the location of the fusional stimulus that lead to differences in objective and subjective fixation disparity measures. Finally, capture of the visual direction of monocular stimuli by adjacent fusional stimuli and the implications for fixation disparity measures are considered.

The Statistics of Eye Movements and Binocular Disparities during VR Gaming: Implications for Headset Design

The human visual system evolved in environments with statistical regularities. Binocular vision is adapted to these such that depth perception and eye movements are more precise, faster, and performed comfortably in environments consistent with the regularities. We measured the statistics of eye movements and binocular disparities in virtual-reality (VR) - gaming environments and found that they are quite different from those in the natural environment. Fixation distance and direction are more restricted in VR, and fixation distance is farther. The pattern of disparity across the visual field is less regular in VR and does not conform to a prominent property of naturally occurring disparities. From this we predict that double vision is more likely in VR than in the natural environment. We also determined the optimal screen distance to minimize discomfort due to the vergence-accommodation conflict, and the optimal nasal-temporal positioning of head-mounted display (HMD) screens to maximize binocular field of view. Finally, in a user study we investigated how VR content affects comfort and performance. Content that is more consistent with the statistics of the natural world yields less discomfort than content that is not. Furthermore, consistent content yields slightly better performance than inconsistent content.

Learning bio-inspired head-centric representations of 3D shapes in an active fixation setting

When exploring the surrounding environment with the eyes, humans and primates need to interpret three-dimensional (3D) shapes in a fast and invariant way, exploiting a highly variant and gaze-dependent visual information. Since they have front-facing eyes, binocular disparity is a prominent cue for depth perception. Specifically, it serves as computational substrate for two ground mechanisms of binocular active vision: stereopsis and binocular coordination. To this aim, disparity information, which is expressed in a retinotopic reference frame, is combined along the visual cortical pathways with gaze information and transformed in a head-centric reference frame. Despite the importance of this mechanism, the underlying neural substrates still remain widely unknown. In this work, we investigate the capabilities of the human visual system to interpret the 3D scene exploiting disparity and gaze information. In a psychophysical experiment, human subjects were asked to judge the depth orientation of a planar surface either while fixating a target point or while freely exploring the surface. Moreover, we used the same stimuli to train a recurrent neural network to exploit the responses of a modelled population of cortical (V1) cells to interpret the 3D scene layout. The results for both human performance and from the model network show that integrating disparity information across gaze directions is crucial for a reliable and invariant interpretation of the 3D geometry of the scene.

Crossed-uncrossed projections from primate retina are adapted to disparities of natural scenes

In mammals with frontal eyes, optic-nerve fibers from nasal retina project to the contralateral hemisphere of the brain, and fibers from temporal retina project ipsilaterally. The division between crossed and uncrossed projections occurs at or near the vertical meridian. If the division was precise, a problem would arise. Small objects near midline, but nearer or farther than current fixation, would produce signals that travel to opposite hemispheres, making the binocular disparity of those objects difficult to compute. However, in species that have been studied, the division is not precise. Rather, there are overlapping crossed and uncrossed projections such that some fibers from nasal retina project ipsilaterally as well as contralaterally and some from temporal retina project contralaterally as well as ipsilaterally. This increases the probability that signals from an object near vertical midline travel to the same hemisphere, thereby aiding disparity estimation. We investigated whether there is a deficit in binocular vision near the vertical meridian in humans and found no evidence for one. We also investigated the effectiveness of the observed decussation pattern, quantified from anatomical data in monkeys and humans. We used measurements of naturally occurring disparities in humans to determine disparity distributions across the visual field. We then used those distributions to calculate the probability of natural disparities transmitting to the same hemisphere, thereby aiding disparity computation. We found that the pattern of overlapping projections is quite effective. Thus, crossed and uncrossed projections from the retinas are well designed for aiding disparity estimation and stereopsis.

The blur horopter: Retinal conjugate surface in binocular viewing

From measurements of wavefront aberrations in 16 emmetropic eyes, we calculated where objects in the world create best-focused images across the central 27∘ (diameter) of the retina. This is the retinal conjugate surface. We calculated how the surface changes as the eye accommodates from near to far and found that it mostly maintains its shape. The conjugate surface is pitched top-back, meaning that the upper visual field is relatively hyperopic compared to the lower field. We extended the measurements of best image quality into the binocular domain by considering how the retinal conjugate surfaces for the two eyes overlap in binocular viewing. We call this binocular extension the blur horopter. We show that in combining the two images with possibly different sharpness, the visual system creates a larger depth of field of apparently sharp images than occurs with monocular viewing. We examined similarities between the blur horopter and its analog in binocular vision: the binocular horopter. We compared these horopters to the statistics of the natural visual environment. The binocular horopter and scene statistics are strikingly similar. The blur horopter and natural statistics are qualitatively, but not quantitatively, similar. Finally, we used the measurements to refine what is commonly referred to as the zone of clear single binocular vision.

Optimal allocation of quantized human eye depth perception for multi-focal 3D display design

Creating immersive 3D stereoscopic, autostereoscopic, and lightfield experiences are becoming the center point of optical design of future head mounted displays and lightfield displays. However, despite the advancement in 3D and light field displays, there is no consensus on what are the necessary quantized depth levels for such emerging displays at stereoscopic or monocular modalities. Here we start from psychophysical theories and work toward defining and prioritizing quantized levels of depth that would saturate the human depth perception. We propose a general optimization framework, which locates the depth levels in a globally optimal way for band limited displays. While the original problem is computationally intractable, we manage to find a tractable reformulation as maximally covering a region of interest with a selection of hypographs corresponding to the monocular depth of field profiles. The results indicate that on average 1731 stereoscopic and 7 monocular depth levels (distributed optimally from 25 cm to infinity) would saturate the visual depth perception. Such that adding further depth levels adds negligible improvement. Also the first 3 depth levels should be allocated at (148), then (83, 170), then (53, 90, 170) distances respectively from the face plane to minimize the monocular error in the entire population. The study further discusses the 3D spatial profile of the quantized stereoscopic and monocular depth levels. The study provides fundamental guidelines for designing optimal near eye displays, light-field monitors, and 3D screens.

The saccade main sequence revised: A fast and repeatable tool for oculomotor analysis

Saccades are rapid ballistic eye movements that humans make to direct the fovea to an object of interest. Their kinematics is well defined, showing regular relationships between amplitude, duration, and velocity: the saccadic 'main sequence'. Deviations of eye movements from the main sequence can be used as markers of specific neurological disorders. Despite its significance, there is no general methodological consensus for reliable and repeatable measurements of the main sequence. In this work, we propose a novel approach for standard indicators of oculomotor performance. The obtained measurements are characterized by high repeatability, allowing for fine assessments of inter- and intra-subject variability, and inter-ocular differences. The designed experimental procedure is natural and non-fatiguing, thus it is well suited for fragile or non-collaborative subjects like neurological patients and infants. The method has been released as a software toolbox for public use. This framework lays the foundation for a normative dataset of healthy oculomotor performance for the assessment of oculomotor dysfunctions.

Natural statistics of depth edges modulate perceptual stability

Binocular fusion relies on matching points in the two eyes that correspond to the same physical feature in the world; however, not all world features are binocularly visible. Near depth edges, some regions of a scene are often visible to only one eye (so-called half occlusions). Accurate detection of these monocularly visible regions is likely to be important for stable visual perception. If monocular regions are not detected as such, the visual system may attempt to binocularly fuse non-corresponding points, which can result in unstable percepts. We investigated the hypothesis that the visual system capitalizes on statistical regularities associated with depth edges in natural scenes to aid binocular fusion and facilitate perceptual stability. By sampling from a large set of stereoscopic natural images with co-registered distance information, we found evidence that monocularly visible regions near depth edges primarily result from background occlusions. Accordingly, monocular regions tended to be more visually similar to the adjacent binocularly visible background region than to the adjacent binocularly visible foreground. Consistent with our hypothesis, perceptual experiments showed that perception tended to be more stable when the image properties of the depth edge were statistically more likely given the probability of occurrence in natural scenes (i.e., when monocular regions were more visually similar to the binocular background). The generality of these results was supported by a parametric study with simulated environments. Exploiting regularities in natural environments may allow the visual system to facilitate fusion and perceptual stability when both binocular and monocular regions are visible.

A Geometric Theory Integrating Human Binocular Vision With Eye Movement

A theory of the binocular system with asymmetric eyes (AEs) is developed in the framework of bicentric perspective projections. The AE accounts for the eyeball's global asymmetry produced by the foveal displacement from the posterior pole, the main source of the eye's optical aberrations, and the crystalline lens' tilt countering some of these aberrations. In this theory, the horopter curves, which specify retinal correspondence of binocular single vision, are conic sections resembling empirical horopters. This advances the classic model of empirical horopters as conic sections introduced in an ad hoc way by Ogle in 1932. In contrast to Ogle's theory, here, anatomically supported horopteric conics vary with the AEs' position in the visual plane of bifoveal fixations and their transformations are visualized in a computer simulation. Integrating horopteric conics with eye movements can help design algorithms for maintaining a stable perceptual world from visual information captured by a mobile robot's camera head. Further, this paper proposes a neurophysiologically meaningful definition for the eyes' primary position, a concept which has remained elusive despite its theoretical importance to oculomotor research. Finally, because the horopteric conic's shape is dependent on the AE's parameters, this theory allows for changes in retinal correspondence, which is usually considered preformed and stable.

A dichoptic feedback-based oculomotor training method to manipulate interocular alignment

Strabismus is a prevalent impairment of binocular alignment that is associated with a spectrum of perceptual deficits and social disadvantages. Current treatments for strabismus involve ocular alignment through surgical or optical methods and may include vision therapy exercises. In the present study, we explore the potential of real-time dichoptic visual feedback that may be used to quantify and manipulate interocular alignment. A gaze-contingent ring was presented independently to each eye of 11 normally-sighted observers as they fixated a target dot presented only to their dominant eye. Their task was to center the rings within 2° of the target for at least 1 s, with feedback provided by the sizes of the rings. By offsetting the ring in the non-dominant eye temporally or nasally, this task required convergence or divergence, respectively, of the non-dominant eye. Eight of 11 observers attained 5° asymmetric convergence and 3 of 11 attained 3° asymmetric divergence. The results suggest that real-time gaze-contingent feedback may be used to quantify and transiently simulate strabismus and holds promise as a method to augment existing therapies for oculomotor alignment disorders.

Near-optimal combination of disparity across a log-polar scaled visual field

The human visual system is foveated: we can see fine spatial details in central vision, whereas resolution is poor in our peripheral visual field, and this loss of resolution follows an approximately logarithmic decrease. Additionally, our brain organizes visual input in polar coordinates. Therefore, the image projection occurring between retina and primary visual cortex can be mathematically described by the log-polar transform. Here, we test and model how this space-variant visual processing affects how we process binocular disparity, a key component of human depth perception. We observe that the fovea preferentially processes disparities at fine spatial scales, whereas the visual periphery is tuned for coarse spatial scales, in line with the naturally occurring distributions of depths and disparities in the real-world. We further show that the visual system integrates disparity information across the visual field, in a near-optimal fashion. We develop a foveated, log-polar model that mimics the processing of depth information in primary visual cortex and that can process disparity directly in the cortical domain representation. This model takes real images as input and recreates the observed topography of human disparity sensitivity. Our findings support the notion that our foveated, binocular visual system has been moulded by the statistics of our visual environment.

Active efficient coding explains the development of binocular vision and its failure in amblyopia

Brains must operate in an energy-efficient manner. The efficient coding hypothesis states that sensory systems achieve this by adapting neural representations to the statistics of sensory input signals. Importantly, however, these statistics are shaped by the organism’s behavior and how it samples information from the environment. Therefore, optimal performance requires jointly optimizing neural representations and behavior, a theory called active efficient coding. Here, we test the plausibility of this theory by proposing a computational model of the development of binocular vision. The model explains the development of accurate binocular vision under healthy conditions. In the case of refractive errors, however, the model develops an amblyopia-like state and suggests conditions for successful treatment.

The development of vision during the first months of life is an active process that comprises the learning of appropriate neural representations and the learning of accurate eye movements. While it has long been suspected that the two learning processes are coupled, there is still no widely accepted theoretical framework describing this joint development. Here, we propose a computational model of the development of active binocular vision to fill this gap. The model is based on a formulation of the active efficient coding theory, which proposes that eye movements as well as stimulus encoding are jointly adapted to maximize the overall coding efficiency. Under healthy conditions, the model self-calibrates to perform accurate vergence and accommodation eye movements. It exploits disparity cues to deduce the direction of defocus, which leads to coordinated vergence and accommodation responses. In a simulated anisometropic case, where the refraction power of the two eyes differs, an amblyopia-like state develops in which the foveal region of one eye is suppressed due to inputs from the other eye. After correcting for refractive errors, the model can only reach healthy performance levels if receptive fields are still plastic, in line with findings on a critical period for binocular vision development. Overall, our model offers a unifying conceptual framework for understanding the development of binocular vision.

Optimal Stereoacuity Reveals More Than Critical Time in Patients With Intermittent Exotropia

Synopsis

Both optimal stereoacuity and integration time to achieve that are impaired in patients with intermittent exotropia. The deterioration of stereoacuity is more revealing since it correlates well with exotropia control score.

Background

Despite the periodic misalignment of two eyes, some intermittent exotropia (IXT) patients exhibit normal stereoacuity, particularly when evaluated with static tests. It is not clear if the temporal integration process of stereopsis is altered in IXT patients, thus warranting further research.

Methods

IXT patients (n = 29) and age-matched normal controls (n = 36) were recruited. Static stereopsis was measured with the Titmus stereoacuity test. In computer-generated random dots tests, stereoacuity was measured with a stimuli presentation duration varying from 100 to 1,200 ms. And the relationship between stereoacuity and stimuli duration was fitted into a quadratic model. Optimal stereoacuity was achieved when fitted curve flattened and the critical integration time was the duration needed to achieve optimal stereoacuity.

Results

IXT patients were not found to differ significantly from control subjects under the Titmus test, while the Random Dots stereotest showed significantly worse optimal stereoacuity and significantly longer critical integration time. Multiple regression analysis showed that age (R = −4.83; P = 0.04) had statistically significant negative correlation on the critical integration time, age (R = −6.45; P = 0.047) and exotropia control scores (R = 60.71; P = 0.007) had statistically significant effects on optimal stereoacuity.

Conclusion

The temporal integration for stereopsis is impaired in IXT patients, requiring longer critical integration time to achieve elevated optimal stereoacuity.

Binocular Eye Movements Are Adapted to the Natural Environment

Humans and many animals make frequent saccades requiring coordinated movements of the eyes. When landing on the new fixation point, the eyes must converge accurately or double images will be perceived. We asked whether the visual system uses statistical regularities in the natural environment to aid eye alignment at the end of saccades. We measured the distribution of naturally occurring disparities in different parts of the visual field. The central tendency of the distributions was crossed (nearer than fixation) in the lower field and uncrossed (farther) in the upper field in male and female participants. It was uncrossed in the left and right fields. We also measured horizontal vergence after completion of vertical, horizontal, and oblique saccades. When the eyes first landed near the eccentric target, vergence was quite consistent with the natural-disparity distribution. For example, when making an upward saccade, the eyes diverged to be aligned with the most probable uncrossed disparity in that part of the visual field. Likewise, when making a downward saccade, the eyes converged to enable alignment with crossed disparity in that part of the field. Our results show that rapid binocular eye movements are adapted to the statistics of the 3D environment, minimizing the need for large corrective vergence movements at the end of saccades. The results are relevant to the debate about whether eye movements are derived from separate saccadic and vergence neural commands that control both eyes or from separate monocular commands that control the eyes independently.SIGNIFICANCE STATEMENT We show that the human visual system incorporates statistical regularities in the visual environment to enable efficient binocular eye movements. We define the oculomotor horopter: the surface of 3D positions to which the eyes initially move when stimulated by eccentric targets. The observed movements maximize the probability of accurate fixation as the eyes move from one position to another. This is the first study to show quantitatively that binocular eye movements conform to 3D scene statistics, thereby enabling efficient processing. The results provide greater insight into the neural mechanisms underlying the planning and execution of saccadic eye movements.

Binocular system with asymmetric eyes

I elaborate binocular geometry with a novel eye model that incorporates the fovea's temporalward displacement and the cornea and the lens' misalignment. The formulated binocular correspondence results in longitudinal horopters that are conic sections resembling empirical horopters. When the eye model's asymmetry parameters' range is that which is observed in healthy eyes, abathic distance also falls within its experimentally observed range. This range in abathic distance is similar to that of the vergence resting position distance. Further, the conic's orientation is specified by the eyes' version angle, integrating binocular geometry with eye movement. This integration presents the possibility for modeling 3D perceptual stability during physiological eye movements.

Depth variation and stereo processing tasks in natural scenes

Local depth variation is a distinctive property of natural scenes, but its effects on perception have only recently begun to be investigated. Depth variation in natural scenes is due to depth edges between objects and surface nonuniformities within objects. Here, we demonstrate how natural depth variation impacts performance in two fundamental tasks related to stereopsis: half-occlusion detection and disparity detection. We report the results of a computational study that uses a large database of natural stereo-images and coregistered laser-based distance measurements. First, we develop a procedure for precisely sampling stereo-image patches from the stereo-images and then quantify the local depth variation in each patch by its disparity contrast. Next, we show that increased disparity contrast degrades half-occlusion detection and disparity detection performance and changes the size and shape of the spatial integration areas ("receptive fields") that optimize performance. Then, we show that a simple image-computable binocular statistic predicts disparity contrast in natural scenes. Finally, we report the most likely spatial patterns of disparity variation and disparity discontinuities (half-occlusions) in natural scenes. Our findings motivate computational and psychophysical investigations of the mechanisms that underlie stereo processing tasks in local regions of natural scenes.

Determinants of neural responses to disparity in natural scenes

We studied disparity-evoked responses in natural scenes using high-density electroencephalography (EEG) in an event-related design. Thirty natural scenes that mainly included outdoor settings with trees and buildings were used. Twenty-four subjects viewed a series of trials composed of sequential two-alternative temporal forced-choice presentation of two different versions (two-dimensional [2D] vs. three-dimensional [3D]) of the same scene interleaved by a scrambled image with the same power spectrum. Scenes were viewed orthostereoscopically at 3 m through a pair of shutter glasses. After each trial, participants indicated with a key press which version of the scene was 3D. Performance on the discrimination was >90%. Participants who were more accurate also tended to respond faster; scenes that were reported more accurately as 3D also led to faster reaction times. We compared visual evoked potentials elicited by scrambled, 2D, and 3D scenes using reliable component analysis to reduce dimensionality. The disparity-evoked response to natural scene stimuli, measured from the difference potential between 2D and 3D scenes, comprised a sustained relative negativity in the dominant response component. The magnitude of the disparity-specific response was correlated with the observer's stereoacuity. Scenes with more homogeneous depth maps also tended to elicit large disparity-specific responses. Finally, the magnitude of the disparity-specific response was correlated with the magnitude of the differential response between scrambled and 2D scenes, suggesting that monocular higher-order scene statistics modulate disparity-specific responses.

A dataset of stereoscopic images and ground-truth disparity mimicking human fixations in peripersonal space

Binocular stereopsis is the ability of a visual system, belonging to a live being or a machine, to interpret the different visual information deriving from two eyes/cameras for depth perception. From this perspective, the ground-truth information about three-dimensional visual space, which is hardly available, is an ideal tool both for evaluating human performance and for benchmarking machine vision algorithms. In the present work, we implemented a rendering methodology in which the camera pose mimics realistic eye pose for a fixating observer, thus including convergent eye geometry and cyclotorsion. The virtual environment we developed relies on highly accurate 3D virtual models, and its full controllability allows us to obtain the stereoscopic pairs together with the ground-truth depth and camera pose information. We thus created a stereoscopic dataset: GENUA PESTO-GENoa hUman Active fixation database: PEripersonal space STereoscopic images and grOund truth disparity. The dataset aims to provide a unified framework useful for a number of problems relevant to human and computer vision, from scene exploration and eye movement studies to 3D scene reconstruction.