Temporal aspects of stereoscopic slant estimation: an evaluation and extension of Howard and Kaneko's theory

Early dynamics of stereoscopic surface slant perception

Surface orientation is an important visual primitive that can be estimated from monocular or binocular (stereoscopic) signals. Changes in motor planning occur within about 200 ms after either type of signal is perturbed, but the time it takes for apparent (perceived) slant to develop from stereoscopic cues is not known. Apparent slant sometimes develops very slowly (Gillam, Chambers, & Russo, 1988; van Ee & Erkelens, 1996). However, these long durations could reflect the time it takes for the visual system to resolve conflicts between slant cues that inevitably specify different slants in laboratory displays (Allison & Howard, 2000). We used a speed–accuracy tradeoff analysis to measure the time it takes to discriminate slant, allowing us to report psychometric functions as a function of response time. Observers reported which side of a slanted surface was farther, with a temporal deadline for responding that varied block-to-block. Stereoscopic slant discrimination rose above chance starting at 200 ms after stimulus onset. Unexpectedly, observers discriminated slant from binocular disparity faster than texture, and for stereoscopic whole-field stimuli faster than stereoscopic slant contrast stimuli. However, performance after the initial deviation from chance increased more rapidly for slant-contrast stimuli than whole-field stimuli. Discrimination latencies were similar for slants about the horizontal and vertical axes, but performance increased faster for slants about the vertical axis. Finally, slant from vertical disparity was somewhat slower than slant from horizontal disparity, which may reflect cue conflict. These results demonstrate, in contradiction with the previous literature, that the perception of slant from disparity happens very quickly—in fact, more quickly than the perception of slant from texture—and in comparable time to the simple perception of brightness from luminance.

A dataset of stereoscopic images and ground-truth disparity mimicking human fixations in peripersonal space

Binocular stereopsis is the ability of a visual system, belonging to a live being or a machine, to interpret the different visual information deriving from two eyes/cameras for depth perception. From this perspective, the ground-truth information about three-dimensional visual space, which is hardly available, is an ideal tool both for evaluating human performance and for benchmarking machine vision algorithms. In the present work, we implemented a rendering methodology in which the camera pose mimics realistic eye pose for a fixating observer, thus including convergent eye geometry and cyclotorsion. The virtual environment we developed relies on highly accurate 3D virtual models, and its full controllability allows us to obtain the stereoscopic pairs together with the ground-truth depth and camera pose information. We thus created a stereoscopic dataset: GENUA PESTO-GENoa hUman Active fixation database: PEripersonal space STereoscopic images and grOund truth disparity. The dataset aims to provide a unified framework useful for a number of problems relevant to human and computer vision, from scene exploration and eye movement studies to 3D scene reconstruction.

Effects of vertical muscle surgery on differences in the orientation of Listing's plane in patients with superior oblique palsy

BACKGROUND

Although scleral search coils are widely and accurately used for the measurement of Listing's plane in both eyes, they require specialized equipment and are invasive. In this study, we describe a convenient and less invasive method that uses a synoptometer to analyze the differences in orientation of Listing's plane (difLP), and the effects of vertical muscle surgery on the difLP tilt in patients with superior oblique palsy (SOP).

METHODS

Seventeen patients with unilateral congenital SOP (CSOP) and four patients with unilateral acquired SOP (ASOP) who had not undergone any strabismus surgeries were examined. Cyclodeviations of 13 vertical and horizontal gaze points within 30° were measured with a synoptometer, and the difLP tilts in the yaw and pitch planes were analyzed before and after vertical muscle surgery.

RESULTS

The difLP tilt in the CSOP patients was significantly tilted nasally (p = 0.02) and forward on the lower side (p = 0.001), whereas that in ASOP patients tended to tilt temporally (p = 0.15). Ipsilateral inferior oblique recession (IOR) performed in seven CSOP patients tended to improve the difLP tilt in both the yaw (p = 0.07) and pitch (p = 0.09) planes, whereas contralateral inferior rectus recession (IRR) performed in three CSOP patients significantly improved the difLP tilt in the pitch plane (p = 0.015). The mean excyclodeviations in the 13 gaze points were significantly improved with both procedures (p < 0.0001 for both).

CONCLUSIONS

The difLP tilt in the SOP patients could be analyzed with a convenient and less invasive method using a synoptometer, and dissimilar difLP tilts were confirmed in the ASOP and CSOP patients. The results of this study suggest that both IOR and IRR are reasonable treatments for improving the difLP tilt in CSOP patients. IOR should be selected for patients with a steep preoperative difLP tilt to the nasal side, whereas IRR should be selected for patients with a gentle preoperative difLP tilt.

Integration of texture and disparity cues to surface slant in dorsal visual cortex

Reliable estimation of three-dimensional (3D) surface orientation is critical for recognizing and interacting with complex 3D objects in our environment. Human observers maximize the reliability of their estimates of surface slant by integrating multiple depth cues. Texture and binocular disparity are two such cues, but they are qualitatively very different. Existing evidence suggests that representations of surface tilt from each of these cues coincide at the single-neuron level in higher cortical areas. However, the cortical circuits responsible for 1) integration of such qualitatively distinct cues and 2) encoding the slant component of surface orientation have not been assessed. We tested for cortical responses related to slanted plane stimuli that were defined independently by texture, disparity, and combinations of these two cues. We analyzed the discriminability of functional MRI responses to two slant angles using multivariate pattern classification. Responses in visual area V3B/KO to stimuli containing congruent cues were more discriminable than those elicited by single cues, in line with predictions based on the fusion of slant estimates from component cues. This improvement was specific to congruent combinations of cues: incongruent cues yielded lower decoding accuracies, which suggests the robust use of individual cues in cases of large cue conflicts. These data suggest that area V3B/KO is intricately involved in the integration of qualitatively dissimilar depth cues.

Probabilistic combination of slant information: weighted averaging and robustness as optimal percepts

Depth perception involves combining multiple, possibly conflicting, sensory measurements to estimate the 3D structure of the viewed scene. Previous work has shown that the perceptual system combines measurements using a statistically optimal weighted average. However, the system should only combine measurements when they come from the same source. We asked whether the brain avoids combining measurements when they differ from one another: that is, whether the system is robust to outliers. To do this, we investigated how two slant cues-binocular disparity and texture gradients-influence perceived slant as a function of the size of the conflict between the cues. When the conflict was small, we observed weighted averaging. When the conflict was large, we observed robust behavior: perceived slant was dictated solely by one cue, the other being rejected. Interestingly, the rejected cue was either disparity or texture, and was not necessarily the more variable cue. We modeled the data in a probabilistic framework, and showed that weighted averaging and robustness are predicted if the underlying likelihoods have heavier tails than Gaussians. We also asked whether observers had conscious access to the single-cue estimates when they exhibited robustness and found they did not, i.e. they completely fused despite the robust percepts.

3D surface orientation based on a novel representation of the orientation disparity field

The orientation disparity field from two orthographic views of an inclined planar surface patch (covered by straight lines) is analyzed, and a new tool to extract the patch orientation is provided: the function coupling the average orientation of each pair of corresponding surface contours with their orientation disparity. This function allows identifying the tilt of the surface, and two indeterminacy functions describing the set of surface inclinations (around the vertical and horizontal axes) over convergence angle values compatible with the orientation disparity field. Results of simulations show that the selection of inclination values matching the difference between the areas below the indeterminacy functions are consistent with some surface orientation effects found in psychophysical and computational experiments, like: the unbiased tilt vs. biased slant estimates, the slant underestimation, the surface orientation anisotropy, and the slant/tilt covariation.

Vertical-size disparities are temporally integrated for slant perception

We investigated temporal properties of vertical-size and horizontal-size disparity processing for slant perception. Subjects indicated perceived slants for a stereoscopic stimulus in which the two magnitudes of vertical-size or horizontal-size disparities were oscillated stepwise with various frequencies (from 0.2 to 10 Hz). For the stimulus with vertical-size disparity oscillation, two slants corresponding to the two magnitudes of disparity were perceived for low-frequency conditions, whereas only a static mean slant of the two slants was perceived for high frequencies (5 and 10 Hz). For the stimulus with horizontal-size disparity oscillation, two slants were perceived for all the temporal frequency conditions. These results indicate that temporal properties of vertical- and horizontal-size disparity processing are clearly different and vertical-size disparities are temporally integrated over a period of around 500 ms for slant perception.

Depth aftereffects mediated by vertical disparities: Evidence for vertical disparity driven calibration of extraretinal signals during stereopsis

Perceptual adaptation often results in a repulsive aftereffect: stimuli are seen as biased away from the adaptation stimulus (). Here we report the absence of a repulsive aftereffect for a vertical gradient of vertical disparity (or vertical size ratio, VSR). We exposed observers to a binocular stimulus consisting of horizontal lines. This stimulus contains vertical, but not horizontal disparities. The visual system was able to measure the VSR of this stimulus: although the lines themselves always appeared unslanted, the VSR carried by the lines had a dramatic effect on the apparent slant of a horizontal row of dots, as predicted by recent accounts of Ogle's (1938) induced effect (e.g., Backus, Banks, van Ee, & Crowell, 1999). Yet we observed no repulsive aftereffect for the VSR signal: after adaptation to horizontal lines that were vertically larger in one eye, we found an attractive aftereffect, the magnitude of which was largest in stimuli that did not contain a VSR signal. We interpret these results as a case of recalibration: disagreement between extra-retinal eye position signals (EP) and VSR causes a recalibration in the use of EP as used in the stereoscopic perception of slant.

Anticipating the three-dimensional consequences of eye movements

Rapid eye movements called saccades give rise to sudden, enormous changes in optic information arriving at the eye; how the world nonetheless appears stable is known as the problem of spatial constancy. One consequence of saccades is that the directions of all visible points shift uniformly; directional or 2D constancy, the fact that we do not perceive this change, has received extensive study for over a century. The problems raised by 3D consequences of saccades, on the other hand, have been neglected. When the eye rotates in space, the 3D orientation of all stationary surfaces undergoes an equal-and-opposite rotation with respect to the eye. When presented with a an optic simulation of a saccade but with the eyes still, observers readily perceive this depth rotation of surfaces; when simultaneously performing the corresponding saccade, the 3D orientations of surfaces are perceived as stable, a phenomenon I propose calling 3D spatial constancy. In experiments presented here, observers viewed ambiguous 3D rotations immediately before, during, or after a saccade. The results show that before the eyes begin to move the brain anticipates the 3D consequences of saccades, preferring to perceive the rotation opposite to the impending eye movement. Further, the anticipation is absent when observers fixate while experiencing optically simulated saccades, and therefore must be evoked by extraretinal signals. Such anticipation could provide a mechanism for 3D spatial constancy and transsaccadic integration of depth information.

Perception can influence the vergence responses associated with open-loop gaze shifts in 3D

We sought to determine if perceived depth can elicit vergence eye movements independent of binocular disparity. A flat surface in the frontal plane appears slanted about a vertical axis when the image in one eye is vertically compressed relative to the image in the other eye: the induced size effect (Ogle, 1938). We show that vergence eye movements accompany horizontal gaze shifts across such surfaces, consistent with the direction of the perceived slant, despite the absence of a horizontal disparity gradient. All images were extinguished during the gaze shifts so that eye movements were executed open-loop. We also used vertical compression of one eye's image to null the perceived slant resulting from prior horizontal compression of that image, and show that this reduces the vergence accompanying horizontal gaze shifts across the surface, even though the horizontal disparity is unchanged. When this last experiment was repeated using vertical expansions in place of the vertical compressions, the perceived slant was increased and so too was the vergence accompanying horizontal gaze shifts, although the horizontal disparity again remained unchanged. We estimate that the perceived depth accounted, on average, for 15-41% of the vergence in our experiments depending on the conditions.

Geometric and induced effects in binocular stereopsis and motion parallax

This paper examines and contrasts motion-parallax analogues of the induced-size and induced-shear effects with the equivalent induced effects from binocular disparity. During lateral head motion or with binocular stereopsis, vertical-shear and vertical-size transformations produced 'induced effects' of apparent inclination and slant that are not predicted geometrically. With vertical head motion, horizontal-shear and horizontal-size transformations produced similar analogues of the disparity induced effects. Typically, the induced effects were opposite in direction and slightly smaller in size than the geometric effects. Local induced-shear and induced-size effects could be elicited from motion parallax, but not from disparity, and were most pronounced when the stimulus contained discontinuities in velocity gradient. The implications of these results are discussed in the context of models of depth perception from disparity and structure from motion.

Vertical disparity can alter perceived direction

It has been well established that vertical disparity is involved in perception of the three-dimensional layout of a visual scene. The goal of this paper was to examine whether vertical disparities can alter perceived direction. We dissociated the common relationship between vertical disparity and the stimulus direction by applying a vertical magnification to the image presented to one eye. We used a staircase paradigm to measure whether perceived straight-ahead depended on the amount of vertical magnification in the stimulus. Subjects judged whether a test dot was flashed to either the left or the right side of straight-ahead. We found that perceived straight-ahead did indeed depend on the amount of vertical magnification but only after subjects adapted (for 5 min) to vertical scale (and only in five out of nine subjects). We argue that vertical disparity is a factor in the calibration of the relationship between eye-position signals and perceived direction.

Perceptual learning without feedback and the stability of stereoscopic slant estimation

Subjects were examined for practice effects in a stereoscopic slant-estimation task involving surfaces that comprised a large portion of the visual field. In most subjects slant estimation was significantly affected by practice, but only when an isolated surface (an absolute disparity gradient) was present in the visual field. When a second, unslanted, surface was visible (providing a second disparity gradient and thereby also a relative disparity gradient) none of the subjects exhibited practice effects. Apparently, stereoscopic slant estimation is more robust or stable over time in the presence of a second surface than in its absence. In order to relate the practice effects, which occurred without feedback, to perceptual learning, results are interpreted within a cue-interaction framework. In this paradigm the contribution of a cue depends on its reliability. It is suggested that normally absolute disparity gradients contribute relatively little to perceived slant and that subjects learn to increase this contribution by utilizing proprioceptive information. It is argued that--given the limited computational power of the brain--a relatively small contribution of absolute disparity gradients in perceived slant enhances the stability of stereoscopic slant perception.

Independent mechanisms produce visually perceived eye level (VPEL) and perceived visual pitch (PVP)

Two aspects of the perception of extrapersonal space undergo systematic changes with variations in the pitch of the visual environment: (1) the physical elevation perceived to correspond to eye level (VPEL); and (2) the perception of the pitch of the visual environment (PVP). Thus, one might assume that both discriminations are controlled by a common mechanism utilizing visual information from the pitched surface. In fact this assumption has been made frequently, and - in different forms - underlies three substantial but very different historical streams in the literature. A quantitative theoretical development shows that two of these streams, although derived from very different viewpoints and appearing very different themselves (it is assumed that the basis for both PVP and VPEL is information about the pitch of the visual field in one, and information about the location of the subject's eye level within the visual field in the other), make identical predictions: each requires that the weighted sum of PVP and VPEL equal the magnitude of physical pitch and that the weighted sum of their first derivatives equal a constant. The third stream, which assumes that an internal representation of the visual field gives rise to both PVP and VPEL, requires that a weighted difference of PVP and VPEL be proportional to physical pitch and that the weighted difference of their derivatives equal a constant. In an experiment designed to examine the relation between VPEL and PVP, psychophysical measurements of VPEL and PVP were made on 20 subjects across a range of pitches from -30 degrees to +20 degrees. Contrary to the predictions from all three interpretations, we find no significant correlation between the two perceptual variables when the influence of pitch itself is removed, despite the fact that VPEL and PVP each increased systematically with increasing visual field pitch. The results not only rule out the specific predictions derived from all three historical streams, they also rule out any theoretical viewpoint that requires control of both perceptual responses by a single mechanism. The statistical independence between VPEL and PVP implies independence between the mechanisms that give rise to them. The correlation observed here and elsewhere between individual PVP and VPEL settings when the influence of the systematic variation of pitch is not eliminated is a consequence of the way in which variations in the two perceptions are generated experimentally, and not on an identity of the mechanisms mediating the generation of the two perceptual variables themselves.

Temporal dependencies in resolving monocular and binocular cue conflict in slant perception

Observers viewed large dichoptic patterns undergoing smooth temporal modulations or step changes in simulated slant or inclination under various conditions of disparity-perspective cue conflict and concordance. After presentation of each test surface, subjects adjusted a comparison surface to match the perceived slant or inclination of the test surface. Addition of conflicting perspective to disparity affected slant and inclination perception more for brief than for long presentations. Perspective had more influence for smooth temporal changes than for step changes in slant or inclination and for surfaces presented in isolation rather than with a zero disparity frame. These results indicate that conflicting perspective information plays a dominant role in determining the temporal properties of perceived slant and inclination.

The influence of large scanning eye movements on stereoscopic slant estimation of large surfaces

The results of several experiments demonstrate that the estimated magnitude of perceived slant of large stereoscopic surfaces increases with the duration of the presentation. In these experiments, subjects were free to make eye movements. A possible explanation for the increase is that the visual system needs to scan the stimulus with eye movements (which take time) before it can make a reliable estimate of slant. We investigated the influence of large scanning eye movements on stereoscopic slant estimation of large surfaces. Six subjects estimated the magnitude of slant about the vertical or horizontal axis induced by large-field stereograms of which one half-image was transformed by horizontal scale, horizontal shear, vertical scale, vertical shear, divergence or rotation relative to the other half-image. The experiment was blocked in three sessions. Each session was devoted to one of the following fixation strategies: central fixation, peripheral (20 deg) fixation and active scanning of the stimulus. The presentation duration in each of the sessions was 0.5, 2 or 8 s. Estimations were done with and without a visual reference. The magnitudes of estimated slant and the perceptual biases were not significantly influenced by the three fixation strategies. Thus, our results provide no support for the hypothesis that the time used for the execution of large scanning eye movements explains the build-up of estimated slant with the duration of the stimulus presentation.

A computational model of depth perception based on headcentric disparity

It is now well established that depth is coded by local horizontal disparity and global vertical disparity. We present a computational model which explains how depth is extracted from these two types of disparities. The model uses the two (one for each eye) headcentric directions of binocular targets, derived from retinal signals and oculomotor signals. Headcentric disparity is defined as the difference between headcentric directions of corresponding features in the left and right eye's images. Using Helmholtz's coordinate systems we decompose headcentric disparity into azimuthal and elevational disparity. Elevational disparities of real objects are zero if the signals which contribute to headcentric disparity do not contain any errors. Azimuthal headcentric disparity is a 1D quantity from which an exact equation relating distance and disparity can be derived. The equation is valid for all headcentric directions and for all binocular fixation positions. Such an equation does not exist if disparity is expressed in retinal coordinates. Possible types of errors in oculomotor signals (six) produce global elevational disparity fields which are characterised by different gradients in the azimuthal and elevational directions. Computations show that the elevational disparity fields uniquely characterise both the type and size of the errors in oculomotor signals. Our model uses a measure of the global elevational disparity field together with local azimuthal disparity to accurately derive headcentric distance throughout the visual field. The model explains existing data on whole-field disparity transformations as well as hitherto unexplained aspects of stereoscopic depth perception.