The interaction of binocular disparity and motion parallax in the computation of depth

The role of binocular disparity and attention in the neural representation of multiple moving stimuli in the visual cortex

Segmenting visual scenes into distinct objects and surfaces is a fundamental visual process, with stereoscopic depth and motion serving as crucial cues. However, how the visual system uses these cues to segment multiple objects is not fully understood. We investigated how neurons in the middle-temporal (MT) cortex of macaque monkeys represent overlapping surfaces at different depths, moving in different directions. Neuronal activity was recorded from three male monkeys during discrimination tasks under varying attention conditions. We found that neuronal responses to overlapping surfaces showed a robust bias toward the binocular disparity of one surface over the other. The disparity bias of a neuron was positively correlated with the neuron's disparity preference for a single surface. In two animals, neurons preferring near disparities of single surfaces (near neurons) showed a near bias for overlapping stimuli, while neurons preferring far disparities (far neurons) showed a far bias. In the third animal, both near and far neurons displayed a near bias, though the near neurons showed a stronger near bias. All three animals exhibited an initial near bias across neurons relative to the average of the responses to the individual surfaces. Although attention modulated neuronal responses, the disparity bias was not caused by attention. We also found that the effect of attention was consistent with object-based, rather than feature-based attention. We proposed a model in which the pool size of the neuron population that weighs the responses to individual stimulus components can be variable. This model is a novel extension of the standard normalization model and provides a unified explanation for the disparity bias across animals. Our results reveal how MT neurons encode multiple stimuli moving at different depths and present new evidence of response modulation by object-based attention. The disparity bias allows subgroups of neurons to preferentially represent individual surfaces of multiple stimuli at different depths, thereby facilitating segmentation.

Effect of binocular disparity on learning anatomy with stereoscopic augmented reality visualization: A double center randomized controlled trial

Binocular disparity provides one of the important depth cues within stereoscopic three-dimensional (3D) visualization technology. However, there is limited research on its effect on learning within a 3D augmented reality (AR) environment. This study evaluated the effect of binocular disparity on the acquisition of anatomical knowledge and perceived cognitive load in relation to visual-spatial abilities. In a double-center randomized controlled trial, first-year (bio)medical undergraduates studied lower extremity anatomy in an interactive 3D AR environment either with a stereoscopic 3D view (n = 32) or monoscopic 3D view (n = 34). Visual-spatial abilities were tested with a mental rotation test. Anatomical knowledge was assessed by a validated 30-item written test and 30-item specimen test. Cognitive load was measured by the NASA-TLX questionnaire. Students in the stereoscopic 3D and monoscopic 3D groups performed equally well in terms of percentage correct answers (written test: 47.9 ± 15.8 vs. 49.1 ± 18.3; P = 0.635; specimen test: 43.0 ± 17.9 vs. 46.3 ± 15.1; P = 0.429), and perceived cognitive load scores (6.2 ± 1.0 vs. 6.2 ± 1.3; P = 0.992). Regardless of intervention, visual-spatial abilities were positively associated with the specimen test scores (η² = 0.13, P = 0.003), perceived representativeness of the anatomy test questions (P = 0.010) and subjective improvement in anatomy knowledge (P < 0.001). In conclusion, binocular disparity does not improve learning anatomy. Motion parallax should be considered as another important depth cue that contributes to depth perception during learning in a stereoscopic 3D AR environment.

Scene-relative object motion biases depth percepts

An important function of the visual system is to represent 3D scene structure from a sequence of 2D images projected onto the retinae. During observer translation, the relative image motion of stationary objects at different distances (motion parallax) provides potent depth information. However, if an object moves relative to the scene, this complicates the computation of depth from motion parallax since there will be an additional component of image motion related to scene-relative object motion. To correctly compute depth from motion parallax, only the component of image motion caused by self-motion should be used by the brain. Previous experimental and theoretical work on perception of depth from motion parallax has assumed that objects are stationary in the world. Thus, it is unknown whether perceived depth based on motion parallax is biased by object motion relative to the scene. Naïve human subjects viewed a virtual 3D scene consisting of a ground plane and stationary background objects, while lateral self-motion was simulated by optic flow. A target object could be either stationary or moving laterally at different velocities, and subjects were asked to judge the depth of the object relative to the plane of fixation. Subjects showed a far bias when object and observer moved in the same direction, and a near bias when object and observer moved in opposite directions. This pattern of biases is expected if subjects confound image motion due to self-motion with that due to scene-relative object motion. These biases were large when the object was viewed monocularly, and were greatly reduced, but not eliminated, when binocular disparity cues were provided. Our findings establish that scene-relative object motion can confound perceptual judgements of depth during self-motion.

Associations Between Binocular Depth Perception and Performance Gains in Laparoscopic Skill Acquisition

The ability to perceive differences in depth is important in many daily life situations. It is also of relevance in laparoscopic surgical procedures that require the extrapolation of three-dimensional visual information from two-dimensional planar images. Besides visual-motor coordination, laparoscopic skills and binocular depth perception are demanding visual tasks for which learning is important. This study explored potential relations between binocular depth perception and individual variations in performance gains during laparoscopic skill acquisition in medical students naïve of such procedures. Individual differences in perceptual learning of binocular depth discrimination when performing a random dot stereogram (RDS) task were measured as variations in the slope changes of the logistic disparity psychometric curves from the first to the last blocks of the experiment. The results showed that not only did the individuals differ in their depth discrimination; the extent with which this performance changed across blocks also differed substantially between individuals. Of note, individual differences in perceptual learning of depth discrimination are associated with performance gains from laparoscopic skill training, both with respect to movement speed and an efficiency score that considered both speed and precision. These results indicate that learning-related benefits for enhancing demanding visual processes are, in part, shared between these two tasks. Future studies that include a broader selection of task-varying monocular and binocular cues as well as visual-motor coordination are needed to further investigate potential mechanistic relations between depth perceptual learning and laparoscopic skill acquisition. A deeper understanding of these mechanisms would be important for applied research that aims at designing behavioral interventions for enhancing technology-assisted laparoscopic skills.

Cue vetoing in depth estimation: Physical and virtual stimuli

Motion parallax and binocular disparity contribute to the perceived depth of three-dimensional (3D) objects. However, depth is often misperceived, even when both cues are available. This may be due in part to conflicts with unmodelled cues endemic to computerized displays. Here we evaluated the impact of display-based cue conflicts on depth cue integration by comparing perceived depth for physical and virtual objects. Truncated square pyramids were rendered using Blender and 3D printed. We assessed perceived depth using a discrimination task with motion parallax, binocular disparity, and their combination. Physical stimuli were presented with precise control over position and lighting. Virtual stimuli were viewed using a head-mounted display. To generate motion parallax, observers made lateral head movements using a chin rest on a motion platform. Observers indicated if the width of the front face appeared greater or less than the distance between this surface and the base. We found that accuracy was similar for virtual and physical pyramids. All estimates were more precise when depth was defined by binocular disparity than motion parallax. Our probabilistic model shows that a linear combination model does not adequately describe performance in either physical or virtual conditions. While there was inter-observer variability in weights, performance in all conditions was best predicted by a veto model that excludes the less reliable depth cue, in this case motion parallax.

Relative Importance of Binocular Disparity and Motion Parallax for Depth Estimation: A Computer Vision Approach

Binocular disparity and motion parallax are the most important cues for depth estimation in human and computer vision. Here, we present an experimental study to evaluate the accuracy of these two cues in depth estimation to stationary objects in a static environment. Depth estimation via binocular disparity is most commonly implemented using stereo vision, which uses images from two or more cameras to triangulate and estimate distances. We use a commercial stereo camera mounted on a wheeled robot to create a depth map of the environment. The sequence of images obtained by one of these two cameras as well as the camera motion parameters serve as the input to our motion parallax-based depth estimation algorithm. The measured camera motion parameters include translational and angular velocities. Reference distance to the tracked features is provided by a LiDAR. Overall, our results show that at short distances stereo vision is more accurate, but at large distances the combination of parallax and camera motion provide better depth estimation. Therefore, by combining the two cues, one obtains depth estimation with greater range than is possible using either cue individually.

Image Segmentation Based on Relative Motion and Relative Disparity Cues in Topographically Organized Areas of Human Visual Cortex

The borders between objects and their backgrounds create discontinuities in image feature maps that can be used to recover object shape. Here we used functional magnetic resonance imaging to identify cortical areas that encode two of the most important image segmentation cues: relative motion and relative disparity. Relative motion and disparity cues were isolated by defining a central 2-degree disk using random-dot kinematograms and stereograms, respectively. For motion, the disk elicited retinotopically organized activations starting in V1 and extending through V2 and V3. In the surrounding region, we observed phase-inverted activations indicative of suppression, extending out to at least 6 degrees of retinal eccentricity. For disparity, disk activations were only found in V3, while suppression was observed in all early visual areas. Outside of early visual cortex, several areas were sensitive to both types of cues, most notably LO1, LO2 and V3B, making them additional candidate areas for motion- and disparity-cue combination. Adding an orthogonal task at fixation did not diminish these effects, and in fact led to small but measurable disk activations in V1 and V2 for disparity. The overall pattern of extra-striate activations is consistent with recent three-stream models of cortical organization.

Combination of Interaural Level and Time Difference in Azimuthal Sound Localization in Owls

A function of the auditory system is to accurately determine the location of a sound source. The main cues for sound location are interaural time (ITD) and level (ILD) differences. Humans use both ITD and ILD to determine the azimuth. Thus far, the conception of sound localization in barn owls was that their facial ruff and asymmetrical ears generate a two-dimensional grid of ITD for azimuth and ILD for elevation. We show that barn owls also use ILD for azimuthal sound localization when ITDs are ambiguous. For high-frequency narrowband sounds, midbrain neurons can signal multiple locations, leading to the perception of an auditory illusion called a phantom source. Owls respond to such an illusory percept by orienting toward it instead of the true source. Acoustical measurements close to the eardrum reveal a small ILD component that changes with azimuth, suggesting that ITD and ILD information could be combined to eliminate the illusion. Our behavioral data confirm that perception was robust against ambiguities if ITD and ILD information was combined. Electrophysiological recordings of ILD sensitivity in the owl’s midbrain support the behavioral findings indicating that rival brain hemispheres drive the decision to orient to either true or phantom sources. Thus, the basis for disambiguation, and reliable detection of sound source azimuth, relies on similar cues across species as similar response to combinations of ILD and narrowband ITD has been observed in humans.

The neural basis of depth perception from motion parallax

In addition to depth cues afforded by binocular vision, the brain processes relative motion signals to perceive depth. When an observer translates relative to their visual environment, the relative motion of objects at different distances (motion parallax) provides a powerful cue to three-dimensional scene structure. Although perception of depth based on motion parallax has been studied extensively in humans, relatively little is known regarding the neural basis of this visual capability. We review recent advances in elucidating the neural mechanisms for representing depth-sign (near versus far) from motion parallax. We examine a potential neural substrate in the middle temporal visual area for depth perception based on motion parallax, and we explore the nature of the signals that provide critical inputs for disambiguating depth-sign.This article is part of the themed issue 'Vision in our three-dimensional world'.

Dependence of Illusory Motion on Directional Consistency in Oblique Components

Pinna and Brelstaff (2000 Vision Research40 2091–2096) reported a motion illusion on viewing two concentric circles consisting of quadrangular components with black and white sides on a grey background. Our results suggest that the illusion is based on the integration of motion signals derived from oblique components, and on the consistency in the direction among those components. Furthermore, arrays of these oblique components can elicit the perception of motion not only for the oblique components themselves, but also for other objects in the picture. We propose that the motion illusion depends not only upon detection of the illusory motion signal at each local oblique component, but also upon the accumulation of the signal all over the stimulus configuration.

If you watch it move, you'll recognize it in 3D: Transfer of depth cues between encoding and retrieval

Viewing objects with stereoscopic displays provides additional depth cues through binocular disparity supporting object recognition. So far, it was unknown whether this results from the representation of specific stereoscopic information in memory or a more general representation of an object's depth structure. Therefore, we investigated whether continuous object rotation acting as depth cue during encoding results in a memory representation that can subsequently be accessed by stereoscopic information during retrieval. In Experiment 1, we found such transfer effects from continuous object rotation during encoding to stereoscopic presentations during retrieval. In Experiments 2a and 2b, we found that the continuity of object rotation is important because only continuous rotation and/or stereoscopic depth but not multiple static snapshots presented without stereoscopic information caused the extraction of an object's depth structure into memory. We conclude that an object's depth structure and not specific depth cues are represented in memory.

Differential responses in dorsal visual cortex to motion and disparity depth cues

We investigated how interactions between monocular motion parallax and binocular cues to depth vary in human motion areas for wide-field visual motion stimuli (110 × 100°). We used fMRI with an extensive 2 × 3 × 2 factorial blocked design in which we combined two types of self-motion (translational motion and translational + rotational motion), with three categories of motion inflicted by the degree of noise (self-motion, distorted self-motion, and multiple object-motion), and two different view modes of the flow patterns (stereo and synoptic viewing). Interactions between disparity and motion category revealed distinct contributions to self- and object-motion processing in 3D. For cortical areas V6 and CSv, but not the anterior part of MT(+) with bilateral visual responsiveness (MT(+)/b), we found a disparity-dependent effect of rotational flow and noise: When self-motion perception was degraded by adding rotational flow and moderate levels of noise, the BOLD responses were reduced compared with translational self-motion alone, but this reduction was cancelled by adding stereo information which also rescued the subject's self-motion percept. At high noise levels, when the self-motion percept gave way to a swarm of moving objects, the BOLD signal strongly increased compared to self-motion in areas MT(+)/b and V6, but only for stereo in the latter. BOLD response did not increase for either view mode in CSv. These different response patterns indicate different contributions of areas V6, MT(+)/b, and CSv to the processing of self-motion perception and the processing of multiple independent motions.

The integration of motion and disparity cues to depth in dorsal visual cortex

Humans exploit a range of visual depth cues to estimate three-dimensional structure. For example, the slant of a nearby tabletop can be judged by combining information from binocular disparity, texture and perspective. Behavioral tests show humans combine cues near-optimally, a feat that could depend on discriminating the outputs from cue-specific mechanisms or on fusing signals into a common representation. Although fusion is computationally attractive, it poses a substantial challenge, requiring the integration of quantitatively different signals. We used functional magnetic resonance imaging (fMRI) to provide evidence that dorsal visual area V3B/KO meets this challenge. Specifically, we found that fMRI responses are more discriminable when two cues (binocular disparity and relative motion) concurrently signal depth, and that information provided by one cue is diagnostic of depth indicated by the other. This suggests a cortical node important when perceiving depth, and highlights computations based on fusion in the dorsal stream.

Decoding conjunctions of direction-of-motion and binocular disparity from human visual cortex

Motion and binocular disparity are two features in our environment that share a common correspondence problem. Decades of psychophysical research dedicated to understanding stereopsis suggest that these features interact early in human visual processing to disambiguate depth. Single-unit recordings in the monkey also provide evidence for the joint encoding of motion and disparity across much of the dorsal visual stream. Here, we used functional MRI and multivariate pattern analysis to examine where in the human brain conjunctions of motion and disparity are encoded. Subjects sequentially viewed two stimuli that could be distinguished only by their conjunctions of motion and disparity. Specifically, each stimulus contained the same feature information (leftward and rightward motion and crossed and uncrossed disparity) but differed exclusively in the way these features were paired. Our results revealed that a linear classifier could accurately decode which stimulus a subject was viewing based on voxel activation patterns throughout the dorsal visual areas and as early as V2. This decoding success was conditional on some voxels being individually sensitive to the unique conjunctions comprising each stimulus, thus a classifier could not rely on independent information about motion and binocular disparity to distinguish these conjunctions. This study expands on evidence that disparity and motion interact at many levels of human visual processing, particularly within the dorsal stream. It also lends support to the idea that stereopsis is subserved by early mechanisms also tuned to direction of motion.

Enhancement of three-dimensional perception of numerical hologram reconstructions of real-world objects by motion and stereo

We investigated the question of how the perception of three-dimensional information reconstructed numerically from digital holograms of real-world objects, and presented on conventional displays, depends on motion and stereoscopic presentation. Perceived depth in an adjustable random pattern stereogram was matched to the depth in hologram reconstructions. The objects in holograms were a microscopic biological cell and a macroscopic metal coil. For control, we used real physical objects in additional to hologram reconstructions of real objects. Stereoscopic presentation increased perceived depth substantially in comparison to non-stereoscopic presentation. When stereoscopic cues were weak or absent e.g. because of blur, motion increased perceived depth considerably. However, when stereoscopic cues were strong, the effect of motion was small. In conclusion, for the maximization of perceived three-dimensional information of holograms on conventional displays, it seems highly beneficial to use the combination of motion and stereoscopic presentation.

Depth-cue integration in grasp programming: no evidence for a binocular specialism

When we grasp with one eye covered, the finger and thumb are typically opened wider than for binocularly guided grasps, as if to build a margin-for-error into the movement. Also, patients with visual form agnosia can have profound deficits in their (otherwise relatively normal) grasping when binocular information is removed. One interpretation of these findings is that there is a functional specialism for binocular vision in the control of grasping. Alternatively, cue-integration theory suggests that binocular and monocular depth cues are combined in the control of grasping, and so impaired performance reflects not the loss of 'critical' binocular cues, but increased uncertainty per se. Unfortunately, removing binocular information confounds removing particular (binocular) depth cues with an overall reduction in the available information, and so such experiments cannot distinguish between these alternatives. We measured the effects on visually open-loop grasping of selectively removing monocular (texture) or binocular depth cues. To allow meaningful comparisons, we made psychophysical measurements of the uncertainty in size estimates in each case, so that the informativeness of binocular and monocular cues was known in each condition. Consistent with cue-integration theory, removing either binocular or monocular cues resulted in similar increases in grip apertures. In a separate experiment, we also confirmed that changes in uncertainty per se (keeping the same depth cues available) resulted in larger grip apertures. Overall, changes in the margin-for-error in grasping movements were determined by the uncertainty in size estimates and not by the presence or absence of particular depth cues. Our data therefore argue against a binocular specialism for grasp programming. Instead, grip apertures were smaller when binocular and monocular cues were available than with either cue alone, providing strong evidence that the visuo-motor system exploits the redundancy available in multiple sources of information, and integrates binocular and monocular cues to improve grasping performance.

A neural representation of depth from motion parallax in macaque visual cortex

Perception of depth is a fundamental challenge for the visual system, particularly for observers moving through their environment. The brain makes use of multiple visual cues to reconstruct the three-dimensional structure of a scene. One potent cue, motion parallax, frequently arises during translation of the observer because the images of objects at different distances move across the retina with different velocities. Human psychophysical studies have demonstrated that motion parallax can be a powerful depth cue, and motion parallax seems to be heavily exploited by animal species that lack highly developed binocular vision. However, little is known about the neural mechanisms that underlie this capacity. Here we show, by using a virtual-reality system to translate macaque monkeys (Macaca mulatta) while they viewed motion parallax displays that simulated objects at different depths, that many neurons in the middle temporal area (area MT) signal the sign of depth (near versus far) from motion parallax in the absence of other depth cues. To achieve this, neurons must combine visual motion with extra-retinal (non-visual) signals related to the animal's movement. Our findings suggest a new neural substrate for depth perception and demonstrate a robust interaction of visual and non-visual cues in area MT. Combined with previous studies that implicate area MT in depth perception based on binocular disparities, our results suggest that area MT contains a more general representation of three-dimensional space that makes use of multiple cues.

A neural model for the integration of stereopsis and motion parallax in structure-from-motion

We introduce a model for the computation of structure-from-motion based on the physiology of visual cortical areas MT and MST. The model assumes that the perception of depth from motion is related to the firing of a subset of MT neurons tuned to both velocity and disparity. The model's MT neurons are connected to each other laterally to form modulatory receptive-field surrounds that are gated by feedback connections from area MST. This allows the building up of a depth map from motion in area MT, even in absence of disparity in the input. Depth maps from motion and from stereo are combined by a weighted average at a final stage. The model's predictions for the interaction between motion and stereo cues agree with previous psychophysical data, both when the cues are consistent with each other or when they are contradictory. In particular, the model shows nonlinearities as a result of early interactions between motion and stereo before their depth maps are averaged. The two cues interact in a way that represents an alternative to the "modified weak fusion" model of depth-cue combination.

Apparent position in depth of stationary moving three-dimensional objects

Motion signals contained within a stationary object projected on the fronto-parallel plane shift the object's apparent spatial position in the direction of the motion [see De Valois, R. L., & De Valois, K. K. (1991). Vernier acuity with stationary moving Gabors. Vision Research, 31(9), 1619-1626]. We report an analogous apparent position shift of three-dimensional objects that contain local elements that move in depth. Our stimulus was a transparent three-dimensional cylinder defined by 150 limited-lifetime dots, oriented such that it was end on and its tangent plane was circular. Dots moved in depth by changes in their binocular disparities. In the first experiment, observers judged the positions of the near and far ends of the cylinder, by moving marker lines in depth, for different dot speeds. The results showed that when dots moved towards the observer, the perceived location of the two ends of the cylinder appeared closer in depth. When dots moved away from the observer, the opposite effect was produced. Additionally, the amount of apparent position shift produced was dependent on dot speed, with faster speeds producing larger positional offsets. However, we found in the second experiment that when the cylinder contained randomly moving dots, or when the cylinder contained equal amounts of dots moving towards and away from the observer, positional shifts were very much reduced, or abolished. Our findings suggest that motion signals can induce a misperception of position in depth that is similar manner to that produced by motion within an object in the two-dimensional image plane.

Visual learning by cue-dependent and cue-invariant mechanisms

We examined learning at multiple levels of the visual system. Subjects were trained and tested on a same/different slant judgment task or a same/different curvature judgment task using simulated planar surfaces or curved surfaces defined by either stereo or monocular (texture and motion) cues. Taken as a whole, the results of four experiments are consistent with the hypothesis that learning takes place at both cue-dependent and cue-invariant levels, and that learning at these levels can have different generalization properties. If so, then cue-invariant mechanisms may mediate the transfer of learning from familiar cue conditions to novel cue conditions, thereby allowing perceptual learning to be robust and efficient. We claim that learning takes place at multiple levels of the visual system, and that a comprehensive understanding of visual perception requires a good understanding of learning at each of these levels.

Surface orientation, modulation frequency and the detection and perception of depth defined by binocular disparity and motion parallax

Binocular disparity and motion parallax provide information about the spatial structure and layout of the world. Descriptive similarities between the two cues have often been noted which have been taken as evidence of a close relationship between them. Here, we report two experiments which investigate the effect of surface orientation and modulation frequency on (i) a threshold detection task and (ii) a supra-threshold depth-matching task using sinusoidally corrugated surfaces defined by binocular disparity or motion parallax. For low frequency corrugations, an orientation anisotropy was observed in both domains, with sensitivity decreasing as surface orientation was varied from horizontal to vertical. In the depth-matching task, for surfaces defined by binocular disparity the greatest depth was seen for oblique orientations. For surfaces defined by motion parallax, perceived depth was found to increase as surface orientation was varied from horizontal to vertical. In neither case was perceived depth for supra-threshold surfaces related to threshold performance in any simple manner. These results reveal clear differences between the perception of depth from binocular disparity or motion parallax, and between perception at threshold and supra-threshold levels of performance.

Gaze tracking accuracy in humans: Two eyes are better than one

Many people viewing a small distant object report the use of a dominant eye, which may change when viewing the same object in right or left viewing fields. This study was designed to determine if the accuracy in left and right gaze recordings from a video-based binocular eye tracking system with eye-head integration would change when observers looked at targets in different viewing fields. We also wanted to determine whether system accuracy was best using the dominant eye, the average of both eyes or an alignment switching strategy for various viewing fields. Bilateral eye and head movements were recorded to determine point of gaze (POG) for subjects viewing targets with the head neutral or rotated left or right with a stationary body (static) or while stepping toward a target and back (dynamic). Constant, absolute and variable POG and rotational errors were determined. Results showed that although errors were highest in the static condition for POG errors and in the dynamic condition for rotational errors, absolute rotational angle errors were similar for static and dynamic conditions, indicating that subject to object distance played an important role in determining POG errors. Furthermore, errors were not consistently lower for a given eye or for a given eye within a specific viewing field for the head rotations performed. Since low errors across the viewing fields were found by averaging left and right errors, it was concluded that this average and should be used in future studies involving relatively small eye-in-head rotations.

Stereoscopic depth aftereffects without retinal position correspondence between adaptation and test stimuli

To clarify whether stereo-slant aftereffects are independent of stimulated retinal position, two experiments compared the magnitude of aftereffects between the following two conditions: when the adaptation and test stimulus fell on (1) the same retinal position, and (2) on different retinal positions separated by 0.5 degrees -20 degrees . In Experiment 1, disc- or ring-shaped surface consisting of random-dots was presented at the central or peripheral visual fields. In Experiment 2, rectangular surface was presented at the upper or lower visual fields. After two minutes inspection of a random-dot stereogram depicting a +/-30 degrees slanted surface, the observer adjusted the slant of the test stimulus to appear fronto-parallel. The results of the experiments showed that significant aftereffects were observed similarly in both conditions. Moreover, the separation nor the stimulus shape scarcely affected the magnitude of the aftereffects. Based on these results we concluded that the depth processing mechanism which operates independently from the stimulated retinal position is responsible for the depth aftereffects we found.

Systematic distortions of perceptual stability investigated using immersive virtual reality

Using an immersive virtual reality system, we measured the ability of observers to detect the rotation of an object when its movement was yoked to the observer's own translation. Most subjects had a large bias such that a static object appeared to rotate away from them as they moved. Thresholds for detecting target rotation were similar to those for an equivalent speed discrimination task carried out by static observers, suggesting that visual discrimination is the predominant limiting factor in detecting target rotation. Adding a stable visual reference frame almost eliminated the bias. Varying the viewing distance of the target had little effect, consistent with observers underestimating distance walked. However, accuracy of walking to a briefly presented visual target was high and not consistent with an underestimation of distance walked. We discuss implications for theories of a task-independent representation of visual space.

Motion repulsion is monocular

Motion repulsion is the illusory enlargement of the angle between objects moving in two different directions of motion. Previous work suggests that motion repulsion occurs under dichoptic conditions, and therefore is binocular. In reference repulsion the direction of motion is misperceived even if only a single direction of motion is presented. In an experiment I show that repulsion under dichoptic conditions is correlated with reference repulsion, but not with binocular motion repulsion. This suggests that motion repulsion proper, which occurs over and beyond reference repulsion, does not occur under dichoptic conditions, implying that motion repulsion is monocular.

Binocular correlation does not improve coherence detection for fronto-parallel motion

We studied the low-level interactions between motion coherence detection and binocular correlation detection. It is well-established that e.g. depth information from motion parallax and from binocular disparities is effectively integrated. The question we aimed to answer is whether such interactions also exist at the very first correlation level that both mechanisms might have in common. First we quantitatively compared motion coherence detection and binocular correlation detection using similar stimuli (random pixels arrays, RPAs) and the same noise masking paradigm (luminance signal to noise ratio, LSNR). This showed that human observers are much more sensitive to motion than to binocular correlation. Adding noise therefore has a much stronger effect on binocular correlation than on motion detection. Next we manipulated the shape of the stimulus aperture to equalize LSNR thresholds for motion and binocular correlation. Motion sensitivity could be progressively reduced by shortening the length of the motion path, while keeping the aperture area constant. Changing the shape of the aperture did not affect binocular correlation sensitivity. A 'balanced' stimulus, one with equal strengths of motion and binocular correlation signals was then used to study the mutual interactions. In accordance with previous results, motion was found to greatly facilitate binocular correlation. Binocular correlation, however did not facilitate motion detection. We conclude that interactions are asymmetrical; fronto-parallel motion is primarily detected monocularly and this information can then be used to facilitate binocular correlation, but binocular correlation cannot improve motion sensitivity.

Integration of binocular disparity and monocular cues at near threshold level

We examined the dependency of the integration of multiple depth cues upon the combined cues and upon the consistency of depth information from different cues. For each observer, depth thresholds were measured by the use of stimuli in which different depth cues (motion parallax, binocular disparity, and monocular configuration) specified the surface undulating sinusoidally with different spatial frequencies and different phases. Analysis of d(') showed that the performance was better than the prediction of probability summation only when parallax and disparity cues specified an undulation with the same spatial frequency and same phase. The probability summation model overestimated the performance for the other conditions of combination of disparity and parallax, and for all of the conditions of combination of disparity and monocular configuration. These results suggest that the improvement in depth perception caused by integration of multiple cues depends on the type of combined cues, and that the visual system possibly integrates the depth information from different cues at different stages of the visual processing.

Information integration in judgements of time to contact

Time to contact (TTC) is specified optically by tau, and studies suggest that observers are sensitive to this information. However, TTC judgements also are influenced by other sources of information, including pictorial depth cues. Therefore, it is useful to identify these sources of information and to determine whether and how their effects combine when multiple sources are available. We evaluated the effect of five depth cues on TTC judgements. Results indicate that relative size, height in field, occlusion, and motion parallax influence TTC judgements. When multiple cues are available, an integration (rather than selection) strategy is used. Finally, the combined effects of multiple cues are not always consistent with a strict additive model and may be task dependent.

Geometric and induced effects in binocular stereopsis and motion parallax

This paper examines and contrasts motion-parallax analogues of the induced-size and induced-shear effects with the equivalent induced effects from binocular disparity. During lateral head motion or with binocular stereopsis, vertical-shear and vertical-size transformations produced 'induced effects' of apparent inclination and slant that are not predicted geometrically. With vertical head motion, horizontal-shear and horizontal-size transformations produced similar analogues of the disparity induced effects. Typically, the induced effects were opposite in direction and slightly smaller in size than the geometric effects. Local induced-shear and induced-size effects could be elicited from motion parallax, but not from disparity, and were most pronounced when the stimulus contained discontinuities in velocity gradient. The implications of these results are discussed in the context of models of depth perception from disparity and structure from motion.

Modification of depth and distance perception caused by long-term wearing of left-right reversing spectacles

For 35 to 39 days, four observers wore continuously left-right reversing spectacles which pseudoscopically reverse the order of binocular disparity and direction of convergence. In three tests, we investigated how the visual system copes with the transformation of depth and distance information due to the reversing spectacles. In stereogram observation, after a few days of wearing the spectacles. the observers sometimes perceived a depth order which was opposite to the depth order that they had perceived in the pre-spectacle-wearing period. Monocular depth cues contributed more to depth perception in the spectacle-wearing period than they did in the pre-spectacle-wearing period. While the perceived distance significantly decreased during the spectacle-wearing period, we found no evidence of adaptive change in distance perception. The results indicate that the visual system adapts itself to the transformed situation by not only changing the processing of disparity but also by changing the relative efficiency of each cue in determining apparent depth.

Effects of luminance and color on apparent motion

The present study explored the effects of luminance, color, and distance on apparent motion direction of some repeated stimuli of alternating colors. Two experiments were performed to specify what affects apparent motion direction when the stroboscopic alternative is between proximity and similarity in color. In Exp. 1 the influence of luminance on apparent motion was systematically studied with seven different background luminance values. Analysis pointed out that apparent motion direction depends on the discriminability among the disks, in relation also to the background values. A formula to predict the apparent motion direction from luminance values was proposed. In Exp. 2 the influence of hue on apparent motion was studied. Stimuli with equiluminant disks on different backgrounds were used. Analysis pointed out that the stroboscopic unification was based mainly on proximity and that discriminability between the disks was rather poor, especially when apparent motion occurred also on an equiluminant background, as already pointed out by Ramachandran and Gregory in 1978. A different influence on apparent motion of the red-blue and red-green disks, as opposed to that of the blue-green ones, was also noted.

The visual control of reaching and grasping: binocular disparity and motion parallax

The primary visual sources of depth and size information are binocular cues and motion parallax. Here, the authors determine the efficacy of these cues to control prehension by presenting them in isolation from other visual cues. When only binocular cues were available, reaches showed normal scaling of the transport and grasp components with object distance and size. However, when only motion parallax was available, only the transport component scaled reliably. No additional increase in scaling was found when both cues were available simultaneously. Therefore, although equivalent information is available from binocular and motion parallax information, the latter may be of relatively limited use for the control of the grasp. Binocular disparity appears selectively important for the control of the grasp.

Adaptation to vertical disparity induced-depth: implications for disparity processing

Depth aftereffects produced by prolonged inspection of an object in depth can be mediated by monocular and binocular depth cues. The adapting mechanisms responsible for such effects have not yet been fully determined. Theories of binocular depth aftereffects typically posit a role of an adaptive horizontal disparity sensitive mechanism, implying multiple cue-specific mechanisms for depth aftereffects. Here we examined whether binocular depth aftereffects can be attributed to such a cue-specific mechanism. In Experiment 1 we did so using a technique allowing us to maintain horizontal disparities and vergence constant for our adaptation stimuli, whilst manipulating simulated depth by virtue of a vertical disparity induced-depth effect. We found that depth aftereffects were almost identical to those produced by adaptation to stimuli of equivalent depth produced by conventional horizontal disparity modulations. In Experiment 2, we examined depth aftereffects following adaptation to apparently frontal surfaces produced by different combinations of horizontal and vertical disparity modulations. Aftereffects were close to zero. These results suggest that binocular depth aftereffects are not due to adaptation of a horizontal disparity sensitive mechanism, and we argue that adaptation occurs at the level of a 3D shape sensitive mechanism derived from multiple cues. Experiment 3 was a control to examine whether the two types of adaptation stimuli in Experiment 1 were indeed perceptually the same, since in theory they may differ if vertical disparities influenced metric depth scaling. We found no evidence of this, and concluded that the two classes of stimuli used in Experiment 1, though consisting of very different patterns of disparity, were perceptually equivalent.

Psychoanatomical substrates of Bálint's syndrome

OBJECTIVES

From a series of glimpses, we perceive a seamless and richly detailed visual world. Cerebral damage, however, can destroy this illusion. In the case of Bálint's syndrome, the visual world is perceived erratically, as a series of single objects. The goal of this review is to explore a range of psychological and anatomical explanations for this striking visual disorder and to propose new directions for interpreting the findings in Bálint's syndrome and related cerebral disorders of visual processing.

METHODS

Bálint's syndrome is reviewed in the light of current concepts and methodologies of vision research.

RESULTS

The syndrome affects visual perception (causing simultanagnosia/visual disorientation) and visual control of eye and hand movement (causing ocular apraxia and optic ataxia). Although it has been generally construed as a biparietal syndrome causing an inability to see more than one object at a time, other lesions and mechanisms are also possible. Key syndrome components are dissociable and comprise a range of disturbances that overlap the hemineglect syndrome. Inouye's observations in similar cases, beginning in 1900, antedated Bálint's initial report. Because Bálint's syndrome is not common and is difficult to assess with standard clinical tools, the literature is dominated by case reports and confounded by case selection bias, non-uniform application of operational definitions, inadequate study of basic vision, poor lesion localisation, and failure to distinguish between deficits in the acute and chronic phases of recovery.

CONCLUSIONS

Studies of Bálint's syndrome have provided unique evidence on neural substrates for attention, perception, and visuomotor control. Future studies should address possible underlying psychoanatomical mechanisms at "bottom up" and "top down" levels, and should specifically consider visual working memory and attention (including object based attention) as well as systems for identification of object structure and depth from binocular stereopsis, kinetic depth, motion parallax, eye movement signals, and other cues.

The design of telepresence systems: the task-dependent use of binocular disparity and motion parallax

The effect of different visual depth cues presented through a head-mounted display in a dark (no pictorial cue) environment was investigated. The relative effects of binocular disparity, motion parallax, and a combination of the 2, were assessed for 3 tasks at 2 viewing distances. These tasks (which varied in the minimum amount of information they required) were a nulling task, setting a triangle to be equilateral and matching the base-to-apex magnitude of 2 triangles at different distances. Performance within the tasks varied considerably but was most accurate for the nulling task. Differences between viewing conditions may be due to a failure in the assessment of absolute viewing distance. It is argued that these results are task specific. Although there was some variation between different cue types, they appear to be largely interchangeable within the tasks. These results have implications for system designers selecting an appropriate display device for a telepresence system.

3D after-effects are due to shape and not disparity adaptation

There are a variety of stereoscopic after-effects in which exposure to a stimulus with a particular slant or curvature affects the perceived slant or curvature of a subsequently presented stimulus. These after-effects have been explained as a consequence of fatigue (a decrease in responsiveness) among neural mechanisms that are tuned to particular disparities or patterns of disparity. In fact, a given disparity pattern is consistent with numerous slants or curvatures; to determine slant or curvature, the visual system must take the viewing distance into account. We took advantage of this property to examine whether the mechanisms underlying the stereoscopic curvature after-effect are tuned to particular disparity patterns or to some other property such as surface curvature. The results clearly support the second hypothesis. Thus, 3D after-effects appear to be caused by adaptation among mechanisms specifying surface shape rather than among mechanisms signaling the disparity pattern.

Perspective based on stereopsis and occlusion

Perspective is usually considered a monocular pictorial cue, distinct from other cues such as occlusion and stereopsis. We cut across these distinctions by asking whether purely binocular (cyclopean) contours, created by stereoscopically shifting a region of homogeneous texture nearer or further than its surround, can act as a linear-perspective cue and whether the contours' ability to do this is influenced by their surface belongingness. We found that the left/right orientation of cyclopean trapezoids nearer than a surround strongly influenced perceived slant, showing that perspective constraints are applied to stereoscopically derived contours. Further regions, however; appeared as surfaces seen through a trapezoidal aperture. Because the aperture "owned" the trapezoidal contours, their orientation had little effect on perceived slant. We conclude that the application of perspective constraints depends critically on how contours are classified by stereo-specified occlusion relationships among surfaces and that perspective, stereopsis, and occlusion are not distinct processing systems.

Motion direction, speed and orientation in binocular matching

The spatial differences between the images seen by the two eyes, called binocular disparities, can be used to recover the volumetric (three-dimensional) aspects of a scene. The computation of disparity depends upon the correct identification of corresponding features in the two images. Understanding what image features are used by the brain to solve this matching problem is one of the main issues in stereoscopic vision. Many cortical neurons in visual areas V1 (ref. 2), MT (refs 3, 4) and MST (refs 5, 6) that are tuned to binocular disparity are also tuned to orientation, motion direction and speed. Although psychophysical work has shown that motion direction can facilitate binocular matching, the psychophysical literature on the role of orientation is mixed, and it has been argued that speed differences are ineffective in aiding correspondence. Here we use a different psychophysical paradigm to show that the visual system uses similarities in orientation, motion direction and speed to achieve binocular correspondence. These results indicate that cells that multiplex orientation, motion direction, speed and binocular disparity may help to solve the binocular matching problem.

The task-dependent use of binocular disparity and motion parallax information

Binocular disparity and motion parallax are powerful cues to the relative depth between objects. However to recover absolute depth, either additional scaling parameters are required to calibrate the information provided by each cue, or it can be recovered through the combination of information from both cues (Richards, W. (1985). Structure from stereo and motion. Journal of the Optical Society of America, 2, 343-349). However, not all tasks necessarily require a full specification of the absolute depth structure of a scene and so psychophysical performance may vary depending on the amount of information available, and the degree to which absolute depth structure is required. The experiments reported here used three different tasks that varied in the type of geometric information required in order for them to be completed successfully. These included a depth nulling task, a depth-matching task, and an absolute depth judgement (shape) task. Real world stimuli were viewed (i) monocularly with head movements, (ii) binocularly and static, or (iii) binocularly with head movements. No effect of viewing condition was found whereas there was a large effect of task. Performance was accurate on the matching and nulling tasks and much less accurate on the shape task. The fact that the same perceptual distortions were not evident in all tasks suggests that the visual system can switch strategy according to the demands of the particular task. No evidence was found to suggest that the visual system could exploit the simultaneous presence of disparity and motion parallax.

Spatial frequency tuning for 3-D corrugations from motion parallax

We provide evidence for the existence of multiple channels tuned to the spatial frequency of depth modulations defined by motion parallax. By linking the distortion of a random dot pattern to the horizontal position of an observer's head horizontally oriented 3-D corrugations were simulated in which the depth function consisted of a range of frequencies. In a baseline experiment thresholds were obtained for detecting depth modulations of single sinewaves for a range of spatial frequencies. In a masking experiment threshold signal strength was determined for detecting a signal frequency in the presence of noise with frequencies restricted to two bands around the signal component ('notched noise'). Threshold elevation was found to decrease with an increase in the spectral difference between signal and noise. By determining thresholds at various noise levels it was further established that the channel responded linearly in the tested range. Estimates of the bandwidth for spatial frequencies of 0.33 and 0.87 cycles/deg were both found to be 1.4 octaves. The results show that motion parallax processing is mediated by a series of narrowly tuned channels with bandwidths similar to those found for processing depth modulations defined by binocular disparity.

Cue combination in the motion correspondence problem

Image motion is a primary source of visual information about the world. However, before this information can be used the visual system must determine the spatio-temporal displacements of the features in the dynamic retinal image, which originate from objects moving in space. This is known as the motion correspondence problem. We investigated whether cross-cue matching constraints contribute to the solution of this problem, which would be consistent with physiological reports that many directionally selective cells in the visual cortex also respond to additional visual cues. We measured the maximum displacement limit (Dmax) for two-frame apparent motion sequences. Dmax increases as the number of elements in such sequences decreases. However, in our displays the total number of elements was kept constant while the number of a subset of elements, defined by a difference in contrast polarity, binocular disparity or colour, was varied. Dmax increased as the number of elements distinguished by a particular cue was decreased. Dmax was affected by contrast polarity for all observers, but only some observers were influenced by binocular disparity and others by colour information. These results demonstrate that the human visual system exploits local, cross-cue matching constraints in the solution of the motion correspondence problem.