The task-dependent use of binocular disparity and motion parallax information

Comparison of Depth-Related Visuomotor Task Performance in Uniocular Individuals and in Binocular Controls With and Without Temporary Monocular Occlusion

Purpose

Do one-eyed (uniocular) humans use monocular depth cues differently from those with intact binocularity to perform depth-related visuomotor tasks that emulate complex activities of daily living? If so, does performance depend on the participant's age, duration of uniocularity and head movements?

Methods

Forty-five uniocular cases (age range 6-37 years; 2.4 months-31.0 years of uniocularity) and 46 age-similar binocular controls performed a task that required them to pass a hoop around an electrified wire convoluted in depth multiple times, while avoiding contact as indicated by auditory feedback. The task was performed with and without head restraint, in random order. The error rate and speed were calculated from the frequency of contact between the hoop and wire and the total task duration (adjusting for error time), respectively, all determined from video recordings of the task. Head movements were analyzed from the videos using face-tracking software.

Results

Error rate decreased with age (P < 0.001) until the late teen years while speed revealed no such trend. Across all ages, the error rate increased and speed decreased in the absence of binocularity (P < 0.001). There was no additional error reduction with duration of uniocularity (P = 0.16). Head movements provided no advantage to task performance, despite generating parallax disparities comparable to binocular viewing.

Conclusions

Performance in a dynamic, depth-related visuomotor task is reduced in the absence of binocular viewing, independent of age-related performance level. This study finds no evidence for a prolonged experience with monocular depth cues being advantageous for such tasks over transient loss of binocularity.

New Approaches to 3D Vision

New approaches to 3D vision are enabling new advances in artificial intelligence and autonomous vehicles, a better understanding of how animals navigate the 3D world, and new insights into human perception in virtual and augmented reality. Whilst traditional approaches to 3D vision in computer vision (SLAM: simultaneous localization and mapping), animal navigation (cognitive maps), and human vision (optimal cue integration) start from the assumption that the aim of 3D vision is to provide an accurate 3D model of the world, the new approaches to 3D vision explored in this issue challenge this assumption. Instead, they investigate the possibility that computer vision, animal navigation, and human vision can rely on partial or distorted models or no model at all. This issue also highlights the implications for artificial intelligence, autonomous vehicles, human perception in virtual and augmented reality, and the treatment of visual disorders, all of which are explored by individual articles. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

The many facets of shape

Shape is an interesting property of objects because it is used in ordinary discourse in ways that seem to have little connection to how it is typically defined in mathematics. The present article describes how the concept of shape can be grounded within Euclidean and non-Euclidean geometry and also to human perception. It considers the formal methods that have been proposed for measuring the differences among shapes and how the performance of those methods compares with shape difference thresholds of human observers. It discusses how different types of shape change can be perceptually categorized. It also evaluates the specific data structures that have been used to represent shape in models of both human and machine vision, and it reviews the psychophysical evidence about the extent to which those models are consistent with human perception. Based on this review of the literature, we argue that shape is not one thing but rather a collection of many object attributes, some of which are more perceptually salient than others. Because the relative importance of these attributes can be context dependent, there is no obvious single definition of shape that is universally applicable in all situations.

Experimentally disambiguating models of sensory cue integration

Sensory cue integration is one of the primary areas in which a normative mathematical framework has been used to define the “optimal” way in which to make decisions based upon ambiguous sensory information and compare these predictions to behavior. The conclusion from such studies is that sensory cues are integrated in a statistically optimal fashion. However, numerous alternative computational frameworks exist by which sensory cues could be integrated, many of which could be described as “optimal” based on different criteria. Existing studies rarely assess the evidence relative to different candidate models, resulting in an inability to conclude that sensory cues are integrated according to the experimenter's preferred framework. The aims of the present paper are to summarize and highlight the implicit assumptions rarely acknowledged in testing models of sensory cue integration, as well as to introduce an unbiased and principled method by which to determine, for a given experimental design, the probability with which a population of observers behaving in accordance with one model of sensory integration can be distinguished from the predictions of a set of alternative models.

Cue vetoing in depth estimation: Physical and virtual stimuli

Motion parallax and binocular disparity contribute to the perceived depth of three-dimensional (3D) objects. However, depth is often misperceived, even when both cues are available. This may be due in part to conflicts with unmodelled cues endemic to computerized displays. Here we evaluated the impact of display-based cue conflicts on depth cue integration by comparing perceived depth for physical and virtual objects. Truncated square pyramids were rendered using Blender and 3D printed. We assessed perceived depth using a discrimination task with motion parallax, binocular disparity, and their combination. Physical stimuli were presented with precise control over position and lighting. Virtual stimuli were viewed using a head-mounted display. To generate motion parallax, observers made lateral head movements using a chin rest on a motion platform. Observers indicated if the width of the front face appeared greater or less than the distance between this surface and the base. We found that accuracy was similar for virtual and physical pyramids. All estimates were more precise when depth was defined by binocular disparity than motion parallax. Our probabilistic model shows that a linear combination model does not adequately describe performance in either physical or virtual conditions. While there was inter-observer variability in weights, performance in all conditions was best predicted by a veto model that excludes the less reliable depth cue, in this case motion parallax.

Taking aim at the perceptual side of motor learning: exploring how explicit and implicit learning encode perceptual error information through depth vision

Motor learning in visuomotor adaptation tasks results from both explicit and implicit processes, each responding differently to an error signal. Although the motor output side of these processes has been extensively studied, the visual input side is relatively unknown. We investigated if and how depth perception affects the computation of error information by explicit and implicit motor learning. Two groups of participants made reaching movements to bring a virtual cursor to a target in the frontoparallel plane. The Delayed group was allowed to reaim and their feedback was delayed to emphasize explicit learning, whereas the camped group received task-irrelevant clamped cursor feedback and continued to aim straight at the target to emphasize implicit adaptation. Both groups played this game in a highly detailed virtual environment (depth condition), leveraging a cover task of playing darts in a virtual tavern, and in an empty environment (no-depth condition). The delayed group showed an increase in error sensitivity under depth relative to no-depth. In contrast, the clamped group adapted to the same degree under both conditions. The movement kinematics of the delayed participants also changed under the depth condition, consistent with the target appearing more distant, unlike the Clamped group. A comparison of the delayed behavioral data with a perceptual task from the same individuals showed that the greater reaiming in the depth condition was consistent with an increase in the scaling of the error distance and size. These findings suggest that explicit and implicit learning processes may rely on different sources of perceptual information.NEW & NOTEWORTHY We leveraged a classic sensorimotor adaptation task to perform a first systematic assessment of the role of perceptual cues in the estimation of an error signal in the 3-D space during motor learning. We crossed two conditions presenting different amounts of depth information, with two manipulations emphasizing explicit and implicit learning processes. Explicit learning responded to the visual conditions, consistent with perceptual reports, whereas implicit learning appeared to be independent of them.

Combining cues to judge distance and direction in an immersive virtual reality environment

When we move, the visual direction of objects in the environment can change substantially. Compared with our understanding of depth perception, the problem the visual system faces in computing this change is relatively poorly understood. Here, we tested the extent to which participants' judgments of visual direction could be predicted by standard cue combination rules. Participants were tested in virtual reality using a head-mounted display. In a simulated room, they judged the position of an object at one location, before walking to another location in the room and judging, in a second interval, whether an object was at the expected visual direction of the first. By manipulating the scale of the room across intervals, which was subjectively invisible to observers, we put two classes of cue into conflict, one that depends only on visual information and one that uses proprioceptive information to scale any reconstruction of the scene. We find that the sensitivity to changes in one class of cue while keeping the other constant provides a good prediction of performance when both cues vary, consistent with the standard cue combination framework. Nevertheless, by comparing judgments of visual direction with those of distance, we show that judgments of visual direction and distance are mutually inconsistent. We discuss why there is no need for any contradiction between these two conclusions.

Size and shape constancy in consumer virtual reality

With the increase in popularity of consumer virtual reality headsets, for research and other applications, it is important to understand the accuracy of 3D perception in VR. We investigated the perceptual accuracy of near-field virtual distances using a size and shape constancy task, in two commercially available devices. Participants wore either the HTC Vive or the Oculus Rift and adjusted the size of a virtual stimulus to match the geometric qualities (size and depth) of a physical stimulus they were able to refer to haptically. The judgments participants made allowed for an indirect measure of their perception of the egocentric, virtual distance to the stimuli. The data show under-constancy and are consistent with research from carefully calibrated psychophysical techniques. There was no difference in the degree of constancy found in the two headsets. We conclude that consumer virtual reality headsets provide a sufficiently high degree of accuracy in distance perception, to allow them to be used confidently in future experimental vision science, and other research applications in psychology.

Lessons from reinforcement learning for biological representations of space

Neuroscientists postulate 3D representations in the brain in a variety of different coordinate frames (e.g. 'head-centred', 'hand-centred' and 'world-based'). Recent advances in reinforcement learning demonstrate a quite different approach that may provide a more promising model for biological representations underlying spatial perception and navigation. In this paper, we focus on reinforcement learning methods that reward an agent for arriving at a target image without any attempt to build up a 3D 'map'. We test the ability of this type of representation to support geometrically consistent spatial tasks such as interpolating between learned locations using decoding of feature vectors. We introduce a hand-crafted representation that has, by design, a high degree of geometric consistency and demonstrate that, in this case, information about the persistence of features as the camera translates (e.g. distant features persist) can improve performance on the geometric tasks. These examples avoid Cartesian (in this case, 2D) representations of space. Non-Cartesian, learned representations provide an important stimulus in neuroscience to the search for alternatives to a 'cognitive map'.

Stereopsis in animals: evolution, function and mechanisms

Stereopsis is the computation of depth information from views acquired simultaneously from different points in space. For many years, stereopsis was thought to be confined to primates and other mammals with front-facing eyes. However, stereopsis has now been demonstrated in many other animals, including lateral-eyed prey mammals, birds, amphibians and invertebrates. The diversity of animals known to have stereo vision allows us to begin to investigate ideas about its evolution and the underlying selective pressures in different animals. It also further prompts the question of whether all animals have evolved essentially the same algorithms to implement stereopsis. If so, this must be the best way to do stereo vision, and should be implemented by engineers in machine stereopsis. Conversely, if animals have evolved a range of stereo algorithms in response to different pressures, that could inspire novel forms of machine stereopsis appropriate for distinct environments, tasks or constraints. As a first step towards addressing these ideas, we here review our current knowledge of stereo vision in animals, with a view towards outlining common principles about the evolution, function and mechanisms of stereo vision across the animal kingdom. We conclude by outlining avenues for future work, including research into possible new mechanisms of stereo vision, with implications for machine vision and the role of stereopsis in the evolution of camouflage.

Summary: Stereopsis has evolved independently in different animals. We review the various functions it serves and the variety of mechanisms that could underlie stereopsis in different species.

A depth illusion supports the model of General Object Constancy: Size and depth constancies related by a same distance-scaling factor

Perceptual constancy refers to the ability to stabilize the representation of an object even though the retinal image of the object undergoes variations. In previous studies, we proposed a General Object Constancy (GOC) hypothesis to demonstrate a common stabilization mechanism for perception of an object's features, such as size, contrast and depth, as the perceived distance varies. In the present study, we report another depth illusion supporting the GOC model. The stimuli comprised pairs of disks moving in a pattern of radial optic flow. Each pair consisted of a white disk positioned upper left to a dark disk, creating a percept of the white disk casting a shadow. As the pairs contracted towards the center of the screen in accordance with motion away from the observer, the two disks in each pair appeared to increase in contrast and separate farther away from each other both in the fronto-parallel plane (angular separation illusion) and in depth (depth separation illusion). While the contrast illusion and the angular separation illusion, which is a variant of the size illusion, replicated our previous findings, the illusion of depth separation revealed a depth constancy phenomenon. We further confirmed that the size and depth perception were related, e.g., the depth separation and the angular separation illusions were highly correlated across observers. Whereas the illusory increase in the angular separation between a disk and its 'shadow' could not be canceled by modulation of depth, decreasing the angular separation could offset the illusory increase in depth separation. The results can be explained by the GOC hypothesis: the visual system uses the same scaling factor to account for contrast, size (angular separation), and depth variations with distance; additionally, the perceived size of the object is used to scale its depth and contrast signals in order to achieve constancy.

Grasping in absence of feedback: systematic biases endure extensive training

Reach-to-grasp movements performed without visual and haptic feedback of the hand are subject to systematic inaccuracies. Grasps directed at an object specified by binocular information usually end at the wrong distance with an incorrect final grip aperture. More specifically, moving the target object away from the observer leads to increasingly larger undershoots and smaller grip apertures. These systematic biases suggest that the visuomotor mapping is based on inaccurate estimates of an object's egocentric distance and 3D structure that compress the visual space. Here we ask whether the appropriate visuomotor mapping can be learned through an extensive exposure to trials where haptic and visual feedback of the hand is provided. By intermixing feedback trials with test trials without feedback, we aimed at maximizing the likelihood that the motor execution of test trials is positively influenced by that of preceding feedback trials. We found that the intermittent presence of feedback trials both (1) largely reduced the positioning error of the hand with respect to the object and (2) affected the shaping of the hand before the final grasp, leading to an overall more accurate performance. While this demonstrates an effective transfer of information from feedback trials to test trials, the remaining biases indicate that a compression of visual space is still taking place. The correct visuomotor mapping, therefore, could not be learned. We speculate that an accurate reconstruction of the scene at movement onset may not actually be needed. Instead, the online monitoring of the hand position relative to the object and the final contact with the object are sufficient for a successful execution of a grasp.

Modeling depth from motion parallax with the motion/pursuit ratio

The perception of unambiguous scaled depth from motion parallax relies on both retinal image motion and an extra-retinal pursuit eye movement signal. The motion/pursuit ratio represents a dynamic geometric model linking these two proximal cues to the ratio of depth to viewing distance. An important step in understanding the visual mechanisms serving the perception of depth from motion parallax is to determine the relationship between these stimulus parameters and empirically determined perceived depth magnitude. Observers compared perceived depth magnitude of dynamic motion parallax stimuli to static binocular disparity comparison stimuli at three different viewing distances, in both head-moving and head-stationary conditions. A stereo-viewing system provided ocular separation for stereo stimuli and monocular viewing of parallax stimuli. For each motion parallax stimulus, a point of subjective equality (PSE) was estimated for the amount of binocular disparity that generates the equivalent magnitude of perceived depth from motion parallax. Similar to previous results, perceived depth from motion parallax had significant foreshortening. Head-moving conditions produced even greater foreshortening due to the differences in the compensatory eye movement signal. An empirical version of the motion/pursuit law, termed the empirical motion/pursuit ratio, which models perceived depth magnitude from these stimulus parameters, is proposed.

Manually locating physical and virtual reality objects

OBJECTIVE

In this study, we compared how users locate physical and equivalent three-dimensional images of virtual objects in a cave automatic virtual environment (CAVE) using the hand to examine how human performance (accuracy, time, and approach) is affected by object size, location, and distance.

BACKGROUND

Virtual reality (VR) offers the promise to flexibly simulate arbitrary environments for studying human performance. Previously, VR researchers primarily considered differences between virtual and physical distance estimation rather than reaching for close-up objects.

METHOD

Fourteen participants completed manual targeting tasks that involved reaching for corners on equivalent physical and virtual boxes of three different sizes. Predicted errors were calculated from a geometric model based on user interpupillary distance, eye location, distance from the eyes to the projector screen, and object.

RESULTS

Users were 1.64 times less accurate (p < .001) and spent 1.49 times more time (p = .01) targeting virtual versus physical box corners using the hands. Predicted virtual targeting errors were on average 1.53 times (p < .05) greater than the observed errors for farther virtual targets but not significantly different for close-up virtual targets.

CONCLUSION

Target size, location, and distance, in addition to binocular disparity, affected virtual object targeting inaccuracy. Observed virtual box inaccuracy was less than predicted for farther locations, suggesting possible influence of cues other than binocular vision.

APPLICATION

Human physical interaction with objects in VR for simulation, training, and prototyping involving reaching and manually handling virtual objects in a CAVE are more accurate than predicted when locating farther objects.

Perceived rigidity in motion-in-depth increases with contour perspective

When observers are asked to match the depth of an object according to its height, they often report systematic errors depending on viewing distance. Systematic biases can also arise while vergence distances are induced by binocular disparities. Observers of stereoscopic images tend to overestimate the depth of objects displayed in front of the screen, while the depth of objects displayed behind the screen plane is underestimated. This phenomenon creates a serious problem in that veridicality in depth perception appears distorted when one attempts to render the metrics of a captured 3-D world. These distortions could also subsist with structure-from-motion information and during motion-in-depth. Observers judged the circularity of transparent rotating cylinders that were either static or moving in depth. Crossed results show that participants could precisely retrieve the best modulation between presented depth and width. As this effect could be amplified with stimuli containing stronger perspective cues (ie contour perspective), participants judged the rigidity of spinning cubes, moving along the line of sight, which were either edges-defined or defined by randomly textured surfaces (dots). The results showed that, although depth constancy was not improved by contour perspective, perceived rigidity was increased by perspective when the best scaling estimate was displayed. This finding suggests that appropriate binocular disparity information in combination to monocular signal is necessary for stereoscopic depth perception.

Reverse correlation reveals how observers sample visual information when estimating three-dimensional shape

Human observers exhibit large systematic distance-dependent biases when estimating the three-dimensional (3D) shape of objects defined by binocular image disparities. This has led some to question the utility of disparity as a cue to 3D shape and whether accurate estimation of 3D shape is at all possible. Others have argued that accurate perception is possible, but only with large continuous perspective transformations of an object. Using a stimulus that is known to elicit large distance-dependent perceptual bias (random dot stereograms of elliptical cylinders) we show that contrary to these findings the simple adoption of a more naturalistic viewing angle completely eliminates this bias. Using behavioural psychophysics, coupled with a novel surface-based reverse correlation methodology, we show that it is binocular edge and contour information that allows for accurate and precise perception and that observers actively exploit and sample this information when it is available.

Misperception of aspect ratio in binocularly viewed surfaces

The horizontal-vertical illusion, in which the vertical dimension is overestimated relative to the horizontal direction, has been explained in terms of the statistical relationship between the lengths of lines in the world, and the lengths of their projections onto the retina (Howe & Purves, 2002). The current study shows that this illusion affects the apparent aspect ratio of shapes, and investigates how it interacts with binocular cues to surface slant. One way in which statistical information could give rise to the horizontal-vertical illusion would be through prior assumptions about the distribution of slant. This prior would then be expected to interact with retinal cues to slant. We determined the aspect ratio of stereoscopically viewed ellipses that appeared circular. We show that observers' judgements of aspect ratio were affected by surface slant, but that the largest image vertical:horizontal aspect ratio that was considered to be a surface with a circular profile was always found for surfaces close to fronto-parallel. This is not consistent with a Bayesian model in which the horizontal-vertical illusion arises from a non-uniform prior probability distribution for slant. Rather, we suggest that assumptions about the slant of surfaces affect apparent aspect ratio in a manner that is more heuristic, and partially dissociated from apparent slant.

Visual depth from motion parallax and eye pursuit

A translating observer viewing a rigid environment experiences "motion parallax", the relative movement upon the observer's retina of variously positioned objects in the scene. This retinal movement of images provides a cue to the relative depth of objects in the environment, however retinal motion alone cannot mathematically determine relative depth of the objects. Visual perception of depth from lateral observer translation uses both retinal image motion and eye movement. In Nawrot and Stroyan (Vision Res 49:1969-1978, 2009) we showed mathematically that the ratio of the rate of retinal motion over the rate of smooth eye pursuit mathematically determines depth relative to the fixation point in central vision. We also reported on psychophysical experiments indicating that this ratio is the important quantity for perception. Here we analyze the motion/pursuit cue for the more general, and more complicated, case when objects are distributed across the horizontal viewing plane beyond central vision. We show how the mathematical motion/pursuit cue varies with different points across the plane and with time as an observer translates. If the time varying retinal motion and smooth eye pursuit are the only signals used for this visual process, it is important to know what is mathematically possible to derive about depth and structure. Our analysis shows that the motion/pursuit ratio determines an excellent description of depth and structure in these broader stimulus conditions, provides a detailed quantitative hypothesis of these visual processes for the perception of depth and structure from motion parallax, and provides a computational foundation to analyze the dynamic geometry of future experiments.

Do we perceive a flattened world on the monitor screen?

The current model of three-dimensional perception hypothesizes that the brain integrates the depth cues in a statistically optimal fashion through a weighted linear combination with weights proportional to the reliabilities obtained for each cue in isolation (Landy, Maloney, Johnston, & Young, 1995). Even though many investigations support such theoretical framework, some recent empirical findings are at odds with this view (e.g., Domini, Caudek, & Tassinari, 2006). Failures of linear cue integration have been attributed to cue-conflict and to unmodelled cues to flatness present in computer-generated displays. We describe two cue-combination experiments designed to test the integration of stereo and motion cues, in the presence of consistent or conflicting blur and accommodation information (i.e., when flatness cues are either absent, with physical stimuli, or present, with computer-generated displays). In both conditions, we replicated the results of Domini et al. (2006): The amount of perceived depth increased as more cues were available, also producing an over-estimation of depth in some conditions. These results can be explained by the Intrinsic Constraint model, but not by linear cue combination.

Depth interval estimates from motion parallax and binocular disparity beyond interaction space

Static and dynamic observers provided binocular and monocular estimates of the depths between real objects lying well beyond interaction space. On each trial, pairs of LEDs were presented inside a dark railway tunnel. The nearest LED was always 40 m from the observer, with the depth separation between LED pairs ranging from 0 up to 248 m. Dynamic binocular viewing was found to produce the greatest (ie most veridical) estimates of depth magnitude, followed next by static binocular viewing, and then by dynamic monocular viewing. (No significant depth was seen with static monocular viewing.) We found evidence that both binocular and monocular dynamic estimates of depth were scaled for the observation distance when the ground plane and walls of the tunnel were visible up to the nearest LED. We conclude that both motion parallax and stereopsis provide useful long-distance depth information and that motion-parallax information can enhance the degree of stereoscopic depth seen.

Integration of disparity and velocity information for haptic and perceptual judgments of object depth

Do reach-to-grasp (prehension) movements require a metric representation of three-dimensional (3D) layouts and objects? We propose a model relying only on direct sensory information to account for the planning and execution of prehension movements in the absence of haptic feedback and when the hand is not visible. In the present investigation, we isolate relative motion and binocular disparity information from other depth cues and we study their efficacy for reach-to-grasp movements and visual judgments. We show that (i) the amplitude of the grasp increases when relative motion is added to binocular disparity information, even if depth from disparity information is already veridical, and (ii) similar distortions of derived depth are found for haptic tasks and perceptual judgments. With a quantitative test, we demonstrate that our results are consistent with the Intrinsic Constraint model and do not require 3D metric inferences (Domini, Caudek, & Tassinari, 2006). By contrast, the linear cue integration model (Landy, Maloney, Johnston, & Young, 1995) cannot explain the present results, even if the flatness cues are taken into account.

Large perspective changes yield perception of metric shape that allows accurate feedforward reaches-to-grasp and it persists after the optic flow has stopped!

Lee et al. (Percept Psychophys 70:1032-1046, 2008a) investigated whether visual perception of metric shape could be calibrated when used to guide feedforward reaches-to-grasp. It could not. Seated participants viewed target objects (elliptical cylinders) in normal lighting using stereo vision and free head movements that allowed small (approximately 10 degrees) perspective changes. The authors concluded that poor perception of metric shape was the reason reaches-to-grasp should be visually guided online. However, Bingham and Lind (Percept Psychophys 70:524-540, 2008) showed that large perspective changes (> or =45 degrees) yield good perception of metric shape. So, now we repeated the Lee et al.'s study with the addition of information from large perspective changes. The results were accurate feedforward reaches-to-grasp reflecting accurate perception of both metric shape and metric size. Large perspective changes occur when one locomotes into a workspace in which reaches-to-grasp are subsequently performed. Does the resulting perception of metric shape persist after the large perspective changes have ceased? Experiments 2 and 3 tested reaches-to-grasp with delays (Exp. 2, 5-s delay; Exp. 3, approximately 16-s delay) and multiple objects to be grasped after a single viewing. Perception of metric shape and metric size persisted yielding accurate reaches-to-grasp. We advocate the study of nested actions using a dynamic approach to perception/action.

Inconsistency of perceived 3D shape

Internal consistency of local depth, slant, and curvature judgments was studied by asking participants to match two 3D surfaces rendered by different mixtures of 3D cues (velocity, texture, and shading). We found that perceptual judgments were not consistent with each other, with cue-specific distortions. Adding multiple cues did not eliminate the inconsistencies of the judgments. These results can be predicted by the Intrinsic Constraint (IC) model according to which the perceptual metric local estimates are a monotonically increasing function of the Signal-to-Noise Ratio of the optimal combination of direct information of 3D shape (Domini, Caudek, & Tassinari, 2006).

Matching perceived depth from disparity and from velocity: Modeling and psychophysics

We asked observers to match in depth a disparity-only stimulus with a velocity-only stimulus. The observers' responses revealed systematic biases: the two stimuli appeared to be matched in depth when they were produced by the projection of different distal depth extents. We discuss two alternative models of depth recovery that could account for these results. (1) Depth matches could be obtained by scaling the image signals by constants not specified by optical information, and (2) depth matches could be obtained by equating the stimuli in terms of their signal-to-noise ratios (see Domini & Caudek, 2009). We show that the systematic failures of shape constancy revealed by observers' judgments are well accounted for by the hypothesis that the apparent depth of a stimulus is determined by the magnitude of the retinal signals relative to the uncertainty (i.e., internal noise) arising from the measurement of those signals.

The motion/pursuit law for visual depth perception from motion parallax

One of vision's most important functions is specification of the layout of objects in the 3D world. While the static optical geometry of retinal disparity explains the perception of depth from binocular stereopsis, we propose a new formula to link the pertinent dynamic geometry to the computation of depth from motion parallax. Mathematically, the ratio of retinal image motion (motion) and smooth pursuit of the eye (pursuit) provides the necessary information for the computation of relative depth from motion parallax. We show that this could have been obtained with the approaches of Nakayama and Loomis [Nakayama, K., & Loomis, J. M. (1974). Optical velocity patterns, velocity-sensitive neurons, and space perception: A hypothesis. Perception, 3, 63-80] or Longuet-Higgens and Prazdny [Longuet-Higgens, H. C., & Prazdny, K. (1980). The interpretation of a moving retinal image. Proceedings of the Royal Society of London Series B, 208, 385-397] by adding pursuit to their treatments. Results of a psychophysical experiment show that changes in the motion/pursuit ratio have a much better relationship to changes in the perception of depth from motion parallax than do changes in motion or pursuit alone. The theoretical framework provided by the motion/pursuit law provides the quantitative foundation necessary to study this fundamental visual depth perception ability.

The intrinsic constraint model for stereo-motion integration

How the visual system integrates the information provided by several depth cues is central for vision research. Here, we present a model for how the human visual system combines disparity and velocity information. The model provides a depth interpretation to a subspace defined by the covariation of the two signals. We show that human performance is consistent with the predictions of the model, and compare them with those of another theoretical approach, the modified weak-fusion model. We discuss the validity of each approach as a model for human perception of 3-D shape from multiple cues to depth.

A neural model for the integration of stereopsis and motion parallax in structure-from-motion

We introduce a model for the computation of structure-from-motion based on the physiology of visual cortical areas MT and MST. The model assumes that the perception of depth from motion is related to the firing of a subset of MT neurons tuned to both velocity and disparity. The model's MT neurons are connected to each other laterally to form modulatory receptive-field surrounds that are gated by feedback connections from area MST. This allows the building up of a depth map from motion in area MT, even in absence of disparity in the input. Depth maps from motion and from stereo are combined by a weighted average at a final stage. The model's predictions for the interaction between motion and stereo cues agree with previous psychophysical data, both when the cues are consistent with each other or when they are contradictory. In particular, the model shows nonlinearities as a result of early interactions between motion and stereo before their depth maps are averaged. The two cues interact in a way that represents an alternative to the "modified weak fusion" model of depth-cue combination.

Dissociation between vergence and binocular disparity cues in the control of prehension

Binocular vision provides important advantages for controlling reach-to-grasp movements. We examined the possible source(s) of these advantages by comparing prehension proficiency under four different binocular viewing conditions, created by randomly placing a neutral lens (control), an eight dioptre prism (Base In or Base Out) or a low-power (2.00-3.75 dioptre) Plus lens over the eye opposite the moving limb. The Base In versus Base Out prisms were intended to selectively alter vergence-specified distance (VSD) information, such that the targets appeared beyond or closer than their actual physical position, respectively. The Plus lens was individually tailored to reduce each subject's disparity sensitivity (to 400-800 arc s), while minimizing effects on distance estimation. In pre-testing, subjects pointed (without visual feedback) to mid-line targets at different distances, and produced the systematic directional errors expected of uncorrected movements programmed under each of the perturbed conditions. For the prehension tasks, subjects reached and precision grasped (with visual feedback available) cylindrical objects (two sizes and three locations), either following a 3 s preview in which to plan their actions or immediately after the object became visible. Viewing condition markedly affected performance, but the planning time allowed did not. Participants made the most errors suggesting premature collision with the object (shortest 'braking' times after peak deceleration; fastest velocity and widest grip at initial contact) under Base In prism viewing, consistent with over-reaching movements programmed to transport the hand beyond the actual target due to its 'further' VSD. Conversely, they produced the longest terminal reaches and grip closure times, with multiple corrections just before and after object contact, under the Plus lens (reduced disparity) condition. Base Out prism performance was intermediate between these two, with significant increases in additional forward movements during the transport end-phase, indicative of initial under-reaching in response to the target's 'nearer' VSD. Our findings suggest dissociations between the role of vergence and binocular disparity in natural prehension movements, with vergence contributing mainly to reach planning and high-grade disparity cues providing particular advantages for grasp-point selection during grip programming and application.

The use of direction and distance information in the perception of approach trajectory

A pair of projectiles travelling on parallel trajectories produce differing patterns of retinal motion when they originate at different distances. For an observer to recognise that the two trajectories are parallel she must "factor out" the effect of distance on retinal motion. The observer faces a similar problem when physically parallel trajectories originate at different lateral positions; here direction must be "factored out". We report the results of a series of experiments designed to determine if observers can do this. The observers' task was to judge whether the direction of travel of an approaching sphere (test trajectory) was to the left or right of parallel to a previously shown trajectory (reference trajectory). In the first set of experiments the reference and test trajectories started from different lateral positions. In the final experiment they started from different distances. From the pattern of judgements we determined a set of perceptually parallel trajectories. Perceptually parallel trajectories deviated significantly from physically parallel. We conclude that under circumstances comparable to those encountered when catching a ball in flight, observers do not have access to accurate estimates of trajectory direction.

Stereo and motion information are not independently processed by the visual system

Many visual tasks are carried out by using multiple sources of sensory information to estimate environmental properties. In this paper, we present a model for how the visual system combines disparity and velocity information. We propose that, in a first stage of processing, the best possible estimate of the affine structure is obtained by computing a composite score from the disparity and velocity signals. In a second stage, a maximum likelihood Euclidean interpretation is assigned to the recovered affine structure. In two experiments, we show that human performance is consistent with the predictions of our model. The present results are also discussed in the framework of another theoretical approach of the depth cue combination process termed Modified Weak Fusion.

Humans ignore motion and stereo cues in favor of a fictional stable world

As we move through the world, our eyes acquire a sequence of images. The information from this sequence is sufficient to determine the structure of a three-dimensional scene, up to a scale factor determined by the distance that the eyes have moved. Previous evidence shows that the human visual system accounts for the distance the observer has walked and the separation of the eyes when judging the scale, shape, and distance of objects. However, in an immersive virtual-reality environment, observers failed to notice when a scene expanded or contracted, despite having consistent information about scale from both distance walked and binocular vision. This failure led to large errors in judging the size of objects. The pattern of errors cannot be explained by assuming a visual reconstruction of the scene with an incorrect estimate of interocular separation or distance walked. Instead, it is consistent with a Bayesian model of cue integration in which the efficacy of motion and disparity cues is greater at near viewing distances. Our results imply that observers are more willing to adjust their estimate of interocular separation or distance walked than to accept that the scene has changed in size.

Disparity-defined objects moving in depth do not elicit three-dimensional shape constancy

Observers generally fail to recover three-dimensional shape accurately from binocular disparity. Typically, depth is overestimated at near distances and underestimated at far distances [Johnston, E. B. (1991). Systematic distortions of shape from stereopsis. Vision Research, 31, 1351-1360]. A simple prediction from this is that disparity-defined objects should appear to expand in depth when moving towards the observer, and compress in depth when moving away. However, additional information is provided when an object moves from which 3D Euclidean shape can be recovered, be this through the addition of structure from motion information [Richards, W. (1985). Structure from stereo and motion. Journal of the Optical Society of America A, 2, 343-349], or the use of non-generic strategies [Todd, J. T., & Norman, J. F. (2003). The visual perception of 3-D shape from multiple cues: Are observers capable of perceiving metric structure? Perception and Psychophysics, 65, 31-47]. Here, we investigated shape constancy for objects moving in depth. We found that to be perceived as constant in shape, objects needed to contract in depth when moving toward the observer, and expand in depth when moving away, countering the effects of incorrect distance scaling (Johnston, 1991). This is a striking example of the failure of shape constancy, but one that is predicted if observers neither accurately estimate object distance in order to recover Euclidean shape, nor are able to base their responses on a simpler processing strategy.

Advantages of binocular vision for the control of reaching and grasping

Theoretical considerations suggest that binocular information should provide advantages, compared to monocular viewing, for the planning and execution of natural reaching and grasping actions, but empirical support for this is quite equivocal. We have examined these predictions on a simple prehension task in which normal subjects reached, grasped and lifted isolated cylindrical household objects (two sizes, four locations) in a well-lit environment, using binocular vision or with one eye occluded. Various kinematic measures reflecting the programming and on-line control of the movements were quantified, in combination with analyses of different types of error occurring in the velocity, spatial path and grip aperture profiles of each trial. There was little consistent effect of viewing condition on the early phase of the reach, up to and including the peak deceleration, but all other aspects of performance were superior under binocular control. Subjects adopted a cautious approach when binocular information was unavailable: they extended the end phase of the reach and pre-shaped their hand with a wider grip aperture further away from the object. Despite these precautions, initial grip application was poorly coordinated with target contact and was inaccurately scaled to the objects' dimensions, with the subsequent post-contact phase of the grasp significantly more prolonged, error-prone and variable compared to binocular performance. These effects were obtained in two separate experiments in which the participants' performed the task under randomized or more predictable (blocked) viewing conditions. Our data suggest that binocular vision offers particular advantages for controlling the terminal reach and the grasp. We argue that these benefits derive from binocular disparity processing linked to changes in relative hand-target distance, and that this depth information is independently used to regulate the progress of the approaching hand and to guide the digits to the (pre-selected) contact points on the object, thereby ensuring that the grip is securely applied.

Systematic distortions of perceptual stability investigated using immersive virtual reality

Using an immersive virtual reality system, we measured the ability of observers to detect the rotation of an object when its movement was yoked to the observer's own translation. Most subjects had a large bias such that a static object appeared to rotate away from them as they moved. Thresholds for detecting target rotation were similar to those for an equivalent speed discrimination task carried out by static observers, suggesting that visual discrimination is the predominant limiting factor in detecting target rotation. Adding a stable visual reference frame almost eliminated the bias. Varying the viewing distance of the target had little effect, consistent with observers underestimating distance walked. However, accuracy of walking to a briefly presented visual target was high and not consistent with an underestimation of distance walked. We discuss implications for theories of a task-independent representation of visual space.

Aging and the perception of depth and 3-D shape from motion parallax

The ability of younger and older observers to perceive 3-D shape and depth from motion parallax was investigated. In Experiment 1, the observers discriminated among differently curved 3-dimensional (3-D) surfaces in the presence of noise. In Experiment 2, the surfaces' shape was held constant and the amount of front-to-back depth was varied; the observers estimated the amount of depth they perceived. The effects of age were strongly task dependent. The younger observers' performance in Experiment 1 was almost 60% higher than that of the older observers. In contrast, no age effect was obtained in Experiment 2. Older observers can effectively perceive variations in depth from patterns of motion parallax, but their ability to discriminate 3-D shape is significantly compromised.

Binocular correlation does not improve coherence detection for fronto-parallel motion

We studied the low-level interactions between motion coherence detection and binocular correlation detection. It is well-established that e.g. depth information from motion parallax and from binocular disparities is effectively integrated. The question we aimed to answer is whether such interactions also exist at the very first correlation level that both mechanisms might have in common. First we quantitatively compared motion coherence detection and binocular correlation detection using similar stimuli (random pixels arrays, RPAs) and the same noise masking paradigm (luminance signal to noise ratio, LSNR). This showed that human observers are much more sensitive to motion than to binocular correlation. Adding noise therefore has a much stronger effect on binocular correlation than on motion detection. Next we manipulated the shape of the stimulus aperture to equalize LSNR thresholds for motion and binocular correlation. Motion sensitivity could be progressively reduced by shortening the length of the motion path, while keeping the aperture area constant. Changing the shape of the aperture did not affect binocular correlation sensitivity. A 'balanced' stimulus, one with equal strengths of motion and binocular correlation signals was then used to study the mutual interactions. In accordance with previous results, motion was found to greatly facilitate binocular correlation. Binocular correlation, however did not facilitate motion detection. We conclude that interactions are asymmetrical; fronto-parallel motion is primarily detected monocularly and this information can then be used to facilitate binocular correlation, but binocular correlation cannot improve motion sensitivity.

No evidence for sequential effects of the interaction of stereo and motion cues in judgements of perceived shape

The interaction of the depth cues of binocular disparity and motion parallax could potentially be used by the visual system to recover an estimate of the viewing distance. The present study investigated whether an interaction of stereo and motion has effects that persist over time to influence the perception of shape from stereo when the motion information is removed. Static stereoscopic ellipsoids were presented following the presentation of rotating stereoscopic ellipsoids, which were located either at the same or a different viewing distance. It was predicted that shape judgements for static stimuli would be better after presentation of a rotating stimulus at the same viewing distance, than after presentation of one at a different viewing distance. No such difference was found. It was concluded that an interaction between stereo and motion depth cues does not influence the perception of subsequently presented static objects.

The visual control of reaching and grasping: binocular disparity and motion parallax

The primary visual sources of depth and size information are binocular cues and motion parallax. Here, the authors determine the efficacy of these cues to control prehension by presenting them in isolation from other visual cues. When only binocular cues were available, reaches showed normal scaling of the transport and grasp components with object distance and size. However, when only motion parallax was available, only the transport component scaled reliably. No additional increase in scaling was found when both cues were available simultaneously. Therefore, although equivalent information is available from binocular and motion parallax information, the latter may be of relatively limited use for the control of the grasp. Binocular disparity appears selectively important for the control of the grasp.

The visual perception of 3-D shape from multiple cues: are observers capable of perceiving metric structure?

Three experiments are reported in which observers judged the three-dimensional (3-D) structures of virtual or real objects defined by various combinations of texture, motion, and binocular disparity under a wide variety of conditions. The tasks employed in these studies involved adjusting the depth of an object to match its width, adjusting the planes of a dihedral angle so that they appeared orthogonal, and adjusting the shape of an object so that it appeared to match another at a different viewing distance. The results obtained on all of these tasks revealed large constant errors and large individual differences among observers. There were also systematic failures of constancy over changes in viewing distance, orientation, or response task. When considered in conjunction with other, similar reports in the literature, these findings provide strong evidence that human observers do not have accurate perceptions of 3-D metric structure.

Reflexive shifts of covert attention operate in an egocentric coordinate frame

Covert shifts of attention have been shown to improve detection and discrimination thresholds for a range of visual stimuli. Although there is some evidence to suggest that the allocation of attention to a particular region of interest occurs in a retinotopic frame of reference, the importance of an allocentric, or object-based, framework has gained widespread empirical support. The current experiment investigates the nature of the spatial representation in which covert shifts of attention occur in response to a reflexive prime. Primes and targets were presented in four conditions designed to vary systematically the validity of the spatial relationship between the prime and target in egocentric or allocentric coordinate frameworks. A significant advantage, in terms of reaction time and correct identification, was found for targets located in positions previously primed in an egocentric (but not allocentric) framework whereas there was no advantage for locations primed in an allocentric (but not egocentric) framework. These results suggest that the allocation of covert spatial attention within an egocentric framework may be more important than previously thought.