The intrinsic constraint model for stereo-motion integration

The case against probabilistic inference: a new deterministic theory of 3D visual processing

How the brain derives 3D information from inherently ambiguous visual input remains the fundamental question of human vision. The past two decades of research have addressed this question as a problem of probabilistic inference, the dominant model being maximum-likelihood estimation (MLE). This model assumes that independent depth-cue modules derive noisy but statistically accurate estimates of 3D scene parameters that are combined through a weighted average. Cue weights are adjusted based on the system representation of each module's output variability. Here I demonstrate that the MLE model fails to account for important psychophysical findings and, importantly, misinterprets the just noticeable difference, a hallmark measure of stimulus discriminability, to be an estimate of perceptual uncertainty. I propose a new theory, termed Intrinsic Constraint, which postulates that the visual system does not derive the most probable interpretation of the visual input, but rather, the most stable interpretation amid variations in viewing conditions. This goal is achieved with the Vector Sum model, which represents individual cue estimates as components of a multi-dimensional vector whose norm determines the combined output. This model accounts for the psychophysical findings cited in support of MLE, while predicting existing and new findings that contradict the MLE model. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Experimentally disambiguating models of sensory cue integration

Sensory cue integration is one of the primary areas in which a normative mathematical framework has been used to define the “optimal” way in which to make decisions based upon ambiguous sensory information and compare these predictions to behavior. The conclusion from such studies is that sensory cues are integrated in a statistically optimal fashion. However, numerous alternative computational frameworks exist by which sensory cues could be integrated, many of which could be described as “optimal” based on different criteria. Existing studies rarely assess the evidence relative to different candidate models, resulting in an inability to conclude that sensory cues are integrated according to the experimenter's preferred framework. The aims of the present paper are to summarize and highlight the implicit assumptions rarely acknowledged in testing models of sensory cue integration, as well as to introduce an unbiased and principled method by which to determine, for a given experimental design, the probability with which a population of observers behaving in accordance with one model of sensory integration can be distinguished from the predictions of a set of alternative models.

Explicit and implicit depth-cue integration: Evidence of systematic biases with real objects

In previous studies using VR, we found evidence that 3D shape estimation agrees to a superadditivity rule of depth-cue combination, by which adding depth cues leads to greater perceived depth and, in principle, to depth overestimation. Superadditivity can be quantitatively accounted for by a normative theory of cue integration, via adapting a model termed Intrinsic Constraint (IC). As for its qualitative nature, it remains unclear whether superadditivity represents the genuine readout of depth-cue integration, as predicted by IC, or alternatively a byproduct of artificial virtual displays, because they carry flatness cues that can bias depth estimates in a Bayesian fashion, or even just a way for observers to express that a scene "looks deeper" with more depth cues by explicitly inflating their depth judgments. In the present study, we addressed this question by testing whether the IC model's prediction of superadditivity generalizes to real world settings. We asked participants to judge the perceived 3D shape of cardboard prisms through a matching task. To control for the potential interference of explicit reasoning, we also asked participants to reach-to-grasp the same objects and we analyzed the in-flight grip size throughout the reaching. We designed a novel technique to carefully control binocular and monocular 3D cues independently, allowing to add or remove depth information seamlessly. Even with real objects, participants exhibited a clear superadditivity effect in both tasks. Furthermore, the magnitude of this effect was accurately predicted by the IC model. These results confirm that superadditivity is an inherent feature of depth estimation.

Combining cues to judge distance and direction in an immersive virtual reality environment

When we move, the visual direction of objects in the environment can change substantially. Compared with our understanding of depth perception, the problem the visual system faces in computing this change is relatively poorly understood. Here, we tested the extent to which participants' judgments of visual direction could be predicted by standard cue combination rules. Participants were tested in virtual reality using a head-mounted display. In a simulated room, they judged the position of an object at one location, before walking to another location in the room and judging, in a second interval, whether an object was at the expected visual direction of the first. By manipulating the scale of the room across intervals, which was subjectively invisible to observers, we put two classes of cue into conflict, one that depends only on visual information and one that uses proprioceptive information to scale any reconstruction of the scene. We find that the sensitivity to changes in one class of cue while keeping the other constant provides a good prediction of performance when both cues vary, consistent with the standard cue combination framework. Nevertheless, by comparing judgments of visual direction with those of distance, we show that judgments of visual direction and distance are mutually inconsistent. We discuss why there is no need for any contradiction between these two conclusions.

Depth cue reweighting requires altered correlations with haptic feedback

Depth cue reweighting is a feedback-driven learning process that modifies the relative influences of different sources of three-dimensional shape information in perceptual judgments and or motor planning. In this study, we investigated the mechanism supporting reweighting of stereo and texture information by manipulating the haptic feedback obtained during a series of grasping movements. At the end of each grasp, the fingers closed down on a physical object that was consistent with one of the two cues, depending on the condition. Previous studies have shown that this style of visuomotor training leads to cue reweighting for perceptual judgments, but the time course has never been documented for a single training session, and many questions remain regarding the underlying mechanism, such as the pattern of feedback signals required to drive reweighting. We address these issues in two experiments, finding short-term changes in the motor response consistent with cue reweighting: the slope of the grip aperture with respect to the reliable cue increased, whereas the slope with respect to the unreliable cue decreased. Critically, Experiment 2 shows that slope changes do not occur when one of the cues is rendered with a constant bias; the grip aperture simply becomes uniformly larger or smaller. Our findings support a model of cue reweighting driven by altered correlations between haptic feedback and individual cues, rather than simple mismatches, which can be resolved by other mechanisms such as sensorimotor adaptation or cue recalibration.

A systematic comparison between visual cues for boundary detection

The detection of object boundaries is a critical first step for many visual processing tasks. Multiple cues (we consider luminance, color, motion and binocular disparity) available in the early visual system may signal object boundaries but little is known about their relative diagnosticity and how to optimally combine them for boundary detection. This study thus aims at understanding how early visual processes inform boundary detection in natural scenes. We collected color binocular video sequences of natural scenes to construct a video database. Each scene was annotated with two full sets of ground-truth contours (one set limited to object boundaries and another set which included all edges). We implemented an integrated computational model of early vision that spans all considered cues, and then assessed their diagnosticity by training machine learning classifiers on individual channels. Color and luminance were found to be most diagnostic while stereo and motion were least. Combining all cues yielded a significant improvement in accuracy beyond that of any cue in isolation. Furthermore, the accuracy of individual cues was found to be a poor predictor of their unique contribution for the combination. This result suggested a complex interaction between cues, which we further quantified using regularization techniques. Our systematic assessment of the accuracy of early vision models for boundary detection together with the resulting annotated video dataset should provide a useful benchmark towards the development of higher-level models of visual processing.

What's in a face? The role of depth undulations in three-dimensional depth-inversion illusions

Upright hollow human faces produce among the strongest depth-inversion illusions (DIIs), but why? We considered the role of depth undulations by comparing four types of hollow objects: an ellipsoid, a human mask, and two symmetric 'Martian'masks, which wavered in depth like the human mask but which lacked face-like features. Illusion strength was quantified either as the critical viewing distance at which the 3-D percept switched between convex and concave (experiment 1) or as the proportion of time ('predominance') that observers experienced DII from a fixed intermediate viewing distance (experiment 2). Critical distances were smallest--and hence the illusion was strongest--for the upright human mask; the remaining objects produced undifferentiated critical distance values. The predominance results were more fine-grained: illusions were experienced most often for the upright human mask, least often for the hollow ellipsoid, and to an intermediate extent for the Martian and upside-down human masks. These results suggest: (1) an upside-down human mask and a surface with nonface features undulating in depth are equivalent for the purposes of generating DIIs; (2) depth undulations contribute to DII; and (3) such undulations are most effective when structured into an upright human face.

Effects of changes in size, speed, and distance on the perception of curved 3-D trajectories

Previous research on the perception of 3-D object motion has considered time to collision, time to passage, collision detection, and judgments of speed and direction of motion but has not directly studied the perception of the overall shape of the motion path. We examined the perception of the magnitude of curvature and sign of curvature of the motion path for objects moving at eye level in a horizontal plane parallel to the line of sight. We considered two sources of information for the perception of motion trajectories: changes in angular size and changes in angular speed. Three experiments examined judgments of relative curvature for objects moving at different distances. At the closest distance studied, accuracy was high with size information alone but near chance with speed information alone. At the greatest distance, accuracy with size information alone decreased sharply, but accuracy for displays with both size and speed information remained high. We found similar results in two experiments with judgments of sign of curvature. Accuracy was higher for displays with both size and speed information than with size information alone, even when the speed information was based on parallel projections and was not informative about sign of curvature. For both magnitude of curvature and sign of curvature judgments, information indicating that the trajectory was curved increased accuracy, even when this information was not directly relevant to the required judgment.

Reverse correlation reveals how observers sample visual information when estimating three-dimensional shape

Human observers exhibit large systematic distance-dependent biases when estimating the three-dimensional (3D) shape of objects defined by binocular image disparities. This has led some to question the utility of disparity as a cue to 3D shape and whether accurate estimation of 3D shape is at all possible. Others have argued that accurate perception is possible, but only with large continuous perspective transformations of an object. Using a stimulus that is known to elicit large distance-dependent perceptual bias (random dot stereograms of elliptical cylinders) we show that contrary to these findings the simple adoption of a more naturalistic viewing angle completely eliminates this bias. Using behavioural psychophysics, coupled with a novel surface-based reverse correlation methodology, we show that it is binocular edge and contour information that allows for accurate and precise perception and that observers actively exploit and sample this information when it is available.

Misperception of aspect ratio in binocularly viewed surfaces

The horizontal-vertical illusion, in which the vertical dimension is overestimated relative to the horizontal direction, has been explained in terms of the statistical relationship between the lengths of lines in the world, and the lengths of their projections onto the retina (Howe & Purves, 2002). The current study shows that this illusion affects the apparent aspect ratio of shapes, and investigates how it interacts with binocular cues to surface slant. One way in which statistical information could give rise to the horizontal-vertical illusion would be through prior assumptions about the distribution of slant. This prior would then be expected to interact with retinal cues to slant. We determined the aspect ratio of stereoscopically viewed ellipses that appeared circular. We show that observers' judgements of aspect ratio were affected by surface slant, but that the largest image vertical:horizontal aspect ratio that was considered to be a surface with a circular profile was always found for surfaces close to fronto-parallel. This is not consistent with a Bayesian model in which the horizontal-vertical illusion arises from a non-uniform prior probability distribution for slant. Rather, we suggest that assumptions about the slant of surfaces affect apparent aspect ratio in a manner that is more heuristic, and partially dissociated from apparent slant.

Independent effects of local and global binocular disparity on the perceived convexity of stereoscopically presented faces in scenes

Evidence suggests that experiencing the hollow-face illusion involves perceptual reversal of the binocular disparities associated with the face even though the rest of the scene appears unchanged. This suggests stereoscopic processing of object shape may be independent of scene-based processing of the layout of objects in depth. We investigated the effects of global scene-based and local object-based disparity on the compellingness of the perceived convexity of the face. We took stereoscopic photographs of people in scenes, and independently reversed the binocular disparities associated with the head and scene. Participants rated perceived convexity of a natural disparity ("convex") or reversed disparity ("concave") face shown either in its original context with reversed or natural disparities or against a black background. Faces with natural disparity were rated as more convincingly convex independent of the background, showing that the local disparities can affect perceived convexity independent of disparities across the rest of the image. However, the apparent convexity of the faces was also greater in natural disparity scenes compared to either a reversed disparity scene or a zero disparity black background. This independent effect of natural scene disparity suggests that the 'solidity' associated with natural scene disparities spread to enhance the perceived convexity of the face itself. Together, these findings suggest that global and local disparity exert independent and additive effects upon the perceived convexity of the face.

Perceived surface slant is systematically biased in the actively-generated optic flow

Humans make systematic errors in the 3D interpretation of the optic flow in both passive and active vision. These systematic distortions can be predicted by a biologically-inspired model which disregards self-motion information resulting from head movements (Caudek, Fantoni, & Domini 2011). Here, we tested two predictions of this model: (1) A plane that is stationary in an earth-fixed reference frame will be perceived as changing its slant if the movement of the observer's head causes a variation of the optic flow; (2) a surface that rotates in an earth-fixed reference frame will be perceived to be stationary, if the surface rotation is appropriately yoked to the head movement so as to generate a variation of the surface slant but not of the optic flow. Both predictions were corroborated by two experiments in which observers judged the perceived slant of a random-dot planar surface during egomotion. We found qualitatively similar biases for monocular and binocular viewing of the simulated surfaces, although, in principle, the simultaneous presence of disparity and motion cues allows for a veridical recovery of surface slant.

Do we perceive a flattened world on the monitor screen?

The current model of three-dimensional perception hypothesizes that the brain integrates the depth cues in a statistically optimal fashion through a weighted linear combination with weights proportional to the reliabilities obtained for each cue in isolation (Landy, Maloney, Johnston, & Young, 1995). Even though many investigations support such theoretical framework, some recent empirical findings are at odds with this view (e.g., Domini, Caudek, & Tassinari, 2006). Failures of linear cue integration have been attributed to cue-conflict and to unmodelled cues to flatness present in computer-generated displays. We describe two cue-combination experiments designed to test the integration of stereo and motion cues, in the presence of consistent or conflicting blur and accommodation information (i.e., when flatness cues are either absent, with physical stimuli, or present, with computer-generated displays). In both conditions, we replicated the results of Domini et al. (2006): The amount of perceived depth increased as more cues were available, also producing an over-estimation of depth in some conditions. These results can be explained by the Intrinsic Constraint model, but not by linear cue combination.

Integration of disparity and velocity information for haptic and perceptual judgments of object depth

Do reach-to-grasp (prehension) movements require a metric representation of three-dimensional (3D) layouts and objects? We propose a model relying only on direct sensory information to account for the planning and execution of prehension movements in the absence of haptic feedback and when the hand is not visible. In the present investigation, we isolate relative motion and binocular disparity information from other depth cues and we study their efficacy for reach-to-grasp movements and visual judgments. We show that (i) the amplitude of the grasp increases when relative motion is added to binocular disparity information, even if depth from disparity information is already veridical, and (ii) similar distortions of derived depth are found for haptic tasks and perceptual judgments. With a quantitative test, we demonstrate that our results are consistent with the Intrinsic Constraint model and do not require 3D metric inferences (Domini, Caudek, & Tassinari, 2006). By contrast, the linear cue integration model (Landy, Maloney, Johnston, & Young, 1995) cannot explain the present results, even if the flatness cues are taken into account.

Inconsistency of perceived 3D shape

Internal consistency of local depth, slant, and curvature judgments was studied by asking participants to match two 3D surfaces rendered by different mixtures of 3D cues (velocity, texture, and shading). We found that perceptual judgments were not consistent with each other, with cue-specific distortions. Adding multiple cues did not eliminate the inconsistencies of the judgments. These results can be predicted by the Intrinsic Constraint (IC) model according to which the perceptual metric local estimates are a monotonically increasing function of the Signal-to-Noise Ratio of the optimal combination of direct information of 3D shape (Domini, Caudek, & Tassinari, 2006).

Matching perceived depth from disparity and from velocity: Modeling and psychophysics

We asked observers to match in depth a disparity-only stimulus with a velocity-only stimulus. The observers' responses revealed systematic biases: the two stimuli appeared to be matched in depth when they were produced by the projection of different distal depth extents. We discuss two alternative models of depth recovery that could account for these results. (1) Depth matches could be obtained by scaling the image signals by constants not specified by optical information, and (2) depth matches could be obtained by equating the stimuli in terms of their signal-to-noise ratios (see Domini & Caudek, 2009). We show that the systematic failures of shape constancy revealed by observers' judgments are well accounted for by the hypothesis that the apparent depth of a stimulus is determined by the magnitude of the retinal signals relative to the uncertainty (i.e., internal noise) arising from the measurement of those signals.

Target Visibility in the Standing Wave Illusion: Is Mask—Target Shape Similarity Important?

The perceptibility of a flickering central bar can be dramatically reduced by the presence of two flanking bars presented in counterphase. This phenomenon, known as the ‘standing wave illusion’, has been suggested to involve local edge interactions (Macknik et al, 2000 Proceedings of the National Academy of Sciences of the USA97 7556–7560). High-level re-entrant mechanisms have also been implicated. Enns (2002, Psychonomic Bulletin & Review9 489–496) reports an association between the reported viability of the centre bar and its similarity in shape with the flanking bars. We find that this relationship between shape similarity and reported visibility seems to be contingent on the degree of experienced apparent motion. When target duration is shortened, so reducing apparent motion, reports of visibility reflect the amount of abutting contour. In a further experiment we find that luminance discriminations of the centre bar are related to the amount of abutting contour not to shape similarity. This is despite experiment 3 being conducted at stimulus durations for which experiment 2 visibility ratings indicated that shape similarity is important and contour is not. We suggest that this perceived motion may be the factor mediating shape ‘effects’ in the reported visibility task. We propose that the absence of such shape effects in the discrimination task may be because the task provides an objective measure of visibility that is immune to bias from perceived motion. We also speculate that while target luminance information may be immune to masking resulting from perceived motion, it may be subject to masking due to lateral inhibition.