Ideal cue combination for localizing texture-defined edges

Bayesian models: the structure of the world, uncertainty, behavior, and the brain

Experiments on humans and other animals have shown that uncertainty due to unreliable or incomplete information affects behavior. Recent studies have formalized uncertainty and asked which behaviors would minimize its effect. This formalization results in a wide range of Bayesian models that derive from assumptions about the world, and it often seems unclear how these models relate to one another. In this review, we use the concept of graphical models to analyze differences and commonalities across Bayesian approaches to the modeling of behavioral and neural data. We review behavioral and neural data associated with each type of Bayesian model and explain how these models can be related. We finish with an overview of different theories that propose possible ways in which the brain can represent uncertainty.

Depth-cue integration in grasp programming: no evidence for a binocular specialism

When we grasp with one eye covered, the finger and thumb are typically opened wider than for binocularly guided grasps, as if to build a margin-for-error into the movement. Also, patients with visual form agnosia can have profound deficits in their (otherwise relatively normal) grasping when binocular information is removed. One interpretation of these findings is that there is a functional specialism for binocular vision in the control of grasping. Alternatively, cue-integration theory suggests that binocular and monocular depth cues are combined in the control of grasping, and so impaired performance reflects not the loss of 'critical' binocular cues, but increased uncertainty per se. Unfortunately, removing binocular information confounds removing particular (binocular) depth cues with an overall reduction in the available information, and so such experiments cannot distinguish between these alternatives. We measured the effects on visually open-loop grasping of selectively removing monocular (texture) or binocular depth cues. To allow meaningful comparisons, we made psychophysical measurements of the uncertainty in size estimates in each case, so that the informativeness of binocular and monocular cues was known in each condition. Consistent with cue-integration theory, removing either binocular or monocular cues resulted in similar increases in grip apertures. In a separate experiment, we also confirmed that changes in uncertainty per se (keeping the same depth cues available) resulted in larger grip apertures. Overall, changes in the margin-for-error in grasping movements were determined by the uncertainty in size estimates and not by the presence or absence of particular depth cues. Our data therefore argue against a binocular specialism for grasp programming. Instead, grip apertures were smaller when binocular and monocular cues were available than with either cue alone, providing strong evidence that the visuo-motor system exploits the redundancy available in multiple sources of information, and integrates binocular and monocular cues to improve grasping performance.

Exogenous attention enhances 2nd-order contrast sensitivity

Natural scenes contain a rich variety of contours that the visual system extracts to segregate the retinal image into perceptually coherent regions. Covert spatial attention helps extract contours by enhancing contrast sensitivity for 1st-order, luminance-defined patterns at attended locations, while reducing sensitivity at unattended locations, relative to neutral attention allocation. However, humans are also sensitive to 2nd-order patterns such as spatial variations of texture, which are predominant in natural scenes and cannot be detected by linear mechanisms. We assess whether and how exogenous attention--the involuntary and transient capture of spatial attention--affects the contrast sensitivity of channels sensitive to 2nd-order, texture-defined patterns. Using 2nd-order, texture-defined stimuli, we demonstrate that exogenous attention increases 2nd-order contrast sensitivity at the attended location, while decreasing it at unattended locations, relative to a neutral condition. By manipulating both 1st- and 2nd-order spatial frequency, we find that the effects of attention depend both on 2nd-order spatial frequency of the stimulus and the observer's 2nd-order spatial resolution at the target location. At parafoveal locations, attention enhances 2nd-order contrast sensitivity to high, but not to low 2nd-order spatial frequencies; at peripheral locations attention also enhances sensitivity to low 2nd-order spatial frequencies. Control experiments rule out the possibility that these effects might be due to an increase in contrast sensitivity at the 1st-order stage of visual processing. Thus, exogenous attention affects 2nd-order contrast sensitivity at both attended and unattended locations.

Multi-sensory weights depend on contextual noise in reference frame transformations

During reach planning, we integrate multiple senses to estimate the location of the hand and the target, which is used to generate a movement. Visual and proprioceptive information are combined to determine the location of the hand. The goal of this study was to investigate whether multi-sensory integration is affected by extraretinal signals, such as head roll. It is believed that a coordinate matching transformation is required before vision and proprioception can be combined because proprioceptive and visual sensory reference frames do not generally align. This transformation utilizes extraretinal signals about current head roll position, i.e., to rotate proprioceptive signals into visual coordinates. Since head roll is an estimated sensory signal with noise, this head roll dependency of the reference frame transformation should introduce additional noise to the transformed signal, reducing its reliability and thus its weight in the multi-sensory integration. To investigate the role of noisy reference frame transformations on multi-sensory weighting, we developed a novel probabilistic (Bayesian) multi-sensory integration model (based on Sober and Sabes, 2003) that included explicit (noisy) reference frame transformations. We then performed a reaching experiment to test the model's predictions. To test for head roll dependent multi-sensory integration, we introduced conflicts between viewed and actual hand position and measured reach errors. Reach analysis revealed that eccentric head roll orientations led to an increase of movement variability, consistent with our model. We further found that the weighting of vision and proprioception depended on head roll, which we interpret as being a result of signal dependant noise. Thus, the brain has online knowledge of the statistics of its internal sensory representations. In summary, we show that sensory reliability is used in a context-dependent way to adjust multi-sensory integration weights for reaching.

Visual-vestibular cue integration for heading perception: applications of optimal cue integration theory

The perception of self-motion is crucial for navigation, spatial orientation and motor control. In particular, estimation of one's direction of translation, or heading, relies heavily on multisensory integration in most natural situations. Visual and nonvisual (e.g., vestibular) information can be used to judge heading, but each modality alone is often insufficient for accurate performance. It is not surprising, then, that visual and vestibular signals converge frequently in the nervous system, and that these signals interact in powerful ways at the level of behavior and perception. Early behavioral studies of visual-vestibular interactions consisted mainly of descriptive accounts of perceptual illusions and qualitative estimation tasks, often with conflicting results. In contrast, cue integration research in other modalities has benefited from the application of rigorous psychophysical techniques, guided by normative models that rest on the foundation of ideal-observer analysis and Bayesian decision theory. Here we review recent experiments that have attempted to harness these so-called optimal cue integration models for the study of self-motion perception. Some of these studies used nonhuman primate subjects, enabling direct comparisons between behavioral performance and simultaneously recorded neuronal activity. The results indicate that humans and monkeys can integrate visual and vestibular heading cues in a manner consistent with optimal integration theory, and that single neurons in the dorsal medial superior temporal area show striking correlates of the behavioral effects. This line of research and other applications of normative cue combination models should continue to shed light on mechanisms of self-motion perception and the neuronal basis of multisensory integration.

Cue integration with categories: Weighting acoustic cues in speech using unsupervised learning and distributional statistics

During speech perception, listeners make judgments about the phonological category of sounds by taking advantage of multiple acoustic cues for each phonological contrast. Perceptual experiments have shown that listeners weight these cues differently. How do listeners weight and combine acoustic cues to arrive at an overall estimate of the category for a speech sound? Here, we present several simulations using mixture of Gaussians (MOG) models that learn cue weights and combine cues on the basis of their distributional statistics. We show that a cue-weighting metric in which cues receive weight as a function of their reliability at distinguishing the phonological categories provides a good fit to the perceptual data obtained from human listeners, but only when these weights emerge through the dynamics of learning. These results suggest that cue weights can be readily extracted from the speech signal through unsupervised learning processes.

Within- and cross-modal distance information disambiguate visual size-change perception

Perception is fundamentally underconstrained because different combinations of object properties can generate the same sensory information. To disambiguate sensory information into estimates of scene properties, our brains incorporate prior knowledge and additional "auxiliary" (i.e., not directly relevant to desired scene property) sensory information to constrain perceptual interpretations. For example, knowing the distance to an object helps in perceiving its size. The literature contains few demonstrations of the use of prior knowledge and auxiliary information in combined visual and haptic disambiguation and almost no examination of haptic disambiguation of vision beyond "bistable" stimuli. Previous studies have reported humans integrate multiple unambiguous sensations to perceive single, continuous object properties, like size or position. Here we test whether humans use visual and haptic information, individually and jointly, to disambiguate size from distance. We presented participants with a ball moving in depth with a changing diameter. Because no unambiguous distance information is available under monocular viewing, participants rely on prior assumptions about the ball's distance to disambiguate their -size percept. Presenting auxiliary binocular and/or haptic distance information augments participants' prior distance assumptions and improves their size judgment accuracy-though binocular cues were trusted more than haptic. Our results suggest both visual and haptic distance information disambiguate size perception, and we interpret these results in the context of probabilistic perceptual reasoning.

Dynamic reweighting of visual and vestibular cues during self-motion perception

The perception of self-motion direction, or heading, relies on integration of multiple sensory cues, especially from the visual and vestibular systems. However, the reliability of sensory information can vary rapidly and unpredictably, and it remains unclear how the brain integrates multiple sensory signals given this dynamic uncertainty. Human psychophysical studies have shown that observers combine cues by weighting them in proportion to their reliability, consistent with statistically optimal integration schemes derived from Bayesian probability theory. Remarkably, because cue reliability is varied randomly across trials, the perceptual weight assigned to each cue must change from trial to trial. Dynamic cue reweighting has not been examined for combinations of visual and vestibular cues, nor has the Bayesian cue integration approach been applied to laboratory animals, an important step toward understanding the neural basis of cue integration. To address these issues, we tested human and monkey subjects in a heading discrimination task involving visual (optic flow) and vestibular (translational motion) cues. The cues were placed in conflict on a subset of trials, and their relative reliability was varied to assess the weights that subjects gave to each cue in their heading judgments. We found that monkeys can rapidly reweight visual and vestibular cues according to their reliability, the first such demonstration in a nonhuman species. However, some monkeys and humans tended to over-weight vestibular cues, inconsistent with simple predictions of a Bayesian model. Nonetheless, our findings establish a robust model system for studying the neural mechanisms of dynamic cue reweighting in multisensory perception.

Is perceptual space inherently non-Euclidean?

It is often assumed that the space we perceive is Euclidean, although this idea has been challenged by many authors. Here we show that, if spatial cues are combined as described by Maximum Likelihood Estimation, Bayesian, or equivalent models, as appears to be the case, then Euclidean geometry cannot describe our perceptual experience. Rather, our perceptual spatial structure would be better described as belonging to an arbitrarily curved Riemannian space.

Humans use visual and remembered information about object location to plan pointing movements

We investigated whether humans use a target's remembered location to plan reaching movements to targets according to the relative reliabilities of visual and remembered information. Using their index finger, subjects moved a virtual object from one side of a table to the other, and then went back to a target. In some trials, the target shifted unnoticed while the finger made the first movement. We regressed subjects' movement trajectories against the initial and shifted target locations to infer the weights that subjects gave to remembered and visual locations. We measured the reliability of vision and memory by adding conditions in which the target only appeared after subjects made the first movement (vision only) and in which the target was initially present but disappeared during the first movement (memory only). When both visual and remembered information were available, movement trajectories were biased to the remembered target location. The different weights that subjects gave to memory and visual information on average matched the weights predicted by the variance associated with the use of vision and memory alone. This suggests that humans integrate remembered information about object locations with peripheral visual information by taking into account the relative reliability of the two sources of information.

Multisensory oddity detection as bayesian inference

A key goal for the perceptual system is to optimally combine information from all the senses that may be available in order to develop the most accurate and unified picture possible of the outside world. The contemporary theoretical framework of ideal observer maximum likelihood integration (MLI) has been highly successful in modelling how the human brain combines information from a variety of different sensory modalities. However, in various recent experiments involving multisensory stimuli of uncertain correspondence, MLI breaks down as a successful model of sensory combination. Within the paradigm of direct stimulus estimation, perceptual models which use Bayesian inference to resolve correspondence have recently been shown to generalize successfully to these cases where MLI fails. This approach has been known variously as model inference, causal inference or structure inference. In this paper, we examine causal uncertainty in another important class of multi-sensory perception paradigm--that of oddity detection and demonstrate how a Bayesian ideal observer also treats oddity detection as a structure inference problem. We validate this approach by showing that it provides an intuitive and quantitative explanation of an important pair of multi-sensory oddity detection experiments--involving cues across and within modalities--for which MLI previously failed dramatically, allowing a novel unifying treatment of within and cross modal multisensory perception. Our successful application of structure inference models to the new 'oddity detection' paradigm, and the resultant unified explanation of across and within modality cases provide further evidence to suggest that structure inference may be a commonly evolved principle for combining perceptual information in the brain.

Efficient integration across spatial frequencies for letter identification in foveal and peripheral vision

Objects in natural scenes are spatially broadband; in contrast, feature detectors in the early stages of visual processing are narrowly tuned in spatial frequency. Earlier studies of feature integration using gratings suggested that integration across spatial frequencies is suboptimal. Here we re-examined this conclusion using a letter identification task at the fovea and at 10 deg in the lower visual field. We found that integration across narrow-band (1-octave) spatial frequency components of letter stimuli is optimal in the fovea. Surprisingly, this optimality is preserved in the periphery, even though feature integration is known to be deficient in the periphery from studies of other form-vision tasks such as crowding. A model that is otherwise a white-noise ideal observer except for a limited spatial resolution defined by the human contrast sensitivity function and using internal templates slightly wider in bandwidth than the stimuli is able to account for the human data. Our findings suggest that deficiency in feature integration found in peripheral vision is not across spatial frequencies.

The statistical determinants of adaptation rate in human reaching

Rapid reaching to a target is generally accurate but also contains random and systematic error. Random errors result from noise in visual measurement, motor planning, and reach execution. Systematic error results from systematic changes in the mapping between the visual estimate of target location and the motor command necessary to reach the target (e.g., new spectacles, muscular fatigue). Humans maintain accurate reaching by recalibrating the visuomotor system, but no widely accepted computational model of the process exists. Given certain boundary conditions, a statistically optimal solution is a Kalman filter. We compared human to Kalman filter behavior to determine how humans take into account the statistical properties of errors and the reliability with which those errors can be measured. For most conditions, human and Kalman filter behavior was similar: Increasing measurement uncertainty caused similar decreases in recalibration rate; directionally asymmetric uncertainty caused different rates in different directions; more variation in systematic error increased recalibration rate. However, behavior differed in one respect: Inserting random error by perturbing feedback position causes slower adaptation in Kalman filters but had no effect in humans. This difference may be due to how biological systems remain responsive to changes in environmental statistics. We discuss the implications of this work.

Implicit knowledge of visual uncertainty guides decisions with asymmetric outcomes

Perception is an "inverse problem," in which the state of the world must be inferred from the sensory neural activity that results. However, this inference is both ill-posed (Helmholtz, 1856; Marr, 1982) and corrupted by noise (Green & Swets, 1989), requiring the brain to compute perceptual beliefs under conditions of uncertainty. Here we show that human observers performing a simple visual choice task under an externally imposed loss function approach the optimal strategy, as defined by Bayesian probability and decision theory (Berger, 1985; Cox, 1961). In concert with earlier work, this suggests that observers possess a model of their internal uncertainty and can utilize this model in the neural computations that underlie their behavior (Knill & Pouget, 2004). In our experiment, optimal behavior requires that observers integrate the loss function with an estimate of their internal uncertainty rather than simply requiring that they use a modal estimate of the uncertain stimulus. Crucially, they approach optimal behavior even when denied the opportunity to learn adaptive decision strategies based on immediate feedback. Our data thus support the idea that flexible representations of uncertainty are pre-existing, widespread, and can be propagated to decision-making areas of the brain.

Can a space-perception conflict be solved with three sense modalities?

A cross-modal conflict over location was resolved in an unexpected way. When vision and proprioception provide conflicting information, which modality should dominate is ambiguous. A visual-proprioceptive conflict was created with a prism and, to logically disambiguate the problem, auditory information was added that either agreed with vision (group 1), agreed with proprioception (group 2), or was absent (group 3). While a scarcity of research addresses the interaction of three modalities, I predicted error should be attributed to the modality in the minority. Instead, the opposite was found: adaptation consisted of a large change in arm proprioception and a small change affecting vision in group 2, and the reverse in group 1. Group 1 was not different than group 3. Findings suggested adaptation to separate two-way conflicts, possibly influenced by direction of attention, rather than a direct solution to a three-way modality problem.

Ecologically valid combinations of first- and second-order surface markings facilitate texture discrimination

Natural scenes contain localized variations in both first-order (luminance) and second-order (contrast and texture) information. There is much evidence that first- and second-order stimuli are detected by distinct mechanisms in the mammalian visual system. However, in natural scenes the two kinds of information tend to be spatially correlated. Do correlated and uncorrelated combinations of first- and second-order stimuli differentially affect perception? To address this question we employed orientation-modulated textures in which observers were required to discriminate the spatial frequency of the texture modulation. The textures consisted of micropatterns defined as either local variations in luminance (first-order) or luminance contrast (second-order). Performance was robust with textures composed of only first-order micropatterns, but impossible with only second-order micropatterns. However, when the second-order micropatterns were combined with the first-order micropatterns, they enhanced performance when the two were spatially correlated, but impaired performance when the two were spatially uncorrelated. We conclude that local second-order information may enhance texture modulation discrimination provided it is combined with first-order information in an ecologically valid manner.

Optimal integration of shape information from vision and touch

Many tasks can be carried out by using several sources of information. For example, an object's size and shape can be judged based on visual as well as haptic cues. It has been shown recently that human observers integrate visual and haptic size information in a statistically optimal fashion, in the sense that the integrated estimate is most reliable (Ernst and Banks in Nature 415:429-433, 2002). In the present study, we tested whether this holds also for visual and haptic shape information. In previous studies virtual stimuli were used to test for optimality in integration. Virtual displays may, however, contain additional inappropriate cues that provide conflicting information and thus affect cue integration. Therefore, we studied optimal integration using real objects. Furthermore, we presented visual information via mirrors to create a spatial separation between visual and haptic cues while observers saw their hand touching the object and thus, knew that they were seeing and feeling the same object. Does this knowledge promote integration even though signals are spatially discrepant which has been shown to lead to a breakdown of integration (Gepshtein et al. in J Vis 5:1013-1023, 2005)? Consistent with the model predictions, observers weighted visual and haptic cues to shape according to their reliability: progressively more weight was given to haptics when visual information became less reliable. Moreover, the integrated visual-haptic estimate was more reliable than either unimodal estimate. These findings suggest that observers integrate visual and haptic shape information of real 3D objects. Thereby, knowledge that multisensory signals arise from the same object seems to promote integration.

Probabilistic population codes and the exponential family of distributions

Many experiments have shown that human behavior is nearly Bayes optimal in a variety of tasks. This implies that neural activity is capable of representing both the value and uncertainty of a stimulus, if not an entire probability distribution, and can also combine such representations in an optimal manner. Moreover, this computation can be performed optimally despite the fact that observed neural activity is highly variable (noisy) on a trial-by-trial basis. Here, we argue that this observed variability is actually expected in a neural system which represents uncertainty. Specifically, we note that Bayes' rule implies that a variable pattern of activity provides a natural representation of a probability distribution, and that the specific form of neural variability can be structured so that optimal inference can be executed using simple operations available to neural circuits.

A Bayesian model of the disambiguation of gravitoinertial force by visual cues

The otoliths are stimulated in the same fashion by gravitational and inertial forces, so otolith signals are ambiguous indicators of self-orientation. The ambiguity can be resolved with added visual information indicating orientation and acceleration with respect to the earth. Here we present a Bayesian model of the statistically optimal combination of noisy vestibular and visual signals. Likelihoods associated with sensory measurements are represented in an orientation/acceleration space. The likelihood function associated with the otolith signal illustrates the ambiguity; there is no unique solution for self-orientation or acceleration. Likelihood functions associated with other sensory signals can resolve this ambiguity. In addition, we propose two priors, each acting on a dimension in the orientation/acceleration space: the idiotropic prior and the no-acceleration prior. We conducted experiments using a motion platform and attached visual display to examine the influence of visual signals on the interpretation of the otolith signal. Subjects made pitch and acceleration judgments as the vestibular and visual signals were manipulated independently. Predictions of the model were confirmed: (1) visual signals affected the interpretation of the otolith signal, (2) less variable signals had more influence on perceived orientation and acceleration than more variable ones, and (3) combined estimates were more precise than single-cue estimates. We also show that the model can explain some well-known phenomena including the perception of upright in zero gravity, the Aubert effect, and the somatogravic illusion.

Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration

In order to maintain a coherent, unified percept of the external environment, the brain must continuously combine information encoded by our different sensory systems. Contemporary models suggest that multisensory integration produces a weighted average of sensory estimates, where the contribution of each system to the ultimate multisensory percept is governed by the relative reliability of the information it provides (maximum-likelihood estimation). In the present study, we investigate interactions between auditory and visual rate perception, where observers are required to make judgments in one modality while ignoring conflicting rate information presented in the other. We show a gradual transition between partial cue integration and complete cue segregation with increasing inter-modal discrepancy that is inconsistent with mandatory implementation of maximum-likelihood estimation. To explain these findings, we implement a simple Bayesian model of integration that is also able to predict observer performance with novel stimuli. The model assumes that the brain takes into account prior knowledge about the correspondence between auditory and visual rate signals, when determining the degree of integration to implement. This provides a strategy for balancing the benefits accrued by integrating sensory estimates arising from a common source, against the costs of conflating information relating to independent objects or events.

Cue combination in a combined feature contrast detection and figure identification task

Target figures defined by feature contrast in spatial frequency, orientation or both cues had to be detected in Gabor random fields and their shape had to be identified in a dual task paradigm. Performance improved with increasing feature contrast and was strongly correlated among both tasks. Subjects performed significantly better with combined cues than with single cues. The improvement due to cue summation was stronger than predicted by the assumption of independent feature specific mechanisms, and increased with the performance level achieved with single cues until it was limited by ceiling effects. Further, cue summation was also strongly correlated among tasks: when there was benefit due to the additional cue in feature contrast detection, there was also benefit in figure identification. For the same performance level achieved with single cues, cue summation was generally larger in figure identification than in feature contrast detection, indicating more benefit when processes of shape and surface formation are involved. Our results suggest that cue combination improves spatial form completion and figure-ground segregation in noisy environments, and therefore leads to more stable object vision.

Combining priors and noisy visual cues in a rapid pointing task

Statistical decision theory suggests that choosing an ideal action requires taking several factors into account: (1) prior knowledge of the probability of various world states, (2) sensory information concerning the world state, (3) the probability of outcomes given a choice of action, and (4) the loss or gain associated with those outcomes. In previous work, we found that, in many circumstances, humans act like ideal decision makers in planning a reaching movement. They select a movement aim point that maximizes expected gain, thus taking into account outcome uncertainty (motor noise) and the consequences of their actions. Here, we ask whether humans can optimally combine prior knowledge and uncertain sensory information in planning a reach. Subjects rapidly pointed at unseen targets, indicated with dots drawn from a distribution centered on the invisible target location. Target location had a prior distribution, the form of which was known to the subject. We varied the number of dots and hence target spatial uncertainty. An analysis of the sources of uncertainty impacting performance in this task indicated that the optimal strategy was to aim between the mean of the prior (the screen center) and the mean stimulus location (centroid of the dot cloud). With increased target location uncertainty, the aim point should have moved closer to the prior. Subjects used near-optimal strategies, combining stimulus uncertainty and prior information appropriately. Observer behavior was well modeled as having three additional sources of inefficiency originating in the motor system, calculation of centroid location, and calculation of aim points.

Stereo and motion information are not independently processed by the visual system

Many visual tasks are carried out by using multiple sources of sensory information to estimate environmental properties. In this paper, we present a model for how the visual system combines disparity and velocity information. We propose that, in a first stage of processing, the best possible estimate of the affine structure is obtained by computing a composite score from the disparity and velocity signals. In a second stage, a maximum likelihood Euclidean interpretation is assigned to the recovered affine structure. In two experiments, we show that human performance is consistent with the predictions of our model. The present results are also discussed in the framework of another theoretical approach of the depth cue combination process termed Modified Weak Fusion.

Why seeing is believing: merging auditory and visual worlds

Vision may dominate our perception of space not because of any inherent physiological advantage of visual over other sensory connections in the brain, but because visual information tends to be more reliable than other sources of spatial information, and the central nervous system integrates information in a statistically optimal fashion. This review discusses recent experiments on audiovisual integration that support this hypothesis. We consider candidate neural codes that would enable optimal integration and the implications of optimal integration for perception and plasticity.

Focus cues affect perceived depth

Depth information from focus cues--accommodation and the gradient of retinal blur--is typically incorrect in three-dimensional (3-D) displays because the light comes from a planar display surface. If the visual system incorporates information from focus cues into its calculation of 3-D scene parameters, this could cause distortions in perceived depth even when the 2-D retinal images are geometrically correct. In Experiment 1 we measured the direct contribution of focus cues to perceived slant by varying independently the physical slant of the display surface and the slant of a simulated surface specified by binocular disparity (binocular viewing) or perspective/texture (monocular viewing). In the binocular condition, slant estimates were unaffected by display slant. In the monocular condition, display slant had a systematic effect on slant estimates. Estimates were consistent with a weighted average of slant from focus cues and slant from disparity/texture, where the cue weights are determined by the reliability of each cue. In Experiment 2, we examined whether focus cues also have an indirect effect on perceived slant via the distance estimate used in disparity scaling. We varied independently the simulated distance and the focal distance to a disparity-defined 3-D stimulus. Perceived slant was systematically affected by changes in focal distance. Accordingly, depth constancy (with respect to simulated distance) was significantly reduced when focal distance was held constant compared to when it varied appropriately with the simulated distance to the stimulus. The results of both experiments show that focus cues can contribute to estimates of 3-D scene parameters. Inappropriate focus cues in typical 3-D displays may therefore contribute to distortions in perceived space.

Explicit estimation of visual uncertainty in human motion processing

We examine whether human observers have explicit access to an estimate of their own uncertainty in extrapolating the motion trajectories of moving objects. Objects moved across a display area at constant speed changing direction at short time intervals. Each new direction was obtained by adding a random perturbation to the previous direction. The perturbation distribution was always symmetric with mean zero (no change in direction) but could differ in variability: objects with low directional variability tended to travel in straight lines while objects with high directional variability moved more erratically. Objects eventually disappeared behind the near edge of an occluder. Observers marked a 'capture region' along the far edge of the occluder that they estimated would contain the object when it re-emerged. We varied both occluder width and directional variability across trials and found that observers correctly compensated for these changes. We present a two-stage model of observer performance in which the visual system first estimates the directional variability of the object and then uses this estimate to set a capture region.

Sensory uncertainty governs the extent of audio-visual interaction

Auditory signals have been shown to exert a marked influence on visual perception in a wide range of tasks. However, the mechanisms of these interactions are, at present, poorly understood. Here we present a series of experiments where a temporal cue within the auditory domain can significantly affect the localisation of a moving visual target. To investigate the mechanism of this interaction, we first modulated the spatial positional uncertainty of the visual target by varying its size. When visual positional uncertainty was low (small target size), auditory signals had little or no influence on perceived visual location. However, with increasing visual uncertainty (larger target sizes), auditory signals exerted a significantly greater influence on perceived visual location. We then altered the temporal profile of the auditory signal by modulating the spread of its Gaussian temporal envelope. Introducing this temporal uncertainty to the auditory signal greatly reduced its effect on visual localisation judgements. These findings support the view that the relative uncertainty in individual sensory domains governs the perceptual outcome of multisensory integration.

Stability and change in perception: spatial organization in temporal context

Perceptual multistability has often been explained using the concepts of adaptation and hysteresis. In this paper we show that effects that would typically be accounted for by adaptation and hysteresis can be explained without assuming the existence of dedicated mechanisms for adaptation and hysteresis. Instead, our data suggest that perceptual multistability reveals lasting states of the visual system rather than changes in the system caused by stimulation. We presented observers with two successive multistable stimuli and found that the probability that they saw the favored organization in the first stimulus was inversely related to the probability that they saw the same organization in the second. This pattern of negative contingency is orientation-tuned and occurs no matter whether the observer had or had not seen the favored organization in the first stimulus. This adaptation-like effect of negative contingency combines multiplicatively with a hysteresis-like effect that increases the likelihood of the just-perceived organization. Both effects are consistent with a probabilistic model in which perception depends on an orientation-tuned intrinsic bias that slowly (and stochastically) changes its orientation tuning over time.

Motion-Sensitive Neurones in V5/MT Modulate Perceived Spatial Position

Until recently, it was widely believed that object position and object motion were represented independently in the visual cortex. However, several studies have shown that adaptation to motion produces substantial shifts in the perceived position of subsequently viewed stationary objects. Two stages of motion adaptation have been proposed: an initial stage at the level of V1 and a secondary stage thought to be located in V5/MT. Indeed, selective adaptation can be demonstrated at each of these levels of motion analysis. What remains unknown is which of these cortical sites are involved in modulating the positional representation of subsequently viewed objects. To answer this question directly, we disrupted cortical activity by using transcranial magnetic stimulation (TMS) immediately after motion adaptation. When TMS was delivered to V5/MT after motion adaptation, the perceived offset of the test stimulus was greatly reduced. In marked contrast, TMS of V1 had no effect on the changes that normally occur in perceived position after motion adaptation. This result demonstrates that the anatomical locus at which motion and positional information interact is area V5/MT rather than V1/V2.

Neuroscience: what you see and hear is what you get

The brain receives signals from a variety of sources; for example, visual and auditory signals can both indicate the direction of a stimulus, but with differing precision. A recent study has shed light on the way that the brain combines these signals to achieve the best estimate possible.

Object perception as Bayesian inference

We perceive the shapes and material properties of objects quickly and reliably despite the complexity and objective ambiguities of natural images. Typical images are highly complex because they consist of many objects embedded in background clutter. Moreover, the image features of an object are extremely variable and ambiguous owing to the effects of projection, occlusion, background clutter, and illumination. The very success of everyday vision implies neural mechanisms, yet to be understood, that discount irrelevant information and organize ambiguous or noisy local image features into objects and surfaces. Recent work in Bayesian theories of visual perception has shown how complexity may be managed and ambiguity resolved through the task-dependent, probabilistic integration of prior object knowledge with image features.

Weighted linear cue combination with possibly correlated error

We test hypotheses concerning human cue combination in a slant estimation task. Observers repeatedly adjusted the slant of a plane to 75 degrees. Feedback was provided after each setting and the observers trained extensively until their setting error stabilized. The slant of the plane was defined by either linear perspective alone (a grid of lines) or texture gradient alone (diamond-shaped texture elements) or the two cues together. We chose a High and Low variance version of each cue type and measured setting variability in four single-cue conditions (Low, High for each cue) and in the four possible combined-cue conditions (Low-Low, Low-High, etc.). We compared performance in the combined-cue conditions to predictions based on single-cue performance. The results were consistent with a linear combination of estimates from cues. Six out of eight observers did better with combined cues than with either cue alone. For three observers, performance was consistent with optimal combination of uncorrelated cues. Three other observers' results were also consistent with optimal combination, but with the assumption that internal cue estimates were correlated. The remaining two observers were consistent with sub-optimal cue combination.

Bayesian models of object perception

The human visual system is the most complex pattern recognition device known. In ways that are yet to be fully understood, the visual cortex arrives at a simple and unambiguous interpretation of data from the retinal image that is useful for the decisions and actions of everyday life. Recent advances in Bayesian models of computer vision and in the measurement and modeling of natural image statistics are providing the tools to test and constrain theories of human object perception. In turn, these theories are having an impact on the interpretation of cortical function.

Viewing geometry determines how vision and haptics combine in size perception

Vision and haptics have different limitations and advantages because they obtain information by different methods. If the brain combined information from the two senses optimally, it would rely more on the one providing more precise information for the current task. In this study, human observers judged the distance between two parallel surfaces in two within-modality experiments (vision-alone and haptics-alone) and in an intermodality experiment (vision and haptics together). In the within-modality experiments, the precision of visual estimates varied with surface orientation, as expected from geometric considerations; the precision of haptic estimates did not. An ideal observer that combines visual and haptic information weights them differently as a function of orientation. In the intermodality experiment, humans adjusted visual and haptic weights in a fashion quite similar to that of the ideal observer. As a result, combined size estimates are finer than is possible with either vision or haptics alone; indeed, they approach statistical optimality.

Combining sensory information: mandatory fusion within, but not between, senses

Humans use multiple sources of sensory information to estimate environmental properties. For example, the eyes and hands both provide relevant information about an object's shape. The eyes estimate shape using binocular disparity, perspective projection, etc. The hands supply haptic shape information by means of tactile and proprioceptive cues. Combining information across cues can improve estimation of object properties but may come at a cost: loss of single-cue information. We report that single-cue information is indeed lost when cues from within the same sensory modality (disparity and texture gradients in vision) are combined, but not when different modalities (vision and haptics) are combined.

What determines visual cue reliability?

Visual environments contain many cues to properties of an observed scene. To integrate information provided by multiple cues in an efficient manner, observers must assess the degree to which each cue provides reliable versus unreliable information. Two hypotheses are reviewed regarding how observers estimate cue reliabilities, namely that the estimated reliability of a cue is related to the ambiguity of the cue, and that people use correlations among cues to estimate cue reliabilities. Cue reliabilities are shown to be important both for cue combination and for aspects of visual learning.