Ideal cue combination for localizing texture-defined edges

Correctly establishing evidence for cue combination via gains in sensory precision: Why the choice of comparator matters

Studying how sensory signals from different sources (sensory cues) are integrated within or across multiple senses allows us to better understand the perceptual computations that lie at the foundation of adaptive behaviour. As such, determining the presence of precision gains - the classic hallmark of cue combination - is important for characterising perceptual systems, their development and functioning in clinical conditions. However, empirically measuring precision gains to distinguish cue combination from alternative perceptual strategies requires careful methodological considerations. Here, we note that the majority of existing studies that tested for cue combination either omitted this important contrast, or used an analysis approach that, unknowingly, strongly inflated false positives. Using simulations, we demonstrate that this approach enhances the chances of finding significant cue combination effects in up to 100% of cases, even when cues are not combined. We establish how this error arises when the wrong cue comparator is chosen and recommend an alternative analysis that is easy to implement but has only been adopted by relatively few studies. By comparing combined-cue perceptual precision with the best single-cue precision, determined for each observer individually rather than at the group level, researchers can enhance the credibility of their reported effects. We also note that testing for deviations from optimal predictions alone is not sufficient to ascertain whether cues are combined. Taken together, to correctly test for perceptual precision gains, we advocate for a careful comparator selection and task design to ensure that cue combination is tested with maximum power, while reducing the inflation of false positives.

Perceiving depth and motion in depth from successive occlusion

Occlusion, or interposition, is one of the strongest and best-known pictorial cues to depth. Furthermore, the successive occlusions of previous objects by newly presented objects produces an impression of increasing depth. Although the perceived motion associated with this illusion has been studied, the depth percept has not. To investigate, participants were presented with two piles of disks with one always static and the other either a static pile or a stacking pile where a new disk was added every 200 ms. We found static piles with equal number of disks appeared equal in height. In contrast, the successive presentation of disks in the stacking condition appeared to enhance the perceived height of the stack-fewer disks were needed to match the static pile. Surprisingly, participants were also more precise when comparing stacking versus static piles of disks. Reversing the stacking by removing rather than adding disks reversed the bias and degraded precision. In follow-up experiments, we used nonoverlapping static and dynamic configurations to show that the effects are not due to simple differences in perceived numerosity. In sum, our results show that successive occlusions generate a greater sense of height than occlusion alone, and we posit that dynamic occlusion may be an underappreciated source of depth information.

Multisensory perception constrains the formation of object categories: a review of evidence from sensory-driven and predictive processes on categorical decisions

Although object categorization is a fundamental cognitive ability, it is also a complex process going beyond the perception and organization of sensory stimulation. Here we review existing evidence about how the human brain acquires and organizes multisensory inputs into object representations that may lead to conceptual knowledge in memory. We first focus on evidence for two processes on object perception, multisensory integration of redundant information (e.g. seeing and feeling a shape) and crossmodal, statistical learning of complementary information (e.g. the 'moo' sound of a cow and its visual shape). For both processes, the importance attributed to each sensory input in constructing a multisensory representation of an object depends on the working range of the specific sensory modality, the relative reliability or distinctiveness of the encoded information and top-down predictions. Moreover, apart from sensory-driven influences on perception, the acquisition of featural information across modalities can affect semantic memory and, in turn, influence category decisions. In sum, we argue that both multisensory processes independently constrain the formation of object categories across the lifespan, possibly through early and late integration mechanisms, respectively, to allow us to efficiently achieve the everyday, but remarkable, ability of recognizing objects. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Multisensory decisions from self to world

Classic Bayesian models of perceptual inference describe how an ideal observer would integrate 'unisensory' measurements (multisensory integration) and attribute sensory signals to their origin(s) (causal inference). However, in the brain, sensory signals are always received in the context of a multisensory bodily state-namely, in combination with other senses. Moreover, sensory signals from both interoceptive sensing of one's own body and exteroceptive sensing of the world are highly interdependent and never occur in isolation. Thus, the observer must fundamentally determine whether each sensory observation is from an external (versus internal, self-generated) source to even be considered for integration. Critically, solving this primary causal inference problem requires knowledge of multisensory and sensorimotor dependencies. Thus, multisensory processing is needed to separate sensory signals. These multisensory processes enable us to simultaneously form a sense of self and form distinct perceptual decisions about the external world. In this opinion paper, we review and discuss the similarities and distinctions between multisensory decisions underlying the sense of self and those directed at acquiring information about the world. We call attention to the fact that heterogeneous multisensory processes take place all along the neural hierarchy (even in forming 'unisensory' observations) and argue that more integration of these aspects, in theory and experiment, is required to obtain a more comprehensive understanding of multisensory brain function. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Measuring uncertainty in human visual segmentation

Segmenting visual stimuli into distinct groups of features and visual objects is central to visual function. Classical psychophysical methods have helped uncover many rules of human perceptual segmentation, and recent progress in machine learning has produced successful algorithms. Yet, the computational logic of human segmentation remains unclear, partially because we lack well-controlled paradigms to measure perceptual segmentation maps and compare models quantitatively. Here we propose a new, integrated approach: given an image, we measure multiple pixel-based same-different judgments and perform model-based reconstruction of the underlying segmentation map. The reconstruction is robust to several experimental manipulations and captures the variability of individual participants. We demonstrate the validity of the approach on human segmentation of natural images and composite textures. We show that image uncertainty affects measured human variability, and it influences how participants weigh different visual features. Because any putative segmentation algorithm can be inserted to perform the reconstruction, our paradigm affords quantitative tests of theories of perception as well as new benchmarks for segmentation algorithms.

Sensitivity to naturalistic texture relies primarily on high spatial frequencies

Natural images contain information at multiple spatial scales. Though we understand how early visual mechanisms split multiscale images into distinct spatial frequency channels, we do not know how the outputs of these channels are processed further by mid-level visual mechanisms. We have recently developed a texture discrimination task that uses synthetic, multi-scale, "naturalistic" textures to isolate these mid-level mechanisms. Here, we use three experimental manipulations (image blur, image rescaling, and eccentric viewing) to show that perceptual sensitivity to naturalistic structure is strongly dependent on features at high object spatial frequencies (measured in cycles/image). As a result, sensitivity depends on a texture acuity limit, a property of the visual system that sets the highest retinal spatial frequency (measured in cycles/degree) at which observers can detect naturalistic features. Analysis of the texture images using a model observer analysis shows that naturalistic image features at high object spatial frequencies carry more task-relevant information than those at low object spatial frequencies. That is, the dependence of sensitivity on high object spatial frequencies is a property of the texture images, rather than a property of the visual system. Accordingly, we find human observers' ability to extract naturalistic information (their efficiency) is similar for all object spatial frequencies. We conclude that the mid-level mechanisms that underlie perceptual sensitivity effectively extract information from all image features below the texture acuity limit, regardless of their retinal and object spatial frequency.

New Approaches to 3D Vision

New approaches to 3D vision are enabling new advances in artificial intelligence and autonomous vehicles, a better understanding of how animals navigate the 3D world, and new insights into human perception in virtual and augmented reality. Whilst traditional approaches to 3D vision in computer vision (SLAM: simultaneous localization and mapping), animal navigation (cognitive maps), and human vision (optimal cue integration) start from the assumption that the aim of 3D vision is to provide an accurate 3D model of the world, the new approaches to 3D vision explored in this issue challenge this assumption. Instead, they investigate the possibility that computer vision, animal navigation, and human vision can rely on partial or distorted models or no model at all. This issue also highlights the implications for artificial intelligence, autonomous vehicles, human perception in virtual and augmented reality, and the treatment of visual disorders, all of which are explored by individual articles. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Overconfidence in visual perception in parkinson's disease

Increased dependence on visual cues in Parkinson's disease (PD) can unbalance the perception-action loop, impair multisensory integration, and affect everyday function of PD patients. It is currently unknown why PD patients seem to be more reliant on their visual cues. We hypothesized that PD patients may be overconfident in the reliability (precision) of their visual cues. In this study we tested coherent visual motion perception in PD, and probed subjective (self-reported) confidence in their visual motion perception. Twenty patients with idiopathic PD, 21 healthy aged-matched controls and 20 healthy young adult participants were presented with visual stimuli of moving dots (random dot kinematograms). They were asked to report: (1) whether the aggregate motion of dots was to the left or to the right, and (2) how confident they were that their perceptual discrimination was correct. Visual motion discrimination thresholds were similar (unimpaired) in PD compared to the other groups. By contrast, PD patients were significantly overconfident in their visual perceptual decisions (p = .002 and p < .001 vs. the age-matched and young adult groups, respectively). These results suggest intact visual motion perception, but overestimation of visual cue reliability, in PD. Overconfidence in visual (vs. other, e.g., somatosensory) cues could underlie increased visual dependence and impaired multisensory/sensorimotor integration in PD. It could thereby contribute to gait and balance impairments, and affect everyday activities, such as driving. Future work should investigate and compare PD confidence in somatosensory function. A better understanding of altered sensory reliance might open up new avenues to treat debilitating PD symptoms.

Pre-Calibrated Visuo-Haptic Co-Location Improves Execution in Virtual Environments

A common approach to visio-haptic human-machine interfaces adopts a simpler design by shifting grounded force feedback away from the virtual scene. The alternative design favors intuitiveness by displaying visual and grounded force feedback at the same location (i.e. visuo-haptic co-location) but requires a sensibly more complex implementation and a tedious kinematic conception. The benefits of one approach over the other had not been fully investigated. Notably, (i) while users seem to better operate under co-located condition, it's not always the case. (ii) In the case of a desktop interface, the cost of a complex implementation to achieve co-location is challenged. We aim here to resolve (i) by conducting a user-centered experiment in which participants performed two generic tasks in co-located and delocated configurations, and comparing their performances. Additionally, we intend to fill the gap (ii) by testing a design without continuous head tracking, i.e., with static co-location. Participants' performances are assessed in terms of execution time, accuracy and force variation, while their subjective experiences are collected via a survey. Findings indicate that co-located configurations lead to shorter execution times, more accurate motions, better management of forces and are largely preferred by users, even when the co-location is pre-calibrated statically.

Visual self-motion cues are impaired yet overweighted during visual-vestibular integration in Parkinson's disease

Parkinson's disease is prototypically a movement disorder. Although perceptual and motor functions are highly interdependent, much less is known about perceptual deficits in Parkinson's disease, which are less observable by nature, and might go unnoticed if not tested directly. It is therefore imperative to seek and identify these, to fully understand the challenges facing patients with Parkinson's disease. Also, perceptual deficits may be related to motor symptoms. Posture, gait and balance, affected in Parkinson's disease, rely on veridical perception of one's own motion (self-motion) in space. Yet it is not known whether self-motion perception is impaired in Parkinson's disease. Using a well-established multisensory paradigm of heading discrimination (that has not been previously applied to Parkinson's disease), we tested unisensory visual and vestibular self-motion perception, as well as multisensory integration of visual and vestibular cues, in 19 Parkinson's disease, 23 healthy age-matched and 20 healthy young-adult participants. After experiencing vestibular (on a motion platform), visual (optic flow) or multisensory (combined visual-vestibular) self-motion stimuli at various headings, participants reported whether their perceived heading was to the right or left of straight ahead. Parkinson's disease participants and age-matched controls were tested twice (Parkinson's disease participants on and off medication). Parkinson's disease participants demonstrated significantly impaired visual self-motion perception compared with age-matched controls on both visits, irrespective of medication status. Young controls performed slightly (but not significantly) better than age-matched controls and significantly better than the Parkinson's disease group. The visual self-motion perception impairment in Parkinson's disease correlated significantly with clinical disease severity. By contrast, vestibular performance was unimpaired in Parkinson's disease. Remarkably, despite impaired visual self-motion perception, Parkinson's disease participants significantly overweighted the visual cues during multisensory (visual-vestibular ) integration (compared with Bayesian predictions of optimal integration) and significantly more than controls. These findings indicate that self-motion perception in Parkinson's disease is affected by impaired visual cues and by suboptimal visual-vestibular integration (overweighting of visual cues). Notably, vestibular self-motion perception was unimpaired. Thus, visual self-motion perception is specifically impaired in early-stage Parkinson's disease. This can impact Parkinson's disease diagnosis and subtyping. Overweighting of visual cues could reflect a general multisensory integration deficit in Parkinson's disease, or specific overestimation of visual cue reliability. Finally, impaired self-motion perception in Parkinson's disease may contribute to impaired balance and gait control. Future investigation into this connection might open up new avenues of alternative therapies to better treat these difficult symptoms.

Resolving the Spatial Profile of Figure Enhancement in Human V1 through Population Receptive Field Modeling

The detection and segmentation of meaningful figures from their background is one of the primary functions of vision. While work in nonhuman primates has implicated early visual mechanisms in this figure-ground modulation, neuroimaging in humans has instead largely ascribed the processing of figures and objects to higher stages of the visual hierarchy. Here, we used high-field fMRI at 7 Tesla to measure BOLD responses to task-irrelevant orientation-defined figures in human early visual cortex (N = 6, four females). We used a novel population receptive field mapping-based approach to resolve the spatial profiles of two constituent mechanisms of figure-ground modulation: a local boundary response, and a further enhancement spanning the full extent of the figure region that is driven by global differences in features. Reconstructing the distinct spatial profiles of these effects reveals that figure enhancement modulates responses in human early visual cortex in a manner consistent with a mechanism of automatic, contextually driven feedback from higher visual areas.SIGNIFICANCE STATEMENT A core function of the visual system is to parse complex 2D input into meaningful figures. We do so constantly and seamlessly, both by processing information about visible edges and by analyzing large-scale differences between figure and background. While influential neurophysiology work has characterized an intriguing mechanism that enhances V1 responses to perceptual figures, we have a poor understanding of how the early visual system contributes to figure-ground processing in humans. Here, we use advanced computational analysis methods and high-field human fMRI data to resolve the distinct spatial profiles of local edge and global figure enhancement in the early visual system (V1 and LGN); the latter is distinct and consistent with a mechanism of automatic, stimulus-driven feedback from higher-level visual areas.

The Effect of Ageing on Optimal Integration of Conflicting and Non-Conflicting Visual–Haptic Stimuli

Multisensory integration typically follows the predictions of a statistically optimal model whereby the contribution of each sensory modality is weighted according to its reliability. Previous research has shown that multisensory integration is affected by ageing, however it is less certain whether older adults follow this statistically optimal model. Additionally, previous studies often present multisensory cues which are conflicting in size, shape or location, yet naturally occurring multisensory cues are usually non-conflicting. Therefore, the mechanisms of integration in older adults might differ depending on whether the multisensory cues are consistent or conflicting. In the current experiment, young () and older () adults were asked to make judgements regarding the height of wooden blocks using visual, haptic or combined visual–haptic information. Dual modality visual–haptic blocks could be presented as equal or conflicting in size. Young and older adults’ size discrimination thresholds (i.e., precision) were not significantly different for visual, haptic or visual–haptic cues. In addition, both young and older adults’ discrimination thresholds and points of subjective equality did not follow model predictions of optimal integration, for both conflicting and non-conflicting cues. Instead, there was considerable between subject variability as to how visual and haptic cues were processed when presented simultaneously. This finding has implications for the development of multisensory therapeutic aids and interventions to assist older adults with everyday activities, where these should be tailored to the needs of each individual.

Effects of visuomotor delays on the control of movement and on perceptual localization in the presence and absence of visual targets

The sensory system constantly deals with delayed feedback. Recent studies showed that playing a virtual game of pong with delayed feedback caused hypermetric reaching movements. We investigated whether this effect is associated with a perceptual bias. In addition, we examined the importance of the target in causing hypermetric movements. In a first experiment, participants played a delayed pong game and blindly reached to presented targets. Following each reaching movement, they assessed the position of the invisible cursor. We found that participants performed hypermetric movements but reported that the invisible cursor reached the target, suggesting that they were unaware of the hypermetria and that their perception was biased toward the target rather than toward their hand position. In a second experiment, we removed the visual target, and strikingly, the hypermetria vanished. Moreover, participants reported that the invisible cursor was located with their hand. Taking these results together, we conclude that the adaptation to the visuomotor delay during the pong game selectively affected the execution of goal directed movements, resulting in hypermetria and perceptual bias when movements are directed toward visual targets but not when such targets are absent.NEW & NOTEWORTHY Recent studies showed that adaptation to visuomotor delays causes hypermetric movements in the absence of visual feedback, suggesting that visuomotor delay is represented using current state information. We report that this adaptation also affects perception. Importantly, both the motor and perceptual effects are selective to the representations that are used in the execution of goal-directed movements toward visual targets.

Auditory motion perception emerges from successive sound localizations integrated over time

Humans rely on auditory information to estimate the path of moving sound sources. But unlike in vision, the existence of motion-sensitive mechanisms in audition is still open to debate. Psychophysical studies indicate that auditory motion perception emerges from successive localization, but existing models fail to predict experimental results. However, these models do not account for any temporal integration. We propose a new model tracking motion using successive localization snapshots but integrated over time. This model is derived from psychophysical experiments on the upper limit for circular auditory motion perception (UL), defined as the speed above which humans no longer identify the direction of sounds spinning around them. Our model predicts ULs measured with different stimuli using solely static localization cues. The temporal integration blurs these localization cues rendering them unreliable at high speeds, which results in the UL. Our findings indicate that auditory motion perception does not require motion-sensitive mechanisms.

Reliance on Haptic Assistance Reflected in Haptic Cue Weighting

When using an automated system, user trust in the automation is an important factor influencing performance. Prior studies have analyzed trust duringsupervisory control of automation, and how trust influences reliance: the behavioral correlate of trust. Here, we investigated how reliance on haptic assistance affects performance during shared control with an automated system. Subjects made reaches towards a hidden target using a visual cue and haptic cue (assistance from the automation). We sought to influence reliance by changing the variability of trial-by-trial random errors in the haptic assistance. Reliance was quantified in terms of the subject's position at the end of the reach relative to the two cues. Our results show that subjects aimed more towards the visual cue when the variability of the haptic cue errors increased, resembling cue weighting behavior. Similar behavior was observed both when subjects had explicit knowledge about the haptic cue error variability, as well as when they had only implicit knowledge (from experience). However, the group with explicit knowledge was able to more quickly adapt their reliance on the haptic assistance. The method we introduce here provides a quantitative way to study user reliance on the information provided by automated systems with shared control.

The development of Bayesian integration in sensorimotor estimation

Examining development is important in addressing questions about whether Bayesian principles are hard coded in the brain. If the brain is inherently Bayesian, then behavior should show the signatures of Bayesian computation from an early stage in life. Children should integrate probabilistic information from prior and likelihood distributions to reach decisions and should be as statistically efficient as adults, when individual reliabilities are taken into account. To test this idea, we examined the integration of prior and likelihood information in a simple position-estimation task comparing children ages 6-11 years and adults. Some combination of prior and likelihood was present in the youngest sample tested (6-8 years old), and in most participants a Bayesian model fit the data better than simple baseline models. However, younger subjects tended to have parameters further from the optimal values, and all groups showed considerable biases. Our findings support some level of Bayesian integration in all age groups, with evidence that children use probabilistic quantities less efficiently than adults do during sensorimotor estimation.

Neck muscle spindle noise biases reaches in a multisensory integration task

Reference frame transformations (RFTs) are crucial components of sensorimotor transformations in the brain. Stochasticity in RFTs has been suggested to add noise to the transformed signal due to variability in transformation parameter estimates (e.g., angle) as well as the stochastic nature of computations in spiking networks of neurons. Here, we varied the RFT angle together with the associated variability and evaluated the behavioral impact in a reaching task that required variability-dependent visual-proprioceptive multisensory integration. Crucially, reaches were performed with the head either straight or rolled 30° to either shoulder, and we also applied neck loads of 0 or 1.8 kg (left or right) in a 3 × 3 design, resulting in different combinations of estimated head roll angle magnitude and variance required in RFTs. A novel three-dimensional stochastic model of multisensory integration across reference frames was fitted to the data and captured our main behavioral findings: 1) neck load biased head angle estimation across all head roll orientations, resulting in systematic shifts in reach errors; 2) increased neck muscle tone led to increased reach variability due to signal-dependent noise; and 3) both head roll and neck load created larger angular errors in reaches to visual targets away from the body compared with reaches toward the body. These results show that noise in muscle spindles and stochasticity in general have a tangible effect on RFTs underlying reach planning. Since RFTs are omnipresent in the brain, our results could have implications for processes as diverse as motor control, decision making, posture/balance control, and perception. NEW & NOTEWORTHY We show that increasing neck muscle tone systematically biases reach movements. A novel three-dimensional multisensory integration across reference frames model captures the data well and provides evidence that the brain must have online knowledge of full-body geometry together with the associated variability to plan reach movements accurately.

Deformation-induced transparency resolves color scission

Dynamic image deformation produces the perception of a transparent material that appears to deform the background image by light refraction. Since past studies on this phenomenon have mainly used subjective judgment about the presence of a transparent layer, it remains unsolved whether this is a real perceptual transparency effect in the sense that it forms surface representations, as do conventional transparency effects. Visual computation for color and luminance transparency, induced mainly by surface-contour information, can be decomposed into two components: surface formation to determine foreground and background layers, and scission to assign color and luminance to each layer. Here we show that deformation-induced perceptual transparency aids surface formation by color transparency and consequently resolves color scission. We asked observers to report the color of the front layer in a spatial region with a neutral physical color. The layer color could be seen as either reddish or greenish depending on the spatial context producing the color transparency, which was, however, ambiguous about the order of layers. We found that adding to the display a deformation-induced transparency that could specify the front layer significantly biased color scission in the predicted way if and only if the deformation-induced transparency was spatially coincident with the interpretation of color transparency. The results indicate that deformation-induced transparency is indeed a novel type of perceptual transparency that plays a role in surface formation in cooperation with color transparency.

Suboptimality in perceptual decision making

Human perceptual decisions are often described as optimal. Critics of this view have argued that claims of optimality are overly flexible and lack explanatory power. Meanwhile, advocates for optimality have countered that such criticisms single out a few selected papers. To elucidate the issue of optimality in perceptual decision making, we review the extensive literature on suboptimal performance in perceptual tasks. We discuss eight different classes of suboptimal perceptual decisions, including improper placement, maintenance, and adjustment of perceptual criteria; inadequate tradeoff between speed and accuracy; inappropriate confidence ratings; misweightings in cue combination; and findings related to various perceptual illusions and biases. In addition, we discuss conceptual shortcomings of a focus on optimality, such as definitional difficulties and the limited value of optimality claims in and of themselves. We therefore advocate that the field drop its emphasis on whether observed behavior is optimal and instead concentrate on building and testing detailed observer models that explain behavior across a wide range of tasks. To facilitate this transition, we compile the proposed hypotheses regarding the origins of suboptimal perceptual decisions reviewed here. We argue that verifying, rejecting, and expanding these explanations for suboptimal behavior - rather than assessing optimality per se - should be among the major goals of the science of perceptual decision making.

Conscious Vision Proceeds from Global to Local Content in Goal-Directed Tasks and Spontaneous Vision

The reverse hierarchy theory (Hochstein and Ahissar, 2002) makes strong, but so far untested, predictions on conscious vision. In this theory, local details encoded in lower-order visual areas are unconsciously processed before being automatically and rapidly combined into global information in higher-order visual areas, where conscious percepts emerge. Contingent on current goals, local details can afterward be consciously retrieved. This model therefore predicts that (1) global information is perceived faster than local details, (2) global information is computed regardless of task demands during early visual processing, and (3) spontaneous vision is dominated by global percepts. We designed novel textured stimuli that are, as opposed to the classic Navon's letters, truly hierarchical (i.e., where global information is solely defined by local information but where local and global orientations can still be manipulated separately). In line with the predictions, observers were systematically faster reporting global than local properties of those stimuli. Second, global information could be decoded from magneto-encephalographic data during early visual processing regardless of task demands. Last, spontaneous subjective reports were dominated by global information and the frequency and speed of spontaneous global perception correlated with the accuracy and speed in the global task. No such correlation was observed for local information. We therefore show that information at different levels of the visual hierarchy is not equally likely to become conscious; rather, conscious percepts emerge preferentially at a global level. We further show that spontaneous reports can be reliable and are tightly linked to objective performance at the global level.

SIGNIFICANCE STATEMENT

Is information encoded at different levels of the visual system (local details in low-level areas vs global shapes in high-level areas) equally likely to become conscious? We designed new hierarchical stimuli and provide the first empirical evidence based on behavioral and MEG data that global information encoded at high levels of the visual hierarchy dominates perception. This result held both in the presence and in the absence of task demands. The preferential emergence of percepts at high levels can account for two properties of conscious vision, namely, the dominance of global percepts and the feeling of visual richness reported independently of the perception of local details.

Trust in haptic assistance: weighting visual and haptic cues based on error history

To effectively interpret and interact with the world, humans weight redundant estimates from different sensory cues to form one coherent, integrated estimate. Recent advancements in physical assistance systems, where guiding forces are computed by an intelligent agent, enable the presentation of augmented cues. It is unknown, however, if cue weighting can be extended to augmented cues. Previous research has shown that cue weighting is determined by the reliability (inversely related to uncertainty) of cues within a trial, yet augmented cues may also be affected by errors that vary over trials. In this study, we investigate whether people can learn to appropriately weight a haptic cue from an intelligent assistance system based on its error history. Subjects held a haptic device and reached to a hidden target using a visual (Gaussian distributed dots) and haptic (force channel) cue. The error of the augmented haptic cue varied from trial to trial based on a Gaussian distribution. Subjects learned to estimate the target location by weighting the visual and augmented haptic cues based on their perceptual uncertainty and experienced errors. With both cues available, subjects were able to find the target with an improved or equal performance compared to what was possible with one cue alone. Our results show that the brain can learn to reweight augmented cues from intelligent agents, akin to previous observations of the reweighting of naturally occurring cues. In addition, these results suggest that the weighting of a cue is not only affected by its within-trial reliability but also the history of errors.

Optimal nonlinear cue integration for sound localization

Integration of multiple sensory cues can improve performance in detection and estimation tasks. There is an open theoretical question of the conditions under which linear or nonlinear cue combination is Bayes-optimal. We demonstrate that a neural population decoded by a population vector requires nonlinear cue combination to approximate Bayesian inference. Specifically, if cues are conditionally independent, multiplicative cue combination is optimal for the population vector. The model was tested on neural and behavioral responses in the barn owl's sound localization system where space-specific neurons owe their selectivity to multiplicative tuning to sound localization cues interaural phase (IPD) and level (ILD) differences. We found that IPD and ILD cues are approximately conditionally independent. As a result, the multiplicative combination selectivity to IPD and ILD of midbrain space-specific neurons permits a population vector to perform Bayesian cue combination. We further show that this model describes the owl's localization behavior in azimuth and elevation. This work provides theoretical justification and experimental evidence supporting the optimality of nonlinear cue combination.

Similarities in Autistic and Neurotypical Visual–Haptic Perception When Making Judgements About Conflicting Sensory Stimuli

A number of studies have shown that multisensory performance is well predicted by a statistically optimal maximum likelihood estimation (MLE) model. Under this model unisensory estimates are combined additively and weighted according to relative reliability. Recent theories have proposed that atypical sensation and perception commonly reported in autism spectrum condition (ASC) may result from differences in the use of reliability information. Furthermore, experimental studies have indicated that multisensory processing is less effective in those with the condition in comparison to neurotypical (NT) controls. In the present study, adults with ASC () and a matched NT group () completed a visual–haptic size judgement task (cf. Gori et al., 2008) in which participants compared the height of wooden blocks using either vision or haptics, and in a dual modality condition in which visual–haptic stimuli were presented in size conflict. Participants with ASC tended to produce more reliable estimates than the NT group. However, dual modality performance was not well predicted by the MLE model for either group. Performance was subsequently compared to alternative models in which the participant either switched between modalities trial to trial (rather than integrating) and a model of non-optimal integration. Performance of both groups was statistically comparable to the cue-switching model. These findings suggest that adults with ASC adopted a similar strategy to NTs when processing conflicting visual–haptic information. Findings are discussed in relation to multisensory perception in ASC and methodological considerations associated with multisensory conflict paradigms.

Dissecting neural circuits for multisensory integration and crossmodal processing

We rely on rich and complex sensory information to perceive and understand our environment. Our multisensory experience of the world depends on the brain's remarkable ability to combine signals across sensory systems. Behavioural, neurophysiological and neuroimaging experiments have established principles of multisensory integration and candidate neural mechanisms. Here we review how targeted manipulation of neural activity using invasive and non-invasive neuromodulation techniques have advanced our understanding of multisensory processing. Neuromodulation studies have provided detailed characterizations of brain networks causally involved in multisensory integration. Despite substantial progress, important questions regarding multisensory networks remain unanswered. Critically, experimental approaches will need to be combined with theory in order to understand how distributed activity across multisensory networks collectively supports perception.

A systematic comparison between visual cues for boundary detection

The detection of object boundaries is a critical first step for many visual processing tasks. Multiple cues (we consider luminance, color, motion and binocular disparity) available in the early visual system may signal object boundaries but little is known about their relative diagnosticity and how to optimally combine them for boundary detection. This study thus aims at understanding how early visual processes inform boundary detection in natural scenes. We collected color binocular video sequences of natural scenes to construct a video database. Each scene was annotated with two full sets of ground-truth contours (one set limited to object boundaries and another set which included all edges). We implemented an integrated computational model of early vision that spans all considered cues, and then assessed their diagnosticity by training machine learning classifiers on individual channels. Color and luminance were found to be most diagnostic while stereo and motion were least. Combining all cues yielded a significant improvement in accuracy beyond that of any cue in isolation. Furthermore, the accuracy of individual cues was found to be a poor predictor of their unique contribution for the combination. This result suggested a complex interaction between cues, which we further quantified using regularization techniques. Our systematic assessment of the accuracy of early vision models for boundary detection together with the resulting annotated video dataset should provide a useful benchmark towards the development of higher-level models of visual processing.

Motor planning poststroke: impairment in vector-coded reach plans

Healthy individuals appear to use both vector‐coded reach plans that encode movements in terms of their desired direction and extent, and target‐coded reach plans that encode the desired endpoint position of the effector. We examined whether these vector and target reach‐planning codes are differentially affected after stroke. Participants with stroke and healthy controls made blocks of reaches that were grouped by target location (providing target‐specific practice) and by movement vector (providing vector‐specific practice). Reach accuracy was impaired in the more affected arm after stroke, but not distinguishable for target‐ versus vector‐grouped reaches. Reach velocity and acceleration were not only impaired in both the less and more affected arms poststroke, but also not distinguishable for target‐ versus vector‐grouped reaches. As previously reported in controls, target‐grouped reaches yielded isotropic (circular) error distributions and vector‐grouped reaches yielded error distributions elongated in the direction of the reach. In stroke, the pattern of variability was similar. However, the more affected arm showed less elongated error ellipses for vector‐grouped reaches compared to the less affected arm, particularly in individuals with right‐hemispheric stroke. The results suggest greater impairment to the vector‐coded movement‐planning system after stroke, and have implications for the development of personalized approaches to poststroke rehabilitation: Motor learning may be enhanced by practice that uses the preserved code or, conversely, by retraining the more impaired code to restore function.

Near-optimal integration of orientation information across saccades

We perceive a stable environment despite the fact that visual information is essentially acquired in a sequence of snapshots separated by saccadic eye movements. The resolution of these snapshots varies-high in the fovea and lower in the periphery-and thus the formation of a stable percept presumably relies on the fusion of information acquired at different resolutions. To test if, and to what extent, foveal and peripheral information are integrated, we examined human orientation-discrimination performance across saccadic eye movements. We found that humans perform best when an oriented target is visible both before (peripherally) and after a saccade (foveally), suggesting that humans integrate the two views. Integration relied on eye movements, as we found no evidence of integration when the target was artificially moved during stationary viewing. Perturbation analysis revealed that humans combine the two views using a weighted sum, with weights assigned based on the relative precision of foveal and peripheral representations, as predicted by ideal observer models. However, our subjects displayed a systematic overweighting of the fovea, relative to the ideal observer, indicating that human integration across saccades is slightly suboptimal.

Multifeature-based surround inhibition improves contour detection in natural images

To effectively perform visual tasks like detecting contours, the visual system normally needs to integrate multiple visual features. Sufficient physiological studies have revealed that for a large number of neurons in the primary visual cortex (V1) of monkeys and cats, neuronal responses elicited by the stimuli placed within the classical receptive field (CRF) are substantially modulated, normally inhibited, when difference exists between the CRF and its surround, namely, non-CRF, for various local features. The exquisite sensitivity of V1 neurons to the center-surround stimulus configuration is thought to serve important perceptual functions, including contour detection. In this paper, we propose a biologically motivated model to improve the performance of perceptually salient contour detection. The main contribution is the multifeature-based center-surround framework, in which the surround inhibition weights of individual features, including orientation, luminance, and luminance contrast, are combined according to a scale-guided strategy, and the combined weights are then used to modulate the final surround inhibition of the neurons. The performance was compared with that of single-cue-based models and other existing methods (especially other biologically motivated ones). The results show that combining multiple cues can substantially improve the performance of contour detection compared with the models using single cue. In general, luminance and luminance contrast contribute much more than orientation to the specific task of contour extraction, at least in gray-scale natural images.

Correlation between human and model observer performance for discrimination task in CT

Although physical metrics can objectively characterize computed tomography (CT) image quality, quantitative approaches to predict human observer performance are more accurate and clinically relevant. This study compared a modified channelized Hotelling model observer (CHO) with human observers in a shape discrimination task. Eight lesion-mimicking rods (two contrasts, two sizes and two shapes) were inserted into a 35 × 26 cm(2) torso-shaped water phantom and scanned 100 times on a 128-slice CT scanner at five dose levels. CT images were reconstructed using filtered backprojection (FBP) and iterative reconstruction (IR) techniques. Two-alternative forced choice studies were constructed with hexagonal and circular rod images put side-by-side in a randomized order. An edge mask was introduced to CHO to reflect the human observers' emphasis on lesion boundaries in discriminating shape. For small size lesions, the performance of three human observers and the modified CHO was highly correlated across lesion contrasts, CT doses and reconstruction algorithms; while for large size lesions, a ceiling effect was observed for both human and model observers' performance at high doses. Our result suggests the potential of CHO to predict human observer performance for both FBP and IR. For this shape discrimination task with uniform background, IR significantly improved human and model observer performance compared to FBP, with the amount of improvement depending on lesion size, contrast and dose.

Is one enough? The case for non-additive influences of visual features on crossmodal Stroop interference

When different perceptual signals arising from the same physical entity are integrated, they form a more reliable sensory estimate. When such repetitive sensory signals are pitted against other competing stimuli, such as in a Stroop Task, this redundancy may lead to stronger processing that biases behavior toward reporting the redundant stimuli. This bias would therefore, be expected to evoke greater incongruency effects than if these stimuli did not contain redundant sensory features. In the present paper we report that this is not the case for a set of three crossmodal, auditory-visual Stroop tasks. In these tasks participants attended to, and reported, either the visual or the auditory stimulus (in separate blocks) while ignoring the other, unattended modality. The visual component of these stimuli could be purely semantic (words), purely perceptual (colors), or the combination of both. Based on previous work showing enhanced crossmodal integration and visual search gains for redundantly coded stimuli, we had expected that relative to the single features, redundant visual features would have induced both greater visual distracter incongruency effects for attended auditory targets, and been less influenced by auditory distracters for attended visual targets. Overall, reaction times were faster for visual targets and were dominated by behavioral facilitation for the cross-modal interactions (relative to interference), but showed surprisingly little influence of visual feature redundancy. Post-hoc analyses revealed modest and trending evidence for possible increases in behavioral interference for redundant visual distracters on auditory targets, however, these effects were substantially smaller than anticipated and were not accompanied by a redundancy effect for behavioral facilitation or for attended visual targets.

Depth perception from dynamic occlusion in motion parallax: roles of expansion-compression versus accretion-deletion

Motion parallax, or differential retinal image motion from observer movement, provides important information for depth perception. We previously measured the contribution of shear motion parallax to depth, which is only composed of relative motion information. Here, we examine the roles of relative motion and accretion-deletion information in dynamic occlusion motion parallax. Observers performed two-alternative forced choice depth-ordering tasks in response to low spatial frequency patterns of horizontal random dot motion that were synchronized to the observer's head movements. We examined conditions that isolated or combined expansion-compression and accretion-deletion across a range of simulated relative depths. At small depths, expansion-compression provided reliable depth perception while accretion-deletion had a minor contribution: When the two were in conflict, the perceived depth was dominated by expansion-compression. At larger depths in the cue-conflict experiment, accretion-deletion determined the depth-ordering performance. Accretion-deletion in isolation did not yield any percept of depth even though, in theory, it provided sufficient information for depth ordering. Thus, accretion-deletion can substantially enhance depth perception at larger depths but only in the presence of relative motion. The results indicate that expansion-compression contributes to depth from motion parallax across a broad range of depths whereas accretion-deletion contributes primarily at larger depths.

Depth and luminance edges attract

The spatial resolution of disparity perception is poor compared to luminance perception, yet we do not notice that depth edges are more blurry than luminance edges. Is this because the two cues are combined by the visual system? Subjects judged the locations of depth-defined or luminance-defined edges, which were separated by up to 5.6 min of arc. The perceived edge location was a function of the depth-defined edge and the luminance-defined edge, with the luminance edge tending to play a larger role. Our data are compatible with but not completely explained by an optimal cue-combination model that gives more reliable cues a heavier weight. Both edge cues (depth and luminance) contribute to the final percept, with an adaptive weighting depending on the task and the acuity with which each cue is perceived.

Seeing and hearing a word: combining eye and ear is more efficient than combining the parts of a word

To understand why human sensitivity for complex objects is so low, we study how word identification combines eye and ear or parts of a word (features, letters, syllables). Our observers identify printed and spoken words presented concurrently or separately. When researchers measure threshold (energy of the faintest visible or audible signal) they may report either sensitivity (one over the human threshold) or efficiency (ratio of the best possible threshold to the human threshold). When the best possible algorithm identifies an object (like a word) in noise, its threshold is independent of how many parts the object has. But, with human observers, efficiency depends on the task. In some tasks, human observers combine parts efficiently, needing hardly more energy to identify an object with more parts. In other tasks, they combine inefficiently, needing energy nearly proportional to the number of parts, over a 60∶1 range. Whether presented to eye or ear, efficiency for detecting a short sinusoid (tone or grating) with few features is a substantial 20%, while efficiency for identifying a word with many features is merely 1%. Why? We show that the low human sensitivity for words is a cost of combining their many parts. We report a dichotomy between inefficient combining of adjacent features and efficient combining across senses. Joining our results with a survey of the cue-combination literature reveals that cues combine efficiently only if they are perceived as aspects of the same object. Observers give different names to adjacent letters in a word, and combine them inefficiently. Observers give the same name to a word’s image and sound, and combine them efficiently. The brain’s machinery optimally combines only cues that are perceived as originating from the same object. Presumably such cues each find their own way through the brain to arrive at the same object representation.

Dynamic weighting of multisensory stimuli shapes decision-making in rats and humans

Stimuli that animals encounter in the natural world are frequently time-varying and activate multiple sensory systems together. Such stimuli pose a major challenge for the brain: Successful multisensory integration requires subjects to estimate the reliability of each modality and use these estimates to weight each signal appropriately. Here, we examined whether humans and rats can estimate the reliability of time-varying multisensory stimuli when stimulus reliability changes unpredictably from trial to trial. Using an existing multisensory decision task that features time-varying audiovisual stimuli, we independently manipulated the signal-to-noise ratios of each modality and measured subjects' decisions on single- and multi-sensory trials. We report three main findings: (a) Sensory reliability influences how subjects weight multisensory evidence even for time-varying, stochastic stimuli. (b) The ability to exploit sensory reliability extends beyond human and nonhuman primates: Rodents and humans both weight incoming sensory information in a reliability-dependent manner. (c) Regardless of sensory reliability, most subjects are disinclined to make "snap judgments" and instead base decisions on evidence presented over the majority of the trial duration. Rare departures from this trend highlight the importance of using time-varying stimuli that permit this analysis. Taken together, these results suggest that the brain's ability to use stimulus reliability to guide decision-making likely relies on computations that are conserved across species and operate over a wide range of stimulus conditions.

Visual size perception and haptic calibration during development

It is still unclear how the visual system perceives accurately the size of objects at different distances. One suggestion, dating back to Berkeley's famous essay, is that vision is calibrated by touch. If so, we may expect different mechanisms involved for near, reachable distances and far, unreachable distances. To study how the haptic system calibrates vision we measured size constancy in children (from 6 to 16 years of age) and adults, at various distances. At all ages, accuracy of the visual size perception changes with distance, and is almost veridical inside the haptic workspace, in agreement with the idea that the haptic system acts to calibrate visual size perception. Outside this space, systematic errors occurred, which varied with age. Adults tended to overestimate visual size of distant objects (over-compensation for distance), while children younger than 14 underestimated their size (under-compensation). At 16 years of age there seemed to be a transition point, with veridical perception of distant objects. When young subjects were allowed to touch the object inside the haptic workspace, the visual biases disappeared, while older subjects showed multisensory integration. All results are consistent with the idea that the haptic system can be used to calibrate visual size perception during development, more effectively within than outside the haptic workspace, and that the calibration mechanisms are different in children than in adults.

Organizing probabilistic models of perception

Probability has played a central role in models of perception for more than a century, but a look at probabilistic concepts in the literature raises many questions. Is being Bayesian the same as being optimal? Are recent Bayesian models fundamentally different from classic signal detection theory models? Do findings of near-optimal inference provide evidence that neurons compute with probability distributions? This review aims to disentangle these concepts and to classify empirical evidence accordingly.

Motor learning reveals the existence of multiple codes for movement planning

Coordinate systems for movement planning are comprised of an anchor point (e.g., retinocentric coordinates) and a representation (encoding) of the desired movement. One of two representations is often assumed: a final-position code describing desired limb endpoint position and a vector code describing movement direction and extent. The existence of movement-planning systems using both representations is controversial. In our experiments, participants completed reaches grouped by target location (providing practice for a final-position code) and the same reaches grouped by movement vector (providing vector-code practice). Target-grouped reaches resulted in the isotropic (circular) distribution of errors predicted for position-coded reaches. The identical reaches grouped by vector resulted in error ellipses aligned with the reach direction, as predicted for vector-coded reaches. Manipulating only recent movement history to provide better learning for one or the other movement code, we provide definitive evidence that both movement representations are used in the identical task.

Differential effects of exogenous and endogenous attention on second-order texture contrast sensitivity

The visual system can use a rich variety of contours to segment visual scenes into distinct perceptually coherent regions. However, successfully segmenting an image is a computationally expensive process. Previously we have shown that exogenous attention--the more automatic, stimulus-driven component of spatial attention--helps extract contours by enhancing contrast sensitivity for second-order, texture-defined patterns at the attended location, while reducing sensitivity at unattended locations, relative to a neutral condition. Interestingly, the effects of exogenous attention depended on the second-order spatial frequency of the stimulus. At parafoveal locations, attention enhanced second-order contrast sensitivity to relatively high, but not to low second-order spatial frequencies. In the present study we investigated whether endogenous attention-the more voluntary, conceptually-driven component of spatial attention--affects second-order contrast sensitivity, and if so, whether its effects are similar to those of exogenous attention. To that end, we compared the effects of exogenous and endogenous attention on the sensitivity to second-order, orientation-defined, texture patterns of either high or low second-order spatial frequencies. The results show that, like exogenous attention, endogenous attention enhances second-order contrast sensitivity at the attended location and reduces it at unattended locations. However, whereas the effects of exogenous attention are a function of the second-order spatial frequency content, endogenous attention affected second-order contrast sensitivity independent of the second-order spatial frequency content. This finding supports the notion that both exogenous and endogenous attention can affect second-order contrast sensitivity, but that endogenous attention is more flexible, benefitting performance under different conditions.

Knowing how much you don't know: a neural organization of uncertainty estimates

How we estimate uncertainty is important in decision neuroscience and has wide-ranging implications in basic and clinical neuroscience, from computational models of optimality to ideas on psychopathological disorders including anxiety, depression and schizophrenia. Empirical research in neuroscience, which has been based on divergent theoretical assumptions, has focused on the fundamental question of how uncertainty is encoded in the brain and how it influences behaviour. Here, we integrate several theoretical concepts about uncertainty into a decision-making framework. We conclude that the currently available evidence indicates that distinct neural encoding (including summary statistic-type representations) of uncertainty occurs in distinct neural systems.

Attention in a bayesian framework

The behavioral phenomena of sensory attention are thought to reflect the allocation of a limited processing resource, but there is little consensus on the nature of the resource or why it should be limited. Here we argue that a fundamental bottleneck emerges naturally within Bayesian models of perception, and use this observation to frame a new computational account of the need for, and action of, attention - unifying diverse attentional phenomena in a way that goes beyond previous inferential, probabilistic and Bayesian models. Attentional effects are most evident in cluttered environments, and include both selective phenomena, where attention is invoked by cues that point to particular stimuli, and integrative phenomena, where attention is invoked dynamically by endogenous processing. However, most previous Bayesian accounts of attention have focused on describing relatively simple experimental settings, where cues shape expectations about a small number of upcoming stimuli and thus convey "prior" information about clearly defined objects. While operationally consistent with the experiments it seeks to describe, this view of attention as prior seems to miss many essential elements of both its selective and integrative roles, and thus cannot be easily extended to complex environments. We suggest that the resource bottleneck stems from the computational intractability of exact perceptual inference in complex settings, and that attention reflects an evolved mechanism for approximate inference which can be shaped to refine the local accuracy of perception. We show that this approach extends the simple picture of attention as prior, so as to provide a unified and computationally driven account of both selective and integrative attentional phenomena.

Impaired visual size-discrimination in children with movement disorders

Multisensory integration of spatial information occurs late in childhood, at around eight years (Gori, Del Viva, Sandini, & Burr, 2008). For younger children, the haptic system dominates size discrimination and vision dominates orientation discrimination: the dominance may reflect sensory calibration, and could have direct consequences on children born with specific sensory disabilities. Here we measure thresholds for visual discrimination of orientation and size in children with movement disorders of upper limbs. Visual orientation discrimination was very similar to the age-matched typical children, but visual size discrimination thresholds were far worse, in all eight individuals with early-onset movement disorder. This surprising and counterintuitive result is readily explained by the cross-sensory calibration hypothesis: when the haptic sense is unavailable for manipulation, it cannot be readily used to estimate size, and hence to calibrate the visual experience of size: visual discrimination is subsequently impaired. This complements a previous study showing that non-sighted children have reduced acuity for haptic orientation, but not haptic size, discriminations (Gori, Sandini, Martinoli, & Burr, 2010). Together these studies show that when either vision or haptic manipulation is impaired, the impairment also impacts on complementary sensory systems that are calibrated by that one.

Multisensory decision-making in rats and humans

We report a novel multisensory decision task designed to encourage subjects to combine information across both time and sensory modalities. We presented subjects, humans and rats, with multisensory event streams, consisting of a series of brief auditory and/or visual events. Subjects made judgments about whether the event rate of these streams was high or low. We have three main findings. First, we report that subjects can combine multisensory information over time to improve judgments about whether a fluctuating rate is high or low. Importantly, the improvement we observed was frequently close to, or better than, the statistically optimal prediction. Second, we found that subjects showed a clear multisensory enhancement both when the inputs in each modality were redundant and when they provided independent evidence about the rate. This latter finding suggests a model where event rates are estimated separately for each modality and fused at a later stage. Finally, because a similar multisensory enhancement was observed in both humans and rats, we conclude that the ability to optimally exploit sequentially presented multisensory information is not restricted to a particular species.

The perception of a face is no more than the sum of its parts

When you see a person's face, how do you go about combining his or her facial features to make a decision about who that person is? Most current theories of face perception assert that the ability to recognize a human face is not simply the result of an independent analysis of individual features, but instead involves a holistic coding of the relationships among features. This coding is thought to enhance people's ability to recognize a face beyond what would be expected if each feature were shown in isolation. In the study reported here, we explicitly tested this idea by comparing human performance on facial-feature integration with that of an optimal Bayesian integrator. Contrary to the predictions of most current notions of face perception, our findings showed that human observers integrate facial features in a manner that is no better than would be predicted by their ability to use each individual feature when shown in isolation. That is, a face is perceived no better than the sum of its individual parts.

Combination of texture and color cues in visual segmentation

The visual system can use various cues to segment the visual scene into figure and background. We studied how human observers combine two of these cues, texture and color, in visual segmentation. In our task, the observers identified the orientation of an edge that was defined by a texture difference, a color difference, or both (cue combination). In a fourth condition, both texture and color information were available, but the texture and color edges were not spatially aligned (cue conflict). Performance markedly improved when the edges were defined by two cues, compared to the single-cue conditions. Observers only benefited from the two cues, however, when they were spatially aligned. A simple signal-detection model that incorporates interactions between texture and color processing accounts for the performance in all conditions. In a second experiment, we studied whether the observers are able to ignore a task-irrelevant cue in the segmentation task or whether it interferes with performance. Observers identified the orientation of an edge defined by one cue and were instructed to ignore the other cue. Three types of trial were intermixed: neutral trials, in which the second cue was absent; congruent trials, in which the second cue signaled the same edge as the target cue; and conflict trials, in which the second cue signaled an edge orthogonal to the target cue. Performance improved when the second cue was congruent with the target cue. Performance was impaired when the second cue was in conflict with the target cue, indicating that observers could not discount the second cue. We conclude that texture and color are not processed independently in visual segmentation.

Weighted integration of visual position information

The ability to localize visual objects is a fundamental component of human behavior and requires the integration of position information from object components. The retinal eccentricity of a stimulus and the locus of spatial attention can affect object localization, but it is unclear whether these factors alter the global localization of the object, the localization of object components, or both. We used psychophysical methods in humans to quantify behavioral responses in a centroid estimation task. Subjects located the centroid of briefly presented random dot patterns (RDPs). A peripheral cue was used to bias attention toward one side of the display. We found that although subjects were able to localize centroid positions reliably, they typically had a bias toward the fovea and a shift toward the locus of attention. We compared quantitative models that explain these effects either as biased global localization of the RDPs or as anisotropic integration of weighted dot component positions. A model that allowed retinal eccentricity and spatial attention to alter the weights assigned to individual dot positions best explained subjects' performance. These results show that global position perception depends on both the retinal eccentricity of stimulus components and their positions relative to the current locus of attention.

Multisensory calibration is independent of cue reliability

Multisensory calibration is fundamental for proficient interaction within a changing environment. Initial studies suggested a visual-dominant mechanism. More recently, a cue-reliability-based model, similar to optimal cue integration, has been proposed. However, a more general, reliability-independent model of fixed-ratio adaptation (of which visual dominance is a subcase) has never been tested. Here, we studied behavior of both humans and monkeys performing a heading-discrimination task. Subjects were presented with either visual (optic-flow), vestibular (motion-platform), or combined (visual-vestibular) stimuli and required to report whether self-motion was to the right/left of straight ahead. A systematic heading discrepancy was introduced between the visual and vestibular cues, without external feedback. Cue calibration was measured by the resulting sensory adaptation. Both visual and vestibular cues significantly adapted in the direction required to reduce cue conflict. However, unlike multisensory cue integration, cue calibration was not reliability based. Rather, a model of fixed-ratio adaptation best described the data, whereby vestibular adaptation was greater than visual adaptation, regardless of relative cue reliability. The average ratio of vestibular to visual adaptation was 1.75 and 2.30 for the human and monkey data, respectively. Furthermore, only through modeling fixed-ratio adaptation (using the ratio extracted from the data) were we able to account for reliability-based cue integration during the adaptation process. The finding that cue calibration does not depend on cue reliability is consistent with the notion that it follows an underlying estimate of cue accuracy. Cue accuracy is generally independent of cue reliability, and its estimate may change with a much slower time constant. Thus, greater vestibular versus visual (fixed-ratio) adaptation suggests lower vestibular versus visual cue accuracy.