Integration of texture and disparity cues to surface slant in dorsal visual cortex

Different types of uncertainty in multisensory perceptual decision making

Efficient decision-making requires accounting for sources of uncertainty (noise, or variability). Many studies have shown how the nervous system is able to account for perceptual uncertainty (noise, variability) that arises from limitations in its own abilities to encode perceptual stimuli. However, many other sources of uncertainty exist, reflecting for example variability in the behaviour of other agents or physical processes. Here we review previous studies on decision making under uncertainty as a function of the different types of uncertainty that the nervous system encounters, showing that noise that is intrinsic to the perceptual system can often be accounted for near-optimally (i.e. not statistically different from optimally), whereas accounting for other types of uncertainty can be much more challenging. As an example, we present a study in which participants made decisions about multisensory stimuli with both intrinsic (perceptual) and extrinsic (environmental) uncertainty and show that the nervous system accounts for these differently when making decisions: they account for internal uncertainty but under-account for external. Human perceptual systems may be well equipped to account for intrinsic (perceptual) uncertainty because, in principle, they have access to this. Accounting for external uncertainty is more challenging because this uncertainty must be learned. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Task feedback suggests a post-perceptual component to serial dependence

Decisions across a range of perceptual tasks are biased toward past stimuli. Such serial dependence is thought to be an adaptive low-level mechanism that promotes perceptual stability across time. However, recent studies suggest post-perceptual mechanisms may also contribute to serially biased responses, calling into question a single locus of serial dependence and the nature of integration of past and present sensory inputs. We measured serial dependence in the context of a three-dimensional (3D) motion perception task where uncertainty in the sensory information varied substantially from trial to trial. We found that serial dependence varied with stimulus properties that impact sensory uncertainty on the current trial. Reduced stimulus contrast was associated with an increased bias toward the stimulus direction of the previous trial. Critically, performance feedback, which reduced sensory uncertainty, abolished serial dependence. These results provide clear evidence for a post-perceptual locus of serial dependence in 3D motion perception and support the role of serial dependence as a response strategy in the face of substantial sensory uncertainty.

Learning to see in depth

Stereopsis provides us with a vivid impression of the depth and distance of objects in our 3- dimensional world. Stereopsis is important for a number of everyday visual tasks, including (but not limited to) reaching and grasping, fine visuo-motor control, and navigating in our world. This review briefly discusses the neural substrate for normal binocular vision and stereopsis and its development in primates; outlines some of the issues and limitations of stereopsis tests and examines some of the factors that limit the typical development of stereopsis and the causes and consequences of stereo-deficiency and stereo-blindness. Finally, we review several approaches to improving or recovering stereopsis in both neurotypical individuals and those with stereo-deficiency and stereo-blindness and outline some emerging strategies for improving stereopsis.

Abnormal effective connectivity in visual cortices underlies stereopsis defects in amblyopia

•

Abnormal effective connectivity inherent stereopsis defects in amblyopia was studied.

•

A weakened connection from V2v to LO2 relates to stereopsis defects in amblyopia.

•

Higher-order visual cortices may serve as key nodes to the stereopsis defects.

•

An independent longitudinal dataset was used to validate the obtained results.

The neural basis underlying stereopsis defects in patients with amblyopia remains unclear, which hinders the development of clinical therapy. This study aimed to investigate visual network abnormalities in patients with amblyopia and their associations with stereopsis function. Spectral dynamic causal modeling methods were employed for resting-state functional magnetic resonance imaging data to investigate the effective connectivity (EC) among 14 predefined regions of interest in the dorsal and ventral visual pathways. We adopted two independent datasets, including a cross-sectional and a longitudinal dataset. In the cross-sectional dataset, we compared group differences in EC between 31 patients with amblyopia (mean age: 26.39 years old) and 31 healthy controls (mean age: 25.71 years old) and investigated the association between EC and stereoacuity. In addition, we explored EC changes after perceptual learning in a novel longitudinal dataset including 9 patients with amblyopia (mean age: 15.78 years old). We found consistent evidence from the two datasets indicating that the aberrant EC from V2v to LO2 is crucial for the stereoscopic deficits in the patients with amblyopia: it was weaker in the patients than in the controls, showed a positive linear relationship with the stereoscopic function, and increased after perceptual learning in the patients. In addition, higher-level dorsal (V3d, V3A, and V3B) and ventral areas (LO1 and LO2) were important nodes in the network of abnormal ECs associated with stereoscopic deficits in the patients with amblyopia. Our research provides insights into the neural mechanism underlying stereopsis deficits in patients with amblyopia and provides candidate targets for focused stimulus interventions to enhance the efficacy of clinical treatment for the improvement of stereopsis deficiency.

Experimentally disambiguating models of sensory cue integration

Sensory cue integration is one of the primary areas in which a normative mathematical framework has been used to define the “optimal” way in which to make decisions based upon ambiguous sensory information and compare these predictions to behavior. The conclusion from such studies is that sensory cues are integrated in a statistically optimal fashion. However, numerous alternative computational frameworks exist by which sensory cues could be integrated, many of which could be described as “optimal” based on different criteria. Existing studies rarely assess the evidence relative to different candidate models, resulting in an inability to conclude that sensory cues are integrated according to the experimenter's preferred framework. The aims of the present paper are to summarize and highlight the implicit assumptions rarely acknowledged in testing models of sensory cue integration, as well as to introduce an unbiased and principled method by which to determine, for a given experimental design, the probability with which a population of observers behaving in accordance with one model of sensory integration can be distinguished from the predictions of a set of alternative models.

Ultra-High-Field Neuroimaging Reveals Fine-Scale Processing for 3D Perception

Binocular disparity provides critical information about three-dimensional (3D) structures to support perception and action. In the past decade significant progress has been made in uncovering human brain areas engaged in the processing of binocular disparity signals. Yet, the fine-scale brain processing underlying 3D perception remains unknown. Here, we use ultra-high-field (7T) functional imaging at submillimeter resolution to examine fine-scale BOLD fMRI signals involved in 3D perception. In particular, we sought to interrogate the local circuitry involved in disparity processing by sampling fMRI responses at different positions relative to the cortical surface (i.e., across cortical depths corresponding to layers). We tested for representations related to 3D perception by presenting participants (male and female, N = 8) with stimuli that enable stable stereoscopic perception [i.e., correlated random dot stereograms (RDS)] versus those that do not (i.e., anticorrelated RDS). Using multivoxel pattern analysis (MVPA), we demonstrate cortical depth-specific representations in areas V3A and V7 as indicated by stronger pattern responses for correlated than for anticorrelated stimuli in upper rather than deeper layers. Examining informational connectivity, we find higher feedforward layer-to-layer connectivity for correlated than anticorrelated stimuli between V3A and V7. Further, we observe disparity-specific feedback from V3A to V1 and from V7 to V3A. Our findings provide evidence for the role of V3A as a key nexus for disparity processing, which is implicated in feedforward and feedback signals related to the perceptual estimation of 3D structures.SIGNIFICANCE STATEMENT Binocular vision plays a significant role in supporting our interactions with the surrounding environment. The fine-scale neural mechanisms that underlie the brain's skill in extracting 3D structures from binocular signals are poorly understood. Here, we capitalize on recent advances in ultra-high-field functional imaging to interrogate human brain circuits involved in 3D perception at submillimeter resolution. We provide evidence for the role of area V3A as a key nexus for disparity processing, which is implicated in feedforward and feedback signals related to the perceptual estimation of 3D structures from binocular signals. These fine-scale measurements help bridge the gap between animal neurophysiology and human fMRI studies investigating cross-scale circuits, from micro circuits to global brain networks for 3D perception.

Disparity in Context: Understanding how monocular image content interacts with disparity processing in human visual cortex

Horizontal disparities between the two eyes' retinal images are the primary cue for depth. Commonly used random ot tereograms (RDS) intentionally camouflage the disparity cue, breaking the correlations between monocular image structure and the depth map that are present in natural images. Because of the nonlinear nature of visual processing, it is unlikely that simple computational rules derived from RDS will be sufficient to explain binocular vision in natural environments. In order to understand the interplay between natural scene structure and disparity encoding, we used a depth-image-based-rendering technique and a library of natural 3D stereo pairs to synthesize two novel stereogram types in which monocular scene content was manipulated independent of scene depth information. The half-images of the novel stereograms comprised either random-dots or scrambled natural scenes, each with the same depth maps as the corresponding natural scene stereograms. Using these stereograms in a simultaneous Event-Related Potential and behavioral discrimination task, we identified multiple disparity-contingent encoding stages between 100 ~ 500 msec. The first disparity sensitive evoked potential was observed at ~100 msec after an earlier evoked potential (between ~50-100 msec) that was sensitive to the structure of the monocular half-images but blind to disparity. Starting at ~150 msec, disparity responses were stereogram-specific and predictive of perceptual depth. Complex features associated with natural scene content are thus at least partially coded prior to disparity information, but these features and possibly others associated with natural scene content interact with disparity information only after an intermediate, 2D scene-independent disparity processing stage.

Unique Neural Activity Patterns Among Lower Order Cortices and Shared Patterns Among Higher Order Cortices During Processing of Similar Shapes With Different Stimulus Types

We investigated the neural mechanism of the processing of three-dimensional (3D) shapes defined by disparity and perspective. We measured blood oxygenation level-dependent signals as participants viewed and classified 3D images of convex-concave shapes. According to the cue (disparity or perspective) and element type (random dots or black and white dotted lines), three types of stimuli were used: random dot stereogram, black and white dotted lines with perspective, and black and white dotted lines with binocular disparity. The blood oxygenation level-dependent images were then classified by multivoxel pattern analysis. To identify areas selective to shape, we assessed convex-concave classification accuracy with classifiers trained and tested using signals evoked by the same stimulus type (same cue and element type). To identify cortical regions with similar neural activity patterns regardless of stimulus type, we assessed the convex-concave classification accuracy of transfer classification in which classifiers were trained and tested using different stimulus types (different cues or element types). Classification accuracy using the same stimulus type was high in the early visual areas and subregions of the intraparietal sulcus (IPS), whereas transfer classification accuracy was high in the dorsal subregions of the IPS. These results indicate that the early visual areas process the specific features of stimuli, whereas the IPS regions perform more generalized processing of 3D shapes, independent of a specific stimulus type.

Combining cues to judge distance and direction in an immersive virtual reality environment

When we move, the visual direction of objects in the environment can change substantially. Compared with our understanding of depth perception, the problem the visual system faces in computing this change is relatively poorly understood. Here, we tested the extent to which participants' judgments of visual direction could be predicted by standard cue combination rules. Participants were tested in virtual reality using a head-mounted display. In a simulated room, they judged the position of an object at one location, before walking to another location in the room and judging, in a second interval, whether an object was at the expected visual direction of the first. By manipulating the scale of the room across intervals, which was subjectively invisible to observers, we put two classes of cue into conflict, one that depends only on visual information and one that uses proprioceptive information to scale any reconstruction of the scene. We find that the sensitivity to changes in one class of cue while keeping the other constant provides a good prediction of performance when both cues vary, consistent with the standard cue combination framework. Nevertheless, by comparing judgments of visual direction with those of distance, we show that judgments of visual direction and distance are mutually inconsistent. We discuss why there is no need for any contradiction between these two conclusions.

Behavioral and Neural Fusion of Expectation with Sensation

Humans perceive expected stimuli faster and more accurately. However, the mechanism behind the integration of expectations with sensory information during perception remains unclear. We investigated the hypothesis that such integration depends on "fusion"-the weighted averaging of different cues informative about stimulus identity. We first trained participants to map a range of tones onto faces spanning a male-female continuum via associative learning. These two features served as expectation and sensory cues to sex, respectively. We then tested specific predictions about the consequences of fusion by manipulating the congruence of these cues in psychophysical and fMRI experiments. Behavioral judgments and patterns of neural activity in auditory association regions revealed fusion of sensory and expectation cues, providing evidence for a precise computational account of how expectations influence perception.

Cue-dependent effects of VR experience on motion-in-depth sensitivity

The visual system exploits multiple signals, including monocular and binocular cues, to determine the motion of objects through depth. In the laboratory, sensitivity to different three-dimensional (3D) motion cues varies across observers and is often weak for binocular cues. However, laboratory assessments may reflect factors beyond inherent perceptual sensitivity. For example, the appearance of weak binocular sensitivity may relate to extensive prior experience with two-dimensional (2D) displays in which binocular cues are not informative. Here we evaluated the impact of experience on motion-in-depth (MID) sensitivity in a virtual reality (VR) environment. We tested a large cohort of observers who reported having no prior VR experience and found that binocular cue sensitivity was substantially weaker than monocular cue sensitivity. As expected, sensitivity was greater when monocular and binocular cues were presented together than in isolation. Surprisingly, the addition of motion parallax signals appeared to cause observers to rely almost exclusively on monocular cues. As observers gained experience in the VR task, sensitivity to monocular and binocular cues increased. Notably, most observers were unable to distinguish the direction of MID based on binocular cues above chance level when tested early in the experiment, whereas most showed statistically significant sensitivity to binocular cues when tested late in the experiment. This result suggests that observers may discount binocular cues when they are first encountered in a VR environment. Laboratory assessments may thus underestimate the sensitivity of inexperienced observers to MID, especially for binocular cues.

Integration of Motion and Form Cues for the Perception of Self-Motion in the Human Brain

When moving around in the world, the human visual system uses both motion and form information to estimate the direction of self-motion (i.e., heading). However, little is known about cortical areas in charge of this task. This brain-imaging study addressed this question by using visual stimuli consisting of randomly distributed dot pairs oriented toward a locus on a screen (the form-defined focus of expansion [FoE]) but moved away from a different locus (the motion-defined FoE) to simulate observer translation. We first fixed the motion-defined FoE location and shifted the form-defined FoE location. We then made the locations of the motion- and the form-defined FoEs either congruent (at the same location in the display) or incongruent (on the opposite sides of the display). The motion- or the form-defined FoE shift was the same in the two types of stimuli, but the perceived heading direction shifted for the congruent, but not for the incongruent stimuli. Participants (both sexes) made a task-irrelevant (contrast discrimination) judgment during scanning. Searchlight and ROI-based multivoxel pattern analysis revealed that early visual areas V1, V2, and V3 responded to either the motion- or the form-defined FoE shift. After V3, only the dorsal areas V3a and V3B/KO responded to such shifts. Furthermore, area V3B/KO shows a significantly higher decoding accuracy for the congruent than the incongruent stimuli. Our results provide direct evidence showing that area V3B/KO does not simply respond to motion and form cues but integrates these two cues for the perception of heading.SIGNIFICANCE STATEMENT Human survival relies on accurate perception of self-motion. The visual system uses both motion (optic flow) and form cues for the perception of the direction of self-motion (heading). Although human brain areas for processing optic flow and form structure are well identified, the areas responsible for integrating these two cues for the perception of self-motion remain unknown. We conducted fMRI experiments and used multivoxel pattern analysis technique to find human brain areas that can decode the shift in heading specified by each cue alone and the two cues combined. We found that motion and form cues are first processed in the early visual areas and then are likely integrated in the higher dorsal area V3B/KO for the final estimation of heading.

Optimized but Not Maximized Cue Integration for 3D Visual Perception

Reconstructing three-dimensional (3D) scenes from two-dimensional (2D) retinal images is an ill-posed problem. Despite this, 3D perception of the world based on 2D retinal images is seemingly accurate and precise. The integration of distinct visual cues is essential for robust 3D perception in humans, but it is unclear whether this is true for non-human primates (NHPs). Here, we assessed 3D perception in macaque monkeys using a planar surface orientation discrimination task. Perception was accurate across a wide range of spatial poses (orientations and distances), but precision was highly dependent on the plane's pose. The monkeys achieved robust 3D perception by dynamically reweighting the integration of stereoscopic and perspective cues according to their pose-dependent reliabilities. Errors in performance could be explained by a prior resembling the 3D orientation statistics of natural scenes. We used neural network simulations based on 3D orientation-selective neurons recorded from the same monkeys to assess how neural computation might constrain perception. The perceptual data were consistent with a model in which the responses of two independent neuronal populations representing stereoscopic cues and perspective cues (with perspective signals from the two eyes combined using nonlinear canonical computations) were optimally integrated through linear summation. Perception of combined-cue stimuli was optimal given this architecture. However, an alternative architecture in which stereoscopic cues, left eye perspective cues, and right eye perspective cues were represented by three independent populations yielded two times greater precision than the monkeys. This result suggests that, due to canonical computations, cue integration for 3D perception is optimized but not maximized.

Generalized Representation of Stereoscopic Surface Shape and Orientation in the Human Visual Cortex

The brain's ability to extract three-dimensional (3D) shape and orientation information from viewed objects is vital in daily life. Stereoscopic 3D surface perception relies on binocular disparity. Neurons selective to binocular disparity are widely distributed among visual areas, but the manner in these areas are involved in stereoscopic 3D surface representation is unclear. To address this, participants were instructed to observe random dot stereograms (RDS) depicting convex and concave curved surfaces and the blood oxygenation level-dependent (BOLD) signal of visual cortices was recorded. Two surface types were: (i) horizontally positioned surfaces defined by shear disparity; and (ii) vertically positioned surfaces defined by compression disparity. The surfaces were presented at different depth positions per trial. Functional magnetic resonance imaging (fMRI) data were classified from early visual areas to higher visual areas. We determined whether cortical areas were selective to shape and orientation by assessing same-type stimuli classification accuracies based on multi-voxel activity patterns per area. To identify whether some areas were related to a more generalized sign of curvature or orientation representation, transfer classification was used by training classifiers on one dataset type and testing classifiers on another type. Same-type stimuli classification results showed that most selected visual areas were selective to shape and all were selective to the orientation of disparity-defined 3D surfaces. Transfer classification results showed that in the dorsal visual area V3A, classification accuracies for the discriminate sign of surface curvature were higher than the baseline of statistical significance for all types of classifications, demonstrating that V3A is related to generalized shape representation. Classification accuracies for discriminating horizontal-vertical surfaces in higher dorsal areas V3A and V7 and ventral area lateral occipital complex (LOC) as well as in some areas of intraparietal sulcus (IPS) were higher than the baseline of statistical significance, indicating their relation to the generalized representation of 3D surface orientation.

Contextual effects on binocular matching are evident in primary visual cortex

Global context can dramatically influence local visual perception. This phenomenon is well-documented for monocular features, e.g., the Kanizsa triangle. It has been demonstrated for binocular matching: the disambiguation of the Wallpaper Illusion via the luminance of the background. For monocular features, there is evidence that global context can influence neuronal responses as early as V1. However, for binocular matching, the activity in this area of the visual cortex is thought to represent local processing, suggesting that the influence of global context may occur at later stages of cortical processing. Here we sought to test if binocular matching is influenced by contextual effects in V1, using fMRI to measure brain activity while participants viewed perceptually ambiguous "wallpaper" stereograms whose depth was disambiguated by the luminance of the surrounding region. We localized voxels in V1 corresponding to the ambiguous region of the pattern, i.e., where the signal received from the eyes was not predictive of depth, and despite the ambiguity of the input signal, using multi-voxel pattern analysis we were able to reliably decode perceived (near/far) depth from the activity of these voxels. These findings indicate that stereoscopic related neural activity is influenced by global context as early as V1.

Areal differences in depth cue integration between monkey and human

Electrophysiological evidence suggested primarily the involvement of the middle temporal (MT) area in depth cue integration in macaques, as opposed to human imaging data pinpointing area V3B/kinetic occipital area (V3B/KO). To clarify this conundrum, we decoded monkey functional MRI (fMRI) responses evoked by stimuli signaling near or far depths defined by binocular disparity, relative motion, and their combination, and we compared results with those from an identical experiment previously performed in humans. Responses in macaque area MT are more discriminable when two cues concurrently signal depth, and information provided by one cue is diagnostic of depth indicated by the other. This suggests that monkey area MT computes fusion of disparity and motion depth signals, exactly as shown for human area V3B/KO. Hence, these data reconcile previously reported discrepancies between depth processing in human and monkey by showing the involvement of the dorsal stream in depth cue integration using the same technique, despite the engagement of different regions.

In everyday life, we interact with a three-dimensional world that we perceive via our two-dimensional retinas. Our brain can reconstruct the third dimension from these flat retinal images using multiple sources of visual information, or cues. The horizontal displacement of the two retinal images, known as binocular disparity, and the relative motion between different objects are two important depth cues. However, to make the most of the information provided by each cue, our brains must efficiently integrate across them. To examine this process, we used neuroimaging in monkeys to record brain responses evoked by stimuli signaling depths defined by either binocular disparity or relative motion in isolation, and also when the two cues are combined congruently or incongruently. We found that cortical area MT in monkeys is involved in the fusion of these two particular depth cues, in contrast to previous human imaging data that pinpoint a more posterior cortical area, V3B/KO. Our findings support the existence of depth cue integration mechanisms in primates; however, this fusion appears to be computed in slightly different areas in humans and monkeys.

Magnetoencephalography adaptation reveals depth-cue-invariant object representations in the visual cortex

Independent of edges and 2-D shape that can be highly informative of object identity, depth cues alone can also give rise to vivid and effective object percepts. The processing of different depth cues engages segregated cortical areas, and an efficient object representation would be one that is invariant to depth cues. Here, we investigated depth-cue invariance of object representations by measuring the category-specific response to faces-the M170 response measured with magnetoencephalography. The M170 response is strongest to faces and is sensitive to adaptation, such that repeated presentation of a face diminishes subsequent M170 responses. We used this feature of the M170 and measured the degree to which the adaptation effect is affected by variations in depth cue and 3-D object shape. Subjects viewed a rapid presentation of two stimuli-an adaptor and a test stimulus. The adaptor was either a face, a chair, or a face-like oval surface, and rendered with a single depth cue (shading, structure from motion, or texture). The test stimulus was always a shaded face of a random identity, thus completely controlling for low-level influences on the M170 response to the test stimulus. In the left fusiform face area, we found strong M170 adaptation when the adaptor was a face regardless of its depth cue. This adaptation was marginal in the right fusiform and negligible in the occipital regions. Our results support the presence of depth-cue-invariant representations in the human visual system, alongside size, position, and viewpoint invariance.

Emergence of symmetry selectivity in the visual areas of the human brain: fMRI responses to symmetry presented in both frontoparallel and slanted planes

Symmetry is effortlessly perceived by humans across changes in viewing geometry. Here, we re-examined the network subserving symmetry processing in the context of up-to-date retinotopic definitions of visual areas. Responses in object selective cortex, as defined by functional localizers, were also examined. We further examined responses to both frontoparallel and slanted symmetry while manipulating attention both toward and away from symmetry. Symmetry-specific responses first emerge in V3 and continue across all downstream areas examined. Of the retinotopic areas, ventral occipital VO1 showed the strongest symmetry response, which was similar in magnitude to the responses observed in object selective cortex. Neural responses were found to increase with both the coherence and folds of symmetry. Compared to passive viewing, drawing attention to symmetry generally increased neural responses and the correspondence of these neural responses with psychophysical performance. Examining symmetry on the slanted plane found responses to again emerge in V3, continue through downstream visual cortex, and be strongest in VO1 and LOB. Both slanted and frontoparallel symmetry evoked similar activity when participants performed a symmetry-related task. However, when a symmetry-unrelated task was performed, fMRI responses to slanted symmetry were reduced relative to their frontoparallel counterparts. These task-related changes provide a neural signature that suggests slant has to be computed ahead of symmetry being appropriately extracted, known as the "normalization" account of symmetry processing. Specifically, our results suggest that normalization occurs naturally when attention is directed toward symmetry and orientation, but becomes interrupted when attention is directed away from these features.

Towards a unified perspective of object shape and motion processing in human dorsal cortex

Although object-related areas were discovered in human parietal cortex a decade ago, surprisingly little is known about the nature and purpose of these representations, and how they differ from those in the ventral processing stream. In this article, we review evidence for the unique contribution of object areas of dorsal cortex to three-dimensional (3-D) shape representation, the localization of objects in space, and in guiding reaching and grasping actions. We also highlight the role of dorsal cortex in form-motion interaction and spatiotemporal integration, possible functional relationships between 3-D shape and motion processing, and how these processes operate together in the service of supporting goal-directed actions with objects. Fundamental differences between the nature of object representations in the dorsal versus ventral processing streams are considered, with an emphasis on how and why dorsal cortex supports veridical (rather than invariant) representations of objects to guide goal-directed hand actions in dynamic visual environments.

Proscription supports robust perceptual integration by suppression in human visual cortex

Perception relies on integrating information within and between the senses, but how does the brain decide which pieces of information should be integrated and which kept separate? Here we demonstrate how proscription can be used to solve this problem: certain neurons respond best to unrealistic combinations of features to provide ‘what not’ information that drives suppression of unlikely perceptual interpretations. First, we present a model that captures both improved perception when signals are consistent (and thus should be integrated) and robust estimation when signals are conflicting. Second, we test for signatures of proscription in the human brain. We show that concentrations of inhibitory neurotransmitter GABA in a brain region intricately involved in integrating cues (V3B/KO) correlate with robust integration. Finally, we show that perturbing excitation/inhibition impairs integration. These results highlight the role of proscription in robust perception and demonstrate the functional purpose of ‘what not’ sensors in supporting sensory estimation.

Perception relies on information integration but it is unclear how the brain decides which information to integrate and which to keep separate. Here, the authors develop and test a biologically inspired model of cue-integration, implicating a key role for GABAergic proscription in robust perception.

The lawful imprecision of human surface tilt estimation in natural scenes

Estimating local surface orientation (slant and tilt) is fundamental to recovering the three-dimensional structure of the environment. It is unknown how well humans perform this task in natural scenes. Here, with a database of natural stereo-images having groundtruth surface orientation at each pixel, we find dramatic differences in human tilt estimation with natural and artificial stimuli. Estimates are precise and unbiased with artificial stimuli and imprecise and strongly biased with natural stimuli. An image-computable Bayes optimal model grounded in natural scene statistics predicts human bias, precision, and trial-by-trial errors without fitting parameters to the human data. The similarities between human and model performance suggest that the complex human performance patterns with natural stimuli are lawful, and that human visual systems have internalized local image and scene statistics to optimally infer the three-dimensional structure of the environment. These results generalize our understanding of vision from the lab to the real world.

Early dynamics of stereoscopic surface slant perception

Surface orientation is an important visual primitive that can be estimated from monocular or binocular (stereoscopic) signals. Changes in motor planning occur within about 200 ms after either type of signal is perturbed, but the time it takes for apparent (perceived) slant to develop from stereoscopic cues is not known. Apparent slant sometimes develops very slowly (Gillam, Chambers, & Russo, 1988; van Ee & Erkelens, 1996). However, these long durations could reflect the time it takes for the visual system to resolve conflicts between slant cues that inevitably specify different slants in laboratory displays (Allison & Howard, 2000). We used a speed–accuracy tradeoff analysis to measure the time it takes to discriminate slant, allowing us to report psychometric functions as a function of response time. Observers reported which side of a slanted surface was farther, with a temporal deadline for responding that varied block-to-block. Stereoscopic slant discrimination rose above chance starting at 200 ms after stimulus onset. Unexpectedly, observers discriminated slant from binocular disparity faster than texture, and for stereoscopic whole-field stimuli faster than stereoscopic slant contrast stimuli. However, performance after the initial deviation from chance increased more rapidly for slant-contrast stimuli than whole-field stimuli. Discrimination latencies were similar for slants about the horizontal and vertical axes, but performance increased faster for slants about the vertical axis. Finally, slant from vertical disparity was somewhat slower than slant from horizontal disparity, which may reflect cue conflict. These results demonstrate, in contradiction with the previous literature, that the perception of slant from disparity happens very quickly—in fact, more quickly than the perception of slant from texture—and in comparable time to the simple perception of brightness from luminance.

Recovering stereo vision by squashing virtual bugs in a virtual reality environment

Stereopsis is the rich impression of three-dimensionality, based on binocular disparity-the differences between the two retinal images of the same world. However, a substantial proportion of the population is stereo-deficient, and relies mostly on monocular cues to judge the relative depth or distance of objects in the environment. Here we trained adults who were stereo blind or stereo-deficient owing to strabismus and/or amblyopia in a natural visuomotor task-a 'bug squashing' game-in a virtual reality environment. The subjects' task was to squash a virtual dichoptic bug on a slanted surface, by hitting it with a physical cylinder they held in their hand. The perceived surface slant was determined by monocular texture and stereoscopic cues, with these cues being either consistent or in conflict, allowing us to track the relative weighting of monocular versus stereoscopic cues as training in the task progressed. Following training most participants showed greater reliance on stereoscopic cues, reduced suppression and improved stereoacuity. Importantly, the training-induced changes in relative stereo weights were significant predictors of the improvements in stereoacuity. We conclude that some adults deprived of normal binocular vision and insensitive to the disparity information can, with appropriate experience, recover access to more reliable stereoscopic information.This article is part of the themed issue 'Vision in our three-dimensional world'.

A dataset of stereoscopic images and ground-truth disparity mimicking human fixations in peripersonal space

Binocular stereopsis is the ability of a visual system, belonging to a live being or a machine, to interpret the different visual information deriving from two eyes/cameras for depth perception. From this perspective, the ground-truth information about three-dimensional visual space, which is hardly available, is an ideal tool both for evaluating human performance and for benchmarking machine vision algorithms. In the present work, we implemented a rendering methodology in which the camera pose mimics realistic eye pose for a fixating observer, thus including convergent eye geometry and cyclotorsion. The virtual environment we developed relies on highly accurate 3D virtual models, and its full controllability allows us to obtain the stereoscopic pairs together with the ground-truth depth and camera pose information. We thus created a stereoscopic dataset: GENUA PESTO-GENoa hUman Active fixation database: PEripersonal space STereoscopic images and grOund truth disparity. The dataset aims to provide a unified framework useful for a number of problems relevant to human and computer vision, from scene exploration and eye movement studies to 3D scene reconstruction.

Differential patterns of 2D location versus depth decoding along the visual hierarchy

Visual information is initially represented as 2D images on the retina, but our brains are able to transform this input to perceive our rich 3D environment. While many studies have explored 2D spatial representations or depth perception in isolation, it remains unknown if or how these processes interact in human visual cortex. Here we used functional MRI and multi-voxel pattern analysis to investigate the relationship between 2D location and position-in-depth information. We stimulated different 3D locations in a blocked design: each location was defined by horizontal, vertical, and depth position. Participants remained fixated at the center of the screen while passively viewing the peripheral stimuli with red/green anaglyph glasses. Our results revealed a widespread, systematic transition throughout visual cortex. As expected, 2D location information (horizontal and vertical) could be strongly decoded in early visual areas, with reduced decoding higher along the visual hierarchy, consistent with known changes in receptive field sizes. Critically, we found that the decoding of position-in-depth information tracked inversely with the 2D location pattern, with the magnitude of depth decoding gradually increasing from intermediate to higher visual and category regions. Representations of 2D location information became increasingly location-tolerant in later areas, where depth information was also tolerant to changes in 2D location. We propose that spatial representations gradually transition from 2D-dominant to balanced 3D (2D and depth) along the visual hierarchy.

Estimating 3D tilt from local image cues in natural scenes

Estimating three-dimensional (3D) surface orientation (slant and tilt) is an important first step toward estimating 3D shape. Here, we examine how three local image cues from the same location (disparity gradient, luminance gradient, and dominant texture orientation) should be combined to estimate 3D tilt in natural scenes. We collected a database of natural stereoscopic images with precisely co-registered range images that provide the ground-truth distance at each pixel location. We then analyzed the relationship between ground-truth tilt and image cue values. Our analysis is free of assumptions about the joint probability distributions and yields the Bayes optimal estimates of tilt, given the cue values. Rich results emerge: (a) typical tilt estimates are only moderately accurate and strongly influenced by the cardinal bias in the prior probability distribution; (b) when cue values are similar, or when slant is greater than 40°, estimates are substantially more accurate; (c) when luminance and texture cues agree, they often veto the disparity cue, and when they disagree, they have little effect; and (d) simplifying assumptions common in the cue combination literature is often justified for estimating tilt in natural scenes. The fact that tilt estimates are typically not very accurate is consistent with subjective impressions from viewing small patches of natural scene. The fact that estimates are substantially more accurate for a subset of image locations is also consistent with subjective impressions and with the hypothesis that perceived surface orientation, at more global scales, is achieved by interpolation or extrapolation from estimates at key locations.

A Major Human White Matter Pathway Between Dorsal and Ventral Visual Cortex

Human visual cortex comprises many visual field maps organized into clusters. A standard organization separates visual maps into 2 distinct clusters within ventral and dorsal cortex. We combined fMRI, diffusion MRI, and fiber tractography to identify a major white matter pathway, the vertical occipital fasciculus (VOF), connecting maps within the dorsal and ventral visual cortex. We use a model-based method to assess the statistical evidence supporting several aspects of the VOF wiring pattern. There is strong evidence supporting the hypothesis that dorsal and ventral visual maps communicate through the VOF. The cortical projection zones of the VOF suggest that human ventral (hV4/VO-1) and dorsal (V3A/B) maps exchange substantial information. The VOF appears to be crucial for transmitting signals between regions that encode object properties including form, identity, and color and regions that map spatial information.

Differential processing of binocular and monocular gloss cues in human visual cortex

The visual impression of an object's surface reflectance (“gloss”) relies on a range of visual cues, both monocular and binocular. Whereas previous imaging work has identified processing within ventral visual areas as important for monocular cues, little is known about cortical areas involved in processing binocular cues. Here, we used human functional MRI (fMRI) to test for brain areas selectively involved in the processing of binocular cues. We manipulated stereoscopic information to create four conditions that differed in their disparity structure and in the impression of surface gloss that they evoked. We performed multivoxel pattern analysis to find areas whose fMRI responses allow classes of stimuli to be distinguished based on their depth structure vs. material appearance. We show that higher dorsal areas play a role in processing binocular gloss information, in addition to known ventral areas involved in material processing, with ventral area lateral occipital responding to both object shape and surface material properties. Moreover, we tested for similarities between the representation of gloss from binocular cues and monocular cues. Specifically, we tested for transfer in the decoding performance of an algorithm trained on glossy vs. matte objects defined by either binocular or by monocular cues. We found transfer effects from monocular to binocular cues in dorsal visual area V3B/kinetic occipital (KO), suggesting a shared representation of the two cues in this area. These results indicate the involvement of mid- to high-level visual circuitry in the estimation of surface material properties, with V3B/KO potentially playing a role in integrating monocular and binocular cues.

Flexible integration of visual cues in adolescents with autism spectrum disorder

Although children with autism spectrum disorder (ASD) show atypical sensory processing, evidence for impaired integration of multisensory information has been mixed. In this study, we took a Bayesian model-based approach to assess within-modality integration of congruent and incongruent texture and disparity cues to judge slant in typical and autistic adolescents. Human adults optimally combine multiple sources of sensory information to reduce perceptual variance but in typical development this ability to integrate cues does not develop until late childhood. While adults cannot help but integrate cues, even when they are incongruent, young children's ability to keep cues separate gives them an advantage in discriminating incongruent stimuli. Given that mature cue integration emerges in later childhood, we hypothesized that typical adolescents would show adult-like integration, combining both congruent and incongruent cues. For the ASD group there were three possible predictions (1) "no fusion": no integration of congruent or incongruent cues, like 6-year-old typical children; (2) "mandatory fusion": integration of congruent and incongruent cues, like typical adults; (3) "selective fusion": cues are combined when congruent but not incongruent, consistent with predictions of Enhanced Perceptual Functioning (EPF) theory. As hypothesized, typical adolescents showed significant integration of both congruent and incongruent cues. The ASD group showed results consistent with "selective fusion," integrating congruent but not incongruent cues. This allowed adolescents with ASD to make perceptual judgments which typical adolescents could not. In line with EPF, results suggest that perception in ASD may be more flexible and less governed by mandatory top-down feedback.

Contribution of binocular vision to the performance of complex manipulation tasks in 5-13years old visually-normal children

Individual studies have shown that visuomotor coordination and aspects of binocular vision, such as stereoacuity and dynamic vergence control, continue to improve in normally developing children between birth and early teenage years. However, no study has systematically addressed the relationship between the development of binocular vision and fine manipulation skills. Thus, the aim of this cross-sectional study was to characterize performance of complex manipulation tasks during binocular and monocular viewing. Fifty-two children, between 5 and 13years old, performed 2 manipulation tasks: peg-board and bead-threading under randomized viewing conditions. Results showed that binocular viewing was associated with a significantly greater improvement in performance on the bead-threading task in comparison to the peg-board task and the youngest children showed the greatest decrement in task performance under the monocular viewing condition when performing the bead-threading task. Thus, the role of binocular vision in performance of fine manipulation skills is both task- and age-dependent. These findings have implications for assessment of visuomotor skills in children with abnormal binocular vision, which occurs in 2-3% of otherwise typically developing children.

Development of Three-Dimensional Perception in Human Infants

The play of light on the retina contains multiple sources of information about the three-dimensional (3D) structure of the world. Some of the best information is derived from differencing operations that act on the images that result from the two eyes’ laterally displaced vantage points. Other information is available in systematic retinal patterns of local texture and motion cues. This article describes what is currently known about the development of sensitivity to these binocular and monocular cues for depth in human infants, and it places the results in the context of what is known about the underlying neural mechanisms from work in nonhuman primates and human neuroimaging studies.

Late development of cue integration is linked to sensory fusion in cortex

Adults optimize perceptual judgements by integrating different types of sensory information [1, 2]. This engages specialized neural circuits that fuse signals from the same [3, 4, 5] or different [6] modalities. Whereas young children can use sensory cues independently, adult-like precision gains from cue combination only emerge around ages 10 to 11 years [7, 8, 9]. Why does it take so long to make best use of sensory information? Existing data cannot distinguish whether this (1) reflects surprisingly late changes in sensory processing (sensory integration mechanisms in the brain are still developing) or (2) depends on post-perceptual changes (integration in sensory cortex is adult-like, but higher-level decision processes do not access the information) [10]. We tested visual depth cue integration in the developing brain to distinguish these possibilities. We presented children aged 6–12 years with displays depicting depth from binocular disparity and relative motion and made measurements using psychophysics, retinotopic mapping, and pattern classification fMRI. Older children (>10.5 years) showed clear evidence for sensory fusion in V3B, a visual area thought to integrate depth cues in the adult brain [3, 4, 5]. By contrast, in younger children (<10.5 years), there was no evidence for sensory fusion in any visual area. This significant age difference was paired with a shift in perceptual performance around ages 10 to 11 years and could not be explained by motion artifacts, visual attention, or signal quality differences. Thus, whereas many basic visual processes mature early in childhood [11, 12], the brain circuits that fuse cues take a very long time to develop.

•

Children aged 6–12 years viewed displays depicting depth via two visual cues

•

We used MVPA fMRI to test for fusion of these cues in retinotopic cortex

•

Perceptual and cortical fusion of depth cues emerged only after age ∼11 years

•

Children’s perception is limited by immature fusion of depth cues in visual cortex

fMRI Analysis-by-Synthesis Reveals a Dorsal Hierarchy That Extracts Surface Slant

The brain's skill in estimating the 3-D orientation of viewed surfaces supports a range of behaviors, from placing an object on a nearby table, to planning the best route when hill walking. This ability relies on integrating depth signals across extensive regions of space that exceed the receptive fields of early sensory neurons. Although hierarchical selection and pooling is central to understanding of the ventral visual pathway, the successive operations in the dorsal stream are poorly understood. Here we use computational modeling of human fMRI signals to probe the computations that extract 3-D surface orientation from binocular disparity. To understand how representations evolve across the hierarchy, we developed an inference approach using a series of generative models to explain the empirical fMRI data in different cortical areas. Specifically, we simulated the responses of candidate visual processing algorithms and tested how well they explained fMRI responses. Thereby we demonstrate a hierarchical refinement of visual representations moving from the representation of edges and figure-ground segmentation (V1, V2) to spatially extensive disparity gradients in V3A. We show that responses in V3A are little affected by low-level image covariates, and have a partial tolerance to the overall depth position. Finally, we show that responses in V3A parallel perceptual judgments of slant. This reveals a relatively short computational hierarchy that captures key information about the 3-D structure of nearby surfaces, and more generally demonstrates an analysis approach that may be of merit in a diverse range of brain imaging domains.

7 tesla FMRI reveals systematic functional organization for binocular disparity in dorsal visual cortex

The binocular disparity between the views of the world registered by the left and right eyes provides a powerful signal about the depth structure of the environment. Despite increasing knowledge of the cortical areas that process disparity from animal models, comparatively little is known about the local architecture of stereoscopic processing in the human brain. Here, we take advantage of the high spatial specificity and image contrast offered by 7 tesla fMRI to test for systematic organization of disparity representations in the human brain. Participants viewed random dot stereogram stimuli depicting different depth positions while we recorded fMRI responses from dorsomedial visual cortex. We repeated measurements across three separate imaging sessions. Using a series of computational modeling approaches, we report three main advances in understanding disparity organization in the human brain. First, we show that disparity preferences are clustered and that this organization persists across imaging sessions, particularly in area V3A. Second, we observe differences between the local distribution of voxel responses in early and dorsomedial visual areas, suggesting different cortical organization. Third, using modeling of voxel responses, we show that higher dorsal areas (V3A, V3B/KO) have properties that are characteristic of human depth judgments: a simple model that uses tuning parameters estimated from fMRI data captures known variations in human psychophysical performance. Together, these findings indicate that human dorsal visual cortex contains selective cortical structures for disparity that may support the neural computations that underlie depth perception.

fMRI evidence for areas that process surface gloss in the human visual cortex

•

Glossiness information is mainly processed along ventral visual pathway.

•

The posterior fusiform sulcus (pFs) is especially selective to surface gloss.

•

V3B/KO responds to gloss, but differentially from the pFs.

Surface gloss is an important cue to the material properties of objects. Recent progress in the study of macaque’s brain has increased our understating of the areas involved in processing information about gloss, however the homologies with the human brain are not yet fully understood. Here we used human functional magnetic resonance imaging (fMRI) measurements to localize brain areas preferentially responding to glossy objects. We measured cortical activity for thirty-two rendered three-dimensional objects that had either Lambertian or specular surface properties. To control for differences in image structure, we overlaid a grid on the images and scrambled its cells. We found activations related to gloss in the posterior fusiform sulcus (pFs) and in area V3B/KO. Subsequent analysis with Granger causality mapping indicated that V3B/KO processes gloss information differently than pFs. Our results identify a small network of mid-level visual areas whose activity may be important in supporting the perception of surface gloss.

Reliability-dependent contributions of visual orientation cues in parietal cortex

Creating accurate 3D representations of the world from 2D retinal images is a fundamental task for the visual system. However, the reliability of different 3D visual signals depends inherently on viewing geometry, such as how much an object is slanted in depth. Human perceptual studies have correspondingly shown that texture and binocular disparity cues for object orientation are combined according to their slant-dependent reliabilities. Where and how this cue combination occurs in the brain is currently unknown. Here, we search for neural correlates of this property in the macaque caudal intraparietal area (CIP) by measuring slant tuning curves using mixed-cue (texture + disparity) and cue-isolated (texture or disparity) planar stimuli. We find that texture cues contribute more to the mixed-cue responses of CIP neurons that prefer larger slants, consistent with theoretical and psychophysical results showing that the reliability of texture relative to disparity cues increases with slant angle. By analyzing responses to binocularly viewed texture stimuli with conflicting texture and disparity information, some cells that are sensitive to both cues when presented in isolation are found to disregard one of the cues during cue conflict. Additionally, the similarity between texture and mixed-cue responses is found to be greater when this cue conflict is eliminated by presenting the texture stimuli monocularly. The present findings demonstrate reliability-dependent contributions of visual orientation cues at the level of the CIP, thus revealing a neural correlate of this property of human visual perception.