Ideal Binocular Disparity Detectors Learned Using Independent Subspace Analysis on Binocular Natural Image Pairs

3D Contrast Image Reconstruction From Human Brain Activity

Several studies demonstrated that functional magnetic resonance imaging (fMRI) signals in early visual cortex can be used to reconstruct 2-dimensional (2D) visual contents. However, it remains unknown how to reconstruct 3-dimensional (3D) visual stimuli from fMRI signals in visual cortex. 3D visual stimuli contain 2D visual features and depth information. Moreover, binocular disparity is an important cue for depth perception. Thus, it is more challenging to reconstruct 3D visual stimuli than 2D visual stimuli from the fMRI signals of visual cortex. This study aimed to reconstruct 3D visual images by constructing three decoding models: contrast-decoding, disparity-decoding and contrast-disparity-decoding models, and testing these models with fMRI data from humans viewing 3D contrast images. The results revealed that the 3D contrast stimuli can be reconstructed from the visual cortex. And the early visual regions (V1, V2) showed predominant advantages in reconstructing the contrast in 3D images for the contrast-decoding model. The dorsal visual regions (V3A, V7 and MT) showed predominant advantages in decoding the disparity in 3D images for the disparity-decoding model. The combination of the early and dorsal visual regions showed predominant advantages in decoding both the contrast and disparity for the contrast-disparity-decoding model. The results suggested that the contrast and disparity in 3D images were mainly represented in the early and dorsal visual regions separately. The two visual systems may interact with each other to decode 3D-contrast images.

Visual Perception: Neural Networks for Stereopsis

How does our brain use differences between the images in our two eyes, binocular disparities, to generate depth perception? New work shows that a type of neural network trained on natural binocular images can learn parameters that match key properties of visual cortex. Most information is conveyed by cells which sense differences between the two eyes' images.

The effect of image position on the Independent Components of natural binocular images

Human visual performance degrades substantially as the angular distance from the fovea increases. This decrease in performance is found for both binocular and monocular vision. Although analysis of the statistics of natural images has provided significant insights into human visual processing, little research has focused on the statistical content of binocular images at eccentric angles. We applied Independent Component Analysis to rectangular image patches cut from locations within binocular images corresponding to different degrees of eccentricity. The distribution of components learned from the varying locations was examined to determine how these distributions varied across eccentricity. We found a general trend towards a broader spread of horizontal and vertical position disparity tunings in eccentric regions compared to the fovea, with the horizontal spread more pronounced than the vertical spread. Eccentric locations above the centroid show a strong bias towards far-tuned components, eccentric locations below the centroid show a strong bias towards near-tuned components. These distributions exhibit substantial similarities with physiological measurements in V1, however in common with previous research we also observe important differences, in particular distributions of binocular phase disparity which do not match physiology.

"What Not" Detectors Help the Brain See in Depth

Binocular stereopsis is one of the primary cues for three-dimensional (3D) vision in species ranging from insects to primates. Understanding how the brain extracts depth from two different retinal images represents a tractable challenge in sensory neuroscience that has so far evaded full explanation. Central to current thinking is the idea that the brain needs to identify matching features in the two retinal images (i.e., solving the “stereoscopic correspondence problem”) so that the depth of objects in the world can be triangulated. Although intuitive, this approach fails to account for key physiological and perceptual observations. We show that formulating the problem to identify “correct matches” is suboptimal and propose an alternative, based on optimal information encoding, that mixes disparity detection with “proscription”: exploiting dissimilar features to provide evidence against unlikely interpretations. We demonstrate the role of these “what not” responses in a neural network optimized to extract depth in natural images. The network combines information for and against the likely depth structure of the viewed scene, naturally reproducing key characteristics of both neural responses and perceptual interpretations. We capture the encoding and readout computations of the network in simple analytical form and derive a binocular likelihood model that provides a unified account of long-standing puzzles in 3D vision at the physiological and perceptual levels. We suggest that marrying detection with proscription provides an effective coding strategy for sensory estimation that may be useful for diverse feature domains (e.g., motion) and multisensory integration.

•

The brain uses “what not” detectors to facilitate 3D vision

•

Binocular mismatches are used to drive suppression of incompatible depths

•

Proscription accounts for depth perception without binocular correspondence

•

A simple analytical model captures perceptual and neural responses

A dataset of stereoscopic images and ground-truth disparity mimicking human fixations in peripersonal space

Binocular stereopsis is the ability of a visual system, belonging to a live being or a machine, to interpret the different visual information deriving from two eyes/cameras for depth perception. From this perspective, the ground-truth information about three-dimensional visual space, which is hardly available, is an ideal tool both for evaluating human performance and for benchmarking machine vision algorithms. In the present work, we implemented a rendering methodology in which the camera pose mimics realistic eye pose for a fixating observer, thus including convergent eye geometry and cyclotorsion. The virtual environment we developed relies on highly accurate 3D virtual models, and its full controllability allows us to obtain the stereoscopic pairs together with the ground-truth depth and camera pose information. We thus created a stereoscopic dataset: GENUA PESTO-GENoa hUman Active fixation database: PEripersonal space STereoscopic images and grOund truth disparity. The dataset aims to provide a unified framework useful for a number of problems relevant to human and computer vision, from scene exploration and eye movement studies to 3D scene reconstruction.