Understanding the cortical specialization for horizontal disparity

Transformations of sensory information in the brain suggest changing criteria for optimality

Neurons throughout the brain modulate their firing rate lawfully in response to sensory input. Theories of neural computation posit that these modulations reflect the outcome of a constrained optimization in which neurons aim to robustly and efficiently represent sensory information. Our understanding of how this optimization varies across different areas in the brain, however, is still in its infancy. Here, we show that neural sensory responses transform along the dorsal stream of the visual system in a manner consistent with a transition from optimizing for information preservation towards optimizing for perceptual discrimination. Focusing on the representation of binocular disparities-the slight differences in the retinal images of the two eyes-we re-analyze measurements characterizing neuronal tuning curves in brain areas V1, V2, and MT (middle temporal) in the macaque monkey. We compare these to measurements of the statistics of binocular disparity typically encountered during natural behaviors using a Fisher Information framework. The differences in tuning curve characteristics across areas are consistent with a shift in optimization goals: V1 and V2 population-level responses are more consistent with maximizing the information encoded about naturally occurring binocular disparities, while MT responses shift towards maximizing the ability to support disparity discrimination. We find that a change towards tuning curves preferring larger disparities is a key driver of this shift. These results provide new insight into previously-identified differences between disparity-selective areas of cortex and suggest these differences play an important role in supporting visually-guided behavior. Our findings emphasize the need to consider not just information preservation and neural resources, but also relevance to behavior, when assessing the optimality of neural codes.

What's special about horizontal disparity

Horizontal disparity has been recognized as the primary signal driving stereoscopic depth since the invention of the stereoscope in the 1830s. It has a unique status in our understanding of binocular vision. The direction of offset of the eyes gives the disparities of corresponding image point locations across the two retinas a strong horizontal bias. Beyond the retina, other factors give shape to the effective disparity direction used by visual mechanisms. The influence of orientation is examined here. I argue that horizontal disparity is an inflection point along a continuum of effective directions, and its role in stereo vision can be reinterpreted. The pointwise geometric justification for its special status neglects the oriented structural elements of spatial vision, its physiological support is equivocal, and psychophysical support of its special status may partially reflect biased stimulus sampling. The literature shows that horizontal disparity plays no particular role in the processing of one-dimensional stimuli, a reflection of the stereo aperture problem. The resulting depth is non-veridical, even non-transitive. Although one-dimensional components contribute to the stereo depth of visual objects generally, two-dimensional stimuli appear not to inherit the aperture problem. However, a look at the two-dimensional stimuli that predominate in experimental studies shows regularities in orientation that give a new perspective on horizontal disparity.

Transformations of sensory information in the brain reflect a changing definition of optimality

Neurons throughout the brain modulate their firing rate lawfully in response to changes in sensory input. Theories of neural computation posit that these modulations reflect the outcome of a constrained optimization: neurons aim to efficiently and robustly represent sensory information under resource limitations. Our understanding of how this optimization varies across the brain, however, is still in its infancy. Here, we show that neural responses transform along the dorsal stream of the visual system in a manner consistent with a transition from optimizing for information preservation to optimizing for perceptual discrimination. Focusing on binocular disparity - the slight differences in how objects project to the two eyes - we re-analyze measurements from neurons characterizing tuning curves in macaque monkey brain regions V1, V2, and MT, and compare these to measurements of the natural visual statistics of binocular disparity. The changes in tuning curve characteristics are computationally consistent with a shift in optimization goals from maximizing the information encoded about naturally occurring binocular disparities to maximizing the ability to support fine disparity discrimination. We find that a change towards tuning curves preferring larger disparities is a key driver of this shift. These results provide new insight into previously-identified differences between disparity-selective regions of cortex and suggest these differences play an important role in supporting visually-guided behavior. Our findings support a key re-framing of optimal coding in regions of the brain that contain sensory information, emphasizing the need to consider not just information preservation and neural resources, but also relevance to behavior.

Humans Perceive Binocular Rivalry and Fusion in a Tristable Dynamic State

Human vision combines inputs from the two eyes into one percept. Small differences "fuse" together, whereas larger differences are seen "rivalrously" from one eye at a time. These outcomes are typically treated as mutually exclusive processes, with paradigms targeting one or the other and fusion being unreported in most rivalry studies. Is fusion truly a default, stable state that only breaks into rivalry for non-fusible stimuli? Or are monocular and fused percepts three sub-states of one dynamical system? To determine whether fusion and rivalry are separate processes, we measured human perception of Gabor patches with a range of interocular orientation disparities. Observers (10 female, 5 male) reported rivalrous, fused, and uncertain percepts over time. We found a dynamic "tristable" zone spanning from ∼25-35° of orientation disparity where fused, left-eye-, or right-eye-dominant percepts could all occur. The temporal characteristics of fusion and non-fusion periods during tristability matched other bistable processes. We tested statistical models with fusion as a higher-level bistable process alternating with rivalry against our findings. None of these fit our data, but a simple bistable model extended to have three states reproduced many of our observations. We conclude that rivalry and fusion are multistable substates capable of direct competition, rather than separate bistable processes.SIGNIFICANCE STATEMENT When inputs to the two eyes differ, they can either fuse together or engage in binocular rivalry, where each eye's view is seen exclusively in turn. Visual stimuli have often been tailored to produce either fusion or rivalry, implicitly treating them as separate mutually-exclusive perceptual processes. We have found that some similar-but-different stimuli can result in both outcomes over time. Comparing various simple models with our results suggests that rivalry and fusion are not independent processes, but compete within a single multistable system. This conceptual shift is a step toward unifying fusion and rivalry, and understanding how they both contribute to the visual system's production of a unified interpretation of the conflicting images cast on the retina by real-world scenes.

Autonomous Development of Active Binocular and Motion Vision Through Active Efficient Coding

We present a model for the autonomous and simultaneous learning of active binocular and motion vision. The model is based on the Active Efficient Coding (AEC) framework, a recent generalization of classic efficient coding theories to active perception. The model learns how to efficiently encode the incoming visual signals generated by an object moving in 3-D through sparse coding. Simultaneously, it learns how to produce eye movements that further improve the efficiency of the sensory coding. This learning is driven by an intrinsic motivation to maximize the system's coding efficiency. We test our approach on the humanoid robot iCub using simulations. The model demonstrates self-calibration of accurate object fixation and tracking of moving objects. Our results show that the model keeps improving until it hits physical constraints such as camera or motor resolution, or limits on its internal coding capacity. Furthermore, we show that the emerging sensory tuning properties are in line with results on disparity, motion, and motion-in-depth tuning in the visual cortex of mammals. The model suggests that vergence and tracking eye movements can be viewed as fundamentally having the same objective of maximizing the coding efficiency of the visual system and that they can be learned and calibrated jointly through AEC.

Motion and binocular disparity processing: Two sides of two different coins

From a mathematical point of view, extracting motion and disparity signals from a binocular visual stream requires very similar operations, applied over time for motion and across eyes for disparity. This similarity is reflected in the theories that have been proposed to describe the neural mechanisms used by the brain to extract these signals. At the behavioral level there are, however, several differences in how humans react to these stimuli, which presumably reflect differences in how these signals are processed by the brain. Here we highlight three such differences: the degree to which different axes of motion/disparity are treated isotropically, the importance of reference signals, and the rules that underlie the combination of 1D signals to extract 2D signals.

Revisiting the functional significance of binocular cues for perceiving motion-in-depth

Binocular differencing of spatial cues required for perceiving depth relationships is associated with decreased sensitivity to the corresponding retinal image displacements. However, binocular summation of contrast signals increases sensitivity. Here, we investigated this divergence in sensitivity by making direct neural measurements of responses to suprathreshold motion in human adults and 5-month-old infants using steady-state visually evoked potentials. Interocular differences in retinal image motion generated suppressed response functions and correspondingly elevated perceptual thresholds compared to motion matched between the two eyes. This suppression was of equal strength for horizontal and vertical motion and therefore not specific to the perception of motion-in-depth. Suppression is strongly dependent on the presence of spatial references in the image and highly immature in infants. Suppression appears to be the manifestation of a succession of spatial and interocular opponency operations that occur at an intermediate processing stage either before or in parallel with the extraction of motion-in-depth.

The effect of image position on the Independent Components of natural binocular images

Human visual performance degrades substantially as the angular distance from the fovea increases. This decrease in performance is found for both binocular and monocular vision. Although analysis of the statistics of natural images has provided significant insights into human visual processing, little research has focused on the statistical content of binocular images at eccentric angles. We applied Independent Component Analysis to rectangular image patches cut from locations within binocular images corresponding to different degrees of eccentricity. The distribution of components learned from the varying locations was examined to determine how these distributions varied across eccentricity. We found a general trend towards a broader spread of horizontal and vertical position disparity tunings in eccentric regions compared to the fovea, with the horizontal spread more pronounced than the vertical spread. Eccentric locations above the centroid show a strong bias towards far-tuned components, eccentric locations below the centroid show a strong bias towards near-tuned components. These distributions exhibit substantial similarities with physiological measurements in V1, however in common with previous research we also observe important differences, in particular distributions of binocular phase disparity which do not match physiology.

A dataset of stereoscopic images and ground-truth disparity mimicking human fixations in peripersonal space

Binocular stereopsis is the ability of a visual system, belonging to a live being or a machine, to interpret the different visual information deriving from two eyes/cameras for depth perception. From this perspective, the ground-truth information about three-dimensional visual space, which is hardly available, is an ideal tool both for evaluating human performance and for benchmarking machine vision algorithms. In the present work, we implemented a rendering methodology in which the camera pose mimics realistic eye pose for a fixating observer, thus including convergent eye geometry and cyclotorsion. The virtual environment we developed relies on highly accurate 3D virtual models, and its full controllability allows us to obtain the stereoscopic pairs together with the ground-truth depth and camera pose information. We thus created a stereoscopic dataset: GENUA PESTO-GENoa hUman Active fixation database: PEripersonal space STereoscopic images and grOund truth disparity. The dataset aims to provide a unified framework useful for a number of problems relevant to human and computer vision, from scene exploration and eye movement studies to 3D scene reconstruction.

The Active Side of Stereopsis: Fixation Strategy and Adaptation to Natural Environments

Depth perception in near viewing strongly relies on the interpretation of binocular retinal disparity to obtain stereopsis. Statistical regularities of retinal disparities have been claimed to greatly impact on the neural mechanisms that underlie binocular vision, both to facilitate perceptual decisions and to reduce computational load. In this paper, we designed a novel and unconventional approach in order to assess the role of fixation strategy in conditioning the statistics of retinal disparity. We integrated accurate realistic three-dimensional models of natural scenes with binocular eye movement recording, to obtain accurate ground-truth statistics of retinal disparity experienced by a subject in near viewing. Our results evidence how the organization of human binocular visual system is finely adapted to the disparity statistics characterizing actual fixations, thus revealing a novel role of the active fixation strategy over the binocular visual functionality. This suggests an ecological explanation for the intrinsic preference of stereopsis for a close central object surrounded by a far background, as an early binocular aspect of the figure-ground segregation process.

Stereopsis is adaptive for the natural environment

Humans and many animals have forward-facing eyes providing different views of the environment. Precise depth estimates can be derived from the resulting binocular disparities, but determining which parts of the two retinal images correspond to one another is computationally challenging. To aid the computation, the visual system focuses the search on a small range of disparities. We asked whether the disparities encountered in the natural environment match that range. We did this by simultaneously measuring binocular eye position and three-dimensional scene geometry during natural tasks. The natural distribution of disparities is indeed matched to the smaller range of correspondence search. Furthermore, the distribution explains the perception of some ambiguous stereograms. Finally, disparity preferences of macaque cortical neurons are consistent with the natural distribution.

Stereo vision and strabismus

Binocular stereopsis, or stereo vision, is the ability to derive information about how far away objects are, based solely on the relative positions of the object in the two eyes. It depends on both sensory and motor abilities. In this review, I briefly outline some of the neuronal mechanisms supporting stereo vision, and discuss how these are disrupted in strabismus. I explain, in some detail, current methods of assessing stereo vision and their pros and cons. Finally, I review the evidence supporting the clinical importance of such measurements.

Binocular stereoscopy in visual areas V-2, V-3, and V-3A of the macaque monkey

Over 40 years ago, Hubel and Wiesel gave a preliminary report of the first account of cells in monkey cerebral cortex selective for binocular disparity. The cells were located outside of V-1 within a region referred to then as "area 18." A full-length manuscript never followed, because the demarcation of the visual areas within this region had not been fully worked out. Here, we provide a full description of the physiological experiments and identify the locations of the recorded neurons using a contemporary atlas generated by functional magnetic resonance imaging; we also perform an independent analysis of the location of the neurons relative to an anatomical landmark (the base of the lunate sulcus) that is often coincident with the border between V-2 and V-3. Disparity-tuned cells resided not only in V-2, the area now synonymous with area 18, but also in V-3 and probably within V-3A. The recordings showed that the disparity-tuned cells were biased for near disparities, tended to prefer vertical orientations, clustered by disparity preference, and often required stimulation of both eyes to elicit responses, features strongly suggesting a role in stereoscopic depth perception.

Adaptation changes stereoscopic depth selectivity in visual cortex

Exposure to specific visual stimuli causes a reduction in sensitivity to similar subsequent stimulation. This adaptation effect is observed behaviorally and for neurons in the primary visual cortex. Here, we explore the effects of adaptation on neurons that encode binocular depth discrimination in the cat's primary visual cortex. Our results show that neuronal preference for binocular depth is altered selectively with appropriate adaptation. At the preferred depth, adaptation causes substantial suppression of subsequent responses. Near the preferred depth, the same procedure causes a shift in depth preference. At the null depth, adaptation has little effect on binocular depth coding. These results demonstrate that prior exposure can change the depth selectivity of binocular neurons. The findings are relevant to the theoretical treatment of binocular depth processing. Specifically, the prevailing notion of binocular depth encoding based on the energy model requires modification.

Spatial stereoresolution for depth corrugations may be set in primary visual cortex

Stereo “3D” depth perception requires the visual system to extract binocular disparities between the two eyes' images. Several current models of this process, based on the known physiology of primary visual cortex (V1), do this by computing a piecewise-frontoparallel local cross-correlation between the left and right eye's images. The size of the “window” within which detectors examine the local cross-correlation corresponds to the receptive field size of V1 neurons. This basic model has successfully captured many aspects of human depth perception. In particular, it accounts for the low human stereoresolution for sinusoidal depth corrugations, suggesting that the limit on stereoresolution may be set in primary visual cortex. An important feature of the model, reflecting a key property of V1 neurons, is that the initial disparity encoding is performed by detectors tuned to locally uniform patches of disparity. Such detectors respond better to square-wave depth corrugations, since these are locally flat, than to sinusoidal corrugations which are slanted almost everywhere. Consequently, for any given window size, current models predict better performance for square-wave disparity corrugations than for sine-wave corrugations at high amplitudes. We have recently shown that this prediction is not borne out: humans perform no better with square-wave than with sine-wave corrugations, even at high amplitudes. The failure of this prediction raised the question of whether stereoresolution may actually be set at later stages of cortical processing, perhaps involving neurons tuned to disparity slant or curvature. Here we extend the local cross-correlation model to include existing physiological and psychophysical evidence indicating that larger disparities are detected by neurons with larger receptive fields (a size/disparity correlation). We show that this simple modification succeeds in reconciling the model with human results, confirming that stereoresolution for disparity gratings may indeed be limited by the size of receptive fields in primary visual cortex.

Stereo depth perception requires the brain to detect displacements of features between the two eyes' images. Several current models use local cross-correlation between the two eyes' images, looking for small patches that are the most similar between the two images. There is evidence that cells in primary visual cortex are doing something very similar. This model captures many aspects of human depth perception, notably why we can see depth variation on much coarser scales than luminance variation. This suggests that the spatial resolution for depth perception is set in primary visual cortex. However, the model as currently implemented cannot explain why humans are as good at detecting sine-waves in depth as they are at detecting square-waves, a fact that we have previously raised as a challenge to the model. Here we show that if we introduce a size/disparity correlation, such that larger patches are used when searching for larger displacements of features between the two images, then simple models based on local cross-correlation can explain human performance for both sine- and square-wave depth corrugations, without needing to invoke more complicated disparity processing. This supports the proposal that spatial resolution for depth perception is set in primary visual cortex.

Complex cells in the cat striate cortex have multiple disparity detectors in the three-dimensional binocular receptive fields

Along the visual pathway, neurons generally become more specialized for signaling a limited subset of stimulus attributes and become more invariant to changes in the stimulus position within the receptive fields (RFs). One of the likely mechanisms underlying such invariance appears to be pooling of detectors located at different positions. Does such spatial pooling occur for disparity-selective neurons in primary visual cortex? To examine whether the three-dimensional (3D) binocular RFs are constructed by pooling detectors for binocular disparity, we investigated binocular interactions in the 3D space for neurons in the cat striate cortex. Approximately one-third of complex cells showed the spatial pooling of disparity detectors to a significant degree, whereas the majority of simple cells did not. The degree of spatial pooling of disparity detectors along the preferred orientation axis was generally larger than that along the axis orthogonal to the orientation axis. We then reconstructed 3D binocular RFs in their complete form and examined their structures. Disparity tuning curves were compared across positions along the orientation axis in the RFs. A small population of cells appeared to show a gradual shift of the preferred disparity along this axis, indicating that they can potentially signal inclination in the 3D space. However, the majority of cells exhibited a position-invariant disparity tuning. Finally, disparity tuning curves were examined for all oblique angles in addition to horizontal and vertical. Tunings were broadest along the orientation axis as the disparity energy model predicts.

Stereo correspondence is optimized for large viewing distances

Stereo '3D' vision depends on correctly matching up the differing images of objects seen by our two eyes. But vertical disparity between the retinal images changes with binocular eye posture, reflecting for example the different convergence angles required for different viewing distances. Thus, stereo correspondence must either dynamically adapt to take account of changes in viewing distance, or be hard-wired to perform best at one particular viewing distance. Here, using psychophysical experiments, we show for the first time that human stereo correspondence does not adapt to changes in physical viewing distance. We examine performance on a stereo correspondence task at a short viewing distance (30 cm) and show that performance is improved when we simulate the disparity pattern for viewing infinity, even though these disparities are impossible at the physical viewing distance. We estimate the vertical extent of the retinally fixed 'search zones' as < 0.6° at 14° eccentricity, suggesting that most V1 neurons must be tuned to near-zero vertical disparity. We also show that performance on our stereo task at 14° eccentricity is affected by the pattern of vertical disparity beyond 20° eccentricity, even though this is irrelevant to the task. Performance is best when vertical disparities within and beyond 20° eccentricity both indicate the same convergence angle (even if not the physical angle), than when the pattern of vertical disparity across the visual field is globally inconsistent with any single convergence angle. This novel effect of the periphery may indicate cooperative interactions between disparity-selective neurons activated by the same eye postures.

Vertical binocular disparity is encoded implicitly within a model neuronal population tuned to horizontal disparity and orientation

Primary visual cortex is often viewed as a "cyclopean retina", performing the initial encoding of binocular disparities between left and right images. Because the eyes are set apart horizontally in the head, binocular disparities are predominantly horizontal. Yet, especially in the visual periphery, a range of non-zero vertical disparities do occur and can influence perception. It has therefore been assumed that primary visual cortex must contain neurons tuned to a range of vertical disparities. Here, I show that this is not necessarily the case. Many disparity-selective neurons are most sensitive to changes in disparity orthogonal to their preferred orientation. That is, the disparity tuning surfaces, mapping their response to different two-dimensional (2D) disparities, are elongated along the cell's preferred orientation. Because of this, even if a neuron's optimal 2D disparity has zero vertical component, the neuron will still respond best to a non-zero vertical disparity when probed with a sub-optimal horizontal disparity. This property can be used to decode 2D disparity, even allowing for realistic levels of neuronal noise. Even if all V1 neurons at a particular retinotopic location are tuned to the expected vertical disparity there (for example, zero at the fovea), the brain could still decode the magnitude and sign of departures from that expected value. This provides an intriguing counter-example to the common wisdom that, in order for a neuronal population to encode a quantity, its members must be tuned to a range of values of that quantity. It demonstrates that populations of disparity-selective neurons encode much richer information than previously appreciated. It suggests a possible strategy for the brain to extract rarely-occurring stimulus values, while concentrating neuronal resources on the most commonly-occurring situations.

Latitude and longitude vertical disparities

The literature on vertical disparity is complicated by the fact that several different definitions of the term "vertical disparity" are in common use, often without a clear statement about which is intended or a widespread appreciation of the properties of the different definitions. Here, we examine two definitions of retinal vertical disparity: elevation-latitude and elevation-longitude disparities. Near the fixation point, these definitions become equivalent, but in general, they have quite different dependences on object distance and binocular eye posture, which have not previously been spelt out. We present analytical approximations for each type of vertical disparity, valid for more general conditions than previous derivations in the literature: we do not restrict ourselves to objects near the fixation point or near the plane of regard, and we allow for non-zero torsion, cyclovergence, and vertical misalignments of the eyes. We use these expressions to derive estimates of the latitude and longitude vertical disparities expected at each point in the visual field, averaged over all natural viewing. Finally, we present analytical expressions showing how binocular eye position-gaze direction, convergence, torsion, cyclovergence, and vertical misalignment-can be derived from the vertical disparity field and its derivatives at the fovea.

Human vergence eye movements to oblique disparity stimuli: evidence for an anisotropy favoring horizontal disparities

Binocular disparities applied to large-field patterns elicit vergence eye movements at ultra-short latencies. We used the electromagnetic search coil technique to record the horizontal and vertical positions of both eyes while subjects briefly viewed (150 ms) large patterns that were identical at the two eyes except for a difference in position (binocular disparity) that was varied in direction from trial to trial. For accurate alignment with the stimuli, the horizontal and vertical disparity vergence responses (HDVRs, VDVRs) should vary as the sine and cosine, respectively, of the direction of the disparity stimulus vector. In a first experiment, using random-dots patterns (RDs) with a binocular disparity of 0.2 degrees , this was indeed the case. In a second experiment, using 1-D sine-wave gratings with a binocular phase difference (disparity) of 1/4-wavelength, it was not the case: HDVRs were maximal when the grating was vertical and showed little decrement until the grating was oriented more than approximately 65 degrees away from vertical, whereas VDVRs were maximal when the grating was horizontal and began to decrement roughly linearly when the grating was oriented away from the horizontal. We attribute these complex directional dependencies with gratings to the aperture problem, and the HDVR data strongly resemble the stereothresholds for 1-D gratings, which are minimal when the gratings are vertical and remain constant for orientations up to approximately 80 degrees away from the vertical when expressed as spatial phase disparities [Morgan, M. J., & Castet, E. (1997). The aperture problem in stereopsis. Vision Research, 37, 2737-2744.]. To explain this constancy of stereothresholds, Morgan and Castet (1997) postulated detectors sensitive to the phase disparity of the gratings seen by the two eyes (rather than their linear separation along some fixed axis, such as the horizontal). However, because (1) our VDVR data with gratings did not show this constancy and (2) the available evidence strongly suggests that there are no major differences in the disparity detectors mediating the initial HDVR and VDVR, we sought an alternative explanation for our data. We show that the dependence of the initial HDVR and VDVR on grating orientation can be successfully modeled by a bias in the number and/or efficacy of the detectors that favors horizontal disparities.

Normalization Enables Robust Validation of Disparity Estimates from Neural Populations

Binocular fusion takes place over a limited region smaller than one degree of visual angle (Panum's fusional area), which is on the order of the range of preferred disparities measured in populations of disparity-tuned neurons in the visual cortex. However, the actual range of binocular disparities encountered in natural scenes extends over tens of degrees. This discrepancy suggests that there must be a mechanism for detecting whether the stimulus disparity is inside or outside the range of the preferred disparities in the population. Here, we compare the efficacy of several features derived from the population responses of phase-tuned disparity energy neurons in differentiating between in-range and out-of-range disparities. Interestingly, some features that might be appealing at first glance, such as the average activation across the population and the difference between the peak and average responses, actually perform poorly. On the other hand, normalizing the difference between the peak and average responses results in a reliable indicator. Using a probabilistic model of the population responses, we improve classification accuracy by combining multiple features. A decision rule that combines the normalized peak to average difference and the peak location significantly improves performance over decision rules based on either measure in isolation. In addition, classifiers using normalized difference are also robust to mismatch between the image statistics assumed by the model and the actual image statistics.

Cyclopean geometry of binocular vision

The geometry of binocular projection is analyzed in relation to the primate visual system. An oculomotor parameterization that includes the classical vergence and version angles is defined. It is shown that the epipolar geometry of the system is constrained by binocular coordination of the eyes. A local model of the scene is adopted in which depth is measured relative to a plane containing the fixation point. These constructions lead to an explicit parameterization of the binocular disparity field involving the gaze angles as well as the scene structure. The representation of visual direction and depth is discussed with reference to the relevant psychophysical and neurophysiological literature.

Binocular energy responses to natural images

The binocular energy model provides a good description of the first stages of cortical binocular processing. Three important determinants of the responses of neurons under this model are the disparity of a stimulus, its spatial variation in disparity and its second-order luminance statistics. The influence of the latter two factors on the disparity tuning of the energy model were investigated. While each can have a significant effect on the energy response, neither presents a significant challenge when one considers the range of variation expected in natural images. The response of the energy model to natural binocular images was also investigated. The strongest responses were found for model neurons tuned to small disparities. This trend was more evident for vertical than for horizontal disparity, and flattened rapidly as image eccentricity increased. These results are predicted on the basis of simple geometrical considerations, and are reflected in both physiological and psychophysical measures of the disparity tuning of the visual system.

Adaptation to natural binocular disparities in primate V1 explained by a generalized energy model

Sensory processing in the brain is thought to have evolved to encode naturally occurring stimuli efficiently. We report an adaptation in binocular cortical neurons that reflects the tight constraints imposed by the geometry of 3D vision. We show that the widely used binocular energy model predicts that neurons dedicate part of their dynamic range to impossible combinations of left and right images. Approximately 42% of the neurons we record from V1 of awake monkeys behave in this way (a powerful confirmation of the model), while about 58% deviate from the model in a manner that concentrates more of their dynamic range on stimuli that obey the constraints of binocular geometry. We propose a simple extension of the energy model, using multiple subunits, that explains the adaptation we observe, as well as other properties of binocular neurons that have been hard to account for, such as the response to anti-correlated stereograms.

Perceptual learning in monocular pattern masking: experiments and explanations by the twin summation gain control model of contrast processing

We investigated practice effects on contrast thresholds for target patterns. Results showed that practice decreased contrast thresholds when targets were presented on maskers. Thresholds tended to decrease more at the higher end of the masker contrast range. At least partially, learning transferred to stimuli of the untrained phase. We simulated changes in threshold versus contrast functions using a contrast-processing model and then fit the model to pre- and posttraining data. The simulation results and model fit suggest that learning in pattern masking can be accounted for by changes in nonlinear transducer functions for divisive inhibitory signals.

Effects of perceptual learning on local stereopsis and neuronal responses of V1 and V2 in prism-reared monkeys

Visual performance improves with practice (perceptual learning). In this study, we sought to determine whether or not adult monkeys reared with early abnormal visual experience improve their stereoacuity by extensive psychophysical training and testing, and if so, whether alterations of neuronal responses in the primary visual cortex (V1) and/or visual area 2 (V2) are involved in such improvement. Strabismus was optically simulated in five macaque monkeys using a prism-rearing procedure between 4 and 14 wk of age. Around 2 yr of age, three of the prism-reared monkeys ("trained" monkeys) were tested for their spatial contrast sensitivity and stereoacuity. Two other prism-reared monkeys received no training or testing ("untrained" monkeys). Microelectrode experiments were conducted around 4 yr of age. All three prism-reared trained monkeys showed improvement in stereoacuity by a factor of 7 or better. However, final stereothresholds were still approximately 10-20 times worse than those in normal monkeys. In V1, disparity sensitivity was drastically reduced in both the trained and untrained prism-reared monkeys and behavioral training had no obvious effect. In V2, the disparity sensitivity in the trained monkeys was better by a factor of approximately 2.0 compared with that in the untrained monkeys. These data suggest that the observed improvement in stereoacuity of the trained prism-reared monkeys may have resulted from better retention of disparity sensitivity in V2 and/or from "learning" by upstream neurons to more efficiently attend to residual local disparity information in V1 and V2.

Neural bases of stereopsis across visual field of the alert macaque monkey

Left and right retinal images of an object seen by the 2 eyes can occupy slightly disparate horizontal and/or vertical locations. The role of horizontal disparity (HD) in stereoscopic vision is well established, but the functional contribution of vertical disparity (VD) remains unclear. Various psychophysical studies have shown that HD and VD are used differently by the visual system depending on their location in the visual field, whether near the center of gaze or more peripheral. We show this horizontal/vertical distinction at the cellular level in monkey primary visual cortex (area V1). The range of VD encoding is reduced in central but not in the peripheral representation of the visual field. Moreover, neurons respond selectively to particular combinations of both types of disparities depending on the coded orientation as predicted by the disparity energy model. The preferred orientations of neurons near the fovea present a vertical bias that is well suited for stereopsis based on HD selectivity alone. In the periphery, instead, preferred orientations are radially biased, which allows a peripheral detector to convey the same depth signal based on either HD or VD. Such an organization has functional implications in both the perceptual and oculomotor domains.

A stereoscopic look at visual cortex

Three recent studies offer new insights into the way visual cortex handles binocular disparity signals. Two of these studies recorded from single neurons in two different visual areas of the monkey brain, one (V5/MT) in dorsal and one (V4) in ventral cortex. While V5/MT neurons respond similarly to neurons in primary visual cortex (V1), V4 neurons appear to reflect a more advanced stage in the analysis of retinal disparity, closer to the perceptual experience of stereoscopic depth. Both studies are consistent with a third study using fMRI to address similar questions in humans. Together with previous evidence, these results suggest a new framework for understanding stereoscopic processing based on the separation between ventral and dorsal streams in visual cortex.

Early computational processing in binocular vision and depth perception

Stereoscopic depth perception is a fascinating ability in its own right and also a useful model of perception. In recent years, considerable progress has been made in understanding the early cortical circuitry underlying this ability. Inputs from left and right eyes are first combined in primary visual cortex (V1), where many cells are tuned for binocular disparity. Although the observation of disparity tuning in V1, combined with psychophysical evidence that stereopsis must occur early in visual processing, led to initial suggestions that V1 was the neural correlate of stereoscopic depth perception, more recent work indicates that this must occur in higher visual areas. The firing of cells in V1 appears to depend relatively simply on the visual stimuli within local receptive fields in each retina, whereas the perception of depth reflects global properties of the stimulus. However, V1 neurons appear to be specialized in a number of respects to encode ecologically relevant binocular disparities. This suggests that they carry out essential pre-processing underlying stereoscopic depth perception in higher areas. This article reviews recent progress in developing accurate models of the computations carried out by these neurons. We seem close to achieving a mathematical description of the initial stages of the brain's stereo algorithm. This is important in itself--for instance, it may enable improved stereopsis in computer vision--and paves the way for a full understanding of how depth perception arises.