Extracting 3D structure from disparity

Depth constancy and the absolute vergence anomaly

Binocular disparity provides information about the depth structure of objects and surfaces in our environment. Since disparity depends on the distance to objects as well as the depth separation of points, information about distance is required to estimate depth from disparity. Our perception of size and shape is biased, such that far objects appear too small and flattened in depth, and near objects too big and stretched in depth. The current study assessed the extent to which the failure of depth constancy can be accounted for by the uncertainty of distance information provided by vergence. We measured individual differences in vergence noise using a nonius line task, and the degree of depth constancy using a task in which observers judged the magnitude of a depth interval relative to the vertical distance between two targets in the image plane. We found no correlation between the two measures, and show that depth constancy was much poorer than would be expected from vergence noise measured in this way. This limited ability to take account of vergence in the perception of depth is, however, consistent with our poor sensitivity to absolute disparity differences. This absolute disparity anomaly thus also applies to our poor ability to make use of vergence information for absolute distance judgements.

Stereoscopic slant contrast revisited

The perceived slant of a stereoscopic surface is altered by the presence of a surrounding surface, a phenomenon termed stereo slant contrast. Previous studies have shown that a slanted surround causes a fronto-parallel surface to appear slanted in the opposite direction, an instance of "bidirectional" contrast. A few studies have examined slant contrast using slanted as opposed to fronto-parallel test surfaces, and these also have shown slant contrast. Here, we use a matching method to examine slant contrast over a wide range of combinations of surround and test slants, one aim being to determine whether stereo slant contrast transfers across opposite directions of test and surround slant. We also examine the effect of the test on the perceived slant of the surround. Test slant contrast was found to be bidirectional in virtually all test-surround combinations and transferred across opposite test and surround slants, with little or no decline in magnitude as the test-surround slant difference approached the limit. There was a weak bidirectional effect of the test slant on the perceived slant of the surround. We consider how our results might be explained by four mechanisms: (a) normalization of stereo slant to vertical; (b) divisive normalization of stereo slant channels in a manner analogous to the tilt illusion; (c) interactions between center and surround disparity-gradient detectors; and (d) uncertainty in slant estimation. We conclude that the third of these (interactions between center and surround disparity-gradient detectors) is the most likely cause of stereo slant contrast.

A focus on the multiple interfaces between action and perception and their neural correlates

Successful behaviour relies on the appropriate interplay between action and perception. The well-established dorsal and ventral stream theories depicted two distinct functional pathways for the processes of action and perception, respectively. In physiological conditions, the two pathways closely cooperate in order to produce successful adaptive behaviour. As the coupling between perception and action exists, this requires an interface that is responsible for a common reading of the two functions. Several studies have proposed different types of perception and action interfaces, suggesting their role in the creation of the shared interaction channel. In the present review, we describe three possible perception and action interfaces: i) the motor code, including common coding approaches, ii) attention, and iii) object affordance; we highlight their potential neural correlates. From this overview, a recurrent neural substrate that underlies all these interface functions appears to be crucial: the parieto-frontal circuit. This network is involved in the mirror mechanism which underlies the perception and action interfaces identified as common coding and motor code theories. The same network is also involved in the spotlight of attention and in the encoding of potential action towards objects; these are manifested in the perception and action interfaces for common attention and object affordance, respectively. Within this framework, most studies were dedicated to the description of the role of the inferior parietal lobule; growing evidence, however, suggests that the superior parietal lobule also plays a crucial role in the interplay between action and perception. The present review proposes a novel model that is inclusive of the superior parietal regions and their relative contribution to the different action and perception interfaces.

Distinct Neural Components of Visually Guided Grasping during Planning and Execution

Selecting suitable grasps on three-dimensional objects is a challenging visuomotor computation, which involves combining information about an object (e.g., its shape, size, and mass) with information about the actor's body (e.g., the optimal grasp aperture and hand posture for comfortable manipulation). Here, we used functional magnetic resonance imaging to investigate brain networks associated with these distinct aspects during grasp planning and execution. Human participants of either sex viewed and then executed preselected grasps on L-shaped objects made of wood and/or brass. By leveraging a computational approach that accurately predicts human grasp locations, we selected grasp points that disentangled the role of multiple grasp-relevant factors, that is, grasp axis, grasp size, and object mass. Representational Similarity Analysis revealed that grasp axis was encoded along dorsal-stream regions during grasp planning. Grasp size was first encoded in ventral stream areas during grasp planning then in premotor regions during grasp execution. Object mass was encoded in ventral stream and (pre)motor regions only during grasp execution. Premotor regions further encoded visual predictions of grasp comfort, whereas the ventral stream encoded grasp comfort during execution, suggesting its involvement in haptic evaluation. These shifts in neural representations thus capture the sensorimotor transformations that allow humans to grasp objects.SIGNIFICANCE STATEMENT Grasping requires integrating object properties with constraints on hand and arm postures. Using a computational approach that accurately predicts human grasp locations by combining such constraints, we selected grasps on objects that disentangled the relative contributions of object mass, grasp size, and grasp axis during grasp planning and execution in a neuroimaging study. Our findings reveal a greater role of dorsal-stream visuomotor areas during grasp planning, and, surprisingly, increasing ventral stream engagement during execution. We propose that during planning, visuomotor representations initially encode grasp axis and size. Perceptual representations of object material properties become more relevant instead as the hand approaches the object and motor programs are refined with estimates of the grip forces required to successfully lift the object.

Digital Therapy for Visual Acuity and Binocular Function in Children with Anisometropic Amblyopia

Amblyopia affects development of children's monocular vision and binocular function and becomes a largely intractable problem with increasing aging. This study is to investigate the binocular function and evaluate efficacy of digital therapy in children 8-13 years of age with anisometropic amblyopia. The patients in the digital therapy group performed the training with the digital amblyopia therapeutic software. The visual acuity and binocular function (perceptual eye position [PEP], suppression, and stereopsis) were examined at the first visit and 3-month post-treatment. Twenty-three cases in the control group and 25 cases in the digital therapy group were enrolled. The results revealed that 3-month digital therapy can effectively improve corrected distance visual acuity (CDVA) and improve the binocular function, including PEP, suppression, and second-order stereopsis in children with anisometropic amblyopia, 8-13 years of age. Digital therapy for amblyopia can effectively improve monocular CDVA of amblyopic eyes and binocular function in older children with anisometropic amblyopia.

Cost-Effective Full-Color 3D Dental Imaging Based on Close-Range Photogrammetry

Dental imaging plays a crucial role in clinical dental practice. Conventional 2D dental imaging serves general-purpose tasks, such as patient documentation, while high-precision 3D dental scanning is tailored for specialized procedures, such as orthodontics and implant surgeries. In this study, we aimed to develop a cost-effective 3D imaging technique that could bridge the gap between conventional dental photography and high-precision 3D dental scanning, with the goal of improving patient dental care. We developed a 3D imaging technique based on close-range photogrammetry and termed it close-range photogrammetry-based dental imaging (CPDI). We evaluated this technique on both in vitro dental models and in vivo teeth. For dental models, we conducted a parametric study to examine the effects of the depth of field and specular reflection on reconstruction quality. We showed that the optimal results were achieved with an f/5.6 lens and without a circular polarizer for reflection suppression. This configuration generated 3D scans with 57.7 ± 3.2% and 82.4 ± 2.7% of reconstructed points falling within ±0.1 mm and ±0.2 mm error margins, respectively. With such accuracy, these 3D dental models can faithfully represent dental morphology and features. During in vivo imaging, we were able to reconstruct high-quality 3D models of the anterior arch, further demonstrating its clinical relevance. The reconstructed models carry both 3D shapes and detail full-color surface textures, which positions CPDI as a versatile imaging tool in different areas of clinical dental care.

Spikiness and animacy as potential organizing principles of human ventral visual cortex

Considerable research has been devoted to understanding the fundamental organizing principles of the ventral visual pathway. A recent study revealed a series of 3-4 topographical maps arranged along the macaque inferotemporal (IT) cortex. The maps articulated a two-dimensional space based on the spikiness and animacy of visual objects, with "inanimate-spiky" and "inanimate-stubby" regions of the maps constituting two previously unidentified cortical networks. The goal of our study was to determine whether a similar functional organization might exist in human IT. To address this question, we presented the same object stimuli and images from "classic" object categories (bodies, faces, houses) to humans while recording fMRI activity at 7 Tesla. Contrasts designed to reveal the spikiness-animacy object space evoked extensive significant activation across human IT. However, unlike the macaque, we did not observe a clear sequence of complete maps, and selectivity for the spikiness-animacy space was deeply and mutually entangled with category-selectivity. Instead, we observed multiple new stimulus preferences in category-selective regions, including functional sub-structure related to object spikiness in scene-selective cortex. Taken together, these findings highlight spikiness as a promising organizing principle of human IT and provide new insights into the role of category-selective regions in visual object processing.

Initial postoperative plasticity as a predictor of mid-term stereoacuity outcome after surgery for intermittent exotropia

BACKGROUND

Intermittent exotropia (IXT) would cause different degrees of damage to stereopsis. We aimed to introduce a visual perception plasticity score (VPPS) that reflects initial postoperative plasticity and evaluate its effectiveness in predicting the mid-term surgical outcome in IXT patients.

METHODS

A total of 149 patients with intermittent exotropia who underwent surgery in November 2018 and October 2019 were recruited. All subjects underwent detailed ocular examinations before and after surgery. VPPS were calculated based on visual perception examination system at one week postoperatively. Demographic, angle of deviation and stereopsis were collected and analyzed with regard to the VPPSs preoperatively and at one week, one month, three months, six months postoperatively. Predictive performances of VPPS were assessed using receiver operating characteristic (ROC) curves, the area under the curve (AUC) and cut-offs were obtained.

RESULTS

Of the 149 patients, the average deviation was 43^Δ at distance and 46^Δ at near. The average rate of normal stereopsis before surgery was 22.81% at distance and 29.53% at near. Higher VPPS was associated with preoperative better near stereoacuity (r = 0.362, p = 0.000), less angle of deviation at distance (r=-0.164, p = 0.046), and better near (r = 0.400, p = 0.000) and distant stereoacuity (r = 0.321, p = 0.000) during the early postoperative period (1 week). The areas under the curves suggested that VPPS could be an effective predictor of sensory outcome(AUC>0.6). Cut-off values of 50 and 80 were calculated for VPPS using ROC curve analysis.

CONCLUSION

Higher VPPSs were associated with a greater possibility of stereopsis improvement in patients with IXT. VPPS is a potentially promising indicator to predict the mid-term surgical outcome of intermittent exotropia.

EEG response in humans for frequency-tagged anticorrelated random-dot stereograms: Increased coherency and alpha oscillations

In humans, the presence of a neural mechanism triggered by anticorrelated random-dot stereograms have been theorized based on animal models from invasive studies, but have not been experimentally verified with the use of electroencephalography. In this study, we employed a phase-consistent, temporally modulated alternating depth stereogram stimulus, where we created anticorrelation by inverting the contrast between the eyes. We recorded the electrical response of the resulting brain oscillations of our four participants using EEG in both the correlated and anticorrelated conditions and whether they perceived depth movement. Our analysis found that the correlated stereograms elicited a strong coherency at the even harmonics of the depth alternation, and the anticorrelated stimulus created lower coherency peaks at the first harmonic of the depth alternation, even when participants did not report the depth movement to be visible. While both conditions created a diminishment of spectral power in the beta band, we found that the anticorrelated condition created increased spectral power in the alpha band. We experimentally verified the presence of a neural mechanism triggered by anticorrelated random-dot stereograms in the human brain with our coherency analysis and that it would not have been detected with the conventional spectral analysis due to the weakness of the response. We hypothesize that the decreased beta oscillations are related to either visual discomfort and visual attention to our stimulus, and that the increased alpha oscillations in the anticorrelated condition is a response to the incorrect depth information created by the stereogram.

A Systematic Review of the Effects of Second-Eye Cataract Surgery on Motor Function

Cataract removal surgery is one of the most commonly performed surgical procedure in developed countries. The financial and staff resource cost that first-eye cataract surgery incurs, leads to restricted access to second-eye cataract surgery (SES) in some areas, including the United Kingdom. These restrictions have been imposed despite a lack of knowledge about the impact of not performing SES on visuo-motor function. To this end, a systematic literature review was carried out, with the aim of synthesising our present understanding of the effects of SES on motor function. Key terms were searched across four databases, PsycINFO, Medline, Web of Science, and CINAHL. Of the screened studies (K = 499) 13 met the eligibility criteria. The homogeneity between participants, study-design and outcome measures across these studies was not sufficient for meta-analyses and a narrative synthesis was carried out. The evidence from objective sources indicates a positive effect of SES on both mobility and fall rates, however, when considering self-report measures, the reduction in falls associated with SES becomes negligible. The evidence for any positive effect of SES on driving is also mixed, whereby SES was associated with improvements in simulated driving performance but was not associated with changes in driving behaviours measured through in vehicle monitoring. Self-report measures of driving performance also returned inconsistent results. Whilst SES appears to be associated with a general trend towards improved motor function, more evidence is needed to reach any firm conclusions and to best advise policy regarding access to SES in an ageing population. Systematic Review Registration: https://osf.io/7hne6/, identifier INPLASY2020100042.

Neural responses to reflection symmetry for shapes defined by binocular disparity, and for shapes perceived as regions of background

Human perception of symmetry is associated with activation in an extended network of extrastriate visual areas. This activation generates an ERP called the Sustained Posterior Negativity (SPN). In most studies so far, the stimuli have been defined by luminance. We tested whether the SPN is present when stimuli are defined by stereoscopic disparity using random dot stereograms (RDS). In Experiment 1, we compared the SPN signal for contours specified by binocular disparity and contours specified by monocular cues. The SPN was equivalent, suggesting that the type of contour does not alter the SPN signal. In Experiment 2 we exploited the unique property of RDS to provide unambiguous figure-ground arrangements. Psychophysical work has shown that symmetry is more easily detected when it is a property of a single object (i.e., within a figure), compared to a property of a gap between two objects (i.e., the ground). Therefore, the target regions in this experiment could either be foreground or background. The SPN onset was delayed when the symmetry was in a ground region. This may be because object formation interferes with the processing of shape information in the ground region.

Exploring the effects of degraded vision on sensorimotor performance

PURPOSE

Many people experience unilateral degraded vision, usually owing to a developmental or age-related disorder. There are unresolved questions regarding the extent to which such unilateral visual deficits impact on sensorimotor performance; an important issue as sensorimotor limitations can constrain quality of life by restricting 'activities of daily living'. Examination of the relationship between visual deficit and sensorimotor performance is essential for determining the functional implications of ophthalmic conditions. This study attempts to explore the effect of unilaterally degraded vision on sensorimotor performance.

METHODS

In Experiment 1 we simulated visual deficits in 30 participants using unilateral and bilateral Bangerter filters to explore whether motor performance was affected in water pouring, peg placing, and aiming tasks. Experiment 2 (n = 74) tested the hypothesis that kinematic measures are associated with visuomotor deficits by measuring the impact of small visual sensitivity decrements created by monocular viewing on sensorimotor interactions with targets presented on a planar surface in aiming, tracking and steering tasks.

RESULTS

In Experiment 1, the filters caused decreased task performance-confirming that unilateral (and bilateral) visual loss has functional implications. In Experiment 2, kinematic measures were affected by monocular viewing in two of three tasks requiring rapid online visual feedback (aiming and steering).

CONCLUSIONS

Unilateral visual loss has a measurable impact on sensorimotor performance. The benefits of binocular vision may be particularly important for some groups (e.g. older adults) where an inability to complete sensorimotor tasks may necessitate assisted living. There is an urgent need to develop rigorous kinematic approaches to the quantification of the functional impact of unilaterally degraded vision and of the benefits associated with treatments for unilateral ophthalmic conditions to enable informed decisions around treatment.

Detection of Static and Dynamic Stereopsis after Femtosecond Laser Small Incision Lenticule Extraction for High Myopia

Purpose

The purpose of this study is to test binocular visual function after femtosecond laser small incision lenticule extraction (SMILE) for high myopia. The traditional Titmus stereotest and dynamic stereotest based on the visual perception biological model were used for comparative analysis.

Methods

A total of 43 patients were enrolled in this prospective study. At Week 1, Month 1, and Month 3 after surgery, the Titmus stereotest and dynamic stereotest generated by MATLAB were conducted. Dynamic stereopsis consists of randomly flickering Gabor spots and is divided into two models of high energy and low energy according to flicker frequency.

Results

The preoperative manifest refraction spherical equivalent was −7.21 ± 0.70 D. The preoperative anisometropia was 0.52 ± 0.54D. The quartiles of static stereoacuity in preoperation and 3 follow-ups were as follows: 50.00 (25.00, 100.00) in preoperation, 63.00 (40.00, 63.00) at Week 1, 40.00 (32.00, 63.00) at Month 1, and 40.00 (25.00, 50.00) at Month 3. Static stereopsis improved at Month 1 and Month 3 compared with preoperation and Week 1 (P < 0.05). There were statistically significant differences in high energy dynamic stereopsis at Week 1 and Month 1 compared to preoperation (P < 0.05). In addition, significant differences in low energy dynamic stereopsis were detected between Month 1 and preoperation and also at Month 3 compared to Month 1 (P < 0.05).

Conclusion

Most high myopia patients have a dynamic stereopsis deficiency before refractive correction. SMILE surgery can improve both static and dynamic stereopsis early in the postoperation period. However, in the long term, there is no significant difference or even a decrease in dynamic stereopsis.

Unique Neural Activity Patterns Among Lower Order Cortices and Shared Patterns Among Higher Order Cortices During Processing of Similar Shapes With Different Stimulus Types

We investigated the neural mechanism of the processing of three-dimensional (3D) shapes defined by disparity and perspective. We measured blood oxygenation level-dependent signals as participants viewed and classified 3D images of convex-concave shapes. According to the cue (disparity or perspective) and element type (random dots or black and white dotted lines), three types of stimuli were used: random dot stereogram, black and white dotted lines with perspective, and black and white dotted lines with binocular disparity. The blood oxygenation level-dependent images were then classified by multivoxel pattern analysis. To identify areas selective to shape, we assessed convex-concave classification accuracy with classifiers trained and tested using signals evoked by the same stimulus type (same cue and element type). To identify cortical regions with similar neural activity patterns regardless of stimulus type, we assessed the convex-concave classification accuracy of transfer classification in which classifiers were trained and tested using different stimulus types (different cues or element types). Classification accuracy using the same stimulus type was high in the early visual areas and subregions of the intraparietal sulcus (IPS), whereas transfer classification accuracy was high in the dorsal subregions of the IPS. These results indicate that the early visual areas process the specific features of stimuli, whereas the IPS regions perform more generalized processing of 3D shapes, independent of a specific stimulus type.

Specialized contributions of mid-tier stages of dorsal and ventral pathways to stereoscopic processing in macaque

The division of labor between the dorsal and ventral visual pathways has been well studied, but not often with direct comparison at the single-neuron resolution with matched stimuli. Here we directly compared how single neurons in MT and V4, mid-tier areas of the two pathways, process binocular disparity, a powerful cue for 3D perception and actions. We found that MT neurons transmitted disparity signals more quickly and robustly, whereas V4 or its upstream neurons transformed the signals into sophisticated representations more prominently. Therefore, signaling speed and robustness were traded for transformation between the dorsal and ventral pathways. The key factor in this tradeoff was disparity-tuning shape: V4 neurons had more even-symmetric tuning than MT neurons. Moreover, the tuning symmetry predicted the degree of signal transformation across neurons similarly within each area, implying a general role of tuning symmetry in the stereoscopic processing by the two pathways.

A unified model for binocular fusion and depth perception

We describe a new unified model to explain both binocular fusion and depth perception, over a broad range of depths. At each location, the model consists of an array of paired spatial frequency filters, with different relative horizontal shifts (position disparity) and interocular phase disparities of 0, 90, ±180, or -90°. The paired filters with different spatial profiles (non-zero phase disparity) compute interocular misalignment and provide phase-disparity energy (binocular fusion energy) to drive selection of the appropriate filters along the position disparity space until the misalignment is eliminated and sensory fusion is achieved locally. The paired filters with identical spatial profiles (0 phase disparity) compute the position-disparity energy. After sensory fusion, the combination of position and possible residual phase disparity energies is calculated for binocular depth perception. Binocular fusion occurs at multiple scales following a coarse-to-fine process. At a given location, the apparent depth is the weighted sum of fusion shifts combined with residual phase disparity in all spatial-frequency channels, and the weights depend on stimulus spatial frequency and stimulus contrast. To test the theory, we measured disparity minimum and maximum thresholds (Dmin and Dmax) at three spatial frequencies and with different intraocular contrast levels. The stimuli were Random-Gabor-Patch (RGP) stereograms consisting of Gabor patches with random positions and phases, but with a fixed spatial frequency. The two eyes viewed identical arrays of patches except that one eye's array could be shifted horizontally and could differ in contrast. Our experiments and modeling reveal two contrast normalization mechanisms: (1) Energy Normalization (EN): Binocular energy is normalized with monocular energy after the site of binocular combination. This predicts constant Dmin thresholds when varying stimulus contrast in the two eyes; (2) DSKL model Interocular interactions: Monocular contrasts are normalized before the binocular combination site through interocular contrast gain-control and gain-enhancement mechanisms. This predicts contrast dependent Dmax thresholds. We tested a range of models and found that a model consisting of a second-order pathway with DSKL interocular interactions and a first-order pathway with EN at each spatial-frequency band can account for both the Dmin and Dmax data very well. Simulations show that the model makes reasonable predictions of suprathreshold depth perception.

What Does Dorsal Cortex Contribute to Perception?

According to the influential "Two Visual Pathways" hypothesis, the cortical visual system is segregated into two pathways, with the ventral, occipitotemporal pathway subserving object perception, and the dorsal, occipitoparietal pathway subserving the visuomotor control of action. However, growing evidence suggests that the dorsal pathway also plays a functional role in object perception. In the current article, we present evidence that the dorsal pathway contributes uniquely to the perception of a range of visuospatial attributes that are not redundant with representations in ventral cortex. We describe how dorsal cortex is recruited automatically during perception, even when no explicit visuomotor response is required. Importantly, we propose that dorsal cortex may selectively process visual attributes that can inform the perception of potential actions on objects and environments, and we consider plausible developmental and cognitive mechanisms that might give rise to these representations. As such, we consider whether naturalistic stimuli, such as real-world solid objects, might engage dorsal cortex more so than simplified or artificial stimuli such as images that do not afford action, and how the use of suboptimal stimuli might limit our understanding of the functional contribution of dorsal cortex to visual perception.

Generalized Representation of Stereoscopic Surface Shape and Orientation in the Human Visual Cortex

The brain's ability to extract three-dimensional (3D) shape and orientation information from viewed objects is vital in daily life. Stereoscopic 3D surface perception relies on binocular disparity. Neurons selective to binocular disparity are widely distributed among visual areas, but the manner in these areas are involved in stereoscopic 3D surface representation is unclear. To address this, participants were instructed to observe random dot stereograms (RDS) depicting convex and concave curved surfaces and the blood oxygenation level-dependent (BOLD) signal of visual cortices was recorded. Two surface types were: (i) horizontally positioned surfaces defined by shear disparity; and (ii) vertically positioned surfaces defined by compression disparity. The surfaces were presented at different depth positions per trial. Functional magnetic resonance imaging (fMRI) data were classified from early visual areas to higher visual areas. We determined whether cortical areas were selective to shape and orientation by assessing same-type stimuli classification accuracies based on multi-voxel activity patterns per area. To identify whether some areas were related to a more generalized sign of curvature or orientation representation, transfer classification was used by training classifiers on one dataset type and testing classifiers on another type. Same-type stimuli classification results showed that most selected visual areas were selective to shape and all were selective to the orientation of disparity-defined 3D surfaces. Transfer classification results showed that in the dorsal visual area V3A, classification accuracies for the discriminate sign of surface curvature were higher than the baseline of statistical significance for all types of classifications, demonstrating that V3A is related to generalized shape representation. Classification accuracies for discriminating horizontal-vertical surfaces in higher dorsal areas V3A and V7 and ventral area lateral occipital complex (LOC) as well as in some areas of intraparietal sulcus (IPS) were higher than the baseline of statistical significance, indicating their relation to the generalized representation of 3D surface orientation.

Areal differences in depth cue integration between monkey and human

Electrophysiological evidence suggested primarily the involvement of the middle temporal (MT) area in depth cue integration in macaques, as opposed to human imaging data pinpointing area V3B/kinetic occipital area (V3B/KO). To clarify this conundrum, we decoded monkey functional MRI (fMRI) responses evoked by stimuli signaling near or far depths defined by binocular disparity, relative motion, and their combination, and we compared results with those from an identical experiment previously performed in humans. Responses in macaque area MT are more discriminable when two cues concurrently signal depth, and information provided by one cue is diagnostic of depth indicated by the other. This suggests that monkey area MT computes fusion of disparity and motion depth signals, exactly as shown for human area V3B/KO. Hence, these data reconcile previously reported discrepancies between depth processing in human and monkey by showing the involvement of the dorsal stream in depth cue integration using the same technique, despite the engagement of different regions.

In everyday life, we interact with a three-dimensional world that we perceive via our two-dimensional retinas. Our brain can reconstruct the third dimension from these flat retinal images using multiple sources of visual information, or cues. The horizontal displacement of the two retinal images, known as binocular disparity, and the relative motion between different objects are two important depth cues. However, to make the most of the information provided by each cue, our brains must efficiently integrate across them. To examine this process, we used neuroimaging in monkeys to record brain responses evoked by stimuli signaling depths defined by either binocular disparity or relative motion in isolation, and also when the two cues are combined congruently or incongruently. We found that cortical area MT in monkeys is involved in the fusion of these two particular depth cues, in contrast to previous human imaging data that pinpoint a more posterior cortical area, V3B/KO. Our findings support the existence of depth cue integration mechanisms in primates; however, this fusion appears to be computed in slightly different areas in humans and monkeys.

Disparity level identification using the voxel-wise Gabor model of fMRI data

Perceiving disparities is the intuitive basis for our understanding of the physical world. Although many electrophysiology studies have revealed the disparity-tuning characteristics of the neurons in the visual areas of the macaque brain, neuron population responses to disparity processing have seldom been investigated. Many disparity studies using functional magnetic resonance imaging (fMRI) have revealed the disparity-selective visual areas in the human brain. However, it is unclear how to characterize neuron population disparity-tuning responses using fMRI technique. In the present study, we constructed three voxel-wise encoding Gabor models to predict the voxel responses to novel disparity levels and used a decoding method to identify the new disparity levels from population responses in the cortex. Among the three encoding models, the fine-coarse model (FCM) that used fine/coarse disparities to fit the voxel responses to disparities outperformed the single model and uncrossed-crossed model. Moreover, the FCM demonstrated high accuracy in predicting voxel responses in V3A complex and high accuracy in identifying novel disparities from responses in V3A complex. Our results suggest that the FCM can better characterize the voxel responses to disparities than the other two models and V3A complex is a critical visual area for representing disparity information.

Microstructural properties of the vertical occipital fasciculus explain the variability in human stereoacuity

Seeing in the three-dimensional world—stereopsis—is an innate human ability, but it varies substantially among individuals. The neurobiological basis of this variability is not understood. We combined diffusion and quantitative MRI imaging with a psychophysical measurements, and found that variability in stereoacuity is associated with microstructural differences in the right vertical occipital fasciculus, a white matter tract connecting dorsal and ventral visual cortex. This result suggests that the microstructure of the pathways that support information transmission across dorsal and ventral visual areas plays an important role human stereopsis.

Stereopsis is a fundamental visual function that has been studied extensively. However, it is not clear why depth discrimination (stereoacuity) varies more significantly among people than other modalities. Previous studies have reported the involvement of both dorsal and ventral visual areas in stereopsis, implying that not only neural computations in cortical areas but also the anatomical properties of white matter tracts connecting those areas can impact stereopsis. Here, we studied how human stereoacuity relates to white matter properties by combining psychophysics, diffusion MRI (dMRI), and quantitative MRI (qMRI). We performed a psychophysical experiment to measure stereoacuity and, in the same participants, we analyzed the microstructural properties of visual white matter tracts on the basis of two independent measurements, dMRI (fractional anisotropy, FA) and qMRI (macromolecular tissue volume; MTV). Microstructural properties along the right vertical occipital fasciculus (VOF), a major tract connecting dorsal and ventral visual areas, were highly correlated with measures of stereoacuity. This result was consistent for both FA and MTV, suggesting that the behavioral–structural relationship reflects differences in neural tissue density, rather than differences in the morphological configuration of fibers. fMRI confirmed that binocular disparity stimuli activated the dorsal and ventral visual regions near VOF endpoints. No other occipital tracts explained the variance in stereoacuity. In addition, the VOF properties were not associated with differences in performance on a different psychophysical task (contrast detection). These series of experiments suggest that stereoscopic depth discrimination performance is, at least in part, constrained by dorso-ventral communication through the VOF.

First- and second-order contributions to depth perception in anti-correlated random dot stereograms

The binocular energy model of neural responses predicts that depth from binocular disparity might be perceived in the reversed direction when the contrast of dots presented to one eye is reversed. While reversed-depth has been found using anti-correlated random-dot stereograms (ACRDS) the findings are inconsistent across studies. The mixed findings may be accounted for by the presence of a gap between the target and surround, or as a result of overlap of dots around the vertical edges of the stimuli. To test this, we assessed whether (1) the gap size (0, 19.2 or 38.4 arc min) (2) the correlation of dots or (3) the border orientation (circular target, or horizontal or vertical edge) affected the perception of depth. Reversed-depth from ACRDS (circular no-gap condition) was seen by a minority of participants, but this effect reduced as the gap size increased. Depth was mostly perceived in the correct direction for ACRDS edge stimuli, with the effect increasing with the gap size. The inconsistency across conditions can be accounted for by the relative reliability of first- and second-order depth detection mechanisms, and the coarse spatial resolution of the latter.

Beyond Rehabilitation of Acuity, Ocular Alignment, and Binocularity in Infantile Strabismus

Infantile strabismus impairs the perception of all attributes of the visual scene. High spatial frequency components are no longer visible, leading to amblyopia. Binocularity is altered, leading to the loss of stereopsis. Spatial perception is impaired as well as detection of vertical orientation, the fastest movements, directions of movement, the highest contrasts and colors. Infantile strabismus also affects other vision-dependent processes such as control of postural stability. But presently, rehabilitative therapies for infantile strabismus by ophthalmologists, orthoptists and optometrists are restricted to preventing or curing amblyopia of the deviated eye, aligning the eyes and, whenever possible, preserving or restoring binocular vision during the critical period of development, i.e., before ~10 years of age. All the other impairments are thus ignored; whether they may recover after strabismus treatment even remains unknown. We argue here that medical and paramedical professionals may extend their present treatments of the perceptual losses associated with infantile strabismus. This hypothesis is based on findings from fundamental research on visual system organization of higher mammals in particular at the cortical level. In strabismic subjects (as in normal-seeing ones), information about all of the visual attributes converge, interact and are thus inter-dependent at multiple levels of encoding ranging from the single neuron to neuronal assemblies in visual cortex. Thus if the perception of one attribute is restored this may help to rehabilitate the perception of other attributes. Concomitantly, vision-dependent processes may also improve. This could occur spontaneously, but still should be assessed and validated. If not, medical and paramedical staff, in collaboration with neuroscientists, will have to break new ground in the field of therapies to help reorganize brain circuitry and promote more comprehensive functional recovery. Findings from fundamental research studies in both young and adult patients already support our hypothesis and are reviewed here. For example, presenting different contrasts to each eye of a strabismic patient during training sessions facilitates recovery of acuity in the amblyopic eye as well as of 3D perception. Recent data also demonstrate that visual recoveries in strabismic subjects improve postural stability. These findings form the basis for a roadmap for future research and clinical development to extend presently applied rehabilitative therapies for infantile strabismus.

Effects of cortical damage on binocular depth perception

Stereoscopic depth perception requires considerable neural computation, including the initial correspondence of the two retinal images, comparison across the local regions of the visual field and integration with other cues to depth. The most common cause for loss of stereoscopic vision is amblyopia, in which one eye has failed to form an adequate input to the visual cortex, usually due to strabismus (deviating eye) or anisometropia. However, the significant cortical processing required to produce the percept of depth means that, even when the retinal input is intact from both eyes, brain damage or dysfunction can interfere with stereoscopic vision. In this review, I examine the evidence for impairment of binocular vision and depth perception that can result from insults to the brain, including both discrete damage, temporal lobectomy and more systemic diseases such as posterior cortical atrophy.

This article is part of the themed issue ‘Vision in our three-dimensional world’.

Binocular depth processing in the ventral visual pathway

One of the most powerful forms of depth perception capitalizes on the small relative displacements, or binocular disparities, in the images projected onto each eye. The brain employs these disparities to facilitate various computations, including sensori-motor transformations (reaching, grasping), scene segmentation and object recognition. In accordance with these different functions, disparity activates a large number of regions in the brain of both humans and monkeys. Here, we review how disparity processing evolves along different regions of the ventral visual pathway of macaques, emphasizing research based on both correlational and causal techniques. We will discuss the progression in the ventral pathway from a basic absolute disparity representation to a more complex three-dimensional shape code. We will show that, in the course of this evolution, the underlying neuronal activity becomes progressively more bound to the global perceptual experience. We argue that these observations most probably extend beyond disparity processing per se, and pertain to object processing in the ventral pathway in general. We conclude by posing some important unresolved questions whose answers may significantly advance the field, and broaden its scope.This article is part of the themed issue 'Vision in our three-dimensional world'.

Weighted parallel contributions of binocular correlation and match signals to conscious perception of depth

Binocular disparity is detected in the primary visual cortex by a process similar to calculation of local cross-correlation between left and right retinal images. As a consequence, correlation-based neural signals convey information about false disparities as well as the true disparity. The false responses in the initial disparity detectors are eliminated at later stages in order to encode only disparities of the features correctly matched between the two eyes. For a simple stimulus configuration, a feed-forward nonlinear process can transform the correlation signal into the match signal. For human observers, depth judgement is determined by a weighted sum of the correlation and match signals rather than depending solely on the latter. The relative weight changes with spatial and temporal parameters of the stimuli, allowing adaptive recruitment of the two computations under different visual circumstances. A full transformation from correlation-based to match-based representation occurs at the neuronal population level in cortical area V4 and manifests in single-neuron responses of inferior temporal and posterior parietal cortices. Neurons in area V5/MT represent disparity in a manner intermediate between the correlation and match signals. We propose that the correlation and match signals in these areas contribute to depth perception in a weighted, parallel manner.This article is part of the themed issue 'Vision in our three-dimensional world'.

A dataset of stereoscopic images and ground-truth disparity mimicking human fixations in peripersonal space

Binocular stereopsis is the ability of a visual system, belonging to a live being or a machine, to interpret the different visual information deriving from two eyes/cameras for depth perception. From this perspective, the ground-truth information about three-dimensional visual space, which is hardly available, is an ideal tool both for evaluating human performance and for benchmarking machine vision algorithms. In the present work, we implemented a rendering methodology in which the camera pose mimics realistic eye pose for a fixating observer, thus including convergent eye geometry and cyclotorsion. The virtual environment we developed relies on highly accurate 3D virtual models, and its full controllability allows us to obtain the stereoscopic pairs together with the ground-truth depth and camera pose information. We thus created a stereoscopic dataset: GENUA PESTO-GENoa hUman Active fixation database: PEripersonal space STereoscopic images and grOund truth disparity. The dataset aims to provide a unified framework useful for a number of problems relevant to human and computer vision, from scene exploration and eye movement studies to 3D scene reconstruction.

Binocular Depth Judgments on Smoothly Curved Surfaces

Binocular disparity is an important cue to depth, allowing us to make very fine discriminations of the relative depth of objects. In complex scenes, this sensitivity depends on the particular shape and layout of the objects viewed. For example, judgments of the relative depths of points on a smoothly curved surface are less accurate than those for points in empty space. It has been argued that this occurs because depth relationships are represented accurately only within a local spatial area. A consequence of this is that, when judging the relative depths of points separated by depth maxima and minima, information must be integrated across separate local representations. This integration, by adding more stages of processing, might be expected to reduce the accuracy of depth judgements. We tested this idea directly by measuring how accurately human participants could report the relative depths of two dots, presented with different binocular disparities. In the first, Two Dot condition the two dots were presented in front of a square grid. In the second, Three Dot condition, an additional dot was presented midway between the target dots, at a range of depths, both nearer and further than the target dots. In the final, Surface condition, the target dots were placed on a smooth surface defined by binocular disparity cues. In some trials, this contained a depth maximum or minimum between the target dots. In the Three Dot condition, performance was impaired when the central dot was presented with a large disparity, in line with predictions. In the Surface condition, performance was worst when the midpoint of the surface was at a similar distance to the targets, and relatively unaffected when there was a large depth maximum or minimum present. These results are not consistent with the idea that depth order is represented only within a local spatial area.

Stereo-Curvature Aftereffect Is Due to More Than Shape Curvature Adaptation

For stereo-curvature aftereffect (sCAE), there is no agreement on whether adaptation occurs at the disparity-specified stage, the percept-specified stage, or both. Additionally, it remains uncertain whether retinal-position-dependent sCAE can be induced by possible adaptation to disparity-specified sources. Our study aimed to investigate the dependency and processing levels of adaptation underlying sCAE using dynamic spherical adaptation stimuli with static fixation. Experiment I examined the dependency by dynamically altering the location or size of adaptation stimuli. Experiment 2 investigated the adaptation levels via three sub-experiments: Experiment 2.1 examined how eccentricity influenced adaptation strength using static adaptation stimuli with different eccentricities, Experiment 2.2 tested a hypothesis about adaptation to a percept-specified primitive shape index (PSI) using dynamic size change of adaptation stimuli, and Experiment 2.3 tested another hypothesis on adapting disparity-specified average disparity information (ADI) using dynamic PSI change of adaptation stimuli. The results showed retinal-position-dependent and scale-independent sCAE. In addition to a possible eccentricity effect, the retinal-position dependence can result from ADI adaptation while the scale independence can be attributed to PSI adaptation. Therefore, sCAE is caused by adaptation at both the disparity-specified and percept-specified stages. Additionally, sCAE endows two coexisting adaptation processes with one dependent on retinal position and one independent of retinal position.

Functional architecture for disparity in macaque inferior temporal cortex and its relationship to the architecture for faces, color, scenes, and visual field

Binocular disparity is a powerful depth cue for object perception. The computations for object vision culminate in inferior temporal cortex (IT), but the functional organization for disparity in IT is unknown. Here we addressed this question by measuring fMRI responses in alert monkeys to stimuli that appeared in front of (near), behind (far), or at the fixation plane. We discovered three regions that showed preferential responses for near and far stimuli, relative to zero-disparity stimuli at the fixation plane. These "near/far" disparity-biased regions were located within dorsal IT, as predicted by microelectrode studies, and on the posterior inferotemporal gyrus. In a second analysis, we instead compared responses to near stimuli with responses to far stimuli and discovered a separate network of "near" disparity-biased regions that extended along the crest of the superior temporal sulcus. We also measured in the same animals fMRI responses to faces, scenes, color, and checkerboard annuli at different visual field eccentricities. Disparity-biased regions defined in either analysis did not show a color bias, suggesting that disparity and color contribute to different computations within IT. Scene-biased regions responded preferentially to near and far stimuli (compared with stimuli without disparity) and had a peripheral visual field bias, whereas face patches had a marked near bias and a central visual field bias. These results support the idea that IT is organized by a coarse eccentricity map, and show that disparity likely contributes to computations associated with both central (face processing) and peripheral (scene processing) visual field biases, but likely does not contribute much to computations within IT that are implicated in processing color.

Modeling the shape hierarchy for visually guided grasping

The monkey anterior intraparietal area (AIP) encodes visual information about three-dimensional object shape that is used to shape the hand for grasping. We modeled shape tuning in visual AIP neurons and its relationship with curvature and gradient information from the caudal intraparietal area (CIP). The main goal was to gain insight into the kinds of shape parameterizations that can account for AIP tuning and that are consistent with both the inputs to AIP and the role of AIP in grasping. We first experimented with superquadric shape parameters. We considered superquadrics because they occupy a role in robotics that is similar to AIP, in that superquadric fits are derived from visual input and used for grasp planning. We also experimented with an alternative shape parameterization that was based on an Isomap dimension reduction of spatial derivatives of depth (i.e., distance from the observer to the object surface). We considered an Isomap-based model because its parameters lacked discontinuities between similar shapes. When we matched the dimension of the Isomap to the number of superquadric parameters, the superquadric model fit the AIP data somewhat more closely. However, higher-dimensional Isomaps provided excellent fits. Also, we found that the Isomap parameters could be approximated much more accurately than superquadric parameters by feedforward neural networks with CIP-like inputs. We conclude that Isomaps, or perhaps alternative dimension reductions of visual inputs to AIP, provide a promising model of AIP electrophysiology data. Further work is needed to test whether such shape parameterizations actually provide an effective basis for grasp control.

Linking brain imaging signals to visual perception

The rapid advances in brain imaging technology over the past 20 years are affording new insights into cortical processing hierarchies in the human brain. These new data provide a complementary front in seeking to understand the links between perceptual and physiological states. Here we review some of the challenges associated with incorporating brain imaging data into such "linking hypotheses," highlighting some of the considerations needed in brain imaging data acquisition and analysis. We discuss work that has sought to link human brain imaging signals to existing electrophysiological data and opened up new opportunities in studying the neural basis of complex perceptual judgments. We consider a range of approaches when using human functional magnetic resonance imaging to identify brain circuits whose activity changes in a similar manner to perceptual judgments and illustrate these approaches by discussing work that has studied the neural basis of 3D perception and perceptual learning. Finally, we describe approaches that have sought to understand the information content of brain imaging data using machine learning and work that has integrated multimodal data to overcome the limitations associated with individual brain imaging approaches. Together these approaches provide an important route in seeking to understand the links between physiological and psychological states.

Linking brain to behavior for the visual perception of figures and objects

The dissociation of a figure from its background is an essential feat of visual perception, as it allows us to detect, recognize, and interact with shapes and objects in our environment. In order to understand how the human brain gives rise to the perception of figures, we here review experiments that explore the links between activity in visual cortex and performance of perceptual tasks related to figure perception. We organize our review according to a proposed model that attempts to contextualize figure processing within the more general framework of object processing in the brain. Overall, the current literature provides us with individual linking hypotheses as to cortical regions that are necessary for particular tasks related to figure perception. Attempts to reach a more complete understanding of how the brain instantiates figure and object perception, however, will have to consider the temporal interaction between the many regions involved, the details of which may vary widely across different tasks.

Reverse correlation reveals how observers sample visual information when estimating three-dimensional shape

Human observers exhibit large systematic distance-dependent biases when estimating the three-dimensional (3D) shape of objects defined by binocular image disparities. This has led some to question the utility of disparity as a cue to 3D shape and whether accurate estimation of 3D shape is at all possible. Others have argued that accurate perception is possible, but only with large continuous perspective transformations of an object. Using a stimulus that is known to elicit large distance-dependent perceptual bias (random dot stereograms of elliptical cylinders) we show that contrary to these findings the simple adoption of a more naturalistic viewing angle completely eliminates this bias. Using behavioural psychophysics, coupled with a novel surface-based reverse correlation methodology, we show that it is binocular edge and contour information that allows for accurate and precise perception and that observers actively exploit and sample this information when it is available.

The ventral visual pathway: an expanded neural framework for the processing of object quality

Since the original characterization of the ventral visual pathway, our knowledge of its neuroanatomy, functional properties, and extrinsic targets has grown considerably. Here we synthesize this recent evidence and propose that the ventral pathway is best understood as a recurrent occipitotemporal network containing neural representations of object quality both utilized and constrained by at least six distinct cortical and subcortical systems. Each system serves its own specialized behavioral, cognitive, or affective function, collectively providing the raison d'être for the ventral visual pathway. This expanded framework contrasts with the depiction of the ventral visual pathway as a largely serial staged hierarchy culminating in singular object representations and more parsimoniously incorporates attentional, contextual, and feedback effects.

Independent effects of local and global binocular disparity on the perceived convexity of stereoscopically presented faces in scenes

Evidence suggests that experiencing the hollow-face illusion involves perceptual reversal of the binocular disparities associated with the face even though the rest of the scene appears unchanged. This suggests stereoscopic processing of object shape may be independent of scene-based processing of the layout of objects in depth. We investigated the effects of global scene-based and local object-based disparity on the compellingness of the perceived convexity of the face. We took stereoscopic photographs of people in scenes, and independently reversed the binocular disparities associated with the head and scene. Participants rated perceived convexity of a natural disparity ("convex") or reversed disparity ("concave") face shown either in its original context with reversed or natural disparities or against a black background. Faces with natural disparity were rated as more convincingly convex independent of the background, showing that the local disparities can affect perceived convexity independent of disparities across the rest of the image. However, the apparent convexity of the faces was also greater in natural disparity scenes compared to either a reversed disparity scene or a zero disparity black background. This independent effect of natural scene disparity suggests that the 'solidity' associated with natural scene disparities spread to enhance the perceived convexity of the face itself. Together, these findings suggest that global and local disparity exert independent and additive effects upon the perceived convexity of the face.

Disparity-specific spatial interactions: evidence from EEG source imaging

Using cortical source estimation techniques based on high-density EEG and fMRI measurements in humans, we measured how a disparity-defined surround influenced the responses to the changing disparity of a central disk within five visual ROIs: V1, V4, lateral occipital complex (LOC), hMT+, and V3A. The responses in the V1 ROI were not consistently affected either by changes in the characteristics of the surround (correlated or uncorrelated) or by its disparity value, consistent with V1 being responsive only to absolute, not relative, disparity. Correlation in the surround increased the responses in the V4, LOC, and hMT+ ROIs over those measured with the uncorrelated surround. Thus, these extrastriate areas contain neurons that are sensitive to disparity differences. However, their evoked responses did not vary systematically with the surround disparity. Responses in the V3A ROI, in contrast, were increased by correlation in the surround and varied with its disparity. We modeled these V3A responses as attributable to a gain modulation of the absolute disparity response, where the gain amplitude is proportional to the center-surround disparity difference. An additional experiment identified a nonlinear center-surround interaction in V3A that facilitates the responses when center and surround are misaligned but suppresses it when they share the same disparity plane.

The integration of motion and disparity cues to depth in dorsal visual cortex

Humans exploit a range of visual depth cues to estimate three-dimensional structure. For example, the slant of a nearby tabletop can be judged by combining information from binocular disparity, texture and perspective. Behavioral tests show humans combine cues near-optimally, a feat that could depend on discriminating the outputs from cue-specific mechanisms or on fusing signals into a common representation. Although fusion is computationally attractive, it poses a substantial challenge, requiring the integration of quantitatively different signals. We used functional magnetic resonance imaging (fMRI) to provide evidence that dorsal visual area V3B/KO meets this challenge. Specifically, we found that fMRI responses are more discriminable when two cues (binocular disparity and relative motion) concurrently signal depth, and that information provided by one cue is diagnostic of depth indicated by the other. This suggests a cortical node important when perceiving depth, and highlights computations based on fusion in the dorsal stream.

Bridging the gap: global disparity processing in the human visual cortex

The human stereoscopic system is remarkable in its ability to utilize widely separated features as references to support fine depth discrimination. In a search for possible neural substrates of this ability, we recorded high-density EEG and used a distributed inverse technique to estimate population-level disparity responses in five regions of interest (ROIs): V1, V3A, hMT+, V4, and lateral occipital complex (LOC). The stimulus was a central modulating disk surrounded by a correlated "reference" annulus presented in the fixation plane. We varied a gap separating the disk from the annulus parametrically from 0 to 5.5° as a test of long-range disparity integration. In the V1, LOC, and hMT+ ROIs, the responses with gaps >0.5° were equal to those obtained in a control condition where the surround was composed of uncorrelated noise (no reference). By contrast, in the V4 and V3A ROIs, responses with gaps as large as 5.5° were still significantly higher than the control. As a test of the spatial distribution of the disparity reference information, we manipulated the properties of the stimulus by placing noise between the center and the surround or throughout the surround. The V3A ROI was particularly sensitive to disparity noise between the center and annulus regions, suggesting an important contribution of disparity edge detectors in this ROI.

Neurons in dorsal visual area V5/MT signal relative disparity

Judgments of visual depth rely crucially on the relative binocular disparity between two visual features. While areas of ventral visual cortex contain neurons that signal the relative disparity between spatially adjacent visual features, the same tests in dorsal visual areas yield little evidence for relative disparity selectivity. We investigated the sensitivity of neurons in dorsal visual area V5/MT of macaque monkeys to relative disparity, using two superimposed, transparent planes composed of dots moving in opposite directions. The separation of the planes in depth specifies their relative disparity, while absolute disparity can be altered independently by changing the binocular depth of the two planes with respect to the monkey's fixation point. Many V5/MT neurons were tuned to relative disparity, independent of the absolute disparities of the individual planes. For the two plane stimulus, neuronal responses were often linearly related to responses to the absolute disparity of each component plane presented individually, but some aspects of relative disparity tuning were not explained by linear combination. Selectivity for relative disparity could not predict whether neuronal firing was related to the monkeys' perceptual reports of the rotation direction of structure-from-motion figures centered on the plane of fixation. In sum, V5/MT neurons are not just selective for absolute disparity, but also code for relative disparity between visual features. This selectivity may be important for segmentation and depth order of moving visual features, particularly the processing of three-dimensional information in scenes viewed by an actively moving observer.

The extraction of 3D shape in the visual system of human and nonhuman primates

Depth structure, the third dimension of object shape, is extracted from disparity, motion, texture, and shading in the optic array. Gradient-selective neurons play a key role in this process. Such neurons occur in CIP, AIP, TEs, and F5 (for first- or second-order disparity gradients), in MT/V5, in FST (for speed gradients), and in CIP and TEs (for texture gradients). Most of these regions are activated during magnetic resonance scanning in alert monkeys by comparing 3D conditions with the 2D controls for the different cues. Similarities in activation patterns of monkeys and humans tested with identical paradigms suggest that like gradient-selective neurons are found in corresponding human cortical areas. This view gains credence as the homologies between such areas become more evident. Furthermore, 3D shape-processing networks are similar in the two species, with the exception of the greater involvement of human posterior parietal cortex in the extraction of 3D shape from motion. Thus we can begin to understand how depth structure is extracted from motion, disparity, and texture in the primate brain, but the extraction of depth structure from shading and that of wire-like objects requires further scrutiny.

A new neural framework for visuospatial processing

The division of cortical visual processing into distinct dorsal and ventral streams is a key framework that has guided visual neuroscience. The characterization of the ventral stream as a 'What' pathway is relatively uncontroversial, but the nature of dorsal stream processing is less clear. Originally proposed as mediating spatial perception ('Where'), more recent accounts suggest it primarily serves non-conscious visually guided action ('How'). Here, we identify three pathways emerging from the dorsal stream that consist of projections to the prefrontal and premotor cortices, and a major projection to the medial temporal lobe that courses both directly and indirectly through the posterior cingulate and retrosplenial cortices. These three pathways support both conscious and non-conscious visuospatial processing, including spatial working memory, visually guided action and navigation, respectively.

Synchronization between the end stages of the dorsal and the ventral visual stream

The end stage areas of the ventral (IT) and the dorsal (AIP) visual streams encode the shape of disparity-defined three-dimensional (3D) surfaces. Recent anatomical tracer studies have found direct reciprocal connections between the 3D-shape selective areas in IT and AIP. Whether these anatomical connections are used to facilitate 3D-shape perception is still unknown. We simultaneously recorded multi-unit activity (MUA) and local field potentials in IT and AIP while monkeys discriminated between concave and convex 3D shapes and measured the degree to which the activity in IT and AIP synchronized during the task. We observed strong beta-band synchronization between IT and AIP preceding stimulus onset that decreased shortly after stimulus onset and became modulated by stereo-signal strength and stimulus contrast during the later portion of the stimulus period. The beta-coherence modulation was unrelated to task-difficulty, regionally specific, and dependent on the MUA selectivity of the pairs of sites under study. The beta-spike-field coherence in AIP predicted the upcoming choice of the monkey. Several convergent lines of evidence suggested AIP as the primary source of the AIP-IT synchronized activity. The synchronized beta activity seemed to occur during perceptual anticipation and when the system has stabilized to a particular perceptual state but not during active visual processing. Our findings demonstrate for the first time that synchronized activity exists between the end stages of the dorsal and ventral stream during 3D-shape discrimination.

Complex cells in the cat striate cortex have multiple disparity detectors in the three-dimensional binocular receptive fields

Along the visual pathway, neurons generally become more specialized for signaling a limited subset of stimulus attributes and become more invariant to changes in the stimulus position within the receptive fields (RFs). One of the likely mechanisms underlying such invariance appears to be pooling of detectors located at different positions. Does such spatial pooling occur for disparity-selective neurons in primary visual cortex? To examine whether the three-dimensional (3D) binocular RFs are constructed by pooling detectors for binocular disparity, we investigated binocular interactions in the 3D space for neurons in the cat striate cortex. Approximately one-third of complex cells showed the spatial pooling of disparity detectors to a significant degree, whereas the majority of simple cells did not. The degree of spatial pooling of disparity detectors along the preferred orientation axis was generally larger than that along the axis orthogonal to the orientation axis. We then reconstructed 3D binocular RFs in their complete form and examined their structures. Disparity tuning curves were compared across positions along the orientation axis in the RFs. A small population of cells appeared to show a gradual shift of the preferred disparity along this axis, indicating that they can potentially signal inclination in the 3D space. However, the majority of cells exhibited a position-invariant disparity tuning. Finally, disparity tuning curves were examined for all oblique angles in addition to horizontal and vertical. Tunings were broadest along the orientation axis as the disparity energy model predicts.

Binocular vision

This essay reviews major developments - empirical and theoretical - in the field of binocular vision during the last 25years. We limit our survey primarily to work on human stereopsis, binocular rivalry and binocular contrast summation, with discussion where relevant of single-unit neurophysiology and human brain imaging. We identify several key controversies that have stimulated important work on these problems. In the case of stereopsis those controversies include position vs. phase encoding of disparity, dependence of disparity limits on spatial scale, role of occlusion in binocular depth and surface perception, and motion in 3D. In the case of binocular rivalry, controversies include eye vs. stimulus rivalry, role of "top-down" influences on rivalry dynamics, and the interaction of binocular rivalry and stereopsis. Concerning binocular contrast summation, the essay focuses on two representative models that highlight the evolving complexity in this field of study.