Disparity selectivity of neurons in monkey inferior temporal cortex

Neural substrate for the engagement of the ventral visual stream in motor control in the macaque monkey

The present study aimed to describe the cortical connectivity of a sector located in the ventral bank of the superior temporal sulcus in the macaque (intermediate area TEa and TEm [TEa/m]), which appears to represent the major source of output of the ventral visual stream outside the temporal lobe. The retrograde tracer wheat germ agglutinin was injected in the intermediate TEa/m in four macaque monkeys. The results showed that 58-78% of labeled cells were located within ventral visual stream areas other than the TE complex. Outside the ventral visual stream, there were connections with the memory-related medial temporal area 36 and the parahippocampal cortex, orbitofrontal areas involved in encoding subjective values of stimuli for action selection, and eye- or hand-movement related parietal (LIP, AIP, and SII), prefrontal (12r, 45A, and 45B) areas, and a hand-related dysgranular insula field. Altogether these data provide a solid substrate for the engagement of the ventral visual stream in large scale cortical networks for skeletomotor or oculomotor control. Accordingly, the role of the ventral visual stream could go beyond pure perceptual processes and could be also finalized to the neural mechanisms underlying the control of voluntary motor behavior.

Context-based modulations of 3D vision are expertise dependent

An object's identity can influence depth-position judgments. The mechanistic underpinnings underlying this phenomenon are largely unknown. Here, we asked whether context-dependent modulations of stereoscopic depth perception are expertise dependent. In 2 experiments, we tested whether training that attaches meaning (i.e. classification labels) to otherwise novel, stereoscopically presented objects changes observers' sensitivity for judging their depth position. In Experiment 1, observers were randomly assigned to 3 groups: a Greeble-classification training group, an orientation-discrimination training group, or a no-training group, and were tested on their stereoscopic depth sensitivity before and after training. In Experiment 2, participants were tested before and after training while fMRI responses were concurrently imaged. Behaviorally, stereoscopic performance was significantly better following Greeble-classification (but not orientation-discrimination, or no-) training. Using the fMRI data, we trained support vector machines to predict whether the data were from the pre- or post-training sessions. Results indicated that classification accuracies in V4 were higher for the Greeble-classification group as compared with the orientation-discrimination group for which accuracies were at chance level. Furthermore, classification accuracies in V4 were negatively correlated with response times for Greeble identification. We speculate that V4 is implicated in an expertise-dependent, object-tuning manner that allows it to better guide stereoscopic depth retrieval.

Modulations of depth responses in the human brain by object context: Does biological relevance matter?

Depth sensitivity has been shown to be modulated by object context (plausibility). It is possible that it is behavioural relevance rather than object plausibility per se which drives this effect. Here, we manipulated the biological relevance of objects (face or a non-face) and tested whether object relevance affects behavioural sensitivity and neural responses to depth-position. In a first experiment, we presented human observers with disparity-defined faces and non-faces, and observers were asked to judge the depth position of the target under signal-noise and clear (fine) task conditions. In the second experiment, we concurrently measured behavioural and fMRI responses to depth. We found that behavioural performance varied across stimulus conditions such that they were significantly worse for the upright face than the inverted face and the random shape in the SNR task, but worse for the random shape than the upright face in the feature task. Pattern analysis of fMRI responses revealed that activity of FFA was distinctly different during depth judgments of the upright face versus the other two stimuli, with its responses (and to a stronger extent, those of V3) appearing functionally-relevant to behavioural performance. We speculate that FFA is not only involved in object analysis, but exerts considerable influence on stereoscopic mechanisms as early as in V3 based on a broader appreciation of the stimulus' behavioural relevance.Significance StatementWe asked how disparity sensitivity is modulated by object (biological) relevance using behavioural and neuroimaging paradigms. We show that behavioural sensitivity to depth-position changes in biological (face) vs non-biological (random surface) contexts, and that these changes are task-dependent. Imaging results highlight a potentially key role of the fusiform region for governing the modulation of stereo encoding by object relevance. These findings highlight powerful interactions between object recognition mechanisms and stereoencoding, such that the utility of disparity information may be up/down weighed depending on the biological relevance of the object.

Surface slant impairs disparity discontinuity discrimination

Binocular disparity signals are highly informative about the three-dimensional structure of visual scenes, including aiding the detection of depth discontinuities between surfaces. Here, we examine factors affecting sensitivity to such surface discontinuities. Participants were presented with random dot stereograms depicting two planar surfaces slanted in opposite directions and were asked to judge the sign of the depth discontinuity created where those surfaces met. Although the judgement was focussed on the adjacent edges, the precision of depth discontinuity discrimination depended upon the slant of the two surfaces: increasing surface slants to ±60° increased discontinuity discrimination thresholds by, on average, a factor of 5. Control experiments examining discontinuity discrimination across surfaces with identical slants showed either biases in discontinuity judgements or reduced threshold elevation. These results suggest that sensitivity to depth discontinuities is affected by processing limitations in both local absolute disparity measurement mechanisms and mechanisms selective for disparity differences. As further evidence in support of this conclusion, we show that our results are well-described by a model of discontinuity discrimination based on the encoding of local differences in relative disparity.

Microstructural properties of the vertical occipital fasciculus explain the variability in human stereoacuity

Seeing in the three-dimensional world—stereopsis—is an innate human ability, but it varies substantially among individuals. The neurobiological basis of this variability is not understood. We combined diffusion and quantitative MRI imaging with a psychophysical measurements, and found that variability in stereoacuity is associated with microstructural differences in the right vertical occipital fasciculus, a white matter tract connecting dorsal and ventral visual cortex. This result suggests that the microstructure of the pathways that support information transmission across dorsal and ventral visual areas plays an important role human stereopsis.

Stereopsis is a fundamental visual function that has been studied extensively. However, it is not clear why depth discrimination (stereoacuity) varies more significantly among people than other modalities. Previous studies have reported the involvement of both dorsal and ventral visual areas in stereopsis, implying that not only neural computations in cortical areas but also the anatomical properties of white matter tracts connecting those areas can impact stereopsis. Here, we studied how human stereoacuity relates to white matter properties by combining psychophysics, diffusion MRI (dMRI), and quantitative MRI (qMRI). We performed a psychophysical experiment to measure stereoacuity and, in the same participants, we analyzed the microstructural properties of visual white matter tracts on the basis of two independent measurements, dMRI (fractional anisotropy, FA) and qMRI (macromolecular tissue volume; MTV). Microstructural properties along the right vertical occipital fasciculus (VOF), a major tract connecting dorsal and ventral visual areas, were highly correlated with measures of stereoacuity. This result was consistent for both FA and MTV, suggesting that the behavioral–structural relationship reflects differences in neural tissue density, rather than differences in the morphological configuration of fibers. fMRI confirmed that binocular disparity stimuli activated the dorsal and ventral visual regions near VOF endpoints. No other occipital tracts explained the variance in stereoacuity. In addition, the VOF properties were not associated with differences in performance on a different psychophysical task (contrast detection). These series of experiments suggest that stereoscopic depth discrimination performance is, at least in part, constrained by dorso-ventral communication through the VOF.

Single-cell responses to three-dimensional structure in a functionally defined patch in macaque area TEO

Both dorsal and ventral visual pathways harbor several areas sensitive to gradients of binocular disparity (i.e., higher-order disparity). Although a wealth of information exists about disparity processing in early visual (V1, V2, and V3) and end-stage areas, TE in the ventral stream, and the anterior intraparietal area (AIP) in the dorsal stream, little is known about midlevel area TEO in the ventral pathway. We recorded single-unit responses to disparity-defined curved stimuli in a functional magnetic resonance imaging (fMRI) activation elicited by curved surfaces compared with flat surfaces in the macaque area TEO. This fMRI activation contained a small proportion of disparity-selective neurons, with very few of them second-order disparity selective. Overall, this population of TEO neurons did not preserve its three-dimensional structure selectivity across positions in depth, indicating a lack of higher-order disparity selectivity, but showed stronger responses to flat surfaces than to curved surfaces, as predicted by the fMRI experiment. The receptive fields of the responsive TEO cells were relatively small and generally foveal. A linear support vector machine classifier showed that this population of disparity-selective TEO neurons contains reliable information about the sign of curvature and the position in depth of the stimulus. NEW & NOTEWORTHY We recorded in a part of the macaque area TEO that is activated more by curved surfaces than by flat surfaces at different disparities using the same stimuli. In contrast to previous studies, this functional magnetic resonance imaging-defined patch did not contain a large number of higher-order disparity-selective neurons. However, a linear support vector machine could reliably classify both the sign of the disparity gradient and the position in depth of the stimuli.

Stereoscopic processing of crossed and uncrossed disparities in the human visual cortex

BACKGROUND

Binocular disparity provides a powerful cue for depth perception in a stereoscopic environment. Despite increasing knowledge of the cortical areas that process disparity from neuroimaging studies, the neural mechanism underlying disparity sign processing [crossed disparity (CD)/uncrossed disparity (UD)] is still poorly understood. In the present study, functional magnetic resonance imaging (fMRI) was used to explore different neural features that are relevant to disparity-sign processing.

METHODS

We performed an fMRI experiment on 27 right-handed healthy human volunteers by using both general linear model (GLM) and multi-voxel pattern analysis (MVPA) methods. First, GLM was used to determine the cortical areas that displayed different responses to different disparity signs. Second, MVPA was used to determine how the cortical areas discriminate different disparity signs.

RESULTS

The GLM analysis results indicated that shapes with UD induced significantly stronger activity in the sub-region (LO) of the lateral occipital cortex (LOC) than those with CD. The results of MVPA based on region of interest indicated that areas V3d and V3A displayed higher accuracy in the discrimination of crossed and uncrossed disparities than LOC. The results of searchlight-based MVPA indicated that the dorsal visual cortex showed significantly higher prediction accuracy than the ventral visual cortex and the sub-region LO of LOC showed high accuracy in the discrimination of crossed and uncrossed disparities.

CONCLUSIONS

The results may suggest the dorsal visual areas are more discriminative to the disparity signs than the ventral visual areas although they are not sensitive to the disparity sign processing. Moreover, the LO in the ventral visual cortex is relevant to the recognition of shapes with different disparity signs and discriminative to the disparity sign.

Binocular depth processing in the ventral visual pathway

One of the most powerful forms of depth perception capitalizes on the small relative displacements, or binocular disparities, in the images projected onto each eye. The brain employs these disparities to facilitate various computations, including sensori-motor transformations (reaching, grasping), scene segmentation and object recognition. In accordance with these different functions, disparity activates a large number of regions in the brain of both humans and monkeys. Here, we review how disparity processing evolves along different regions of the ventral visual pathway of macaques, emphasizing research based on both correlational and causal techniques. We will discuss the progression in the ventral pathway from a basic absolute disparity representation to a more complex three-dimensional shape code. We will show that, in the course of this evolution, the underlying neuronal activity becomes progressively more bound to the global perceptual experience. We argue that these observations most probably extend beyond disparity processing per se, and pertain to object processing in the ventral pathway in general. We conclude by posing some important unresolved questions whose answers may significantly advance the field, and broaden its scope.This article is part of the themed issue 'Vision in our three-dimensional world'.

Weighted parallel contributions of binocular correlation and match signals to conscious perception of depth

Binocular disparity is detected in the primary visual cortex by a process similar to calculation of local cross-correlation between left and right retinal images. As a consequence, correlation-based neural signals convey information about false disparities as well as the true disparity. The false responses in the initial disparity detectors are eliminated at later stages in order to encode only disparities of the features correctly matched between the two eyes. For a simple stimulus configuration, a feed-forward nonlinear process can transform the correlation signal into the match signal. For human observers, depth judgement is determined by a weighted sum of the correlation and match signals rather than depending solely on the latter. The relative weight changes with spatial and temporal parameters of the stimuli, allowing adaptive recruitment of the two computations under different visual circumstances. A full transformation from correlation-based to match-based representation occurs at the neuronal population level in cortical area V4 and manifests in single-neuron responses of inferior temporal and posterior parietal cortices. Neurons in area V5/MT represent disparity in a manner intermediate between the correlation and match signals. We propose that the correlation and match signals in these areas contribute to depth perception in a weighted, parallel manner.This article is part of the themed issue 'Vision in our three-dimensional world'.

Postnatal Development of Intrinsic Horizontal Axons in Macaque Inferior Temporal and Primary Visual Cortices

Two distinct areas along the ventral visual stream of monkeys, the primary visual (V1) and inferior temporal (TE) cortices, exhibit different projection patterns of intrinsic horizontal axons with patchy terminal fields in adult animals. The differences between the patches in these 2 areas may reflect differences in cortical representation and processing of visual information. We studied the postnatal development of patches by injecting an anterograde tracer into TE and V1 in monkeys of various ages. At 1 week of age, labeled patches with distribution patterns reminiscent of those in adults were already present in both areas. The labeling intensity of patches decayed exponentially with projection distance in monkeys of all ages in both areas, but this trend was far less evident in TE. The number and extent of patches gradually decreased with age in V1, but not in TE. In V1, axonal and bouton densities increased postnatally only in patches with short projection distances, whereas in TE this density change occurred in patches with various projection distances. Thus, patches with area-specific distribution patterns are formed early in life, and area-specific postnatal developmental processes shape the connectivity of patches into adulthood.

Pooled, but not single-neuron, responses in macaque V4 represent a solution to the stereo correspondence problem

Binocular disparity is an important cue for depth perception. To correctly represent disparity, neurons must find corresponding visual features between the left- and right-eye images. The visual pathway ascending from V1 to inferior temporal cortex solves the correspondence problem. An intermediate area, V4, has been proposed to be a critical stage in the correspondence process. However, the distinction between V1 and V4 is unclear, because accumulating evidence suggests that the process begins within V1. In this article, we report that the pooled responses in macaque V4, but not responses of individual neurons, represent a solution to the correspondence problem. We recorded single-unit responses of V4 neurons to random-dot stereograms of varying degrees of anticorrelation. To achieve gradual anticorrelation, we reversed the contrast of an increasing proportion of dots as in our previous psychophysical studies, which predicted that the neural correlates of the solution to correspondence problem should gradually eliminate their disparity modulation as the level of anticorrelation increases. Inconsistent with this prediction, the tuning amplitudes of individual V4 neurons quickly decreased to a nonzero baseline with small anticorrelation. By contrast, the shapes of individual tuning curves changed more gradually so that the amplitude of population-pooled responses gradually decreased toward zero over the entire range of graded anticorrelation. We explain these results by combining multiple energy-model subunits. From a comparison with the population-pooled responses in V1, we suggest that disparity representation in V4 is distinctly advanced from that in V1. Population readout of V4 responses provides disparity information consistent with the correspondence solution.

Computation of Object Size in Visual Cortical Area V4 as a Neural Basis for Size Constancy

Even when we view an object from different distances, so that the size of its projection onto the retina varies, we perceive its size to be relatively unchanged. In this perceptual phenomenon known as size constancy, the brain uses both distance and retinal image size to estimate the size of an object. Given that binocular disparity, the small positional difference between the retinal images in the two eyes, is a powerful visual cue for distance, we examined how it affects neuronal tuning to retinal image size in visual cortical area V4 of macaque monkeys. Depending on the imposed binocular disparity of a circular patch embedded in random dot stereograms, most neurons adjusted their preferred size in a manner consistent with size constancy. They preferred larger retinal image sizes when stimuli were stereoscopically presented nearer and preferred smaller retinal image sizes when stimuli were presented farther away. This disparity-dependent shift of preferred image size was not affected by the vergence angle, a cue for the fixation distance, suggesting that different V4 neurons compute object size for different fixation distances rather than that individual neurons adjust the shift based on vergence. This interpretation was supported by a simple circuit model, which could simulate the shift of preferred image size without any information about the fixation distance. We suggest that a population of V4 neurons encodes the actual size of objects, rather than simply the size of their retinal images, and that these neurons thereby contribute to size constancy.

SIGNIFICANCE STATEMENT

We perceive the size of an object to be relatively stable despite changes in the size of its retinal image that accompany changes in viewing distance. This phenomenon, called size constancy, is accomplished by combining retinal image size and distance information in our brain. We demonstrate that a large population of V4 neurons changes their size tuning depending on the perceived distance of a visual stimulus derived from binocular disparity. They prefer larger or smaller retinal image sizes when stimuli are stereoscopically presented nearer or farther away, respectively. This property makes V4 neurons suitable for encoding the actual size of objects, not simply the retinal image sizes, and providing a possible mechanism for perceptual size constancy.

The body and objects represented in the ventral stream of the parieto-premotor network

The network between the parietal cortex and premotor cortex has a pivotal role in sensory-motor control. Grasping-related neurons in the anterior intraparietal area (AIP) and the ventral premotor cortex (F5) showed complementary properties each other. The object information for grasping is sent from the parietal cortex to the premotor cortex for sensory-motor transformation, and the backward signal from the premotor cortex to parietal cortex can be considered an efference copy/corollary discharge that is used to predict sensory outcome during motor behavior. Mirror neurons that represent both own action and other's action are involved in this system. This system also very well fits with body schema that reflects online state of the body during motor execution. We speculate that the parieto-premotor network, which includes the mirror neuron system, is key for mapping one's own body and the bodies of others. This means that the neuronal substrates that control one's own action and the mirror neuron system are shared with the "who" system, which is related to the recognition of action contribution, i.e., sense of agency. Representation of own and other's body in the parieto-premotor network is key to link between sensory-motor control and higher-order cognitive functions.

Integration of Multiple Spatial Frequency Channels in Disparity-Sensitive Neurons in the Primary Visual Cortex

For our vivid perception of a 3-D world, the stereoscopic function begins in our brain by detecting slight shifts of image features between the two eyes, called binocular disparity. The primary visual cortex is the first stage of this processing, and neurons there are tuned to a limited range of spatial frequencies (SFs). However, our visual world is generally highly complex, composed of numerous features at a variety of scales, thereby having broadband SF spectra. This means that binocular information signaled by individual neurons is highly incomplete, and combining information across multiple SF bands must be essential for the visual system to function in a robust and reliable manner. In this study, we investigated whether the integration of information from multiple SF channels begins in the cat primary visual cortex. We measured disparity-selective responses in the joint left-right SF domain using sequences of dichoptically flashed grating stimuli consisting of various combinations of SFs and phases. The obtained interaction map in the joint SF domain reflects the degree of integration across different SF channels. Our data are consistent with the idea that disparity information is combined from multiple SF channels in a substantial fraction of complex cells. Furthermore, for the majority of these neurons, the optimal disparity is matched across the SF bands. These results suggest that a highly specific SF integration process for disparity detection starts in the primary visual cortex.

SIGNIFICANCE STATEMENT

Our visual world is broadband, containing features with a wide range of object scales. On the other hand, single neurons in the primary visual cortex are narrow-band, being tuned narrowly for a specific scale. For robust visual perception, narrow-band information of single neurons must be integrated eventually at some stage. We have examined whether such an integration process begins in the primary visual cortex with respect to binocular processing. The results suggest that a subset of cells appear to combine binocular information across multiple scales. Furthermore, for the majority of these neurons, an optimal parameter of binocular tuning is matched across multiple scales, suggesting the presence of a highly specific neural integration mechanism.

Functional architecture for disparity in macaque inferior temporal cortex and its relationship to the architecture for faces, color, scenes, and visual field

Binocular disparity is a powerful depth cue for object perception. The computations for object vision culminate in inferior temporal cortex (IT), but the functional organization for disparity in IT is unknown. Here we addressed this question by measuring fMRI responses in alert monkeys to stimuli that appeared in front of (near), behind (far), or at the fixation plane. We discovered three regions that showed preferential responses for near and far stimuli, relative to zero-disparity stimuli at the fixation plane. These "near/far" disparity-biased regions were located within dorsal IT, as predicted by microelectrode studies, and on the posterior inferotemporal gyrus. In a second analysis, we instead compared responses to near stimuli with responses to far stimuli and discovered a separate network of "near" disparity-biased regions that extended along the crest of the superior temporal sulcus. We also measured in the same animals fMRI responses to faces, scenes, color, and checkerboard annuli at different visual field eccentricities. Disparity-biased regions defined in either analysis did not show a color bias, suggesting that disparity and color contribute to different computations within IT. Scene-biased regions responded preferentially to near and far stimuli (compared with stimuli without disparity) and had a peripheral visual field bias, whereas face patches had a marked near bias and a central visual field bias. These results support the idea that IT is organized by a coarse eccentricity map, and show that disparity likely contributes to computations associated with both central (face processing) and peripheral (scene processing) visual field biases, but likely does not contribute much to computations within IT that are implicated in processing color.

The place of human psychophysics in modern neuroscience

Human psychophysics is the quantitative measurement of our own perceptions. In essence, it is simply a more sophisticated version of what humans have done since time immemorial: noticed and reflected upon what we can see, hear, and feel. In the 21st century, when hugely powerful techniques are available that enable us to probe the innermost structure and function of nervous systems, is human psychophysics still relevant? I argue that it is, and that in combination with other techniques, it will continue to be a key part of neuroscience for the foreseeable future. I discuss these points in detail using the example of binocular stereopsis, where human psychophysics in combination with physiology and computational vision, has made a substantial contribution.

Binocular stereoscopy in visual areas V-2, V-3, and V-3A of the macaque monkey

Over 40 years ago, Hubel and Wiesel gave a preliminary report of the first account of cells in monkey cerebral cortex selective for binocular disparity. The cells were located outside of V-1 within a region referred to then as "area 18." A full-length manuscript never followed, because the demarcation of the visual areas within this region had not been fully worked out. Here, we provide a full description of the physiological experiments and identify the locations of the recorded neurons using a contemporary atlas generated by functional magnetic resonance imaging; we also perform an independent analysis of the location of the neurons relative to an anatomical landmark (the base of the lunate sulcus) that is often coincident with the border between V-2 and V-3. Disparity-tuned cells resided not only in V-2, the area now synonymous with area 18, but also in V-3 and probably within V-3A. The recordings showed that the disparity-tuned cells were biased for near disparities, tended to prefer vertical orientations, clustered by disparity preference, and often required stimulation of both eyes to elicit responses, features strongly suggesting a role in stereoscopic depth perception.

Spatially invariant computations in stereoscopic vision

PERCEPTION OF STEREOSCOPIC DEPTH REQUIRES THAT VISUAL SYSTEMS SOLVE A CORRESPONDENCE PROBLEM: find parts of the left-eye view of the visual scene that correspond to parts of the right-eye view. The standard model of binocular matching implies that similarity of left and right images is computed by inter-ocular correlation. But the left and right images of the same object are normally distorted relative to one another by the binocular projection, in particular when slanted surfaces are viewed from close distance. Correlation often fails to detect correct correspondences between such image parts. We investigate a measure of inter-ocular similarity that takes advantage of spatially invariant computations similar to the computations performed by complex cells in biological visual systems. This measure tolerates distortions of corresponding image parts and yields excellent performance over a much larger range of surface slants than the standard model. The results suggest that, rather than serving as disparity detectors, multiple binocular complex cells take part in the computation of inter-ocular similarity, and that visual systems are likely to postpone commitment to particular binocular disparities until later stages in the visual process.

Selectivity for three-dimensional contours and surfaces in the anterior intraparietal area

The macaque anterior intraparietal area (AIP) is crucial for visually guided grasping. AIP neurons respond during the visual presentation of real-world objects and encode the depth profile of disparity-defined curved surfaces. We investigated the neural representation of curved surfaces in AIP using a stimulus-reduction approach. The stimuli consisted of three-dimensional (3-D) shapes curved along the horizontal axis, the vertical axis, or both the horizontal and the vertical axes of the shape. The depth profile was defined solely by binocular disparity that varied along either the boundary or the surface of the shape or along both the boundary and the surface of the shape. The majority of AIP neurons were selective for curved boundaries along the horizontal or the vertical axis, and neural selectivity emerged at short latencies. Stimuli in which disparity varied only along the surface of the shape (with zero disparity on the boundaries) evoked selectivity in a smaller proportion of AIP neurons and at considerably longer latencies. AIP neurons were not selective for 3-D surfaces composed of anticorrelated disparities. Thus the neural selectivity for object depth profile in AIP is present when only the boundary is curved in depth, but not for disparity in anticorrelated stereograms.

Bringing the real world into the fMRI scanner: repetition effects for pictures versus real objects

Our understanding of the neural underpinnings of perception is largely built upon studies employing 2-dimensional (2D) planar images. Here we used slow event-related functional imaging in humans to examine whether neural populations show a characteristic repetition-related change in haemodynamic response for real-world 3-dimensional (3D) objects, an effect commonly observed using 2D images. As expected, trials involving 2D pictures of objects produced robust repetition effects within classic object-selective cortical regions along the ventral and dorsal visual processing streams. Surprisingly, however, repetition effects were weak, if not absent on trials involving the 3D objects. These results suggest that the neural mechanisms involved in processing real objects may therefore be distinct from those that arise when we encounter a 2D representation of the same items. These preliminary results suggest the need for further research with ecologically valid stimuli in other imaging designs to broaden our understanding of the neural mechanisms underlying human vision.

An investigation of surface reconstruction from binocular disparity based on standard regularization theory: comparison between "membrane" and "thin-plate" potential energy models

Visible surfaces of three-dimensional objects are reconstructed from two-dimensional retinal images in the early stages of human visual processing. In the computational model of surface reconstruction based on the standard regularization theory, an energy function is minimized. Two types of model have been proposed, called "membrane" and "thin-plate" after their function formulas, in which the first or the second derivative of depth information is used. In this study, the threshold of surface reconstruction from binocular disparity was investigated using a sparse random dot stereogram, and the predictive accuracy of these models was evaluated. It was found that the thin-plate model reconstructed surfaces more accurately than the membrane model and showed good agreement with experimental results. The likelihood that these models imitate human processing of visual information is discussed in terms of the size of receptive fields in the visual pathways of the human cortex.

Unmixing binocular signals

Incompatible images presented to the two eyes lead to perceptual oscillations in which one image at a time is visible. Early models portrayed this binocular rivalry as involving reciprocal inhibition between monocular representations of images, occurring at an early visual stage prior to binocular mixing. However, psychophysical experiments found conditions where rivalry could also occur at a higher, more abstract level of representation. In those cases, the rivalry was between image representations dissociated from eye-of-origin information, rather than between monocular representations from the two eyes. Moreover, neurophysiological recordings found the strongest rivalry correlate in inferotemporal cortex, a high-level, predominantly binocular visual area involved in object recognition, rather than early visual structures. An unresolved issue is how can the separate identities of the two images be maintained after binocular mixing in order for rivalry to be possible at higher levels? Here we demonstrate that after the two images are mixed, they can be unmixed at any subsequent stage using a physiologically plausible non-linear signal-processing algorithm, non-negative matrix factorization, previously proposed for parsing object parts during object recognition. The possibility that unmixed left and right images can be regenerated at late stages within the visual system provides a mechanism for creating various binocular representations and interactions de novo in different cortical areas for different purposes, rather than inheriting then from early areas. This is a clear example how non-linear algorithms can lead to highly non-intuitive behavior in neural information processing.

A distinct representation of three-dimensional shape in macaque anterior intraparietal area: fast, metric, and coarse

Differences in the horizontal positions of retinal images--binocular disparity--provide important cues for three-dimensional object recognition and manipulation. We investigated the neural coding of three-dimensional shape defined by disparity in anterior intraparietal (AIP) area. Robust selectivity for disparity-defined slanted and curved surfaces was observed in a high proportion of AIP neurons, emerging at relatively short latencies. The large majority of AIP neurons preserved their three-dimensional shape preference over different positions in depth, a hallmark of higher-order disparity selectivity. Yet both stimulus type (concave-convex) and position in depth could be reliably decoded from the AIP responses. The neural coding of three-dimensional shape was based on first-order (slanted surfaces) and second-order (curved surfaces) disparity selectivity. Many AIP neurons tolerated the presence of disparity discontinuities in the stimulus, but the population of AIP neurons provided reliable information on the degree of curvedness of the stimulus. Finally, AIP neurons preserved their three-dimensional shape preference over different positions in the frontoparallel plane. Thus, AIP neurons extract or have access to three-dimensional object information defined by binocular disparity, consistent with previous functional magnetic resonance imaging data. Unlike the known representation of three-dimensional shape in inferior temporal cortex, the neural representation in AIP appears to emphasize object parameters required for the planning of grasping movements.

The phenomenon of task-irrelevant perceptual learning

Task-irrelevant perceptual learning (TIPL) has captured a growing interest in the field of perceptual learning. The basic phenomenon is that stimulus features that are irrelevant to a subject's task (i.e. convey no useful information to that task) can be learned due to their consistent presentation during task-performance. Here we review recent research on TIPL and focus on two key aspects of TIPL; (1) the mechanisms gating learning in TIPL, and (2) what is learned through TIPL. We show that TIPL is gated by learning signals that are triggered from task processing or by rewards. These learning signals operate to enhance processing of individual stimulus features and appear to result in plasticity in early stages of visual processing. Furthermore, we discuss recent research that demonstrates that TIPL is not in opposition to theories of attention but instead that TIPL operates in concert with attention. Where attentional learning is best to enhance (or suppress) processing of stimuli of known task relevance, TIPL serves to enhance perception of stimuli that are originally inadequately processed by the brain.

Parietal reach region encodes reach depth using retinal disparity and vergence angle signals

Performing a visually guided reach requires the ability to perceive the egocentric distance of a target in three-dimensional space. Previous studies have shown that the parietal reach region (PRR) encodes the two-dimensional location of frontoparallel targets in an eye-centered reference frame. To investigate how a reach target is represented in three dimensions, we recorded the spiking activity of PRR neurons from two rhesus macaques trained to fixate and perform memory reaches to targets at different depths. Reach and fixation targets were configured to explore whether neural activity directly reflects egocentric distance as the amplitude of the required motor command, which is the absolute depth of the target, or rather the relative depth of the target with reference to fixation depth. We show that planning activity in PRR represents the depth of the reach target as a function of disparity and fixation depth, the spatial parameters important for encoding the depth of a reach goal in an eye centered reference frame. The strength of modulation by disparity is maintained across fixation depth. Fixation depth gain modulates disparity tuning while preserving the location of peak tuning features in PRR neurons. The results show that individual PRR neurons code depth with respect to the fixation point, that is, in eye centered coordinates. However, because the activity is gain modulated by vergence angle, the absolute depth can be decoded from the population activity.

Parallel processing strategies of the primate visual system

Incoming sensory information is sent to the brain along modality-specific channels corresponding to the five senses. Each of these channels further parses the incoming signals into parallel streams to provide a compact, efficient input to the brain. Ultimately, these parallel input signals must be elaborated on and integrated in the cortex to provide a unified and coherent percept. Recent studies in the primate visual cortex have greatly contributed to our understanding of how this goal is accomplished. Multiple strategies including retinal tiling, hierarchical and parallel processing and modularity, defined spatially and by cell type-specific connectivity, are used by the visual system to recover the intricate detail of our visual surroundings.

Rewards evoke learning of unconsciously processed visual stimuli in adult humans

The study of human learning is complicated by the myriad of processing elements involved in conducting any behavioral task. In the case of visual perceptual learning, there has been significant controversy regarding the task processes that guide the formation of this learning. However, there is a developing consensus that top-down, task-related factors are required for such learning to take place. Here we challenge this idea by use of a novel procedure in which human participants, who were deprived of food and water, passively viewed visual stimuli while receiving occasional drops of water as rewards. Visual orientation stimuli, which were temporally paired with the liquid rewards, were viewed monocularly and rendered imperceptible by continuously flashing contour-rich patterns to the other eye. Results show that visual learning can be formed in human adults through stimulus-reward pairing in the absence of a task and without awareness of the stimulus presentation or reward contingencies.

Multivoxel pattern selectivity for perceptually relevant binocular disparities in the human brain

Processing of binocular disparity is thought to be widespread throughout cortex, highlighting its importance for perception and action. Yet the computations and functional roles underlying this activity across areas remain largely unknown. Here, we trace the neural representations mediating depth perception across human brain areas using multivariate analysis methods and high-resolution imaging. Presenting disparity-defined planes, we determine functional magnetic resonance imaging (fMRI) selectivity to near versus far depth positions. First, we test the perceptual relevance of this selectivity, comparing the pattern-based decoding of fMRI responses evoked by random dot stereograms that support depth perception (correlated RDS) with the decoding of stimuli containing disparities to which the perceptual system is blind (anticorrelated RDS). Preferential disparity selectivity for correlated stimuli in dorsal (visual and parietal) areas and higher ventral area LO (lateral occipital area) suggests encoding of perceptually relevant information, in contrast to early (V1, V2) and intermediate ventral (V3v, V4) visual cortical areas that show similar selectivity for both correlated and anticorrelated stimuli. Second, manipulating disparity parametrically, we show that dorsal areas encode the metric disparity structure of the viewed stimuli (i.e., disparity magnitude), whereas ventral area LO appears to represent depth position in a categorical manner (i.e., disparity sign). Our findings suggest that activity in both visual streams is commensurate with the use of disparity for depth perception but the neural computations may differ. Intriguingly, perceptually relevant responses in the dorsal stream are tuned to disparity content and emerge at a comparatively earlier stage than categorical representations for depth position in the ventral stream.

A neural code for three-dimensional object shape in macaque inferotemporal cortex

Previous investigations of the neural code for complex object shape have focused on two-dimensional (2D) pattern representation. This might be the primary mode for object vision, based on simplicity and direct relation to the retinal image. In contrast, 3D shape representation requires higher-dimensional coding based on extensive computation. Here, for the first time, we provide evidence of an explicit neural code for complex 3D object shape. We used a novel evolutionary stimulus strategy and linear/nonlinear response models to characterize 3D shape responses in macaque monkey inferotemporal cortex (IT). We found widespread tuning for 3D spatial configurations of surface fragments characterized by their 3D orientations and joint principal curvatures. Configural representation of 3D shape could provide specific knowledge of object structure critical for guidance of complex physical interactions and evaluation of object functionality and utility.

Higher order visual processing in macaque extrastriate cortex

The extrastriate cortex of primates encompasses a substantial portion of the cerebral cortex and is devoted to the higher order processing of visual signals and their dispatch to other parts of the brain. A first step towards the understanding of the function of this cortical tissue is a description of the selectivities of the various neuronal populations for higher order aspects of the image. These selectivities present in the various extrastriate areas support many diverse representations of the scene before the subject. The list of the known selectivities includes that for pattern direction and speed gradients in middle temporal/V5 area; for heading in medial superior temporal visual area, dorsal part; for orientation of nonluminance contours in V2 and V4; for curved boundary fragments in V4 and shape parts in infero-temporal area (IT); and for curvature and orientation in depth from disparity in IT and CIP. The most common putative mechanism for generating such emergent selectivity is the pattern of excitatory and inhibitory linear inputs from the afferent area combined with nonlinear mechanisms in the afferent and receiving area.

A computational model of perceptual grouping and 3D surface completion in the mime effect

We propose a computational model of perceptual grouping for explaining the 3D shape representation of an illusory percept called "mime effect." This effect is associated with the generation of an illusory, volumetric perception that can be induced by particular distributions of inducing stimuli such as cones, whose orientations affect the stability of illusory perception. The authors have attempted to explain the characteristics of the shape representation of the mime effect using a neural network model that consists of four types of cells-encoding (E), normalizing (N), energetic (EN), and geometric (G) cells. E cells represent both the positions and orientations of inducing stimuli and the mime-effect shape, and N cells regulate the activity of E cells. The interactions of E cells generate dynamics whose mode indicates the stability of illusory perception; a stable dynamics mode indicates a stable perception, whereas a chaotic dynamics mode indicates an unstable perception. EN cells compute the Liapunov energetic exponent (LEE) from an energy function of the system of E cells. The stable and chaotic dynamics modes are identified by strictly negative and strictly positive values of LEE, respectively. In case of stability, G cells perform a particular surface interpolation for completing the mime effect shape. The authors confirm the model behaviour by means of computer-simulated experiments. The relation between the model behaviour and the shape representation in the human brain is also discussed.

Long-term speeding in perceptual switches mediated by attention-dependent plasticity in cortical visual processing

Binocular rivalry has been extensively studied to understand the mechanisms that control switches in visual awareness and much has been revealed about the contributions of stimulus and cognitive factors. Because visual processes are fundamentally adaptive, however, it is also important to understand how experience alters the dynamics of perceptual switches. When observers viewed binocular rivalry repeatedly over many days, the rate of perceptual switches increased as much as 3-fold. This long-term rivalry speeding exhibited a pattern of image-feature specificity that ruled out primary contributions from strategic and nonsensory factors and implicated neural plasticity occurring in both low- and high-level visual processes in the ventral stream. Furthermore, the speeding occurred only when the rivaling patterns were voluntarily attended, suggesting that the underlying neural plasticity selectively engages when stimuli are behaviorally relevant. Long-term rivalry speeding may thus reflect broader mechanisms that facilitate quick assessments of signals that contain multiple behaviorally relevant interpretations.

Disparity channels in early vision

The past decade has seen a dramatic increase in our knowledge of the neural basis of stereopsis. New cortical areas have been found to represent binocular disparities, new representations of disparity information (e.g., relative disparity signals) have been uncovered, the first topographic maps of disparity have been measured, and the first causal links between neural activity and depth perception have been established. Equally exciting is the finding that training and experience affects how signals are channeled through different brain areas, a flexibility that may be crucial for learning, plasticity, and recovery of function. The collective efforts of several laboratories have established stereo vision as one of the most productive model systems for elucidating the neural basis of perception. Much remains to be learned about how the disparity signals that are initially encoded in primary visual cortex are routed to and processed by extrastriate areas to mediate the diverse capacities of three-dimensional vision that enhance our daily experience of the world.

Object recognition by artificial cortical maps

Object recognition is one of the most important functions of the human visual system, yet one of the least understood, this despite the fact that vision is certainly the most studied function of the brain. We understand relatively well how several processes in the cortical visual areas that support recognition capabilities take place, such as orientation discrimination and color constancy. This paper proposes a model of the development of object recognition capability, based on two main theoretical principles. The first is that recognition does not imply any sort of geometrical reconstruction, it is instead fully driven by the two dimensional view captured by the retina. The second assumption is that all the processing functions involved in recognition are not genetically determined or hardwired in neural circuits, but are the result of interactions between epigenetic influences and basic neural plasticity mechanisms. The model is organized in modules roughly related to the main visual biological areas, and is implemented mainly using the LISSOM architecture, a recent neural self-organizing map model that simulates the effects of intercortical lateral connections. This paper shows how recognition capabilities, similar to those found in brain ventral visual areas, can develop spontaneously by exposure to natural images in an artificial cortical model.

Cortical Connections of the Macaque Anterior Intraparietal (AIP) Area

We traced the cortical connections of the anterior intraparietal (AIP) area, which is known to play a crucial role in visuomotor transformations for grasping. AIP displayed major connections with 1) areas of the inferior parietal lobule convexity, the rostral part of the lateral intraparietal area and the SII region; 2) ventral visual stream areas of the lower bank of the superior temporal sulcus and the middle temporal gyrus; and 3) the premotor area F5 and prefrontal areas 46 and 12. Additional connections were observed with the caudal intraparietal area and the ventral part of the frontal eye field. This study suggests that visuomotor transformations for object-oriented actions, processed in AIP, rely not only on dorsal visual stream information related to the object's physical properties but also on ventral visual stream information related to object identity. The identification of direct anatomical connections with the inferotemporal cortex suggests that AIP also has a unique role in linking the parietofrontal network of areas involved in sensorimotor transformations for grasping with areas involved in object recognition. Thus, AIP could represent a crucial node in a cortical circuit in which hand-related sensory and motor signals gain access to representations of object identity for tactile object recognition.

Representation of stereoscopic depth based on relative disparity in macaque area V4

Stereoscopic vision is characterized by greater visual acuity when a background feature serves as a reference. When a reference is present, the perceived depth of an object is predominantly dependent on this reference. Neural representations of stereoscopic depth are expected to have a relative frame of reference. The conversion of absolute disparity encoded in area V1 to relative disparity begins in area V2, although the information encoded in this area appears to be insufficient for stereopsis. This study examines whether relative disparity is encoded in a higher cortical area. We recorded the responses of V4 neurons from macaque monkeys to various combinations of the absolute disparities of two features: the center patch and surrounding annulus of a dynamic random-dot stereogram. We analyzed the effects of the disparity of the surrounding annulus on the tuning for the disparity of the center patch; the tuning curves of relative-disparity-selective neurons for disparities of the center patch should shift with changes in the disparity of the surrounding annulus. Most V4 tuning curves exhibited significant shifts. The magnitudes of the shifts were larger than those reported for V2 neurons and smaller than that expected for an ideal relative-disparity-selective cell. No correlation was found between the shift magnitude and the degree of size suppression, suggesting that the two phenomena are not the result of a common mechanism. Our results suggest that the coding of relative disparity advances as information flows along the cortical pathway that includes areas V2 and V4.

Neural correlates of disparity-defined shape discrimination in the human brain

Binocular disparity, the slight differences between the images registered by our two eyes, provides an important cue when estimating the three-dimensional (3D) structure of the complex environment we inhabit. Sensitivity to binocular disparity is evident at multiple levels of the visual hierarchy in the primate brain, from early visual cortex to parietal and temporal areas. However, the relationship between activity in these areas and key perceptual functions that exploit disparity information for 3D shape perception remains an important open question. Here we investigate the link between human cortical activity and the perception of disparity-defined shape, measuring fMRI responses concurrently with psychophysical shape judgments. We parametrically degraded the coherence of shapes by shuffling the spatial position of dots whose disparity defined the 3D structure and investigated the effect of this stimulus manipulation on both cortical activity and shape discrimination. We report significant relationships between shape coherence and fMRI response in both dorsal (V3, hMT+/V5) and ventral (LOC) visual areas that correspond to the observers' discrimination performance. In contrast to previous suggestions of a dichotomy of disparity-related processes in the ventral and dorsal streams, these findings are consistent with proposed interactions between these pathways that may mediate a continuum of processes important in perceiving 3D shape from coarse contour segmentation to fine curvature estimation.

Extracting 3D structure from disparity

The neural mechanisms of stereoscopic 3D shape perception have only recently been investigated. Here we review the two cortical regions in which these mechanisms have been studied so far in macaques: a small subpart of inferotemporal cortex called TEs, and the caudal intraparietal (CIP) region. Neurons in TEs respond selectively to the orientation and curvature in depth of stereoscopic surfaces and this region provides a detailed 3D shape description of surface boundaries and surface content. This description is evoked only by binocular stimuli in which subjects see depth and it does not vary if depth is specified by different cues. Neurons in CIP are a selective for orientation in depth of surfaces and elongated objects, and their responses are also unaffected by changes in depth cues. Thus, stereoscopic 3D shape is processed in both the dorsal, occipito-parietal and the ventral, occipito-temporal streams.

Linking neural representation to function in stereoscopic depth perception: roles of the middle temporal area in coarse versus fine disparity discrimination

Neurons selective for binocular disparity form the neural substrate for stereoscopic depth perception and are found in several areas of primate visual cortex. Presumably, multiple representations of disparity exist to serve different functions, but the specific contributions of different visual areas to depth perception remain poorly understood. We examine this issue by comparing the contributions of the middle temporal (MT) area to performance of two depth discrimination tasks: a "coarse" task that involves discrimination between absolute disparities in the presence of noise, and a "fine" task that involves discrimination of very small differences in relative disparity between two stimuli in the absence of noise. In the fine task, we find that electrical microstimulation of MT does not affect perceptual decisions, although many individual MT neurons have sufficient sensitivity to account for behavioral performance. In contrast, microstimulation at the same recording sites does bias depth percepts in the coarse task. We hypothesized that these results may be explained by the fact that MT neurons do not represent relative disparity signals that are thought to be essential for the fine task. This hypothesis was supported by single-unit recordings that show that MT neurons signal absolute, but not relative, disparities in a stimulus configuration similar to that used in the fine task. This work establishes a link between the neural representation of disparity in MT and the functional contributions of this area to depth perception.

Neural correlates of fine depth discrimination in monkey inferior temporal cortex

Binocular disparity is an important visual cue that gives rise to the perception of depth. Disparity signals are widely spread across the visual cortex, but their relative role is poorly understood. Here, we addressed the correlation between the responses of disparity-selective neurons in the occipitotemporal (ventral) visual pathway and the behavioral discrimination of stereoscopic depth. We recorded activity of disparity-selective neurons in the inferior temporal cortex (IT) while monkeys were engaged in a fine stereoscopic depth discrimination (stereoacuity) task. We found that trial-to-trial fluctuations in neuronal responses correlated with the monkey's perceptual choice. We suggest that disparity signals in the IT, located in the ventral visual pathway, are functionally linked to the discrimination of fine-grain depth.

At what stage of neural processing do perspective depth cues make a difference?

The present study investigated the cortical processing of three-dimensional (3D) perspective cues in humans, to determine how the brain computes depth from a bidimensional retinal image. We recorded visual evoked potentials in 12 subjects in response to flat and in-perspective stimuli, which evoked biphasic potentials over posterior electrodes. The first, positive component (P1, at 90 ms) was not sensitive to perspective, while the second, negative peak (N1 at approximately 150 ms) was significantly larger for 3D stimuli, regardless of attention. The amplitude increase due to perspective was seen on all posterior electrodes, but was largest over the right hemisphere, particularly at parietal sites. Source modeling low-resolution electromagnetic tomography (LORETA) confirmed that among the different areas participating in two- and three-dimensional stimuli processing, the right parietal source is the most enhanced by perspective depth cues. We conclude that the extraction of depth from perspective cues occurs at a second level of stimulus processing, by increasing the activity of the regions involved in 2D stimuli processing, particularly in the right hemisphere, possibly through feedback loops from higher cortical areas. These modulations would participate in the fine-tuned analysis of the 3D features of stimuli.

Disparity-Tuning Characteristics of Neuronal Responses to Dynamic Random-Dot Stereograms in Macaque Visual Area V4

Stereo processing begins in the striate cortex and involves several extrastriate visual areas. We quantitatively analyzed the disparity-tuning characteristics of neurons in area V4 of awake, fixating monkeys. Approximately half of the analyzed V4 cells were tuned for horizontal binocular disparities embedded in dynamic random-dot stereograms (RDSs). Their response preferences were strongly biased for crossed disparities. To characterize the disparity-tuning profile, we fitted a Gabor function to the disparity-tuning data. The distribution of V4 cells showed a single dense cluster in a joint parameter space of the center and the phase parameters of the fitted Gabor function; most V4 neurons were maximally sensitive to fine stereoscopic depth increments near zero disparity. Comparing single-cell responses with background multiunit responses at the same sites showed that disparity-sensitive cells were clustered within V4 and that nearby cells possessed similar preferred disparities. Consistent with a recent report by Hegdé and Van Essen, the disparity tuning for an RDS drastically differed from that for a solid-figure stereogram (SFS). Disparity-tuning curves were generally broader for SFSs than for RDSs, and there was no correlation between the fitted Gabor functions' amplitudes, widths, or peaks for the two types of stereograms. The differences were partially attributable to shifts in the monocular images of an SFS. Our results suggest that the representation of stereoscopic depth in V4 is suited for detecting fine structural features protruding from a background. The representation is not generic and differs when the stimulus is broad-band noise or a solid figure.