Role of primate visual area V4 in the processing of 3-D shape characteristics defined by disparity

How Shape Perception Works, in Two Dimensions and Three Dimensions

The ventral visual pathway transforms retinal images into neural representations that support object understanding, including exquisite appreciation of precise 2D pattern shape and 3D volumetric shape. We articulate a framework for understanding the goals of this transformation and how they are achieved by neural coding at successive ventral pathway stages. The critical goals are (a) radical compression to make shape information communicable across axonal bundles and storable in memory, (b) explicit coding to make shape information easily readable by the rest of the brain and thus accessible for cognition and behavioral control, and (c) representational stability to maintain consistent perception across highly variable viewing conditions. We describe how each transformational step in ventral pathway vision serves one or more of these goals. This three-goal framework unifies discoveries about ventral shape processing into a neural explanation for our remarkable experience of shape as a vivid, richly detailed aspect of the natural world.

Hue and orientation pinwheels in macaque area V4

Area V4 is an intermediate-level area of the macaque visual cortical hierarchy that serves key functions in visual processing by integrating inputs from lower areas such as V1 and V2 and providing feedforward inputs to many higher cortical areas. Previous V4 imaging studies have focused on differential responses to color, orientation, disparity, and motion stimuli, but many details of the spatial organization of significant hue and orientation tuning have not been fully described. We used support vector machine (SVM) decoding of intrinsic cortical single-condition responses to generate high-resolution maps of hue and orientation tuning and to describe the organization of hue and orientation pinwheels in V4. Like V1 and V2, V4 contains maps of orientation that are organized as pinwheels. V4 also contains maps of hue that are organized as pinwheels, whose circular organization more closely represents the perception of hue than is observed in antecedent cortical areas. Unlike V1, where orientation is continuously mapped across the surface, V4 hue and orientation pinwheels are organized in limited numbers of pinwheel sequences. The organization of these sequences and the distance between pinwheels may provide insight into the functional organization of V4. Regions significantly tuned for hue occupy roughly four times that of the orientation, are largely separated from each other, and overlap by roughly 5%. This spatial organization is largely consistent with segregated inputs arising from V2 thin and interstripes. This modular organization of V4 suggests that further integration of color and shape might occur in higher areas in inferotemporal cortical.NEW & NOTEWORTHY The representation of hue and orientation in macaque monkey area V4 was determined by intrinsic cortical imaging of responses to isoluminant hues and achromatic grating stimuli. Vector summation of support vector machine (SVM) decoded single-condition responses was used to generate hue and orientation maps that, like V1 orientation maps, were both characterized by distinct pinwheel patterns. These data suggest that pinwheels are an important structure to represent different stimulus features across multiple visual cortical areas.

Neuronal Representations Supporting Three-Dimensional Vision in Nonhuman Primates

The visual system must reconstruct the dynamic, three-dimensional (3D) world from ambiguous two-dimensional (2D) retinal images. In this review, we synthesize current literature on how the visual system of nonhuman primates performs this transformation through multiple channels within the classically defined dorsal (where) and ventral (what) pathways. Each of these channels is specialized for processing different 3D features (e.g., the shape, orientation, or motion of objects, or the larger scene structure). Despite the common goal of 3D reconstruction, neurocomputational differences between the channels impose distinct information-limiting constraints on perception. Convergent evidence further points to the little-studied area V3A as a potential branchpoint from which multiple 3D-fugal processing channels diverge. We speculate that the expansion of V3A in humans may have supported the emergence of advanced 3D spatial reasoning skills. Lastly, we discuss future directions for exploring 3D information transmission across brain areas and experimental approaches that can further advance the understanding of 3D vision.

Parallel processing, hierarchical transformations, and sensorimotor associations along the 'where' pathway

Visually guided behaviors require the brain to transform ambiguous retinal images into object-level spatial representations and implement sensorimotor transformations. These processes are supported by the dorsal ‘where’ pathway. However, the specific functional contributions of areas along this pathway remain elusive due in part to methodological differences across studies. We previously showed that macaque caudal intraparietal (CIP) area neurons possess robust 3D visual representations, carry choice- and saccade-related activity, and exhibit experience-dependent sensorimotor associations (Chang et al., 2020b). Here, we used a common experimental design to reveal parallel processing, hierarchical transformations, and the formation of sensorimotor associations along the ‘where’ pathway by extending the investigation to V3A, a major feedforward input to CIP. Higher-level 3D representations and choice-related activity were more prevalent in CIP than V3A. Both areas contained saccade-related activity that predicted the direction/timing of eye movements. Intriguingly, the time course of saccade-related activity in CIP aligned with the temporally integrated V3A output. Sensorimotor associations between 3D orientation and saccade direction preferences were stronger in CIP than V3A, and moderated by choice signals in both areas. Together, the results explicate parallel representations, hierarchical transformations, and functional associations of visual and saccade-related signals at a key juncture in the ‘where’ pathway.

Modulations of depth responses in the human brain by object context: Does biological relevance matter?

Depth sensitivity has been shown to be modulated by object context (plausibility). It is possible that it is behavioural relevance rather than object plausibility per se which drives this effect. Here, we manipulated the biological relevance of objects (face or a non-face) and tested whether object relevance affects behavioural sensitivity and neural responses to depth-position. In a first experiment, we presented human observers with disparity-defined faces and non-faces, and observers were asked to judge the depth position of the target under signal-noise and clear (fine) task conditions. In the second experiment, we concurrently measured behavioural and fMRI responses to depth. We found that behavioural performance varied across stimulus conditions such that they were significantly worse for the upright face than the inverted face and the random shape in the SNR task, but worse for the random shape than the upright face in the feature task. Pattern analysis of fMRI responses revealed that activity of FFA was distinctly different during depth judgments of the upright face versus the other two stimuli, with its responses (and to a stronger extent, those of V3) appearing functionally-relevant to behavioural performance. We speculate that FFA is not only involved in object analysis, but exerts considerable influence on stereoscopic mechanisms as early as in V3 based on a broader appreciation of the stimulus' behavioural relevance.Significance StatementWe asked how disparity sensitivity is modulated by object (biological) relevance using behavioural and neuroimaging paradigms. We show that behavioural sensitivity to depth-position changes in biological (face) vs non-biological (random surface) contexts, and that these changes are task-dependent. Imaging results highlight a potentially key role of the fusiform region for governing the modulation of stereo encoding by object relevance. These findings highlight powerful interactions between object recognition mechanisms and stereoencoding, such that the utility of disparity information may be up/down weighed depending on the biological relevance of the object.

Specialized contributions of mid-tier stages of dorsal and ventral pathways to stereoscopic processing in macaque

The division of labor between the dorsal and ventral visual pathways has been well studied, but not often with direct comparison at the single-neuron resolution with matched stimuli. Here we directly compared how single neurons in MT and V4, mid-tier areas of the two pathways, process binocular disparity, a powerful cue for 3D perception and actions. We found that MT neurons transmitted disparity signals more quickly and robustly, whereas V4 or its upstream neurons transformed the signals into sophisticated representations more prominently. Therefore, signaling speed and robustness were traded for transformation between the dorsal and ventral pathways. The key factor in this tradeoff was disparity-tuning shape: V4 neurons had more even-symmetric tuning than MT neurons. Moreover, the tuning symmetry predicted the degree of signal transformation across neurons similarly within each area, implying a general role of tuning symmetry in the stereoscopic processing by the two pathways.

Early Emergence of Solid Shape Coding in Natural and Deep Network Vision

Area V4 is the first object-specific processing stage in the ventral visual pathway, just as area MT is the first motion-specific processing stage in the dorsal pathway. For almost 50 years, coding of object shape in V4 has been studied and conceived in terms of flat pattern processing, given its early position in the transformation of 2D visual images. Here, however, in awake monkey recording experiments, we found that roughly half of V4 neurons are more tuned and responsive to solid, 3D shape-in-depth, as conveyed by shading, specularity, reflection, refraction, or disparity cues in images. Using 2-photon functional microscopy, we found that flat- and solid-preferring neurons were segregated into separate modules across the surface of area V4. These findings should impact early shape-processing theories and models, which have focused on 2D pattern processing. In fact, our analyses of early object processing in AlexNet, a standard visual deep network, revealed a similar distribution of sensitivities to flat and solid shape in layer 3. Early processing of solid shape, in parallel with flat shape, could represent a computational advantage discovered by both primate brain evolution and deep-network training.

Optimized but Not Maximized Cue Integration for 3D Visual Perception

Reconstructing three-dimensional (3D) scenes from two-dimensional (2D) retinal images is an ill-posed problem. Despite this, 3D perception of the world based on 2D retinal images is seemingly accurate and precise. The integration of distinct visual cues is essential for robust 3D perception in humans, but it is unclear whether this is true for non-human primates (NHPs). Here, we assessed 3D perception in macaque monkeys using a planar surface orientation discrimination task. Perception was accurate across a wide range of spatial poses (orientations and distances), but precision was highly dependent on the plane's pose. The monkeys achieved robust 3D perception by dynamically reweighting the integration of stereoscopic and perspective cues according to their pose-dependent reliabilities. Errors in performance could be explained by a prior resembling the 3D orientation statistics of natural scenes. We used neural network simulations based on 3D orientation-selective neurons recorded from the same monkeys to assess how neural computation might constrain perception. The perceptual data were consistent with a model in which the responses of two independent neuronal populations representing stereoscopic cues and perspective cues (with perspective signals from the two eyes combined using nonlinear canonical computations) were optimally integrated through linear summation. Perception of combined-cue stimuli was optimal given this architecture. However, an alternative architecture in which stereoscopic cues, left eye perspective cues, and right eye perspective cues were represented by three independent populations yielded two times greater precision than the monkeys. This result suggests that, due to canonical computations, cue integration for 3D perception is optimized but not maximized.

Stochastic resonance model of synaesthesia

In synaesthesia, stimulation of one sensory modality evokes additional experiences in another modality (e.g. sounds evoking colours). Along with these cross-sensory experiences, there are several cognitive and perceptual differences between synaesthetes and non-synaesthetes. For example, synaesthetes demonstrate enhanced imagery, increased cortical excitability and greater perceptual sensitivity in the concurrent modality. Previous models suggest that synaesthesia results from increased connectivity between corresponding sensory regions or disinhibited feedback from higher cortical areas. While these models explain how one sense can evoke qualitative experiences in another, they fail to predict the broader phenotype of differences observed in synaesthetes. Here, we propose a novel model of synaesthesia based on the principles of stochastic resonance. Specifically, we hypothesize that synaesthetes have greater neural noise in sensory regions, which allows pre-existing multisensory pathways to elicit supra-threshold activation (i.e. synaesthetic experiences). The strengths of this model are (a) it predicts the broader cognitive and perceptual differences in synaesthetes, (b) it provides a unified framework linking developmental and induced synaesthesias, and (c) it explains why synaesthetic associations are inconsistent at onset but stabilize over time. We review research consistent with this model and propose future studies to test its limits. This article is part of a discussion meeting issue 'Bridging senses: novel insights from synaesthesia'.

Single-cell responses to three-dimensional structure in a functionally defined patch in macaque area TEO

Both dorsal and ventral visual pathways harbor several areas sensitive to gradients of binocular disparity (i.e., higher-order disparity). Although a wealth of information exists about disparity processing in early visual (V1, V2, and V3) and end-stage areas, TE in the ventral stream, and the anterior intraparietal area (AIP) in the dorsal stream, little is known about midlevel area TEO in the ventral pathway. We recorded single-unit responses to disparity-defined curved stimuli in a functional magnetic resonance imaging (fMRI) activation elicited by curved surfaces compared with flat surfaces in the macaque area TEO. This fMRI activation contained a small proportion of disparity-selective neurons, with very few of them second-order disparity selective. Overall, this population of TEO neurons did not preserve its three-dimensional structure selectivity across positions in depth, indicating a lack of higher-order disparity selectivity, but showed stronger responses to flat surfaces than to curved surfaces, as predicted by the fMRI experiment. The receptive fields of the responsive TEO cells were relatively small and generally foveal. A linear support vector machine classifier showed that this population of disparity-selective TEO neurons contains reliable information about the sign of curvature and the position in depth of the stimulus. NEW & NOTEWORTHY We recorded in a part of the macaque area TEO that is activated more by curved surfaces than by flat surfaces at different disparities using the same stimuli. In contrast to previous studies, this functional magnetic resonance imaging-defined patch did not contain a large number of higher-order disparity-selective neurons. However, a linear support vector machine could reliably classify both the sign of the disparity gradient and the position in depth of the stimuli.

Representation of Rapid Image Sequences in V4 Networks

Natural viewing often consists of sequences of brief fixations to image patches of different structure. Whether and how briefly presented sequential stimuli are encoded in a temporal-position manner is poorly understood. Here, we performed multiple-electrode recordings in the visual cortex (area V4) of nonhuman primates (Macaca mulatta) viewing a sequence of 7 briefly flashed natural images, and measured correlations between the cue-triggered population response in the presence and absence of the stimulus. Surprisingly, we found significant correlations for images occurring at the beginning and the end of a sequence, but not for those in the middle. The correlation strength increased with stimulus exposure and favored the image position in the sequence rather than image identity. These results challenge the commonly held view that images are represented in visual cortex exclusively based on their informational content, and indicate that, in the absence of sensory information, neuronal populations exhibit reactivation of stimulus-evoked responses in a way that reflects temporal position within a stimulus sequence.

Binocular depth processing in the ventral visual pathway

One of the most powerful forms of depth perception capitalizes on the small relative displacements, or binocular disparities, in the images projected onto each eye. The brain employs these disparities to facilitate various computations, including sensori-motor transformations (reaching, grasping), scene segmentation and object recognition. In accordance with these different functions, disparity activates a large number of regions in the brain of both humans and monkeys. Here, we review how disparity processing evolves along different regions of the ventral visual pathway of macaques, emphasizing research based on both correlational and causal techniques. We will discuss the progression in the ventral pathway from a basic absolute disparity representation to a more complex three-dimensional shape code. We will show that, in the course of this evolution, the underlying neuronal activity becomes progressively more bound to the global perceptual experience. We argue that these observations most probably extend beyond disparity processing per se, and pertain to object processing in the ventral pathway in general. We conclude by posing some important unresolved questions whose answers may significantly advance the field, and broaden its scope.This article is part of the themed issue 'Vision in our three-dimensional world'.

Weighted parallel contributions of binocular correlation and match signals to conscious perception of depth

Binocular disparity is detected in the primary visual cortex by a process similar to calculation of local cross-correlation between left and right retinal images. As a consequence, correlation-based neural signals convey information about false disparities as well as the true disparity. The false responses in the initial disparity detectors are eliminated at later stages in order to encode only disparities of the features correctly matched between the two eyes. For a simple stimulus configuration, a feed-forward nonlinear process can transform the correlation signal into the match signal. For human observers, depth judgement is determined by a weighted sum of the correlation and match signals rather than depending solely on the latter. The relative weight changes with spatial and temporal parameters of the stimuli, allowing adaptive recruitment of the two computations under different visual circumstances. A full transformation from correlation-based to match-based representation occurs at the neuronal population level in cortical area V4 and manifests in single-neuron responses of inferior temporal and posterior parietal cortices. Neurons in area V5/MT represent disparity in a manner intermediate between the correlation and match signals. We propose that the correlation and match signals in these areas contribute to depth perception in a weighted, parallel manner.This article is part of the themed issue 'Vision in our three-dimensional world'.

Rapid learning in visual cortical networks

Although changes in brain activity during learning have been extensively examined at the single neuron level, the coding strategies employed by cell populations remain mysterious. We examined cell populations in macaque area V4 during a rapid form of perceptual learning that emerges within tens of minutes. Multiple single units and LFP responses were recorded as monkeys improved their performance in an image discrimination task. We show that the increase in behavioral performance during learning is predicted by a tight coordination of spike timing with local population activity. More spike-LFP theta synchronization is correlated with higher learning performance, while high-frequency synchronization is unrelated with changes in performance, but these changes were absent once learning had stabilized and stimuli became familiar, or in the absence of learning. These findings reveal a novel mechanism of plasticity in visual cortex by which elevated low-frequency synchronization between individual neurons and local population activity accompanies the improvement in performance during learning.

DOI:http://dx.doi.org/10.7554/eLife.08417.001

Throughout life, we learn and become better at many skills through repeated practice. However, how the brain cells enable us to adapt to changes in the environment and improve cognitive performance is poorly understood.

The activity of a neuron can be recorded as a ‘spike’ of electrical activity. In the nervous system, neurons work together in networks. If a group of neurons fire in a synchronized manner, waves of activity may be recorded from that brain region. One important issue in neuroscience is whether the spikes of individual neurons are synchronized with the local network activity. Indeed, it is generally believed that it is functionally important for individual cells to synchronize their responses to the waves of population activity.

The vast majority of studies aimed at understanding the behavior of neurons during learning have only recorded the activity of single neurons. This activity does not change much during learning, which suggests that learning may instead be encoded by the combined activity of a group of neurons. However, it is difficult to examine the same population of neurons as an animal practices and improves a skill. This is because the learning process typically takes longer than the length of time for which a single cell can be held in a stable condition and recorded from.

To overcome these limitations, Wang and Dragoi briefly flashed images at monkeys and trained them to report when the images have been rotated. Monkeys learn to do this within a single-training session, which allows the responses of the same group of neurons—found in a part of the brain called the mid-level visual cortex—to be recorded throughout the learning process.

Wang and Dragoi found that the improvement in behavioral performance during learning was accompanied by a tight synchronization between the spikes produced by individual neurons and the activity of groups of cells within a specific low-frequency band. This low-frequency activity had previously been linked to changes in the strength of functional connections between neurons in the hippocampus, which may be important for learning. The more synchronized this neural activity was, the better the monkeys were at the task. However, changes to the synchronization of spiking responses to local population activity in the higher frequency bands were unrelated to changes in performance. The changes to the level of synchronization were abolished once learning had stabilized and stimuli had become familiar.

Although Wang and Dragoi have found that the mid-level visual cortex neurons fire in a more synchronized way throughout learning, it remains to be confirmed whether these changes in synchronization are causally related to learning. Future studies could test whether this is the case by electrically or optically stimulating neurons so that their activity synchronizes with the local population activity, and investigating whether this manipulation improves learning ability.

DOI:http://dx.doi.org/10.7554/eLife.08417.002

Neuroimaging of amblyopia and binocular vision: a review

Amblyopia is a cerebral visual impairment considered to derive from abnormal visual experience (e.g., strabismus, anisometropia). Amblyopia, first considered as a monocular disorder, is now often seen as a primarily binocular disorder resulting in more and more studies examining the binocular deficits in the patients. The neural mechanisms of amblyopia are not completely understood even though they have been investigated with electrophysiological recordings in animal models and more recently with neuroimaging techniques in humans. In this review, we summarize the current knowledge about the brain regions that underlie the visual deficits associated with amblyopia with a focus on binocular vision using functional magnetic resonance imaging. The first studies focused on abnormal responses in the primary and secondary visual areas whereas recent evidence shows that there are also deficits at higher levels of the visual pathways within the parieto-occipital and temporal cortices. These higher level areas are part of the cortical network involved in 3D vision from binocular cues. Therefore, reduced responses in these areas could be related to the impaired binocular vision in amblyopic patients. Promising new binocular treatments might at least partially correct the activation in these areas. Future neuroimaging experiments could help to characterize the brain response changes associated with these treatments and help devise them.

The what, where and how of auditory-object perception

The fundamental perceptual unit in hearing is the 'auditory object'. Similar to visual objects, auditory objects are the computational result of the auditory system's capacity to detect, extract, segregate and group spectrotemporal regularities in the acoustic environment; the multitude of acoustic stimuli around us together form the auditory scene. However, unlike the visual scene, resolving the component objects within the auditory scene crucially depends on their temporal structure. Neural correlates of auditory objects are found throughout the auditory system. However, neural responses do not become correlated with a listener's perceptual reports until the level of the cortex. The roles of different neural structures and the contribution of different cognitive states to the perception of auditory objects are not yet fully understood.

Binocular stereoscopy in visual areas V-2, V-3, and V-3A of the macaque monkey

Over 40 years ago, Hubel and Wiesel gave a preliminary report of the first account of cells in monkey cerebral cortex selective for binocular disparity. The cells were located outside of V-1 within a region referred to then as "area 18." A full-length manuscript never followed, because the demarcation of the visual areas within this region had not been fully worked out. Here, we provide a full description of the physiological experiments and identify the locations of the recorded neurons using a contemporary atlas generated by functional magnetic resonance imaging; we also perform an independent analysis of the location of the neurons relative to an anatomical landmark (the base of the lunate sulcus) that is often coincident with the border between V-2 and V-3. Disparity-tuned cells resided not only in V-2, the area now synonymous with area 18, but also in V-3 and probably within V-3A. The recordings showed that the disparity-tuned cells were biased for near disparities, tended to prefer vertical orientations, clustered by disparity preference, and often required stimulation of both eyes to elicit responses, features strongly suggesting a role in stereoscopic depth perception.

Three-dimensional Shape Coding in Grasping Circuits: A Comparison between the Anterior Intraparietal Area and Ventral Premotor Area F5a

Depth information is necessary for adjusting the hand to the three-dimensional (3-D) shape of an object to grasp it. The transformation of visual information into appropriate distal motor commands is critically dependent on the anterior intraparietal area (AIP) and the ventral premotor cortex (area F5), particularly the F5p sector. Recent studies have demonstrated that both AIP and the F5a sector of the ventral premotor cortex contain neurons that respond selectively to disparity-defined 3-D shape. To investigate the neural coding of 3-D shape and the behavioral role of 3-D shape-selective neurons in these two areas, we recorded single-cell activity in AIP and F5a during passive fixation of curved surfaces and during grasping of real-world objects. Similar to those in AIP, F5a neurons were either first- or second-order disparity selective, frequently showed selectivity for discrete approximations of smoothly curved surfaces that contained disparity discontinuities, and exhibited mostly monotonic tuning for the degree of disparity variation. Furthermore, in both areas, 3-D shape-selective neurons were colocalized with neurons that were active during grasping of real-world objects. Thus, area AIP and F5a contain highly similar representations of 3-D shape, which is consistent with the proposed transfer of object information from AIP to the motor system through the ventral premotor cortex.

Image sequence reactivation in awake V4 networks

In the absence of sensory input, neuronal networks are far from being silent. Whether spontaneous changes in ongoing activity reflect previous sensory experience or stochastic fluctuations in brain activity is not well understood. Here we describe reactivation of stimulus-evoked activity in awake visual cortical networks. We found that continuous exposure to randomly flashed image sequences induces reactivation in macaque V4 cortical networks in the absence of visual stimulation. This reactivation of previously evoked activity is stimulus-specific, occurs only in the same temporal order as the original response, and strengthens with increased stimulus exposures. Importantly, cells exhibiting significant reactivation carry more information about the stimulus than cells that do not reactivate. These results demonstrate a surprising degree of experience-dependent plasticity in visual cortical networks as a result of repeated exposure to unattended information. We suggest that awake reactivation in visual cortex may underlie perceptual learning by passive stimulus exposure.

Neural activity in cortical area V4 underlies fine disparity discrimination

Primates are capable of discriminating depth with remarkable precision using binocular disparity. Neurons in area V4 are selective for relative disparity, which is the crucial visual cue for discrimination of fine disparity. Here, we investigated the contribution of V4 neurons to fine disparity discrimination. Monkeys discriminated whether the center disk of a dynamic random-dot stereogram was in front of or behind its surrounding annulus. We first behaviorally tested the reference frame of the disparity representation used for performing this task. After learning the task with a set of surround disparities, the monkey generalized its responses to untrained surround disparities, indicating that the perceptual decisions were generated from a disparity representation in a relative frame of reference. We then recorded single-unit responses from V4 while the monkeys performed the task. On average, neuronal thresholds were higher than the behavioral thresholds. The most sensitive neurons reached thresholds as low as the psychophysical thresholds. For subthreshold disparities, the monkeys made frequent errors. The variable decisions were predictable from the fluctuation in the neuronal responses. The predictions were based on a decision model in which each V4 neuron transmits the evidence for the disparity it prefers. We finally altered the disparity representation artificially by means of microstimulation to V4. The decisions were systematically biased when microstimulation boosted the V4 responses. The bias was toward the direction predicted from the decision model. We suggest that disparity signals carried by V4 neurons underlie precise discrimination of fine stereoscopic depth.

Toward a unified theory of visual area V4

Visual area V4 is a midtier cortical area in the ventral visual pathway. It is crucial for visual object recognition and has been a focus of many studies on visual attention. However, there is no unifying view of V4's role in visual processing. Neither is there an understanding of how its role in feature processing interfaces with its role in visual attention. This review captures our current knowledge of V4, largely derived from electrophysiological and imaging studies in the macaque monkey. Based on recent discovery of functionally specific domains in V4, we propose that the unifying function of V4 circuitry is to enable selective extraction of specific functional domain-based networks, whether it be by bottom-up specification of object features or by top-down attentionally driven selection.

Neurons in dorsal visual area V5/MT signal relative disparity

Judgments of visual depth rely crucially on the relative binocular disparity between two visual features. While areas of ventral visual cortex contain neurons that signal the relative disparity between spatially adjacent visual features, the same tests in dorsal visual areas yield little evidence for relative disparity selectivity. We investigated the sensitivity of neurons in dorsal visual area V5/MT of macaque monkeys to relative disparity, using two superimposed, transparent planes composed of dots moving in opposite directions. The separation of the planes in depth specifies their relative disparity, while absolute disparity can be altered independently by changing the binocular depth of the two planes with respect to the monkey's fixation point. Many V5/MT neurons were tuned to relative disparity, independent of the absolute disparities of the individual planes. For the two plane stimulus, neuronal responses were often linearly related to responses to the absolute disparity of each component plane presented individually, but some aspects of relative disparity tuning were not explained by linear combination. Selectivity for relative disparity could not predict whether neuronal firing was related to the monkeys' perceptual reports of the rotation direction of structure-from-motion figures centered on the plane of fixation. In sum, V5/MT neurons are not just selective for absolute disparity, but also code for relative disparity between visual features. This selectivity may be important for segmentation and depth order of moving visual features, particularly the processing of three-dimensional information in scenes viewed by an actively moving observer.

Representation of 3-D surface orientation by velocity and disparity gradient cues in area MT

Neural coding of the three-dimensional (3-D) orientation of planar surface patches may be an important intermediate step in constructing representations of complex 3-D surface structure. Spatial gradients of binocular disparity, image velocity, and texture provide potent cues to the 3-D orientation (tilt and slant) of planar surfaces. Previous studies have described neurons in both dorsal and ventral stream areas that are selective for surface tilt based on one or more of these gradient cues. However, relatively little is known about whether single neurons provide consistent information about surface orientation from multiple gradient cues. Moreover, it is unclear how neural responses to combinations of surface orientation cues are related to responses to the individual cues. We measured responses of middle temporal (MT) neurons to random dot stimuli that simulated planar surfaces at a variety of tilts and slants. Four cue conditions were tested: disparity, velocity, and texture gradients alone, as well as all three gradient cues combined. Many neurons showed robust tuning for surface tilt based on disparity and velocity gradients, with relatively little selectivity for texture gradients. Some neurons showed consistent tilt preferences for disparity and velocity cues, whereas others showed large discrepancies. Responses to the combined stimulus were generally well described as a weighted linear sum of responses to the individual cues, even when disparity and velocity preferences were discrepant. These findings suggest that area MT contains a rudimentary representation of 3-D surface orientation based on multiple cues, with single neurons implementing a simple cue integration rule.

The extraction of 3D shape in the visual system of human and nonhuman primates

Depth structure, the third dimension of object shape, is extracted from disparity, motion, texture, and shading in the optic array. Gradient-selective neurons play a key role in this process. Such neurons occur in CIP, AIP, TEs, and F5 (for first- or second-order disparity gradients), in MT/V5, in FST (for speed gradients), and in CIP and TEs (for texture gradients). Most of these regions are activated during magnetic resonance scanning in alert monkeys by comparing 3D conditions with the 2D controls for the different cues. Similarities in activation patterns of monkeys and humans tested with identical paradigms suggest that like gradient-selective neurons are found in corresponding human cortical areas. This view gains credence as the homologies between such areas become more evident. Furthermore, 3D shape-processing networks are similar in the two species, with the exception of the greater involvement of human posterior parietal cortex in the extraction of 3D shape from motion. Thus we can begin to understand how depth structure is extracted from motion, disparity, and texture in the primate brain, but the extraction of depth structure from shading and that of wire-like objects requires further scrutiny.

Neural computations underlying depth perception

Neural mechanisms underlying depth perception are reviewed with respect to three computational goals: determining surface depth order, gauging depth intervals, and representing 3D surface geometry and object shape. Accumulating evidence suggests that these three computational steps correspond to different stages of cortical processing. Early visual areas appear to be involved in depth ordering, while depth intervals, expressed in terms of relative disparities, are likely represented at intermediate stages. Finally, 3D surfaces appear to be processed in higher cortical areas, including an area in which individual neurons encode 3D surface geometry, and a population of these neurons may therefore represent 3D object shape. How these processes are integrated to form a coherent 3D percept of the world remains to be understood.

Coding of shape from shading in area V4 of the macaque monkey

Background

The shading of an object provides an important cue for recognition, especially for determining its 3D shape. However, neuronal mechanisms that allow the recovery of 3D shape from shading are poorly understood. The aim of our study was to determine the neuronal basis of 3D shape from shading coding in area V4 of the awake macaque monkey.

Results

We recorded the responses of V4 cells to stimuli presented parafoveally while the monkeys fixated a central spot. We used a set of stimuli made of 8 different 3D shapes illuminated from 4 directions (from above, the left, the right and below) and different 2D controls for each stimulus. The results show that V4 neurons present a broad selectivity to 3D shape and illumination direction, but without a preference for a unique illumination direction. However, 3D shape and illumination direction selectivities are correlated suggesting that V4 neurons can use the direction of illumination present in complex patterns of shading present on the surface of objects. In addition, a vast majority of V4 neurons (78%) have statistically different responses to the 3D and 2D versions of the stimuli, while responses to 3D are not systematically stronger than those to 2D controls. However, a hierarchical cluster analysis showed that the different classes of stimuli (3D, 2D controls) are clustered in the V4 cells response space suggesting a coding of 3D stimuli based on the population response. The different illumination directions also tend to be clustered in this space.

Conclusion

Together, these results show that area V4 participates, at the population level, in the coding of complex shape from the shading patterns coming from the illumination of the surface of corrugated objects. Hence V4 provides important information for one of the steps of cortical processing of the 3D aspect of objects in natural light environment.

A distinct representation of three-dimensional shape in macaque anterior intraparietal area: fast, metric, and coarse

Differences in the horizontal positions of retinal images--binocular disparity--provide important cues for three-dimensional object recognition and manipulation. We investigated the neural coding of three-dimensional shape defined by disparity in anterior intraparietal (AIP) area. Robust selectivity for disparity-defined slanted and curved surfaces was observed in a high proportion of AIP neurons, emerging at relatively short latencies. The large majority of AIP neurons preserved their three-dimensional shape preference over different positions in depth, a hallmark of higher-order disparity selectivity. Yet both stimulus type (concave-convex) and position in depth could be reliably decoded from the AIP responses. The neural coding of three-dimensional shape was based on first-order (slanted surfaces) and second-order (curved surfaces) disparity selectivity. Many AIP neurons tolerated the presence of disparity discontinuities in the stimulus, but the population of AIP neurons provided reliable information on the degree of curvedness of the stimulus. Finally, AIP neurons preserved their three-dimensional shape preference over different positions in the frontoparallel plane. Thus, AIP neurons extract or have access to three-dimensional object information defined by binocular disparity, consistent with previous functional magnetic resonance imaging data. Unlike the known representation of three-dimensional shape in inferior temporal cortex, the neural representation in AIP appears to emphasize object parameters required for the planning of grasping movements.

Representation of binocular surfaces by cortical neurons

Useful representations of the three-dimensional (3D) world go beyond assigning depth to individual points, building maps of surfaces and shapes. Studies in a wide range of extrastriate cortical areas have shown that single neurons show selective responses to 3D surfaces. The extent to which this advances the representation beyond that provided by the earliest binocular signals requires careful evaluation. We conclude that current data are not sufficient to identify distinctive contributions from different cortical areas to the binocular representation of 3D surfaces.

Higher order visual processing in macaque extrastriate cortex

The extrastriate cortex of primates encompasses a substantial portion of the cerebral cortex and is devoted to the higher order processing of visual signals and their dispatch to other parts of the brain. A first step towards the understanding of the function of this cortical tissue is a description of the selectivities of the various neuronal populations for higher order aspects of the image. These selectivities present in the various extrastriate areas support many diverse representations of the scene before the subject. The list of the known selectivities includes that for pattern direction and speed gradients in middle temporal/V5 area; for heading in medial superior temporal visual area, dorsal part; for orientation of nonluminance contours in V2 and V4; for curved boundary fragments in V4 and shape parts in infero-temporal area (IT); and for curvature and orientation in depth from disparity in IT and CIP. The most common putative mechanism for generating such emergent selectivity is the pattern of excitatory and inhibitory linear inputs from the afferent area combined with nonlinear mechanisms in the afferent and receiving area.

Representation of stereoscopic depth based on relative disparity in macaque area V4

Stereoscopic vision is characterized by greater visual acuity when a background feature serves as a reference. When a reference is present, the perceived depth of an object is predominantly dependent on this reference. Neural representations of stereoscopic depth are expected to have a relative frame of reference. The conversion of absolute disparity encoded in area V1 to relative disparity begins in area V2, although the information encoded in this area appears to be insufficient for stereopsis. This study examines whether relative disparity is encoded in a higher cortical area. We recorded the responses of V4 neurons from macaque monkeys to various combinations of the absolute disparities of two features: the center patch and surrounding annulus of a dynamic random-dot stereogram. We analyzed the effects of the disparity of the surrounding annulus on the tuning for the disparity of the center patch; the tuning curves of relative-disparity-selective neurons for disparities of the center patch should shift with changes in the disparity of the surrounding annulus. Most V4 tuning curves exhibited significant shifts. The magnitudes of the shifts were larger than those reported for V2 neurons and smaller than that expected for an ideal relative-disparity-selective cell. No correlation was found between the shift magnitude and the degree of size suppression, suggesting that the two phenomena are not the result of a common mechanism. Our results suggest that the coding of relative disparity advances as information flows along the cortical pathway that includes areas V2 and V4.

Temporal dynamics of 2D and 3D shape representation in macaque visual area V4

We studied the temporal dynamics of shape representation in area V4 of the alert macaque monkey. Analyses were based on two large stimulus sets, one equivalent to the 2D shape stimuli used in a previous study of V2, and the other a set of stereoscopic 3D shape stimuli. As in V2, we found that information conveyed by individual V4 neurons about the stimuli tended to be maximal during the initial transient response and generally lower, albeit statistically significant, afterwards. The population response was substantially correlated from one stimulus to the next during the transients, and decorrelated as responses decayed. V4 responses showed significantly longer latencies than in V2, especially for the 3D stimulus set. Recordings from area V1 in a single animal revealed temporal dynamic patterns in response to the 2D shape stimuli that were largely similar to those in V2 and V4. Together with earlier results, these findings provide evidence for a distributed process of coarse-to-fine representation of shape stimuli in the visual cortex.

Extracting 3D structure from disparity

The neural mechanisms of stereoscopic 3D shape perception have only recently been investigated. Here we review the two cortical regions in which these mechanisms have been studied so far in macaques: a small subpart of inferotemporal cortex called TEs, and the caudal intraparietal (CIP) region. Neurons in TEs respond selectively to the orientation and curvature in depth of stereoscopic surfaces and this region provides a detailed 3D shape description of surface boundaries and surface content. This description is evoked only by binocular stimuli in which subjects see depth and it does not vary if depth is specified by different cues. Neurons in CIP are a selective for orientation in depth of surfaces and elongated objects, and their responses are also unaffected by changes in depth cues. Thus, stereoscopic 3D shape is processed in both the dorsal, occipito-parietal and the ventral, occipito-temporal streams.

Neural basis of shape representation in the primate brain

Visual shape recognition--the ability to recognize a wide variety of shapes regardless of their size, position, view, clutter and ambient lighting--is a remarkable ability essential for complex behavior. In the primate brain, this depends on information processing in a multistage pathway running from primary visual cortex (V1), where cells encode local orientation and spatial frequency information, to the inferotemporal cortex (IT), where cells respond selectively to complex shapes. A fundamental question yet to be answered is how the local orientation signals (in V1) are transformed into selectivity for complex shapes (in IT). To gain insights into the underlying mechanisms we investigated the neural basis of shape representation in area V4, an intermediate stage in this processing hierarchy. Theoretical considerations and psychophysical evidence suggest that contour features, i.e. angles and curves along an object contour, may serve as the basis of representation at intermediate stages of shape processing. To test this hypothesis we studied the response properties of single units in area V4 of primates. We first demonstrated that V4 neurons show strong systematic tuning for the orientation and acuteness of angles and curves when presented in isolation within the cells' receptive field. Next, we found that responses to complex shapes were dictated by the curvature at a specific boundary location within the shape. Finally, using basis function decoding, we demonstrated that an ensemble of V4 neurons could successfully encode complete shapes as aggregates of boundary fragments. These findings identify curvature as a basis of shape representation in area V4 and provide insights into the neurophysiological basis for the salience of convex curves in shape perception.