The role of binocular disparity in stereoscopic images of objects in the macaque anterior intraparietal area

Disparity in Context: Understanding how monocular image content interacts with disparity processing in human visual cortex

Horizontal disparities between the two eyes' retinal images are the primary cue for depth. Commonly used random ot tereograms (RDS) intentionally camouflage the disparity cue, breaking the correlations between monocular image structure and the depth map that are present in natural images. Because of the nonlinear nature of visual processing, it is unlikely that simple computational rules derived from RDS will be sufficient to explain binocular vision in natural environments. In order to understand the interplay between natural scene structure and disparity encoding, we used a depth-image-based-rendering technique and a library of natural 3D stereo pairs to synthesize two novel stereogram types in which monocular scene content was manipulated independent of scene depth information. The half-images of the novel stereograms comprised either random-dots or scrambled natural scenes, each with the same depth maps as the corresponding natural scene stereograms. Using these stereograms in a simultaneous Event-Related Potential and behavioral discrimination task, we identified multiple disparity-contingent encoding stages between 100 ~ 500 msec. The first disparity sensitive evoked potential was observed at ~100 msec after an earlier evoked potential (between ~50-100 msec) that was sensitive to the structure of the monocular half-images but blind to disparity. Starting at ~150 msec, disparity responses were stereogram-specific and predictive of perceptual depth. Complex features associated with natural scene content are thus at least partially coded prior to disparity information, but these features and possibly others associated with natural scene content interact with disparity information only after an intermediate, 2D scene-independent disparity processing stage.

Face Representations via Tensorfaces of Various Complexities

Neurons selective for faces exist in humans and monkeys. However, characteristics of face cell receptive fields are poorly understood. In this theoretical study, we explore the effects of complexity, defined as algorithmic information (Kolmogorov complexity) and logical depth, on possible ways that face cells may be organized. We use tensor decompositions to decompose faces into a set of components, called tensorfaces, and their associated weights, which can be interpreted as model face cells and their firing rates. These tensorfaces form a high-dimensional representation space in which each tensorface forms an axis of the space. A distinctive feature of the decomposition algorithm is the ability to specify tensorface complexity. We found that low-complexity tensorfaces have blob-like appearances crudely approximating faces, while high-complexity tensorfaces appear clearly face-like. Low-complexity tensorfaces require a larger population to reach a criterion face reconstruction error than medium- or high-complexity tensorfaces, and thus are inefficient by that criterion. Low-complexity tensorfaces, however, generalize better when representing statistically novel faces, which are faces falling beyond the distribution of face description parameters found in the tensorface training set. The degree to which face representations are parts based or global forms a continuum as a function of tensorface complexity, with low and medium tensorfaces being more parts based. Given the computational load imposed in creating high-complexity face cells (in the form of algorithmic information and logical depth) and in the absence of a compelling advantage to using high-complexity cells, we suggest face representations consist of a mixture of low- and medium-complexity face cells.

Representation of shape, space, and attention in monkey cortex

Attentional deficits are core to numerous developmental, neurological, and psychiatric disorders. At the single-cell level, much knowledge has been garnered from studies of shape and spatial properties, as well as from numerous demonstrations of attentional modulation of those properties. Despite this wealth of knowledge of single-cell responses across many brain regions, little is known about how these cellular characteristics relate to population level representations and how such representations relate to behavior; in particular, how these cellular responses relate to the representation of shape, space, and attention, and how these representations differ across cortical areas and streams. Here we will emphasize the role of population coding as a missing link for connecting single-cell properties with behavior. Using a data-driven intrinsic approach to population decoding, we show that both 'what' and 'where' cortical visual streams encode shape, space, and attention, yet demonstrate striking differences in these representations. We suggest that both pathways fully process shape and space, but that differences in representation may arise due to their differing functions and input and output constraints. Moreover, differences in the effects of attention on shape and spatial population representations in the two visual streams suggest two distinct strategies: in a ventral area, attention or task demands modulate the population representations themselves (perhaps to expand or enhance one part at the expense of other parts) while in a dorsal area, at a population representation level, attention effects are weak and nearly non-existent, perhaps in order to maintain veridical representations needed for visuomotor control. We show that an intrinsic approach, as opposed to theory-driven and labeled approaches, is useful for understanding how representations develop and differ across brain regions. Most importantly, these approaches help link cellular properties more tightly with behavior, a much-needed step to better understand and interpret cellular findings and key to providing insights to improve interventions in human disorders.

Attention Effects on Neural Population Representations for Shape and Location Are Stronger in the Ventral than Dorsal Stream

We examined how attention causes neural population representations of shape and location to change in ventral stream (AIT) and dorsal stream (LIP). Monkeys performed two identical delayed-match-to-sample (DMTS) tasks, attending either to shape or location. In AIT, shapes were more discriminable when directing attention to shape rather than location, measured by an increase in mean distance between population response vectors. In LIP, attending to location rather than shape did not increase the discriminability of different stimulus locations. Even when factoring out the change in mean vector response distance, multidimensional scaling (MDS) still showed a significant task difference in AIT, but not LIP, indicating that beyond increasing discriminability, attention also causes a nonlinear warping of representation space in AIT. Despite single-cell attentional modulations in both areas, our data show that attentional modulations of population representations are weaker in LIP, likely due to a need to maintain veridical representations for visuomotor control.

A brief comparative review of primate posterior parietal cortex: A novel hypothesis on the human toolmaker

The primate visual system contains two major cortical pathways: a ventral-temporal pathway that has been associated with object processing and recognition, and a dorsal-parietal pathway that has been associated with spatial processing and action guidance. Our understanding of the role of the dorsal pathway, in particular, has greatly evolved within the framework of the two-pathway hypothesis since its original conception. Here, we present a comparative review of the primate dorsal pathway in humans and monkeys based on electrophysiological, neuroimaging, neuropsychological, and neuroanatomical studies. We consider similarities and differences across species in terms of the topographic representation of visual space; specificity for eye, reaching, or grasping movements; multi-modal response properties; and the representation of objects and tools. We also review the relative anatomical location of functionally- and topographically-defined regions of the posterior parietal cortex. An emerging theme from this comparative analysis is that non-spatial information is represented to a greater degree, and with increased complexity, in the human dorsal visual system. We propose that non-spatial information in the primate parietal cortex contributes to the perception-to-action system aimed at manipulating objects in peripersonal space. In humans, this network has expanded in multiple ways, including the development of a dorsal object vision system mirroring the complexity of the ventral stream, the integration of object information with parietal working memory systems, and the emergence of tool-specific object representations in the anterior intraparietal sulcus and regions of the inferior parietal lobe. We propose that these evolutionary changes have enabled the emergence of human-specific behaviors, such as the sophisticated use of tools.

Through the forest of motor representations

Following neuroscience, and using different labels, several philosophers have addressed the idea of the presence of a single representational mechanism lying in between (visual) perceptual processes and motor processes involved in different functions and useful for shaping suitable action performances: a motor representation (MR). MRs are the naturalized mental antecedents of action. This paper presents a new, non-monolithic view of MRs, according to which, contrarily to the received view, when looking at in between (visual) perceptual processes and motor processes, we find not only a single representational mechanism with different functions, but an ensemble of different sub-representational phenomena, each of which with a different function. This new view is able to avoid several issues emerging from the literature and to address something the literature is silent about, which however turns out to be crucial for a theory of MRs.

The neural representation of objects formed through the spatiotemporal integration of visual transients

Oftentimes, objects are only partially and transiently visible as parts of them become occluded during observer or object motion. The visual system can integrate such object fragments across space and time into perceptual wholes or spatiotemporal objects. This integrative and dynamic process may involve both ventral and dorsal visual processing pathways, along which shape and spatial representations are thought to arise. We measured fMRI BOLD response to spatiotemporal objects and used multi-voxel pattern analysis (MVPA) to decode shape information across 20 topographic regions of visual cortex. Object identity could be decoded throughout visual cortex, including intermediate (V3A, V3B, hV4, LO1-2,) and dorsal (TO1-2, and IPS0-1) visual areas. Shape-specific information, therefore, may not be limited to early and ventral visual areas, particularly when it is dynamic and must be integrated. Contrary to the classic view that the representation of objects is the purview of the ventral stream, intermediate and dorsal areas may play a distinct and critical role in the construction of object representations across space and time.

Receptive field properties of neurons in the macaque anterior intraparietal area

Visual object information is necessary for grasping. In primates, the anterior intraparietal area (AIP) plays an essential role in visually guided grasping. Neurons in AIP encode features of objects, but no study has systematically investigated the receptive field (RF) of AIP neurons. We mapped the RF of posterior AIP (pAIP) neurons in the central visual field, using images of objects and small line fragments that evoked robust responses, together with less effective stimuli. The RF sizes we measured varied between 3°(2)and 90°(2), with the highest response either at the fixation point or at parafoveal positions. A large fraction of pAIP neurons showed nonuniform RFs, with multiple local maxima in both ipsilateral and contralateral hemifields. Moreover, the RF profile could depend strongly on the stimulus used to map the RF. Highly similar results were obtained with the smallest stimulus that evoked reliable responses (line fragments measuring 1-2°). The nonuniformity and dependence of the RF profile on the stimulus in pAIP were comparable to previous observations in the anterior part of the lateral intraparietal area (aLIP), but the average RF of pAIP neurons was located at the fovea whereas the average RF of aLIP neurons was located parafoveally. Thus nonuniformity and stimulus dependence of the RF may represent general RF properties of neurons in the dorsal visual stream involved in object analysis, which contrast markedly with those of neurons in the ventral visual stream.

Shape representations in the primate dorsal visual stream

The primate visual system extracts object shape information for object recognition in the ventral visual stream. Recent research has demonstrated that object shape is also processed in the dorsal visual stream, which is specialized for spatial vision and the planning of actions. A number of studies have investigated the coding of 2D shape in the anterior intraparietal area (AIP), one of the end-stage areas of the dorsal stream which has been implicated in the extraction of affordances for the purpose of grasping. These findings challenge the current understanding of area AIP as a critical stage in the dorsal stream for the extraction of object affordances. The representation of three-dimensional (3D) shape has been studied in two interconnected areas known to be critical for object grasping: area AIP and area F5a in the ventral premotor cortex (PMv), to which AIP projects. In both areas neurons respond selectively to 3D shape defined by binocular disparity, but the latency of the neural selectivity is approximately 10 ms longer in F5a compared to AIP, consistent with its higher position in the hierarchy of cortical areas. Furthermore, F5a neurons were more sensitive to small amplitudes of 3D curvature and could detect subtle differences in 3D structure more reliably than AIP neurons. In both areas, 3D-shape selective neurons were co-localized with neurons showing motor-related activity during object grasping in the dark, indicating a close convergence of visual and motor information on the same clusters of neurons.

Effective connectivity of depth-structure-selective patches in the lateral bank of the macaque intraparietal sulcus

Extrastriate cortical areas are frequently composed of subpopulations of neurons encoding specific features or stimuli, such as color, disparity, or faces, and patches of neurons encoding similar stimulus properties are typically embedded in interconnected networks, such as the attention or face-processing network. The goal of the current study was to examine the effective connectivity of subsectors of neurons in the same cortical area with highly similar neuronal response properties. We first recorded single- and multi-unit activity to identify two neuronal patches in the anterior part of the macaque intraparietal sulcus (IPS) showing the same depth structure selectivity and then employed electrical microstimulation during functional magnetic resonance imaging in these patches to determine the effective connectivity of these patches. The two IPS subsectors we identified-with the same neuronal response properties and in some cases separated by only 3 mm-were effectively connected to remarkably distinct cortical networks in both dorsal and ventral stream in three macaques. Conversely, the differences in effective connectivity could account for the known visual-to-motor gradient within the anterior IPS. These results clarify the role of the anterior IPS as a pivotal brain region where dorsal and ventral visual stream interact during object analysis. Thus, in addition to the anatomical connectivity of cortical areas and the properties of individual neurons in these areas, the effective connectivity provides novel key insights into the widespread functional networks that support behavior.

Modeling the shape hierarchy for visually guided grasping

The monkey anterior intraparietal area (AIP) encodes visual information about three-dimensional object shape that is used to shape the hand for grasping. We modeled shape tuning in visual AIP neurons and its relationship with curvature and gradient information from the caudal intraparietal area (CIP). The main goal was to gain insight into the kinds of shape parameterizations that can account for AIP tuning and that are consistent with both the inputs to AIP and the role of AIP in grasping. We first experimented with superquadric shape parameters. We considered superquadrics because they occupy a role in robotics that is similar to AIP, in that superquadric fits are derived from visual input and used for grasp planning. We also experimented with an alternative shape parameterization that was based on an Isomap dimension reduction of spatial derivatives of depth (i.e., distance from the observer to the object surface). We considered an Isomap-based model because its parameters lacked discontinuities between similar shapes. When we matched the dimension of the Isomap to the number of superquadric parameters, the superquadric model fit the AIP data somewhat more closely. However, higher-dimensional Isomaps provided excellent fits. Also, we found that the Isomap parameters could be approximated much more accurately than superquadric parameters by feedforward neural networks with CIP-like inputs. We conclude that Isomaps, or perhaps alternative dimension reductions of visual inputs to AIP, provide a promising model of AIP electrophysiology data. Further work is needed to test whether such shape parameterizations actually provide an effective basis for grasp control.

Coding of shape features in the macaque anterior intraparietal area

The exquisite ability of primates to grasp and manipulate objects relies on the transformation of visual information into motor commands. To this end, the visual system extracts object affordances that can be used to program and execute the appropriate grip. The macaque anterior intraparietal (AIP) area has been implicated in the extraction of affordances for the purpose of grasping. Neurons in the AIP area respond during visually guided grasping and to the visual presentation of objects. A subset of AIP neurons is also activated by two-dimensional images of objects and even by outline contours defining the object shape, but it is unknown how AIP neurons actually represent object shape. In this study, we used a stimulus reduction approach to determine the minimum effective shape feature evoking AIP responses. AIP neurons responding to outline shapes also responded selectively to very small fragment stimuli measuring only 1-2°. This fragment selectivity could not be explained by differences in eye movements or simple orientation selectivity, but proved to be highly dependent on the relative position of the stimulus in the receptive field. Our findings challenge the current understanding of the AIP area as a critical stage in the dorsal stream for the extraction of object affordances.

Surface roughness detection of arteries via texture analysis of ultrasound images for early diagnosis of atherosclerosis

There is a strong research interest in identifying the surface roughness of the carotid arterial inner wall via texture analysis for early diagnosis of atherosclerosis. The purpose of this study is to assess the efficacy of texture analysis methods for identifying arterial roughness in the early stage of atherosclerosis. Ultrasound images of common carotid arteries of 15 normal mice fed a normal diet and 28 apoE^−/− mice fed a high-fat diet were recorded by a high-frequency ultrasound system (Vevo 2100, frequency: 40 MHz). Six different texture feature sets were extracted based on the following methods: first-order statistics, fractal dimension texture analysis, spatial gray level dependence matrix, gray level difference statistics, the neighborhood gray tone difference matrix, and the statistical feature matrix. Statistical analysis indicates that 11 of 19 texture features can be used to distinguish between normal and abnormal groups (p<0.05). When the 11 optimal features were used as inputs to a support vector machine classifier, we achieved over 89% accuracy, 87% sensitivity and 93% specificity. The accuracy, sensitivity and specificity for the k-nearest neighbor classifier were 73%, 75% and 70%, respectively. The results show that it is feasible to identify arterial surface roughness based on texture features extracted from ultrasound images of the carotid arterial wall. This method is shown to be useful for early detection and diagnosis of atherosclerosis.

Human fMRI reveals that delayed action re-recruits visual perception

Behavioral and neuropsychological research suggests that delayed actions rely on different neural substrates than immediate actions; however, the specific brain areas implicated in the two types of actions remain unknown. We used functional magnetic resonance imaging (fMRI) to measure human brain activation during delayed grasping and reaching. Specifically, we examined activation during visual stimulation and action execution separated by a 18-s delay interval in which subjects had to remember an intended action toward the remembered object. The long delay interval enabled us to unambiguously distinguish visual, memory-related, and action responses. Most strikingly, we observed reactivation of the lateral occipital complex (LOC), a ventral-stream area implicated in visual object recognition, and early visual cortex (EVC) at the time of action. Importantly this reactivation was observed even though participants remained in complete darkness with no visual stimulation at the time of the action. Moreover, within EVC, higher activation was observed for grasping than reaching during both vision and action execution. Areas in the dorsal visual stream were activated during action execution as expected and, for some, also during vision. Several areas, including the anterior intraparietal sulcus (aIPS), dorsal premotor cortex (PMd), primary motor cortex (M1) and the supplementary motor area (SMA), showed sustained activation during the delay phase. We propose that during delayed actions, dorsal-stream areas plan and maintain coarse action goals; however, at the time of execution, motor programming requires re-recruitment of detailed visual information about the object through reactivation of (1) ventral-stream areas involved in object perception and (2) early visual areas that contain richly detailed visual representations, particularly for grasping.