Functional architecture in monkey inferotemporal cortex revealed by in vivo optical imaging

Functional ultrasound imaging reveals 3D structure of orientation domains in ferret primary visual cortex

BACKGROUND AND PURPOSE

The sensory cortex is organized into "maps" that represent sensory space across cortical space. In primary visual cortex (V1) of highly visual mammals, multiple visual feature maps are organized into a functional architecture anchored by orientation domains: regions containing neurons preferring the same stimulus orientation. Although the pinwheel-like structure of orientation domains is well-characterized in the superficial cortical layers in dorsal regions of V1, the 3D shape of orientation domains spanning all 6 cortical layers and across dorsal and ventral regions of V1 has never been revealed.

METHODS

We utilized an emerging research method in neuroscience, functional ultrasound imaging (fUS), to resolve the 3D structure of orientation domains throughout V1 in anesthetized female ferrets. fUS measures blood flow from which neuronal population activity is inferred with improved spatial resolution over fMRI.

RESULTS

fUS activations in response to drifting gratings placed at multiple locations in visual space generated unique activation patterns in V1 and visual thalamus, confirming prior observations that fUS can resolve retinotopy. Iso-orientation domains, determined from clusters of activations driven by large oriented gratings, were cone-shaped and present in both dorsal and ventral regions of V1. The spacing between iso-orientation domains was consistent with spacing measured previously using optical imaging methods.

CONCLUSIONS

Orientation domains are cones rather than columns. Their width and intra-domain distances may vary across dorsal and ventral regions of V1. These findings demonstrate the power of fUS at revealing 3D functional architecture in cortical regions not accessible to traditional surface imaging methods.

Early Emergence of Solid Shape Coding in Natural and Deep Network Vision

Area V4 is the first object-specific processing stage in the ventral visual pathway, just as area MT is the first motion-specific processing stage in the dorsal pathway. For almost 50 years, coding of object shape in V4 has been studied and conceived in terms of flat pattern processing, given its early position in the transformation of 2D visual images. Here, however, in awake monkey recording experiments, we found that roughly half of V4 neurons are more tuned and responsive to solid, 3D shape-in-depth, as conveyed by shading, specularity, reflection, refraction, or disparity cues in images. Using 2-photon functional microscopy, we found that flat- and solid-preferring neurons were segregated into separate modules across the surface of area V4. These findings should impact early shape-processing theories and models, which have focused on 2D pattern processing. In fact, our analyses of early object processing in AlexNet, a standard visual deep network, revealed a similar distribution of sensitivities to flat and solid shape in layer 3. Early processing of solid shape, in parallel with flat shape, could represent a computational advantage discovered by both primate brain evolution and deep-network training.

Mapping of language and motor function during awake neurosurgery with intraoperative optical imaging

Intraoperative optical imaging (IOI) is a marker-free, contactless, and noninvasive imaging technique that is able to visualize metabolic changes of the brain surface following neuronal activation. Although it has been used in the past mainly for the identification of functional brain areas under general anesthesia, the authors investigated the potential of the method during awake surgery. Measurements were performed in 10 patients who underwent resection of lesions within or adjacent to cortical language or motor sites. IOI was applied in 3 different scenarios: identification of motor areas by using finger-tapping tasks, identification of language areas by using speech tasks (overt and silent speech), and a novel approach-the application of IOI as a feedback tool during direct electrical stimulation (DES) mapping of language. The functional maps, which were calculated from the IOI data (activity maps), were qualitatively compared with the functional MRI (fMRI) and the electrophysiological testing results during the surgical procedure to assess their potential benefit for surgical decision-making.The results reveal that the intraoperative identification of motor sites with IOI in good agreement with the preoperatively acquired fMRI and the intraoperative electrophysiological measurements is possible. Because IOI provides spatially highly resolved maps with minimal additional hardware effort, the application of the technique for motor site identification seems to be beneficial in awake procedures. The identification of language processing sites with IOI was also possible, but in the majority of cases significant differences between fMRI, IOI, and DES were visible, and therefore according to the authors' findings the IOI results are too unspecific to be useful for intraoperative decision-making with respect to exact language localization. For this purpose, DES mapping will remain the method of choice.Nevertheless, the IOI technique can provide additional value during the language mapping procedure with DES. Using a simple difference imaging approach, the authors were able to visualize and calculate the spatial extent of activation for each stimulation. This might enable surgeons in the future to optimize the mapping process. Additionally, differences between tumor and nontumor stimulation sites were observed with respect to the spatial extent of the changes in cortical optical properties. These findings provide further evidence that the method allows the assessment of the functional state of neurovascular coupling and is therefore suited for the delineation of pathologically altered tissue.

Reversible Inactivation of Different Millimeter-Scale Regions of Primate IT Results in Different Patterns of Core Object Recognition Deficits

Extensive research suggests that the inferior temporal (IT) population supports visual object recognition behavior. However, causal evidence for this hypothesis has been equivocal, particularly beyond the specific case of face-selective subregions of IT. Here, we directly tested this hypothesis by pharmacologically inactivating individual, millimeter-scale subregions of IT while monkeys performed several core object recognition subtasks, interleaved trial-by trial. First, we observed that IT inactivation resulted in reliable contralateral-biased subtask-selective behavioral deficits. Moreover, inactivating different IT subregions resulted in different patterns of subtask deficits, predicted by each subregion's neuronal object discriminability. Finally, the similarity between different inactivation effects was tightly related to the anatomical distance between corresponding inactivation sites. Taken together, these results provide direct evidence that the IT cortex causally supports general core object recognition and that the underlying IT coding dimensions are topographically organized.

Orientation Encoding and Viewpoint Invariance in Face Recognition: Inferring Neural Properties from Large-Scale Signals

Viewpoint-invariant face recognition is thought to be subserved by a distributed network of occipitotemporal face-selective areas that, except for the human anterior temporal lobe, have been shown to also contain face-orientation information. This review begins by highlighting the importance of bilateral symmetry for viewpoint-invariant recognition and face-orientation perception. Then, monkey electrophysiological evidence is surveyed describing key tuning properties of face-selective neurons-including neurons bimodally tuned to mirror-symmetric face-views-followed by studies combining functional magnetic resonance imaging (fMRI) and multivariate pattern analyses to probe the representation of face-orientation and identity information in humans. Altogether, neuroimaging studies suggest that face-identity is gradually disentangled from face-orientation information along the ventral visual processing stream. The evidence seems to diverge, however, regarding the prevalent form of tuning of neural populations in human face-selective areas. In this context, caveats possibly leading to erroneous inferences regarding mirror-symmetric coding are exposed, including the need to distinguish angular from Euclidean distances when interpreting multivariate pattern analyses. On this basis, this review argues that evidence from the fusiform face area is best explained by a view-sensitive code reflecting head angular disparity, consistent with a role of this area in face-orientation perception. Finally, the importance is stressed of explicit models relating neural properties to large-scale signals.

Learned Value Shapes Responses to Objects in Frontal and Ventral Stream Networks in Macaque Monkeys

We have an incomplete picture of how the brain links object representations to reward value, and how this information is stored and later retrieved. The orbitofrontal cortex (OFC), medial frontal cortex (MFC), and ventrolateral prefrontal cortex (VLPFC), together with the amygdala, are thought to play key roles in these processes. There is an apparent discrepancy, however, regarding frontal areas thought to encode value in macaque monkeys versus humans. To address this issue, we used fMRI in macaque monkeys to localize brain areas encoding recently learned image values. Each week, monkeys learned to associate images of novel objects with a high or low probability of water reward. Areas responding to the value of recently learned reward-predictive images included MFC area 10 m/32, VLPFC area 12, and inferior temporal visual cortex (IT). The amygdala and OFC, each thought to be involved in value encoding, showed little such effect. Instead, these 2 areas primarily responded to visual stimulation and reward receipt, respectively. Strong image value encoding in monkey MFC compared with OFC is surprising, but agrees with results from human imaging studies. Our findings demonstrate the importance of VLPFC, MFC, and IT in representing the values of recently learned visual images.

Space-Time Dynamics of Membrane Currents Evolve to Shape Excitation, Spiking, and Inhibition in the Cortex at Small and Large Scales

In the cerebral cortex, membrane currents, i.e., action potentials and other membrane currents, express many forms of space-time dynamics. In the spontaneous asynchronous irregular state, their space-time dynamics are local non-propagating fluctuations and sparse spiking appearing at unpredictable positions. After transition to active spiking states, larger structured zones with active spiking neurons appear, propagating through the cortical network, driving it into various forms of widespread excitation, and engaging the network from microscopic scales to whole cortical areas. At each engaged cortical site, the amount of excitation in the network, after a delay, becomes matched by an equal amount of space-time fine-tuned inhibition that might be instrumental in driving the dynamics toward perception and action.

Neurophysiological Organization of the Middle Face Patch in Macaque Inferior Temporal Cortex

While early cortical visual areas contain fine scale spatial organization of neuronal properties, such as orientation preference, the spatial organization of higher-level visual areas is less well understood. The fMRI demonstration of face-preferring regions in human ventral cortex and monkey inferior temporal cortex ("face patches") raises the question of how neural selectivity for faces is organized. Here, we targeted hundreds of spatially registered neural recordings to the largest fMRI-identified face-preferring region in monkeys, the middle face patch (MFP), and show that the MFP contains a graded enrichment of face-preferring neurons. At its center, as much as 93% of the sites we sampled responded twice as strongly to faces than to nonface objects. We estimate the maximum neurophysiological size of the MFP to be ∼6 mm in diameter, consistent with its previously reported size under fMRI. Importantly, face selectivity in the MFP varied strongly even between neighboring sites. Additionally, extremely face-selective sites were ∼40 times more likely to be present inside the MFP than outside. These results provide the first direct quantification of the size and neural composition of the MFP by showing that the cortical tissue localized to the fMRI defined region consists of a very high fraction of face-preferring sites near its center, and a monotonic decrease in that fraction along any radial spatial axis.

SIGNIFICANCE STATEMENT

The underlying organization of neurons that give rise to the large spatial regions of activity observed with fMRI is not well understood. Neurophysiological studies that have targeted the fMRI identified face patches in monkeys have provided evidence for both large-scale clustering and a heterogeneous spatial organization. Here we used a novel x-ray imaging system to spatially map the responses of hundreds of sites in and around the middle face patch. We observed that face-selective signal localized to the middle face patch was characterized by a gradual spatial enrichment. Furthermore, strongly face-selective sites were ∼40 times more likely to be found inside the patch than outside of the patch.

The neural code for face orientation in the human fusiform face area

Humans recognize faces and objects with high speed and accuracy regardless of their orientation. Recent studies have proposed that orientation invariance in face recognition involves an intermediate representation where neural responses are similar for mirror-symmetric views. Here, we used fMRI, multivariate pattern analysis, and computational modeling to investigate the neural encoding of faces and vehicles at different rotational angles. Corroborating previous studies, we demonstrate a representation of face orientation in the fusiform face-selective area (FFA). We go beyond these studies by showing that this representation is category-selective and tolerant to retinal translation. Critically, by controlling for low-level confounds, we found the representation of orientation in FFA to be compatible with a linear angle code. Aspects of mirror-symmetric coding cannot be ruled out when FFA mean activity levels are considered as a dimension of coding. Finally, we used a parametric family of computational models, involving a biased sampling of view-tuned neuronal clusters, to compare different face angle encoding models. The best fitting model exhibited a predominance of neuronal clusters tuned to frontal views of faces. In sum, our findings suggest a category-selective and monotonic code of face orientation in the human FFA, in line with primate electrophysiology studies that observed mirror-symmetric tuning of neural responses at higher stages of the visual system, beyond the putative homolog of human FFA.

Object representation in inferior temporal cortex is organized hierarchically in a mosaic-like structure

There are two dominant models for the functional organization of brain regions underlying object recognition. One model postulates category-specific modules while the other proposes a distributed representation of objects with generic visual features. Functional imaging techniques relying on metabolic signals, such as fMRI and optical intrinsic signal imaging (OISI), have been used to support both models, but due to the indirect nature of the measurements in these techniques, the existing data for one model cannot be used to support the other model. Here, we used large-scale multielectrode recordings over a large surface of anterior inferior temporal (IT) cortex, and densely mapped stimulus-evoked neuronal responses. We found that IT cortex is subdivided into distinct domains characterized by similar patterns of responses to the objects in our stimulus set. Each domain spanned several millimeters on the cortex. Some of these domains represented faces ("face" domains) or monkey bodies ("monkey-body" domains). We also identified domains with low responsiveness to faces ("anti-face" domains). Meanwhile, the recording sites within domains that displayed category selectivity showed heterogeneous tuning profiles to different exemplars within each category. This local heterogeneity was consistent with the stimulus-evoked feature columns revealed by OISI. Taken together, our study revealed that regions with common functional properties (domains) consist of a finer functional structure (columns) in anterior IT cortex. The "domains" and previously proposed "patches" are rather like "mosaics" where a whole mosaic is characterized by overall similarity in stimulus responses and pieces of the mosaic correspond to feature columns.

Exemplar selectivity reflects perceptual similarities in the human fusiform cortex

While brain imaging studies emphasized the category selectivity of face-related areas, the underlying mechanisms of our remarkable ability to discriminate between different faces are less understood. Here, we recorded intracranial local field potentials from face-related areas in patients presented with images of faces and objects. A highly significant exemplar tuning within the category of faces was observed in high-Gamma (80-150 Hz) responses. The robustness of this effect was supported by single-trial decoding of face exemplars using a minimal (n = 5) training set. Importantly, exemplar tuning reflected the psychophysical distance between faces but not their low-level features. Our results reveal a neuronal substrate for the establishment of perceptual distance among faces in the human brain. They further imply that face neurons are anatomically grouped according to well-defined functional principles, such as perceptual similarity.

Has brain imaging discovered anything new about how the brain works?

There have now been roughly 130,000 papers on fMRI. While these have clearly contributed to our understanding of the functional anatomy of the human brain, it is less clear that they have changed the way in which we think about the brain. The issue, in other words, is whether they have established new principles about how the brain works. In this paper we offer as an example one new principle, partly to lay down the criteria that are required for establishing a new principle, and partly to encourage others to offer other principles. Our example concerns the flexible flow of information through the cortex that must occur according to the demands of the task or current context. We suggest that this flexibility is achieved by feedback connections from the prefrontal and parietal cortex, and that these include connections to sensory and motor areas. However, the nature of the selective effect differs. The parietal cortex can select both within and across processing streams. By across streams we mean that it can have the same influence on different streams, for example the dorsal and ventral visual systems. However, only the prefrontal cortex can also select between processing streams. The difference between the prefrontal and parietal effects is due to their different positions within the processing hierarchy.

Characterizing responses of translation-invariant neurons to natural stimuli: maximally informative invariant dimensions

The human visual system is capable of recognizing complex objects even when their appearances change drastically under various viewing conditions. Especially in the higher cortical areas, the sensory neurons reflect such functional capacity in their selectivity for complex visual features and invariance to certain object transformations, such as image translation. Due to the strong nonlinearities necessary to achieve both the selectivity and invariance, characterizing and predicting the response patterns of these neurons represents a formidable computational challenge. A related problem is that such neurons are poorly driven by randomized inputs, such as white noise, and respond strongly only to stimuli with complex high-order correlations, such as natural stimuli. Here we describe a novel two-step optimization technique that can characterize both the shape selectivity and the range and coarseness of position invariance from neural responses to natural stimuli. One step in the optimization is finding the template as the maximally informative dimension given the estimated spatial location where the response could have been triggered within each image. The estimates of the locations that triggered the response are updated in the next step. Under the assumption of a monotonic relationship between the firing rate and stimulus projections on the template at a given position, the most likely location is the one that has the largest projection on the estimate of the template. The algorithm shows quick convergence during optimization, and the estimation results are reliable even in the regime of small signal-to-noise ratios. When we apply the algorithm to responses of complex cells in the primary visual cortex (V1) to natural movies, we find that responses of the majority of cells were significantly better described by translation-invariant models based on one template compared with position-specific models with several relevant features.

Insights into cortical mechanisms of behavior from microstimulation experiments

Even the simplest behaviors depend on a large number of neurons that are distributed across many brain regions. Because electrical microstimulation can change the activity of localized subsets of neurons, it has provided valuable evidence that specific neurons contribute to particular behaviors. Here we review what has been learned about cortical function from behavioral studies using microstimulation in animals and humans. Experiments that examine how microstimulation affects the perception of stimuli have shown that the effects of microstimulation are usually highly specific and can be related to the stimuli preferred by neurons at the stimulated site. Experiments that ask subjects to detect cortical microstimulation in the absence of other stimuli have provided further insights. Although subjects typically can detect microstimulation of primary sensory or motor cortex, they are generally unable to detect stimulation of most of cortex without extensive practice. With practice, however, stimulation of any part of cortex can become detected. These training effects suggest that some patterns of cortical activity cannot be readily accessed to guide behavior, but that the adult brain retains enough plasticity to learn to process novel patterns of neuronal activity arising anywhere in cortex.

Whatever Happened to Object-Centered Representations?

David Marr's approach to the study of vision has been tremendously influential. However, the approach proposes the goal of computing invariant shape descriptions from image-based information, a task that appears implausible, given the tremendous variation that can occur between images displaying a single object. Theorists in the field of three-dimensional object recognition have rejected the approach of computing object-centered representations, and instead propose representations of objects from the perspective of a viewer. If object-centered descriptions of objects exist in the brain, they are more likely to underlie motor interaction with objects rather than visual object understanding.

Similarity dependency of the change in ERP component N1 accompanying with the object recognition learning

Performance during object recognition across views is largely dependent on inter-object similarity. The present study was designed to investigate the similarity dependency of object recognition learning on the changes in ERP component N1. Human subjects were asked to train themselves to recognize novel objects with different inter-object similarity by performing object recognition tasks. During the tasks, images of an object had to be discriminated from the images of other objects irrespective of the viewpoint. When objects had a high inter-object similarity, the ERP component, N1 exhibited a significant increase in both the amplitude and the latency variation across objects during the object recognition learning process, and the N1 amplitude and latency variation across the views of the same objects decreased significantly. In contrast, no significant changes were found during the learning process when using objects with low inter-object similarity. The present findings demonstrate that the changes in the variation of N1 that accompany the object recognition learning process are dependent upon the inter-object similarity and imply that there is a difference in the neuronal representation for object recognition when using objects with high and low inter-object similarity.

A head view-invariant representation of gaze direction in anterior superior temporal sulcus

Humans show a remarkable ability to discriminate others' gaze direction, even though a given direction can be conveyed by many physically dissimilar configurations of different eye positions and head views. For example, eye contact can be signaled by a rightward glance in a left-turned head or by direct gaze in a front-facing head. Such acute gaze discrimination implies considerable perceptual invariance. Previous human research found that superior temporal sulcus (STS) responds preferentially to gaze shifts [1], but the underlying representation that supports such general responsiveness remains poorly understood. Using multivariate pattern analysis (MVPA) of human functional magnetic resonance imaging (fMRI) data, we tested whether STS contains a higher-order, head view-invariant code for gaze direction. The results revealed a finely graded gaze direction code in right anterior STS that was invariant to head view and physical image features. Further analyses revealed similar gaze effects in left anterior STS and precuneus. Our results suggest that anterior STS codes the direction of another's attention regardless of how this information is conveyed and demonstrate how high-level face areas carry out fine-grained, perceptually relevant discrimination through invariance to other face features.

Orientation dependency of intrinsic optical signal dynamics in cat area 18

Imaging studies based on intrinsic optical signals have been primarily performed by analyses of the signal amplitude as opposed to using the temporal information of the intrinsic optical signals. The present study focused on the dynamics of these signals in cat area 18, and quantitatively compared the waveforms after presentation of different stimuli across the same cortical regions. Optical imaging based on intrinsic signals was conducted on 18 hemispheres of 9 cats. For the visual stimuli, gratings with orientations that changed from horizontal (0°) to 157.5° in 22.5° steps were used. The signal time course was examined at each pixel, with the peak delay defined as the amount of time required after the stimulus onset for the intrinsic optical signal to reach its negative maximum. In the area that showed significant orientation preference to 0° and 90° but not to their 22.5° separated nearby angles, the delays were 1.92 ± 0.22s and 1.99 ± 0.29s (mean ± SE, n = 18), respectively. Delays of 2.31 ± 0.20s and 2.28 ± 0.25s were observed in the cortical areas that selectively responded to the orientation gratings of 45°and 135° but not their nearby angles. Statistically, the delays in areas exhibiting oblique orientation preferences were significantly longer than those showing cardinal orientation preferences. These results demonstrate anisotropy for the intrinsic optical signal dynamics in the cat area 18. The possible neural mechanisms underlying were discussed.

Multiple scales of organization for object selectivity in ventral visual cortex

Object knowledge is hierarchical. Several hypotheses have proposed that this property might be reflected in the spatial organization of ventral visual cortex. For example, all exemplars of a category might activate the same patches of cortex, but with a slightly different position of the peak of activation in each patch. According to this view, category selectivity would be organized at a larger spatial scale compared to exemplar selectivity. No empirical evidence for such proposals is available from experiments with human subjects. Here, we compare the relative scale of organization for category and exemplar selectivity in two datasets with two methods: (i) by investigating the previously reported beneficial effect of spatial smoothing of the fMRI data on the reliability of multi-voxel selectivity patterns; and (ii) by comparing the relative weight of lower and higher spatial frequencies in the spatial frequency spectrum of these selectivity patterns. The findings are consistent with the proposal that selectivity for stimulus properties that underlie finer distinctions between objects is organized at a finer scale than selectivity for stimulus properties that differentiate categories. This finding confirms the existence of multiple scales of organization in the ventral visual pathway.

Investigating object representations during change detection in human extrastriate cortex

Detecting a change in a visual stimulus is particularly difficult when it is accompanied by a visual disruption such as a saccade or flicker. In order to say whether a stimulus has changed across such a disruption, some neural trace must persist. Here we investigated whether two different regions of the human extrastriate visual cortex contain neuronal populations encoding such a trace. Participants viewed a stimulus that included various objects and a short blank period (flicker) made it difficult to distinguish whether an object in the stimulus had changed or not. By applying transcranial magnetic stimulation (TMS) during the visual disruption we show that the lateral occipital (LO) cortex, but not the occipital face area, contains a sustained representation of a visual stimulus. TMS over LO improved the sensitivity and response bias for detecting changes by selectively reducing false alarms. We suggest that TMS enhanced the initial object representation and thus boosted neural events associated with object repetition. Our findings show that neuronal signals in the human LO cortex carry a sustained neural trace that is necessary for detecting the repetition of a stimulus.

A Gaussian attractor network for memory and recognition with experience-dependent learning

Attractor networks are widely believed to underlie the memory systems of animals across different species. Existing models have succeeded in qualitatively modeling properties of attractor dynamics, but their computational abilities often suffer from poor representations for realistic complex patterns, spurious attractors, low storage capacity, and difficulty in identifying attractive fields of attractors. We propose a simple two-layer architecture, gaussian attractor network, which has no spurious attractors if patterns to be stored are uncorrelated and can store as many patterns as the number of neurons in the output layer. Meanwhile the attractive fields can be precisely quantified and manipulated. Equipped with experience-dependent unsupervised learning strategies, the network can exhibit both discrete and continuous attractor dynamics. A testable prediction based on numerical simulations is that there exist neurons in the brain that can discriminate two similar stimuli at first but cannot after extensive exposure to physically intermediate stimuli. Inspired by this network, we found that adding some local feedbacks to a well-known hierarchical visual recognition model, HMAX, can enable the model to reproduce some recent experimental results related to high-level visual perception.

Six principles of visual cortical dynamics

A fundamental goal in vision science is to determine how many neurons in how many areas are required to compute a coherent interpretation of the visual scene. Here I propose six principles of cortical dynamics of visual processing in the first 150 ms following the appearance of a visual stimulus. Fast synaptic communication between neurons depends on the driving neurons and the biophysical history and driving forces of the target neurons. Under these constraints, the retina communicates changes in the field of view driving large populations of neurons in visual areas into a dynamic sequence of feed-forward communication and integration of the inward current of the change signal into the dendrites of higher order area neurons (30-70 ms). Simultaneously an even larger number of neurons within each area receiving feed-forward input are pre-excited to sub-threshold levels. The higher order area neurons communicate the results of their computations as feedback adding inward current to the excited and pre-excited neurons in lower areas. This feedback reconciles computational differences between higher and lower areas (75-120 ms). This brings the lower area neurons into a new dynamic regime characterized by reduced driving forces and sparse firing reflecting the visual areas interpretation of the current scene (140 ms). The population membrane potentials and net-inward/outward currents and firing are well behaved at the mesoscopic scale, such that the decoding in retinotopic cortical space shows the visual areas' interpretation of the current scene. These dynamics have plausible biophysical explanations. The principles are theoretical, predictive, supported by recent experiments and easily lend themselves to experimental tests or computational modeling.

View-invariant object recognition ability develops after discrimination, not mere exposure, at several viewing angles

One usually fails to recognize an unfamiliar object across changes in viewing angle when it has to be discriminated from similar distractor objects. Previous work has demonstrated that after long-term experience in discriminating among a set of objects seen from the same viewing angle, immediate recognition of the objects across 30-60 degrees changes in viewing angle becomes possible. The capability for view-invariant object recognition should develop during the within-viewing-angle discrimination, which includes two kinds of experience: seeing individual views and discriminating among the objects. The aim of the present study was to determine the relative contribution of each factor to the development of view-invariant object recognition capability. Monkeys were first extensively trained in a task that required view-invariant object recognition (Object task) with several sets of objects. The animals were then exposed to a new set of objects over 26 days in one of two preparatory tasks: one in which each object view was seen individually, and a second that required discrimination among the objects at each of four viewing angles. After the preparatory period, we measured the monkeys' ability to recognize the objects across changes in viewing angle, by introducing the object set to the Object task. Results indicated significant view-invariant recognition after the second but not first preparatory task. These results suggest that discrimination of objects from distractors at each of several viewing angles is required for the development of view-invariant recognition of the objects when the distractors are similar to the objects.

Perceptual learning of object shape

Recognition of objects is accomplished through the use of cues that depend on internal representations of familiar shapes. We used a paradigm of perceptual learning during visual search to explore what features human observers use to identify objects. Human subjects were trained to search for a target object embedded in an array of distractors, until their performance improved from near-chance levels to over 80% of trials in an object-specific manner. We determined the role of specific object components in the recognition of the object as a whole by measuring the transfer of learning from the trained object to other objects sharing components with it. Depending on the geometric relationship of the trained object with untrained objects, transfer to untrained objects was observed. Novel objects that shared a component with the trained object were identified at much higher levels than those that did not, and this could be used as an indicator of which features of the object were important for recognition. Training on an object also transferred to the components of the object when these components were embedded in an array of distractors of similar complexity. These results suggest that objects are not represented in a holistic manner during learning but that their individual components are encoded. Transfer between objects was not complete and occurred for more than one component, regardless of how well they distinguish the object from distractors. This suggests that a joint involvement of multiple components was necessary for full performance.

A model for learning topographically organized parts-based representations of objects in visual cortex: topographic nonnegative matrix factorization

Object representation in the inferior temporal cortex (IT), an area of visual cortex critical for object recognition in the primate, exhibits two prominent properties: (1) objects are represented by the combined activity of columnar clusters of neurons, with each cluster representing component features or parts of objects, and (2) closely related features are continuously represented along the tangential direction of individual columnar clusters. Here we propose a learning model that reflects these properties of parts-based representation and topographic organization in a unified framework. This model is based on a nonnegative matrix factorization (NMF) basis decomposition method. NMF alone provides a parts-based representation where nonnegative inputs are approximated by additive combinations of nonnegative basis functions. Our proposed model of topographic NMF (TNMF) incorporates neighborhood connections between NMF basis functions arranged on a topographic map and attains the topographic property without losing the parts-based property of the NMF. The TNMF represents an input by multiple activity peaks to describe diverse information, whereas conventional topographic models, such as the self-organizing map (SOM), represent an input by a single activity peak in a topographic map. We demonstrate the parts-based and topographic properties of the TNMF by constructing a hierarchical model for object recognition where the TNMF is at the top tier for learning high-level object features. The TNMF showed better generalization performance over NMF for a data set of continuous view change of an image and more robustly preserving the continuity of the view change in its object representation. Comparison of the outputs of our model with actual neural responses recorded in the IT indicates that the TNMF reconstructs the neuronal responses better than the SOM, giving plausibility to the parts-based learning of the model.

Cortical connections to area TE in monkey: hybrid modular and distributed organization

To investigate the fine anatomical organization of cortical inputs to visual association area TE, 2–3 small injections of retrograde tracers were made in macaque monkeys. Injections were made as a terminal procedure, after optical imaging and electrophysiological recording, and targeted to patches physiologically identified as object-selective. Retrogradely labeled neurons occurred in several unimodal visual areas, the superior temporal sulcus, intraparietal sulcus (IPS), and prefrontal cortex (PFC), consistent with previous studies. Despite the small injection size (<0.5 mm wide), the projection foci in visual areas, but not in IPS or PFC, were spatially widespread (4–6 mm in extent), and predominantly consisted of neurons labeled by only one of the injections. This can be seen as a quasi-modular organization. In addition, within each projection focus, there were scattered neurons projecting to one of the other injections, together with some double-labeled (DL) neurons, in a more distributed pattern. Finally, projection foci included smaller “hotspots,” consisting of intermixed neurons, single-labeled by the different injections, and DL neurons. DL neurons are likely the result of axons having extended, spatially separated terminal arbors, as demonstrated by anterograde experiments. These results suggest a complex, hybrid connectivity architecture, with both modular and distributed components.

Fine-scale spatial organization of face and object selectivity in the temporal lobe: do functional magnetic resonance imaging, optical imaging, and electrophysiology agree?

The spatial organization of the brain's object and face representations in the temporal lobe is critical for understanding high-level vision and cognition but is poorly understood. Recently, exciting progress has been made using advanced imaging and physiology methods in humans and nonhuman primates, and the combination of such methods may be particularly powerful. Studies applying these methods help us to understand how neuronal activity, optical imaging, and functional magnetic resonance imaging signals are related within the temporal lobe, and to uncover the fine-grained and large-scale spatial organization of object and face representations in the primate brain.

Using the light scattering component of optical intrinsic signals to visualize in vivo functional structures of neural tissues

Visualization of changes in reflected light from in vivo brain tissues reveals spatial patterns of neural activity. An important factor which influences the degree of light reflected includes the change in light scattering elicited by neural activation. Microstructures of neural tissues generally cause light scattering, and neural activities are associated with some changes in the microstructures. Here, we show that the optical properties unique to light scattering enable us to visualize spatial patterns of retinal activity non-invasively (FRG: functional retinography), and resolve functional structures in depth (fOCT: functional optical coherence tomography).

Cortical columnar organization is reconsidered in inferior temporal cortex

The object selectivity of nearby cells in inferior temporal (IT) cortex is often different. To elucidate the relationship between columnar organization in IT cortex and the variability among neurons with respect to object selectivity, we used optical imaging technique to locate columnar regions (activity spots) and systematically compared object selectivity of individual neurons within and across the spots. The object selectivity of a given cell in a spot was similar to that of the averaged cellular activity within the spot. However, there was not such similarity among different spots (>600 μm apart). We suggest that each cell is characterized by 1) a cell-specific response property that cause cell-to-cell variability in object selectivity and 2) one or potentially a few numbers of response properties common across the cells within a spot, which provide the basis for columnar organization in IT cortex. Furthermore, similarity in object selectivity among cells within a randomly chosen site was lower than that for a cell in an activity spot identified by optical imaging beforehand. We suggest that the cortex may be organized in a region where neurons with similar response properties were densely clustered and a region where neurons with similar response properties were sparsely clustered.

Multiple topological representation self-organized by spike-timing-dependent synaptic learning rule

Position-and-scale-free representations of shapes are acquired by neurons in the inferior temporal (IT) cortex. So each neuron receives information from the whole visual field. Familiar shapes are extremely restricted from all the possible shapes on the whole visual field. So they must be clustered in the shape space to have mixed structure of continuity and discreteness. We demonstrate that multiple representation can be acquired in a spike-based model for topological maps based on the spike-timing-dependent synaptic plasticity (STDP), subjected to a set of inputs on multiple rings, which is a simple example of mixed structure. In this representation, the position on each ring is represented by a center of active neurons and the difference of rings is represented by a detailed pattern of active neurons. Neurons in the same region exhibit high activities for an input on the other ring. The result is consistent with the fact observed in IT cortex that neighboring neurons exhibit different preferences while the region of active neurons is continuously shifted for continuous changes of object.

Hemodynamic changes in the infant cortex during the processing of featural and spatiotemporal information

Over the last 20 years neuroscientists have learned a great deal about the ventral and dorsal object processing pathways in the adult brain, yet little is known about the functional development of these pathways. The present research assessed the extent to which different patterns of neural activation, as measured by changes in blood volume and oxygenation, are observed in infant visual and temporal cortex in response to events that involve processing of featural differences or spatiotemporal discontinuities. Infants aged 6.5 months were tested. Increased neural activation was observed in visual cortex in response to a featural-difference and a spatiotemporal-discontinuity event. In addition, increased neural activation was observed in temporal cortex in response to the featural-difference but not the spatiotemporal-discontinuity event. The outcome of this experiment reveals early functional specialization of temporal cortex and lays the foundation for future investigation of the maturation of object processing pathways in humans.

Object representations for multiple visual categories overlap in lateral occipital and medial fusiform cortex

How representations of visual objects are maintained across changes in viewpoint is a central issue in visual perception. Whether neural processes underlying view-invariant recognition involve distinct subregions within extrastriate visual cortex for distinct categories of visual objects remains unresolved. We used event-related functional magnetic resonance imaging in 16 healthy volunteers to map visual cortical areas responding to a large set (156) of exemplars from 3 object categories (faces, houses, and chairs), each repeated once after a variable time lag (3-7 intervening stimuli). Exemplars were repeated with the same viewpoint (but different retinal size) or with different viewpoint and size. The task was kept constant across object categories (judging items as "young" vs. "old"). We identified object-selective adaptation effects by comparing neural responses to the first presentation versus repetition of each individual exemplar. We found that exemplar-specific adaptation effects partly overlapped with regions showing category-selective responses (as identified using a separate localizer scan). These included the lateral fusiform gyrus (FG) for faces, parahippocampal gyrus for houses, and lateral occipital complex (LOC) for chairs. In face-selective fusiform gyrus (FG), adaptation effects occurred only for faces repeated with the same viewpoint, but not with a different viewpoint, confirming previous studies using faces only. By contrast, a region in right medial FG, adjacent to but nonoverlapping with the more lateral and face-selective FG, showed repetition effects for faces and to a lesser extent for other objects, regardless of changes in viewpoint or in retinal image-size. Category- and viewpoint-independent repetition effects were also found in bilateral LOC. Our results reveal a common neural substrate in bilateral LOC and right medial FG underlying view-invariant and category-independent recognition for multiple object identities, with only a relative preference for faces in medial FG but no selectivity in LOC.

Mid-fusiform activation during object discrimination reflects the process of differentiating structural descriptions

The present study explored constraints on mid-fusiform activation during object discrimination. In three experiments, participants performed a matching task on simple line configurations, nameable objects, three dimensional (3-D) shapes, and colors. Significant bilateral mid-fusiform activation emerged when participants matched objects and 3-D shapes, as compared to when they matched two-dimensional (2-D) line configurations and colors, indicating that the mid-fusiform is engaged more strongly for processing structural descriptions (e.g., comparing 3-D volumetric shape) than perceptual descriptions (e.g., comparing 2-D or color information). In two of the experiments, the same mid-fusiform regions were also modulated by the degree of structural similarity between stimuli, implicating a role for the mid-fusiform in fine differentiation of similar visual object representations. Importantly, however, this process of fine differentiation occurred at the level of structural, but not perceptual, descriptions. Moreover, mid-fusiform activity was more robust when participants matched shape compared to color information using the identical stimuli, indicating that activity in the mid-fusiform gyrus is not driven by specific stimulus properties, but rather by the process of distinguishing stimuli based on shape information. Taken together, these findings further clarify the nature of object processing in the mid-fusiform gyrus. This region is engaged specifically in structural differentiation, a critical component process of object recognition and categorization.

Hemodynamic response to featural changes in the occipital and inferior temporal cortex in infants: a preliminary methodological exploration

Over the past 30 years researchers have learned a great deal about the development of object processing in infancy. In contrast, little is understood about the neural mechanisms that underlie this capacity, in large part because there are few techniques available to measure brain functioning in human infants. The present research examined the extent to which near-infrared spectroscopy (NIRS), an optical imaging technique, could be used to assess the relation between object processing and brain functioning. Infants aged 6.5 months were presented with an occlusion event involving objects that differed on many feature dimensions (multi-featural change), differed on shape only (shape change) or color only (color change), or did not differ (control). NIRS data were collected in the occipital and inferior temporal cortex. In the occipital cortex, a significant increase in oxyhemoglobin (HbO(2)) was observed in response to all four events and these responses did not differ significantly from each other. In the inferior temporal cortex, a significant increase in HbO(2 )was observed in the multi-featural and the shape change condition but not in the control condition. An increase was also observed in the color change condition but this increase did not differ significantly from baseline nor did it differ significantly from the response obtained in the control condition. These data were discussed in terms of (a) what they suggest about the neural basis of feature processing in infants and (b) the viability of using NIRS to study brain-behavior relations in infants.

Category-specific responses to faces and objects in primate auditory cortex

Auditory and visual signals often occur together, and the two sensory channels are known to influence each other to facilitate perception. The neural basis of this integration is not well understood, although other forms of multisensory influences have been shown to occur at surprisingly early stages of processing in cortex. Primary visual cortex neurons can show frequency-tuning to auditory stimuli, and auditory cortex responds selectively to certain somatosensory stimuli, supporting the possibility that complex visual signals may modulate early stages of auditory processing. To elucidate which auditory regions, if any, are responsive to complex visual stimuli, we recorded from auditory cortex and the superior temporal sulcus while presenting visual stimuli consisting of various objects, neutral faces, and facial expressions generated during vocalization. Both objects and conspecific faces elicited robust field potential responses in auditory cortex sites, but the responses varied by category: both neutral and vocalizing faces had a highly consistent negative component (N100) followed by a broader positive component (P180) whereas object responses were more variable in time and shape, but could be discriminated consistently from the responses to faces. The face response did not vary within the face category, i.e., for expressive vs. neutral face stimuli. The presence of responses for both objects and neutral faces suggests that auditory cortex receives highly informative visual input that is not restricted to those stimuli associated with auditory components. These results reveal selectivity for complex visual stimuli in a brain region conventionally described as non-visual “unisensory” cortex.

Feedback decoding of spatially structured population activity in cortical maps

A mechanism is proposed by which feedback pathways model spatial patterns of feedforward activity in cortical maps. The mechanism can be viewed equivalently as readout of a content-addressable memory or as decoding of a population code. The model is based on the evidence that cortical receptive fields can often be described as a separable product of functions along several dimensions, each represented in a spatially ordered map. Given this, it is shown that for an N-dimensional map, accurate modeling and decoding of x(N) feedforward activity patterns can be done with Nx fibers, N of which must be active at any one time. The proposed mechanism explains several known properties of the cortex and pyramidal neurons: (1) the integration of signals by dendrites with a narrow tangential distribution, that is, apical dendrites; (2) the presence of fast-conducting feedback projections with broad tangential distributions; (3) the multiplicative effects of attention on receptive field profiles; and (4) the existence of multiplicative interactions between subthreshold feedforward inputs to basal dendrites and inputs to apical dendrites.

A stable topography of selectivity for unfamiliar shape classes in monkey inferior temporal cortex

The inferior temporal (IT) cortex in monkeys plays a central role in visual object recognition and learning. Previous studies have observed patches in IT cortex with strong selectivity for highly familiar object classes (e.g., faces), but the principles behind this functional organization are largely unknown due to the many properties that distinguish different object classes. To unconfound shape from meaning and memory, we scanned monkeys with functional magnetic resonance imaging while they viewed classes of initially novel objects. Our data revealed a topography of selectivity for these novel object classes across IT cortex. We found that this selectivity topography was highly reproducible and remarkably stable across a 3-month interval during which monkeys were extensively trained to discriminate among exemplars within one of the object classes. Furthermore, this selectivity topography was largely unaffected by changes in behavioral task and object retinal position, both of which preserve shape. In contrast, it was strongly influenced by changes in object shape. The topography was partially related to, but not explained by, the previously described pattern of face selectivity. Together, these results suggest that IT cortex contains a large-scale map of shape that is largely independent of meaning, familiarity, and behavioral task.

Object category structure in response patterns of neuronal population in monkey inferior temporal cortex

Our mental representation of object categories is hierarchically organized, and our rapid and seemingly effortless categorization ability is crucial for our daily behavior. Here, we examine responses of a large number (>600) of neurons in monkey inferior temporal (IT) cortex with a large number (>1,000) of natural and artificial object images. During the recordings, the monkeys performed a passive fixation task. We found that the categorical structure of objects is represented by the pattern of activity distributed over the cell population. Animate and inanimate objects created distinguishable clusters in the population code. The global category of animate objects was divided into bodies, hands, and faces. Faces were divided into primate and nonprimate faces, and the primate-face group was divided into human and monkey faces. Bodies of human, birds, and four-limb animals clustered together, whereas lower animals such as fish, reptile, and insects made another cluster. Thus the cluster analysis showed that IT population responses reconstruct a large part of our intuitive category structure, including the global division into animate and inanimate objects, and further hierarchical subdivisions of animate objects. The representation of categories was distributed in several respects, e.g., the similarity of response patterns to stimuli within a category was maintained by both the cells that maximally responded to the category and the cells that responded weakly to the category. These results advance our understanding of the nature of the IT neural code, suggesting an inherently categorical representation that comprises a range of categories including the amply investigated face category.

Long-term optical imaging of intrinsic signals in anesthetized and awake monkeys

Some exciting new efforts to use intrinsic signal optical imaging methods for long-term studies in anesthetized and awake monkeys are reviewed. The development of such methodologies opens the door for studying behavioral states such as attention, motivation, memory, emotion, and other higher-order cognitive functions. Long-term imaging is also ideal for studying changes in the brain that accompany development, plasticity, and learning. Although intrinsic imaging lacks the temporal resolution offered by dyes, it is a high spatial resolution imaging method that does not require application of any external agents to the brain. The bulk of procedures described here have been developed in the monkey but can be applied to the study of surface structures in any in vivo preparation.

Object recognition learning differentiates the representations of objects at the ERP component N1

OBJECTIVE

Even if viewed from different angles, one can identify an object among similar distractors through learning. This study was designed to investigate the changes in neuronal activity related to learning.

METHODS

Human subjects were asked to train themselves with novel objects by performing an object recognition task, in which the images of an object had to be discriminated from those of other objects regardless of the viewpoint.

RESULTS

The ERP component-N1, the first negative peak at posterior electrodes, showed a significant increase in the amplitude variation across the objects during the learning process, while the variation across viewpoints decreased.

CONCLUSIONS

These results suggest that object recognition learning differentiates between the representations of the objects, at least, at the N1 level.

SIGNIFICANCE

The results may support the notion that object recognition differentiates among the functional representations of the trained objects in our brain.

Representation of the Spatial Relationship Among Object Parts by Neurons in Macaque Inferotemporal Cortex

We investigated object representation in area TE, the anterior part of monkey inferotemporal (IT) cortex, with a combination of optical and extracellular recordings in anesthetized monkeys. We found neurons that respond to visual stimuli composed of naturally distinguishable parts. These neurons were sensitive to a particular spatial arrangement of parts but less sensitive to differences in local features within individual parts. Thus these neurons were activated when arbitrary local features were arranged in a particular spatial configuration, suggesting that they may be responsible for representing the spatial configuration of object images. Previously it has been reported that many neurons in area TE respond to visual features less complex than natural objects, but it has remained unclear whether these features are related to local features of object images or to more global features. These results indicate that TE neurons represent not only local features but also global features such as the spatial relationship among object parts.