How position dependent is visual object recognition?

The power of connectivity: identity preserving transformations on visual streams in the spike domain

We investigate neural architectures for identity preserving transformations (IPTs) on visual stimuli in the spike domain. The stimuli are encoded with a population of spiking neurons; the resulting spikes are processed and finally decoded. A number of IPTs are demonstrated including faithful stimulus recovery, as well as simple transformations on the original visual stimulus such as translations, rotations and zoomings. We show that if the set of receptive fields satisfies certain symmetry properties, then IPTs can easily be realized and additionally, the same basic stimulus decoding algorithm can be employed to recover the transformed input stimulus. Using group theoretic methods we advance two different neural encoding architectures and discuss the realization of exact and approximate IPTs. These are realized in the spike domain processing block by a "switching matrix" that regulates the input/output connectivity between the stimulus encoding and decoding blocks. For example, for a particular connectivity setting of the switching matrix, the original stimulus is faithfully recovered. For other settings, translations, rotations and dilations (or combinations of these operations) of the original video stream are obtained. We evaluate our theoretical derivations through extensive simulations on natural video scenes, and discuss implications of our results on the problem of invariant object recognition in the spike domain.

Visual interhemispheric communication and callosal connections of the occipital lobes

Callosal connections of the occipital lobes, coursing in the splenium of the corpus callosum, have long been thought to be crucial for interactions between the cerebral hemispheres in vision in both experimental animals and humans. Yet the callosal connections of the temporal and parietal lobes appear to have more important roles than those of the occipital callosal connections in at least some high-order interhemispheric visual functions. The partial intermixing and overlap of temporal, parietal and occipital callosal connections within the splenium has made it difficult to attribute the effects of splenial pathological lesions or experimental sections to splenial components specifically related to select cortical areas. The present review describes some current contributions from the modern techniques for the tracking of commissural fibers within the living human brain to the tentative assignation of specific visual functions to specific callosal tracts, either occipital or extraoccipital.

The ventral visual pathway: an expanded neural framework for the processing of object quality

Since the original characterization of the ventral visual pathway, our knowledge of its neuroanatomy, functional properties, and extrinsic targets has grown considerably. Here we synthesize this recent evidence and propose that the ventral pathway is best understood as a recurrent occipitotemporal network containing neural representations of object quality both utilized and constrained by at least six distinct cortical and subcortical systems. Each system serves its own specialized behavioral, cognitive, or affective function, collectively providing the raison d'être for the ventral visual pathway. This expanded framework contrasts with the depiction of the ventral visual pathway as a largely serial staged hierarchy culminating in singular object representations and more parsimoniously incorporates attentional, contextual, and feedback effects.

The transfer of object learning across exemplars and their orientation is related to perceptual similarity

Recognition of objects improves after training. The exact characteristics of this visual learning process remain unclear. We examined to which extent object learning depends on the exact exemplar and orientation used during training. Participants were trained to name object pictures at as short a picture presentation time as possible. The required presentation time diminished over training. After training participants were tested with a completely new set of objects as well as with two variants of the trained object set, namely an orientation change and a change of the exact exemplar shown. Both manipulations led to a decrease in performance compared to the original picture set. Nevertheless, performance with the manipulated versions of the trained stimuli was better than performance with the completely new set, at least when only one manipulation was performed. Amount of transfer to new images of an object was related to perceptual similarity, but not to pixel overlap or to measurements of similarity in the different layers of a popular hierarchical object recognition model (HMAX). Thus, object learning generalizes only partially over changes in exemplars and orientation, which is consistent with the tuning properties of neurons in object-selective cortical regions and the role of perceptual similarity in these representations.

Renewing the respect for similarity

In psychology, the concept of similarity has traditionally evoked a mixture of respect, stemming from its ubiquity and intuitive appeal, and concern, due to its dependence on the framing of the problem at hand and on its context. We argue for a renewed focus on similarity as an explanatory concept, by surveying established results and new developments in the theory and methods of similarity-preserving associative lookup and dimensionality reduction-critical components of many cognitive functions, as well as of intelligent data management in computer vision. We focus in particular on the growing family of algorithms that support associative memory by performing hashing that respects local similarity, and on the uses of similarity in representing structured objects and scenes. Insofar as these similarity-based ideas and methods are useful in cognitive modeling and in AI applications, they should be included in the core conceptual toolkit of computational neuroscience. In support of this stance, the present paper (1) offers a discussion of conceptual, mathematical, computational, and empirical aspects of similarity, as applied to the problems of visual object and scene representation, recognition, and interpretation, (2) mentions some key computational problems arising in attempts to put similarity to use, along with their possible solutions, (3) briefly states a previously developed similarity-based framework for visual object representation, the Chorus of Prototypes, along with the empirical support it enjoys, (4) presents new mathematical insights into the effectiveness of this framework, derived from its relationship to locality-sensitive hashing (LSH) and to concomitant statistics, (5) introduces a new model, the Chorus of Relational Descriptors (ChoRD), that extends this framework to scene representation and interpretation, (6) describes its implementation and testing, and finally (7) suggests possible directions in which the present research program can be extended in the future.

Position sensitivity in the visual word form area

Seeing words involves the activity of neural circuitry within a small region in human ventral temporal cortex known as the visual word form area (VWFA). It is widely asserted that VWFA responses, which are essential for skilled reading, do not depend on the visual field position of the writing (position invariant). Such position invariance supports the hypothesis that the VWFA analyzes word forms at an abstract level, far removed from specific stimulus features. Using functional MRI pattern-classification techniques, we show that position information is encoded in the spatial pattern of VWFA responses. A right-hemisphere homolog (rVWFA) shows similarly position-sensitive responses. Furthermore, electrophysiological recordings in the human brain show position-sensitive VWFA response latencies. These findings show that position-sensitive information is present in the neural circuitry that conveys visual word form information to language areas. The presence of position sensitivity in the VWFA has implications for how word forms might be learned and stored within the reading circuitry.

Transformation-tolerant object recognition in rats revealed by visual priming

Successful use of rodents as models for studying object vision crucially depends on the ability of their visual system to construct representations of visual objects that tolerate (i.e., remain relatively unchanged with respect to) the tremendous changes in object appearance produced, for instance, by size and viewpoint variation. Whether this is the case is still controversial, despite some recent demonstration of transformation-tolerant object recognition in rats. In fact, it remains unknown to what extent such a tolerant recognition has a spontaneous, perceptual basis, or, alternatively, mainly reflects learning of arbitrary associative relations among trained object appearances. In this study, we addressed this question by training rats to categorize a continuum of morph objects resulting from blending two object prototypes. The resulting psychometric curve (reporting the proportion of responses to one prototype along the morph line) served as a reference when, in a second phase of the experiment, either prototype was briefly presented as a prime, immediately before a test morph object. The resulting shift of the psychometric curve showed that recognition became biased toward the identity of the prime. Critically, this bias was observed also when the primes were transformed along a variety of dimensions (i.e., size, position, viewpoint, and their combination) that the animals had never experienced before. These results indicate that rats spontaneously perceive different views/appearances of an object as similar (i.e., as instances of the same object) and argue for the existence of neuronal substrates underlying formation of transformation-tolerant object representations in rats.

How does the brain solve visual object recognition?

Mounting evidence suggests that 'core object recognition,' the ability to rapidly recognize objects despite substantial appearance variation, is solved in the brain via a cascade of reflexive, largely feedforward computations that culminate in a powerful neuronal representation in the inferior temporal cortex. However, the algorithm that produces this solution remains poorly understood. Here we review evidence ranging from individual neurons and neuronal populations to behavior and computational models. We propose that understanding this algorithm will require using neuronal and psychophysical data to sift through many computational models, each based on building blocks of small, canonical subnetworks with a common functional goal.

Grounding word learning in space

Humans and objects, and thus social interactions about objects, exist within space. Words direct listeners' attention to specific regions of space. Thus, a strong correspondence exists between where one looks, one's bodily orientation, and what one sees. This leads to further correspondence with what one remembers. Here, we present data suggesting that children use associations between space and objects and space and words to link words and objects—space binds labels to their referents. We tested this claim in four experiments, showing that the spatial consistency of where objects are presented affects children's word learning. Next, we demonstrate that a process model that grounds word learning in the known neural dynamics of spatial attention, spatial memory, and associative learning can capture the suite of results reported here. This model also predicts that space is special, a prediction supported in a fifth experiment that shows children do not use color as a cue to bind words and objects. In a final experiment, we ask whether spatial consistency affects word learning in naturalistic word learning contexts. Children of parents who spontaneously keep objects in a consistent spatial location during naming interactions learn words more effectively. Together, the model and data show that space is a powerful tool that can effectively ground word learning in social contexts.

Division of labor between lateral and ventral extrastriate representations of faces, bodies, and objects

The occipito-temporal cortex is strongly implicated in carrying out the high-level computations associated with vision. In human neuroimaging studies, focal regions are consistently found within this broad region that respond strongly and selectively to faces, bodies, or objects. A notable feature of these selective regions is that they are found in pairs. In the posterior-lateral occipito-temporal cortex, focal selectivity is found for faces (occipital face area), bodies (extrastriate body area), and objects (lateral occipital). These three areas are found bilaterally and at close quarters to each other. Likewise, in the ventro-medial occipito-temporal cortex, three similar category-selective regions are found, also in proximity to each other: for faces (fusiform face area), bodies (fusiform body area), and objects (posterior fusiform). Here we review some of the extensive evidence on the functional properties of these areas with two aims. First, we seek to identify principles that distinguish the posterior-lateral and ventro-medial clusters of selective regions but that apply generally within each cluster across the three stimulus kinds. Our review identifies and elaborates several principles by which these relationships hold. In brief, the posterior-lateral representations are more primitive, local, and stimulus-driven relative to the ventro-medial representations, which in contrast are more invariant to visual features, global, and linked to the subjective percept. Second, because the evidence base of studies that compare both posterior-lateral and ventro-medial representations of faces, bodies, and objects is still relatively small, we seek to provoke more cross-talk among the research strands that are traditionally separate. We identify several promising approaches for such future work.

Imaging retinotopic maps in the human brain

A quarter-century ago visual neuroscientists had little information about the number and organization of retinotopic maps in human visual cortex. The advent of functional magnetic resonance imaging (MRI), a non-invasive, spatially-resolved technique for measuring brain activity, provided a wealth of data about human retinotopic maps. Just as there are differences amongst non-human primate maps, the human maps have their own unique properties. Many human maps can be measured reliably in individual subjects during experimental sessions lasting less than an hour. The efficiency of the measurements and the relatively large amplitude of functional MRI signals in visual cortex make it possible to develop quantitative models of functional responses within specific maps in individual subjects. During this last quarter-century, there has also been significant progress in measuring properties of the human brain at a range of length and time scales, including white matter pathways, macroscopic properties of gray and white matter, and cellular and molecular tissue properties. We hope the next 25years will see a great deal of work that aims to integrate these data by modeling the network of visual signals. We do not know what such theories will look like, but the characterization of human retinotopic maps from the last 25years is likely to be an important part of future ideas about visual computations.

Population coding of visual space: modeling

We examine how the representation of space is affected by receptive field (RF) characteristics of the encoding population. Spatial responses were defined by overlapping Gaussian RFs. These responses were analyzed using multidimensional scaling to extract the representation of global space implicit in population activity. Spatial representations were based purely on firing rates, which were not labeled with RF characteristics (tuning curve peak location, for example), differentiating this approach from many other population coding models. Because responses were unlabeled, this model represents space using intrinsic coding, extracting relative positions amongst stimuli, rather than extrinsic coding where known RF characteristics provide a reference frame for extracting absolute positions. Two parameters were particularly important: RF diameter and RF dispersion, where dispersion indicates how broadly RF centers are spread out from the fovea. For large RFs, the model was able to form metrically accurate representations of physical space on low-dimensional manifolds embedded within the high-dimensional neural population response space, suggesting that in some cases the neural representation of space may be dimensionally isomorphic with 3D physical space. Smaller RF sizes degraded and distorted the spatial representation, with the smallest RF sizes (present in early visual areas) being unable to recover even a topologically consistent rendition of space on low-dimensional manifolds. Finally, although positional invariance of stimulus responses has long been associated with large RFs in object recognition models, we found RF dispersion rather than RF diameter to be the critical parameter. In fact, at a population level, the modeling suggests that higher ventral stream areas with highly restricted RF dispersion would be unable to achieve positionally-invariant representations beyond this narrow region around fixation.

Population coding of visual space: comparison of spatial representations in dorsal and ventral pathways

Although the representation of space is as fundamental to visual processing as the representation of shape, it has received relatively little attention from neurophysiological investigations. In this study we characterize representations of space within visual cortex, and examine how they differ in a first direct comparison between dorsal and ventral subdivisions of the visual pathways. Neural activities were recorded in anterior inferotemporal cortex (AIT) and lateral intraparietal cortex (LIP) of awake behaving monkeys, structures associated with the ventral and dorsal visual pathways respectively, as a stimulus was presented at different locations within the visual field. In spatially selective cells, we find greater modulation of cell responses in LIP with changes in stimulus position. Further, using a novel population-based statistical approach (namely, multidimensional scaling), we recover the spatial map implicit within activities of neural populations, allowing us to quantitatively compare the geometry of neural space with physical space. We show that a population of spatially selective LIP neurons, despite having large receptive fields, is able to almost perfectly reconstruct stimulus locations within a low-dimensional representation. In contrast, a population of AIT neurons, despite each cell being spatially selective, provide less accurate low-dimensional reconstructions of stimulus locations. They produce instead only a topologically (categorically) correct rendition of space, which nevertheless might be critical for object and scene recognition. Furthermore, we found that the spatial representation recovered from population activity shows greater translation invariance in LIP than in AIT. We suggest that LIP spatial representations may be dimensionally isomorphic with 3D physical space, while in AIT spatial representations may reflect a more categorical representation of space (e.g., "next to" or "above").

The n170 shows differential repetition effects for faces, objects, and orthographic stimuli

Previous event-related potentials research has suggested that the N170 component has a larger amplitude to faces and words than to other stimuli, but it remains unclear whether it indexes the same cognitive processes for faces and for words. The present study investigated how category-level repetition effects on the N170 differ across stimulus categories. Faces, cars, words, and non-words were presented in homogeneous (1 category) or mixed blocks (2 intermixed categories). We found a significant repetition effect of N170 amplitude for successively presented faces and cars (in homogeneous blocks), but not for words and unpronounceable consonant strings, suggesting that the N170 indexes different underlying cognitive processes for objects (including faces) and orthographic stimuli. The N170 amplitude was significantly smaller when multiple faces or multiple cars were presented in a row than when these stimuli were preceded by a stimulus of a different category. Moreover, the large N170 repetition effect for faces may be important to consider when comparing the relative N170 amplitude for different stimulus categories. Indeed, a larger N170 deflection for faces than for other stimulus categories was observed only when stimuli were preceded by a stimulus of a different category (in mixed blocks), suggesting that an enhanced N170 to faces may be more reliably observed when faces are presented within the context of some non-face stimuli.

Spatial heterogeneity in the perception of face and form attributes

The identity of an object is a fixed property, independent of where it appears, and an effective visual system should capture this invariance [1-3]. However, we now report that the perceived gender of a face is strongly biased toward male or female at different locations in the visual field. The spatial pattern of these biases was distinctive and stable for each individual. Identical neutral faces looked different when they were presented simultaneously at locations maximally biased to opposite genders. A similar effect was observed for perceived age of faces. We measured the magnitude of this perceptual heterogeneity for four other visual judgments: perceived aspect ratio, orientation discrimination, spatial-frequency discrimination, and color discrimination. The effect was sizeable for the aspect ratio task but substantially smaller for the other three tasks. We also evaluated perceptual heterogeneity for facial gender and orientation tasks at different spatial scales. Strong heterogeneity was observed even for the orientation task when tested at small scales. We suggest that perceptual heterogeneity is a general property of visual perception and results from undersampling of the visual signal at spatial scales that are small relative to the size of the receptive fields associated with each visual attribute.

Encoding the identity and location of objects in human LOC

We are able to recognize objects independent of their location in the visual field. At the same time, we also keep track of the location of objects to orient ourselves and to interact with the environment. The lateral occipital complex (LOC) has been suggested as the prime cortical region for representation of object identity. However, the extent to which LOC also represents object location has remained debated. In this study we used high-resolution fMRI in combination with multivoxel pattern classification to investigate the cortical encoding of three object exemplars from four different categories presented in two different locations. This approach allowed us to study location-tolerant object information and object-tolerant location information in LOC, both at the level of categories and exemplars. We found evidence for both location-tolerant object information and object-tolerant location information in LOC at the level of categories and exemplars. Our results further highlight the mixing of identity and location information in the ventral visual pathway.

Equal degrees of object selectivity for upper and lower visual field stimuli

Functional MRI (fMRI) studies of the human object recognition system commonly identify object-selective cortical regions by comparing blood oxygen level-dependent (BOLD) responses to objects versus those to scrambled objects. Object selectivity distinguishes human lateral occipital cortex (LO) from earlier visual areas. Recent studies suggest that, in addition to being object selective, LO is retinotopically organized; LO represents both object and location information. Although LO responses to objects have been shown to depend on location, it is not known whether responses to scrambled objects vary similarly. This is important because it would suggest that the degree of object selectivity in LO does not vary with retinal stimulus position. We used a conventional functional localizer to identify human visual area LO by comparing BOLD responses to objects versus scrambled objects presented to either the upper (UVF) or lower (LVF) visual field. In agreement with recent findings, we found evidence of position-dependent responses to objects. However, we observed the same degree of position dependence for scrambled objects and thus object selectivity did not differ for UVF and LVF stimuli. We conclude that, in terms of BOLD response, LO discriminates objects from non-objects equally well in either visual field location, despite stronger responses to objects in the LVF.

Broad and narrow conceptual tuning in the human frontal lobes

Previous work has implicated prefrontal cortices in selecting among and retrieving conceptual information stored elsewhere. However, recent neurophysiological work in monkeys suggests that prefrontal cortex may play a more direct role in representing conceptual information in a flexible context-specific manner. Here, we investigate the nature of visual object representations from perceptual to conceptual levels in an unbiased data-driven manner using a functional magnetic resonance imaging adaptation paradigm with pictures of animals. Throughout much of occipital cortex, activity was highly sensitive to changes in 2D stimulus form, consistent with tuning to form and position within retinotopic coordinates and matching an automated measure of shape similarity. Broad superordinate conceptual information was represented as early as extrastriate and posterior ventral temporal cortex. These regions were not completely invariant to form, suggesting that form similarity remains an important organizational constraint into the temporal cortex. Separate sites within prefrontal cortex represented broad and narrow conceptual tuning, with more anterior sites tuned narrowly to close conceptual associates in a manner that was invariant to stimulus form/position and that matched independent similarity ratings of the stimuli. The combination of broad and narrow conceptual tuning within prefrontal cortex may support flexible selection, retrieval, and classification of objects at different levels of categorical abstraction.

Continuous transformation learning of translation invariant representations

We show that spatial continuity can enable a network to learn translation invariant representations of objects by self-organization in a hierarchical model of cortical processing in the ventral visual system. During 'continuous transformation learning', the active synapses from each overlapping transform are associatively modified onto the set of postsynaptic neurons. Because other transforms of the same object overlap with previously learned exemplars, a common set of postsynaptic neurons is activated by the new transforms, and learning of the new active inputs onto the same postsynaptic neurons is facilitated. We show that the transforms must be close for this to occur; that the temporal order of presentation of each transformed image during training is not crucial for learning to occur; that relatively large numbers of transforms can be learned; and that such continuous transformation learning can be usefully combined with temporal trace training.

Invariance in visual object recognition requires training: a computational argument

Visual object recognition is remarkably accurate and robust, yet its neurophysiological underpinnings are poorly understood. Single cells in brain regions thought to underlie object recognition code for many stimulus aspects, which poses a limit on their invariance. Combining the responses of multiple non-invariant neurons via weighted linear summation offers an optimal decoding strategy, which may be able to achieve invariant object recognition. However, because object identification is essentially parameter optimization in this model, the characteristics of the identification task trained to perform are critically important. If this task does not require invariance, a neural population-code is inherently more selective but less tolerant than the single-neurons constituting the population. Nevertheless, tolerance can be learned - provided that it is trained for - at the cost of selectivity. We argue that this model is an interesting null-hypothesis to compare behavioral results with and conclude that it may explain several experimental findings.

High-level visual object representations are constrained by position

It is widely assumed that high-level visual object representations are position-independent (or invariant). While there is sensitivity to position in high-level object-selective cortex, position and object identity are thought to be encoded independently in the population response such that position information is available across objects and object information is available across positions. Contrary to this view, we show, with both behavior and neuroimaging, that visual object representations are position-dependent (tied to limited portions of the visual field). Behaviorally, we show that the effect of priming an object was greatly reduced with any change in position (within- or between-hemifields), indicating nonoverlapping representations of the same object across different positions. Furthermore, using neuroimaging, we show that object-selective cortex is not only highly sensitive to object position but also the ability to differentiate objects based on its response is greatly reduced across different positions, consistent with the observed behavior and the receptive field properties observed in macaque object-selective neurons. Thus, even at the population level, the object information available in response of object-selective cortex is constrained by position. We conclude that even high-level visual object representations are position-dependent.

Emotion separation is completed early and it depends on visual field presentation

It is now apparent that the visual system reacts to stimuli very fast, with many brain areas activated within 100 ms. It is, however, unclear how much detail is extracted about stimulus properties in the early stages of visual processing. Here, using magnetoencephalography we show that the visual system separates different facial expressions of emotion well within 100 ms after image onset, and that this separation is processed differently depending on where in the visual field the stimulus is presented. Seven right-handed males participated in a face affect recognition experiment in which they viewed happy, fearful and neutral faces. Blocks of images were shown either at the center or in one of the four quadrants of the visual field. For centrally presented faces, the emotions were separated fast, first in the right superior temporal sulcus (STS; 35-48 ms), followed by the right amygdala (57-64 ms) and medial pre-frontal cortex (83-96 ms). For faces presented in the periphery, the emotions were separated first in the ipsilateral amygdala and contralateral STS. We conclude that amygdala and STS likely play a different role in early visual processing, recruiting distinct neural networks for action: the amygdala alerts sub-cortical centers for appropriate autonomic system response for fight or flight decisions, while the STS facilitates more cognitive appraisal of situations and links appropriate cortical sites together. It is then likely that different problems may arise when either network fails to initiate or function properly.

Translational and reflectional priming invariance: a retrospective

Biederman and Cooper (1991a) showed that the presentation of a briefly presented image of an object at one position in the visual field facilitated its identification, as assessed by naming speed and accuracy, several minutes later. The facilitation was unaffected by a translation or a reflection of the stimulus. A component of this priming was visual rather than basic-level conceptual or lexical in that there was less facilitation for an object with the same name (and basic-level class) but a different shape. The invariance of priming to view variables has stood up well over the years and appears to be a general phenomenon--as long as the original structural description can be readily resolved--in that it has also been observed for variations in size and orientation in depth. Although priming was unaffected by a change in position, we documented that there was explicit memory for the position (and orientation and size) of the stimulus. The existence of two forms of representation from the identical stimulus presentation--one invariant and the other dependent on view variables--poses a challenge as to what can be concluded about view invariance from single-unit activity.

Ready-Made and Self-Made Facilitation Effects of Arrays

The study investigates the relationship between array priming and the self-generated conceptualization of arrays in spatial memory. Nursery and primary school age children and adults (N = 70) were tested with an object and place memory reaction-time/accuracy task, once first using a frame (containment and figurative thought) and (in another session) using a grid (explicit boundaries around places). They were also given the Common Region Test (CRT) with which drawing of object-place vs. object-region binding was tested. Object memory was better than place memory in 5-year-olds, but place memory had caught up in older children. Ten-year-olds showed already an accuracy comparable to young adults, and they also remembered places somewhat better and faster than object shapes. Experiencing the explicitly denoted places in the grid in the first session improved place memory in the second session with the frame, but not vice versa. Object-place binding in the CRT predicted better object than place memory, while object-region binding predicted place memory equal to or better than object memory. These binding strategies statistically eliminated the array priming effects of the grid, showing a trade-off between “self-made” spatial encoding strategy and priming with a “ready-made” spatial array structure.

Neural representations that support invariant object recognition

Neural mechanisms underlying invariant behaviour such as object recognition are not well understood. For brain regions critical for object recognition, such as inferior temporal cortex (ITC), there is now ample evidence indicating that single cells code for many stimulus aspects, implying that only a moderate degree of invariance is present. However, recent theoretical and empirical work seems to suggest that integrating responses of multiple non-invariant units may produce invariant representations at population level. We provide an explicit test for the hypothesis that a linear read-out mechanism of a pool of units resembling ITC neurons may achieve invariant performance in an identification task. A linear classifier was trained to decode a particular value in a 2-D stimulus space using as input the response pattern across a population of units. Only one dimension was relevant for the task, and the stimulus location on the irrelevant dimension (ID) was kept constant during training. In a series of identification tests, the stimulus location on the relevant dimension (RD) and ID was manipulated, yielding estimates for both the level of sensitivity and tolerance reached by the network. We studied the effects of several single-cell characteristics as well as population characteristics typically considered in the literature, but found little support for the hypothesis. While the classifier averages out effects of idiosyncratic tuning properties and inter-unit variability, its invariance is very much determined by the (hypothetical) ‘average’ neuron. Consequently, even at population level there exists a fundamental trade-off between selectivity and tolerance, and invariant behaviour does not emerge spontaneously.

Does learned shape selectivity in inferior temporal cortex automatically generalize across retinal position?

Biological visual systems have the remarkable ability to recognize objects despite confounding factors such as object position, size, pose, and lighting. In primates, this ability likely results from neuronal responses at the highest stage of the ventral visual stream [inferior temporal cortex (IT)] that signal object identity while tolerating these factors. However, for even the apparently simplest IT tolerance ("invariance"), tolerance to object position on the retina, little is known about how this feat is achieved. One possibility is that IT position tolerance is innate in that discriminatory power for newly learned objects automatically generalizes across position. Alternatively, visual experience plays a role in developing position tolerance. To test these ideas, we trained adult monkeys in a difficult object discrimination task in which their visual experience with novel objects was restricted to a single retinal position. After training, we recorded the spiking activity of an unbiased population of IT neurons and found that it contained significantly greater selectivity among the newly learned objects at the experienced position compared with a carefully matched, non-experienced position. Interleaved testing with other objects shows that this difference cannot be attributed to a bias in spatial attention or neuronal sampling. We conclude from these results that, at least under some conditions, full transfer of IT neuronal selectivity across retinal position is not automatic. This finding raises the possibility that visual experience plays a role in building neuronal tolerance in the ventral visual stream and the recognition abilities it supports.