Rapid serial visual presentation for the determination of n eural selectivity in area STSa

Recognition of brief sounds in rapid serial auditory presentation

Two experiments were conducted to test the role of participant factors (i.e., musical sophistication, working memory capacity) and stimulus factors (i.e., sound duration, timbre) on auditory recognition using a rapid serial auditory presentation paradigm. Participants listened to a rapid stream of very brief sounds ranging from 30 to 150 milliseconds and were tested on their ability to distinguish the presence from the absence of a target sound selected from various sound sources placed amongst the distracters. Experiment 1a established that brief exposure to stimuli (60 to 150 milliseconds) does not necessarily correspond to impaired recognition. In Experiment 1b we found evidence that 30 milliseconds of exposure to the stimuli significantly impairs recognition of single auditory targets, but the recognition for voice and sine tone targets impaired the least, suggesting that the lower limit required for successful recognition could be lower than 30 milliseconds for voice and sine tone targets. Critically, the effect of sound duration on recognition completely disappeared when differences in musical sophistication were controlled for. Participants' working memory capacities did not seem to predict their recognition performances. Our behavioral results extend the studies oriented to understand the processing of brief timbres under temporal constraint by suggesting that the musical sophistication may play a larger role than previously thought. These results can also provide a working hypothesis for future research, namely, that underlying neural mechanisms for the processing of various sound sources may have different temporal constraints.

Varying Stimulus Duration Reveals Consistent Neural Activity and Behavior for Human Face Individuation

Establishing consistent relationships between neural activity and behavior is a challenge in human cognitive neuroscience research. We addressed this issue using variable time constraints in an oddball frequency-sweep design for visual discrimination of complex images (face exemplars). Sixteen participants viewed sequences of ascending presentation durations, from 25 to 333 ms (40-3 Hz stimulation rate) while their electroencephalogram (EEG) was recorded. Throughout each sequence, the same unfamiliar face picture was repeated with variable size and luminance changes while different unfamiliar facial identities appeared every 1 s (1 Hz). A neural face individuation response, tagged at 1 Hz and its unique harmonics, emerged over the occipito-temporal cortex at 50 ms stimulus duration (25-100 ms across individuals), with an optimal response reached at 170 ms stimulus duration. In a subsequent experiment, identity changes appeared non-periodically within fixed-frequency sequences while the same participants performed an explicit face individuation task. The behavioral face individuation response also emerged at 50 ms presentation time, and behavioral accuracy correlated with individual participants' neural response amplitude in a weighted middle stimulus duration range (50-125 ms). Moreover, the latency of the neural response peaking between 180 and 200 ms correlated strongly with individuals' behavioral accuracy in this middle duration range, as measured independently. These observations point to the minimal (50 ms) and optimal (170 ms) stimulus durations for human face individuation and provide novel evidence that inter-individual differences in the magnitude and latency of early, high-level neural responses are predictive of behavioral differences in performance at this function.

All-or-none face categorization in the human brain

Visual categorization is integral for our interaction with the natural environment. In this process, similar selective responses are produced to a class of variable visual inputs. Whether categorization is supported by partial (graded) or absolute (all-or-none) neural responses in high-level human brain regions is largely unknown. We address this issue with a novel frequency-sweep paradigm probing the evolution of face categorization responses between the minimal and optimal stimulus presentation times. In a first experiment, natural images of variable non-face objects were progressively swept from 120 to 3 ​Hz (8.33-333 ​ms duration) in rapid serial visual presentation sequences. Widely variable face exemplars appeared every 1 ​s, enabling an implicit frequency-tagged face-categorization electroencephalographic (EEG) response at 1 ​Hz. Face-categorization activity emerged with stimulus durations as brief as 17 ​ms (17-83 ​ms across individual participants) but was significant with 33 ​ms durations at the group level. The face categorization response amplitude increased until 83 ​ms stimulus duration (12 ​Hz), implying graded categorization responses. In a second EEG experiment, faces appeared non-periodically throughout such sequences at fixed presentation rates, while participants explicitly categorized faces. A strong correlation between response amplitude and behavioral accuracy across frequency rates suggested that dilution from missed categorizations, rather than a decreased response to each face stimulus, accounted for the graded categorization responses as found in Experiment 1. This was supported by (1) the absence of neural responses to faces that participants failed to categorize explicitly in Experiment 2 and (2) equivalent amplitudes and spatio-temporal signatures of neural responses to behaviorally categorized faces across presentation rates. Overall, these observations provide original evidence that high-level visual categorization of faces, starting at about 100 ​ms following stimulus onset in the human brain, is variable across observers tested under tight temporal constraints, but occurs in an all-or-none fashion.

Stimulus context alters neural representations of faces in inferotemporal cortex

One goal of our nervous system is to form predictions about the world around us to facilitate our responses to upcoming events. One basis for such predictions could be the recently encountered visual stimuli, or the recent statistics of the visual environment. We examined the effect of recently experienced stimulus statistics on the visual representation of face stimuli by recording the responses of face-responsive neurons in the final stage of visual object recognition, the inferotemporal (IT) cortex, during blocks in which the probability of seeing a particular face was either 100% or only 12%. During the block with only face images, ∼30% of IT neurons exhibit enhanced anticipatory activity before the evoked visual response. This anticipatory modulation is followed by greater activity, broader view tuning, more distributed processing, and more reliable responses of IT neurons to the face stimuli. These changes in the visual response were sufficient to improve the ability of IT neurons to represent a variable property of the predictable face images (viewing angle), as measured by the performance of a simple linear classifier. These results demonstrate that the recent statistics of the visual environment can facilitate processing of stimulus information in the population neuronal representation.

NEW & NOTEWORTHY

Neurons in inferotemporal (IT) cortex anticipate the arrival of a predictable stimulus, and visual responses to an expected stimulus are more distributed throughout the population of IT neurons, providing an enhanced representation of second-order stimulus information (in this case, viewing angle). The findings reveal a potential neural basis for the behavioral benefits of contextual expectation.

Temporal dynamics of visual category representation in the macaque inferior temporal cortex

Object categories are recognized at multiple levels of hierarchical abstractions. Psychophysical studies have shown a more rapid perceptual access to the mid-level category information (e.g., human faces) than the higher (superordinate; e.g., animal) or the lower (subordinate; e.g., face identity) level. Mid-level category members share many features, whereas few features are shared among members of different mid-level categories. To understand better the neural basis of expedited access to mid-level category information, we examined neural responses of the inferior temporal (IT) cortex of macaque monkeys viewing a large number of object images. We found an earlier representation of mid-level categories in the IT population and single-unit responses compared with superordinate- and subordinate-level categories. The short-latency representation of mid-level category information shows that visual cortex first divides the category shape space at its sharpest boundaries, defined by high/low within/between-group similarity. This short-latency, mid-level category boundary map may be a prerequisite for representation of other categories at more global and finer scales.

Image familiarization sharpens response dynamics of neurons in inferotemporal cortex

Repeated viewing of an image over days and weeks induces a marked reduction in the strength with which neurons in monkey inferotemporal cortex respond to it. The processing advantage that attaches to this reduction is unknown. One possibility is that truncation of the response to a familiar image leaves neurons in a state of readiness to respond to ensuing images and thereby enhances their ability to track rapidly changing displays. We explored this possibility by assessing neuronal responses to familiar and novel images in rapid serial visual displays. Inferotemporal neurons responded more strongly to familiar than to novel images in such displays. The effect was stronger among putative inhibitory neurons than among putative excitatory neurons. A comparable effect occurred at the level of the scalp potential in humans. We conclude that long-term familiarization sharpens the response dynamics of neurons in both monkey and human extrastriate visual cortex.

Dimensionality of object representations in monkey inferotemporal cortex

We have calculated the intrinsic dimensionality of visual object representations in anterior inferotemporal (AIT) cortex, based on responses of a large sample of cells stimulated with photographs of diverse objects. Because dimensionality was dependent on data set size, we determined asymptotic dimensionality as both the number of neurons and number of stimulus image approached infinity. Our final dimensionality estimate was 93 (SD: ± 11), indicating that there is basis set of approximately 100 independent features that characterize the dimensions of neural object space. We believe this is the first estimate of the dimensionality of neural visual representations based on single-cell neurophysiological data. The dimensionality of AIT object representations was much lower than the dimensionality of the stimuli. We suggest that there may be a gradual reduction in the dimensionality of object representations in neural populations going from retina to inferotemporal cortex as receptive fields become increasingly complex.

Categorical, yet graded--single-image activation profiles of human category-selective cortical regions

Human inferior temporal cortex contains category-selective visual regions, including the fusiform face area (FFA) and the parahippocampal place area (PPA). These regions are defined by their greater category-average activation to the preferred category (faces and places, respectively) relative to nonpreferred categories. The approach of investigating category-average activation has left unclear to what extent category selectivity holds for individual object images. Here we investigate single-image activation profiles to address (1) whether each image from the preferred category elicits greater activation than any image outside the preferred category (categorical ranking), (2) whether there are activation differences within and outside the preferred category (gradedness), and (3) whether the activation profile falls off continuously across the category boundary or exhibits a discontinuity at the boundary (category step). We used functional magnetic resonance imaging to measure the activation elicited in the FFA and PPA by each of 96 object images from a wide range of categories, including faces and places, but also humans and animals, and natural and manmade objects. Results suggest that responses in FFA and PPA exhibit almost perfect categorical ranking, are graded within and outside the preferred category, and exhibit a category step. The gradedness within the preferred category was more pronounced in FFA; the category step was more pronounced in PPA. These findings support the idea that these regions have category-specific functions, but are also consistent with a distributed object representation emphasizing categories while still distinguishing individual images.

Face selectivity properties of monkey temporal cortical neurons predicted by computer vision

In this issue of Neuron, Ohayon et al. (2012) utilize fMRI-guided single-cell recordings to demonstrate the importance of contrast polarity features for face-selective responses in macaque temporal cortex, as predicted by a computer vision face detection algorithm.

Visual cortex combines a stimulus and an error-like signal with a proportion that is dependent on time, space, and stimulus contrast

Even though the visual cortex is one of the most studied brain areas, the neuronal code in this area is still not fully understood. In the literature, two codes are commonly hypothesized, namely stimulus and predictive (error) codes. Here, we examined whether and how these two codes can coexist in a neuron. To this end, we assumed that neurons could predict a constant stimulus across time or space, since this is the most fundamental type of prediction. Prediction was examined in time using electrophysiology and voltage-sensitive dye imaging in the supragranular layers in area 18 of the anesthetized cat, and in space using a computer model. The distinction into stimulus and error code was made by means of the orientation tuning of the recorded unit. The stimulus was constructed as such that a maximum response to the non-preferred orientation indicated an error signal, and the maximum response to the preferred orientation indicated a stimulus signal. We demonstrate that a single neuron combines stimulus and error-like coding. In addition, we observed that the duration of the error coding varies as a function of stimulus contrast. For low contrast the error-like coding was prolonged by around 60-100%. Finally, the combination of stimulus and error leads to a suboptimal free energy in a recent predictive coding model. We therefore suggest a straightforward modification that can be applied to the free energy model and other predictive coding models. Combining stimulus and error might be advantageous because the stimulus code enables a direct stimulus recognition that is free of assumptions whereas the error code enables an experience dependent inference of ambiguous and non-salient stimuli.

Relationship between functional magnetic resonance imaging-identified regions and neuronal category selectivity

Functional magnetic resonance imaging (fMRI) has been used extensively to identify regions in the inferior temporal (IT) cortex that are selective for categories of visual stimuli. However, comparatively little is known about the neuronal responses relative to these fMRI-defined regions. Here, we compared in nonhuman primates the distribution and response properties of IT neurons recorded within versus outside fMRI regions selective for four different visual categories: faces, body parts, objects, and places. Although individual neurons that preferred each of the four categories were found throughout the sampled regions, they were most concentrated within the corresponding fMRI region, decreasing significantly within 1-4 mm from the edge of these regions. Furthermore, the correspondence between fMRI and neuronal distributions was specific to neurons that increased their firing rates in response to the visual stimuli but not to neurons suppressed by visual stimuli, suggesting that the processes associated with inhibiting neuronal activity did not contribute strongly to the fMRI signal in this experiment.

From single cells to social perception

Research describing the cellular coding of faces in non-human primates often provides the underlying physiological framework for our understanding of face processing in humans. Models of face perception, explanations of perceptual after-effects from viewing particular types of faces, and interpretation of human neuroimaging data rely on monkey neurophysiological data and the assumption that neurophysiological responses of humans are comparable to those recorded in the non-human primate. Here, we review studies that describe cells that preferentially respond to faces, and assess the link between the physiological characteristics of single cells and social perception. Principally, we describe cells recorded from the non-human primate, although a limited number of cells have been recorded in humans, and are included in order to appraise the validity of non-human physiological data for our understanding of human face and social perception.

Neural control: closed-loop human brain reading

Closed-loop experimental testing of single medial temporal lobe neurons in humans reveals top-down effects, opening new possibilities for describing neural representations at the highest level.

Statistics of visual responses in primate inferotemporal cortex to object stimuli

We have characterized selectivity and sparseness in anterior inferotemporal cortex, using a large data set. Responses were collected from 674 monkey inferotemporal cells, each stimulated by 806 object photographs. This 806 × 674 matrix was examined in two ways: columnwise, looking at responses of a single neuron to all images (single-neuron selectivity), and rowwise, looking at the responses of all neurons caused by a single image (population sparseness). Selectivity and sparseness were measured as kurtosis of probability distributions. Population sparseness exceeded single-neuron selectivity, with specific values dependent on the size of the data sample. This difference was principally caused by inclusion, within the population, of neurons with a variety of dynamic ranges (standard deviations of responses over all images). Statistics of large responses were examined by quantifying how quickly the upper tail of the probability distribution decreased (tail heaviness). This analysis demonstrated that population responses had heavier tails than single-neuron responses, consistent with the difference between sparseness and selectivity measurements. Population responses with spontaneous activity subtracted had the heaviest tails, following a power law. The very light tails of single-neuron responses indicate that the critical feature for each neuron is simple enough to have a high probability of occurring within a limited stimulus set. Heavy tails of population responses indicate that there are a large number of different critical features to which different neurons are tuned. These results are inconsistent with some structural models of object recognition that posit that objects are decomposed into a small number of standard features.

Internal curvature signal and noise in low- and high-level vision

How does internal processing contribute to visual pattern perception? By modeling visual search performance, we estimated internal signal and noise relevant to perception of curvature, a basic feature important for encoding of three-dimensional surfaces and objects. We used isolated, sparse, crowded, and face contexts to determine how internal curvature signal and noise depended on image crowding, lateral feature interactions, and level of pattern processing. Observers reported the curvature of a briefly flashed segment, which was presented alone (without lateral interaction) or among multiple straight segments (with lateral interaction). Each segment was presented with no context (engaging low-to-intermediate-level curvature processing), embedded within a face context as the mouth (engaging high-level face processing), or embedded within an inverted-scrambled-face context as a control for crowding. Using a simple, biologically plausible model of curvature perception, we estimated internal curvature signal and noise as the mean and standard deviation, respectively, of the Gaussian-distributed population activity of local curvature-tuned channels that best simulated behavioral curvature responses. Internal noise was increased by crowding but not by face context (irrespective of lateral interactions), suggesting prevention of noise accumulation in high-level pattern processing. In contrast, internal curvature signal was unaffected by crowding but modulated by lateral interactions. Lateral interactions (with straight segments) increased curvature signal when no contextual elements were added, but equivalent interactions reduced curvature signal when each segment was presented within a face. These opposing effects of lateral interactions are consistent with the phenomena of local-feature contrast in low-level processing and global-feature averaging in high-level processing.

Visual stimulation decorrelates neuronal activity

The accuracy of neuronal encoding depends on the response statistics of individual neurons and the correlation of the activity between different neurons. Here, the dynamics of the neuronal response statistics in the anterior superior temporal sulcus of the macaque monkey is described. A transient reduction in the normalized trial-by-trial variability and decorrelation of the responses with both the activity of other neurons and previous activity of the same neuron are found at response onset. The variability of neuronal activity and its correlation structure return to the levels observed in the resting state 50-100 ms after response onset, except for marked increases in the signal correlation between neurons. The transient changes in the response statistics are seen even if there is little or no stimulus-elicited activity, indicating the effect is due to network properties rather than to activity changes per se. Modeling also indicates that the observed variations in response variability and correlation structure of the neuronal activity over time cannot be attributed to changes in firing rate. However, a reset of the underlying spike-generating process, possibly due to the driving input changing from recurrent to feedforward inputs, captures most of the observed changes. The nonstationarity indicated by the changes in correlation structure around response onset increases coding efficiency: compared with the mutual information calculated without regard to the transitory changes, the decorrelation increases the information conveyed by the initial response of modeled neuronal pairs by ≤ 4% and suggests that an integration time of as little as 50 ms is sufficient to extract 95% the available information during the initial response period.

Modelling spike trains and extracting response latency with Bayesian binning

The peristimulus time histogram (PSTH) and the spike density function (SDF) are commonly used in the analysis of neurophysiological data. The PSTH is usually obtained by binning spike trains, the SDF being a (Gaussian) kernel smoothed version of the PSTH. While selection of the bin width or kernel size is often relatively arbitrary there have been recent attempts to remedy this situation (Shimazaki and Shinomoto, 2007c,b,a). We further develop an exact Bayesian generative model approach to estimating PSTHs (Endres et al., 2008) and demonstrate its superiority to competing methods using data from early (LGN) and late (STSa) visual areas. We also highlight the advantages of our scheme's automatic complexity control and generation of error bars. Additionally, our approach allows extraction of excitatory and inhibitory response latency from spike trains in a principled way, both on repeated and single trial data. We show that the method can be applied to data with high background firing rates and inhibitory responses (LGN) as well as to data with low firing rate and excitatory responses (STSa). Furthermore, we demonstrate on simulated data that our latency extraction method works for a range of signal-to-noise ratios and background firing rates. While further studies are needed to examine the sensitivity of our method to, for example, gradual changes in firing rate and adaptation, the current results suggest that Bayesian binning is a powerful method for the estimation of firing rate and the extraction response latency from neuronal spike trains.

Seeing the future: Natural image sequences produce “anticipatory” neuronal activity and bias perceptual report

This paper relates human perception to the functioning of cells in the temporal cortex that are engaged in high-level pattern processing. We review historical developments concerning (a) the functional organization of cells processing faces and (b) the selectivity for faces in cell responses. We then focus on (c) the comparison of perception and cell responses to images of faces presented in sequences of unrelated images. Specifically the paper concerns the cell function and perception in circumstances where meaningful patterns occur momentarily in the context of a naturally or unnaturally changing visual environment. Experience of visual sequences allows anticipation, yet one sensory stimulus also “masks” perception and neural processing of subsequent stimuli. To understand this paradox we compared cell responses in monkey temporal cortex to body images presented individually, in pairs and in action sequences. Responses to one image suppressed responses to similar images for ∼500 ms. This suppression led to responses peaking 100 ms earlier to image sequences than to isolated images (e.g., during head rotation, face-selective activity peaks before the face confronts the observer). Thus forward masking has unrecognized benefits for perception because it can transform neuronal activity to make it predictive during natural change.

Visual Adaptation to Goal-directed Hand Actions

Prolonged exposure to visual stimuli, or adaptation, often results in an adaptation “aftereffect” which can profoundly distort our perception of subsequent visual stimuli. This technique has been commonly used to investigate mechanisms underlying our perception of simple visual stimuli, and more recently, of static faces. We tested whether humans would adapt to movies of hands grasping and placing different weight objects. After adapting to hands grasping light or heavy objects, subsequently perceived objects appeared relatively heavier, or lighter, respectively. The aftereffects increased logarithmically with adaptation action repetition and decayed logarithmically with time. Adaptation aftereffects also indicated that perception of actions relies predominantly on view-dependent mechanisms. Adapting to one action significantly influenced the perception of the opposite action. These aftereffects can only be explained by adaptation of mechanisms that take into account the presence/absence of the object in the hand. We tested if evidence on action processing mechanisms obtained using visual adaptation techniques confirms underlying neural processing. We recorded monkey superior temporal sulcus (STS) single-cell responses to hand actions. Cells sensitive to grasping or placing typically responded well to the opposite action; cells also responded during different phases of the actions. Cell responses were sensitive to the view of the action and were dependent upon the presence of the object in the scene. We show here that action processing mechanisms established using visual adaptation parallel the neural mechanisms revealed during recording from monkey STS. Visual adaptation techniques can thus be usefully employed to investigate brain mechanisms underlying action perception.

Mechanisms of face perception

Faces are among the most informative stimuli we ever perceive: Even a split-second glimpse of a person's face tells us his identity, sex, mood, age, race, and direction of attention. The specialness of face processing is acknowledged in the artificial vision community, where contests for face-recognition algorithms abound. Neurological evidence strongly implicates a dedicated machinery for face processing in the human brain to explain the double dissociability of face- and object-recognition deficits. Furthermore, recent evidence shows that macaques too have specialized neural machinery for processing faces. Here we propose a unifying hypothesis, deduced from computational, neurological, fMRI, and single-unit experiments: that what makes face processing special is that it is gated by an obligatory detection process. We clarify this idea in concrete algorithmic terms and show how it can explain a variety of phenomena associated with face processing.

Turning the other cheek: the viewpoint dependence of facial expression after-effects

How do we visually encode facial expressions? Is this done by viewpoint-dependent mechanisms representing facial expressions as two-dimensional templates or do we build more complex viewpoint independent three-dimensional representations? Recent facial adaptation techniques offer a powerful way to address these questions. Prolonged viewing of a stimulus (adaptation) changes the perception of subsequently viewed stimuli (an after-effect). Adaptation to a particular attribute is believed to target those neural mechanisms encoding that attribute. We gathered images of facial expressions taken simultaneously from five different viewpoints evenly spread from the three-quarter leftward to the three-quarter rightward facing view. We measured the strength of expression after-effects as a function of the difference between adaptation and test viewpoints. Our data show that, although there is a decrease in after-effect over test viewpoint, there remains a substantial after-effect when adapt and test are at differing three-quarter views. We take these results to indicate that neural systems encoding facial expressions contain a mixture of viewpoint-dependent and viewpoint-independent elements. This accords with evidence from single cell recording studies in macaque and is consonant with a view in which viewpoint-independent expression encoding arises from a combination of view-dependent expression-sensitive responses.

Sex-contingent face aftereffects depend on perceptual category rather than structural encoding

Many studies have used visual adaptation to investigate how recent experience with faces influences perception. While faces similar to those seen during adaptation phases are typically perceived as more 'normal' after adaptation, it is possible to induce aftereffects in one direction for one category (e.g. female) and simultaneously induce aftereffects in the opposite direction for another category (e.g. male). Such aftereffects could reflect 'category-contingent' adaptation of neurons selective for perceptual category (e.g. male or female) or 'structure-contingent' adaptation of lower-level neurons coding the physical characteristics of different face patterns. We compared these explanations by testing for simultaneous opposite after effects following adaptation to (a) two groups of faces from distinct sex categories (male and female) or (b) two groups of faces from the same sex category (female and hyper-female) where the structural differences between the female and hyper-female groups were mathematically identical to those between male and female groups. We were able to induce opposite aftereffects following adaptation between sex categories but not after adaptation within a sex category. These findings indicate the involvement of neurons coding perceptual category in sex-contingent face aftereffects and cannot be explained by neurons coding only the physical aspects of face patterns.

From response to stimulus: adaptive sampling in sensory physiology

Sensory systems extract behaviorally relevant information from a continuous stream of complex high-dimensional input signals. Understanding the detailed dynamics and precise neural code, even of a single neuron, is therefore a non-trivial task. Automated closed-loop approaches that integrate data analysis in the experimental design ease the investigation of sensory systems in three directions: First, adaptive sampling speeds up the data acquisition and thus increases the yield of an experiment. Second, model-driven stimulus exploration improves the quality of experimental data needed to discriminate between alternative hypotheses. Third, information-theoretic data analyses open up novel ways to search for those stimuli that are most efficient in driving a given neuron in terms of its firing rate or coding quality. Examples from different sensory systems show that, in all three directions, substantial progress can be achieved once rapid online data analysis, adaptive sampling, and computational modeling are tightly integrated into experiments.

Exact Bayesian bin classification: a fast alternative to Bayesian classification and its application to neural response analysis

We investigate the general problem of signal classification and, in particular, that of assigning stimulus labels to neural spike trains recorded from single cortical neurons. Finding efficient ways of classifying neural responses is especially important in experiments involving rapid presentation of stimuli. We introduce a fast, exact alternative to Bayesian classification. Instead of estimating the class-conditional densities p(x|y) (where x is a scalar function of the feature[s], y the class label) and converting them to P(y|x) via Bayes' theorem, this probability is evaluated directly and without the need for approximations. This is achieved by integrating over all possible binnings of x with an upper limit on the number of bins. Computational time is quadratic in both the number of observed data points and the number of bins. The algorithm also allows for the computation of feedback signals, which can be used as input to subsequent stages of inference, e.g. neural network training. Responses of single neurons from high-level visual cortex (area STSa) to rapid sequences of complex visual stimuli are analysed. Information latency and response duration increase nonlinearly with presentation duration, suggesting that neural processing speeds adapt to presentation speeds.

Object category structure in response patterns of neuronal population in monkey inferior temporal cortex

Our mental representation of object categories is hierarchically organized, and our rapid and seemingly effortless categorization ability is crucial for our daily behavior. Here, we examine responses of a large number (>600) of neurons in monkey inferior temporal (IT) cortex with a large number (>1,000) of natural and artificial object images. During the recordings, the monkeys performed a passive fixation task. We found that the categorical structure of objects is represented by the pattern of activity distributed over the cell population. Animate and inanimate objects created distinguishable clusters in the population code. The global category of animate objects was divided into bodies, hands, and faces. Faces were divided into primate and nonprimate faces, and the primate-face group was divided into human and monkey faces. Bodies of human, birds, and four-limb animals clustered together, whereas lower animals such as fish, reptile, and insects made another cluster. Thus the cluster analysis showed that IT population responses reconstruct a large part of our intuitive category structure, including the global division into animate and inanimate objects, and further hierarchical subdivisions of animate objects. The representation of categories was distributed in several respects, e.g., the similarity of response patterns to stimuli within a category was maintained by both the cells that maximally responded to the category and the cells that responded weakly to the category. These results advance our understanding of the nature of the IT neural code, suggesting an inherently categorical representation that comprises a range of categories including the amply investigated face category.

Properties of shape tuning of macaque inferior temporal neurons examined using rapid serial visual presentation

We used rapid serial visual presentation (RSVP) to examine the tuning of macaque inferior temporal cortical (IT) neurons to five sets of 25 shapes each that varied systematically along predefined shape dimensions. A comparison of the RSVP technique using 100-ms presentations with that using a longer duration showed that shape preference can be determined with RSVP. Using relatively complex shapes that vary along relatively simple shape dimensions, we found that the large majority of neurons preferred extremes of the shape configuration, extending the results of a previous study using simpler shapes and a standard testing paradigm. A population analysis of the neuronal responses demonstrated that, in general, IT neurons can represent the similarities among the shapes at an ordinal level, extending a previous study that used a smaller number of shapes and a categorization task. However, the same analysis showed that IT neurons do not faithfully represent the physical similarities among the shapes. The responses to the two-part shapes could be predicted, virtually perfectly, from the average of the responses to the respective two parts presented in isolation. We also showed that IT neurons adapt to the stimulus distribution statistics. The neural shape discrimination improved when a shape set with a narrower stimulus range was presented, suggesting that the tuning of IT neurons is not static but adapts to the stimulus distribution statistics, at least when stimulated at a high rate with a restricted set of stimuli.

Invariance and selectivity in the ventral visual pathway

Pattern recognition systems that are invariant to shape, pose, lighting and texture are never sufficiently selective; they suffer a high rate of "false alarms". How are biological vision systems both invariant and selective? Specifically, how are proper arrangements of sub-patterns distinguished from the chance arrangements that defeat selectivity in artificial systems? The answer may lie in the nonlinear dynamics that characterize complex and other invariant cell types: these cells are temporarily more receptive to some inputs than to others (functional connectivity). One consequence is that pairs of such cells with overlapping receptive fields will possess a related property that might be termed functional common input. Functional common input would induce high correlation exactly when there is a match in the sub-patterns appearing in the overlapping receptive fields. These correlations, possibly expressed as a partial and highly local synchrony, would preserve the selectivity otherwise lost to invariance.

The fusiform face area: a cortical region specialized for the perception of faces

Faces are among the most important visual stimuli we perceive, informing us not only about a person's identity, but also about their mood, sex, age and direction of gaze. The ability to extract this information within a fraction of a second of viewing a face is important for normal social interactions and has probably played a critical role in the survival of our primate ancestors. Considerable evidence from behavioural, neuropsychological and neurophysiological investigations supports the hypothesis that humans have specialized cognitive and neural mechanisms dedicated to the perception of faces (the face-specificity hypothesis). Here, we review the literature on a region of the human brain that appears to play a key role in face perception, known as the fusiform face area (FFA). Section 1 outlines the theoretical background for much of this work. The face-specificity hypothesis falls squarely on one side of a longstanding debate in the fields of cognitive science and cognitive neuroscience concerning the extent to which the mind/brain is composed of: (i) special-purpose ('domain-specific') mechanisms, each dedicated to processing a specific kind of information (e.g. faces, according to the face-specificity hypothesis), versus (ii) general-purpose ('domain-general') mechanisms, each capable of operating on any kind of information. Face perception has long served both as one of the prime candidates of a domain-specific process and as a key target for attack by proponents of domain-general theories of brain and mind. Section 2 briefly reviews the prior literature on face perception from behaviour and neurophysiology. This work supports the face-specificity hypothesis and argues against its domain-general alternatives (the individuation hypothesis, the expertise hypothesis and others). Section 3 outlines the more recent evidence on this debate from brain imaging, focusing particularly on the FFA. We review the evidence that the FFA is selectively engaged in face perception, by addressing (and rebutting) five of the most widely discussed alternatives to this hypothesis. In section 4, we consider recent findings that are beginning to provide clues into the computations conducted in the FFA and the nature of the representations the FFA extracts from faces. We argue that the FFA is engaged both in detecting faces and in extracting the necessary perceptual information to recognize them, and that the properties of the FFA mirror previously identified behavioural signatures of face-specific processing (e.g. the face-inversion effect). Section 5 asks how the computations and representations in the FFA differ from those occurring in other nearby regions of cortex that respond strongly to faces and objects. The evidence indicates clear functional dissociations between these regions, demonstrating that the FFA shows not only functional specificity but also area specificity. We end by speculating in section 6 on some of the broader questions raised by current research on the FFA, including the developmental origins of this region and the question of whether faces are unique versus whether similarly specialized mechanisms also exist for other domains of high-level perception and cognition.

Coding of visual objects in the ventral stream

How are objects represented in the brain? Two facets of this question are currently under investigation. First, are objects represented by activity in a relatively small number of neurons that are each selective for the shape or identity of a specific object (a 'sparse code'), or are they represented by a pattern of activity across a large number of less selective neurons (a 'population code')? Second, how are the neurons that code for an object distributed across the cortex: are they clustered together in patches, or are they scattered widely across the cortex? The results from neurophysiology and functional magnetic resonance imaging studies are beginning to provide preliminary answers to both questions.

Microstimulation of inferotemporal cortex influences face categorization

The inferior temporal cortex (IT) of primates is thought to be the final visual area in the ventral stream of cortical areas responsible for object recognition. Consistent with this hypothesis, single IT neurons respond selectively to highly complex visual stimuli such as faces. However, a direct causal link between the activity of face-selective neurons and face perception has not been demonstrated. In the present study of macaque monkeys, we artificially activated small clusters of IT neurons by means of electrical microstimulation while the monkeys performed a categorization task, judging whether noisy visual images belonged to 'face' or 'non-face' categories. Here we show that microstimulation of face-selective sites, but not other sites, strongly biased the monkeys' decisions towards the face category. The magnitude of the effect depended upon the degree of face selectivity of the stimulation site, the size of the stimulated cluster of face-selective neurons, and the exact timing of microstimulation. Our results establish a causal relationship between the activity of face-selective neurons and face perception.

Snakes as agents of evolutionary change in primate brains

Current hypotheses that use visually guided reaching and grasping to explain orbital convergence, visual specialization, and brain expansion in primates are open to question now that neurological evidence reveals no correlation between orbital convergence and the visual pathway in the brain that is associated with reaching and grasping. An alternative hypothesis proposed here posits that snakes were ultimately responsible for these defining primate characteristics. Snakes have a long, shared evolutionary existence with crown-group placental mammals and were likely to have been their first predators. Mammals are conservative in the structures of the brain that are involved in vigilance, fear, and learning and memory associated with fearful stimuli, e.g., predators. Some of these areas have expanded in primates and are more strongly connected to visual systems. However, primates vary in the extent of brain expansion. This variation is coincident with variation in evolutionary co-existence with the more recently evolved venomous snakes. Malagasy prosimians have never co-existed with venomous snakes, New World monkeys (platyrrhines) have had interrupted co-existence with venomous snakes, and Old World monkeys and apes (catarrhines) have had continuous co-existence with venomous snakes. The koniocellular visual pathway, arising from the retina and connecting to the lateral geniculate nucleus, the superior colliculus, and the pulvinar, has expanded along with the parvocellular pathway, a visual pathway that is involved with color and object recognition. I suggest that expansion of these pathways co-occurred, with the koniocellular pathway being crucially involved (among other tasks) in pre-attentional visual detection of fearful stimuli, including snakes, and the parvocellular pathway being involved (among other tasks) in protecting the brain from increasingly greater metabolic demands to evolve the neural capacity to detect such stimuli quickly. A diet that included fruits or nectar (though not to the exclusion of arthropods), which provided sugars as a neuroprotectant, may have been a required preadaptation for the expansion of such metabolically active brains. Taxonomic differences in evolutionary exposure to venomous snakes are associated with similar taxonomic differences in rates of evolution in cytochrome oxidase genes and in the metabolic activity of cytochrome oxidase proteins in at least some visual areas in the brains of primates. Raptors that specialize in eating snakes have larger eyes and greater binocularity than more generalized raptors, and provide non-mammalian models for snakes as a selective pressure on primate visual systems. These models, along with evidence from paleobiogeography, neuroscience, ecology, behavior, and immunology, suggest that the evolutionary arms race begun by constrictors early in mammalian evolution continued with venomous snakes. Whereas other mammals responded by evolving physiological resistance to snake venoms, anthropoids responded by enhancing their ability to detect snakes visually before the strike.

Adaptation: from single cells to BOLD signals

Functional magnetic resonance imaging adaptation (fMRIa) is an increasingly popular method that aims to provide insight into the functional properties of subpopulations of neurons within an imaging voxel. The technique relies on the assumption that neural adaptation reduces activity when two successive stimuli activate the same subpopulation but not when they stimulate different subpopulations. Here, we assess the validity of fMRIa by comparing single-cell recordings with functional imaging of orientation, motion and face processing. We find that fMRIa provides novel insight into neural representations in the human brain. However, network responses in general and adaptation in particular are more complex than is often assumed, and an unequivocal interpretation of fMRIa results can be achieved only with great care.

The sensitivity of primate STS neurons to walking sequences and to the degree of articulation in static images

We readily use the form of human figures to determine if they are moving. Human figures that have arms and legs outstretched (articulated) appear to be moving more than figures where the arms and legs are near the body (standing). We tested whether neurons in the macaque monkey superior temporal sulcus (STS), a region known to be involved in processing social stimuli, were sensitive to the degree of articulation of a static human figure. Additionally, we tested sensitivity to the same stimuli within forward and backward walking sequences. We found that 57% of cells that responded to the static image of a human figure was also sensitive to the degree of articulation of the figure. Some cells displayed selective responses for articulated postures, while others (in equal numbers) displayed selective responses for standing postures. Cells selective for static images of articulated figures were more likely to respond to movies of walking forwards than walking backwards. Cells selective for static images of standing figures were more likely to respond to movies of walking backwards than forwards. An association between form sensitivity and walking sensitivity could be consistent with an interpretation that cell responses to articulated figures act as an implied motion signal.

Domain Specificity in Visual Cortex

We investigated the prevalence and specificity of category-selective regions in human visual cortex. In the broadest survey to date of category selectivity in visual cortex, 12 participants were scanned with functional magnetic resonance imaging while viewing scenes and 19 different object categories in a blocked-design experiment. As expected, we found selectivity for faces in the fusiform face area (FFA), for scenes in the parahippocampal place area (PPA), and for bodies in the extrastriate body area (EBA). In addition, we describe 3 main new findings. First, evidence for the selectivity of the FFA, PPA, and EBA was strengthened by the finding that each area responded significantly more strongly to its preferred category than to the next most effective of the remaining 19 stimulus categories tested. Second, a region in the middle temporal gyrus that has been reported to respond significantly more strongly to tools than to animals did not respond significantly more strongly to tools than to other nontool categories (such as fruits and vegetables), casting doubt on the characterization of this region as tool selective. Finally, we did not find any new regions in the occipitotemporal pathway that were strongly selective for other categories. Taken together, these results demonstrate both the strong selectivity of a small number of regions and the scarcity of such regions in visual cortex.

Do we know what the early visual system does?

We can claim that we know what the visual system does once we can predict neural responses to arbitrary stimuli, including those seen in nature. In the early visual system, models based on one or more linear receptive fields hold promise to achieve this goal as long as the models include nonlinear mechanisms that control responsiveness, based on stimulus context and history, and take into account the nonlinearity of spike generation. These linear and nonlinear mechanisms might be the only essential determinants of the response, or alternatively, there may be additional fundamental determinants yet to be identified. Research is progressing with the goals of defining a single "standard model" for each stage of the visual pathway and testing the predictive power of these models on the responses to movies of natural scenes. These predictive models represent, at a given stage of the visual pathway, a compact description of visual computation. They would be an invaluable guide for understanding the underlying biophysical and anatomical mechanisms and relating neural responses to visual perception.

How close are we to understanding v1?

A wide variety of papers have reviewed what is known about the function of primary visual cortex. In this review, rather than stating what is known, we attempt to estimate how much is still unknown about V1 function. In particular, we identify five problems with the current view of V1 that stem largely from experimental and theoretical biases, in addition to the contributions of nonlinearities in the cortex that are not well understood. Our purpose is to open the door to new theories, a number of which we describe, along with some proposals for testing them.

Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions

Processing of complex visual stimuli comprising facial movements, hand actions, and body movements is known to occur in the superior temporal sulcus (STS) of humans and nonhuman primates. The STS is also thought to play a role in the integration of multimodal sensory input. We investigated whether STS neurons coding the sight of actions also integrated the sound of those actions. For 23% of neurons responsive to the sight of an action, the sound of that action significantly modulated the visual response. The sound of the action increased or decreased the visually evoked response for an equal number of neurons. In the neurons whose visual response was increased by the addition of sound (but not those neurons whose responses were decreased), the audiovisual integration was dependent upon the sound of the action matching the sight of the action. These results suggest that neurons in the STS form multisensory representations of observed actions.