The distributed human neural system for face perception

Intersubject synchronization of cortical activity during natural vision

To what extent do all brains work alike during natural conditions? We explored this question by letting five subjects freely view half an hour of a popular movie while undergoing functional brain imaging. Applying an unbiased analysis in which spatiotemporal activity patterns in one brain were used to "model" activity in another brain, we found a striking level of voxel-by-voxel synchronization between individuals, not only in primary and secondary visual and auditory areas but also in association cortices. The results reveal a surprising tendency of individual brains to "tick collectively" during natural vision. The intersubject synchronization consisted of a widespread cortical activation pattern correlated with emotionally arousing scenes and regionally selective components. The characteristics of these activations were revealed with the use of an open-ended "reverse-correlation" approach, which inverts the conventional analysis by letting the brain signals themselves "pick up" the optimal stimuli for each specialized cortical area.

Adaptation to speaker's voice in right anterior temporal lobe

Little is known on how voices are represented in the brain. We used fMRI to investigate whether parts of auditory cortex would be sensitive to the repetition of a speaker's voice. Subjects were scanned while passively listening to spoken syllables, presented in blocs in which either syllable or speaker were repeated. Only one cortical region, located in the anterior part of the right superior temporal sulcus (STS), responded differently to the two conditions: activation relative to the silent baseline was significantly reduced when syllables were spoken by a single voice than when they were spoken by different voices. This result suggest that the right anterior STS plays an important role in the representation of individual voices.

No troubles with bubbles: a reply to Murray and Gold

Murray and Gold discuss two "shortcomings" of the Bubbles method [Vision Research 41 (2001) 2261]. The first one is theoretical: Bubbles would not fully characterize the LAM (Linear Amplifier Model) observer, whereas reverse correlation would. The second "shortcoming" is practical: the apertures that partly reveal information in a typical Bubbles experiment would induce atypical strategies in human observers, whereas the additive Gaussian white noise used by Murray and Gold (and others) in conjunction with reverse correlation would not. Here, we show that these claims are unfounded.

Eye Movements during Task Switching: Reflexive, Symbolic, and Affective Contributions to Response Selection

Active vision is a dynamic process involving the flexible coordination of different gaze strategies to achieve behavioral goals. Although many complex behaviors rely on an ability to efficiently switch between gaze-control strategies, few studies to date have examined mechanisms of task level oculomotor control in detail. Here, we report five experiments in which subjects alternated between conflicting stimulus-saccade mappings within a block of trials. The first experiment showed that there is no performance cost associated with switching between pro and anti saccades. However, follow-up experiments demonstrate that whenever subjects alternate between arbitrary stimulus-saccade mappings, latency costs are apparent on the first trial after a task change. More detailed analysis of switch costs showed that latencies were particularly elevated for saccades directed toward the same location that had been the target for a saccade on the preceeding trial. This saccade “inhibition of return” effect was most marked when unexpected error feedbacks cued task switches, suggesting that saccade selection processes are modulated by reward. We conclude that there are two systems for saccade control that differ in their characteristics following a task switch. The “reflexive” control system can be enabled/disabled in advance of saccade execution without incurring any performance cost. Switch costs are only observed when two or more arbitrary stimulus-saccade mappings have to be coordinated by a “symbolic” control system.

Watching social interactions produces dorsomedial prefrontal and medial parietal BOLD fMRI signal increases compared to a resting baseline

Some human brain areas are tonically active in a resting state when subjects are not engaged in any overt task. The activity of these areas decreases when subjects are engaged in a wide variety of laboratory tasks designed to study cognitive operations. It has been suggested that these areas, among them the medial parietal (precyneus) and the dorsomedial prefrontal cortices, may support a "default state" of the human brain. Passive visual observation of laboratory stimuli typically yields no change in activity in these default areas compared to rest. Here we report functional magnetic resonance imaging (fMRI) data on normal subjects watching realistic movie clips depicting everyday social interactions. In contrast with previous findings on default state brain areas, the observation of the relational segment of the movie clip, during which two persons interact, yielded increased activity in the medial parietal (precuneus) and dorsomedial prefrontal cortices, compared to rest and to observation of the segment of the movie clip depicting a single individual engaged in everyday activities. To the best of our knowledge, this is the first report of joint increased activity in medial parietal and dorsomedial prefrontal cortices. We suggest that the default state areas may participate in the processing of social relations in concert with regions previously identified as critical for social cognition that were also activated by our stimuli, including the inferior frontal cortex, the superior temporal cortex, and the fusiform gyrus.

Understanding Other Minds: Linking Developmental Psychology and Functional Neuroimaging

Evidence from developmental psychology suggests that understanding other minds constitutes a special domain of cognition with at least two components: an early-developing system for reasoning about goals, perceptions, and emotions, and a later-developing system for representing the contents of beliefs. Neuroimaging reinforces and elaborates upon this view by providing evidence that (a) domain-specific brain regions exist for representing belief contents, (b) these regions are apparently distinct from other regions engaged in reasoning about goals and actions (suggesting that the two developmental stages reflect the emergence of two distinct systems, rather than the elaboration of a single system), and (c) these regions are distinct from brain regions engaged in inhibitory control and in syntactic processing. The clear neural distinction between these processes is evidence that belief attribution is not dependent on either inhibitory control or syntax, but is subserved by a specialized neural system for theory of mind.

Rapid completion effects in human high-order visual areas

Object completion is an inherent property of visual recognition in which objects can be accurately perceived in the presence of substantial obstructions. We have previously shown [Cereb. Cortex 12 (2002) 163] that high-order human object areas are driven partially by local object fragments and partially by global completion effects. Here we explored, through a backward masking paradigm, whether the balance of local and global processing is time dependent, that is, to what extent completion effects evolve at a different time compared to local image representations. In two separate experiments, subjects were presented with three types of images: (a) unobstructed line drawings of animal shapes ("whole"), (b) the same shapes obstructed by a set of parallel stripes ("grid"), and (c) a scrambled version of b in which the stripe position was shifted horizontally, disrupting the relative position of image regions but maintaining the local feature distribution ("scrambled"). Images were presented either for 60 or 250 ms followed by a mask. Both behavioral and fMRI findings from high-order occipitotemporal object areas showed consistently that object selectivity emerges at the same time as the local feature representation. Thus, object completion effects were evident at the same relative magnitude (LO: 0.5 +/- 0.3 and 0.58 +/- 0.04; pFs: 0.62 +/- 0.3 and 0.6 +/- 0.04; 60 and 250 ms, respectively) even at the short presentation durations when overall object activation was greatly reduced.

Neural substrates participating in acquisition of facial familiarity: an fMRI study

The amygdala is related to recognition of faces and emotions, and functional magnetic resonance imaging (fMRI) studies have reported that the amygdala is habituated over time with repetition of facial stimuli. When subjects are presented repeatedly with unfamiliar faces, they come to gradually recognize the unfamiliar faces as familiar. To investigate the brain areas participating in the acquisition of familiarity to repeatedly presented unfamiliar faces, we conducted an fMRI study in 16 healthy subjects. During the task periods, the subjects were instructed to see presented unfamiliar faces repeatedly and to judge whether the face was male or female or whether the face had emotional valences. The experiment consisted of nine sessions. To clarify the brain areas that showed increasing or decreasing activation as the experimental session proceeded, we analyzed the fMRI data using specified linear covariates in the face recognition task from the first session to the ninth session. Imaging data were investigated on a voxel-by-voxel basis for single-group analysis according to the random effect model using Statistical Parametric Mapping. The bilateral posterior cingulate cortices showed significant increases in activity as the experimental sessions proceeded, while the activation in the right amygdala and the left medial fusiform gyrus decreased. Thus, the posterior cingulate cortex may play an important role in the acquisition of facial familiarity.

Neuroimaging studies of attention and the processing of emotion-laden stimuli

Because the processing capacity of the visual system is limited, selective attention to one part of the visual field comes at the cost of neglecting other parts. In this paper, we review evidence from single-cell studies in monkeys and functional magnetic resonance imaging (fMRI) studies in humans for neural competition and how competition is biased by attention. We suggest that, at the neural level, an important consequence of attention is to enhance the influence of behaviorally relevant stimuli at the expense of irrelevant ones, providing a mechanism for the filtering of distracting information in cluttered visual scenes. Psychophysical evidence suggests that processing outside the focus of attention is attenuated and may be even eliminated under some conditions. A major exception to the critical role of attention may be in the neural processing of emotion-laden stimuli, which are reported to be processed automatically, namely, without attention. Contrary to this prevailing view, in a recent study we found that all brain regions responding differentially to faces with emotional content, including the amygdala, did so only when sufficient resources were available to process those faces. After reviewing our findings, we discuss their implications, in particular (1) how emotional stimuli can bias competition for processing resources; (2) the source of the biasing signal for emotional stimuli; (3) how visual information reaches the amygdala; and finally (4) the relationship between attention and awareness.

Effects of Familiarity on the Perceptual Integrality of the Identity and Expression of Faces: The Parallel-Route Hypothesis Revisited

The effects of familiarity on selective attention for the identity and expression of faces were tested using Garner's speeded-classification task. In 2 experiments, participants classified expression (or identity) of familiar and unfamiliar faces while the irrelevant dimension of identity (or expression) was either held constant (baseline condition) or varied randomly (filtering condition). Selective attention was measured by the difference in performance between these 2 conditions. Failure of selective attention was larger for familiar than for unfamiliar faces. In addition, failure of selective attention was found both for identity and for expression judgments. These findings show that familiarity increases (he perceptual integrality between identity and expression, and they question previous studies arguing that identity judgments are always resistant to irrelevant variations in expression. The authors suggest that the systems processing identity and expression are interconnected in that facial identity serves as a reference from which expressions can be more easily derived.

Neuronal Correlates of Face Identification in the Monkey Anterior Temporal Cortical Areas

To investigate the neuronal basis underlying face identification, the activity of face neurons in the anterior superior temporal sulcus (STS) and the anterior inferior temporal gyrus (ITG) of macaque monkeys was analyzed during their performance of a face-identification task. The face space was composed by the activities of face neurons during the face-identification task, based on a multidimensional scaling (MDS) method; the face space composed by the anterior STS neurons represented facial views, whereas that composed by the anterior ITG neurons represented facial identity. The temporal correlation between the behavioral reaction time of the animal and the latency of face-related neuronal responses was also analyzed. The response latency of some of the face neurons in the anterior ITG exhibited a significant correlation with the behavioral reaction time, whereas this correlation was not significant in the anterior STS. The correlation of the latency of face-related neuronal responses in the anterior ITG with the behavioral reaction time was not found to be attributed to the correlation between the response latency and the magnitude of the neuronal responses. The present results suggest that the anterior ITG is closely related to judgments of facial identity, and that the anterior STS is closely related to analyses of incoming perceptual information; face identification in monkeys might involve interactions between the two areas.

The Facilitated Processing of Threatening Faces: An ERP Analysis

Threatening, friendly, and neutral faces were presented to test the hypothesis of the facilitated perceptual processing of threatening faces. Dense sensor event-related brain potentials were measured while subjects viewed facial stimuli. Subjects had no explicit task for emotional categorization of the faces. Assessing early perceptual stimulus processing, threatening faces elicited an early posterior negativity compared with nonthreatening neutral or friendly expressions. Moreover, at later stages of stimulus processing, facial threat also elicited augmented late positive potentials relative to the other facial expressions, indicating the more elaborate perceptual analysis of these stimuli. Taken together, these data demonstrate the facilitated perceptual processing of threatening faces. Results are discussed within the context of an evolved module of fear (A. Ohman & S. Mineka, 2001).

Processing faces and facial expressions

This paper reviews processing of facial identity and expressions. The issue of independence of these two systems for these tasks has been addressed from different approaches over the past 25 years. More recently, neuroimaging techniques have provided researchers with new tools to investigate how facial information is processed in the brain. First, findings from "traditional" approaches to identity and expression processing are summarized. The review then covers findings from neuroimaging studies on face perception, recognition, and encoding. Processing of the basic facial expressions is detailed in light of behavioral and neuroimaging data. Whereas data from experimental and neuropsychological studies support the existence of two systems, the neuroimaging literature yields a less clear picture because it shows considerable overlap in activation patterns in response to the different face-processing tasks. Further, activation patterns in response to facial expressions support the notion of involved neural substrates for processing different facial expressions.

Successful episodic memory retrieval of newly learned faces activates a left fronto-parietal network

It is easier to recognize a familiar face than a newly learned face. The neural basis of familiar face recognition has been elucidated in functional imaging and lesion studies. Behavioural and neuropsychological data indicate, however, that brain systems involved in episodic retrieval of familiar and newly learned faces are distinct. In our study, 12 subjects viewed 30 novel faces in an encoding session. In the study condition, event-related functional magnetic resonance imaging (fMRI) was used to compare brain activation during correct recognition of the recently learned faces to that observed during correct rejection of unknown control faces. Differences were present in the left inferior parietal (BA 40) and left medial frontal/anterior cingulate (BA 32/9) cortex. These two regions may be part of a pathway in the dorsal visual stream, responsible for a "feeling of familiarity" in contrast to the ventral pathway in the temporal lobes, which is mainly involved in the recognition of personal identity.

Area-Specific Amblyopic Effects in Human Occipitotemporal Object Representations

The role of early visual experience in the establishment of human high-order visual areas is poorly understood. Here we investigated this issue using human amblyopia--a developmental visual disorder, which manifests a central vision (acuity) deficit. Previous fMRI studies of amblyopes have described abnormal functional activations in early retinotopic areas. Here we report the surprising finding of a selective object-related abnormality in high-order occipitotemporal cortex. Specifically, we found that face-related cortical areas show a severe disconnection from the amblyopic eye, while building-related regions remain essentially normal. The selectivity of the deficit highlights the differential computations performed in the different object-related areas and is compatible with the suggested association of face regions with analysis of fine detail.

Utilisation of spatial frequency information in face search

In previous studies the utilisation of spatial frequency information in face perception has been investigated by using static recognition tasks. In this study we used a visual search task, which requires eye movements and fast identification of previously learned facial photographs. Using Fourier phase randomisation, spatial information was selectively removed without changing the amplitude spectrum of the image. Fourier phase was randomised within one-octave wide bands of nine different centre spatial frequencies (2-32 c/face width, 0.63-10.1 c/deg). In a control condition no randomisation was used. All stimuli had similar contrast. Search times and eye movements during the search were measured. The removal of spatial information by phase randomisation at medium spatial frequencies resulted in a considerable increase of search times. In the main experiment the maximum of the search times occurred between 8 and 11 c/face width. The number of eye fixations behaved similarly. In an additional experiment with a threefold viewing distance the search times increased and the maximum of the search times shifted slightly to lower object spatial frequencies (5.6-8 c/face width). This suggests that the band of spatial frequencies used in face search is not completely scale invariant. The results show that information most important to face search is located at a limited band of mid spatial frequencies. This is consistent with earlier studies, in which non-dynamical face recognition tasks and low-contrast stimuli have been used.

Face memory and its disorders

Face recognition is an essential biologic and social skill. Accurate recognition depends on the ability to encode, store, and retrieve distinct memory representations for the faces of countless individuals encountered in everyday life. In addition, face memory records must be integrated with specific biographic and name information in order to allow the recognition of each person's unique identity. Converging evidence from functional imaging, cortical electrical recording, and neuropsychologic studies suggests that face memory operations in the human brain are mediated by a distributed neural system. Components of this network include specialized memory storage sites within temporal neocortex that interact with medial temporal lobe and prefrontal cortical areas during face memory encoding and retrieval. Selective damage to these neuroanatomic regions gives rise to face recognition disorders characterized by memory loss or memory distortion.

Early lateralization and orientation tuning for face, word, and object processing in the visual cortex

Event-related potential (ERP) studies of the human brain have shown that object categories can be reliably distinguished as early as 130-170 ms on the surface of occipito-temporal cortex, peaking at the level of the N170 component. Consistent with this finding, neuropsychological and neuroimaging studies suggest major functional distinctions within the human object recognition system, particularly in hemispheric advantage, between the processing of words (left), faces (right), and objects (bilateral). Given these observations, our aim was to (1) characterize the differential response properties of the N170 to pictures of faces, objects, and words across hemispheres; and (2) test whether an effect of inversion for highly familiar and monooriented nonface stimuli such as printed words can be observed at the level of the N170. Scalp EEG (53 channels) was recorded in 15 subjects performing an orientation decision task with pictures of faces, words, and cars presented upright or inverted. All three categories elicited at the same latency a robust N170 component associated with a positive counterpart at centro-frontal sites (vertex-positive potential, VPP). While there were minor amplitude differences at the level of the occipital medial P1 between linguistic and nonlinguistic categories, scalp topographies and source analyses indicated strong hemispheric and orientation effects starting at the level of the N170, which was right lateralized for faces, smaller and bilateral for cars, and as large for printed words in the left hemisphere as for faces. The entire N170/VPP complex was accounted for by two dipolar sources located in the lateral inferior occipital cortex/posterior fusiform gyrus. These two locations were roughly equivalent across conditions but differed in strength and lateralization. Inversion delayed the N170 (and VPP) response for all categories, with an increasing delay for cars, words, and faces, respectively, as suggested by source modeling analysis. Such results show that early processes in object recognition respond to category-specific visual information, and are associated with strong lateralization and orientation bias.

A modulatory role for facial expressions in prosopagnosia

Brain-damaged patients experience difficulties in recognizing a face (prosopagnosics), but they can still recognize its expression. The dissociation between these two face-related skills has served as a keystone of models of face processing. We now report that the presence of a facial expression can influence face identification. For normal viewers, the presence of a facial expression influences performance negatively, whereas for prosopagnosic patients, it improves performance dramatically. Accordingly, although prosopagnosic patients show a failure to process the facial configuration in the interest of face identification, that ability returns when the face shows an emotional expression. Accompanying brain-imaging results indicate activation in brain areas (amygdala, superior temporal sulcus, parietal cortex) outside the occipitotemporal areas normally activated for face identification and lesioned in these patients. This finding suggests a modulatory role of these areas in face identification that is independent of occipitotemporal face areas.

Perceptual dominance of oriented faces mirrors the distribution of orientation tunings in inferotemporal neurons

In three experiments participants viewed pairs of overlapping transparent faces, with one face upright and the other oriented, and they reported which face was dominant. In each trial, an upright face was presented with a face at 45, 90, 135 or 180 degrees, with transparency set using a linear weighted algorithm, so that relative contrast across faces was biased in favour of oriented faces. Exposure duration was restricted in experiment 1 to 250, 500 or 1000 ms, but was unlimited in experiments 2 and 3. Adults were tested in experiments 1 and 2 and children aged 6-9 years of age were tested in experiment 3. Irrespective of exposure duration, the results showed the probability of dominance being ceded by oriented faces to upright faces was a function of orientation. In comparable conditions, the function found with young children was flatter than with adults. These patterns, and those of earlier perceptual studies, can be explained by the distribution of different orientation tunings found in physiological studies of inferotemporal cortex in macaques.

Perception of global facial geometry in the inversion effect and prosopagnosia

We investigated how efficiently combinations of positional shifts in facial features were perceived and whether the effects of combinations on the overall geometry of the face were reflected in discriminative performance. We moved the eyes closer together or further apart, and moved the mouth up or down. Trials with combinations of changes to both the mouth and the eyes were contrasted with trials with single changes to either the mouth or the eyes. As a contrast, we also examined combinations of changes in eye colour (brightness) and the same spatial manipulations. In addition, we specifically contrasted spatial combinations that more severely distorted the original triangular relation of the mouth and eyes (e.g. eyes closer and mouth down) to those that better preserved the original aspect ratio (e.g. eyes farther and mouth down). This we termed the "geometric context effect". We found that combinations of two spatial changes were detected more quickly and accurately by normal subjects viewing upright faces but not when faces were inverted. In contrast, combinations of spatial shifts and eye colour changes showed no advantage over faces with only one type of change. Combinations of spatial changes that distorted overall facial geometry more were detected more efficiently than less distorting combinations, showing that the spatial shifts were perceived in the context of the global facial structure. Again, this was found for upright but not inverted faces. We also tested a prosopagnosic patient, who showed the advantage for two spatial changes over one but lacked this geometric context effect, implying that she did not integrate local spatial information into overall facial structure.

Common and distinct neural responses during direct and incidental processing of multiple facial emotions

Whether common or distinct neural systems underpin perception of different emotions and the degree to which these systems are automatically engaged during emotional perception are unresolved. We performed an event-related fMRI experiment in which subjects viewed morphed emotional faces displaying low or high intensities of disgust, fear, happiness, or sadness under two task conditions. The amygdala and fusiform cortex responded to high-intensity expressions of all emotions, independent of task. Right superior temporal sulcus showed an additive effect of the emotion-directed task and high-intensity emotion. Ventromedial prefrontal and somatosensory cortices, regions implicated in providing representations of somatic states, showed enhanced activity during explicit emotional judgments. We failed to find predicted differences between emotions. The results suggest that amygdala contributes to task-independent perceptual processing of a range of emotions. We interpret ventromedial prefrontal and somatosensory cortex activations as evidence that these regions contribute to explicit emotion processing through linking emotion perception with representations of somatic states previously engendered by emotions.

Electrophysiological Correlates of Age and Gender Perception on Human Faces

In a previous experiment using scalp event-related potentials (ERPs), we have described the neuroelectric activities associated with the processing of gender information on human faces (Mouchetant-Rostaing, Giard, Bentin, Aguera, & Pernier, 2000). Here we extend this study by examining the processing of age on faces using a similar experimental paradigm, and we compare age and gender processing. In one session, faces were of the same gender (women) and of one age range (young or old), to reduce gender and age processing. In a second session, faces of young and old women were randomly intermixed but age was irrelevant for the task, hence, age discrimination, if any, was assumed to be incidental. In the third and fourth sessions, faces had to be explicitly categorized according to their age or gender, respectively (intentional discrimination). Neither age nor gender processing affected the occipito-temporal N170 component often associated with the detection of physiognomic features and global structural encoding of faces. Rather, the three age and gender discrimination conditions induced similar fronto-central activities around 145–185 msec. In our previous experiment, this ERP pattern was also found for implicit and explicit categorization of gender from faces but not in a control condition manipulating hand stimuli (Mouchetant-Rostaing, Giard, Bentin, et al., 2000). Whatever their exact nature, these 145–185 msec effects therefore suggest, first, that similar mechanisms could be engaged in age and gender perception, and second, that age and gender may be implicitly processed irrespective of their relevance to the task, through somewhat specialized mechanisms. Additional ERP effects were found at early latencies (45–90 msec) in all three discrimination conditions, and around 200–400 msec during explicit age and gender discrimination. These effects have been previously found in control conditions manipulating nonfacial stimuli and may therefore be related to more general categorization processes.

The use of facial motion and facial form during the processing of identity

Previous research has shown that facial motion can carry information about age, gender, emotion and, at least to some extent, identity. By combining recent computer animation techniques with psychophysical methods, we show that during the computation of identity the human face recognition system integrates both types of information: individual non-rigid facial motion and individual facial form. This has important implications for cognitive and neural models of face perception, which currently emphasize a separation between the processing of invariant aspects (facial form) and changeable aspects (facial motion) of faces.

Amygdala damage impairs emotion recognition from scenes only when they contain facial expressions

Bilateral damage to the human amygdala impairs recognition of negatively valenced emotions from facial expressions, but it is unclear if this finding generalizes to richer visual stimuli that contain cues in addition to faces. We investigated this issue in 4 subjects with bilateral amygdala damage, 23 with unilateral amygdala damage, 22 brain-damaged controls and 16 normal individuals. Subjects were shown two blocks of complex social scenes; all stimuli in the two blocks were identical, except that the first block had all facial expressions in the image erased. While control subjects were more accurate in recognizing emotions when facial expressions were present, subjects with bilateral amygdala damage did not show the same benefit for negative emotions, often performing equivalently across the two conditions. Most striking, subjects with bilateral amygdala damage were more accurate in recognizing scenes showing anger with faces erased than with faces present, an effect resulting in part from highly abnormal recognition of certain angry facial expressions. All four subjects with bilateral amygdala damage were impaired in recognizing angry faces shown in isolation, and frequently mistook expressions of anger for smiles, a mistake never made by any control subject. Bilateral amygdala damage thus disproportionately impairs recognition of certain emotions from complex visual stimuli when subjects utilize information from facial expressions.

Dissociation of human and computer voices in the brain: evidence for a preattentive gestalt-like perception

We investigated the early ("preattentive") cortical processing of voice information, using the so-called "mismatch response". This brain potential allows inferences to be made about the sensory short-term store. Most importantly, the mismatch potential also provides information about the organization of long-term memory traces in the auditory system. Such traces have reliably been reported for phonemes. However, it is unclear whether they also exist for human voice information. To explore this issue, 10 healthy subjects were presented with a single word stimulus uttered by voices of different prototypicality (natural, manipulated, synthetic) in a mismatch experiment (stimulus duration 380 msec, onset-to-onset interval 900 msec). The event-related magnetic fields were recorded by a 148-channel whole-head magnetometer and a source current density modeling of the magnetic field data was performed using a minimum-norm estimate. Each deviating voice signal in a series of standard-voice stimuli evoked a mismatch response that was localized in temporal brain regions bilaterally. Increased mismatch related magnetic flux was observed in response to decreased prototypicality of a presented voice signal, but did not correspond to the acoustic similarity of standard voice and deviant voices. We, therefore, conclude that the mismatch activation predominantly reflects the ecological validity of the voice signals. We further demonstrate that the findings cannot be explained by mere acoustic feature processing, but rather point towards a holistic mapping of the incoming voice signal onto long-term representations in the auditory memory.

The functionally defined right occipital and fusiform "face areas" discriminate novel from visually familiar faces

Neuroimaging (PET and fMRI) studies have identified a set of brain areas responding more to faces than to other object categories in the visual extrastriate cortex of humans. This network includes the middle lateral fusiform gyrus (the fusiform face area, or FFA) as well as the inferior occipital gyrus (occipital face area, OFA). The exact functions of these areas in face processing remain unclear although it has been argued that their primary function is to distinguish faces from nonface object categories-"face detection"-or also to discriminate among faces, irrespective of their visual familiarity to the observer. Here, we combined the data from two previous positron emission tomography (PET) studies to show that the functionally defined face areas are involved in the automatic discrimination between unfamiliar faces and familiar faces. Consistent with previous studies, a face localizer contrast (faces-objects) revealed bilateral activation in the middle lateral fusiform gyrus (FFA, BA37) and in the right inferior occipital cortex (OFA, BA19). Within all the regions of the right hemisphere, larger levels of activation were found for unfamiliar as compared to familiar faces. These results suggest that the very same areas involved in categorizing faces at the basic or individual level, play a role in differentiating familiar faces from new faces, showing an overlap between visual and presemantic mnesic representations of faces in the right hemisphere.

The visual word form area: expertise for reading in the fusiform gyrus

Brain imaging studies reliably localize a region of visual cortex that is especially responsive to visual words. This brain specialization is essential to rapid reading ability because it enhances perception of words by becoming specifically tuned to recurring properties of a writing system. The origin of this specialization poses a challenge for evolutionary accounts involving innate mechanisms for functional brain organization. We propose an alternative account, based on studies of other forms of visual expertise (i.e. bird and car experts) that lead to functional reorganization. We argue that the interplay between the unique demands of word reading and the structural constraints of the visual system lead to the emergence of the Visual Word Form Area.

Brain networks for analyzing eye gaze

The eyes convey a wealth of information in social interactions. This information is analyzed by multiple brain networks, which we identified using functional magnetic resonance imaging (MRI). Subjects attempted to detect a particular directional cue provided either by gaze changes on an image of a face or by an arrow presented alone or by an arrow superimposed on the face. Another control condition was included in which the eyes moved without providing meaningful directional information. Activation of the superior temporal sulcus accompanied extracting directional information from gaze relative to directional information from an arrow and relative to eye motion without relevant directional information. Such selectivity for gaze processing was not observed in face-responsive fusiform regions. Brain activations were also investigated while subjects viewed the same face but attempted to detect when the eyes gazed directly at them. Most notably, amygdala activation was greater during periods when direct gaze never occurred than during periods when direct gaze occurred on 40% of the trials. In summary, our results suggest that increases in neural processing in the amygdala facilitate the analysis of gaze cues when a person is actively monitoring for emotional gaze events, whereas increases in neural processing in the superior temporal sulcus support the analysis of gaze cues that provide socially meaningful spatial information.

Right lateral fusiform gyrus dysfunction during facial information processing in schizophrenia

BACKGROUND

Patients with schizophrenia exhibit facial information processing deficits that likely contribute to their social dysfunction. Whether the deficits involve facial affect and/or identity processing or result from other cognitive abnormalities in schizophrenia remains controversial, and a brain dysfunction specifically related to them has never been reported. If such dysfunction existed, it should be consistently observed across groups of patients and during performance of different facial information processing tasks, independently of whether such tasks demand working memory (WM), semantic, or other cognitive processes. We hypothesized that the right lateral fusiform gyrus (rLFG), one of several human brain areas involved in facial information processing, would consistently show activation abnormalities during both facial affect and identity discrimination in schizophrenia.

METHODS

We used functional magnetic resonance imaging to measure brain activation in two groups of six chronic, stable schizophrenic outpatients and two of six age- and gender-matched healthy controls. One group of patients and one of controls performed facial affect-with or without semantic processing-and identity discrimination tasks, and the other two groups WM tasks with facial expression cues and varying attentional demands.

RESULTS

Patients from either group failed to activate the rLFG when compared to controls in any task. Other activation abnormalities were task-specific (i.e., seen only during performance of one set of tasks) and not consistently observed in both groups of patients, and thus could not be directly and solely linked to facial information discrimination.

CONCLUSIONS

These results indicate a specific rLFG dysfunction during early facial information--identity or affect--processing, independent from other cognitive deficits, in schizophrenia.

Distinct spatial frequency sensitivities for processing faces and emotional expressions

High and low spatial frequency information in visual images is processed by distinct neural channels. Using event-related functional magnetic resonance imaging (fMRI) in humans, we show dissociable roles of such visual channels for processing faces and emotional fearful expressions. Neural responses in fusiform cortex, and effects of repeating the same face identity upon fusiform activity, were greater with intact or high-spatial-frequency face stimuli than with low-frequency faces, regardless of emotional expression. In contrast, amygdala responses to fearful expressions were greater for intact or low-frequency faces than for high-frequency faces. An activation of pulvinar and superior colliculus by fearful expressions occurred specifically with low-frequency faces, suggesting that these subcortical pathways may provide coarse fear-related inputs to the amygdala.

Rapid extraction of emotional expression: evidence from evoked potential fields during brief presentation of face stimuli

Although the emotional expression of faces is believed to be accessed rapidly, previous ERP studies hardly found correlates of these processes. Here, we report findings from a study that investigated dichoptic binocular interaction using emotional face stimuli. Thirty-one subjects were briefly presented with schematic normal and scrambled faces (of neutral, positive, or negative expression) that occurred simultaneously in the left and right visual fields. Stimuli for both eyes could be congruent (control) or incongruent (dichoptic). Subjects decided which of the superimposed images in both hemi-fields appeared more "face-like" and during this task, the EEG was recorded from 30 channels. VEPs were analysed topographically according to the influence of the different experimental conditions (defined by presentation form, emotional expression, and location). Behavioural responses to the ambiguous dichoptic stimuli demonstrated a functional eye dominance not related to visual acuity and conventional eye preference. Electrophysiological data revealed three components with mean latencies of 85, 160, and 310 ms. Topography of the second component (equivalent to the face-related N170) differed in left-right and anterior-posterior direction compared with simple checkerboard stimuli. Dichoptic presentation caused reduced field strength of all three, and increased latency of the first component. Faces with negative expression yielded largest field strength of the second and third components. Besides that, emotional expression affected topography not only of late, but also the first component. This provides new evidence about the timing of perceptual processes related to facial expression, indicating that already VEP components occurring at 80-90 ms are sensitive to emotional content.

Brain regions sensitive to the face inversion effect: a functional magnetic resonance imaging study in humans

Perception of upright faces relies on configural processing. Therefore recognition of inverted, compared to upright faces is impaired. In a functional magnetic resonance imaging experiment we investigated the neural correlate of a face inversion task. Thirteen healthy subjects were presented with a equal number of upright and inverted faces alternating with a low level baseline with an upright and inverted picture of an abstract symbol. Brain activation was calculated for upright minus inverted faces. For this differential contrast, we found a signal change in the right superior temporal sulcus and right insula. Configural properties are processed in a network comprising right superior temporal and insular cortex.

Time course of regional brain activations during facial emotion recognition in humans

Recognition of facial expressions of emotions is very important for communication and social cognition. Neuroimaging studies showed that numerous brain regions participate in this complex function. To study spatiotemporal aspects of the neural representation of facial emotion recognition we recorded neuromagnetic activity in 12 healthy individuals by means of a whole head magnetoencephalography system. Source reconstructions revealed that several cortical and subcortical brain regions produced strong neural activity in response to emotional faces at latencies between 100 and 360 ms that were much stronger than those to neutral as well as to blurred faces. Orbitofrontal cortex and amygdala showed affect-related activity at short latencies already within 180 ms after stimulus onset. Some of the emotion-responsive regions were repeatedly activated during the stimulus presentation period pointing to the assumption that these reactivations represent indicators of a distributed interacting circuitry.

Disorders of face perception and recognition

Face recognition is one of the most complex visual tasks performed by the human brain. Data from monkeys suggest that area IT may play a key role in identifying faces, and functional imaging research suggests that the human homologue of IT may be located in the medial occipitotemporal cortex, where a FFA has been located. Damage to medial occipitotemporal structures on the right or bilaterally leads to prosopagnosia, the failure to recognize facial identity. Prosopagnosia is not a single functional disorder but a family of dysfunctions, with different patients having different degrees of impairments to various perceptual and memory stages involved in face processing. Understanding the perceptual basis of this disorder and epiphenomena, such as covert recognition, is a goal of current research. Deficits in face perception also may contribute to Capgras syndrome and may be related to the impaired social development of patients with Asperger syndrome. More recently, identified deficits in face processing include the false recognition of unfamiliar faces and the impaired extraction of social information from faces, independent of the recognition of identity. Many of these prosopagnosia and other face processing deficits can be placed in the context of cognitive models of face processing stages, which are being refined continually by data from neurologic patients and functional imaging in normal subjects.

Face-selective activation in a congenital prosopagnosic subject

Congenital prosopagnosia is a severe impairment in face identification manifested from early childhood in the absence of any evident brain lesion. In this study, we used fMRI to compare the brain activity elicited by faces in a congenital prosopagnosic subject (YT) relative to a control group of 12 subjects in an attempt to shed more light on the nature of the brain mechanisms subserving face identification. The face-related activation pattern of YT in the ventral occipito-temporal cortex was similar to that observed in the control group on several parameters: anatomical location, activation profiles, and hemispheric laterality. In addition, using a modified vase-face illusion, we found that YT's brain activity in the face-related regions manifested global grouping processes. However, subtle differences in the degree of selectivity between objects and faces were observed in the lateral occipital cortex. These data suggest that face-related activation in the ventral occipito-temporal cortex, although necessary, might not be sufficient by itself for normal face identification.

The neural basis of object perception

Humans can recognize an object within a fraction of a second, even if there are no clues about what kind of object it might be. Recent findings have identified functional properties of extrastriate regions in the ventral visual pathway that are involved in the representation and perception of objects and faces. The functional properties of these regions, and the correlation between the activation of these regions and visual recognition, indicate that the lateral and ventral occipito-temporal areas are important in perceiving and recognizing objects and faces.

Matching expression variant faces

Several models have been proposed that attempt to explain how the brain identifies people by looking at their faces. However, to date, it is still not clear by which mechanism the brain successfully accomplishes the matching of two or more face images when differences in facial expression make the (local and global) appearance of these images different from one another. There seems to be a consensus that faces are processed holistically rather than locally, but there is not yet consensus on whether information on facial expression is passed to the identification process to aid recognition of individuals or not. Models have been proposed that exploit each of these two views, and psychophysical data exist in favor of and against each view. In this article, we show how the experimental data of these two opposite views can be explained by incorporating a key process of motion estimation in the classical feedforward model of face processing. This new model will then lead us to hypothesize that to successfully match expression variant faces, it is convenient to use the information supplied by this motion estimation process within the matching task. We will show experimental results in favor of this hypothesis. Finally, we will show how we can also use the same motion estimator to recognize facial expressions.

Facial expressions, their communicatory functions and neuro-cognitive substrates

Human emotional expressions serve a crucial communicatory role allowing the rapid transmission of valence information from one individual to another. This paper will review the literature on the neural mechanisms necessary for this communication: both the mechanisms involved in the production of emotional expressions and those involved in the interpretation of the emotional expressions of others. Finally, reference to the neuro-psychiatric disorders of autism, psychopathy and acquired sociopathy will be made. In these conditions, the appropriate processing of emotional expressions is impaired. In autism, it is argued that the basic response to emotional expressions remains intact but that there is impaired ability to represent the referent of the individual displaying the emotion. In psychopathy, the response to fearful and sad expressions is attenuated and this interferes with socialization resulting in an individual who fails to learn to avoid actions that result in harm to others. In acquired sociopathy, the response to angry expressions in particular is attenuated resulting in reduced regulation of social behaviour.

Large-scale mirror-symmetry organization of human occipito-temporal object areas

We have combined functional maps of retinotopy (eccentricity and meridian mapping), object category, and motion in a group of subjects to explore the large-scale topography of higher-order object areas. Our results reveal seven consistent category-related entities situated in the occipito-temporal cortex adjoining early visual areas. These include two face-related regions, three object-related regions, and two building-related regions. Interestingly, this complex category-related pattern is organized in a large-scale dorso-ventral mirror symmetry of object category. Furthermore, correlating this pattern to the map of visual field eccentricity, we found that the entire network of areas could be related to a single and unified eccentricity map. We hypothesize that this large-scale organization points to a possible development of high-order object areas through extension and specialization of a single proto-representation.

Brain activation evoked by perception of gaze shifts: the influence of context

Prior studies from our laboratory [Journal of Neuroscience 18 (1998) 2188; Cognitive Neuropsychology 17 (2000) 221] have demonstrated that discrete regions of the superior temporal sulcus (STS) are activated when a subject views a face in which the eyes shift their gaze. Here we investigated the degree to which activity in the STS and other brain regions is modulated by the context of the perceived gaze shift; that is, when the shift correctly or incorrectly acquires a visual target. Fifteen subjects participated in an event-related functional magnetic resonance imaging experiment in which they viewed an animated face that remained present throughout each run. On each of 21 trials within each run, a small checkerboard appeared and flickered at one of six locations within the character's visual field. On "correct" trials, the character shifted its gaze towards the checkerboard after a delay of 1 or 3s. On "incorrect" trials, the character shifted its gaze towards empty space after the same delays. On "no shift" trials, the character's eyes did not move. Significantly larger hemodynamic responses (HDR) were evoked by gaze shifts compared to no gaze shifts in primarily right hemisphere STS. The gaze-evoked HDR was significantly delayed in peak amplitude for 3s compared to 1s shifts. For 1s shifts, a strong effect of context was observed in which errors evoked a HDR with extended duration. Although this study focused upon STS, similar effects were also observed in the intraparietal sulcus and fusiform gyrus.

Spatio-temporal dynamics of brain mechanisms in aversive classical conditioning: high-density event-related potential and brain electrical tomography analyses

Social cognition, including complex social judgments and attitudes, is shaped by individual learning experiences, where affect often plays a critical role. Aversive classical conditioning-a form of associative learning involving a relationship between a neutral event (conditioned stimulus, CS) and an aversive event (unconditioned stimulus, US)-represents a well-controlled paradigm to study how the acquisition of socially relevant knowledge influences behavior and the brain. Unraveling the temporal unfolding of brain mechanisms involved appears critical for an initial understanding about how social cognition operates. Here, 128-channel ERPs were recorded in 50 subjects during the acquisition phase of a differential aversive classical conditioning paradigm. The CS+ (two fearful faces) were paired 50% of the time with an aversive noise (CS upward arrow + /Paired), whereas in the remaining 50% they were not (CS upward arrow + /Unpaired); the CS- (two different fearful faces) were never paired with the noise. Scalp ERP analyses revealed differences between CS upward arrow + /Unpaired and CS- as early as approximately 120 ms post-stimulus. Tomographic source localization analyses revealed early activation modulated by the CS+ in the ventral visual pathway (e.g. fusiform gyrus, approximately 120 ms), right middle frontal gyrus (approximately 176 ms), and precuneus (approximately 240 ms). At approximately 120 ms, the CS- elicited increased activation in the left insula and left middle frontal gyrus. These findings not only confirm a critical role of prefrontal, insular, and precuneus regions in aversive conditioning, but they also suggest that biologically and socially salient information modulates activation at early stages of the information processing flow, and thus furnish initial insight about how affect and social judgments operate.

Neural response suppression, haemodynamic repetition effects, and behavioural priming

Repeated stimulus processing is often associated with a reduction in neural activity, as measured by single-cell recording or by haemodynamic imaging techniques like PET and fMRI. These reductions are sometimes linked to the behavioural phenomenon of priming. In this article, we discuss issues relevant to theories that attempt to relate these phenomena, concentrating in particular on the interpretative limitations of current imaging techniques.

Cognitive neuroscience of human social behaviour

We are an intensely social species--it has been argued that our social nature defines what makes us human, what makes us conscious or what gave us our large brains. As a new field, the social brain sciences are probing the neural underpinnings of social behaviour and have produced a banquet of data that are both tantalizing and deeply puzzling. We are finding new links between emotion and reason, between action and perception, and between representations of other people and ourselves. No less important are the links that are also being established across disciplines to understand social behaviour, as neuroscientists, social psychologists, anthropologists, ethologists and philosophers forge new collaborations.

Novelty responses and differential effects of order in the amygdala, substantia innominata, and inferior temporal cortex

Recent studies of amygdala function have focused on examining responses to emotionally valenced versus neutral stimuli. However, electrophysiologic and neuroimaging studies also suggest that novel neutral faces activate the amygdala, though few investigations have examined the effects of novelty and its relation to changes in stimulus condition. To further investigate how the human amygdala and related structures react to novel neutral faces and to stimulus condition changes, we evaluated human brain responses to blocks containing multiple novel and single repeated face stimuli, presented in two different orders, using functional magnetic resonance imaging (fMRI). Significantly increased signal was present in the amygdala, substantia innominata (SI), and inferior temporal cortex (ITC) to the contrast of multiple novel versus single faces. However, these regions differed in their responses based on whether a stimulus condition was presented 1st or 2nd, with the amygdala and SI having significantly different response profiles than the ITC. Specifically, greater responses to stimuli presented 2nd (i.e., after a condition change) were found in the amygdala and SI, but not in the ITC. Furthermore, the response difference to the Multiple versus Single contrast was greatest in the amygdala and SI, when single faces were presented 1st, and multiple faces presented 2nd, but this pattern was the reverse in the ITC. We speculate that the signal changes to neutral faces in the amygdala and SI with respect to condition (multiple or single faces) and stimulus order may relate to the involvement of these structures in novelty detection and the orienting response.

Face versus non-face object perception and the 'other-race' effect: a spatio-temporal event-related potential study

OBJECTIVE

To investigate a modulation of the N170 face-sensitive component related to the perception of other-race (OR) and same-race (SR) faces, as well as differences in face and non-face object processing, by combining different methods of event-related potential (ERP) signal analysis.

METHODS

Sixty-two channel ERPs were recorded in 12 Caucasian subjects presented with Caucasian and Asian faces along with non-face objects. Surface data were submitted to classical waveforms and ERP map topography analysis. Underlying brain sources were estimated with two inverse solutions (BESA and LORETA).

RESULTS

The N170 face component was identical for both race faces. This component and its topography revealed a face specific pattern regardless of race. However, in this time period OR faces evoked significantly stronger medial occipital activity than SR faces. Moreover, in terms of maps, at around 170 ms face-specific activity significantly preceded non-face object activity by 25 ms. These ERP maps were followed by similar activation patterns across conditions around 190-300 ms, most likely reflecting the activation of visually derived semantic information.

CONCLUSIONS

The N170 was not sensitive to the race of the faces. However, a possible pre-attentive process associated to the relatively stronger unfamiliarity for OR faces was found in medial occipital area. Moreover, our data provide further information on the time-course of face and non-face object processing.