Electrophysiological studies of human face perception. II: Response properties of face-specific potentials generated in occipitotemporal cortex

Electrophysiological study of contextual variations in a short-term face recognition task

Event-related potentials (ERPs) were recorded during two short-term recognition tasks using unfamiliar faces. These experiments are based on the process dissociation procedure (PDP), whereby the exclusion criterion was an intrinsic context or extrinsic context, the facial expression (Experiment 1) or background (Experiment 2), respectively. The results indicate that retrieval orientation, in addition to extensive strategic control, affects both the frontal (N250) and temporoparietal (P3b) components. Furthermore, these data indicate that an early frontal modulation interacts between processing that bears on the face (interactive intrinsic context) and processing that bears on two objects at the same time (interactive extrinsic context), in which, in the latter case, that the background change led to an early modulation at the frontal sites in the left hemisphere. These results are consistent with the idea that frontal effects reflect differences in the nature of the information during retrieval and postretrieval processes involved. Furthermore, that the left posterior repetition effect appears to be a manifestation of the retrieval of relevant contextual information that perturbs the recognition decision, whereas the right posterior repetition effect reflects to be the outcome of the retrieval of the face as a whole. Finally, results are in concordance with the hypothesis that the difference during recognition with or without source memory may be in the strength of the relationship between the target and the contextual information to be retrieved. In essence, that automatic and controlled processes in a given context depends on both task-related and target-related constraints.

The many faces of the gamma band response to complex visual stimuli

While much is known about the functional architecture of the visual system, little is known about its large-scale dynamics during perception. This study describes this dynamics with a high spatial, temporal and spectral resolution. We recorded depth EEG of epileptic patients performing a face detection task and found that the stimuli induced strong modulations in the gamma band (40 Hz to 200 Hz) in selective occipital, parietal and temporal sites, in particular the fusiform gyrus, the lateral occipital gyrus and the intra-parietal sulcus. Occipito-temporal sites were the first to be activated, closely followed by the parietal sites, while portions of the primary visual cortex seemed to deactivate temporarily. Some of those effects were found to be correlated across distant sites, suggesting that a coordinated balance between regional gamma activations and deactivations could be involved during visual perception.

Face perception: domain specific, not process specific

Evidence that face perception is mediated by special cognitive and neural mechanisms comes from fMRI studies of the fusiform face area (FFA) and behavioral studies of the face inversion effect. Here, we used these two methods to ask whether face perception mechanisms are stimulus specific, process specific, or both. Subjects discriminated pairs of upright or inverted faces or house stimuli that differed in either the spatial distance among parts (configuration) or the shape of the parts. The FFA showed a much higher response to faces than to houses, but no preference for the configuration task over the part task. Similarly, the behavioral inversion effect was as large in the part task as the configuration task for faces, but absent in both part and configuration tasks for houses. These findings indicate that face perception mechanisms are not process specific for parts or configuration but are domain specific for face stimuli per se.

The Influence of Repetition and Famousness on the Intracranially Recorded Temporobasal N200

It is still a matter of debate at which time point faces are recognized as familiar, with some studies claiming a relatively early face recognition and others later effects of familiarity. The authors report on effects of famousness of depicted persons and stimulus repetition on intracranially recorded event-related potentials. Famousness resulted in an increased latency of the N200 component, as well as in an increased amplitude of a later long-lasting potential (N700). In contrast, repetition led to an increased amplitude of the N200 but no increase of its latency. They suppose that the recognition of faces becomes observable within the N200 latency range and that the increased N200 latency reflects a feature processing additional to the holistic face processing.

Neuroanatomical substrates of social cognition dysfunction in autism

In this review article, we summarize recent progress toward understanding the neural structures and circuitry underlying dysfunctional social cognition in autism. We review selected studies from the growing literature that has used the functional neuroimaging techniques of cognitive neuroscience to map out the neuroanatomical substrates of social cognition in autism. We also draw upon functional neuroimaging studies with neurologically normal individuals and individuals with brain lesions to highlight the insights these studies offer that may help elucidate the search for the neural basis of social cognition deficits in autism. We organize this review around key brain structures that have been implicated in the social cognition deficits in autism: (1) the amygdala, (2) the superior temporal sulcus region, and (3) the fusiform gyrus. We review some of what is known about the contribution of each structure to social cognition and then review autism studies that implicate that particular structure. We conclude with a discussion of several potential future directions in the cognitive neuroscience of social deficits in autism.

Deviant gaze processing in children with autism: an ERP study

This study investigated event-related potentials (ERP) during an oddball task in which detection of specific eye direction was required of children with and without autism. The detection of a change in eye direction elicited occipito-temporal negativity, which had two major differences between children with and without autism. First, while this occipito-temporal negativity predominated in the right hemisphere of typically developed children, it was distributed equally bilaterally in children with autism. Second, the amplitude of this negativity was more pronounced in typically developed children in response to the detection of direct gaze as compared to averted gaze, but was not sensitive to direct/averted gaze direction in children with autism, which converges with behavioral reports. The results concur with previous literature, suggesting the importance of the right hemisphere, especially the superior temporal sulcus, in gaze processing. Results indicate that deviant neural substrates might be involved in gaze processing in individuals with autism.

Seeing faces in the noise: stochastic activity in perceptual regions of the brain may influence the perception of ambiguous stimuli

Research on binocular rivalry and motion direction discrimination suggests that stochastic activity early in visual processing influences the perception of ambiguous stimuli. Here, we extend this to higher level tasks of word and face processing. In Experiment 1, we used blocked gender and word discrimination tasks, and in Experiment 2, we used a face versus word discrimination task. Stimuli were embedded in noise, and some trials contained only noise. In Experiment 1, we found a larger response in the N170, an ERP component associated with faces, to the noise-alone stimulus when observers were performing the gender discrimination task. The noise-alone trials in Experiment 2 were binned according to the observer's behavioral response, and there was a greater response in the N170 when they reported seeing a face. After considering various top-down and priming-related explanations, we raise the possibility that seeing a face in noise may result from greater stochastic activity in neural face-processing regions.

Morphing Marilyn into Maggie dissociates physical and identity face representations in the brain

How the brain represents different aspects of faces remains controversial. Here we presented subjects with stimuli drawn from morph continua between pairs of famous faces. In the paired presentations, a second face could be identical to the first, could share perceived identity but differ physically (30% along the morph continuum), or could differ physically by the same distance along the continuum (30%) but in the other direction. We show that, behaviorally, subjects are more likely to classify face pairs in the third paired presentation as different and that this effect is more pronounced for subjects who are more familiar with the faces. In functional magnetic resonance imaging (fMRI), inferior occipital gyrus (IOG) shows sensitivity to physical rather than to identity changes, whereas right fusiform gyrus (FFG) shows sensitivity to identity rather than to physical changes. Bilateral anterior temporal regions show sensitivity to identity change that varies with the subjects' pre-experimental familiarity with the faces. These findings provide neurobiological support for a hierarchical model of face perception.

Recognition of facial expression and visual P300 in schizophrenic patients: differences between paranoid type patients and non-paranoid patients

This study compared the effects of facial affective stimuli on visual event-related potentials (ERP) in schizophrenic patients and healthy subjects using photographs of babies depicting sadness (crying face), neutrality (neutral face), and pleasure (smiling face). Visual ERP were recorded using an oddball paradigm in 32 schizophrenic patients (16 paranoid type and 16 non-paranoid patients) and 32 age-matched healthy subjects. The P300 amplitude, latency, and the subject's reaction time were recorded. The P300 amplitude when viewing a photograph of a smiling baby was the smallest registered of three photographs for healthy subjects and paranoid type patients with successively greater amplitudes for neutrality and sadness. However, the P300 amplitude was the smallest while viewing crying photographs and was the largest while viewing a smiling photograph for non-paranoid patients. These results suggest that the P300 amplitude is influenced by viewing emotionally moving facial expressions and that the effect is different for different subtypes of schizophrenia. These differences may reflect differences in information processing resulted from emotional influences caused by visual-affective stimuli.

Parametric design and correlational analyses help integrating fMRI and electrophysiological data during face processing

Face perception is typically associated with activation in the inferior occipital, superior temporal (STG), and fusiform gyri (FG) and with an occipitotemporal electrophysiological component peaking around 170 ms on the scalp, the N170. However, the relationship between the N170 and the multiple face-sensitive activations observed in neuroimaging is unclear. It has been recently shown that the amplitude of the N170 component monotonically decreases as gaussian noise is added to a picture of a face [Jemel et al., 2003]. To help clarify the sources of the N170 without a priori assumptions regarding their number and locations, ERPs and fMRI were recorded in five subjects in the same experiment, in separate sessions. We used a parametric paradigm in which the amplitude of the N170 was modulated by varying the level of noise in a picture, and identified regions where the percent signal change in fMRI correlated with the ERP data. N170 signals were observed for pictures of both cars and faces but were stronger for faces. A monotonic decrease with added noise was observed for the N170 at right hemisphere sites but was less clear on the left and occipital central sites. Correlations between fMRI signal and N170 amplitudes for faces were highly significant (P < 0.001) in bilateral fusiform gyrus and superior temporal gyrus. For cars, the strongest correlations were observed in the parahippocampal region and in the STG (P < 0.005). Besides contributing to clarify the spatiotemporal course of face processing, this study illustrates how ERP information may be used synergistically in fMRI analyses. Parametric designs may be developed further to provide some timing information on fMRI activity and help identify the generators of ERP signals.

The Faces of Development: A Review of Early Face Processing over Childhood

The understanding of the adult proficiency in recognizing and extracting information from faces is still limited despite the number of studies over the last decade. Our knowledge on the development of these capacities is even more restricted, as only a handful of such studies exist. Here we present a combined reanalysis of four ERP studies in children from 4 to 15 years of age and adults (n = 424, across the studies), which investigated face processing in implicit and explicit tasks. We restricted these analyses to what was common across studies: early ERP components and upright face processing across all four studies and the inversion effect, investigated in three of the studies. These data demonstrated that processing faces implicates very rapid neural activity, even in young children— at the P1 component—with protracted age-related change in both P1 and N170, that were sensitive to the different task demands. Inversion produced latency and amplitude effects on the P1 from the youngest group, but on N170 only starting in mid childhood. These developmental data suggest that there are functionally different sources of the P1 and N170, related to the processing of different aspects of faces.

Source analysis of the N170 to faces and objects

To clarify the neural sources for the face-sensitive N170, ERPs were recorded in 16 subjects viewing upright and inverted faces, and compared to seven object categories. Source analyses were performed for each category and each subject at the latency of the N170. Larger source intensities were found in the posterior superior temporal sulcus region (STS) for faces compared to objects. STS intensities were highly correlated to the N170 amplitude over both hemispheres only for faces. The results suggest that one of the major sources of the N170 is the STS region and that the larger N170 amplitude to inverted faces is due to increased activation of that source for inverted faces rather than a recruitment of additional areas.

Interaction of gaze direction and facial expressions processing: ERP study

Event related potentials (ERPs) were recorded from 11 subjects watching photographs of angry and happy faces with different gaze directions. ERPs to right averted gaze differed from those to straight and left averted gaze at 85 and 460 ms whereas ERPs to happy and angry expressions differed at 115, 330 and 380 ms. We suggest that short-latency effects, maximal over occipital cortex, reflect the involvement of visual cortex in the early analysis of socially-relevant stimuli. Interaction of gaze and expressions was reflected in ERPs at 270 - 450 ms. We conclude that gaze and emotional expressions are analyzed in parallel at the early stages of visual processing. The interaction of these two processing streams starts no earlier than at 270 ms.

Magnetoencephalographic study of occipitotemporal activity elicited by viewing mouth movements

OBJECTIVE

We studied the temporal and spatial characteristics of neural responses elicited by viewing mouth movements using magnetoencephalography.

METHODS

We focused on differences in responses to mouth opening and closing movements by apparent motion, using an averting eyes condition as a control.

RESULTS

A large clear MEG component, 1 M (mean peak latency of approximately 160 ms), was elicited by both mouth movements. We modeled the neural sources using a brain electric source analysis (BESA) method and placed the sources around: (1) the occipitotemporal border at human MT/V5, (2) the primary visual cortex (V1), and (3) fusiform gyrus. The calculated activity of Source (1) was large whereas the activity of the others was small or negligible. Source (1), as calculated separately for mouth closing and opening movements and eye movement, showed no significant different amplitude and locations. We did not find any activity in the superior temporal sulcus (STS).

CONCLUSIONS

Our results indicate that human MT/V5 is active in the perception of both mouth and eye motions. Viewing mouth and eye movements elicits no significant differences in MT/V5 activity, indicating that the perception of movement of facial parts is probably processed in the same manner.

SIGNIFICANCE

Characteristic activities in the human MT/V5 elicited by viewing mouth movement were clarified by MEG.

An earlier component of face perception detected by seeing-as-face task

To investigate the time sequence of the neural processes underlying face perception, magnetoencephalography was performed using a seeing-as-face task, in which visual inputs were identical across two conditions, but subject perceptions differed: one being a non-specific pattern of geographical shapes, the other being a percept of a face. Subtraction between the two conditions revealed a response occurring 120 ms after stimulus onset in right occipital, approximately 50 ms earlier than previously reported response at a latency of 170 ms at the right fusiform gyrus. As our novel task completely excluded differences in low-level properties of visual stimuli between control and face conditions, these two responses were considered specific to face perception. The result supported the two-stage theory of face processing.

Face recognition memory and configural processing: a developmental ERP study using upright, inverted, and contrast-reversed faces

The effects of configural changes on faces were investigated in children to determine their role in encoding and recognition processes. Upright, inverted, and contrast-reversed unfamiliar faces were presented in blocks in which one-third of the pictures repeated immediately or after one intervening face. Subjects (8-16 years) responded to repeated faces; event-related potentials were recorded throughout the procedure. Recognition improved steadily with age and all components studied showed age effects reflecting differing maturation processes occurring until adulthood. All children were affected by inversion and contrast-reversal, and face-type effects were seen on latencies and amplitudes of early components (P1 and N170), as well as on later frontal amplitudes. The "old-new" repetition effects (larger amplitude for repeated stimuli) were found at frontal sites and were similar across age groups and face types, suggesting a general working memory system comparably involved in all age groups. These data demonstrate that (1) there is quantitative development in face processing, (2) both face encoding and recognition improve with age, but (3) only encoding is affected by configural changes. The data also suggest a gradual tuning of face processing towards the upright orientation.

Scale invariant adaptation in fusiform face-responsive regions

Several functional neuroimaging studies have observed response adaptation in face-sensitive regions when repeating identical face stimuli. To address whether this was due to low-level stimulus properties or facial identity, we decomposed pictures of faces into pictures preserving only the lower or higher parts of the normal frequency spectrum. In an event-related functional neuroimaging study, pairs of such pictures were sequentially presented that showed the same or different persons in the same or different frequency bands. This factorial design allowed to separate effects related to repetition of personal identity from those related to identical stimulus properties. In a random effects group analysis, activation in the right fusiform region was affected by repetition of personal identity regardless of changing or constant spatial scale. Responses in the more medial and posterior fusiform and lingual regions adapted with repetition of the same frequency band. An analysis in regions of interest determined individually as face responsive showed that repetition decreases for the same faces in fusiform face-responsive regions generalized across spatial frequency bands. Our results therefore point to a role of this area in discriminating individual faces at a level of representation that is invariant to changes in low-level stimulus properties, as spatial scale. The same invariance could not be detected in more posterior occipital face-responsive regions.

fMRI-adaptation reveals dissociable neural representations of identity and expression in face perception

The distributed model of face processing proposes an anatomical dissociation between brain regions that encode invariant aspects of faces, such as identity, and those that encode changeable aspects of faces, such as expression. We tested for a neuroanatomical dissociation for identity and expression in face perception using a functional MRI (fMRI) adaptation paradigm. Repeating identity across face pairs led to reduced fMRI signal in fusiform cortex and posterior superior temporal sulcus (STS), whereas repeating emotional expression across pairs led to reduced signal in a more anterior region of STS. These results provide neuroanatomical evidence for the distributed model of face processing and highlight a dissociation within right STS between a caudal segment coding identity and a more rostral region coding emotional expression.

Looking for faces: Attention modulates early occipitotemporal object processing

Looking for somebody's face in a crowd is one of the most important examples of visual search. For this goal, attention has to be directed to a well-defined perceptual category. When this categorically selective process starts is, however, still unknown. To this end, we used magnetoencephalography (MEG) recorded over right human occipitotemporal cortex to investigate the time course of attentional modulation of perceptual processes elicited by faces and by houses. The first face-distinctive MEG response was observed at 160-170 ms (M170). Nevertheless, attention did not start to modulate face processing before 190 ms. The first house-distinctive MEG activity was also found around 160-170 ms. However, house processing was not modulated by attention before 280 ms (90 ms later than face processing). Further analysis revealed that the attentional modulation of face processing is not due to later, for example, back-propagated activation of the M170 generator. Rather, subsequent stages of occipitotemporal object processing were modulated in a category-specific manner and with preferential access to face processing.

Neuronal Correlates of Face Identification in the Monkey Anterior Temporal Cortical Areas

To investigate the neuronal basis underlying face identification, the activity of face neurons in the anterior superior temporal sulcus (STS) and the anterior inferior temporal gyrus (ITG) of macaque monkeys was analyzed during their performance of a face-identification task. The face space was composed by the activities of face neurons during the face-identification task, based on a multidimensional scaling (MDS) method; the face space composed by the anterior STS neurons represented facial views, whereas that composed by the anterior ITG neurons represented facial identity. The temporal correlation between the behavioral reaction time of the animal and the latency of face-related neuronal responses was also analyzed. The response latency of some of the face neurons in the anterior ITG exhibited a significant correlation with the behavioral reaction time, whereas this correlation was not significant in the anterior STS. The correlation of the latency of face-related neuronal responses in the anterior ITG with the behavioral reaction time was not found to be attributed to the correlation between the response latency and the magnitude of the neuronal responses. The present results suggest that the anterior ITG is closely related to judgments of facial identity, and that the anterior STS is closely related to analyses of incoming perceptual information; face identification in monkeys might involve interactions between the two areas.

Processing faces and facial expressions

This paper reviews processing of facial identity and expressions. The issue of independence of these two systems for these tasks has been addressed from different approaches over the past 25 years. More recently, neuroimaging techniques have provided researchers with new tools to investigate how facial information is processed in the brain. First, findings from "traditional" approaches to identity and expression processing are summarized. The review then covers findings from neuroimaging studies on face perception, recognition, and encoding. Processing of the basic facial expressions is detailed in light of behavioral and neuroimaging data. Whereas data from experimental and neuropsychological studies support the existence of two systems, the neuroimaging literature yields a less clear picture because it shows considerable overlap in activation patterns in response to the different face-processing tasks. Further, activation patterns in response to facial expressions support the notion of involved neural substrates for processing different facial expressions.

Early processing of the six basic facial emotional expressions

Facial emotions represent an important part of non-verbal communication used in everyday life. Recent studies on emotional processing have implicated differing brain regions for different emotions, but little has been determined on the timing of this processing. Here we presented a large number of unfamiliar faces expressing the six basic emotions, plus neutral faces, to 26 young adults while recording event-related potentials (ERPs). Subjects were naive with respect to the specific questions investigated; it was an implicit emotional task. ERPs showed global effects of emotion from 90 ms (P1), while latency and amplitude differences among emotional expressions were seen from 140 ms (N170 component). Positive emotions evoked N170 significantly earlier than negative emotions and the amplitude of N170 evoked by fearful faces was larger than neutral or surprised faces. At longer latencies (330-420 ms) at fronto-central sites, we also found a different pattern of effects among emotions. Localization analyses confirmed the superior and middle-temporal regions for early processing of facial expressions; the negative emotions elicited later, distinctive activations. The data support a model of automatic, rapid processing of emotional expressions.

Effect of head orientation on gaze processing in fusiform gyrus and superior temporal sulcus

We used functional MRI with an event-related design to dissociate the brain activation in the fusiform gyrus (FG) and posterior superior temporal sulcus (STS) for multiple face and gaze orientations. The event-related design allowed for concurrent behavioral analysis, which revealed a significant effect of both head and gaze orientation on the speed of gaze processing, with the face and gaze forward condition showing the fastest reaction times. In conjunction with this behavioral finding, the FG responded with the greatest activation to face and gaze forward, perhaps reflecting the unambiguous social salience of congruent face and gaze directed toward the viewer. Random effects analysis showed greater activation in both the FG and posterior STS when the subjects viewed a direct face compared to an angled face, regardless of gaze direction. Additionally, the FG showed greater activation for forward gaze compared to angled gaze, but only when the face was forward. Together, these findings suggest that head orientation has a significant effect on gaze processing and these effects are manifest not only in the STS, but also the FG.

Electrophysiological Correlates of Age and Gender Perception on Human Faces

In a previous experiment using scalp event-related potentials (ERPs), we have described the neuroelectric activities associated with the processing of gender information on human faces (Mouchetant-Rostaing, Giard, Bentin, Aguera, & Pernier, 2000). Here we extend this study by examining the processing of age on faces using a similar experimental paradigm, and we compare age and gender processing. In one session, faces were of the same gender (women) and of one age range (young or old), to reduce gender and age processing. In a second session, faces of young and old women were randomly intermixed but age was irrelevant for the task, hence, age discrimination, if any, was assumed to be incidental. In the third and fourth sessions, faces had to be explicitly categorized according to their age or gender, respectively (intentional discrimination). Neither age nor gender processing affected the occipito-temporal N170 component often associated with the detection of physiognomic features and global structural encoding of faces. Rather, the three age and gender discrimination conditions induced similar fronto-central activities around 145–185 msec. In our previous experiment, this ERP pattern was also found for implicit and explicit categorization of gender from faces but not in a control condition manipulating hand stimuli (Mouchetant-Rostaing, Giard, Bentin, et al., 2000). Whatever their exact nature, these 145–185 msec effects therefore suggest, first, that similar mechanisms could be engaged in age and gender perception, and second, that age and gender may be implicitly processed irrespective of their relevance to the task, through somewhat specialized mechanisms. Additional ERP effects were found at early latencies (45–90 msec) in all three discrimination conditions, and around 200–400 msec during explicit age and gender discrimination. These effects have been previously found in control conditions manipulating nonfacial stimuli and may therefore be related to more general categorization processes.

Stimulus repetition and hemodynamic response refractoriness in event-related fMRI

We investigated the extent of hemodynamic recovery following the paired presentation of either identical or different faces at two different inter-stimulus intervals (ISI). Signal recovery was consistently better at an ISI of 6 sec compared to 3 sec. Significantly less signal recovery was associated with identical faces compared to different faces in bilateral mid-fusiform and right prefrontal regions but not in the calcarine and posterior fusiform regions. Repetition suppression effects contributed significantly to incomplete signal recovery in a region-specific manner. Simulations using empirically derived data suggest that experiments with shorter ISI (average 4.5-6.0 sec) are as sensitive as experiments with intermediate ISI (average 9 sec) in detecting response differences if experimental duration is equivalent. However, designs using intermediate ISI may be more appropriate if the expected difference in responses is small and if the number of suitable stimuli is limited.

The human temporal lobe integrates facial form and motion: evidence from fMRI and ERP studies

Physiological studies in humans and monkeys indicate that the posterior temporal cortex is active when viewing the movements of others. Here we tested the premise that this region integrates form and motion information by presenting both natural and line-drawn displays of moving faces and motion controls where motion was continuously presented in the same part of the visual field. The cortex in and near the STS and on the fusiform gyrus (FG) responded to both types of face stimuli, but not to the controls, in a functional magnetic resonance imaging study in 10 normal subjects. The response in the STS to both types of facial motion was equal in magnitude, whereas in the FG the natural image of the face produced a significantly greater response than that of the line-drawn face. In a subsequent recording session, the electrical activity of the brain was recorded in the same subjects to the same activation task. Significantly larger event-related potentials (ERPs) to both types of moving faces were observed over the posterior temporal scalp compared to the motion controls at around 200 ms postmotion onset. Taken together, these data suggest that regions of temporal cortex actively integrate form and motion information-a process largely independent of low-level visual processes such as changes in local luminance and contrast.

The spatiotemporal dynamics of the face inversion effect: a magneto- and electro-encephalographic study

The neurophysiological basis of the face inversion effect was studied with magneto- and electro-encephalography in 10 normal subjects. Spatiotemporal analyses using dipole modeling was performed on combined evoked magneto and electro-encephalography data to hemifield presentation of upright and inverted faces and objects. Inferior temporal cortex, i.e. fusiform gyrus, and lateral temporal cortex near the superior temporal sulcus were activated simultaneously, but independently, at 140-200 ms post-stimulus to upright and inverted unfamiliar faces. Right hemisphere inferior temporal cortex and lateral temporal cortex were active in all subjects, and in the left hemisphere in half the subjects. Latencies to inverted relative to upright faces were longer in the right hemisphere, and shorter in the left hemisphere. For right hemifield stimulation ipsilateral activation delay was around 18-19 ms for both upright and inverted faces and was calculated from all 10 subjects. For left hemifield stimulation, and the data from 7 of 10 subjects, it was 22 and 29 ms to upright and inverted faces, respectively. In sum, the methods used in this study did not identify clear differences in anatomical location of activated regions to upright and inverted faces. We believe, however, that the differences in processing upright versus inverted faces are attributable to temporal processing differences rather than to processing of information by different brain regions.

Enhanced Extrastriate Activation during Observation of Distorted Finger Postures

Hand and finger postures of other people are important body language cues that strongly contribute to the observer's decision about the person's intentions, thoughts, and attentional state. We compared neuromagnetic cortical activation elicited by color images of natural and distorted finger postures. The distorted postures contained computer-deformed joint angles and thereby easily caught the observer's attention. From about 260 msec onwards, extrastriate occipital areas of both hemispheres were activated more strongly by distorted than natural finger postures. We interpret this result as an early topdown effect of emotional valence on the processing of unusual hand shapes in the extrastriate visual cortex.

Face-selective activation in a congenital prosopagnosic subject

Congenital prosopagnosia is a severe impairment in face identification manifested from early childhood in the absence of any evident brain lesion. In this study, we used fMRI to compare the brain activity elicited by faces in a congenital prosopagnosic subject (YT) relative to a control group of 12 subjects in an attempt to shed more light on the nature of the brain mechanisms subserving face identification. The face-related activation pattern of YT in the ventral occipito-temporal cortex was similar to that observed in the control group on several parameters: anatomical location, activation profiles, and hemispheric laterality. In addition, using a modified vase-face illusion, we found that YT's brain activity in the face-related regions manifested global grouping processes. However, subtle differences in the degree of selectivity between objects and faces were observed in the lateral occipital cortex. These data suggest that face-related activation in the ventral occipito-temporal cortex, although necessary, might not be sufficient by itself for normal face identification.

The processing of emotional facial expression is gated by spatial attention: evidence from event-related brain potentials

To investigate whether the processing of faces and emotional facial expression can be modulated by spatial attention, ERPs were recorded in response to stimulus arrays containing two faces and two non-face stimuli (houses). In separate trials, attention was focused on the face pair or on the house pair, and facial expression was either fearful or neutral. When faces were attended, a greater frontal positivity in response to arrays containing fearful faces was obtained, starting about 100 ms after stimulus onset. In contrast, with faces located outside the attentional focus, this emotional expression effect was completely eliminated. This differential result demonstrates for the first time a strong attentional gating of brain processes involved in the analysis of emotional facial expression. It is argued that while an initial detection of emotionally relevant events mediated by the amygdala may occur pre-attentively, subsequent stages of emotional processing require focal spatial attention. The face-sensitive N170 component was unaffected by emotional facial expression, but N170 amplitudes were enhanced when faces were attended, suggesting that spatial attention can modulate the structural encoding of faces.

Mapping of the neuronal networks of human cortical brain functions

OBJECTIVE

The principles and methodology of event-related fMRI, electromagnetic source imaging and intracranial evoked potentials will be described along with some examples of the mapping of the neuronal networks of human cortical brain functions with the use of these techniques.

INTRODUCTION

Functional brain mapping using PET or fMRI has provided clues on the functioning brain and notably on the functional neuroanatomy of cognitive functions. These mapping possibilities can be used to delineate in an individual patient the brain areas subserving a cerebral function that might be compromised by a surgery in a nearby location, or to target a functional neurosurgical procedure.

BACKGROUND

Brain functions and notably "higher brain functions" are served by a complex network of interrelating brain regions. Deeper insights into the functioning of a neuronal network can be gained by adding dynamic, i.e. temporal, information to the functional maps. This will demonstrate the orchestration of the activation of the different brain areas constituting the network, which gives clues to the information processing and therefore to the functioning of the different modules of the network. In order to track the flow of information and the sequential activation of the different brain regions constituting the network, brain activity has to be recorded at the speed of transfer of activation from one neuronal population to the other. The temporal resolution needed to achieve this is not in the range of traditional subtractive or comparative PET or fMRI techniques.

NEW DEVELOPMENTS

Novel fMRI methods that record haemodynamic signal changes after single events (event-related fMRI) are now able to determine sequential neural processing by distinguishing the relative onset-time of activity between different areas. The temporal resolution of event-related (ER) fMRI is sufficient to detect changes of mental activity within the order of several hundreds of milliseconds. This allows the exploration of a broad range of cognitive functions. Nevertheless, this technique is currently not rapid enough to observe the transient coordinations and oscillations of neuronal activities occurring across certain cortical areas during the performance of cognitive tasks. The temporal resolution needed for that is within the order of tens or a few milliseconds and is only accessible by EEG or MEG that allow true real-time measurements of the neuronal activity elicited by a stimulus. Surface recordings of multichannel EEG or MEG combined with novel electromagnetic source localisation algorithms allow a relatively precise estimation of the activated areas. A more direct localisation of electric activity is achieved by intracranial recordings in patients having implanted electrodes for diagnostic reasons. In these cases, a high temporal and spatial resolution is achieved but with a limited sampling of brain regions.

CONCLUSION

Although the temporal resolution of ER fMRI is due to improve, the temporal measures provided by EEG, MEG or intracranial event-related potentials (ERPs) are absolute, which remains a unique feature of these techniques. Therefore, ER fMRI and electromagnetic source imaging are complementary. The maps obtained with ER fMRI may be refined by electromagnetic ERPs that provide further insights into the temporal coordination or orchestration between the cortical areas already detected by ER fMRI and constituting a neuronal network, and ER fMRI can be used to precisely locate the areas coarsely situated and delineated by electromagnetic source imaging. Thus, the combination of ER fMRI and electromagnetic ERPs is essential in order to produce a mapping method with a millimetre spatial resolution and a millisecond temporal resolution. Future applications should combine these techniques to localise precisely and non-invasively relevant sensory, motor and cognitive processes in order to adequately tailor any brain surgery.

Brain activation evoked by perception of gaze shifts: the influence of context

Prior studies from our laboratory [Journal of Neuroscience 18 (1998) 2188; Cognitive Neuropsychology 17 (2000) 221] have demonstrated that discrete regions of the superior temporal sulcus (STS) are activated when a subject views a face in which the eyes shift their gaze. Here we investigated the degree to which activity in the STS and other brain regions is modulated by the context of the perceived gaze shift; that is, when the shift correctly or incorrectly acquires a visual target. Fifteen subjects participated in an event-related functional magnetic resonance imaging experiment in which they viewed an animated face that remained present throughout each run. On each of 21 trials within each run, a small checkerboard appeared and flickered at one of six locations within the character's visual field. On "correct" trials, the character shifted its gaze towards the checkerboard after a delay of 1 or 3s. On "incorrect" trials, the character shifted its gaze towards empty space after the same delays. On "no shift" trials, the character's eyes did not move. Significantly larger hemodynamic responses (HDR) were evoked by gaze shifts compared to no gaze shifts in primarily right hemisphere STS. The gaze-evoked HDR was significantly delayed in peak amplitude for 3s compared to 1s shifts. For 1s shifts, a strong effect of context was observed in which errors evoked a HDR with extended duration. Although this study focused upon STS, similar effects were also observed in the intraparietal sulcus and fusiform gyrus.

Face versus non-face object perception and the 'other-race' effect: a spatio-temporal event-related potential study

OBJECTIVE

To investigate a modulation of the N170 face-sensitive component related to the perception of other-race (OR) and same-race (SR) faces, as well as differences in face and non-face object processing, by combining different methods of event-related potential (ERP) signal analysis.

METHODS

Sixty-two channel ERPs were recorded in 12 Caucasian subjects presented with Caucasian and Asian faces along with non-face objects. Surface data were submitted to classical waveforms and ERP map topography analysis. Underlying brain sources were estimated with two inverse solutions (BESA and LORETA).

RESULTS

The N170 face component was identical for both race faces. This component and its topography revealed a face specific pattern regardless of race. However, in this time period OR faces evoked significantly stronger medial occipital activity than SR faces. Moreover, in terms of maps, at around 170 ms face-specific activity significantly preceded non-face object activity by 25 ms. These ERP maps were followed by similar activation patterns across conditions around 190-300 ms, most likely reflecting the activation of visually derived semantic information.

CONCLUSIONS

The N170 was not sensitive to the race of the faces. However, a possible pre-attentive process associated to the relatively stronger unfamiliarity for OR faces was found in medial occipital area. Moreover, our data provide further information on the time-course of face and non-face object processing.

Covert recognition and the neural system for face processing

In this viewpoint, we discuss the new evidence on covert face recognition in prosopagnosia presented by Bobes et al. (2003, this issue) and by Sperber and Spinnler (2003, this issue). Contrary to earlier hypotheses, both papers agree that covert and overt face recognition are based on the same mechanism. In line with this suggestion, an analysis of reported cases with prosopagnosia indicates that a degree of successful encoding of facial representations is a prerequisite for covert recognition to occur. While we agree with this general conclusion as far as Bobes et al.'s and Sperber and Spinnler's data are concerned, we also discuss evidence for a dissociation between different measures of covert recognition. Specifically, studies in patients with Capgras delusion and patients with prosopagnosia suggest that skin conductance and behavioural indexes of covert face recognition are mediated by partially different mechanisms. We also discuss implications of the new data for models of normal face recognition that have been successful in simulating covert recognition phenomena (e.g., Young and Burton, 1999, and O'Reilly et al., 1999). Finally, in reviewing recent neurophysiological and brain imaging evidence concerning the neural system for face processing, we argue that the relationship between ERP components (specifically, N170, N250r, and N400) and different cognitive processes in face recognition is beginning to emerge.

Non-spatial attentional effects on P1

OBJECTIVE

It has been suggested that P1, the earliest endogenous visual potential, is influenced primarily by spatial location. However, we have found that attention to non-spatial visual features can affect both the latency and amplitude of this component.

METHODS

A series of studies are reviewed, starting with 4 using simple geometric forms, and either serial presentation of single stimuli or presentation of stimulus arrays followed by two studies using natural complex images.

RESULTS

With simple stimuli, latency and amplitude effects are seen on the P1, but differ among the paradigms, depending on the demands of the task. The data further showed a facilitation effect and that binding occurs in parallel with single feature processing. For complex stimuli we found P1 shorter to faces than inverted faces, eyes or non-face stimuli, and larger to animal than non-animal pictures. The above effects were present in children as well as in adults.

CONCLUSIONS

These findings demonstrate that very early stages of processing can be modified by top-down attentional influences across a range of ages and experimental paradigms, concordant with visual processing models showing very rapid and dispersed activation with feedback at early cortical levels.

Category-sensitive excitatory and inhibitory processes in human extrastriate cortex

Single-cell recordings from the temporal lobe of monkeys viewing stimuli show that cells may be highly selective, responding for example to particular objects such as faces. However, stimulus-selective cells may be inhibited by nonpreferred stimuli. Can such inhibitory mechanisms be detected in human visual cortex? In previous recordings from the surface of human ventral extrastriate cortex, we found that specific categories of stimuli such as faces and words generate category-specific negative event-related potentials (ERPs) with a peak latency of about 200 ms (N200). Laminar recordings in animal cortex suggest that the human N200 reflects excitatory depolarizing potentials in apical dendrites of pyramidal cells. In this study we found that, at about half of word-specific N200 sites, faces generated a positive ERP (P200); conversely, at about half of face-specific sites, words generated P200s. The electrogenesis of N200 implies that P200 ERPs reflect hyperpolarizing inhibition of apical dendrites. These recordings, together with the prior animal recordings, provide strong circumstantial evidence that in human cortex populations of cells responsive to one stimulus category (such as faces) inhibit cells responsive to another category (such as words), probably by a type of lateral inhibition. Of the stimulus categories studied quantitatively, face-specific cells are maximally inhibited by words and vice versa, but other categories of stimuli may generate smaller P200s, suggesting that inhibition of category-specific cells by nonpreferred stimuli is a general feature of human extrastriate cortex involved in object recognition.

Event-related brain potential evidence for a response of inferior temporal cortex to familiar face repetitions

We investigated immediate repetition effects in the recognition of famous faces by recording event-related brain potentials (ERPs) and reaction times (RTs). Participants recognized celebrities' faces that were preceded by either the same picture, a different picture of the same celebrity, or a different famous face. Face repetition caused two distinct ERP modulations. Repetitions elicited a strong modulation of an N250 component ( approximately 200-300 ms) over inferior temporal regions. The N250 modulation showed a degree of image specificity in that it was still significant for repetitions across different pictures, though reduced in amplitude. ERPs to repeated faces were also more positive than those to unprimed faces at parietal sites from 400 to 600 ms, but these later effects were largely independent of whether the same or a different image of the celebrity had served as prime. Finally, no influence of repetition was observed for the N170 component. Dipole source modelling suggested that the N250 repetition effect (N250r) may originate from the fusiform gyrus. In contrast, source localisation of the N170 implicated a significantly more posterior location, corresponding to a lateral occipitotemporal source outside the fusiform gyrus.

Affective judgments of faces modulate early activity (approximately 160 ms) within the fusiform gyri

Functional neuroimaging studies have implicated the fusiform gyri (FG) in structural encoding of faces, while event-related potential (ERP) and magnetoencephalography studies have shown that such encoding occurs approximately 170 ms poststimulus. Behavioral and functional neuroimaging studies suggest that processes involved in face recognition may be strongly modulated by socially relevant information conveyed by faces. To test the hypothesis that affective information indeed modulates early stages of face processing, ERPs were recorded to individually assessed liked, neutral, and disliked faces and checkerboard-reversal stimuli. At the N170 latency, the cortical three-dimensional distribution of current density was computed in stereotactic space using a tomographic source localization technique. Mean activity was extracted from the FG, defined by structure-probability maps, and a meta-cluster delineated by the coordinates of the voxel with the strongest face-sensitive response from five published functional magnetic resonance imaging studies. In the FG, approximately 160 ms poststimulus, liked faces elicited stronger activation than disliked and neutral faces and checkerboard-reversal stimuli. Further, confirming recent results, affect-modulated brain electrical activity started very early in the human brain (approximately 112 ms). These findings suggest that affective features conveyed by faces modulate structural face encoding. Behavioral results from an independent study revealed that the stimuli were not biased toward particular facial expressions and confirmed that liked faces were rated as more attractive. Increased FG activation for liked faces may thus be interpreted as reflecting enhanced attention due to their saliency.

Gaze direction affects face perception in humans

We recorded event-related potentials (ERP) in response to images of faces with a straight gaze (straight eyes) and eyes averted (averted eyes). Peak latencies of ERP components showed no significant change between straight eyes and averted eyes, but amplitude for averted eyes, particularly when averted to the right, was significantly larger than that for straight eyes at the lateral temporal electrode of the right hemisphere. Single-unit recordings in monkeys and neuroimaging studies in humans have revealed activity in the lateral temporal region, mainly the superior temporal sulcus, and a clinical study demonstrated the importance of the right hemisphere when viewing gaze direction. This is the first systematic neurophysiological report to confirm these findings using ERP.

Structural and functional magnetic resonance imaging of autism

Magnetic resonance imaging (MRI) of brain structures and function is uniquely suited to characterize the range of neuroanatomical and physiological changes that characterize the autism phenotype as it develops over time. In this review, we examine the scientific literature in MRI as applied to autism and related areas, over approximately the last decade, discussing findings which have emerged, methodological stumbling blocks which have been identified, and potential future directions. Structural MRI studies have recently begun to elucidate the neurodevelopmental underpinnings and brain-behavior relationships in autism while fMRI studies, building on the wealth of data in normal individuals, are beginning to characterize the underlying neuropsychological deficits of the disorder. Together, these two methods combine to contribute to a better understanding of the neural basis and brain phenotype of this disorder.

Dynamics of visual feature analysis and object-level processing in face versus letter-string perception

Neurones in the human inferior occipitotemporal cortex respond to specific categories of images, such as numbers, letters and faces, within 150-200 ms. Here we identify the locus in time when stimulus-specific analysis emerges by comparing the dynamics of face and letter-string perception in the same 10 individuals. An ideal paradigm was provided by our previous study on letter-strings, in which noise-masking of stimuli revealed putative visual feature processing at 100 ms around the occipital midline followed by letter-string-specific activation at 150 ms in the left inferior occipitotemporal cortex. In the present study, noise-masking of cartoon-like faces revealed that the response at 100 ms increased linearly with the visual complexity of the images, a result that was similar for faces and letter-strings. By 150 ms, faces and letter-strings had entered their own stimulus-specific processing routes in the inferior occipitotemporal cortex, with identical timing and large spatial overlap. However, letter-string analysis lateralized to the left hemisphere, whereas face processing occurred more bilaterally or with right-hemisphere preponderance. The inferior occipitotemporal activations at approximately 150 ms, which take place after the visual feature analysis at approximately 100 ms, are likely to represent a general object-level analysis stage that acts as a rapid gateway to higher cognitive processing.

Neural basis of prosopagnosia: an fMRI study

Brain imaging research has identified at least two regions in human extrastriate cortex responding selectively to faces. One of these is located in the mid-fusiform gyrus (FFA), the other in the inferior occipital gyrus (IOG). We studied activation of these areas using fMRI in three individuals with severely impaired face recognition (one pure developmental and two childhood prosopagnosics). None of the subjects showed the normal pattern of higher fMRI activity to faces than to objects in the FFA and IOG or elsewhere. Moreover, in two of the patients, faces and objects produced similar activations in the regions corresponding to where the FFA and IOG are found in normal subjects. Our study casts light on the important role of FFA and IOG in the network of areas involved in face recognition, and indicates limits of brain plasticity.

Neural systems for recognizing emotion

Recognition of emotion draws on a distributed set of structures that include the occipitotemporal neocortex, amygdala, orbitofrontal cortex and right frontoparietal cortices. Recognition of fear may draw especially on the amygdala and the detection of disgust may rely on the insula and basal ganglia. Two important mechanisms for recognition of emotions are the construction of a simulation of the observed emotion in the perceiver, and the modulation of sensory cortices via top-down influences.

Priming visual face-processing mechanisms: electrophysiological evidence

Accumulated evidence from electrophysiology and neuroimaging suggests that face perception involves extrastriate visual mechanisms specialized in processing physiognomic features and building a perceptual representation that is categorically distinct and can be identified by face-recognition units. In the present experiment, we recorded event-related brain potentials in order to explore possible contextual influences on the activity of this perceptual mechanism. Subjects werefirst exposed to pairs of small shapes, which did not elicit any face-specific brain activity. The same stimuli, however, elicitedface-specific brain activity after subjects saw them embedded in schematic faces, which probably primed the subjects to interpret the shapes as schematic eyes. No face-specific activity was observed when objects rather than faces were used to form the context. We conclude that the activity of face-specific extrastriate perceptual mechanisms can be modulated by contextual constraints that determine the significance of the visual input.

Testing cognitive models of visual attention with fMRI and MEG

Neuroimaging techniques can be used not only to identify the neural substrates of attention, but also to test cognitive theories of attention. Here we consider four classic questions in the psychology of visual attention: (i) Are some 'special' classes of stimuli (e.g. faces) immune to attentional modulation?; (ii) What are the information units on which attention operates?; (iii) How early in stimulus processing are attentional effects observed?; and (iv) Are common mechanisms involved in different modes of attentional selection (e.g. spatial and non-spatial selection)? We describe studies from our laboratory that illustrate the ways in which fMRI and MEG can provide key evidence in answering these questions. A central methodological theme in many of our fMRI studies is the use of analyses in which the activity in certain functionally-defined regions of interest (ROIs) is used to test specific cognitive hypotheses. An analogous sensor-of-interest (SOI) approach is applied to MEG. Our results include: evidence for the modulation of face representations by attention; confirmation of the independent contributions of object-based and location-based selection; evidence for modulation of face representations by non-spatial selection within the first 170 ms of processing; and implication of the intraparietal sulcus in functions general to spatial and non-spatial visual selection.