Neural correlates of shape and surface reflectance information in individual faces

The Neural Dynamics of Face Ensemble and Central Face Processing

Extensive work has investigated the neural processing of single faces, including the role of shape and surface properties. However, much less is known about the neural basis of face ensemble perception (e.g., simultaneously viewing several faces in a crowd). Importantly, the contribution of shape and surface properties have not been elucidated in face ensemble processing. Furthermore, how single central faces are processed within the context of an ensemble remains unclear. Here, we probe the neural dynamics of ensemble representation using pattern analyses as applied to electrophysiology data in healthy adults (seven males, nine females). Our investigation relies on a unique set of stimuli, depicting different facial identities, which vary parametrically and independently along their shape and surface properties. These stimuli were organized into ensemble displays consisting of six surround faces arranged in a circle around one central face. Overall, our results indicate that both shape and surface properties play a significant role in face ensemble encoding, with the latter demonstrating a more pronounced contribution. Importantly, we find that the neural processing of the center face precedes that of the surround faces in an ensemble. Further, the temporal profile of center face decoding is similar to that of single faces, while those of single faces and face ensembles diverge extensively from each other. Thus, our work capitalizes on a new center-surround paradigm to elucidate the neural dynamics of ensemble processing and the information that underpins it. Critically, our results serve to bridge the study of single and ensemble face perception.

Face processing in the temporal lobe

Face perception is a socially important but complex process with many stages and many facets. There is substantial evidence from many sources that it involves a large extent of the temporal lobe, from the ventral occipitotemporal cortex and superior temporal sulci to anterior temporal regions. While early human neuroimaging work suggested a core face network consisting of the occipital face area, fusiform face area, and posterior superior temporal sulcus, studies in both humans and monkeys show a system of face patches stretching from posterior to anterior in both the superior temporal sulcus and inferotemporal cortex. Sophisticated techniques such as fMRI adaptation have shown that these face-activated regions show responses that have many of the attributes of human face processing. Lesions of some of these regions in humans lead to variants of prosopagnosia, the inability to recognize the identity of a face. Lesion, imaging, and electrophysiologic data all suggest that there is a segregation between identity and expression processing, though some suggest this may be better characterized as a distinction between static and dynamic facial information.

The role of stimulus-based cues and conceptual information in processing facial expressions of emotion

Face shape and surface textures are two important cues that aid in the perception of facial expressions of emotion. Additionally, this perception is also influenced by high-level emotion concepts. Across two studies, we use representational similarity analysis to investigate the relative roles of shape, surface, and conceptual information in the perception, categorisation, and neural representation of facial expressions. In Study 1, 50 participants completed a perceptual task designed to measure the perceptual similarity of expression pairs, and a categorical task designed to measure the confusability between expression pairs when assigning emotion labels to a face. We used representational similarity analysis and constructed three models of the similarities between emotions using distinct information. Two models were based on stimulus-based cues (face shapes and surface textures) and one model was based on emotion concepts. Using multiple linear regression, we found that behaviour during both tasks was related with the similarity of emotion concepts. The model based on face shapes was more related with behaviour in the perceptual task than in the categorical, and the model based on surface textures was more related with behaviour in the categorical than the perceptual task. In Study 2, 30 participants viewed facial expressions while undergoing fMRI, allowing for the measurement of brain representational geometries of facial expressions of emotion in three core face-responsive regions (the Fusiform Face Area, Occipital Face Area, and Superior Temporal Sulcus), and a region involved in theory of mind (Medial Prefrontal Cortex). Across all four regions, the representational distances between facial expression pairs were related to the similarities of emotion concepts, but not to either of the stimulus-based cues. Together, these results highlight the important top-down influence of high-level emotion concepts both in behavioural tasks and in the neural representation of facial expressions.

The composite face effect is robust against perceptual misfit

The composite face effect—the failure of selective attention toward a target face half—is frequently used to study mechanisms of feature integration in faces. Here we studied how this effect depends on the perceptual fit between attended and unattended halves. We used composite faces that were rated by trained observers as either a seamless fit (i.e., close to a natural and homogeneous face) or as a deliberately bad quality of fit (i.e., unnatural, strongly segregated face halves). In addition, composites created by combining face halves randomly were tested. The composite face effect was measured as the alignment × congruency interaction (Gauthier and Bukach Cognition, 103, 322–330 2007), but also with alternative data analysis procedures (Rossion and Boremanse Journal of Vision, 8, 1–13 2008). We found strong but identical composite effects in all fit conditions. Fit quality neither increased the composite face effect nor was it attenuated by bad or random fit quality. The implications for a Gestalt account of holistic face processing are discussed.

The Face of Image Reconstruction: Progress, Pitfalls, Prospects

Recent research has demonstrated that neural and behavioral data acquired in response to viewing face images can be used to reconstruct the images themselves. However, the theoretical implications, promises, and challenges of this direction of research remain unclear. We evaluate the potential of this research for elucidating the visual representations underlying face recognition. Specifically, we outline complementary and converging accounts of the visual content, the representational structure, and the neural dynamics of face processing. We illustrate how this research addresses fundamental questions in the study of normal and impaired face recognition, and how image reconstruction provides a powerful framework for uncovering face representations, for unifying multiple types of empirical data, and for facilitating both theoretical and methodological progress.

A Flexible Neural Representation of Faces in the Human Brain

An important question in human face perception research is to understand whether the neural representation of faces is dynamically modulated by context. In particular, although there is a plethora of neuroimaging literature that has probed the neural representation of faces, few studies have investigated what low-level structural and textural facial features parametrically drive neural responses to faces and whether the representation of these features is modulated by the task. To answer these questions, we employed 2 task instructions when participants viewed the same faces. We first identified brain regions that parametrically encoded high-level social traits such as perceived facial trustworthiness and dominance, and we showed that these brain regions were modulated by task instructions. We then employed a data-driven computational face model with parametrically generated faces and identified brain regions that encoded low-level variation in the faces (shape and skin texture) that drove neural responses. We further analyzed the evolution of the neural feature vectors along the visual processing stream and visualized and explained these feature vectors. Together, our results showed a flexible neural representation of faces for both low-level features and high-level social traits in the human brain.

Contributions of shape and reflectance information to social judgments from faces

Face perception is based on both shape and reflectance information. However, we know little about the relative contribution of these kinds of information to social judgments of faces. In Experiment 1, we generated faces using validated computational models of attractiveness, competence, dominance, extroversion, and trustworthiness. Faces were manipulated orthogonally on five levels of shape and reflectance for each model. Both kinds of information had linear and additive effects on participants' social judgments. Shape information was more predictive of dominance, extroversion, and trustworthiness judgments, whereas reflectance information was more predictive of competence judgments. In Experiment 2, to test whether the amount of visual information alters the relative contribution of shape and reflectance information, we presented faces - varied on attractiveness, competence, and dominance - for five different durations (33-500 ms). For all judgments, the linear effect of both shape and reflectance increased as duration increased. Importantly, the relative contribution did not change across durations. These findings show that that the judged dimension is critical for which kind of information is weighted more heavily in judgments and that the relative contribution of shape and reflectance is stable across the amount of visual information available.

Amygdala responds to direct gaze in real but not in computer-generated faces

Computer-generated (CG) faces are an important visual interface for human-computer interaction in social contexts. Here we investigated whether the human brain processes emotion and gaze similarly in real and carefully matched CG faces. Real faces evoked greater responses in the fusiform face area than CG faces, particularly for fearful expressions. Emotional (angry and fearful) facial expressions evoked similar activations in the amygdala in real and CG faces. Direct as compared with averted gaze elicited greater fMRI responses in the amygdala regardless of facial expression but only for real and not for CG faces. We observed an interaction effect between gaze and emotion (i.e., the shared signal effect) in the right posterior temporal sulcus and other regions, but not in the amygdala, and we found no evidence for different shared signal effects in real and CG faces. Taken together, the present findings highlight similarities (emotional processing in the amygdala) and differences (overall processing in the fusiform face area, gaze processing in the amygdala) in the neural processing of real and CG faces.

Diagnosing Prosopagnosia: The Utility of Visual Noise in the Cambridge Face Recognition Test

Adding visual noise to facial images has been used to increase reliance on configural processing. Whether this enhances the ability of tests to diagnose prosopagnosia is not known. We examined 15 subjects with developmental prosopagnosia, 13 subjects with acquired prosopagnosia, and 38 control subjects with the Cambridge Face Memory Test. We compared their performance on the second phase, without visual noise, and on the third phase, which adds visual noise. We analyzed the results with signal detection theory methods. The performance of controls worsened more than did that of prosopagnosic subjects when noise was added. The second phase showed better ability to discriminate between prosopagnosic and control subjects than did the third phase. For developmental prosopagnosia, a test using only the 48 trials of the first and second phases yielded sensitivity of 88% and specificity of 91% with a criterion of 33/48 correct, performance characteristics that are similar for a criterion of 43/72 for the whole test. We conclude that a shortened Cambridge Face Memory Test without the noisy images may be a quicker yet equally effective instrument for diagnosing prosopagnosia. The theoretical advantage of noisy images is outweighed by the poorer performance of control subjects with visual noise.

3D faces are recognized more accurately and faster than 2D faces, but with similar inversion effects

Recognition of faces typically occurs via holistic processing where individual features are combined to provide an overall facial representation. However, when faces are inverted, there is greater reliance on featural processing where faces are recognized based on their individual features. These findings are based on a substantial number of studies using 2-dimensional (2D) faces and it is unknown whether these results can be extended to 3-dimensional (3D) faces, which have more depth information that is absent in the typical 2D stimuli used in face recognition literature. The current study used the face inversion paradigm as a means to investigate how holistic and featural processing are differentially influenced by 2D and 3D faces. Twenty-five participants completed a delayed face-matching task consisting of upright and inverted faces that were presented as both 2D and 3D stereoscopic images. Recognition accuracy was significantly higher for 3D upright faces compared to 2D upright faces, providing support that the enriched visual information in 3D stereoscopic images facilitates holistic processing that is essential for the recognition of upright faces. Typical face inversion effects were also obtained, regardless of whether the faces were presented in 2D or 3D. Moreover, recognition performances for 2D inverted and 3D inverted faces did not differ. Taken together, these results demonstrated that 3D stereoscopic effects influence face recognition during holistic processing but not during featural processing. Our findings therefore provide a novel perspective that furthers our understanding of face recognition mechanisms, shedding light on how the integration of stereoscopic information in 3D faces influences face recognition processes.

Visual adaptation provides objective electrophysiological evidence of facial identity discrimination

Discrimination of facial identities is a fundamental function of the human brain that is challenging to examine with macroscopic measurements of neural activity, such as those obtained with functional magnetic resonance imaging (fMRI) and electroencephalography (EEG). Although visual adaptation or repetition suppression (RS) stimulation paradigms have been successfully implemented to this end with such recording techniques, objective evidence of an identity-specific discrimination response due to adaptation at the level of the visual representation is lacking. Here, we addressed this issue with fast periodic visual stimulation (FPVS) and EEG recording combined with a symmetry/asymmetry adaptation paradigm. Adaptation to one facial identity is induced through repeated presentation of that identity at a rate of 6 images per second (6 Hz) over 10 sec. Subsequently, this identity is presented in alternation with another facial identity (i.e., its anti-face, both faces being equidistant from an average face), producing an identity repetition rate of 3 Hz over a 20 sec testing sequence. A clear EEG response at 3 Hz is observed over the right occipito-temporal (ROT) cortex, indexing discrimination between the two facial identities in the absence of an explicit behavioral discrimination measure. This face identity discrimination occurs immediately after adaptation and disappears rapidly within 20 sec. Importantly, this 3 Hz response is not observed in a control condition without the single-identity 10 sec adaptation period. These results indicate that visual adaptation to a given facial identity produces an objective (i.e., at a pre-defined stimulation frequency) electrophysiological index of visual discrimination between that identity and another, and provides a unique behavior-free quantification of the effect of visual adaptation.

Global Shape Information Increases but Color Information Decreases the Composite Face Effect

The separation of visual shape and surface information may be useful for understanding holistic face perception—that is, the perception of a face as a single unit (Jiang, Blanz, & Rossion, 2011, Visual Cognition, 19, 1003–1034). A widely used measure of holistic face perception is the composite face effect (CFE), in which identical top face halves appear different when aligned with bottom face halves from different identities. In the present study the influences of global face shape (ie contour of the face) and color information on the CFE are investigated, with the hypothesis that global face shape supports but color impairs holistic face perception as measured in this paradigm. In experiment 1 the CFE is significantly increased when face stimuli possess natural global shape information than when cropped to a generic (ie oval) global shape; this effect is not found when the stimuli are presented inverted. In experiment 2 the CFE is significantly decreased when face stimuli are presented with color information than when presented in grayscale. These findings indicate that grayscale stimuli maintaining natural global face shape information provide the most adept measure of holistic face perception in the behavioral composite face paradigm. More generally, they show that reducing different types of information diagnostic for individual face perception can have opposite effects on the CFE, illustrating the functional dissociation between shape and surface information in face perception.

Facial color processing in the face-selective regions: an fMRI study

Facial color is important information for social communication as it provides important clues to recognize a person's emotion and health condition. Our previous EEG study suggested that N170 at the left occipito-temporal site is related to facial color processing (Nakajima et al., [2012]: Neuropsychologia 50:2499-2505). However, because of the low spatial resolution of EEG experiment, the brain region is involved in facial color processing remains controversial. In the present study, we examined the neural substrates of facial color processing using functional magnetic resonance imaging (fMRI). We measured brain activity from 25 subjects during the presentation of natural- and bluish-colored face and their scrambled images. The bilateral fusiform face (FFA) area and occipital face area (OFA) were localized by the contrast of natural-colored faces versus natural-colored scrambled images. Moreover, region of interest (ROI) analysis showed that the left FFA was sensitive to facial color, whereas the right FFA and the right and left OFA were insensitive to facial color. In combination with our previous EEG results, these data suggest that the left FFA may play an important role in facial color processing.

Brain regions involved in processing facial identity and expression are differentially selective for surface and edge information

Although different brain regions are widely considered to be involved in the recognition of facial identity and expression, it remains unclear how these regions process different properties of the visual image. Here, we ask how surface-based reflectance information and edge-based shape cues contribute to the perception and neural representation of facial identity and expression. Contrast-reversal was used to generate images in which normal contrast relationships across the surface of the image were disrupted, but edge information was preserved. In a behavioural experiment, contrast-reversal significantly attenuated judgements of facial identity, but only had a marginal effect on judgements of expression. An fMR-adaptation paradigm was then used to ask how brain regions involved in the processing of identity and expression responded to blocks comprising all normal, all contrast-reversed, or a mixture of normal and contrast-reversed faces. Adaptation in the posterior superior temporal sulcus--a region directly linked with processing facial expression--was relatively unaffected by mixing normal with contrast-reversed faces. In contrast, the response of the fusiform face area--a region linked with processing facial identity--was significantly affected by contrast-reversal. These results offer a new perspective on the reasons underlying the neural segregation of facial identity and expression in which brain regions involved in processing invariant aspects of faces, such as identity, are very sensitive to surface-based cues, whereas regions involved in processing changes in faces, such as expression, are relatively dependent on edge-based cues.

Perceptual and category processing of the Uncanny Valley hypothesis' dimension of human likeness: some methodological issues

Mori's Uncanny Valley Hypothesis^1,2 proposes that the perception of humanlike characters such as robots and, by extension, avatars (computer-generated characters) can evoke negative or positive affect (valence) depending on the object's degree of visual and behavioral realism along a dimension of human likeness (DHL) (Figure 1). But studies of affective valence of subjective responses to variously realistic non-human characters have produced inconsistent findings ^{3, 4, 5, 6}. One of a number of reasons for this is that human likeness is not perceived as the hypothesis assumes. While the DHL can be defined following Mori's description as a smooth linear change in the degree of physical humanlike similarity, subjective perception of objects along the DHL can be understood in terms of the psychological effects of categorical perception (CP) ⁷. Further behavioral and neuroimaging investigations of category processing and CP along the DHL and of the potential influence of the dimension's underlying category structure on affective experience are needed. This protocol therefore focuses on the DHL and allows examination of CP. Based on the protocol presented in the video as an example, issues surrounding the methodology in the protocol and the use in "uncanny" research of stimuli drawn from morph continua to represent the DHL are discussed in the article that accompanies the video. The use of neuroimaging and morph stimuli to represent the DHL in order to disentangle brain regions neurally responsive to physical human-like similarity from those responsive to category change and category processing is briefly illustrated.

Holistic Face Processing is Induced by Shape and Texture

There is increasing evidence that shape and texture are integral parts of face identity. However, it is less clear whether face-specific processing mechanisms are triggered by face shape alone, or if texture might play an important role. We address this question by studying mechanisms involved in holistic face processing. Face stimuli were either full-color pictures of real faces (shape and texture) or line drawings of the same faces (shape without texture). In a change detection task subjects judged whether eyes and eyebrows in two otherwise identical, sequentially presented faces were different in size or not. Afterwards, subjects had to identify the just presented face among two distractor faces (forced-choice identification task). The results obtained from the two tasks give rise to the conclusion that face identification and change detection tasks engage different processing strategies, which capture different aspects of holistic processing. Real faces were processed holistically, irrespective of task requirements, whereas line drawings were processed holistically only if face identification was required. On the basis of the data we conclude that face shape is relevant for the initial processing stage and feature binding, whereas face texture seems to be involved in processing of face configuration more specifically. Moreover, results demonstrate considerable flexibility of the face processing systems allowing for goal-directed and task-specific recall of face information.

The role of the amygdala in face perception and evaluation

Faces are one of the most significant social stimuli and the processes underlying face perception are at the intersection of cognition, affect, and motivation. Vision scientists have had a tremendous success of mapping the regions for perceptual analysis of faces in posterior cortex. Based on evidence from (a) single unit recording studies in monkeys and humans; (b) human functional localizer studies; and (c) meta-analyses of neuroimaging studies, I argue that faces automatically evoke responses not only in these regions but also in the amygdala. I also argue that (a) a key property of faces represented in the amygdala is their typicality; and (b) one of the functions of the amygdala is to bias attention to atypical faces, which are associated with higher uncertainty. This framework is consistent with a number of other amygdala findings not involving faces, suggesting a general account for the role of the amygdala in perception.

The human likeness dimension of the "uncanny valley hypothesis": behavioral and functional MRI findings

The uncanny valley hypothesis (Mori, 1970) predicts differential experience of negative and positive affect as a function of human likeness. Affective experience of humanlike robots and computer-generated characters (avatars) dominates "uncanny" research, but findings are inconsistent. Importantly, it is unknown how objects are actually perceived along the hypothesis' dimension of human likeness (DOH), defined in terms of human physical similarity. To examine whether the DOH can also be defined in terms of effects of categorical perception (CP), stimuli from morph continua with controlled differences in physical human likeness between avatar and human faces as endpoints were presented. Two behavioral studies found a sharp category boundary along the DOH and enhanced visual discrimination (i.e., CP) of fine-grained differences between pairs of faces at the category boundary. Discrimination was better for face pairs presenting category change in the human-to-avatar than avatar-to-human direction along the DOH. To investigate brain representation of physical change and category change along the DOH, an event-related functional magnetic resonance imaging study used the same stimuli in a pair-repetition priming paradigm. Bilateral mid-fusiform areas and a different right mid-fusiform area were sensitive to physical change within the human and avatar categories, respectively, whereas entirely different regions were sensitive to the human-to-avatar (caudate head, putamen, thalamus, red nucleus) and avatar-to-human (hippocampus, amygdala, mid-insula) direction of category change. These findings show that Mori's DOH definition does not reflect subjective perception of human likeness and suggest that future "uncanny" studies consider CP and the DOH's category structure in guiding experience of non-human objects.

Face categorization in visual scenes may start in a higher order area of the right fusiform gyrus: evidence from dynamic visual stimulation in neuroimaging

How a visual stimulus is initially categorized as a face by the cortical face-processing network remains largely unclear. In this study we used functional MRI to study the dynamics of face detection in visual scenes by using a paradigm in which scenes containing faces or cars are revealed progressively as they emerge from visual noise. Participants were asked to respond as soon as they detected a face or car during the noise sequence. Among the face-sensitive regions identified based on a standard localizer, a high-level face-sensitive area, the right fusiform face area (FFA), showed the earliest difference between face and car activation. Critically, differential activation in FFA was observed before differential activation in the more posteriorly located occipital face area (OFA). A whole brain analysis confirmed these findings, with a face-sensitive cluster in the right fusiform gyrus being the only cluster showing face preference before successful behavioral detection. Overall, these findings indicate that following generic low-level visual analysis, a face stimulus presented in a gradually revealed visual scene is first detected in the right middle fusiform gyrus, only after which further processing spreads to a network of cortical and subcortical face-sensitive areas (including the posteriorly located OFA). These results provide further evidence for a nonhierarchical organization of the cortical face-processing network.

The role of skin texture and facial shape in representations of age and identity

Faces have both shape and skin texture, but the relative importance of the two in face representations is unclear. Our goals were first, to determine the contribution of shape versus texture to aftereffects for facial age and identity and second, to assess whether adaptation transferred between shape and texture, suggesting integration in a single representation. In our first experiment we examined age aftereffects. We obtained young and old images of two celebrities and created hybrid images, one combining the structure of the old face with the skin texture of the young face, the other combining the young structure with the old skin texture. This allowed us to create adaptation contrasts where the two adapting faces had the same facial structure but different skin texture, and vice versa. In the second experiment, we performed a similar study but this time examining identity aftereffects between two people of a similar age. We found that both skin texture and facial shape generated significant age aftereffects, but the contribution was greater from texture than from shape. Both texture and shape also generated significant identity aftereffects, but the contribution was greater from shape than from texture. In the last experiment, we used the normal and hybrid images to determine if adaptation to one property (i.e., texture) could create aftereffects in the perception of age in the other property (i.e., shape). While there was significant within-component adaptation for texture and shape, there was no evidence of cross-component adaptation. We conclude that shape and texture contribute differently to different face representations, with texture dominating for age. The lack of cross-component adaptation transfer suggests independent encoding of shape and texture, at least for age representations.

Event-related potential and functional MRI measures of face-selectivity are highly correlated: a simultaneous ERP-fMRI investigation

A face-selective neural signal is reliably found in humans with functional MRI and event-related potential (ERP) measures, which provide complementary information about the spatial and temporal properties of the neural response. However, because most neuroimaging studies so far have studied ERP and fMRI face-selective markers separately, the relationship between them is still unknown. Here we simultaneously recorded fMRI and ERP responses to faces and chairs to examine the correlations across subjects between the magnitudes of fMRI and ERP face-selectivity measures. Findings show that the face-selective responses in the temporal lobe (i.e., fusiform gyrus--FFA) and superior temporal sulcus (fSTS), but not the face-selective response in the occipital cortex (OFA), were highly correlated with the face-selective N170 component. In contrast, the OFA was correlated with earlier ERPs at about 110 ms after stimulus-onset. Importantly, these correlations reveal a temporal dissociation between the face-selective area in the occipital lobe and face-selective areas in the temporal lobe. Despite the very different time-scale of the fMRI and EEG signals, our data show that a correlation analysis across subjects may be informative with respect to the latency in which different brain regions process information.

Personally familiar faces are perceived categorically in face-selective regions other than the fusiform face area

Neuroimaging studies of humans have provided inconsistent evidence with respect to the response properties of the fusiform face area (FFA). It has been claimed that neural populations within this region are sensitive to subtle differences between individual faces only when they are perceived as distinct identities [P. Rotshtein et al. (2005)Nature Neuroscience, 8, 107-113]. However, sensitivity to subtle changes of identity was found in previous studies using unfamiliar faces, for which categorical perception is less pronounced. Using functional magnetic resonance adaptation and morph continua of personally familiar faces, we investigated sensitivity to subtle changes between faces that were located either on the same or opposite sides of a categorical perceptual boundary. We found no evidence for categorical perception within the FFA, which exhibited reliable sensitivity to subtle changes of face identity whether these were perceived as distinct identities, or not. On the contrary, both the posterior superior temporal sulcus and prefrontal cortex exhibited categorical perception, as subtle changes between faces perceived as different identities yielded larger release from adaptation than those perceived as the same identity. These observations suggest that, whereas the FFA discriminates subtle physical changes of personally familiar faces, other regions encode faces in a categorical fashion.