Differential sensitivity for viewpoint between familiar and unfamiliar faces in human visual cortex

Distinct patterns of neural response to faces from different races in humans and deep networks

Social categories such as the race or ethnicity of an individual are typically conveyed by the visual appearance of the face. The aim of this study was to explore how these differences in facial appearance are represented in human and artificial neural networks. First, we compared the similarity of faces from different races using a neural network trained to discriminate identity. We found that the differences between races were most evident in the fully connected layers of the network. Although these layers were also able to predict behavioural judgements of face identity from human participants, performance was biased toward White faces. Next, we measured the neural response in face-selective regions of the human brain to faces from different races in Asian and White participants. We found distinct patterns of response to faces from different races in face-selective regions. We also found that the spatial pattern of response was more consistent across participants for own-race compared to other-race faces. Together, these findings show that faces from different races elicit different patterns of response in human and artificial neural networks. These differences may underlie the ability to make categorical judgements and explain the behavioural advantage for the recognition of own-race faces.

A neural marker of the human face identity familiarity effect

Human adults associate different views of an identity much better for familiar than for unfamiliar faces. However, a robust and consistent neural index of this behavioral face identity familiarity effect (FIFE)-not found in non-human primate species-is lacking. Here we provide such a neural FIFE index, measured implicitly and with one fixation per face. Fourteen participants viewed 70 s stimulation sequences of a large set (n = 40) of widely variable natural images of a face identity at a rate of 6 images/second (6 Hz). Different face identities appeared every 5th image (1.2 Hz). In a sequence, face images were either familiar (i.e., famous) or unfamiliar, participants performing a non-periodic task unrelated to face recognition. The face identity recognition response identified at 1.2 Hz over occipital-temporal regions in the frequency-domain electroencephalogram was 3.4 times larger for familiar than unfamiliar faces. The neural response to familiar faces-which emerged at about 180 ms following face onset-was significant in each individual but a case of prosopdysgnosia. Besides potential clinical and forensic applications to implicitly measure one's knowledge of a face identity, these findings open new perspectives to clarify the neurofunctional source of the FIFE and understand the nature of human face identity recognition.

Single neuron responses underlying face recognition in the human midfusiform face-selective cortex

Faces are critical for social interactions and their recognition constitutes one of the most important and challenging functions of the human brain. While neurons responding selectively to faces have been recorded for decades in the monkey brain, face-selective neural activations have been reported with neuroimaging primarily in the human midfusiform gyrus. Yet, the cellular mechanisms producing selective responses to faces in this hominoid neuroanatomical structure remain unknown. Here we report single neuron recordings performed in 5 human subjects (1 male, 4 females) implanted with intracerebral microelectrodes in the face-selective midfusiform gyrus, while they viewed pictures of familiar and unknown faces and places. We observed similar responses to faces and places at the single cell level, but a significantly higher number of neurons responding to faces, thus offering a mechanistic account for the face-selective activations observed in this region. Although individual neurons did not respond preferentially to familiar faces, a population level analysis could consistently determine whether or not the faces (but not the places) were familiar, only about 50 ms after the initial recognition of the stimuli as faces. These results provide insights into the neural mechanisms of face processing in the human brain.

Naturalistic face adaptation: How we adapt to freckles fast and sustainably

While sunbathing, our skin becomes susceptible to quite remarkable changes in visual appearance, that is, freckles appear or increase in intensity-most obviously on the face. Research on face adaptation repeatedly showed that the inspection of manipulated versions of faces (so-called adaptor faces) leads to robust and sustainable changes in the perception of subsequently presented faces. Therefore, during the adaptation phase of the present study, participants saw faces with either strongly increased or decreased intensities of freckles. After a 5-minute break, during the test phase, participants had to identify the veridical (non-manipulated) face out of two faces (a slightly manipulated face combined with a non-manipulated face). Results showed strong adaptation effects to increased and decreased levels of freckles. We conclude that updating facial representations in memory is relatively fast, and these representation updates seem to sustain over a certain time span (at least 5 minutes). Face-specificity of our effects will be discussed. The results align with our everyday experience that the appearance of freckles in spring is a salient change in a familiar face; however, we seem to not register these changes after a few exposures due to a loss of information quality.

Intracerebral Electrophysiological Recordings to Understand the Neural Basis of Human Face Recognition

Understanding how the human brain recognizes faces is a primary scientific goal in cognitive neuroscience. Given the limitations of the monkey model of human face recognition, a key approach in this endeavor is the recording of electrophysiological activity with electrodes implanted inside the brain of human epileptic patients. However, this approach faces a number of challenges that must be overcome for meaningful scientific knowledge to emerge. Here we synthesize a 10 year research program combining the recording of intracerebral activity (StereoElectroEncephaloGraphy, SEEG) in the ventral occipito-temporal cortex (VOTC) of large samples of participants and fast periodic visual stimulation (FPVS), to objectively define, quantify, and characterize the neural basis of human face recognition. These large-scale studies reconcile the wide distribution of neural face recognition activity with its (right) hemispheric and regional specialization and extend face-selectivity to anterior regions of the VOTC, including the ventral anterior temporal lobe (VATL) typically affected by magnetic susceptibility artifacts in functional magnetic resonance imaging (fMRI). Clear spatial dissociations in category-selectivity between faces and other meaningful stimuli such as landmarks (houses, medial VOTC regions) or written words (left lateralized VOTC) are found, confirming and extending neuroimaging observations while supporting the validity of the clinical population tested to inform about normal brain function. The recognition of face identity - arguably the ultimate form of recognition for the human brain - beyond mere differences in physical features is essentially supported by selective populations of neurons in the right inferior occipital gyrus and the lateral portion of the middle and anterior fusiform gyrus. In addition, low-frequency and high-frequency broadband iEEG signals of face recognition appear to be largely concordant in the human association cortex. We conclude by outlining the challenges of this research program to understand the neural basis of human face recognition in the next 10 years.

Twenty years of investigation with the case of prosopagnosia PS to understand human face identity recognition. Part II: Neural basis

Patient PS sustained her dramatic brain injury in 1992, the same year as the first report of a neuroimaging study of human face recognition. The present paper complements the review on the functional nature of PS's prosopagnosia (part I), illustrating how her case study directly, i.e., through neuroimaging investigations of her brain structure and activity, but also indirectly, through neural studies performed on other clinical cases and neurotypical individuals, inspired and constrained neural models of human face recognition. In the dominant right hemisphere for face recognition in humans, PS's main lesion concerns (inputs to) the inferior occipital gyrus (IOG), in a region where face-selective activity is typically found in normal individuals ('Occipital Face Area', OFA). Her case study initially supported the criticality of this region for face identity recognition (FIR) and provided the impetus for transcranial magnetic stimulation (TMS), intracerebral electrical stimulation, and cortical surgery studies that have generally supported this view. Despite PS's right IOG lesion, typical face-selectivity is found anteriorly in the middle portion of the fusiform gyrus, a hominoid structure (termed the right 'Fusiform Face Area', FFA) that is widely considered to be the most important region for human face recognition. This finding led to the original proposal of direct anatomico-functional connections from early visual cortices to the FFA, bypassing the IOG/OFA (lulu), a hypothesis supported by further neuroimaging studies of PS, other neurological cases and neuro-typical individuals with original visual stimulation paradigms, data recordings and analyses. The proposal of a lack of sensitivity to face identity in PS's right FFA due to defective reentrant inputs from the IOG/FFA has also been supported by other cases, functional connectivity and cortical surgery studies. Overall, neural studies of, and based on, the case of prosopagnosia PS strongly question the hierarchical organization of the human neural face recognition system, supporting a more flexible and dynamic view of this key social brain function.

Lesion-behaviour mapping reveals multifactorial neurocognitive processes in recognition memory for unfamiliar faces

Face recognition abilities, which play a critical role in social interactions, involve face processing and identifying familiar faces, but also remembering one-off encounters with previously unfamiliar faces. Previous functional imaging and lesion studies have found evidence for temporal, frontal, and parietal contributions to episodic recognition memory for previously unfamiliar faces. However, the functional contributions of these regions remain unclear. We, therefore, conducted a systematic group analysis of this memory function using lesion-behavior mapping. 95 first-event stroke patients (53 with right- and 42 with left-hemisphere damage) in the sub-acute phase performed the Wechsler Memory Scale (WMS-III) face recognition memory subtest. We analyzed their performance relative to 75 healthy controls, using signal detection measures. To identify brain lesions specifically implicated in face recognition deficits, we used voxel-based lesion-behavior mapping (VLBM; an analysis comparing the performance of participants with and without damage affecting a given voxel). Behavioral analysis disclosed a pronounced impairment in the performance of patients with right hemisphere damage. Frontal damage was associated with an increased amount of false alarms (i.e., failed rejection of new face items) and overly liberal criterion setting, without affecting the recognition of studied faces. In contrast, parietal damage was associated with impaired recognition of studied faces, which was more pronounced in immediate than in delayed retrieval. These findings suggest the existence of multifactorial neurocognitive processes in recognition memory for unfamiliar faces.

Is left-behind a real reason for children's social cognition deficit? An fNIRS study on the effect of social interaction on left-behind preschooler's prefrontal activation

The left-behind phenomenon, caused by parent out-migration, has become a common social issue and might lead to long-term and potential risks for children in rural areas of China. It is important to investigate the effect of social interaction on prefrontal activation of left-behind children in China because of possible effects of parent out-migration on children’s social cognition. We recruited 81 rural Chinese preschoolers aged 52–76 months (mean = 64.98 ± 6.321 months) preschoolers with three different statuses of parental out-migration (including non-, partially, and completely left-behind children). Using functional Near-Infrared Spectroscopy (fNIRS), we compared behavior and brain activation and in three groups (non-, partially-, completely-left-behind children) under two different social interaction conditions (child-teacher and child-stranger situation). Results revealed that initiating joint attention (IJA) may evoke higher brain activation than responding to joint attention (RJA) in the prefrontal cortex (PFC), especially in the case of initiating joint attention with the stranger. In addition, the activation of joint attention was positively correlated with children’s language score, cognitive flexibility, and facial expression recognition. More importantly, partially-left-behind children evoked higher brain activation in the IJA condition and presented a higher language level than completely/non-left-behind children. The current study provides insight into the neural basis of left-behind children’s development and revealed for the first time that family economic level and left-behind status may contribute to the lower social cognition.

Holistic face recognition is an emergent phenomenon of spatial processing in face-selective regions

Spatial processing by receptive fields is a core property of the visual system. However, it is unknown how spatial processing in high-level regions contributes to recognition behavior. As face inversion is thought to disrupt typical holistic processing of information in faces, we mapped population receptive fields (pRFs) with upright and inverted faces in the human visual system. Here we show that in face-selective regions, but not primary visual cortex, pRFs and overall visual field coverage are smaller and shifted downward in response to face inversion. From these measurements, we successfully predict the relative behavioral detriment of face inversion at different positions in the visual field. This correspondence between neural measurements and behavior demonstrates how spatial processing in face-selective regions may enable holistic perception. These results not only show that spatial processing in high-level visual regions is dynamically used towards recognition, but also suggest a powerful approach for bridging neural computations by receptive fields to behavior.

Visual mechanisms for voice-identity recognition flexibly adjust to auditory noise level

Recognising the identity of voices is a key ingredient of communication. Visual mechanisms support this ability: recognition is better for voices previously learned with their corresponding face (compared to a control condition). This so‐called ‘face‐benefit’ is supported by the fusiform face area (FFA), a region sensitive to facial form and identity. Behavioural findings indicate that the face‐benefit increases in noisy listening conditions. The neural mechanisms for this increase are unknown. Here, using functional magnetic resonance imaging, we examined responses in face‐sensitive regions while participants recognised the identity of auditory‐only speakers (previously learned by face) in high (SNR −4 dB) and low (SNR +4 dB) levels of auditory noise. We observed a face‐benefit in both noise levels, for most participants (16 of 21). In high‐noise, the recognition of face‐learned speakers engaged the right posterior superior temporal sulcus motion‐sensitive face area (pSTS‐mFA), a region implicated in the processing of dynamic facial cues. The face‐benefit in high‐noise also correlated positively with increased functional connectivity between this region and voice‐sensitive regions in the temporal lobe in the group of 16 participants with a behavioural face‐benefit. In low‐noise, the face‐benefit was robustly associated with increased responses in the FFA and to a lesser extent the right pSTS‐mFA. The findings highlight the remarkably adaptive nature of the visual network supporting voice‐identity recognition in auditory‐only listening conditions.

Getting to Know Someone: Familiarity, Person Recognition, and Identification in the Human Brain

In our everyday life, we continuously get to know people, dominantly through their faces. Several neuroscientific experiments showed that familiarization changes the behavioral processing and underlying neural representation of faces of others. Here, we propose a model of the process of how we actually get to know someone. First, the purely visual familiarization of unfamiliar faces occurs. Second, the accumulation of associated, nonsensory information refines person representation, and finally, one reaches a stage where the effortless identification of very well-known persons occurs. We offer here an overview of neuroimaging studies, first evaluating how and in what ways the processing of unfamiliar and familiar faces differs and, second, by analyzing the fMRI adaptation and multivariate pattern analysis results we estimate where identity-specific representation is found in the brain. The available neuroimaging data suggest that different aspects of the information emerge gradually as one gets more and more familiar with a person within the same network. We propose a novel model of familiarity and identity processing, where the differential activation of long-term memory and emotion processing areas is essential for correct identification.

Predicting Performances on Processing and Memorizing East Asian Faces from Brain Activities in Face-Selective Regions: A Neurocomputational Approach

For more than two decades, a network of face-selective brain regions has been identified as the core system for face processing, including occipital face area (OFA), fusiform face area (FFA), and posterior region of superior temporal sulcus (pSTS). Moreover, recent studies have suggested that the ventral route of face processing and memory should end at the anterior temporal lobes (i.e., vATLs), which may play an important role bridging face perception and face memory. It is not entirely clear, however, the extent to which neural activities in these face-selective regions can effectively predict behavioral performance on tasks that are frequently used to investigate face processing and face memory test that requires recognition beyond variation in pose and lighting, especially when non-Caucasian East Asian faces are involved. To address these questions, we first identified during a functional scan the core face network by asking participants to perform a one-back task, while viewing either static images or dynamic videos. Dynamic localizers were effective in identifying regions of interest (ROIs) in the core face-processing system. We then correlated the brain activities of core ROIs with performances on face-processing tasks (component, configural, and composite) and face memory test (Taiwanese Face Memory Test, TFMT) and found evidence for limited predictability. We next adopted an multi-voxel pattern analysis (MVPA) approach to further explore the predictability of face-selective brain regions on TFMT performance and found evidence suggesting that a basic visual processing area such as calcarine and an area for structural face processing such as OFA may play an even greater role in memorizing faces. Implications regarding how differences in processing demands between behavioral and neuroimaging tasks and cultural specificity in face-processing and memory strategies among participants may have contributed to the findings reported here are discussed.

The 4D Space-Time Dimensions of Facial Perception

Facial information is a powerful channel for human-to-human communication. Characteristically, faces can be defined as biological objects that are four-dimensional (4D) patterns, whereby they have concurrently a spatial structure and surface as well as temporal dynamics. The spatial characteristics of facial objects contain a volume and surface in three dimensions (3D), namely breadth, height and importantly, depth. The temporal properties of facial objects are defined by how a 3D facial structure and surface evolves dynamically over time; where time is referred to as the fourth dimension (4D). Our entire perception of another’s face, whether it be social, affective or cognitive perceptions, is therefore built on a combination of 3D and 4D visual cues. Counterintuitively, over the past few decades of experimental research in psychology, facial stimuli have largely been captured, reproduced and presented to participants with two dimensions (2D), while remaining largely static. The following review aims to advance and update facial researchers, on the recent revolution in computer-generated, realistic 4D facial models produced from real-life human subjects. We delve in-depth to summarize recent studies which have utilized facial stimuli that possess 3D structural and surface cues (geometry, surface and depth) and 4D temporal cues (3D structure + dynamic viewpoint and movement). In sum, we have found that higher-order perceptions such as identity, gender, ethnicity, emotion and personality, are critically influenced by 4D characteristics. In future, it is recommended that facial stimuli incorporate the 4D space-time perspective with the proposed time-resolved methods.

The occipital face area is causally involved in identity-related visual-semantic associations

Faces are processed in a network of areas within regions of the ventral visual stream. However, familiar faces typically are characterized by additional associated information, such as episodic memories or semantic biographical information as well. The acquisition of such non-sensory, identity-specific knowledge plays a crucial role in our ability to recognize and identify someone we know. The occipital face area (OFA), an early part of the core face-processing network, is recently found to be involved in the formation of identity-specific memory traces but it is currently unclear if this role is limited to unimodal visual information. The current experiments used transcranial magnetic stimulation (TMS) to test whether the OFA is involved in the association of a face with identity-specific semantic information, such as the name or job title of a person. We applied an identity-learning task where unfamiliar faces were presented together with a name and a job title in the first encoding phase. Simultaneously, TMS pulses were applied either to the left or right OFA or to Cz, as a control. In the subsequent retrieval phase, the previously seen faces were presented either with two names or with two job titles and the task of the participants was to select the semantic information previously learned. We found that the stimulation of the right or left OFA reduced subsequent retrieval performance for the face-associated job titles. This suggests a causal role of the OFA in the association of faces and related semantic information. Furthermore, in contrast to prior findings, we did not observe hemispherical differences of the TMS intervention, suggesting a similar role of the left and right OFAs in the formation of the visual-semantic associations. Our results suggest the necessity to reconsider the hierarchical face-perception models and support the distributed and recurrent models.

Convergent evolution of face spaces across human face-selective neuronal groups and deep convolutional networks

The discovery that deep convolutional neural networks (DCNNs) achieve human performance in realistic tasks offers fresh opportunities for linking neuronal tuning properties to such tasks. Here we show that the face-space geometry, revealed through pair-wise activation similarities of face-selective neuronal groups recorded intracranially in 33 patients, significantly matches that of a DCNN having human-level face recognition capabilities. This convergent evolution of pattern similarities across biological and artificial networks highlights the significance of face-space geometry in face perception. Furthermore, the nature of the neuronal to DCNN match suggests a role of human face areas in pictorial aspects of face perception. First, the match was confined to intermediate DCNN layers. Second, presenting identity-preserving image manipulations to the DCNN abolished its correlation to neuronal responses. Finally, DCNN units matching human neuronal group tuning displayed view-point selective receptive fields. Our results demonstrate the importance of face-space geometry in the pictorial aspects of human face perception.

Deep convolutional neural networks (DCNNs) are able to identify faces on par with humans. Here, the authors record neuronal activity from higher visual areas in humans and show that face-selective responses in the brain show similarity to those in the intermediate layers of the DCNN.

Symmetrical Viewpoint Representations in Face-Selective Regions Convey an Advantage in the Perception and Recognition of Faces

Learning new identities is crucial for effective social interaction. A critical aspect of this process is the integration of different images from the same face into a view-invariant representation that can be used for recognition. The representation of symmetrical viewpoints has been proposed to be a key computational step in achieving view-invariance. The aim of this study was to determine whether the representation of symmetrical viewpoints in face-selective regions is directly linked to the perception and recognition of face identity. In Experiment 1, we measured fMRI responses while male and female human participants viewed images of real faces from different viewpoints (-90, -45, 0, 45, and 90° from full-face view). Within the face regions, patterns of neural response to symmetrical views (-45 and 45° or -90 and 90°) were more similar than responses to nonsymmetrical views in the fusiform face area and superior temporal sulcus, but not in the occipital face area. In Experiment 2, participants made perceptual similarity judgements to pairs of face images. Images with symmetrical viewpoints were reported as being more similar than nonsymmetric views. In Experiment 3, we asked whether symmetrical views also convey an advantage when learning new faces. We found that recognition was best when participants were tested with novel face images that were symmetrical to the learning viewpoint. Critically, the pattern of perceptual similarity and recognition across different viewpoints predicted the pattern of neural response in face-selective regions. Together, our results provide support for the functional value of symmetry as an intermediate step in generating view-invariant representations.SIGNIFICANCE STATEMENT The recognition of identity from faces is crucial for successful social interactions. A critical step in this process is the integration of different views into a unified, view-invariant representation. The representation of symmetrical views (e.g., left profile and right profile) has been proposed as an important intermediate step in computing view-invariant representations. We found view symmetric representations were specific to some face-selective regions, but not others. We also show that these neural representations influence the perception of faces. Symmetric views were perceived to be more similar and were recognized more accurately than nonsymmetric views. Moreover, the perception and recognition of faces at different viewpoints predicted patterns of response in those face regions with view symmetric representations.

A Robust Neural Index of High Face Familiarity

Humans are remarkably accurate at recognizing familiar faces, whereas their ability to recognize, or even match, unfamiliar faces is much poorer. However, previous research has failed to identify neural correlates of this striking behavioral difference. Here, we found a clear difference in brain potentials elicited by highly familiar faces versus unfamiliar faces. This effect starts 200 ms after stimulus onset and reaches its maximum at 400 to 600 ms. This sustained-familiarity effect was substantially larger than previous candidates for a neural familiarity marker and was detected in almost all participants, representing a reliable index of high familiarity. Whereas its scalp distribution was consistent with a generator in the ventral visual pathway, its modulation by repetition and degree of familiarity suggests an integration of affective and visual information.

Distinct neural processes for the perception of familiar versus unfamiliar faces along the visual hierarchy revealed by EEG

Humans recognize faces with ease, despite the complexity of the task and of the visual system which underlies it. Different spatial regions, including both the core and extended face processing networks, and distinct temporal stages of processing have been implicated in face recognition, but there is ongoing controversy regarding the extent to which the mechanisms for recognizing a familiar face differ from those for an unfamiliar face. Here, we used electroencephalogram (EEG) and flicker SSVEP, a high signal-to-noise approach, and searchlight decoding methods to elucidate the mechanisms mediating the processing of familiar and unfamiliar faces in the time domain. Familiar and unfamiliar faces were presented periodically at 15 Hz, 6 Hz and 3.75 Hz either upright or inverted in separate blocks, with the rationale that faster frequencies require shorter processing times per image and tap into fundamentally different levels of visual processing. The 15 Hz trials, likely to reflect early visual processing, exhibited enhanced neural responses for familiar over unfamiliar face trials, but only when the faces were upright. In contrast, decoding methods revealed similar classification accuracies for upright and inverted faces for both familiar and unfamiliar faces. For the 6 Hz frequency, familiar faces had lower amplitude responses than unfamiliar faces, and decoding familiarity was more accurate for upright compared with inverted faces. Finally, the 3.75 Hz frequency revealed no main effects of familiarity, but decoding showed significant correlations with behavioral ratings of face familiarity, suggesting that activity evoked by this slow presentation frequency reflected higher-level, cognitive aspects of familiarity processing. This three-way dissociation between frequencies reveals that fundamentally different stages of the visual hierarchy are modulated by face familiarity. The combination of experimental and analytical approaches used here represent a novel method for elucidating spatio-temporal characteristics within the visual system.

Age-related increase of image-invariance in the fusiform face area

•

Image invariance in the FFA increases from age seven to adulthood.

•

These results are confirmed by two independent ROI analyses.

•

Adaptation in the FFA relates to the ability to recognize a face in multiple images.

Face recognition undergoes prolonged development from childhood to adulthood, thereby raising the question which neural underpinnings are driving this development. Here, we address the development of the neural foundation of the ability to recognize a face across naturally varying images. Fourteen children (ages, 7–10) and 14 adults (ages, 20–23) watched images of either the same or different faces in a functional magnetic resonance imaging adaptation paradigm. The same face was either presented in exact image repetitions or in varying images. Additionally, a subset of participants completed a behavioral task, in which they decided if the face in consecutively presented images belonged to the same person. Results revealed age-related increases in neural sensitivity to face identity in the fusiform face area. Importantly, ventral temporal face-selective regions exhibited more image-invariance – as indicated by stronger adaptation for different images of the same person – in adults compared to children. Crucially, the amount of adaptation to face identity across varying images was correlated with the ability to recognize individual faces in different images. These results suggest that the increase of image-invariance in face-selective regions might be related to the development of face recognition skills.

Temporal Contiguity Training Influences Behavioral and Neural Measures of Viewpoint Tolerance

Humans can often recognize faces across viewpoints despite the large changes in low-level image properties a shift in viewpoint introduces. We present a behavioral and an fMRI adaptation experiment to investigate whether this viewpoint tolerance is reflected in the neural visual system and whether it can be manipulated through training. Participants saw training sequences of face images creating the appearance of a rotating head. Half of the sequences showed faces undergoing veridical changes in appearance across the rotation (non-morph condition). The other half were non-veridical: during rotation, the face simultaneously morphed into another face. This procedure should successfully associate frontal face views with side views of the same or a different identity, and, according to the temporal contiguity hypothesis, thus enhance viewpoint tolerance in the non-morph condition and/or break tolerance in the morph condition. Performance on the same/different task in the behavioral experiment (N = 20) was affected by training. There was a significant interaction between training (associated/not associated) and identity (same/different), mostly reflecting a higher confusion of different identities when they were associated during training. In the fMRI study (N = 20), fMRI adaptation effects were found for same-viewpoint images of untrained faces, but no adaptation for untrained faces was present across viewpoints. Only trained faces which were not morphed during training elicited a slight adaptation across viewpoints in face-selective regions. However, both in the behavioral and in the neural data the effects were small and weak from a statistical point of view. Overall, we conclude that the findings are not inconsistent with the proposal that temporal contiguity can influence viewpoint tolerance, with more evidence for tolerance when faces are not morphed during training.

The Functional Neuroanatomy of Human Face Perception

Face perception is critical for normal social functioning and is mediated by a network of regions in the ventral visual stream. In this review, we describe recent neuroimaging findings regarding the macro- and microscopic anatomical features of the ventral face network, the characteristics of white matter connections, and basic computations performed by population receptive fields within face-selective regions composing this network. We emphasize the importance of the neural tissue properties and white matter connections of each region, as these anatomical properties may be tightly linked to the functional characteristics of the ventral face network. We end by considering how empirical investigations of the neural architecture of the face network may inform the development of computational models and shed light on how computations in the face network enable efficient face perception.

Event-related potentials reveal the development of stable face representations from natural variability

Natural variability between instances of unfamiliar faces can make it difficult to reconcile two images as the same person. Yet for familiar faces, effortless recognition occurs even with considerable variability between images. To explore how stable face representations develop, we employed incidental learning in the form of a face sorting task. In each trial, multiple images of two facial identities were sorted into two corresponding piles. Following the sort, participants showed evidence of having learnt the faces performing more accurately on a matching task with seen than with unseen identities. Furthermore, ventral temporal event-related potentials were more negative in the N250 time range for previously seen than for previously unseen identities. These effects appear to demonstrate some degree of abstraction, rather than simple picture learning, as the neurophysiological and behavioural effects were observed with novel images of the previously seen identities. The results provide evidence of the development of facial representations, allowing a window onto natural mechanisms of face learning.

Development of Neural Sensitivity to Face Identity Correlates with Perceptual Discriminability

Face perception is subserved by a series of face-selective regions in the human ventral stream, which undergo prolonged development from childhood to adulthood. However, it is unknown how neural development of these regions relates to the development of face-perception abilities. Here, we used functional magnetic resonance imaging (fMRI) to measure brain responses of ventral occipitotemporal regions in children (ages, 5-12 years) and adults (ages, 19-34 years) when they viewed faces that parametrically varied in dissimilarity. Since similar faces generate lower responses than dissimilar faces due to fMRI adaptation, this design objectively evaluates neural sensitivity to face identity across development. Additionally, a subset of subjects participated in a behavioral experiment to assess perceptual discriminability of face identity. Our data reveal three main findings: (1) neural sensitivity to face identity increases with age in face-selective but not object-selective regions; (2) the amplitude of responses to faces increases with age in both face-selective and object-selective regions; and (3) perceptual discriminability of face identity is correlated with the neural sensitivity to face identity of face-selective regions. In contrast, perceptual discriminability is not correlated with the amplitude of response in face-selective regions or of responses of object-selective regions. These data suggest that developmental increases in neural sensitivity to face identity in face-selective regions improve perceptual discriminability of faces. Our findings significantly advance the understanding of the neural mechanisms of development of face perception and open new avenues for using fMRI adaptation to study the neural development of high-level visual and cognitive functions more broadly.

SIGNIFICANCE STATEMENT

Face perception, which is critical for daily social interactions, develops from childhood to adulthood. However, it is unknown what developmental changes in the brain lead to improved performance. Using fMRI in children and adults, we find that from childhood to adulthood, neural sensitivity to changes in face identity increases in face-selective regions. Critically, subjects' perceptual discriminability among faces is linked to neural sensitivity: participants with higher neural sensitivity in face-selective regions demonstrate higher perceptual discriminability. Thus, our results suggest that developmental increases in face-selective regions' sensitivity to face identity improve perceptual discrimination of faces. These findings significantly advance understanding of the neural mechanisms underlying the development of face perception and have important implications for assessing both typical and atypical development.

Contributions of individual face features to face discrimination

Faces are highly complex stimuli that contain a host of information. Such complexity poses the following questions: (a) do observers exhibit preferences for specific information? (b) how does sensitivity to individual face parts compare? These questions were addressed by quantifying sensitivity to different face features. Discrimination thresholds were determined for synthetic faces under the following conditions: (i) 'full face': all face features visible; (ii) 'isolated feature': single feature presented in isolation; (iii) 'embedded feature': all features visible, but only one feature modified. Mean threshold elevations for isolated features, relative to full-faces, were 0.84x, 1.08, 2.12, 3.34, 4.07 and 4.47 for head-shape, hairline, nose, mouth, eyes and eyebrows respectively. Hence, when two full faces can be discriminated at threshold, the difference between the eyes is about four times less than what is required when discriminating between isolated eyes. In all cases, sensitivity was higher when features were presented in isolation than when they were embedded within a face context (threshold elevations of 0.94x, 1.74, 2.67, 2.90, 5.94 and 9.94). This reveals a specific pattern of sensitivity to face information. Observers are between two and four times more sensitive to external than internal features. The pattern for internal features (higher sensitivity for the nose, compared to mouth, eyes and eyebrows) is consistent with lower sensitivity for those parts affected by facial dynamics (e.g. facial expressions). That isolated features are easier to discriminate than embedded features supports a holistic face processing mechanism which impedes extraction of information about individual features from full faces.

An image-invariant neural response to familiar faces in the human medial temporal lobe

The ability to recognise familiar faces with ease across different viewing conditions contrasts with the inherent difficulty in the perception of unfamiliar faces across similar image manipulations. Models of face processing suggest that this difference is based on the neural representation for familiar faces being more invariant to changes in the image, than it is for unfamiliar faces. Here, we used an fMR-adaptation paradigm to investigate neural correlates of image-invariant face recognition in face-selective regions of the human brain. Participants viewed faces presented in a blocked design. Each block contained different images of the same identity or different images from different identities. Faces in each block were either familiar or unfamiliar to the participants. First, we defined face-selective regions by comparing the response to faces with the response to scenes and scrambled faces. Next, we asked whether any of these face-selective regions showed image-invariant adaptation to the identity of a face. The core face-selective regions showed image-invariant adaptation to familiar and unfamiliar faces. However, there was no difference in the adaptation to familiar compared to unfamiliar faces. In contrast, image-invariant adaptation for familiar faces, but not for unfamiliar faces, was found in face-selective regions of the medial temporal lobe (MTL). Taken together, our results suggest that the marked differences in the perception of familiar and unfamiliar faces may depend critically on neural processes in the medial temporal lobe.

Socially Important Faces Are Processed Preferentially to Other Familiar and Unfamiliar Faces in a Priming Task across a Range of Viewpoints

Using a priming paradigm, we investigate whether socially important faces are processed preferentially compared to other familiar and unfamiliar faces, and whether any such effects are affected by changes in viewpoint. Participants were primed with frontal images of personally familiar, famous or unfamiliar faces, and responded to target images of congruent or incongruent identity, presented in frontal, three quarter or profile views. We report that participants responded significantly faster to socially important faces (a friend's face) compared to other highly familiar (famous) faces or unfamiliar faces. Crucially, responses to famous and unfamiliar faces did not differ. This suggests that, when presented in the context of a socially important stimulus, socially unimportant familiar faces (famous faces) are treated in a similar manner to unfamiliar faces. This effect was not tied to viewpoint, and priming did not affect socially important face processing differently to other faces.

MEG Adaptation Resolves the Spatiotemporal Characteristics of Face-Sensitive Brain Responses

An unresolved goal in face perception is to identify brain areas involved in face processing and simultaneously understand the timing of their involvement. Currently, high spatial resolution imaging techniques identify the fusiform gyrus as subserving processing of invariant face features relating to identity. High temporal resolution imaging techniques localize an early latency evoked component-the N/M170-as having a major generator in the fusiform region; however, this evoked component is not believed to be associated with the processing of identity. To resolve this, we used novel magnetoencephalographic beamformer analyses to localize cortical regions in humans spatially with trial-by-trial activity that differentiated faces and objects and to interrogate their functional sensitivity by analyzing the effects of stimulus repetition. This demonstrated a temporal sequence of processing that provides category-level and then item-level invariance. The right fusiform gyrus showed adaptation to faces (not objects) at ∼150 ms after stimulus onset regardless of face identity; however, at the later latency of ∼200-300 ms, this area showed greater adaptation to repeated identity faces than to novel identities. This is consistent with an involvement of the fusiform region in both early and midlatency face-processing operations, with only the latter showing sensitivity to invariant face features relating to identity.

SIGNIFICANCE STATEMENT

Neuroimaging techniques with high spatial-resolution have identified brain structures that are reliably activated when viewing faces and techniques with high temporal resolution have identified the time-varying temporal signature of the brain's response to faces. However, until now, colocalizing face-specific mechanisms in both time and space has proven notoriously difficult. Here, we used novel magnetoencephalographic analysis techniques to spatially localize cortical regions with trial-by-trial temporal activity that differentiates between faces and objects and to interrogate their functional sensitivity by analyzing effects of stimulus repetition on the time-locked signal. These analyses confirm a role for the right fusiform region in early to midlatency responses consistent with face identity processing and convincingly deliver upon magnetoencephalography's promise to resolve brain signals in time and space simultaneously.

Visual adaptation provides objective electrophysiological evidence of facial identity discrimination

Discrimination of facial identities is a fundamental function of the human brain that is challenging to examine with macroscopic measurements of neural activity, such as those obtained with functional magnetic resonance imaging (fMRI) and electroencephalography (EEG). Although visual adaptation or repetition suppression (RS) stimulation paradigms have been successfully implemented to this end with such recording techniques, objective evidence of an identity-specific discrimination response due to adaptation at the level of the visual representation is lacking. Here, we addressed this issue with fast periodic visual stimulation (FPVS) and EEG recording combined with a symmetry/asymmetry adaptation paradigm. Adaptation to one facial identity is induced through repeated presentation of that identity at a rate of 6 images per second (6 Hz) over 10 sec. Subsequently, this identity is presented in alternation with another facial identity (i.e., its anti-face, both faces being equidistant from an average face), producing an identity repetition rate of 3 Hz over a 20 sec testing sequence. A clear EEG response at 3 Hz is observed over the right occipito-temporal (ROT) cortex, indexing discrimination between the two facial identities in the absence of an explicit behavioral discrimination measure. This face identity discrimination occurs immediately after adaptation and disappears rapidly within 20 sec. Importantly, this 3 Hz response is not observed in a control condition without the single-identity 10 sec adaptation period. These results indicate that visual adaptation to a given facial identity produces an objective (i.e., at a pre-defined stimulation frequency) electrophysiological index of visual discrimination between that identity and another, and provides a unique behavior-free quantification of the effect of visual adaptation.

ACC and IPL networks in the perception of the faces of parents during selective tasks

The perception of the faces of parents is different from that of other faces. Mother's face perception is associated with childbearing and parenting and has specific cognitive processing properties. Yet, the underlying mechanism remains unknown. We investigated the neural network that is involved in the perception of parent's face based upon parental face selective tasks that were completed by 23 normal adults. Resting state functional magnetic resonance imaging (fMRI) and a task block design with fixation cross sign control fMRI were used. The faces activated the right inferior frontal gyrus, the left and right middle frontal gyri, and the left and right inferior parietal lobule. In addition, the parental faces task resulted in more activation than the control tasks in the left and right anterior cingulate, left middle (superior) temporal gyrus, right and left inferior frontal gyrus. The anterior cingulate cortex (ACC) and the inferior parietal lobe (IPL) are central nodes of the brain network, while the parental face perception network includes the anterior cingulate cortex and superior temporal gyrus (ACC-STG)/parahippocampal network as well as the inferior frontal and inferior parietal lobule (IFG-IPL) network. The network of father's face perception involves the left inferior parietal lobule and left middle frontal gyrus/right middle frontal gyrus/right inferior frontal gyrus, while mother's face perception involves the right inferior parietal lobe and frontal network (MFG.L/IFG.R). The experiments showed that the ACC-STG/parahippocampal network and IFG-IPL network are neural networks used in parental face perception, and the ACC and IPL are central nodes in the network. The neural pathway of parental face perception is similar to the perception of a stranger's face, as both include the STG, but is different in that the perception of the stranger's face involves a connected STG and IPL as a specific neural pathway.

A Revised Neural Framework for Face Processing

Face perception relies on computations carried out in face-selective cortical areas. These areas have been intensively investigated for two decades, and this work has been guided by an influential neural model suggested by Haxby and colleagues in 2000. Here, we review new findings about face-selective areas that suggest the need for modifications and additions to the Haxby model. We suggest a revised framework based on (a) evidence for multiple routes from early visual areas into the face-processing system, (b) information about the temporal characteristics of these areas, (c) indications that the fusiform face area contributes to the perception of changeable aspects of faces, (d) the greatly elevated responses to dynamic compared with static faces in dorsal face-selective brain areas, and (e) the identification of three new anterior face-selective areas. Together, these findings lead us to suggest that face perception depends on two separate pathways: a ventral stream that represents form information and a dorsal stream driven by motion and form information.

Face recognition in schizophrenia disorder: A comprehensive review of behavioral, neuroimaging and neurophysiological studies

Facial emotion processing has been extensively studied in schizophrenia patients while general face processing has received less attention. The already published reviews do not address the current scientific literature in a complete manner. Therefore, here we tried to answer some questions that remain to be clarified, particularly: are the non-emotional aspects of facial processing in fact impaired in schizophrenia patients? At the behavioral level, our key conclusions are that visual perception deficit in schizophrenia patients: are not specific to faces; are most often present when the cognitive (e.g. attention) and perceptual demands of the tasks are important; and seems to worsen with the illness chronification. Although, currently evidence suggests impaired second order configural processing, more studies are necessary to determine whether or not holistic processing is impaired in schizophrenia patients. Neural and neurophysiological evidence suggests impaired earlier levels of visual processing, which might involve the deficits in interaction of the magnocellular and parvocellular pathways impacting on further processing. These deficits seem to be present even before the disorder out-set. Although evidence suggests that this deficit may be not specific to faces, further evidence on this question is necessary, in particularly more ecological studies including context and body processing.

Successful decoding of famous faces in the fusiform face area

What are the neural mechanisms of face recognition? It is believed that the network of face-selective areas, which spans the occipital, temporal, and frontal cortices, is important in face recognition. A number of previous studies indeed reported that face identity could be discriminated based on patterns of multivoxel activity in the fusiform face area and the anterior temporal lobe. However, given the difficulty in localizing the face-selective area in the anterior temporal lobe, its role in face recognition is still unknown. Furthermore, previous studies limited their analysis to occipito-temporal regions without testing identity decoding in more anterior face-selective regions, such as the amygdala and prefrontal cortex. In the current high-resolution functional Magnetic Resonance Imaging study, we systematically examined the decoding of the identity of famous faces in the temporo-frontal network of face-selective and adjacent non-face-selective regions. A special focus has been put on the face-area in the anterior temporal lobe, which was reliably localized using an optimized scanning protocol. We found that face-identity could be discriminated above chance level only in the fusiform face area. Our results corroborate the role of the fusiform face area in face recognition. Future studies are needed to further explore the role of the more recently discovered anterior face-selective areas in face recognition.

Spatiotemporal changes in neural response patterns to faces varying in visual familiarity

Increasing experience with a previously unfamiliar face improves human ability to recognize it in challenging and novel viewing conditions. Differential neural responses to familiar versus unfamiliar faces in multiple regions of the ventral-temporal and parietal cortex have been reported in previous work, but with limited attention to how behavioral and neural measures change with increasing familiarity. We examined changes in the spatial and temporal characteristics of neural response patterns elicited by faces that vary in their degree of visual familiarity. First, we developed a behavioral paradigm to familiarize participants to low-, medium-, and high-levels of familiarity with faces. Recognition of novel, naturalistic images of the learned individuals improved with increasing familiarity with faces. Next, a new set of participants learned faces using the behavioral paradigm, outside the fMRI scanner, and subsequently viewed blocks of whole-body images of the learned and novel people, inside the scanner. We found that the face-selective FFA and OFA, and a combination of the ventral-temporal areas (e.g., fusiform gyrus) and parietal areas (e.g., precuneus) contained patterns useful for classifying highly familiar versus unfamiliar faces. Classification along the temporal-sequence of the face blocks revealed an early separation of neural patterns elicited in response to highly familiar versus unfamiliar faces in the FFA and OFA, but not in other regions of interest. This indicates the potential for a rapid assessment of the "known versus unknown" status of faces in core face-selective regions of the brain. The present study provides a first look at the perceptual and neural correlates underlying experience gains with faces as they become familiar.

Predictive codes of familiarity and context during the perceptual learning of facial identities

Face recognition is a key component of successful social behaviour. However, the computational processes that underpin perceptual learning and recognition as faces transition from unfamiliar to familiar are poorly understood. In predictive coding, learning occurs through prediction errors that update stimulus familiarity, but recognition is a function of both stimulus and contextual familiarity. Here we show that behavioural responses on a two-option face recognition task can be predicted by the level of contextual and facial familiarity in a computational model derived from predictive-coding principles. Using fMRI, we show that activity in the superior temporal sulcus varies with the contextual familiarity in the model, whereas activity in the fusiform face area covaries with the prediction error parameter that updated facial familiarity. Our results characterize the key computations underpinning the perceptual learning of faces, highlighting that the functional properties of face-processing areas conform to the principles of predictive coding.

The neural mechanisms for the recognition of face identity in humans

Every day we encounter dozens of people, and in order to interact with them appropriately we need to recognize their identity. The face is a crucial source of information to recognize a person’s identity. However, recognizing the identity of a face is challenging because it requires distinguishing between very similar images (e.g., the front views of two different faces) while categorizing very different images (e.g., a front view and a profile) as the same person. Neuroimaging has the whole-brain coverage needed to investigate where representations of face identity are encoded, but it is limited in terms of spatial and temporal resolution. In this article, we review recent neuroimaging research that attempted to investigate the representation of face identity, the challenges it faces, and the proposed solutions, to conclude that given the current state of the evidence the right anterior temporal lobe is the most promising candidate region for the representation of face identity.

Beyond the FFA: The role of the ventral anterior temporal lobes in face processing

Extensive research has supported the existence of a specialized face-processing network that is distinct from the visual processing areas used for general object recognition. The majority of this work has been aimed at characterizing the response properties of the fusiform face area (FFA) and the occipital face area (OFA), which together are thought to constitute the core network of brain areas responsible for facial identification. Although accruing evidence has shown that face-selective patches in the ventral anterior temporal lobes (vATLs) are interconnected with the FFA and OFA, and that they play a role in facial identification, the relative contribution of these brain areas to the core face-processing network has remained unarticulated. Here we review recent research critically implicating the vATLs in face perception and memory. We propose that current models of face processing should be revised such that the ventral anterior temporal lobes serve a centralized role in the visual face-processing network. We speculate that a hierarchically organized system of face processing areas extends bilaterally from the inferior occipital gyri to the vATLs, with facial representations becoming increasingly complex and abstracted from low-level perceptual features as they move forward along this network. The anterior temporal face areas may serve as the apex of this hierarchy, instantiating the final stages of face recognition. We further argue that the anterior temporal face areas are ideally suited to serve as an interface between face perception and face memory, linking perceptual representations of individual identity with person-specific semantic knowledge.