An image-invariant neural response to familiar faces in the human medial temporal lobe

The anterior fusiform gyrus: The ghost in the cortical face machine

Face-selective regions in the human ventral occipito-temporal cortex (VOTC) have been defined for decades mainly with functional magnetic resonance imaging. This face-selective VOTC network is traditionally divided in a posterior 'core' system thought to subtend face perception, and regions of the anterior temporal lobe as a semantic memory component of an extended general system. In between these two putative systems lies the anterior fusiform gyrus and surrounding sulci, affected by magnetic susceptibility artifacts. Here we suggest that this methodological gap overlaps with and contributes to a conceptual gap between (visual) perception and semantic memory for faces. Filling this gap with intracerebral recordings and direct electrical stimulation reveals robust face-selectivity in the anterior fusiform gyrus and a crucial role of this region, especially in the right hemisphere, in identity recognition for both familiar and unfamiliar faces. Based on these observations, we propose an integrated theoretical framework for human face (identity) recognition according to which face-selective regions in the anterior fusiform gyrus join the dots between posterior and anterior cortical face memories.

Personal familiarity of faces, animals, objects, and scenes: Distinct perceptual and overlapping conceptual representations

While face, object, and scene recognition are often studied at a basic categorization level (e.g. "a face", "a car", "a kitchen"), we frequently recognise individual items of these categories as unique entities (e.g. "my mother", "my car", "my kitchen"). This recognition of individual identity is essential to appropriate behaviour in our world. However, relatively little is known about how we recognise individually familiar visual stimuli. Using event-related brain potentials, the present study examined whether and to what extent the underlying neural representations of personally familiar items are similar or different across different categories. In three experiments, we examined the recognition of personally highly familiar faces, animals, indoor scenes, and objects. We observed relatively distinct familiarity effects in an early time window (200-400 ms), with a clearly right-lateralized occipito-temporal scalp distribution for human faces and more bilateral and posterior distributions for other stimulus categories, presumably reflecting access to at least partly discrete visual long-term representations. In contrast, we found clearly overlapping familiarity effects in a later time window (starting 400 to 500 ms after stimulus onset), again with a mainly right occipito-temporal scalp distribution, for all stimulus categories. These later effects appear to reflect the sustained activation of conceptual properties relevant to any potential interaction. We conclude that familiarity for items from the various visual stimulus categories tested here is represented differently at the perceptual level, while relatively overlapping conceptual mechanisms allow for the preparation of impending potential interaction with the environment.

A neural marker of the human face identity familiarity effect

Human adults associate different views of an identity much better for familiar than for unfamiliar faces. However, a robust and consistent neural index of this behavioral face identity familiarity effect (FIFE)-not found in non-human primate species-is lacking. Here we provide such a neural FIFE index, measured implicitly and with one fixation per face. Fourteen participants viewed 70 s stimulation sequences of a large set (n = 40) of widely variable natural images of a face identity at a rate of 6 images/second (6 Hz). Different face identities appeared every 5th image (1.2 Hz). In a sequence, face images were either familiar (i.e., famous) or unfamiliar, participants performing a non-periodic task unrelated to face recognition. The face identity recognition response identified at 1.2 Hz over occipital-temporal regions in the frequency-domain electroencephalogram was 3.4 times larger for familiar than unfamiliar faces. The neural response to familiar faces-which emerged at about 180 ms following face onset-was significant in each individual but a case of prosopdysgnosia. Besides potential clinical and forensic applications to implicitly measure one's knowledge of a face identity, these findings open new perspectives to clarify the neurofunctional source of the FIFE and understand the nature of human face identity recognition.

Single neuron responses underlying face recognition in the human midfusiform face-selective cortex

Faces are critical for social interactions and their recognition constitutes one of the most important and challenging functions of the human brain. While neurons responding selectively to faces have been recorded for decades in the monkey brain, face-selective neural activations have been reported with neuroimaging primarily in the human midfusiform gyrus. Yet, the cellular mechanisms producing selective responses to faces in this hominoid neuroanatomical structure remain unknown. Here we report single neuron recordings performed in 5 human subjects (1 male, 4 females) implanted with intracerebral microelectrodes in the face-selective midfusiform gyrus, while they viewed pictures of familiar and unknown faces and places. We observed similar responses to faces and places at the single cell level, but a significantly higher number of neurons responding to faces, thus offering a mechanistic account for the face-selective activations observed in this region. Although individual neurons did not respond preferentially to familiar faces, a population level analysis could consistently determine whether or not the faces (but not the places) were familiar, only about 50 ms after the initial recognition of the stimuli as faces. These results provide insights into the neural mechanisms of face processing in the human brain.

Face familiarity revealed by fixational eye movements and fixation-related potentials in free viewing

Event-related potentials (ERPs) and the oculomotor inhibition (OMI) in response to visual transients are known to be sensitive to stimulus properties, attention, and expectation. We have recently found that the OMI is also sensitive to face familiarity. In natural vision, stimulation of the visual cortex is generated primarily by saccades, and it has been recently suggested that fixation-related potentials (FRPs) share similar components with the ERPs. Here, we investigated whether FRPs and microsaccade inhibition (OMI) in free viewing are sensitive to face familiarity. Observers freely watched a slideshow of seven unfamiliar and one familiar facial images presented randomly for 4-s periods, with multiple images per identity. We measured the occipital fixation-related N1 relative to the P1 magnitude as well as the associated fixation-triggered OMI. We found that the average N1-P1 was significantly smaller and the OMI was shorter for the familiar face, compared with any of the seven unfamiliar faces. Moreover, the P1 was suppressed across saccades for the familiar but not for the unfamiliar faces. Our results highlight the sensitivity of the occipital FRPs to stimulus properties such as face familiarity and advance our understanding of the integration process across successive saccades in natural vision.

Top-down modulation and cortical-AMG/HPC interaction in familiar face processing

Humans can accurately recognize familiar faces in only a few hundred milliseconds, but the underlying neural mechanism remains unclear. Here, we recorded intracranial electrophysiological signals from ventral temporal cortex (VTC), superior/middle temporal cortex (STC/MTC), medial parietal cortex (MPC), and amygdala/hippocampus (AMG/HPC) in 20 epilepsy patients while they viewed faces of famous people and strangers as well as common objects. In posterior VTC and MPC, familiarity-sensitive responses emerged significantly later than initial face-selective responses, suggesting that familiarity enhances face representations after they are first being extracted. Moreover, viewing famous faces increased the coupling between cortical areas and AMG/HPC in multiple frequency bands. These findings advance our understanding of the neural basis of familiar face perception by identifying the top-down modulation in local face-selective response and interactions between cortical face areas and AMG/HPC.

The representation of shape and texture in category-selective regions of ventral-temporal cortex

Neuroimaging studies using univariate and multivariate approaches have shown that the fusiform face area (FFA) and parahippocampal place area (PPA) respond selectively to images of faces and places. The aim of this study was to determine the extent to which this selectivity to faces or places is based on the shape or texture properties of the images. Faces and houses were filtered to manipulate their texture properties, while preserving the shape properties (spatial envelope) of the images. In Experiment 1, multivariate pattern analysis (MVPA) showed that patterns of fMRI response to faces and houses in FFA and PPA were predicted by the shape properties, but not by the texture properties of the image. In Experiment 2, a univariate analysis (fMR‐adaptation) showed that responses in the FFA and PPA were sensitive to changes in both the shape and texture properties of the image. These findings can be explained by the spatial scale of the representation of images in the FFA and PPA. At a coarser scale (revealed by MVPA), the neural selectivity to faces and houses is sensitive to variation in the shape properties of the image. However, at a finer scale (revealed by fMR‐adaptation), the neural selectivity is sensitive to the texture properties of the image. By combining these neuroimaging paradigms, our results provide insights into the spatial scale of the neural representation of faces and places in the ventral‐temporal cortex.

The emergence of view-symmetric neural responses to familiar and unfamiliar faces

Successful recognition of familiar faces is thought to depend on the ability to integrate view-dependent representations of a face into a view-invariant representation. It has been proposed that a key intermediate step in achieving view invariance is the representation of symmetrical views. However, key unresolved questions remain, such as whether these representations are specific for naturally occurring changes in viewpoint and whether view-symmetric representations exist for familiar faces. To address these issues, we compared behavioural and neural responses to natural (canonical) and unnatural (noncanonical) rotations of the face. Similarity judgements revealed that symmetrical viewpoints were perceived to be more similar than non-symmetrical viewpoints for both canonical and non-canonical rotations. Next, we measured patterns of neural response from early to higher level regions of visual cortex. Early visual areas showed a view-dependent representation for natural or canonical rotations of the face, such that the similarity between patterns of response were related to the difference in rotation. View symmetric patterns of neural response to canonically rotated faces emerged in higher visual areas, particularly in face-selective regions. The emergence of a view-symmetric representation from a view-dependent representation for canonical rotations of the face was also evident for familiar faces, suggesting that view-symmetry is an important intermediate step in generating view-invariant representations. Finally, we measured neural responses to unnatural or non-canonical rotations of the face. View-symmetric patterns of response were also found in face-selective regions. However, in contrast to natural or canonical rotations of the face, these view-symmetric responses did not arise from an initial view-dependent representation in early visual areas. This suggests differences in the way that view-symmetrical representations emerge with canonical or non-canonical rotations. The similarity in the neural response to canonical views of familiar and unfamiliar faces in the core face network suggests that the neural correlates of familiarity emerge at later stages of processing.

Spatially Adjacent Regions in Posterior Cingulate Cortex Represent Familiar Faces at Different Levels of Complexity

Extensive research has shown that perceptual information of faces is processed in a network of hierarchically-organized areas within ventral temporal cortex. For familiar and famous faces, perceptual processing of faces is normally accompanied by extraction of semantic knowledge about the social status of persons. Semantic processing of familiar faces could entail progressive stages of information abstraction. However, the cortical mechanisms supporting multistage processing of familiar faces have not been characterized. Here, using an event-related fMRI experiment, familiar faces from four celebrity groups (actors, singers, politicians, and football players) and unfamiliar faces were presented to the human subjects (both males and females) while they were engaged in a face categorization task. We systematically explored the cortical representations for faces, familiar faces, subcategories of familiar faces, and familiar face identities using whole-brain univariate analysis, searchlight-based multivariate pattern analysis (MVPA), and functional connectivity analysis. Convergent evidence from all these analyses revealed a set of overlapping regions within posterior cingulate cortex (PCC) that contained decodable fMRI responses for representing different levels of semantic knowledge about familiar faces. Our results suggest a multistage pathway in PCC for processing semantic information of faces, analogous to the multistage pathway in ventral temporal cortex for processing perceptual information of faces.SIGNIFICANCE STATEMENT Recognizing familiar faces is an important component of social communications. Previous research has shown that a distributed network of brain areas is involved in processing the semantic information of familiar faces. However, it is not clear how different levels of semantic information are represented in the brain. Here, we evaluated the multivariate response patterns across the entire cortex to discover the areas that contain information for familiar faces, subcategories of familiar faces, and identities of familiar faces. The searchlight maps revealed that different levels of semantic information are represented in topographically adjacent areas within posterior cingulate cortex (PCC). The results suggest that semantic processing of faces is mediated through progressive stages of information abstraction in PCC.

Detecting a viewer's familiarity with a face: Evidence from event-related brain potentials and classifier analyses

Human observers recognize the faces of people they know efficiently and without apparent effort. Consequently, recognizing a familiar face is often assumed to be an automatic process beyond voluntary control. However, there are circumstances in which a person might seek to hide their recognition of a particular face. The present study therefore used event-related potentials (ERPs) and a classifier based on logistic regression to determine if it is possible to detect whether a viewer is familiar with a particular face, regardless of whether the participant is willing to acknowledge it or not. In three experiments, participants were presented with highly variable "ambient" images of personally familiar and unfamiliar faces, while performing an incidental butterfly detection task (Experiment 1), an explicit familiarity judgment task (Experiment 2), and a concealed familiarity task in which they were asked to deny familiarity with one truly known facial identity while acknowledging familiarity with a second known identity (Experiment 3). In all three experiments, we observed substantially more negative ERP amplitudes at occipito-temporal electrodes for familiar relative to unfamiliar faces starting approximately 200 ms after stimulus onset. Both the earlier N250 familiarity effect, reflecting visual recognition of a known face, and the later sustained familiarity effect, reflecting the integration of visual with additional identity-specific information, were similar across experiments and thus independent of task demands. These results were further supported by the classifier analysis. We conclude that ERP correlates of familiar face recognition are largely independent of voluntary control and discuss potential applications in forensic settings.

Getting to Know Someone: Familiarity, Person Recognition, and Identification in the Human Brain

In our everyday life, we continuously get to know people, dominantly through their faces. Several neuroscientific experiments showed that familiarization changes the behavioral processing and underlying neural representation of faces of others. Here, we propose a model of the process of how we actually get to know someone. First, the purely visual familiarization of unfamiliar faces occurs. Second, the accumulation of associated, nonsensory information refines person representation, and finally, one reaches a stage where the effortless identification of very well-known persons occurs. We offer here an overview of neuroimaging studies, first evaluating how and in what ways the processing of unfamiliar and familiar faces differs and, second, by analyzing the fMRI adaptation and multivariate pattern analysis results we estimate where identity-specific representation is found in the brain. The available neuroimaging data suggest that different aspects of the information emerge gradually as one gets more and more familiar with a person within the same network. We propose a novel model of familiarity and identity processing, where the differential activation of long-term memory and emotion processing areas is essential for correct identification.

Predicting Interpersonal Outcomes From Information Processing Tasks Using Personally Relevant and Generic Stimuli: A Methodology Study

Despite evidence of differential processing of personally relevant stimuli (PR), most studies investigating attentional biases in processing emotional content use generic stimuli. We sought to examine differences in the processing of PR, relative to generic, stimuli across information processing tasks and to validate their use in predicting concurrent interpersonal functioning. Fifty participants (25 female) viewed generic and PR (i.e., their intimate partner’s face) emotional stimuli during tasks assessing selective attention (using a modified version of the Spatial Cueing Task) and inhibition (using the Negative Affective Priming task) of emotional content. Ratings of relationship quality were also collected. Evidence of increased selective attention during controlled and greater avoidance during automatic stages of processing emerged when viewing PR, relative to generic, emotional faces. We also found greater inhibition of PR sad faces. Finally, male, but not female, participants who displayed greater difficulty disengaging from the sad face of their partner reported more conflict in their relationships. Taken together, findings from information processing studies using generic emotional stimuli may not be representative of how we process PR stimuli in naturalistic settings.

From concepts to percepts in human and machine face recognition: A reply to Blauch, Behrmann & Plaut

Intact recognition of familiar faces is critical for appropriate social interactions. Thus, the human face processing system should be optimized for familiar face recognition. Blauch et al. (2020) used face recognition deep convolutional neural networks (DCNNs) that are trained to maximize recognition of the trained (familiar) identities, to model human unfamiliar and familiar face recognition. In line with this model, we discuss behavioral, neuroimaging and computational findings that indicate that human face recognition develops from the generation of identity-specific concepts of familiar faces that are learned in a supervised manner, to the generation of view-invariant identity-general perceptual representations. Face-trained DCNNs seem to share some fundamental similarities with this framework.

How face perception unfolds over time

Within a fraction of a second of viewing a face, we have already determined its gender, age and identity. A full understanding of this remarkable feat will require a characterization of the computational steps it entails, along with the representations extracted at each. Here, we used magnetoencephalography (MEG) to measure the time course of neural responses to faces, thereby addressing two fundamental questions about how face processing unfolds over time. First, using representational similarity analysis, we found that facial gender and age information emerged before identity information, suggesting a coarse-to-fine processing of face dimensions. Second, identity and gender representations of familiar faces were enhanced very early on, suggesting that the behavioral benefit for familiar faces results from tuning of early feed-forward processing mechanisms. These findings start to reveal the time course of face processing in humans, and provide powerful new constraints on computational theories of face perception.

A Robust Neural Index of High Face Familiarity

Humans are remarkably accurate at recognizing familiar faces, whereas their ability to recognize, or even match, unfamiliar faces is much poorer. However, previous research has failed to identify neural correlates of this striking behavioral difference. Here, we found a clear difference in brain potentials elicited by highly familiar faces versus unfamiliar faces. This effect starts 200 ms after stimulus onset and reaches its maximum at 400 to 600 ms. This sustained-familiarity effect was substantially larger than previous candidates for a neural familiarity marker and was detected in almost all participants, representing a reliable index of high familiarity. Whereas its scalp distribution was consistent with a generator in the ventral visual pathway, its modulation by repetition and degree of familiarity suggests an integration of affective and visual information.

Child brain exhibits a multi-rhythmic response to attachment cues

Research on the human parental brain implicated brain networks involved in simulation, mentalization and emotion processing and indicated that stimuli of own parent-child interaction elicit greater integration among networks supporting attachment. Here, we examined children's neural activation while viewing own parent-child interactions and asked whether similar networks activate when children are exposed to attachment stimuli. Sixty-five 11-year-old children underwent magnetoencephalography (MEG) while observing own vs unfamiliar mother-child interaction. Own mother-child interactions elicited a greater neural response across distributed brain areas including alpha suppression in posterior regions, theta enhancement in the fusiform gyrus and beta- and gamma-band oscillations across a wide cluster in the right temporal cortex, comprising the superior temporal sulcus/superior temporal gyrus and insula. Theta and gamma activations were associated with the degree of mother-child social synchrony in the home ecology. Findings from this exploratory study are the first to show activations in children that are similar to previous findings in parents and comparable associations between social synchrony and gamma oscillations in temporal regions. Results indicate that attachment stimuli elicit a strong neural response in children that spreads across a wide range of oscillations, underscoring the considerable neural resources allocated to this fundamental, survival-related cue.

Pupil mimicry promotes trust through the theory-of-mind network

The human eye can provide powerful insights into the emotions and intentions of others; however, how pupillary changes influence observers' behavior remains largely unknown. The present fMRI-pupillometry study revealed that when the pupils of interacting partners synchronously dilate, trust is promoted, which suggests that pupil mimicry affiliates people. Here we provide evidence that pupil mimicry modulates trust decisions through the activation of the theory-of-mind network (precuneus, temporo-parietal junction, superior temporal sulcus, and medial prefrontal cortex). This network was recruited during pupil-dilation mimicry compared with interactions without mimicry or compared with pupil-constriction mimicry. Furthermore, the level of theory-of-mind engagement was proportional to individual's susceptibility to pupil-dilation mimicry. These data reveal a fundamental mechanism by which an individual's pupils trigger neurophysiological responses within an observer: when interacting partners synchronously dilate their pupils, humans come to feel reflections of the inner states of others, which fosters trust formation.

Distinct neural processes for the perception of familiar versus unfamiliar faces along the visual hierarchy revealed by EEG

Humans recognize faces with ease, despite the complexity of the task and of the visual system which underlies it. Different spatial regions, including both the core and extended face processing networks, and distinct temporal stages of processing have been implicated in face recognition, but there is ongoing controversy regarding the extent to which the mechanisms for recognizing a familiar face differ from those for an unfamiliar face. Here, we used electroencephalogram (EEG) and flicker SSVEP, a high signal-to-noise approach, and searchlight decoding methods to elucidate the mechanisms mediating the processing of familiar and unfamiliar faces in the time domain. Familiar and unfamiliar faces were presented periodically at 15 Hz, 6 Hz and 3.75 Hz either upright or inverted in separate blocks, with the rationale that faster frequencies require shorter processing times per image and tap into fundamentally different levels of visual processing. The 15 Hz trials, likely to reflect early visual processing, exhibited enhanced neural responses for familiar over unfamiliar face trials, but only when the faces were upright. In contrast, decoding methods revealed similar classification accuracies for upright and inverted faces for both familiar and unfamiliar faces. For the 6 Hz frequency, familiar faces had lower amplitude responses than unfamiliar faces, and decoding familiarity was more accurate for upright compared with inverted faces. Finally, the 3.75 Hz frequency revealed no main effects of familiarity, but decoding showed significant correlations with behavioral ratings of face familiarity, suggesting that activity evoked by this slow presentation frequency reflected higher-level, cognitive aspects of familiarity processing. This three-way dissociation between frequencies reveals that fundamentally different stages of the visual hierarchy are modulated by face familiarity. The combination of experimental and analytical approaches used here represent a novel method for elucidating spatio-temporal characteristics within the visual system.

Age-related increase of image-invariance in the fusiform face area

•

Image invariance in the FFA increases from age seven to adulthood.

•

These results are confirmed by two independent ROI analyses.

•

Adaptation in the FFA relates to the ability to recognize a face in multiple images.

Face recognition undergoes prolonged development from childhood to adulthood, thereby raising the question which neural underpinnings are driving this development. Here, we address the development of the neural foundation of the ability to recognize a face across naturally varying images. Fourteen children (ages, 7–10) and 14 adults (ages, 20–23) watched images of either the same or different faces in a functional magnetic resonance imaging adaptation paradigm. The same face was either presented in exact image repetitions or in varying images. Additionally, a subset of participants completed a behavioral task, in which they decided if the face in consecutively presented images belonged to the same person. Results revealed age-related increases in neural sensitivity to face identity in the fusiform face area. Importantly, ventral temporal face-selective regions exhibited more image-invariance – as indicated by stronger adaptation for different images of the same person – in adults compared to children. Crucially, the amount of adaptation to face identity across varying images was correlated with the ability to recognize individual faces in different images. These results suggest that the increase of image-invariance in face-selective regions might be related to the development of face recognition skills.

Faces, people and the brain: The 45th Sir Frederic Bartlett Lecture

The fact that the face is a source of diverse social signals allows us to use face and person perception as a model system for asking important psychological questions about how our brains are organised. A key issue concerns whether we rely primarily on some form of generic representation of the common physical source of these social signals (the face) to interpret them, or instead create multiple representations by assigning different aspects of the task to different specialist components. Variants of the specialist components hypothesis have formed the dominant theoretical perspective on face perception for more than three decades, but despite this dominance of formally and informally expressed theories, the underlying principles and extent of any division of labour remain uncertain. Here, I discuss three important sources of constraint: first, the evolved structure of the brain; second, the need to optimise responses to different everyday tasks; and third, the statistical structure of faces in the perceiver’s environment. I show how these constraints interact to determine the underlying functional organisation of face and person perception.