Modulation of face processing by top-down attention: Insights from early ERP waveforms

The face fusiform area (FFA) plays a pivotal role in face recognition, yet the precise timeline of its activity remains debated. Using EEG, we conducted three experiments to investigate how expectancy-consistent versus expectancy-inconsistent visual stimuli influence processing dynamics. Participants viewed images of faces, houses, and tools (Experiment 1), celebrity faces (Experiment 2), or animal faces (Experiment 3), preceded by a priming question. Notably, both conditions presented identical visual stimulation, ensuring that observed differences stemmed from cognitive processing rather than sensory input. Our results from Experiments 2 and 3 reveal that while the initial 150 ms period, crucial for unconscious face detection, remained unaffected, subsequent processing exhibited a delay of several milliseconds for expectancy-inconsistent stimuli, indicating additional processing time required for unexpected recognition. Importantly, no significant differences were observed in Experiment 1, where less demanding tasks or generic mental imagery were used, suggesting that the priming effect was not as pronounced in this context. These findings underscore the critical role of the period immediately following the first 150 ms in face identification and individuation, highlighting the influence of top-down attention on face recognition dynamics. This study provides novel insights into the temporal dynamics of face processing and the neural mechanisms underlying top-down attentional modulation.

The temporal dynamics of familiar face recognition: Event-related brain potentials reveal the efficient activation of facial identity representations

While it is widely known that humans are typically highly accurate at recognizing familiar faces, it is less clear how efficiently recognition is achieved. In a series of three experiments, we used event-related brain potentials (ERP) in a repetition priming paradigm to examine the efficiency of familiar face recognition. Specifically, we varied the presentation time of the prime stimulus between 500 ms and 33 ms (Experiments 1 and 2), and additionally used backward masks (Experiment 3) to prevent the potential occurrence of visual aftereffects. Crucially, to test for the recognition of facial identity rather than a specific picture, we used different images of the same facial identities in repetition conditions. We observed clear ERP repetition priming effects between 300 and 500 ms after target onset at all prime durations, which suggests that the prime stimulus was sufficiently well processed to allow for facilitated recognition of the target in all conditions. This finding held true even in severely restricted viewing conditions including very brief prime durations and backward masks. We conclude that the facial recognition system is both highly effective and efficient, thus allowing for our impressive ability to recognise the faces that we know.

Shared neural codes of recognition memory

Recognition memory research has identified several electrophysiological indicators of successful memory retrieval, known as old-new effects. These effects have been observed in different sensory domains using various stimulus types, but little attention has been given to their similarity or distinctiveness and the underlying processes they may share. Here, a data-driven approach was taken to investigate the temporal evolution of shared information content between different memory conditions using openly available EEG data from healthy human participants of both sexes, taken from six experiments. A test dataset involving personally highly familiar and unfamiliar faces was used. The results show that neural signals of recognition memory for face stimuli were highly generalized starting from around 200 ms following stimulus onset. When training was performed on non-face datasets, an early (around 200-300 ms) to late (post-400 ms) differentiation was observed over most regions of interest. Successful cross-classification for non-face stimuli (music and object/scene associations) was most pronounced in late period. Additionally, a striking dissociation was observed between familiar and remembered objects, with shared signals present only in the late window for correctly remembered objects, while cross-classification for familiar objects was successful in the early period as well. These findings suggest that late neural signals of memory retrieval generalize across sensory modalities and stimulus types, and the dissociation between familiar and remembered objects may provide insight into the underlying processes.

Familiarity enhances functional connectivity between visual and nonvisual regions of the brain during natural viewing

We explored the neural correlates of familiarity with people and places using a naturalistic viewing paradigm. Neural responses were measured using functional magnetic resonance imaging, while participants viewed a movie taken from Game of Thrones. We compared inter-subject correlations and functional connectivity in participants who were either familiar or unfamiliar with the TV series. Higher inter-subject correlations were found between familiar participants in regions, beyond the visual brain, that are typically associated with the processing of semantic, episodic, and affective information. However, familiarity also increased functional connectivity between face and scene regions in the visual brain and the nonvisual regions of the familiarity network. To determine whether these regions play an important role in face recognition, we measured responses in participants with developmental prosopagnosia (DP). Consistent with a deficit in face recognition, the effect of familiarity was significantly attenuated across the familiarity network in DP. The effect of familiarity on functional connectivity between face regions and the familiarity network was also attenuated in DP. These results show that the neural response to familiarity involves an extended network of brain regions and that functional connectivity between visual and nonvisual regions of the brain plays an important role in the recognition of people and places during natural viewing.

The neuropsychological evaluation of face identity recognition

Facial identity recognition (FIR) is arguably the ultimate form of recognition for the adult human brain. Even if the term prosopagnosia is reserved for exceptionally rare brain-damaged cases with a category-specific abrupt loss of FIR at adulthood, subjective and objective impairments or difficulties of FIR are common in the neuropsychological population. Here we provide a critical overview of the evaluation of FIR both for clinicians and researchers in neuropsychology. FIR impairments occur following many causes that should be identified objectively by both general and specific, behavioral and neural examinations. We refute the commonly used dissociation between perceptual and memory deficits/tests for FIR, since even a task involving the discrimination of unfamiliar face images presented side-by-side relies on cortical memories of faces in the right-lateralized ventral occipito-temporal cortex. Another frequently encountered confusion is between specific deficits of the FIR function and a more general impairment of semantic memory (of people), the latter being most often encountered following anterior temporal lobe damage. Many computerized tests aimed at evaluating FIR have appeared over the last two decades, as reviewed here. However, despite undeniable strengths, they often suffer from ecological limitations, difficulties of instruction, as well as a lack of consideration for processing speed and qualitative information. Taking into account these issues, a recently developed behavioral test with natural images manipulating face familiarity, stimulus inversion, and correct response times as a key variable appears promising. The measurement of electroencephalographic (EEG) activity in the frequency domain from fast periodic visual stimulation also appears as a particularly promising tool to complete and enhance the neuropsychological assessment of FIR.

Neural computations in prosopagnosia

We report an investigation of the neural processes involved in the processing of faces and objects of brain-lesioned patient PS, a well-documented case of pure acquired prosopagnosia. We gathered a substantial dataset of high-density electrophysiological recordings from both PS and neurotypicals. Using representational similarity analysis, we produced time-resolved brain representations in a format that facilitates direct comparisons across time points, different individuals, and computational models. To understand how the lesions in PS's ventral stream affect the temporal evolution of her brain representations, we computed the temporal generalization of her brain representations. We uncovered that PS's early brain representations exhibit an unusual similarity to later representations, implying an excessive generalization of early visual patterns. To reveal the underlying computational deficits, we correlated PS' brain representations with those of deep neural networks (DNN). We found that the computations underlying PS' brain activity bore a closer resemblance to early layers of a visual DNN than those of controls. However, the brain representations in neurotypicals became more akin to those of the later layers of the model compared to PS. We confirmed PS's deficits in high-level brain representations by demonstrating that her brain representations exhibited less similarity with those of a DNN of semantics.

Multiple images captured from a single encounter do not promote face learning

Viewing multiple images of a newly encountered face improves recognition of that identity in new instances. Studies examining face learning have presented high-variability (HV) images that incorporate changes that occur from moment-to-moment (e.g., head orientation and expression) and over time (e.g., lighting, hairstyle, and health). We examined whether low-variability (LV) images (i.e., images that incorporate only moment-to-moment changes) also promote generalisation of learning such that novel instances are recognised. Participants viewed a single image, six LV images, or six HV images of a target identity before being asked to recognise novel images of that identity in a face matching task (training stimuli remained visible) or a memory task (training stimuli were removed). In Experiment 1 (n = 71), participants indicated which image(s) in 8-image arrays belonged to the target identity. In Experiment 2 (n = 73), participants indicated whether sequentially presented images belonged to the target identity. Relative to the single-image condition, sensitivity to identity improved and response biases were less conservative in the HV condition; we found no evidence of generalisation of learning in the LV condition regardless of testing protocol. Our findings suggest that day-to-day variability in appearance plays an essential role in acquiring expertise with a novel face.

How neural representations of newly learnt faces change over time: Event-related brain potential evidence for overnight consolidation

Previous experiments have shown that a brief encounter with a previously unfamiliar person leads to the establishment of new facial representations, which can be activated by completely novel pictures of the newly learnt face. The present study examined how stable such novel neural representations are over time, and, specifically, how they become consolidated within the first 24 h after learning. Using event-related brain potentials (ERPs) in a between-participants design, we demonstrate that clear face familiarity effects in the occipito-temporal N250 are evident immediately after learning. These effects then undergo change, with a nearly complete absence of familiarity-related ERP differences 4 h after the initial encounter. Critically, 24 h after learning, the original familiarity effect re-emerges. These findings suggest that the neural correlates of novel face representations are not stable over time but change during the first day after learning. The resulting pattern of change is consistent with a process of consolidation.

Your place or mine? The neural dynamics of personally familiar scene recognition suggests category independent familiarity encoding

Recognizing a stimulus as familiar is an important capacity in our everyday life. Recent investigation of visual processes has led to important insights into the nature of the neural representations of familiarity for human faces. Still, little is known about how familiarity affects the neural dynamics of non-face stimulus processing. Here we report the results of an EEG study, examining the representational dynamics of personally familiar scenes. Participants viewed highly variable images of their own apartments and unfamiliar ones, as well as personally familiar and unfamiliar faces. Multivariate pattern analyses were used to examine the time course of differential processing of familiar and unfamiliar stimuli. Time-resolved classification revealed that familiarity is decodable from the EEG data similarly for scenes and faces. The temporal dynamics showed delayed onsets and peaks for scenes as compared to faces. Familiarity information, starting at 200 ms, generalized across stimulus categories and led to a robust familiarity effect. In addition, familiarity enhanced category representations in early (250-300 ms) and later (>400 ms) processing stages. Our results extend previous face familiarity results to another stimulus category and suggest that familiarity as a construct can be understood as a general, stimulus-independent processing step during recognition.

Personal familiarity of faces, animals, objects, and scenes: Distinct perceptual and overlapping conceptual representations

While face, object, and scene recognition are often studied at a basic categorization level (e.g. "a face", "a car", "a kitchen"), we frequently recognise individual items of these categories as unique entities (e.g. "my mother", "my car", "my kitchen"). This recognition of individual identity is essential to appropriate behaviour in our world. However, relatively little is known about how we recognise individually familiar visual stimuli. Using event-related brain potentials, the present study examined whether and to what extent the underlying neural representations of personally familiar items are similar or different across different categories. In three experiments, we examined the recognition of personally highly familiar faces, animals, indoor scenes, and objects. We observed relatively distinct familiarity effects in an early time window (200-400 ms), with a clearly right-lateralized occipito-temporal scalp distribution for human faces and more bilateral and posterior distributions for other stimulus categories, presumably reflecting access to at least partly discrete visual long-term representations. In contrast, we found clearly overlapping familiarity effects in a later time window (starting 400 to 500 ms after stimulus onset), again with a mainly right occipito-temporal scalp distribution, for all stimulus categories. These later effects appear to reflect the sustained activation of conceptual properties relevant to any potential interaction. We conclude that familiarity for items from the various visual stimulus categories tested here is represented differently at the perceptual level, while relatively overlapping conceptual mechanisms allow for the preparation of impending potential interaction with the environment.

Beyond facial expressions: A systematic review on effects of emotional relevance of faces on the N170

The N170 is the most prominent electrophysiological signature of face processing. While facial expressions reliably modulate the N170, there is considerable variance in N170 modulations by other sources of emotional relevance. Therefore, we systematically review and discuss this research area using different methods to manipulate the emotional relevance of inherently neutral faces. These methods were categorized into (1) existing pre-experimental affective person knowledge (e.g., negative attitudes towards outgroup faces), (2) experimentally instructed affective person knowledge (e.g., negative person information), (3) contingency-based affective learning (e.g., fear-conditioning), or (4) the immediate affective context (e.g., emotional information directly preceding the face presentation). For all categories except the immediate affective context category, the majority of studies reported significantly increased N170 amplitudes depending on the emotional relevance of faces. Furthermore, the potentiated N170 was observed across different attention conditions, supporting the role of the emotional relevance of faces on the early prioritized processing of configural facial information, regardless of low-level differences. However, we identified several open research questions and suggest venues for further research.

A neural marker of the human face identity familiarity effect

Human adults associate different views of an identity much better for familiar than for unfamiliar faces. However, a robust and consistent neural index of this behavioral face identity familiarity effect (FIFE)-not found in non-human primate species-is lacking. Here we provide such a neural FIFE index, measured implicitly and with one fixation per face. Fourteen participants viewed 70 s stimulation sequences of a large set (n = 40) of widely variable natural images of a face identity at a rate of 6 images/second (6 Hz). Different face identities appeared every 5th image (1.2 Hz). In a sequence, face images were either familiar (i.e., famous) or unfamiliar, participants performing a non-periodic task unrelated to face recognition. The face identity recognition response identified at 1.2 Hz over occipital-temporal regions in the frequency-domain electroencephalogram was 3.4 times larger for familiar than unfamiliar faces. The neural response to familiar faces-which emerged at about 180 ms following face onset-was significant in each individual but a case of prosopdysgnosia. Besides potential clinical and forensic applications to implicitly measure one's knowledge of a face identity, these findings open new perspectives to clarify the neurofunctional source of the FIFE and understand the nature of human face identity recognition.

Increased N250 elicited by facial familiarity: An ERP study including the face inversion effect and facial emotion processing

The present study aims to explore how familiarity modulates the neural processing of faces under different conditions: upright or inverted, neutral or emotional. To this purpose, 32 participants (25 female; age: M = 27.7 years, SD = 9.3) performed two face/emotion identification tasks during EEG recording. In the first task, to study facial processing, three different categories of facial stimuli were presented during a target detection task: famous familiar faces, faces of loved ones, and unfamiliar faces. To explore the face inversion effect according to each level of familiarity, these facial stimuli were also presented upside down. In the second task, to study emotional face processing, an emotional identification task on personally familiar and unfamiliar faces was conducted. The behavioural results showed an improved performance in the identification of facial expressions of emotion with the increase of facial familiarity, consistent with the previous literature. Regarding electrophysiological results, we found increased amplitudes of the P100, N170, and N250 for inverted compared to upright faces, independently of their degree of familiarity. Moreover, we did not find familiarity effects at the P100 and N170 time-windows, but we found that N250 amplitude was larger for personally familiar compared to unfamiliar faces. This result supports the reasoning that the facial familiarity increases the neural activity during the N250 time-window, which may be explained by the processing of additional information prompted by the viewing of our loved ones faces, in contrast to what happens with unfamiliar individuals.

The neural dynamics of familiarity-dependent face identity representation

Recognizing a face as belonging to a given identity is essential in our everyday life. Clearly, the correct identification of a face is only possible for familiar people, but 'familiarity' covers a wide range-from people we see every day to those we barely know. Although several studies have shown that the processing of familiar and unfamiliar faces is substantially different, little is known about how the degree of familiarity affects the neural dynamics of face identity processing. Here, we report the results of a multivariate EEG analysis, examining the representational dynamics of face identity across several familiarity levels. Participants viewed highly variable face images of 20 identities, including the participants' own face, personally familiar (PF), celebrity and unfamiliar faces. Linear discriminant classifiers were trained and tested on EEG patterns to discriminate pairs of identities of the same familiarity level. Time-resolved classification revealed that the neural representations of identity discrimination emerge around 100 ms post-stimulus onset, relatively independently of familiarity level. In contrast, identity decoding between 200 and 400 ms is determined to a large extent by familiarity: it can be recovered with higher accuracy and for a longer duration in the case of more familiar faces. In addition, we found no increased discriminability for faces of PF persons compared to those of highly familiar celebrities. One's own face benefits from processing advantages only in a relatively late time-window. Our findings provide new insights into how the brain represents face identity with various degrees of familiarity and show that the degree of familiarity modulates the available identity-specific information at a relatively early time window.

Differences between high and low performers in face recognition in electrophysiological correlates of face familiarity and distance-to-norm

Valentine's influential norm-based multidimensional face-space model (nMDFS) predicts that perceived distinctiveness of a face increases with its distance to the norm. Occipito-temporal event-related potentials (ERPs) have been recently shown to respond selectively to variations in distance-to-norm (P200) or familiarity (N250, late negativity), respectively (Wuttke & Schweinberger, 2019). Despite growing evidence on interindividual differences in face perception skills at the behavioral level, little research has focused on their electrophysiological correlates. To reveal potential interindividual differences in face spaces, we contrasted high and low performers in face recognition in regards to distance-to-norm (P200) and familiarity (N250). We replicated both the P200 distance-to-norm and the N250 familiarity effect. Importantly, we observed: i) reduced responses in low compared to high performers of face recognition, especially in terms of smaller distance-to-norm effects in the P200, possibly indicating less 'expanded' face spaces in low compared to high performers; ii) increased N250 responses to familiar original faces in high performers, suggesting more robust face identity representations. In summary, these findings suggest the contribution of both early norm-based face coding and robust face representations to individual face recognition skills, and indicate that ERPs can offer a promising route to understand individual differences in face perception and their neurocognitive correlates.

Developing familiarity during the first eight months of knowing a person: A longitudinal EEG study on face and identity learning

It is well-established that familiar and unfamiliar faces are processed differently, but surprisingly little is known about how familiarity builds up over time and how novel faces gradually become represented in the brain. Here, we used event-related brain potentials (ERPs) in a pre-registered, longitudinal study to examine the neural processes accompanying face and identity learning during the first eight months of knowing a person. Specifically, we examined how increasing real-life familiarity affects visual recognition (N250 Familiarity Effect) and the integration of person-related knowledge (Sustained Familiarity Effect, SFE). Sixteen first-year undergraduates were tested in three sessions, approximately one, five, and eight months after the start of the academic year, with highly variable "ambient" images of a new friend they had met at university and of an unfamiliar person. We observed clear ERP familiarity effects for the new friend after one month of familiarity. While there was an increase in the N250 effect over the course of the study, no change in the SFE was observed. These results suggest that visual face representations develop faster relative to the integration of identity-specific knowledge.

The effect of familiarity on behavioral oscillations in face perception

Studies on behavioral oscillations demonstrate that visual sensitivity fluctuates over time and visual processing varies periodically, mirroring neural oscillations at the same frequencies. Do these behavioral oscillations reflect fixed and relatively automatic sensory sampling, or top-down processes such as attention or predictive coding? To disentangle these theories, the current study used a dual-target rapid serial visual presentation paradigm, where participants indicated the gender of a face target embedded in streams of distractors presented at 30 Hz. On critical trials, two identical targets were presented with varied stimulus onset asynchrony from 200 to 833 ms. The target was either familiar or unfamiliar faces, divided into different blocks. We found a 4.6 Hz phase-coherent fluctuation in gender discrimination performance across both trial types, consistent with previous reports. In addition, however, we found an effect at the alpha frequency, with behavioral oscillations in the familiar blocks characterized by a faster high-alpha peak than for the unfamiliar face blocks. These results are consistent with the combination of both a relatively stable modulation in the theta band and faster modulation of the alpha oscillations. Therefore, the overall pattern of perceptual sampling in visual perception may depend, at least in part, on task demands. PROTOCOL REGISTRATION: The stage 1 protocol for this Registered Report was accepted in principle on 16/08/2022. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/A98UF .

Neural Correlates of Voice Learning with Distinctive and Non-Distinctive Faces

Recognizing people from their voices may be facilitated by a voice's distinctiveness, in a manner similar to that which has been reported for faces. However, little is known about the neural time-course of voice learning and the role of facial information in voice learning. Based on evidence for audiovisual integration in the recognition of familiar people, we studied the behavioral and electrophysiological correlates of voice learning associated with distinctive or non-distinctive faces. We repeated twelve unfamiliar voices uttering short sentences, together with either distinctive or non-distinctive faces (depicted before and during voice presentation) in six learning-test cycles. During learning, distinctive faces increased early visually-evoked (N170, P200, N250) potentials relative to non-distinctive faces, and face distinctiveness modulated voice-elicited slow EEG activity at the occipito-temporal and fronto-central electrodes. At the test, unimodally-presented voices previously learned with distinctive faces were classified more quickly than were voices learned with non-distinctive faces, and also more quickly than novel voices. Moreover, voices previously learned with faces elicited an N250-like component that was similar in topography to that typically observed for facial stimuli. The preliminary source localization of this voice-induced N250 was compatible with a source in the fusiform gyrus. Taken together, our findings provide support for a theory of early interaction between voice and face processing areas during both learning and voice recognition.

Unfamiliar face matching ability predicts the slope of face learning

We provide the first examination of individual differences in the efficiency of face learning. Investigating individual differences in face learning can illuminate potential mechanisms and provide greater understanding of why certain individuals might be more efficient face learners. Participants completed two unfamiliar face matching tasks and a learning task in which learning was assessed after viewing 1, 3, 6, and 9 images of to-be-learned identities. Individual differences in the slope of face learning (i.e., increases in sensitivity to identity) were predicted by the ability to discriminate between matched (same-identity) vs. mismatched (different-identity) pairs of wholly unfamiliar faces. A Dual Process Signal Detection model showed that three parameters increased with learning: Familiarity (an unconscious type of memory that varies in strength), recollection-old (conscious recognition of a learned identity), and recollection-new (conscious/confident rejection of novel identities). Good (vs. poor) matchers had higher Recollection-Old scores throughout learning and showed a steeper increase in Recollection-New. We conclude that good matchers are better able to capitalize on exposure to within-person variability in appearance, an effect that is attributable to their conscious memory for both learned and novel faces. These results have applied implications and will inform contemporary and traditional models of face identification.

The role of the eye region for familiar face recognition: Evidence from spatial low-pass filtering and contrast negation

What information is used for familiar face recognition? While previous research suggests a particular importance of the eye region, information from the rest of the face also needs to be integrated. What type of information is used in conjunction with the eyes is largely unclear. In three experiments, participants were asked to recognise so-called face chimeras, in which the eye region was not manipulated, while the rest of the face was either presented in negative contrast (contrast chimeras) or low-pass filtered (blur chimeras). We show (1) that both chimeras are recognised substantially better than fully blurred faces, (2) that the recognition advantage for blur chimeras is specific to the eye region but cannot be explained by cues available in this part of the face alone, and (3) that a combination of negative contrast and blurring outside of the eye region eliminates the chimera advantage. We conclude that full-frequency but distorted surface reflectance cues (in contrast chimeras) or coarse shape information (in blur chimeras) can be used in combination with the eye region for effective face recognition. Our findings further suggest that the face recognition system can flexibly use both types of information, depending on availability.

How quickly do we learn new faces in everyday life? Neurophysiological evidence for face identity learning after a brief real-life encounter

Faces learnt in a single experimental session elicit a familiarity effect in event-related brain potentials (ERPs), with more negative amplitudes for newly learnt relative to unfamiliar faces in the N250 component. However, no ERP study has examined face learning following a brief real-life encounter, and it is not clear how long it takes to learn new faces in such ecologically more valid conditions. To investigate these questions, the present study examined whether robust image-independent representations, as reflected in the N250 familiarity effect, could be established after a brief unconstrained social interaction by analysing the ERPs elicited by highly variable images of the newly learnt identity and an unfamiliar person. Significant N250 familiarity effects were observed after a 30-min (Experiment 1) and a 10-min (Experiment 2) encounter, and a trend was observed after 5 min of learning (Experiment 3), demonstrating that 5-10 min of exposure were sufficient for the initial establishment of image-independent representations. Additionally, the magnitude of the effects reported after 10 and 30 min was comparable suggesting that the first 10 min of a social encounter might be crucial, with extra 20 min from the same encounter not adding further benefit for the initial formation of robust face representations.

No semantic information is necessary to evoke general neural signatures of face familiarity: evidence from cross-experiment classification

Recent theories on the neural correlates of face identification stressed the importance of the available identity-specific semantic and affective information. However, whether such information is essential for the emergence of neural signal of familiarity has not yet been studied in detail. Here, we explored the shared representation of face familiarity between perceptually and personally familiarized identities. We applied a cross-experiment multivariate pattern classification analysis (MVPA), to test if EEG patterns for passive viewing of personally familiar and unfamiliar faces are useful in decoding familiarity in a matching task where familiarity was attained thorough a short perceptual task. Importantly, no additional semantic, contextual, or affective information was provided for the familiarized identities during perceptual familiarization. Although the two datasets originate from different sets of participants who were engaged in two different tasks, familiarity was still decodable in the sorted, same-identity matching trials. This finding indicates that the visual processing of the faces of personally familiar and purely perceptually familiarized identities involve similar mechanisms, leading to cross-classifiable neural patterns.

Top-down modulation and cortical-AMG/HPC interaction in familiar face processing

Humans can accurately recognize familiar faces in only a few hundred milliseconds, but the underlying neural mechanism remains unclear. Here, we recorded intracranial electrophysiological signals from ventral temporal cortex (VTC), superior/middle temporal cortex (STC/MTC), medial parietal cortex (MPC), and amygdala/hippocampus (AMG/HPC) in 20 epilepsy patients while they viewed faces of famous people and strangers as well as common objects. In posterior VTC and MPC, familiarity-sensitive responses emerged significantly later than initial face-selective responses, suggesting that familiarity enhances face representations after they are first being extracted. Moreover, viewing famous faces increased the coupling between cortical areas and AMG/HPC in multiple frequency bands. These findings advance our understanding of the neural basis of familiar face perception by identifying the top-down modulation in local face-selective response and interactions between cortical face areas and AMG/HPC.

Characterizing the shared signals of face familiarity: Long-term acquaintance, voluntary control, and concealed knowledge

In a recent study using cross-experiment multivariate classification of EEG patterns, we found evidence for a shared familiarity signal for faces, patterns of neural activity that successfully separate trials for familiar and unfamiliar faces across participants and modes of familiarization. Here, our aim was to expand upon this research to further characterize the spatio-temporal properties of this signal. By utilizing the information content present for incidental exposure to personally familiar and unfamiliar faces, we tested how the information content in the neural signal unfolds over time under different task demands - giving truthful or deceptive responses to photographs of genuinely familiar and unfamiliar individuals. For this goal, we re-analyzed data from two previously published experiments using within-experiment leave-one-subject-out and cross-experiment classification of face familiarity. We observed that the general face familiarity signal, consistent with its previously described spatio-temporal properties, is present for long-term personally familiar faces under passive viewing, as well as for acknowledged and concealed familiarity responses. Also, central-posterior regions contain information related to deception. We propose that signals in the 200-400 ms window are modulated by top-down task-related anticipation, while the patterns in the 400-600 ms window are influenced by conscious effort to deceive. To our knowledge, this is the first report describing the representational dynamics of concealed knowledge for faces, using time-resolved multivariate classification.

What happens to our representation of identity as familiar faces age? Evidence from priming and identity aftereffects

Matching identity in images of unfamiliar faces is error prone, but we can easily recognize highly variable images of familiar faces - even images taken decades apart. Recent theoretical development based on computational modelling can account for how we recognize extremely variable instances of the same identity. We provide complementary behavioural data by examining older adults' representation of older celebrities who were also famous when young. In Experiment 1, participants completed a long-lag repetition priming task in which primes and test stimuli were the same age or different ages. In Experiment 2, participants completed an identity after effects task in which the adapting stimulus was an older or young photograph of one celebrity and the test stimulus was a morph between the adapting identity and a different celebrity; the adapting stimulus was the same age as the test stimulus on some trials (e.g., both old) or a different age (e.g., adapter young, test stimulus old). The magnitude of priming and identity after effects were not influenced by whether the prime and adapting stimulus were the same age or different age as the test face. Collectively, our findings suggest that humans have one common mental representation for a familiar face (e.g., Paul McCartney) that incorporates visual changes across decades, rather than multiple age-specific representations. These findings make novel predictions for state-of-the-art algorithms (e.g., Deep Convolutional Neural Networks).

Twenty years of investigation with the case of prosopagnosia PS to understand human face identity recognition. Part II: Neural basis

Patient PS sustained her dramatic brain injury in 1992, the same year as the first report of a neuroimaging study of human face recognition. The present paper complements the review on the functional nature of PS's prosopagnosia (part I), illustrating how her case study directly, i.e., through neuroimaging investigations of her brain structure and activity, but also indirectly, through neural studies performed on other clinical cases and neurotypical individuals, inspired and constrained neural models of human face recognition. In the dominant right hemisphere for face recognition in humans, PS's main lesion concerns (inputs to) the inferior occipital gyrus (IOG), in a region where face-selective activity is typically found in normal individuals ('Occipital Face Area', OFA). Her case study initially supported the criticality of this region for face identity recognition (FIR) and provided the impetus for transcranial magnetic stimulation (TMS), intracerebral electrical stimulation, and cortical surgery studies that have generally supported this view. Despite PS's right IOG lesion, typical face-selectivity is found anteriorly in the middle portion of the fusiform gyrus, a hominoid structure (termed the right 'Fusiform Face Area', FFA) that is widely considered to be the most important region for human face recognition. This finding led to the original proposal of direct anatomico-functional connections from early visual cortices to the FFA, bypassing the IOG/OFA (lulu), a hypothesis supported by further neuroimaging studies of PS, other neurological cases and neuro-typical individuals with original visual stimulation paradigms, data recordings and analyses. The proposal of a lack of sensitivity to face identity in PS's right FFA due to defective reentrant inputs from the IOG/FFA has also been supported by other cases, functional connectivity and cortical surgery studies. Overall, neural studies of, and based on, the case of prosopagnosia PS strongly question the hierarchical organization of the human neural face recognition system, supporting a more flexible and dynamic view of this key social brain function.

Face learning via brief real-world social interactions includes changes in face-selective brain areas and hippocampus

Making new acquaintances requires learning to recognise previously unfamiliar faces. In the current study, we investigated this process by staging real-world social interactions between actors and the participants. Participants completed a face-matching behavioural task in which they matched photographs of the actors (whom they had yet to meet), or faces similar to the actors (henceforth called foils). Participants were then scanned using functional magnetic resonance imaging (fMRI) while viewing photographs of actors and foils. Immediately after exiting the scanner, participants met the actors for the first time and interacted with them for 10 min. On subsequent days, participants completed a second behavioural experiment and then a second fMRI scan. Prior to each session, actors again interacted with the participants for 10 min. Behavioural results showed that social interactions improved performance accuracy when matching actor photographs, but not foil photographs. The fMRI analysis revealed a difference in the neural response to actor photographs and foil photographs across all regions of interest (ROIs) only after social interactions had occurred. Our results demonstrate that short social interactions were sufficient to learn and discriminate previously unfamiliar individuals. Moreover, these learning effects were present in brain areas involved in face processing and memory.

Familiarity, orientation, and realism increase face uncanniness by sensitizing to facial distortions

The uncanny valley predicts aversive reactions toward near-humanlike entities. Greater uncanniness is elicited by distortions in realistic than unrealistic faces, possibly due to familiarity. Experiment 1 investigated how familiarity and inversion affect uncanniness of facial distortions and the ability to detect differences between the distorted variants of the same face (distortion sensitivity). Familiar or unfamiliar celebrity faces were incrementally distorted and presented either upright or inverted. Uncanniness ratings increased across the distortion levels, and were stronger for familiar and upright faces. Distortion sensitivity increased with increasing distortion difference levels, again stronger for familiar and upright faces. Experiment 2 investigated how face realism, familiarity, and face orientation interacted for the increase of uncanniness across distortions. Realism increased the increase of uncanniness across the distortion levels, further enhanced by upright orientation and familiarity. The findings show that familiarity, upright orientation, and high face realism increase the sensitivity of uncanniness, likely by increasing distortion sensitivity. Finally, a moderated linear function of face realism and deviation level could explain the uncanniness of stimuli better than a quadratic function. A re-interpretation of the uncanny valley as sensitivity toward deviations from familiarized patterns is discussed.

Face-evoked thoughts

The thoughts that come to mind when viewing a face depend partly on the face and partly on the viewer. This basic interaction raises the question of how much common ground there is in face-evoked thoughts, and how this compares to viewers' expectations. Previous analyses have focused on early perceptual stages of face processing. Here we take a more expansive approach that encompasses later associative stages. In Experiment 1 (free association), participants exhibited strong egocentric bias, greatly overestimating the extent to which other people's thoughts resembled their own. In Experiment 2, we show that viewers' familiarity with a face can be decoded from their face-evoked thoughts. In Experiment 3 (person association), participants reported who came to mind when viewing a face-a task that emphasises connections in a social network rather than nodes. Here again, viewers' estimates of common ground exceeded actual common ground by a large margin. We assume that a face elicits much the same thoughts in other people as it does in us, but that is a mistake. In this respect, we are more isolated than we think.

Evidence for a General Neural Signature of Face Familiarity

We explored the neural signatures of face familiarity using cross-participant and cross-experiment decoding of event-related potentials, evoked by unknown and experimentally familiarized faces from a set of experiments with different participants, stimuli, and familiarization-types. Human participants of both sexes were either familiarized perceptually, via media exposure, or by personal interaction. We observed significant cross-experiment familiarity decoding involving all three experiments, predominantly over posterior and central regions of the right hemisphere in the 270-630 ms time window. This shared face familiarity effect was most prominent across the Media and the Personal, as well as between the Perceptual and Personal experiments. Cross-experiment decodability makes this signal a strong candidate for a general neural indicator of face familiarity, independent of familiarization methods, participants, and stimuli. Furthermore, the sustained pattern of temporal generalization suggests that it reflects a single automatic processing cascade that is maintained over time.

Detecting a viewer's familiarity with a face: Evidence from event-related brain potentials and classifier analyses

Human observers recognize the faces of people they know efficiently and without apparent effort. Consequently, recognizing a familiar face is often assumed to be an automatic process beyond voluntary control. However, there are circumstances in which a person might seek to hide their recognition of a particular face. The present study therefore used event-related potentials (ERPs) and a classifier based on logistic regression to determine if it is possible to detect whether a viewer is familiar with a particular face, regardless of whether the participant is willing to acknowledge it or not. In three experiments, participants were presented with highly variable "ambient" images of personally familiar and unfamiliar faces, while performing an incidental butterfly detection task (Experiment 1), an explicit familiarity judgment task (Experiment 2), and a concealed familiarity task in which they were asked to deny familiarity with one truly known facial identity while acknowledging familiarity with a second known identity (Experiment 3). In all three experiments, we observed substantially more negative ERP amplitudes at occipito-temporal electrodes for familiar relative to unfamiliar faces starting approximately 200 ms after stimulus onset. Both the earlier N250 familiarity effect, reflecting visual recognition of a known face, and the later sustained familiarity effect, reflecting the integration of visual with additional identity-specific information, were similar across experiments and thus independent of task demands. These results were further supported by the classifier analysis. We conclude that ERP correlates of familiar face recognition are largely independent of voluntary control and discuss potential applications in forensic settings.

Who speaks next? Adaptations to speaker identity in processing spoken sentences

When listening to a speaker, we need to adapt to her individual speaking characteristics, such as error proneness, accent, etc. The present study investigated two aspects of adaptation to speaker identity during processing spoken sentences in multi-speaker situations: the effect of speaker sequence across sentences and the effect of learning speaker-specific error probability. Spoken sentences were presented, cued, and accompanied by one of three portraits that were labeled as the speakers' faces. In Block 1 speaker-specific probabilities of syntax errors were 10%, 50%, or 90%; in Block 2 they were uniformly 50%. In both blocks, speech errors elicited P600 effects in the scalp recorded ERP. We found a speaker sequence effect only in Block 1: the P600 to target words was larger after speaker switches than after speaker repetitions, independent of sentence correctness. In Block 1, listeners showed higher accuracy in judging sentence correctness spoken by speakers with lower error proportions. No speaker-specific differences in target word P600 and accuracy were found in Block 2. When speakers differ in error proneness, listeners seem to flexibly adapt their speech processing for the upcoming sentence through attention reorientation and resource reallocation if the speaker is about to change, and through proactive maintenance of neural resources if the speaker remains the same.

Faces are not always special for attention: Effects of response-relevance and identity

Research over the past 25 years indicates that stimulus processing is diminished when attention is engaged in a perceptually demanding task of high 'perceptual load'. These results have generalized across a variety of stimulus categories, but a controversy evolved over the question of whether perception of distractor faces (or other categories of perceptual expertise) can proceed irrespective of the level of perceptual load in the attended task. Here we identify task-relevance, and in particular identity-relevance, as a potentially important factor in explaining prior inconsistencies. In four experiments, we tested whether perceptual load in an attended letter or word task modulates the processing of famous face distractors, while varying their task-relevance. Distractor interference effects on task RTs was reduced by perceptual load not only when the faces were entirely task-irrelevant, but also when the face gender was task relevant, within a name gender classification response-competition task, using famous female or male distractor faces. However, when the identity associated with the famous faces was primed by the task using their names, as in prior demonstrations that face distractors are immune to the effects of perceptual load, we were able to replicate these prior findings. Our findings demonstrate a role for identity-priming by the relevant task in determining attentional capture by faces under high perceptual load. Our results also highlight the importance of considering even relatively subtle forms of task-relevance in selective attention research.

Getting to Know You: Emerging Neural Representations during Face Familiarization

The successful recognition of familiar persons is critical for social interactions. Despite extensive research on the neural representations of familiar faces, we know little about how such representations unfold as someone becomes familiar. In three EEG experiments on human participants of both sexes, we elucidated how representations of face familiarity and identity emerge from different qualities of familiarization: brief perceptual exposure (Experiment 1), extensive media familiarization (Experiment 2), and real-life personal familiarization (Experiment 3). Time-resolved representational similarity analysis revealed that familiarization quality has a profound impact on representations of face familiarity: they were strongly visible after personal familiarization, weaker after media familiarization, and absent after perceptual familiarization. Across all experiments, we found no enhancement of face identity representation, suggesting that familiarity and identity representations emerge independently during face familiarization. Our results emphasize the importance of extensive, real-life familiarization for the emergence of robust face familiarity representations, constraining models of face perception and recognition memory.SIGNIFICANCE STATEMENT Despite extensive research on the neural representations of familiar faces, we know little about how such representations unfold as someone becomes familiar. To elucidate how face representations change as we get familiar with someone, we conducted three EEG experiments where we used brief perceptual exposure, extensive media familiarization, or real-life personal familiarization. Using multivariate representational similarity analysis, we demonstrate that the method of familiarization has a profound impact on face representations, and emphasize the importance of real-life familiarization. Additionally, familiarization shapes representations of face familiarity and identity differently: as we get to know someone, familiarity signals seem to appear before the formation of identity representations.

The N250 event-related potential as an index of face familiarity: a replication study

The neural correlates of face individuation-the acquisition of memory representations for novel faces-have been studied only in coarse detail and disregarding individual differences between learners. In their seminal study, Tanaka et al. (Tanaka et al. 2006 J. Cogn. Neurosci. 18, 1488-1497. (doi:10.1162/jocn.2006.18.9.1488)) required the identification of a particular novel face across 70 trials and found that the N250 component in the EEG event-related potentials became more negative from the first to the second half of the experiment, where it reached a similar amplitude as a well-known face. We were unable to directly replicate this finding in our study when we used the original split of trials. However, when we applied a different split of trials we observed very similar changes in N250 amplitude. We conclude that the N250 component is indeed sensitive to the build-up of a robust representation of a face in memory; the time course of this process appears to vary as a function of variables that may be determined in future research.

Inhibition of the occipital face area modulates the electrophysiological signals of face familiarity: A combined cTBS-EEG study

The occipital face area (OFA) is hierarchically one of the first stages of the face processing network. It has originally been thought to be involved in early, structural processing steps, but currently more and more studies challenge this view and propose that it also takes part in higher level face processing, such as identification and recognition. Here we tested whether the OFA is involved in the initial steps of recognition memory and plays a causal role in the differential processing of familiar and unfamiliar faces. We used an offline, inhibitory continuous theta-burst stimulation (cTBS) protocol over the right OFA and the vertex as control site. Electroencephalographic (EEG) recording of event-related potentials (ERPs), elicited by visually presented familiar (famous) and unfamiliar faces was performed before and after stimulation. We observed a difference in ERPs for famous and unfamiliar faces in a time-window corresponding to the N250 component. Importantly, this difference was significantly increased by cTBS of the right OFA, suggesting its causal role in the differential processing of familiar and unfamiliar faces. The enhancement occurred focally, at electrodes close to the right hemispheric cTBS site, as well as over similar occipito-temporal sites of the contralateral hemisphere. To the best of our knowledge, this is the first study showing the causal role of the rOFA in the differential processing of familiar and unfamiliar faces, using combined cTBS and EEG recording methods. These results are discussed with respect to the nature of familiar face representations, supported by an extensive, bilateral network.

Perceptual difficulty modulates the direction of information flow in familiar face recognition

Humans are fast and accurate when they recognize familiar faces. Previous neurophysiological studies have shown enhanced representations for the dichotomy of familiar vs. unfamiliar faces. As familiarity is a spectrum, however, any neural correlate should reflect graded representations for more vs. less familiar faces along the spectrum. By systematically varying familiarity across stimuli, we show a neural familiarity spectrum using electroencephalography. We then evaluated the spatiotemporal dynamics of familiar face recognition across the brain. Specifically, we developed a novel informational connectivity method to test whether peri-frontal brain areas contribute to familiar face recognition. Results showed that feed-forward flow dominates for the most familiar faces and top-down flow was only dominant when sensory evidence was insufficient to support face recognition. These results demonstrate that perceptual difficulty and the level of familiarity influence the neural representation of familiar faces and the degree to which peri-frontal neural networks contribute to familiar face recognition.

Recognising Familiar Faces Out of Context

Expertise in familiar face recognition has been well-documented in several studies. Here, we examined the role of context using a surprise lecturer recognition test. Across two experiments, we found few students recognised their lecturer when they were unexpected, but accuracy was higher when the lecturer was preceded by a prompt. Our findings suggest that familiar face recognition can be poor in unexpected contexts.

The N170 is Sensitive to Long-term (Personal) Familiarity of a Face Identity

The N170 is a large deflection of the human electroencephalogram (EEG), peaking at about 170 milliseconds over the occipito-temporal cortex after the sudden onset of a face stimulus. The N170 reflects perceptual awareness of a face and its onset corresponds to the emergence of reliable face-selectivity in the human brain. However, whether sensitivity to the long-term familiarity of a face identity emerges already at this early time-point remains debated. Here we provide a brief survey of the 45 published studies comparing the N170 response to unfamiliar and familiar (famous, experimentally familiarized, personally familiar and own) faces. Even though effects of familiarity on the N170 are relatively small and inconsistent across studies, this overview indicates that face familiarity significantly increases the N170 amplitude. This effect is especially present for personally familiar faces, learned in natural conditions. In the human brain, effects linked to familiarity with specific facial identities therefore appear to emerge between 150 and 200 ms in occipito-temporal brain regions, i.e., shortly after the onset of face-selectivity but at the same time as the earliest high-level effects of immediate unfamiliar face identity repetition. This observation challenges standard neurocognitive models with a clear-cut distinction between perceptual and memory stages in human face recognition.

Temporal Dynamics of the Neural Representation of Social Relationships

Humans can rapidly encode information from faces to support social judgments and facilitate interactions with others. We can also recall complex knowledge about those individuals, such as their social relationships with others, but the time course of this process has not been examined in detail. This study addressed the temporal dynamics of emerging visual and social relationship information using EEG and representational similarity analysis. Participants (female = 23, male = 10) became familiar with a 10-person social network, and were then shown faces of that network's members while EEG was recorded. To examine the temporal dynamics of the cognitive processes related to face perception, we compared the similarity structure of neural pattern responses to models of visual processing, face shape similarity, person identity, and social relationships. We found that all types of information are associated with neural patterns after a face is seen. Visual models became significant early after image onset, and identity across a change in facial expression was uniquely associated with neural patterns at several points throughout the time course. Additionally, a model reflecting perceived frequency of social interaction was present beginning at ∼110 ms, even in the absence of an explicit task to think about the relationships among the network members. This study highlights the speed and salience of social information relating to group dynamics that are present in the brain during person perception.SIGNIFICANCE STATEMENT We live our lives in social groups where complex relationships form among and around us. It is likely that some of the information about social relationships that we observe is integral during person perception, to better help us interact in differing situations with a variety of people. However, when exactly this information becomes relevant has been unclear. In this study, we present evidence that information reflecting observed relationships among a social network is spontaneously represented in whole-brain patterns shortly following presentation of a face. These results are consistent with neuroimaging studies showing spontaneous spatial representation of social network characteristics, and contribute novel insights into the timing of these neural processes.

Mechanisms of face specificity - Differentiating speed and accuracy in face cognition by event-related potentials of central processing

Given the crucial role of face recognition in social life, it is hardly surprising that cognitive processes specific for faces have been identified. In previous individual differences studies, the speed (measured in easy tasks) and accuracy (difficult tasks) of face cognition (FC, involving perception and recognition of faces) have been shown to form distinct abilities, going along with divergent factorial structures. This result has been replicated, but remained unexplained. To fill this gap, we first parameterized the sub-processes underlying speed vs. accuracy in easy and difficult memory tasks for faces and houses in a large sample. Then, we analyzed event-related potentials (ERPs) extracted from the EEG by using residue iteration decomposition (RIDE), yielding a central (C) component that is comparable to a purified P300. Structural equation modeling (SEM) was applied to estimate face specificity of C component latencies and amplitudes. If performance in easy tasks relies on purely general processes that are insensitive to stimulus content, there should be no specificity of individual differences in the latency recorded in easy tasks. However, in difficult tasks specificity was expected. Results indicated that, contrary to our predictions, specificity occurred in the C component latency of both speed-based and accuracy-based measures, but was stronger in accuracy. Further analyses suggested specific relationships between the face-related C latency and FC ability. Finally, we detected specificity in RTs of easy tasks when single tasks were modeled, but not when multiple tasks were jointly modeled. This suggests that the mechanisms leading to face specificity in performance speed are distinct across tasks.

Eye spy a liar: assessing the utility of eye fixations and confidence judgments for detecting concealed recognition of faces, scenes and objects

BACKGROUND

In criminal investigations, uncooperative witnesses might deny knowing a perpetrator, the location of a murder scene or knowledge of a weapon. We sought to identify markers of recognition in eye fixations and confidence judgments whilst participants told the truth and lied about recognising faces (Experiment 1) and scenes and objects (Experiment 2) that varied in familiarity. To detect recognition we calculated effect size differences in markers of recognition between familiar and unfamiliar items that varied in familiarity (personally familiar, newly learned).

RESULTS

In Experiment 1, recognition of personally familiar faces was reliably detected across multiple fixation markers (e.g. fewer fixations, fewer interest areas viewed, fewer return fixations) during honest and concealed recognition. In Experiment 2, recognition of personally familiar non-face items (scenes and objects) was detected solely by fewer fixations during honest and concealed recognition; differences in other fixation measures were not consistent. In both experiments, fewer fixations exposed concealed recognition of newly learned faces, scenes and objects, but the same pattern was not observed during honest recognition. Confidence ratings were higher for recognition of personally familiar faces than for unfamiliar faces.

CONCLUSIONS

Robust memories of personally familiar faces were detected in patterns of fixations and confidence ratings, irrespective of task demands required to conceal recognition. Crucially, we demonstrate that newly learned faces should not be used as a proxy for real-world familiarity, and that conclusions should not be generalised across different types of familiarity or stimulus class.