Familiar Face Detection in 180 ms

Shared neural codes of recognition memory

Recognition memory research has identified several electrophysiological indicators of successful memory retrieval, known as old-new effects. These effects have been observed in different sensory domains using various stimulus types, but little attention has been given to their similarity or distinctiveness and the underlying processes they may share. Here, a data-driven approach was taken to investigate the temporal evolution of shared information content between different memory conditions using openly available EEG data from healthy human participants of both sexes, taken from six experiments. A test dataset involving personally highly familiar and unfamiliar faces was used. The results show that neural signals of recognition memory for face stimuli were highly generalized starting from around 200 ms following stimulus onset. When training was performed on non-face datasets, an early (around 200-300 ms) to late (post-400 ms) differentiation was observed over most regions of interest. Successful cross-classification for non-face stimuli (music and object/scene associations) was most pronounced in late period. Additionally, a striking dissociation was observed between familiar and remembered objects, with shared signals present only in the late window for correctly remembered objects, while cross-classification for familiar objects was successful in the early period as well. These findings suggest that late neural signals of memory retrieval generalize across sensory modalities and stimulus types, and the dissociation between familiar and remembered objects may provide insight into the underlying processes.

Neuronal response of the primate striatum tail to face of socially familiar persons

Recent studies have suggested that the basal ganglia, the center of stimulus-reward associative learning, are involved in social behavior. However, the role of the basal ganglia in social information processing remains unclear. Here, we demonstrate that the striatum tail (STRt) in macaque monkeys, which is sensitive to visual objects with long-term reward history (i.e., stable object value), is also sensitive to socially familiar persons. Many STRt neurons responded to face images of persons, especially those who took daily care of the subject monkeys. These face-responsive neurons also encoded stable object value. The strength of the neuronal modulation of social familiarity and stable object value biases were positively correlated. These results suggest that both social familiarity and stable object value information are mediated by a common neuronal mechanism. Thus, the representation of social information is linked to reward information in the STRt, not in the dedicated social information circuit.

Fast saccades to faces during the feedforward sweep

Saccadic choice tasks use eye movements as a response method, typically in a task where observers are asked to saccade as quickly as possible to an image of a prespecified target category. Using this approach, face-selective saccades have been observed within 100 ms poststimulus. When taking into account oculomotor processing, this suggests that faces can be detected in as little as 70 to 80 ms. It has therefore been suggested that face detection must occur during the initial feedforward sweep, since this latency leaves little time for feedback processing. In the current experiment, we tested this hypothesis using backward masking-a technique shown to primarily disrupt feedback processing while leaving feedforward activation relatively intact. Based on minimum saccadic reaction time, we found that face detection benefited from ultra-fast, accurate saccades within 110 to 160 ms and that these eye movements are obtainable even under extreme masking conditions that limit perceptual awareness. However, masking did significantly increase the median SRT for faces. In the manual responses, we found remarkable detection accuracy for faces and houses, even when participants indicated having no visual experience of the test images. These results provide evidence for the view that the saccadic bias to faces is initiated by coarse information used to categorize faces in the feedforward sweep but that, in most cases, additional processing is required to quickly reach the threshold for saccade initiation.

Modeling naturalistic face processing in humans with deep convolutional neural networks

Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The ways in which the internal face representations in DCNNs relate to human cognitive representations and brain activity are not well understood. Nearly all previous studies focused on static face image processing with rapid display times and ignored the processing of naturalistic, dynamic information. To address this gap, we developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces). We used this naturalistic dataset to compare representational geometries estimated from DCNNs, behavioral responses, and brain responses. We found that DCNN representational geometries were consistent across architectures, cognitive representational geometries were consistent across raters in a behavioral arrangement task, and neural representational geometries in face areas were consistent across brains. Representational geometries in late, fully connected DCNN layers, which are optimized for individuation, were much more weakly correlated with cognitive and neural geometries than were geometries in late-intermediate layers. The late-intermediate face-DCNN layers successfully matched cognitive representational geometries, as measured with a behavioral arrangement task that primarily reflected categorical attributes, and correlated with neural representational geometries in known face-selective topographies. Our study suggests that current DCNNs successfully capture neural cognitive processes for categorical attributes of faces but less accurately capture individuation and dynamic features.

The effect of familiarity on behavioral oscillations in face perception

Studies on behavioral oscillations demonstrate that visual sensitivity fluctuates over time and visual processing varies periodically, mirroring neural oscillations at the same frequencies. Do these behavioral oscillations reflect fixed and relatively automatic sensory sampling, or top-down processes such as attention or predictive coding? To disentangle these theories, the current study used a dual-target rapid serial visual presentation paradigm, where participants indicated the gender of a face target embedded in streams of distractors presented at 30 Hz. On critical trials, two identical targets were presented with varied stimulus onset asynchrony from 200 to 833 ms. The target was either familiar or unfamiliar faces, divided into different blocks. We found a 4.6 Hz phase-coherent fluctuation in gender discrimination performance across both trial types, consistent with previous reports. In addition, however, we found an effect at the alpha frequency, with behavioral oscillations in the familiar blocks characterized by a faster high-alpha peak than for the unfamiliar face blocks. These results are consistent with the combination of both a relatively stable modulation in the theta band and faster modulation of the alpha oscillations. Therefore, the overall pattern of perceptual sampling in visual perception may depend, at least in part, on task demands. PROTOCOL REGISTRATION: The stage 1 protocol for this Registered Report was accepted in principle on 16/08/2022. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/A98UF .

Familiarity Facilitates Detection of Angry Expressions

Personal familiarity facilitates rapid and optimized detection of faces. In this study, we investigated whether familiarity associated with faces can also facilitate the detection of facial expressions. Models of face processing propose that face identity and face expression detection are mediated by distinct pathways. We used a visual search paradigm to assess if facial expressions of emotion (anger and happiness) were detected more rapidly when produced by familiar as compared to unfamiliar faces. We found that participants detected an angry expression 11% more accurately and 135 ms faster when produced by familiar as compared to unfamiliar faces while happy expressions were detected with equivalent accuracies and at equivalent speeds for familiar and unfamiliar faces. These results suggest that detectors in the visual system dedicated to processing features of angry expressions are optimized for familiar faces.

Intracerebral Electrophysiological Recordings to Understand the Neural Basis of Human Face Recognition

Understanding how the human brain recognizes faces is a primary scientific goal in cognitive neuroscience. Given the limitations of the monkey model of human face recognition, a key approach in this endeavor is the recording of electrophysiological activity with electrodes implanted inside the brain of human epileptic patients. However, this approach faces a number of challenges that must be overcome for meaningful scientific knowledge to emerge. Here we synthesize a 10 year research program combining the recording of intracerebral activity (StereoElectroEncephaloGraphy, SEEG) in the ventral occipito-temporal cortex (VOTC) of large samples of participants and fast periodic visual stimulation (FPVS), to objectively define, quantify, and characterize the neural basis of human face recognition. These large-scale studies reconcile the wide distribution of neural face recognition activity with its (right) hemispheric and regional specialization and extend face-selectivity to anterior regions of the VOTC, including the ventral anterior temporal lobe (VATL) typically affected by magnetic susceptibility artifacts in functional magnetic resonance imaging (fMRI). Clear spatial dissociations in category-selectivity between faces and other meaningful stimuli such as landmarks (houses, medial VOTC regions) or written words (left lateralized VOTC) are found, confirming and extending neuroimaging observations while supporting the validity of the clinical population tested to inform about normal brain function. The recognition of face identity - arguably the ultimate form of recognition for the human brain - beyond mere differences in physical features is essentially supported by selective populations of neurons in the right inferior occipital gyrus and the lateral portion of the middle and anterior fusiform gyrus. In addition, low-frequency and high-frequency broadband iEEG signals of face recognition appear to be largely concordant in the human association cortex. We conclude by outlining the challenges of this research program to understand the neural basis of human face recognition in the next 10 years.

Superior neural individuation of mother's than stranger's faces by five months of age

Human adults are better at recognizing different views of a given face as belonging to the same person when that person is familiar rather than unfamiliar. To clarify the developmental origin of this well-established phenomenon, one group of five-month-olds (N = 22) was presented with pictures of four different unfamiliar female faces at a fixed rate (6 Hz, 166 msec stimulus onset asynchrony), interrupted every 5th stimulus (1.2 Hz) by either their mother's face (mother oddball condition) or, in different stimulation sequences, a stranger's face (stranger oddball condition). In another group of five-month-olds (N = 17), stimulation sequences were reversed such that their mothers' or a given stranger's face were repeated at 6 Hz and interrupted every 5 stimuli by pictures of different female faces (mother standard, stranger standard conditions, respectively). Twelve variable images of each identity served as stimulus material. Besides clear frequency-tagged EEG responses at the 6 Hz stimulation rate over the medial occipital region in all conditions, significant activity at 1.2 Hz and harmonics (2.4 Hz, etc.) was observed in this region, reflecting selective responses to facial identity across changes of views. This effect was strongest when the mother's face was immediately repeated at every stimulation cycle (mother standard). Overall, these observations point to an early developmental advantage of identifying a familiar face presented from different views during immediate stimulus repetition.

High-level visual search in children with autism

Visual search has been classified as easy feature search, with rapid target detection and little set size dependence, versus slower difficult search with focused attention, with set size–dependent speed. Reverse hierarchy theory attributes these classes to rapid high cortical-level vision at a glance versus low-level vision with scrutiny, attributing easy search to high-level representations. Accordingly, faces “pop out” of heterogeneous object photographs. Individuals with autism have difficulties recognizing faces, and we now asked if this disability disturbs their search for faces. We compare search times and set size slopes for children with autism spectrum disorders (ASDs) and those with neurotypical development (NT) when searching for faces. Human face targets were found rapidly, with shallow set size slopes. The between-group difference between slopes (18.8 vs. 11.3 ms/item) is significant, suggesting that faces may not “pop out” as easily, but in our view does not warrant classifying ASD face search as categorically different from that of NT children. We also tested search for different target categories, dog and lion faces, and nonface basic categories, cars and houses. The ASD group was generally a bit slower than the NT group, and their slopes were somewhat steeper. Nevertheless, the overall dependencies on target category were similar: human face search fastest, nonface categories slowest, and dog and lion faces in between. We conclude that autism may spare vision at a glance, including face detection, despite its reported effects on face recognition, which may require vision with scrutiny. This dichotomy is consistent with the two perceptual modes suggested by reverse hierarchy theory.

Infrequent faces bias social attention differently in manual and oculomotor measures

Although attention is thought to be spontaneously biased by social cues like faces and eyes, recent data have demonstrated that when extraneous content, context, and task factors are controlled, attentional biasing is abolished in manual responses while still occurring sparingly in oculomotor measures. Here, we investigated how social attentional biasing was affected by face novelty by measuring responses to frequently presented (i.e., those with lower novelty) and infrequently presented (i.e., those with higher novelty) face identities. Using a dot-probe task, participants viewed either the same face and house identity that was frequently presented on half of the trials or sixteen different face and house identities that were infrequently presented on the other half of the trials. A response target occurred with equal probability at the previous location of the eyes or mouth of the face or the top or bottom of the house. Experiment 1 measured manual responses to the target while participants maintained central fixation. Experiment 2 additionally measured participants' natural oculomotor behaviour when their eye movements were not restricted. Across both experiments, no evidence of social attentional biasing was found in manual data. However, in Experiment 2, there was a reliable oculomotor bias towards the eyes of infrequently presented upright faces. Together, these findings suggest that face novelty does not facilitate manual measures of social attention, but it appears to promote spontaneous oculomotor biasing towards the eyes of infrequently presented novel faces.

Shared neural codes for visual and semantic information about familiar faces in a common representational space

Our brain processes faces of close others differently than faces of visually familiar individuals. While both types of faces activate similar visual areas, faces of close others activate areas involved in processing social and semantic information. Here, we used between-subject linear classifiers trained on hyperaligned brain data to investigate the neural code for visual and semantic information about familiar others. The identity of both visually and personally familiar faces could be decoded across participants from brain activity in visual areas. Instead, only the identity of personally familiar faces could be decoded in areas involved in social cognition. Our results suggest that individually distinctive information associated with familiar faces is embedded in a neural code that is shared across brains.

Processes evoked by seeing a personally familiar face encompass recognition of visual appearance and activation of social and person knowledge. Whereas visual appearance is the same for all viewers, social and person knowledge may be more idiosyncratic. Using between-subject multivariate decoding of hyperaligned functional magnetic resonance imaging data, we investigated whether representations of personally familiar faces in different parts of the distributed neural system for face perception are shared across individuals who know the same people. We found that the identities of both personally familiar and merely visually familiar faces were decoded accurately across brains in the core system for visual processing, but only the identities of personally familiar faces could be decoded across brains in the extended system for processing nonvisual information associated with faces. Our results show that personal interactions with the same individuals lead to shared neural representations of both the seen and unseen features that distinguish their identities.

Rapid saccadic categorization of other-race faces

The human visual system is very fast and efficient at extracting socially relevant information from faces. Visual studies employing foveated faces have consistently reported faster categorization by race response times for other-race compared with same-race faces. However, in everyday life we typically encounter faces outside the foveated visual field. In study 1, we explored whether and how race is categorized extrafoveally in same- and other-race faces normalized for low-level properties by tracking eye movements of Western Caucasian and East Asian observers in a saccadic response task. The results show that not only are people sensitive to race in faces presented outside of central vision, but the speed advantage in categorizing other-race faces occurs astonishingly quickly in as little as 200 ms. Critically, this visual categorization process was approximately 300 ms faster than the typical button press responses on centrally presented foveated faces. Study 2 investigated the genesis of the extrafoveal saccadic response speed advantage by comparing the influences of the response modality (button presses and saccadic responses), as well as the potential contribution of the impoverished low-spatial frequency spectrum characterizing extrafoveal visual information processing. Button press race categorization was not significantly faster with reconstructed retinal-filtered low spatial frequency faces, regardless of the visual field presentation. The speed of race categorization was significantly boosted only by extrafoveal saccades and not centrally foveated faces. Race is a potent, rapid, and effective visual signal transmitted by faces used for the categorization of ingroup/outgroup members. This fast universal visual categorization can occur outside central vision, igniting a cascade of social processes.

Coarse-to-Fine(r) Automatic Familiar Face Recognition in the Human Brain

At what level of spatial resolution can the human brain recognize a familiar face in a crowd of strangers? Does it depend on whether one approaches or rather moves back from the crowd? To answer these questions, 16 observers viewed different unsegmented images of unfamiliar faces alternating at 6 Hz, with spatial frequency (SF) content progressively increasing (i.e., coarse-to-fine) or decreasing (fine-to-coarse) in different sequences. Variable natural images of celebrity faces every sixth stimulus generated an objective neural index of single-glanced automatic familiar face recognition (FFR) at 1 Hz in participants' electroencephalogram (EEG). For blurry images increasing in spatial resolution, the neural FFR response over occipitotemporal regions emerged abruptly with additional cues at about 6.3-8.7 cycles/head width, immediately reaching amplitude saturation. When the same images progressively decreased in resolution, the FFR response disappeared already below 12 cycles/head width, thus providing no support for a predictive coding hypothesis. Overall, these observations indicate that rapid automatic recognition of heterogenous natural views of familiar faces is achieved from coarser visual inputs than generally thought, and support a coarse-to-fine FFR dynamics in the human brain.

Are You Able to Trust Me? Analysis of the Relationships Between Personality Traits and the Assessment of Attractiveness and Trust

Behavioral and neuroimaging studies show that people trust and collaborate with others based on a quick assessment of the facial appearance. Based on the morphological characteristics of the face, i.e., features, shape, or color, it is possible to determine health, attractiveness, trust, and some personality traits. The study attempts to indicate the features influencing the perception of attractiveness and trust. In order to select individual factors, a model of backward stepwise logistic regression was used, analyzing the results of the psychological tests and the attractiveness and trust survey. Statistical analysis made it possible to select the most important personality traits related to attractiveness and trust assessments.

A fast link between face perception and memory in the temporal pole

The question of how the brain recognizes the faces of familiar individuals has been important throughout the history of neuroscience. Cells linking visual processing to person memory have been proposed but not found. Here, we report the discovery of such cells through recordings from an area in the macaque temporal pole identified with functional magnetic resonance imaging. These cells responded to faces that were personally familiar. They responded nonlinearly to stepwise changes in face visibility and detail and holistically to face parts, reflecting key signatures of familiar face recognition. They discriminated between familiar identities, as fast as a general face identity area. The discovery of these cells establishes a new pathway for the fast recognition of familiar individuals.

Getting to Know You: Emerging Neural Representations during Face Familiarization

The successful recognition of familiar persons is critical for social interactions. Despite extensive research on the neural representations of familiar faces, we know little about how such representations unfold as someone becomes familiar. In three EEG experiments on human participants of both sexes, we elucidated how representations of face familiarity and identity emerge from different qualities of familiarization: brief perceptual exposure (Experiment 1), extensive media familiarization (Experiment 2), and real-life personal familiarization (Experiment 3). Time-resolved representational similarity analysis revealed that familiarization quality has a profound impact on representations of face familiarity: they were strongly visible after personal familiarization, weaker after media familiarization, and absent after perceptual familiarization. Across all experiments, we found no enhancement of face identity representation, suggesting that familiarity and identity representations emerge independently during face familiarization. Our results emphasize the importance of extensive, real-life familiarization for the emergence of robust face familiarity representations, constraining models of face perception and recognition memory.SIGNIFICANCE STATEMENT Despite extensive research on the neural representations of familiar faces, we know little about how such representations unfold as someone becomes familiar. To elucidate how face representations change as we get familiar with someone, we conducted three EEG experiments where we used brief perceptual exposure, extensive media familiarization, or real-life personal familiarization. Using multivariate representational similarity analysis, we demonstrate that the method of familiarization has a profound impact on face representations, and emphasize the importance of real-life familiarization. Additionally, familiarization shapes representations of face familiarity and identity differently: as we get to know someone, familiarity signals seem to appear before the formation of identity representations.

Rapid Neural Representations of Personally Relevant Faces

The faces of those most personally relevant to us are our primary source of social information, making their timely perception a priority. Recent research indicates that gender, age and identity of faces can be decoded from EEG/MEG data within 100 ms. Yet, the time course and neural circuitry involved in representing the personal relevance of faces remain unknown. We applied simultaneous EEG-fMRI to examine neural responses to emotional faces of female participants' romantic partners, friends, and a stranger. Combining EEG and fMRI in cross-modal representational similarity analyses, we provide evidence that representations of personal relevance start prior to structural encoding at 100 ms, with correlated representations in visual cortex, but also in prefrontal and midline regions involved in value representation, and monitoring and recall of self-relevant information. Our results add to an emerging body of research that suggests that models of face perception need to be updated to account for rapid detection of personal relevance in cortical circuitry beyond the core face processing network.

Intracerebral electrical stimulation to understand the neural basis of human face identity recognition

Recognizing people's identity by their faces is a key function in the human species, supported by regions of the ventral occipito-temporal cortex (VOTC). In the last decade, there have been several reports of perceptual face distortion during direct electrical stimulation (DES) with subdural electrodes positioned over a well-known face-selective VOTC region of the right lateral middle fusiform gyrus (LatMidFG; i.e., the "Fusiform Face Area", FFA). However, transient impairments of face identity recognition (FIR) have been extremely rare and only behaviorally quantified during DES with intracerebral (i.e., depth) electrodes in stereo-electroencephalography (SEEG). The three detailed cases reported so far, summarized here, were specifically impaired at FIR during DES inside different anatomical VOTC regions of the right hemisphere: the inferior occipital gyrus (IOG) and the LatMidFG, as well as a region that lies at the heart of a large magnetic susceptibility artifact in functional magnetic resonance imaging (fMRI): the anterior fusiform gyrus (AntFG). In the first two regions, the eloquent electrode contacts were systematically associated with the highest face-selective and (unfamiliar) face individuation responses as measured with intracerebral electrophysiology. Stimulation in the right AntFG did not lead to perceptual changes but also caused an inability to remember having been presented face pictures, as if the episode was never recorded in memory. These observations support the view of an extensive network of face-selective VOTC regions subtending human FIR, with at least three critical nodes in the right hemisphere associated with differential intrinsic and extrinsic patterns of reentrant connectivity.

Rapid and coarse face detection: With a lack of evidence for a nasal-temporal asymmetry

Humans have structures dedicated to the processing of faces, which include cortical components (e.g., areas in occipital and temporal lobes) and subcortical components (e.g., superior colliculus and amygdala). Although faces are processed more quickly than stimuli from other categories, there is a lack of consensus regarding whether subcortical structures are responsible for rapid face processing. In order to probe this, we exploited the asymmetry in the strength of projections to subcortical structures between the nasal and temporal hemiretina. Participants detected faces from unrecognizable control stimuli and performed the same task for houses. In Experiments 1 and 3, at the fastest reaction times, participants detected faces more accurately than houses. However, there was no benefit of presenting to the subcortical pathway. In Experiment 2, we probed the coarseness of the rapid pathway, making the foil stimuli more similar to faces and houses. This eliminated the rapid detection advantage, suggesting that rapid face processing is limited to coarse representations. In Experiment 4, we sought to determine whether the natural difference between spatial frequencies of faces and houses were driving the effects seen in Experiments 1 and 3. We spatially filtered the faces and houses so that they were matched. Better rapid detection was again found for faces relative to houses, but we found no benefit of preferentially presenting to the subcortical pathway. Taken together, the results of our experiments suggest a coarse rapid detection mechanism, which was not dependent on spatial frequency, with no advantage for presenting preferentially to subcortical structures.

An objective, sensitive and ecologically valid neural measure of rapid human individual face recognition

Humans may be the only species able to rapidly and automatically recognize a familiar face identity in a crowd of unfamiliar faces, an important social skill. Here, by combining electroencephalography (EEG) and fast periodic visual stimulation (FPVS), we introduce an ecologically valid, objective and sensitive neural measure of this human individual face recognition function. Natural images of various unfamiliar faces are presented at a fast rate of 6 Hz, allowing one fixation per face, with variable natural images of a highly familiar face identity, a celebrity, appearing every seven images (0.86 Hz). Following a few minutes of stimulation, a high signal-to-noise ratio neural response reflecting the generalized discrimination of the familiar face identity from unfamiliar faces is observed over the occipito-temporal cortex at 0.86 Hz and harmonics. When face images are presented upside-down, the individual familiar face recognition response is negligible, being reduced by a factor of 5 over occipito-temporal regions. Differences in the magnitude of the individual face recognition response across different familiar face identities suggest that factors such as exposure, within-person variability and distinctiveness mediate this response. Our findings of a biological marker for fast and automatic recognition of individual familiar faces with ecological stimuli open an avenue for understanding this function, its development and neural basis in neurotypical individual brains along with its pathology. This should also have implications for the use of facial recognition measures in forensic science.

How face perception unfolds over time

Within a fraction of a second of viewing a face, we have already determined its gender, age and identity. A full understanding of this remarkable feat will require a characterization of the computational steps it entails, along with the representations extracted at each. Here, we used magnetoencephalography (MEG) to measure the time course of neural responses to faces, thereby addressing two fundamental questions about how face processing unfolds over time. First, using representational similarity analysis, we found that facial gender and age information emerged before identity information, suggesting a coarse-to-fine processing of face dimensions. Second, identity and gender representations of familiar faces were enhanced very early on, suggesting that the behavioral benefit for familiar faces results from tuning of early feed-forward processing mechanisms. These findings start to reveal the time course of face processing in humans, and provide powerful new constraints on computational theories of face perception.

Familiarity revealed by involuntary eye movements on the fringe of awareness

Involuntary eye movements during fixation of gaze are typically transiently inhibited following stimulus onset. This oculomotor inhibition (OMI), which includes microsaccades and spontaneous eye blinks, is modulated by stimulus saliency and anticipation, but it is currently unknown whether it is sensitive to familiarity. To investigate this, we measured the OMI while observers passively viewed a slideshow of one familiar and 7 unfamiliar facial images presented briefly at 1 Hz in random order. Since the initial experiments indicated that OMI was occasionally insensitive to familiarity when the facial images were highly visible, and to prevent top-down strategies and potential biases, we limited visibility by backward masking making the faces barely visible or at the fringe of awareness. Under these conditions, we found prolonged inhibition of both microsaccades and eye-blinks, as well as earlier onset of microsaccade inhibition with familiarity. These findings demonstrate, for the first time, the sensitivity of OMI to familiarity. Because this is based on involuntary eye movements and can be measured on the fringe of awareness and in passive viewing, our results provide direct evidence that OMI can be used as a novel physiological measure for studying hidden memories with potential implications for health, legal, and security purposes.

Decisional space modulates visual categorization - Evidence from saccadic reaction times

Manual and saccadic reaction times (SRTs) have been used to determine the minimum time required for different types of visual categorizations. Such studies have demonstrated extremely rapid detection of faces within natural scenes, whereas increasingly complex decisions (i.e. levels of processing) require longer processing times. We reasoned that visual categorization speed is not only dependent on the processing level, but is further affected by decisional space constraints. In the context of two different tasks, observers performed choice saccades towards female (gender categorization) or personally familiar (familiarity categorization) faces. Additionally, familiarity categorizations were completed with stimulus sets that differed in the number of individuals presented (3 vs. 7 identities) to investigate the effect of decisional space constraints. We observe an inverse relationship between visual categorization proficiency and decisional space. Observers were most accurate for categorization of gender, which could be achieved in as little as 140 ms. Categorization of highly predictable targets was more error-prone and required an additional ∼100 ms processing time. Our findings add to an increasing body of evidence demonstraing that pre-activation of identity-information can modulate early visual processing in a top-down manner. They also emphasize the importance of considering procedural aspects, as well as terminology when aiming to characterize cognitive processes.

Idiosyncratic, Retinotopic Bias in Face Identification Modulated by Familiarity

The perception of gender and age of unfamiliar faces is reported to vary idiosyncratically across retinal locations such that, for example, the same androgynous face may appear to be male at one location but female at another. Here, we test spatial heterogeneity for the recognition of the identity of personally familiar faces in human participants. We found idiosyncratic biases that were stable within participants and that varied more across locations for low as compared to high familiar faces. These data suggest that like face gender and age, face identity is processed, in part, by independent populations of neurons monitoring restricted spatial regions and that the recognition responses vary for the same face across these different locations. Moreover, repeated and varied social interactions appear to lead to adjustments of these independent face recognition neurons so that the same familiar face is eventually more likely to elicit the same recognition response across widely separated visual field locations. We provide a mechanistic account of this reduced retinotopic bias based on computational simulations.

Distinct neural processes for the perception of familiar versus unfamiliar faces along the visual hierarchy revealed by EEG

Humans recognize faces with ease, despite the complexity of the task and of the visual system which underlies it. Different spatial regions, including both the core and extended face processing networks, and distinct temporal stages of processing have been implicated in face recognition, but there is ongoing controversy regarding the extent to which the mechanisms for recognizing a familiar face differ from those for an unfamiliar face. Here, we used electroencephalogram (EEG) and flicker SSVEP, a high signal-to-noise approach, and searchlight decoding methods to elucidate the mechanisms mediating the processing of familiar and unfamiliar faces in the time domain. Familiar and unfamiliar faces were presented periodically at 15 Hz, 6 Hz and 3.75 Hz either upright or inverted in separate blocks, with the rationale that faster frequencies require shorter processing times per image and tap into fundamentally different levels of visual processing. The 15 Hz trials, likely to reflect early visual processing, exhibited enhanced neural responses for familiar over unfamiliar face trials, but only when the faces were upright. In contrast, decoding methods revealed similar classification accuracies for upright and inverted faces for both familiar and unfamiliar faces. For the 6 Hz frequency, familiar faces had lower amplitude responses than unfamiliar faces, and decoding familiarity was more accurate for upright compared with inverted faces. Finally, the 3.75 Hz frequency revealed no main effects of familiarity, but decoding showed significant correlations with behavioral ratings of face familiarity, suggesting that activity evoked by this slow presentation frequency reflected higher-level, cognitive aspects of familiarity processing. This three-way dissociation between frequencies reveals that fundamentally different stages of the visual hierarchy are modulated by face familiarity. The combination of experimental and analytical approaches used here represent a novel method for elucidating spatio-temporal characteristics within the visual system.

The automaticity of face perception is influenced by familiarity

In this study, we explore the automaticity of encoding for different facial characteristics and ask whether it is influenced by face familiarity. We used a matching task in which participants had to report whether the gender, identity, race, or expression of two briefly presented faces was the same or different. The task was made challenging by allowing nonrelevant dimensions to vary across trials. To test for automaticity, we compared performance on trials in which the task instruction was given at the beginning of the trial, with trials in which the task instruction was given at the end of the trial. As a strong criterion for automatic processing, we reasoned that if perception of a given characteristic (gender, race, identity, or emotion) is fully automatic, the timing of the instruction should not influence performance. We compared automaticity for the perception of familiar and unfamiliar faces. Performance with unfamiliar faces was higher for all tasks when the instruction was given at the beginning of the trial. However, we found a significant interaction between instruction and task with familiar faces. Accuracy of gender and identity judgments to familiar faces was the same regardless of whether the instruction was given before or after the trial, suggesting automatic processing of these properties. In contrast, there was an effect of instruction for judgments of expression and race to familiar faces. These results show that familiarity enhances the automatic processing of some types of facial information more than others.

Decisional space determines saccadic reaction times in healthy observers and acquired prosopagnosia

Determining the familiarity and identity of a face have been considered as independent processes. Covert face recognition in cases of acquired prosopagnosia, as well as rapid detection of familiarity have been taken to support this view. We tested P.S. a well-described case of acquired prosopagnosia, and two healthy controls (her sister and daughter) in two saccadic reaction time (SRT) experiments. Stimuli depicted their family members and well-matched unfamiliar distractors in the context of binary gender, or familiarity decisions. Observers' minimum SRTs were estimated with Bayesian approaches. For gender decisions, P.S. and her daughter achieved sufficient performance, but displayed different SRT distributions. For familiarity decisions, her daughter exhibited above chance level performance and minimum SRTs corresponding to those reported previously in healthy observers, while P.S. performed at chance. These findings extend previous observations, indicating that decisional space determines performance in both the intact and impaired face processing system.

Conserved features of the primate face code

A recent paper demonstrated that the pattern of firing rates across ∼100 neurons in the anterior medial face patch is closely related to which human face (of 2,000) had been presented to a monkey [Chang L, Tsao DY (2017) Cell 169:1013-1028]. In addition, the firing rates for these neurons can be predicted for a novel human face. Although it is clear from this work that the firing rates of these face patch neurons encode faces, the properties of the face code have not yet been fully described. Based on an analysis of 98 neurons responding to 2,000 faces, I conclude that the anterior medial face patch uses a combinatorial rate code, one with an exponential distribution of neuron rates that has a mean rate conserved across faces. Thus, the face code is maximally informative (technically, maximum entropy) and is very similar to the code used by the fruit fly olfactory system.

The neural representation of personally familiar and unfamiliar faces in the distributed system for face perception

Personally familiar faces are processed more robustly and efficiently than unfamiliar faces. The human face processing system comprises a core system that analyzes the visual appearance of faces and an extended system for the retrieval of person-knowledge and other nonvisual information. We applied multivariate pattern analysis to fMRI data to investigate aspects of familiarity that are shared by all familiar identities and information that distinguishes specific face identities from each other. Both identity-independent familiarity information and face identity could be decoded in an overlapping set of areas in the core and extended systems. Representational similarity analysis revealed a clear distinction between the two systems and a subdivision of the core system into ventral, dorsal and anterior components. This study provides evidence that activity in the extended system carries information about both individual identities and personal familiarity, while clarifying and extending the organization of the core system for face perception.

Familiarity facilitates feature-based face processing

Recognition of personally familiar faces is remarkably efficient, effortless and robust. We asked if feature-based face processing facilitates detection of familiar faces by testing the effect of face inversion on a visual search task for familiar and unfamiliar faces. Because face inversion disrupts configural and holistic face processing, we hypothesized that inversion would diminish the familiarity advantage to the extent that it is mediated by such processing. Subjects detected personally familiar and stranger target faces in arrays of two, four, or six face images. Subjects showed significant facilitation of personally familiar face detection for both upright and inverted faces. The effect of familiarity on target absent trials, which involved only rejection of unfamiliar face distractors, suggests that familiarity facilitates rejection of unfamiliar distractors as well as detection of familiar targets. The preserved familiarity effect for inverted faces suggests that facilitation of face detection afforded by familiarity reflects mostly feature-based processes.

Social Saliency of the Cue Slows Attention Shifts

Eye gaze is a powerful cue that indicates where another person's attention is directed in the environment. Seeing another person's eye gaze shift spontaneously and reflexively elicits a shift of one's own attention to the same region in space. Here, we investigated whether reallocation of attention in the direction of eye gaze is modulated by personal familiarity with faces. On the one hand, the eye gaze of a close friend should be more effective in redirecting our attention as compared to the eye gaze of a stranger. On the other hand, the social relevance of a familiar face might itself hold attention and, thereby, slow lateral shifts of attention. To distinguish between these possibilities, we measured the efficacy of the eye gaze of personally familiar and unfamiliar faces as directional attention cues using adapted versions of the Posner paradigm with saccadic and manual responses. We found that attention shifts were slower when elicited by a perceived change in the eye gaze of a familiar individual as compared to attention shifts elicited by unfamiliar faces at short latencies (100 ms). We also measured simple detection of change in direction of gaze in personally familiar and unfamiliar faces to test whether slower attention shifts were due to slower detection. Participants detected changes in eye gaze faster for familiar faces than for unfamiliar faces. Our results suggest that personally familiar faces briefly hold attention due to their social relevance, thereby slowing shifts of attention, even though the direction of eye movements are detected faster in familiar faces.

Expectations about person identity modulate the face-sensitive N170

Identifying familiar faces is a fundamentally important aspect of social perception that requires the ability to assign very different (ambient) images of a face to a common identity. The current consensus is that the brain processes face identity at approximately 250-300 msec following stimulus onset, as indexed by the N250 event related potential. However, using two experiments we show compelling evidence that where experimental paradigms induce expectations about person identity, changes in famous face identity are in fact detected at an earlier latency corresponding to the face-sensitive N170. In Experiment 1, using a rapid periodic stimulation paradigm presenting highly variable ambient images, we demonstrate robust effects of low frequency, periodic face-identity changes in N170 amplitude. In Experiment 2, we added infrequent aperiodic identity changes to show that the N170 was larger to both infrequent periodic and infrequent aperiodic identity changes than to high frequency identities. Our use of ambient stimulus images makes it unlikely that these effects are due to adaptation of low-level stimulus features. In line with current ideas about predictive coding, we therefore suggest that when expectations about the identity of a face exist, the visual system is capable of detecting identity mismatches at a latency consistent with the N170.

The Role of Familiarity for Representations in Norm-Based Face Space

According to the norm-based version of the multidimensional face space model (nMDFS, Valentine, 1991), any given face and its corresponding anti-face (which deviates from the norm in exactly opposite direction as the original face) should be equidistant to a hypothetical prototype face (norm), such that by definition face and anti-face should bear the same level of perceived typicality. However, it has been argued that familiarity affects perceived typicality and that representations of familiar faces are qualitatively different (e.g., more robust and image-independent) from those for unfamiliar faces. Here we investigated the role of face familiarity for rated typicality, using two frequently used operationalisations of typicality (deviation-based: DEV), and distinctiveness (face in the crowd: FITC) for faces of celebrities and their corresponding anti-faces. We further assessed attractiveness, likeability and trustworthiness ratings of the stimuli, which are potentially related to typicality. For unfamiliar faces and their corresponding anti-faces, in line with the predictions of the nMDFS, our results demonstrate comparable levels of perceived typicality (DEV). In contrast, familiar faces were perceived much less typical than their anti-faces. Furthermore, familiar faces were rated higher than their anti-faces in distinctiveness, attractiveness, likability and trustworthiness. These findings suggest that familiarity strongly affects the distribution of facial representations in norm-based face space. Overall, our study suggests (1) that familiarity needs to be considered in studies of mental representations of faces, and (2) that familiarity, general distance-to-norm and more specific vector directions in face space make different and interactive contributions to different types of facial evaluations.