Why has research in face recognition progressed so slowly? The importance of variability

Humans' extreme face recognition abilities challenge the well-established familiarity effect

Classification performance is better for learned than unlearned stimuli. This was also reported for faces, where identity matching of unfamiliar faces is worse than for familiar faces. This familiarity advantage led to the conclusion that variability across appearances of the same identity is partly idiosyncratic and cannot be generalized from familiar to unfamiliar identities. Recent advances in machine vision challenge this claim by showing that the performance for untrained (unfamiliar) identities reached the level of trained identities as the number of identities that the algorithm is trained with increases. We therefore asked whether humans who reportedly can identify a vast number of identities, such as super recognizers, may close the gap between familiar and unfamiliar face classification. Consistent with this prediction, super recognizers classified unfamiliar faces just as well as typical participants who are familiar with the same faces, on a task that generates a sizable familiarity effect in controls. Additionally, prosopagnosics' performance for familiar faces was as bad as that of typical participants who were unfamiliar with the same faces, indicating that they struggle to learn even identity-specific information. Overall, these findings demonstrate that by studying the extreme ends of a system's ability we can gain novel insights into its actual capabilities.

The importance of conceptual knowledge when becoming familiar with faces during naturalistic viewing

Although the ability to recognise familiar faces is a critical part of everyday life, the process by which a face becomes familiar in the real world is not fully understood. Previous research has focussed on the importance of perceptual experience. However, in natural viewing, perceptual experience with faces is accompanied by increased knowledge about the person and the context in which they are encountered. Although conceptual information is known to be crucial for the formation of new episodic memories, it requires a period of consolidation. It is unclear, however, whether a similar process occurs when we learn new faces. Using a natural viewing paradigm, we investigated how the context in which events are presented influences our understanding of those events and whether, after a period of consolidation, this has a subsequent effect on face recognition. The context was manipulated by presenting events in 1) the original sequence, or 2) a scrambled sequence. Although this manipulation was predicted to have a significant effect on conceptual understanding of events, it had no effect on overall visual experience with the faces. Our prediction was that this contextual manipulation would affect face recognition after the information has been consolidated into memory. We found that understanding of the narrative was greater for participants who viewed the movie in the original sequence compared to those that viewed the movie in a scrambled order. To determine if the context in which the movie was viewed had an effect on face recognition, we compared recognition in the original and scrambled condition. We found an overall effect of conceptual knowledge on face recognition. That is, participants who viewed the original sequence had higher face recognition compared to participants who viewed the scrambled sequence. However, our planned comparisons did not reveal a greater effect of conceptual knowledge on face recognition after consolidation. In an exploratory analysis, we found that overlap in conceptual knowledge between participants was significantly correlated with the overlap in face recognition. We also found that this relationship was greater after a period of consolidation. Together, these findings provide new insights into the role of non-visual, conceptual knowledge for face recognition during natural viewing.

Normative data of the Italian Famous Face Test

The faces we see in daily life exist on a continuum of familiarity, ranging from personally familiar to famous to unfamiliar faces. Thus, when assessing face recognition abilities, adequate evaluation measures should be employed to discriminate between each of these processes and their relative impairments. We here developed the Italian Famous Face Test (IT-FFT), a novel assessment tool for famous face recognition in typical and clinical populations. Normative data on a large sample (N = 436) of Italian individuals were collected, assessing both familiarity (d') and recognition accuracy. Furthermore, this study explored whether individuals possess insights into their overall face recognition skills by correlating the Prosopagnosia Index-20 (PI-20) with the IT-FFT; a negative correlation between these measures suggests that people have a moderate insight into their face recognition skills. Overall, our study provides the first online-based Italian test for famous faces (IT-FFT), a test that could be used alongside other standard tests of face recognition because it complements them by evaluating real-world face familiarity, providing a more comprehensive assessment of face recognition abilities. Testing different aspects of face recognition is crucial for understanding both typical and atypical face recognition.

The impact of the use of masks on trait judgments and face recognition

Although effective in reducing virus transmission, face masks might compromise face recognition and trait judgments. With this study, we aimed to observe the influence of masks on face recognition and trait judgments-more specifically, in trustworthiness, dominance, and distinctiveness judgments. Also, we wanted to observe the possible influence of trait judgments on facial recognition for masked and unmasked faces, which has never been done before. For that, we conducted an online study where 140 participants observed and made trait judgments of masked and unmasked faces in a within-subjects design. After a distractive task, participants performed a recognition memory test. As expected, we observed a better recognition of faces shown without a mask during the study phase, which allowed the holistic processing of the faces. The worst performance was found for faces encoded with a mask but tested without it, occurring simultaneity disruption in holistic face processing and the violation of the encoding specificity principle. Regarding the trait judgments, unmasked faces were considered more distinctive, and masked faces were considered more trustworthy. More interestingly, we can conclude that facial distinctiveness predicts face recognition, regardless of mask use. In contrast, dominance judgments only predicted face recognition when faces were presented without a mask. When faces were exposed with masks, trustworthiness overrides dominance, becoming more critical to recognizing faces. We can interpret these results from an evolutionary perspective.

Multiple images captured from a single encounter do not promote face learning

Viewing multiple images of a newly encountered face improves recognition of that identity in new instances. Studies examining face learning have presented high-variability (HV) images that incorporate changes that occur from moment-to-moment (e.g., head orientation and expression) and over time (e.g., lighting, hairstyle, and health). We examined whether low-variability (LV) images (i.e., images that incorporate only moment-to-moment changes) also promote generalisation of learning such that novel instances are recognised. Participants viewed a single image, six LV images, or six HV images of a target identity before being asked to recognise novel images of that identity in a face matching task (training stimuli remained visible) or a memory task (training stimuli were removed). In Experiment 1 (n = 71), participants indicated which image(s) in 8-image arrays belonged to the target identity. In Experiment 2 (n = 73), participants indicated whether sequentially presented images belonged to the target identity. Relative to the single-image condition, sensitivity to identity improved and response biases were less conservative in the HV condition; we found no evidence of generalisation of learning in the LV condition regardless of testing protocol. Our findings suggest that day-to-day variability in appearance plays an essential role in acquiring expertise with a novel face.

Diagnosing eyewitness identifications with reaction time‑based Concealed Information Test: the effect of viewpoint congruency between test and encoding

Mistaken eyewitness identifications continue to be a major contributor to miscarriages of justice. Previous experiments have suggested that implicit identification procedures such as the Concealed Information Test (CIT) might be a promising alternative to classic lineups when encoding conditions during the crime were favorable. We tested this idea by manipulating view congruency (frontal vs. profile view) between encoding and test. Participants witnessed a videotaped mock theft that showed the thief and victim almost exclusively from frontal or profile view. At test, viewing angle was either congruent or incongruent with the view during encoding. We tested eyewitness identification with the RT-CIT (N = 74), and with a traditional simultaneous photo lineup (N = 97). The CIT showed strong capacity to diagnose face recognition (d = 0.91 [0.64; 1.18]), but unexpectedly, view congruency did not moderate this effect. View congruency moderated lineup performance for one of the two lineups. Following these unexpected findings, we conducted a replication with a stronger congruency manipulation and larger sample size. CIT (N = 156) showed moderate capacity to diagnose face recognition (d = 0.63 [0.46; 0.80]) and now view congruency did moderate the CIT effect. For lineups (N = 156), view congruency again moderated performance for one of the two lineups. Capacity for diagnosing face recognition was similar for lineups and RT-CIT in our first comparison, but much stronger for lineups in our second comparison. Future experiments might investigate more conditions that affect performance in lineups vs. the RT-CIT differentially.

Using deep neural networks to disentangle visual and semantic information in human perception and memory

Mental representations of familiar categories are composed of visual and semantic information. Disentangling the contributions of visual and semantic information in humans is challenging because they are intermixed in mental representations. Deep neural networks that are trained either on images or on text or by pairing images and text enable us now to disentangle human mental representations into their visual, visual-semantic and semantic components. Here we used these deep neural networks to uncover the content of human mental representations of familiar faces and objects when they are viewed or recalled from memory. The results show a larger visual than semantic contribution when images are viewed and a reversed pattern when they are recalled. We further reveal a previously unknown unique contribution of an integrated visual-semantic representation in both perception and memory. We propose a new framework in which visual and semantic information contribute independently and interactively to mental representations in perception and memory.

Familiarity Is Key: Exploring the Effect of Familiarity on the Face-Voice Correlation

Recent research has examined the extent to which face and voice processing are associated by virtue of the fact that both tap into a common person perception system. However, existing findings do not yet fully clarify the role of familiarity in this association. Given this, two experiments are presented that examine face-voice correlations for unfamiliar stimuli (Experiment 1) and for familiar stimuli (Experiment 2). With care being taken to use tasks that avoid floor and ceiling effects and that use realistic speech-based voice clips, the results suggested a significant positive but small-sized correlation between face and voice processing when recognizing unfamiliar individuals. In contrast, the correlation when matching familiar individuals was significant and positive, but much larger. The results supported the existing literature suggesting that face and voice processing are aligned as constituents of an overarching person perception system. However, the difference in magnitude of their association here reinforced the view that familiar and unfamiliar stimuli are processed in different ways. This likely reflects the importance of a pre-existing mental representation and cross-talk within the neural architectures when processing familiar faces and voices, and yet the reliance on more superficial stimulus-based and modality-specific analysis when processing unfamiliar faces and voices.

Investigating people's metacognitive insight into their own face abilities

Within the domain of face processing, researchers have been interested in quantifying the relationship between objective (i.e., performance on laboratory tests of recognition and matching) and subjective measures of ability (typically, self-report questionnaires). Put simply, do people show high levels of metacognitive insight into their own abilities with faces? Although several studies have suggested that the association between these two types of measures may only be moderate, there remain several important issues that require consideration before this question can be sensibly investigated. First, specificity is needed regarding both objective and subjective measurements because both tend to span a wide range of potentially separable abilities. Second, experimental tasks appear to focus on different contexts to those tapped in self-report questionnaire items. Third, recent issues with statistical approaches and visualisation can result in numerical artefacts and misinterpretations. Finally, the sizes of population-level insights suggested by recent work provide only limited information regarding individuals within these populations, and so researchers aiming to identify people at the extremes of ability must be careful when drawing conclusions. Taken together, we argue that more attention to these issues is needed when attempting to investigate metacognitive insight within this domain.

A narrow band of image dimensions is critical for face recognition

A key challenge in human and computer face recognition is to differentiate information that is diagnostic for identity from other sources of image variation. Here, we used a combined computational and behavioural approach to reveal critical image dimensions for face recognition. Behavioural data were collected using a sorting and matching task with unfamiliar faces and a recognition task with familiar faces. Principal components analysis was used to reveal the dimensions across which the shape and texture of faces in these tasks varied. We then asked which image dimensions were able to predict behavioural performance across these tasks. We found that the ability to predict behavioural responses in the unfamiliar face tasks increased when the early PCA dimensions (i.e. those accounting for most variance) of shape and texture were removed from the analysis. Image similarity also predicted the output of a computer model of face recognition, but again only when the early image dimensions were removed from the analysis. Finally, we found that recognition of familiar faces increased when the early image dimensions were removed, decreased when intermediate dimensions were removed, but then returned to baseline recognition when only later dimensions were removed. Together, these findings suggest that early image dimensions reflect ambient changes, such as changes in viewpoint or lighting, that do not contribute to face recognition. However, there is a narrow band of image dimensions for shape and texture that are critical for the recognition of identity in humans and computer models of face recognition.

The impact of stability in appearance on the development of facial representations

The way faces become familiar and what information is represented as familiarity develops has puzzled researchers in the field of human face recognition for decades. In this paper, we present three experiments serving as proof of concept for a cost-efficient mechanism of face learning describing how facial representations form over time and accounting for recognition errors. We propose that the encoding of facial information is dynamic and modulated by the intrinsic stability in individual faces' appearance. We drew on a robust and ecological method using a proxy of exposure to famous faces in the real world and manipulated test images to assess the prediction that recognition of famous faces is affected by their relative stability in appearance. We consistently show that stable facial appearances (like Tom Cruise's) facilitate recognition in early stages of familiarisation but that performance does not improve much over time. In contrast, variations in appearance (like Jared Leto's) hinder recognition at first but improve performance with further media exposure. This pattern of results is consistent with the proposed cost-efficient face learning mechanism whereby facial representations build on a foundation of large-scale diagnostic information and refine over time if needed. When coarse information loses its diagnostic value through the experience of variations in appearance across encounters, diagnostic facial details and/or their spatial relationships must receive more weights, leading to refined representations that are more discriminative and reliable than representations of stable faces.

Frequency of adapting events affects face aftereffects but not blur aftereffects

The dynamics of visual adaptation remain poorly understood. Recent studies have found that the strength of adaptation aftereffects in the perception of numerosity depends more strongly on the number of adaptation events than on the duration of the adaptation. We investigated whether such effects can be observed for other visual attributes. We measured blur (perceived focus-sharp vs blurred adapt) and face (perceived race- Asian vs. White adapt) aftereffects by varying the number of adaptation events (4 or 16) and the duration of each adaptation event (0.25 s or 1 s). We found evidence for an effect of event number on face but not on blur adaptation, though the effect for faces was significant for only one of the two face adapt conditions (Asian). Our results suggest that different perceptual dimensions may vary in how adaptation effects accrue, potentially because of differences in factors such as the sites (early or late) of the sensitivity changes or nature of the stimulus. These differences may impact how and how rapidly the visual system can adjust to different visual properties.

Knowledge of identity reduces variability in trait judgements across face images

Faces vary from image to image, eliciting different judgements of traits and often different judgements of identity. Knowledge that two face images belong to the same person facilitates the processing of identity information across images, but it is unclear if this also applies to trait judgements. In this preregistered study, participants (N = 100) rated the same 340 face images on perceived trustworthiness, dominance, or attractiveness presented in randomised order and again later presented in sets consisting of the same identity. We also explored the role of implicit person theory beliefs in the variability of social judgements across images. We found that judgements of trustworthiness varied less when images were presented in sets consisting of the same identity than in randomised order and were more consistent for images presented later in a set than those presented earlier. However, knowledge of identity had little effect on perceptions of dominance and attractiveness. Finally, implicit person theory beliefs were not associated with variability in social judgements and did not account for effects of knowledge of identity. Our findings suggest that knowledge of identity and perceptual familiarity stabilises judgements of trustworthiness, but not perceptions of dominance and attractiveness.

Why can people with developmental prosopagnosia recognise some familiar faces? Insights from subjective experience

Developmental prosopagnosia is a relatively common visuo-cognitive condition, characterised by impaired facial identity recognition. Impairment severity appears to reside on a continuum, however, it is unknown whether instances of milder deficits reflect the successful use of spontaneous (typical) face recognition strategies, or the application of extraneous compensatory cues to recognition. Here, we explore this issue in two studies. First, 23 adults with developmental prosopagnosia were asked about their use of spontaneous versus compensatory face recognition techniques in everyday life, using a series of closed- and open-ended questions. Second, the same participants performed a computerised famous face recognition task where they were asked to provide reasons why they could make any successful identifications. Findings from both studies suggest that people with developmental prosopagnosia can successfully, and quite frequently, use compensatory strategies to recognition, and that these cues support the majority of instances of preserved familiar face recognition. In contrast, 16 of the 23 participants were able to spontaneously recognise familiar faces on at least some occasions, but there were vast individual differences in frequencies of success. These findings have important implications for our conceptualisation of the condition, as well as for diagnostic practice.

Not the norm: Face likeness is not the same as similarity to familiar face prototypes

Face images depicting the same individual can differ substantially from one another. Ecological variation in pose, expression, lighting, and other sources of appearance variability complicates the recognition and matching of unfamiliar faces, but acquired familiarity leads to the ability to cope with these challenges. Among the many ways that face of the same individual can vary, some images are judged to be better likenesses of familiar individuals than others. Simply put, these images look more like the individual under consideration than others. But what does it mean for an image to be a better likeness than another? Does likeness entail typicality, or is it something distinct from this? We examined the relationship between the likeness of face images and the similarity of those images to average images of target individuals using a set of famous faces selected for reciprocal familiarity/unfamiliarity across US and UK participants. We found that though likeness judgments are correlated with similarity-to-prototype judgments made by both familiar and unfamiliar participants, this correlation was smaller than the correlation between similarity judgments made by different participant groups. This implies that while familiarity weakens the relationship between likeness and similarity-to-prototype judgments, it does not change similarity-to-prototype judgments to the same degree.

Provoked overt recognition in acquired prosopagnosia using multiple different images of famous faces

Provoked overt recognition refers to the fact that patients with acquired prosopagnosia can sometimes recognize faces when presented in arrays of individuals from the same category (e.g., actors or politicians). We ask whether a prosopagnosic patient might experience recognition when presented with multiple different images of the same face simultaneously. Over two sessions, patient Herschel, a 66-year-old British man with acquired prosopagnosia, viewed face images individually or in arrays. On several occasions he failed to recognize single photos of an individual but successfully identified that person when the same photos were presented together. For example, Herschel failed to recognize any individual images of King Charles or Paul McCartney but recognised both in arrays of the same photos. Like reports based on category membership, overt recognition was transient and inconsistent. These findings are discussed in terms of models of covert recognition, alongside more recent research on within-person variability for face perception.

Understanding face matching

Many security settings rely on the identity matching of unfamiliar people, which has led this task to be studied extensively in Cognitive Psychology. In these experiments, observers typically decide whether pairs of faces depict one person (an identity match) or two different people (an identity mismatch). The visual similarity of the to-be-compared faces must play a primary role in how observers accurately resolve this task, but the nature of this similarity-accuracy relationship is unclear. The current study investigated the association between accuracy and facial similarity at the level of individual items (Experiments 1 and 2) and facial features (Experiments 3 and 4). All experiments demonstrate a strong link between similarity and matching accuracy, indicating that this forms the basis of identification decisions. At a feature level, however, similarity exhibited distinct relationships with match and mismatch accuracy. In matches, similarity information was generally shared across the features of a face pair under comparison, with greater similarity linked to higher accuracy. Conversely, features within mismatching face pairs exhibited greater variation in similarity information. This indicates that identity matches and mismatches are characterised by different similarity profiles, which present distinct challenges to the cognitive system. We propose that these identification decisions can be resolved through the accumulation of convergent featural information in matches and the evaluation of divergent featural information in mismatches.

Stable individual differences in unfamiliar face identification: Evidence from simultaneous and sequential matching tasks

Matching identity in images of unfamiliar faces is difficult: Images of the same person can look different and images of different people can look similar. Recent studies have capitalized on individual differences in the ability to distinguish match (same ID) vs. mismatch (different IDs) face pairs to inform models of face recognition. We addressed two significant gaps in the literature by examining the stability of individual differences in both sensitivity to identity and response bias. In Study 1, 210 participants completed a battery of four tasks in each of two sessions separated by one week. Tasks varied in protocol (same/different, lineup, sorting) and stimulus characteristics (low vs. high within-person variability in appearance). In Study 2, 148 participants completed a battery of three tasks in a single session. Stimuli were presented simultaneously on some trials and sequentially on others, introducing short-term memory demands. Principal components analysis revealed two components that were stable across time and tasks: sensitivity to identity and bias. Analyses of response times suggest that individual differences in bias reflect decision-making processes. We discuss the implications of our findings in applied settings and for models of face recognition.

The Curious Case of Impersonators and Singers: Telling Voices Apart and Telling Voices Together under Naturally Challenging Listening Conditions

Vocal identity processing depends on the ability to tell apart two instances of different speakers whilst also being able to tell together two instances of the same speaker. Whilst previous research has examined these voice processing capabilities under relatively common listening conditions, it has not yet tested the limits of these capabilities. Here, two studies are presented that employ challenging listening tasks to determine just how good we are at these voice processing tasks. In Experiment 1, 54 university students were asked to distinguish between very similar sounding, yet different speakers (celebrity targets and their impersonators). Participants completed a 'Same/Different' task and a 'Which is the Celebrity?' task to pairs of speakers, and a 'Real or Not?' task to individual speakers. In Experiment 2, a separate group of 40 university students was asked to pair very different sounding instances of the same speakers (speaking and singing). Participants were presented with an array of voice clips and completed a 'Pairs Task' as a variant of the more traditional voice sorting task. The results of Experiment 1 suggested that significantly more mistakes were made when distinguishing celebrity targets from their impersonators than when distinguishing the same targets from control voices. Nevertheless, listeners were significantly better than chance in all three tasks despite the challenge. Similarly, the results of Experiment 2 suggested that it was significantly more difficult to pair singing and speaking clips than to pair two speaking clips, particularly when the speakers were unfamiliar. Again, however, the performance was significantly above zero, and was again better than chance in a cautious comparison. Taken together, the results suggest that vocal identity processing is a highly adaptable task, assisted by familiarity with the speaker. However, the fact that performance remained above chance in all tasks suggests that we had not reached the limit of our listeners' capability, despite the considerable listening challenges introduced. We conclude that voice processing is far better than previous research might have presumed.

How quickly do we learn new faces in everyday life? Neurophysiological evidence for face identity learning after a brief real-life encounter

Faces learnt in a single experimental session elicit a familiarity effect in event-related brain potentials (ERPs), with more negative amplitudes for newly learnt relative to unfamiliar faces in the N250 component. However, no ERP study has examined face learning following a brief real-life encounter, and it is not clear how long it takes to learn new faces in such ecologically more valid conditions. To investigate these questions, the present study examined whether robust image-independent representations, as reflected in the N250 familiarity effect, could be established after a brief unconstrained social interaction by analysing the ERPs elicited by highly variable images of the newly learnt identity and an unfamiliar person. Significant N250 familiarity effects were observed after a 30-min (Experiment 1) and a 10-min (Experiment 2) encounter, and a trend was observed after 5 min of learning (Experiment 3), demonstrating that 5-10 min of exposure were sufficient for the initial establishment of image-independent representations. Additionally, the magnitude of the effects reported after 10 and 30 min was comparable suggesting that the first 10 min of a social encounter might be crucial, with extra 20 min from the same encounter not adding further benefit for the initial formation of robust face representations.

From Pictures to the People in Them: Averaging Within-Person Variability Leads to Face Familiarization

Familiar faces can be confidently recognized despite sometimes radical changes in their appearance. Exposure to within-person variability-differences in facial characteristics over successive encounters-contributes to face familiarization. Research also suggests that viewers create mental averages of the different views of faces they encounter while learning them. Averaging over within-person variability is thus a promising mechanism for face familiarization. In Experiment 1, 153 Canadian undergraduates (88 female; age: M = 21 years, SD = 5.24) learned six target identities from eight different photos of each target interspersed among 32 distractor identities. Face-matching accuracy improved similarly irrespective of awareness of the target's identity, confirming that target faces presented among distractors can be learned incidentally. In Experiment 2, 170 Canadian undergraduates (125 female; age: M = 22.6 years, SD = 6.02) were tested using a novel indirect measure of learning. The results show that viewers update a mental average of a person's face as it becomes learned. Our findings are the first to show how averaging within-person variability over time leads to face familiarization.

Diagnosing eyewitness identifications with reaction time-based concealed information test: the effect of observation time

Eyewitness identification procedures rely heavily on explicit identification from lineups. Lineups have been criticized because they have a considerable error rate. We tested the potential of implicit identifications in a Concealed Information Test (CIT) as an alternative. Previous experiments have suggested that implicit identification procedures might be suited when viewing conditions were favorable. In two experiments, mock eyewitnesses (Ns = 94, 509) witnessed a videotaped mock theft with longer or shorter observation time. We derived eyewitness identifications either implicitly from CIT reaction times or explicitly from simultaneous photo lineups. In Experiment 2, we also manipulated perpetrator presence. In both experiments, the perpetrator-present CIT showed capacity to diagnose face recognition, with large effect sizes (d_E1 = 0.85 [0.51; 1.18]; d_E2 = 0.74 [0.52; 0.96]), as expected. Unexpectedly, no moderation by observation time was found. In line with our hypothesis, no CIT effect emerged in the perpetrator-absent condition, indicating the absence of recognition (d_E2 = 0.02 [- 0.17; 0.20]). We found no compelling evidence that one method would outperform the other. This work adds to accumulating evidence that suggests that, under favorable viewing conditions and replication provided, the RT-CIT might be diagnostic of facial recognition, for example when witnesses are hesitant of making an explicit identification. Future work might investigate conditions that affect performance in one, but not the other identification method.

How do we describe other people from voices and faces?

When seeing someone's face or hearing their voice, perceivers routinely infer information about a person's age, sex and social traits. While many experiments have explored how individual person characteristics are perceived in isolation, less is known about which person characteristics are described spontaneously from voices and faces and how descriptions may differ across modalities. In Experiment 1, participants provided free descriptions for voices and faces. These free descriptions followed similar patterns for voices and faces - and for individual identities: Participants spontaneously referred to a wide range of descriptors. Psychological descriptors, such as character traits, were used most frequently; physical characteristics, such as age and sex, were notable as they were mentioned earlier than other types of descriptors. After finding primarily similarities between modalities when analysing person descriptions across identities, Experiment 2 asked whether free descriptions encode how individual identities differ. For this purpose, the measures derived from the free descriptions were linked to voice/face discrimination judgements that are known to describe differences in perceptual properties between identity pairs. Significant relationships emerged within and across modalities, showing that free descriptions indeed encode differences between identities - information that is shared with discrimination judgements. This suggests that the two tasks tap into similar, high-level person representations. These findings show that free description data can offer valuable insights into person perception and underline that person perception is a multivariate process during which perceivers rapidly and spontaneously infer many different person characteristics to form a holistic impression of a person.

Contributions of low- and high-level contextual mechanisms to human face perception

Contextual modulations at primary stages of visual processing depend on the strength of local input. Contextual modulations at high-level stages of (face) processing show a similar dependence to local input strength. Namely, the discriminability of a facial feature determines the amount of influence of the face context on that feature. How high-level contextual modulations emerge from primary mechanisms is unclear due to the scarcity of empirical research systematically addressing the functional link between the two. We tested (62) young adults' ability to process local input independent of the context using contrast detection and (upright and inverted) morphed facial feature matching tasks. We first investigated contextual modulation magnitudes across tasks to address their shared variance. A second analysis focused on the profile of performance across contextual conditions. In upright eye matching and contrast detection tasks, contextual modulations only correlated at the level of their profile (averaged Fisher-Z transformed r = 1.18, BF10 > 100), but not magnitude (r = .15, BF10 = .61), suggesting the functional independence but similar working principles of the mechanisms involved. Both the profile (averaged Fisher-Z transformed r = .32, BF10 = 9.7) and magnitude (r = .28, BF10 = 4.58) of the contextual modulations correlated between inverted eye matching and contrast detection tasks. Our results suggest that non-face-specialized high-level contextual mechanisms (inverted faces) work in connection to primary contextual mechanisms, but that the engagement of face-specialized mechanisms for upright faces obscures this connection. Such combined study of low- and high-level contextual modulations sheds new light on the functional relationship between different levels of the visual processing hierarchy, and thus on its functional organization.

Perceptual similarity modulates effects of learning from variability on face recognition

Face recognition is a challenging classification task that humans perform effortlessly for familiar faces. Recent studies have emphasized the importance of exposure to high variability appearances of the same identity to perform this task. However, these studies did not explicitly measure the perceptual similarity between the learned images and the images presented at test, which may account for the advantage of learning from high variability. Particularly, randomly selected test images are more likely to be perceptually similar to learned high variability images, and dissimilar to learned low variability images. Here we dissociated effects of learning from variability and study-test perceptual similarity, by collecting human similarity ratings for the study and test images. Using these measures, we independently manipulated the variability between the learning images and their perceptual similarity to the test images. Different groups of participants learned face identities from a low or high variability set of images. The learning phase was followed by a face matching test (Experiment 1) or a face recognition task (Experiment 2) that presented novel images of the learned identities that were perceptually dissimilar or similar to the learned images. Results of both experiments show that perceptual similarity between study and test, rather than image variability at learning per se, predicts face recognition. We conclude that learning from high variability improves face recognition for perceptually similar but not for perceptually dissimilar images. These findings may not be specific to faces and should be similarly evaluated for other domains.

Illumination and gaze effects on face evaluation: The Bi-AGI database

Face evaluation and first impression generation can be affected by multiple face elements such as invariant facial features, gaze direction and environmental context; however, the composite modulation of eye gaze and illumination on faces of different gender and ages has not been previously investigated. We aimed at testing how these different facial and contextual features affect ratings of social attributes. Thus, we created and validated the Bi-AGI Database, a freely available new set of male and female face stimuli varying in age across lifespan from 18 to 87 years, gaze direction and illumination conditions. Judgments on attractiveness, femininity-masculinity, dominance and trustworthiness were collected for each stimulus. Results evidence the interaction of the different variables in modulating social trait attribution, in particular illumination differently affects ratings across age, gaze and gender, with less impact on older adults and greater effect on young faces.

Event-related brain potential correlates of the other-race effect: A review

People are better at remembering own-race relative to other-race faces. Here, we review event-related brain potential (ERP) correlates of this so-called other-'race' effect (ORE) by discussing three critical aspects that characterize the neural signature of this phenomenon. First, difficulties with other-race faces initially emerge during perceptual processing, which is indexed by an increased N170. Second, as evidenced by 'difference due to subsequent memory' effects, more effortful processing of other-race faces is needed for successful encoding into long-term memory. Third, ERP old/new effects reveal that a stronger engagement of processing resources is also required for successful retrieval of other-race faces from memory. The ERP evidence available to date thus suggests widespread ethnicity-related modulations during both perceptual and mnemonic processing stages. We further discuss how findings from the ORE compared with potentially related memory biases (e.g. other-gender or other-age effects) and how ERP findings inform the ongoing debate regarding the mechanisms underlying the ORE. Finally, we outline open questions and potential future directions with an emphasis on using multiple, ecologically more valid 'ambient' images for each face to assess the ORE in paradigms that capture identity rather than image recognition.

Image variability and face matching

This study investigates whether variability in perceived trait judgements disrupts our ability to match unfamiliar faces. In this preregistered study, 174 participants completed a face matching task where they were asked to indicate whether two ambient face images belonged to the same person or different people (17,748 total data points). Participants completed 51 match trials consisting of images of the same person that differed substantially on one trait (either trustworthiness, dominance or attractiveness) with minimal differences in the alternate traits. Participants also completed 51 mismatch trials which contained two photos of similar-looking individuals. We hypothesised that participants would make more errors on match trials when images differed in terms of attractiveness ratings than when they differed on trustworthiness or dominance. Contrary to expectations, images that differed in terms of attractiveness were matched most accurately, and there was no relationship between the extent of differences in attractiveness ratings and accuracy. There was some evidence that differences in perceived dominance and, to a lesser extent, trustworthiness were associated with lower face matching performance. However, these relationships were not significant when alternate traits were accounted for. The findings of our study suggest that face matching performance is largely robust against variation in trait judgements.

What happens to our representation of identity as familiar faces age? Evidence from priming and identity aftereffects

Matching identity in images of unfamiliar faces is error prone, but we can easily recognize highly variable images of familiar faces - even images taken decades apart. Recent theoretical development based on computational modelling can account for how we recognize extremely variable instances of the same identity. We provide complementary behavioural data by examining older adults' representation of older celebrities who were also famous when young. In Experiment 1, participants completed a long-lag repetition priming task in which primes and test stimuli were the same age or different ages. In Experiment 2, participants completed an identity after effects task in which the adapting stimulus was an older or young photograph of one celebrity and the test stimulus was a morph between the adapting identity and a different celebrity; the adapting stimulus was the same age as the test stimulus on some trials (e.g., both old) or a different age (e.g., adapter young, test stimulus old). The magnitude of priming and identity after effects were not influenced by whether the prime and adapting stimulus were the same age or different age as the test face. Collectively, our findings suggest that humans have one common mental representation for a familiar face (e.g., Paul McCartney) that incorporates visual changes across decades, rather than multiple age-specific representations. These findings make novel predictions for state-of-the-art algorithms (e.g., Deep Convolutional Neural Networks).

A systematic survey of face stimuli used in psychological research 2000-2020

For decades, psychology has relied on highly standardized images to understand how people respond to faces. Many of these stimuli are rigorously generated and supported by excellent normative data; as such, they have played an important role in the development of face science. However, there is now clear evidence that testing with ambient images (i.e., naturalistic images "in the wild") and including expressions that are spontaneous can lead to new and important insights. To precisely quantify the extent to which our current knowledge base has relied on standardized and posed stimuli, we systematically surveyed the face stimuli used in 12 key journals in this field across 2000-2020 (N = 3374 articles). Although a small number of posed expression databases continue to dominate the literature, the use of spontaneous expressions seems to be increasing. However, there has been no increase in the use of ambient or dynamic stimuli over time. The vast majority of articles have used highly standardized and nonmoving pictures of faces. An emerging trend is that virtual faces are being used as stand-ins for human faces in research. Overall, the results of the present survey highlight that there has been a significant imbalance in favor of standardized face stimuli. We argue that psychology would benefit from a more balanced approach because ambient and spontaneous stimuli have much to offer. We advocate a cognitive ethological approach that involves studying face processing in natural settings as well as the lab, incorporating more stimuli from "the wild".

Children's ability to recognize their parent's face improves with age

Adults are experts at recognizing familiar faces across images that incorporate natural within-person variability in appearance (i.e., ambient images). Little is known about children's ability to do so. In the current study, we investigated whether 4- to 7-year-olds (n = 56) could recognize images of their own parent-a person with whom children have had abundant exposure in a variety of different contexts. Children were asked to identify images of their parent that were intermixed with images of other people. We included images of each parent taken both before and after their child was born to manipulate how close the images were to the child's own experience. When viewing before-birth images, 4- and 5-year-olds were less sensitive to identity than were older children; sensitivity did not differ when viewing images taken after the child was born. These findings suggest that with even the most familiar face, 4- and 5-year-olds have difficulty recognizing instances that go beyond their direct experience. We discuss two factors that may contribute to the prolonged development of familiar face recognition.

Effects of Emotional Expression on Face Recognition May Be Accounted for by Image Similarity

We examined the degree to which differences in face recognition rates across emotional expression conditions varied concomitantly with differences in mean objective image similarity. Effects of emotional expression on face recognition performance were measured via an old/new recognition paradigm in which stimuli at both learning and testing had happy, neutral, and angry expressions. Results showed an advantage for faces learned with neutral expressions, as well as for angry faces at testing. Performance data was compared to three quantitative image-similarity indices. Findings showed that mean human performance was strongly correlated with mean image similarity, suggesting that the former may be at least partly explained by the latter. Our findings sound a cautionary note regarding the necessity of considering low-level stimulus properties as explanations for findings that otherwise may be prematurely attributed to higher order phenomena such as attention or emotional arousal.

Multi-cultural cities reduce disadvantages in recognizing naturalistic images of other-race faces: evidence from a novel face learning task

People often find it more difficult to recognize other- than own-race faces. This other-race effect is robust across numerous ethnic groups. Yet, it remains unclear how this effect changes in people who live in a multiracial environment, and in immigrants whose lifetime perceptual experience changes over time. In the present study, we developed a novel face recognition test that approximates face recognition in the real world. We tested five groups of White and East Asian adults (n = 120) living in racially homogeneous versus heterogeneous cities and East Asians who immigrated to a multiracial city between infancy and adulthood. Multiracial cities reduce the other-race effect. The magnitude of the other-race effect changes as a function of experience, mirroring the racial diversity in perceivers' living environment. Our study highlights the challenge of forming reliable face representations across naturalistic facial variability and suggests a facilitative role of multiracial environments in eliminating the other-race effect.

Picture this: Photographers no better than controls for recognizing unfamiliar faces

With the exception of super recognizers and forensic examiners, people make a surprising number of errors when deciding whether photographs of unfamiliar faces belong to the same person or different people. Training protocols designed to improve professionals’ (e.g., passport officers) performance often include photography. We evaluated the influence of life-time photography experience on the ability to distinguish matched versus mismatched face pairs. Expert photographers were not more sensitive to identity than hobbyists or novices—despite specializing in human subjects; Hobbyists were more liberal (more same responses) than Experts. We conclude that photography experience is not a route to expertise.

The pairs training effect in unfamiliar face matching

A wealth of studies have shown that humans are remarkably poor at determining whether two face images show the same person or not (face matching). Given the prevalence of photo-ID, and the fact that people employed to check photo-ID are typically unfamiliar with the person pictured, there is a need to improve unfamiliar face matching accuracy. One method of improvement is to have participants complete the task in a pair, which results in subsequent improvements in the low performer (“the pairs training effect”). Here, we sought to replicate the original finding, to test the longevity of the pairs training effect, and to shed light on the potential underlying mechanisms. In two experiments, we replicated the pairs training effect and showed it is maintained after a delay (Experiment 1). We found no differences between high and low performers in confidence (Experiment 1) or response times (Experiment 2), and the content of the pairs’ discussions (Experiment 2) did not explain the results. The pairs training effect in unfamiliar face matching is robust, but the mechanisms underlying the effects remain as yet unexplained.

Face learning via brief real-world social interactions includes changes in face-selective brain areas and hippocampus

Making new acquaintances requires learning to recognise previously unfamiliar faces. In the current study, we investigated this process by staging real-world social interactions between actors and the participants. Participants completed a face-matching behavioural task in which they matched photographs of the actors (whom they had yet to meet), or faces similar to the actors (henceforth called foils). Participants were then scanned using functional magnetic resonance imaging (fMRI) while viewing photographs of actors and foils. Immediately after exiting the scanner, participants met the actors for the first time and interacted with them for 10 min. On subsequent days, participants completed a second behavioural experiment and then a second fMRI scan. Prior to each session, actors again interacted with the participants for 10 min. Behavioural results showed that social interactions improved performance accuracy when matching actor photographs, but not foil photographs. The fMRI analysis revealed a difference in the neural response to actor photographs and foil photographs across all regions of interest (ROIs) only after social interactions had occurred. Our results demonstrate that short social interactions were sufficient to learn and discriminate previously unfamiliar individuals. Moreover, these learning effects were present in brain areas involved in face processing and memory.

Improving face identification of mask-wearing individuals

Research has consistently shown that concealing facial features can hinder subsequent identification. The widespread adoption of face masks due to the COVID-19 pandemic has highlighted the critical and urgent need to discover techniques to improve identification of people wearing face coverings. Despite years of research on face recognition and eyewitness identifications, there are currently no evidence-based recommendations for lineup construction for cases involving masked individuals. The purpose of this study was to examine identification accuracy of a masked perpetrator as a function of lineup type (i.e., unmasked or masked lineups) and perpetrator presence (i.e., absent or present). In both experiments, discriminability was superior for masked lineups, a result that was due almost exclusively to higher hits rates in target-present conditions. These data suggest that presenting a masked lineup can enhance identification of masked faces, and they have important implications for both eyewitness identification and everyday face recognition of people with face coverings.

Attentional Features of Mindfulness are Better Predictors of Face Recognition than Empathy and Compassion-Based Constructs

Recent research has employed measures of either empathy, compassion or mindfulness and linked better face recognition memory to higher scores of identification with all humanity and mindfulness but not empathy or compassion. Additionally, empathy, compassion and mindfulness have been suggested as concepts that intertwine, but research has not yet examined how their respective personality questionnaires map onto latent concepts. We employed these measures together to explore their factor structure and, using structural equation modelling, we investigated if the suggested latent variables predict recognition memory performance for face and non-face stimuli. Attentional notions of mindfulness described a latent factor that predicted face recognition. All self-compassion facets and the non-react mindfulness facet described a latent factor, which predicted false alarms in face recognition. Finally, empathy and compassion-based notions described one latent factor, which did not predict recognition performance. None of the latent variables predicted performance in either object or voice recognition. Collectively, findings indicate attention-based mindfulness to benefit face recognition, prompting further research into the potential of mindfulness to support the face recognition process.

Learning faces from variability: Four- and five-year-olds differ from older children and adults

Children under 6 years of age have difficulty recognizing a familiar face across changes in appearance and telling the face apart from similar-looking people. Understanding the process by which newly encountered faces become familiar can provide insights into these difficulties. Exposure to the ways in which a person varies in appearance is one mechanism by which adults and older children (≥6 years) learn new faces. We provide the first investigation of whether this mechanism for face learning functions in younger children. Children aged 4 and 5 years were read two storybooks featuring an unfamiliar character. Participants viewed six images of the character in one story and one image of the character in the other story. After each story, children were asked to identify novel images of the character that were intermixed with images of a similar-looking distractor. Like older children, 4- and 5-year-olds were more sensitive to identity in the 6-image condition, but they also adapted a less conservative criterion. Young children identified more images of the character after viewing six images versus one image. However, many also incorrectly identified more images of the distractor after viewing six images versus one image, an effect not previously found for older children and adults. These results suggest that this mechanism for face learning is not fully refined before 6 years of age.

Naturalistic Face Learning in Infants and Adults

Everyday face recognition presents a difficult challenge because faces vary naturally in appearance as a result of changes in lighting, expression, viewing angle, and hairstyle. We know little about how humans develop the ability to learn faces despite natural facial variability. In the current study, we provide the first examination of attentional mechanisms underlying adults' and infants' learning of naturally varying faces. Adults (n = 48) and 6- to 12-month-old infants (n = 48) viewed videos of models reading a storybook; the facial appearance of these models was either high or low in variability. Participants then viewed the learned face paired with a novel face. Infants showed adultlike prioritization of face over nonface regions; both age groups fixated the face region more in the high- than low-variability condition. Overall, however, infants showed less ability to resist contextual distractions during learning, which potentially contributed to their lack of discrimination between the learned and novel faces. Mechanisms underlying face learning across natural variability are discussed.

An update of the Benton Facial Recognition Test

The Benton Facial Recognition Test (BFRT) is a paper-and-pen task that is traditionally used to assess face perception skills in neurological, clinical and psychiatric conditions. Despite criticisms of its stimuli, the task enjoys a simple procedure and is rapid to administer. Further, it has recently been computerised (BFRT-c), allowing reliable measurement of completion times and the need for online testing. Here, in response to calls for repeat screening for the accurate detection of face processing deficits, we present the BFRT-Revised (BFRT-r): a new version of the BFRT-c that maintains the task's basic paradigm, but employs new, higher-quality stimuli that reflect recent theoretical advances in the field. An initial validation study with typical participants indicated that the BFRT-r has good internal reliability and content validity. A second investigation indicated that while younger and older participants had comparable accuracy, completion times were longer in the latter, highlighting the need for age-matched norms. Administration of the BFRT-r and BFRT-c to 32 individuals with developmental prosopagnosia resulted in improved sensitivity in diagnostic screening for the BFRT-r compared to the BFRT-c. These findings are discussed in relation to current diagnostic screening protocols for face perception deficits. The BFRT-r is stored in an open repository and is freely available to other researchers.

Comparing the sensitivity of face matching assessments to detect face perception impairments

Numerous neurological, developmental, and psychiatric conditions demonstrate impaired face recognition, which can be socially debilitating. These impairments can be caused by either deficient face perception or face memory mechanisms. Though there are well-validated, sensitive measures of face memory impairments, it currently remains unclear which assessments best measure face perception impairments. A sensitive, validated face perception measure could help with diagnosing causes of face recognition deficits and be useful in characterizing individual differences in unimpaired populations. Here, we compared the computerized Benton Face Recognition Test (BFRT-c) and Cambridge Face Perception Test (CFPT) in their ability to differentiate developmental prosopagnosics (DPs, N = 30) and age-matched controls (N = 30). Participants completed the BFRT-c, CFPT, and two additional face perception assessments: the University of Southern California Face Perception Test (USCFPT) and a novel same/different face matching test (SDFMT). Participants were also evaluated on objective and subjective face recognition tasks including the Cambridge Face Memory Test, famous faces test, and Prosopagnosia Index-20. We performed a logistic regression with the perception tests predicting DP vs. control group membership and used multiple linear regressions to predict continuous objective and subjective face recognition memory. Our results show that the BFRT-c performed as well as, if not better than, the CFPT, and that both tests clearly outperformed the USCFPT and SDFMT. Further, exploratory analyses revealed that face lighting-change conditions better predicted DP group membership and face recognition abilities than viewpoint-change conditions. Together, these results support the combined use of the BFRT-c and CFPT to best assess face perception impairments.

Partitioning natural face image variability emphasises within-identity over between-identity representation for understanding accurate recognition

Accurately recognising faces enables social interactions. In recent years it has become clear that people's accuracy differs markedly depending on viewer's familiarity with a face and their individual skill, but the cognitive and neural bases of these accuracy differences are not understood. We examined cognitive representations underlying these accuracy differences by measuring similarity ratings to natural facial image variation. Natural variation was sampled from uncontrolled images on the internet to reflect the appearance of faces as they are encountered in daily life. Using image averaging, and inspired by the computation of Analysis of Variance, we partitioned this variation into differences between faces (between-identity variation) and differences between photos of the same face (within-identity variation). This allowed us to compare modulation of these two sources of variation attributable to: (i) a person's familiarity with a face and, (ii) their face recognition ability. Contrary to prevailing accounts of human face recognition and perceptual learning, we found that modulation of within-identity variation - rather than between-identity variation - was associated with high accuracy. First, familiarity modulated similarity ratings to within-identity variation more than to between-face variation. Second, viewers that are extremely accurate in face recognition - 'super-recognisers' - differed from typical perceivers mostly in their ratings of within-identity variation, compared to between-identity variation. In a final computational analysis, we found evidence that transformations of between- and within-identity variation make separable contributions to perceptual expertise in face recognition. We conclude that inter- and intra-individual accuracy differences primarily arise from differences in the representation of within-identity image variation.

The Unique and Interactive Effects of Faces, Postures, and Scenes on Emotion Categorization

There is ongoing debate as to whether emotion perception is determined by facial expressions or context (i.e., non-facial cues). The present investigation examined the independent and interactive effects of six emotions (anger, disgust, fear, joy, sadness, neutral) conveyed by combinations of facial expressions, bodily postures, and background scenes in a fully crossed design. Participants viewed each face-posture-scene (FPS) combination for 5 s and were then asked to categorize the emotion depicted in the image. Four key findings emerged from the analyses: (1) For fully incongruent FPS combinations, participants categorized images using the face in 61% of instances and the posture and scene in 18% and 11% of instances, respectively; (2) postures (with neutral scenes) and scenes (with neutral postures) exerted differential influences on emotion categorizations when combined with incongruent facial expressions; (3) contextual asymmetries were observed for some incongruent face-posture pairings and their inverse (e.g., anger-fear vs. fear-anger), but not for face-scene pairings; (4) finally, scenes exhibited a boosting effect of posture when combined with a congruent posture and attenuated the effect of posture when combined with a congruent face. Overall, these findings highlight independent and interactional roles of posture and scene in emotion face perception. Theoretical implications for the study of emotions in context are discussed.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s42761-021-00061-x.

Are women always better able to recognize faces? The unveiling role of exposure time

A longer exposure time generally improves individuals’ ability to recognize faces. The current research investigates whether this effect varies between genders and whether it is influenced by the gender of the exposed faces. Based on a set of four experimental studies, we advance our knowledge of face recognition, gender, gender distribution of exposed faces, and exposure time in three main ways. First, the results reveal that women are more likely than men to suffer from a decrease in face recognition ability due to a lower exposure time. Second, the findings show that when exposure time is short (vs. long) women recognize a larger proportion of same gender faces and also recognize a larger proportion of same gender faces as compared with the proportion of same gender faces recognized by men. Third, findings reveal that when individuals are only exposed to same gender faces, women recognize more faces than men regardless whether exposure time is short, or long. In short, the findings of this research suggest that insight into the interplay between gender and exposure time length is critical to appropriately determine human beings’ ability to recognize faces.

The rapid and automatic categorization of facial expression changes in highly variable natural images

Emotional expressions are quickly and automatically read from human faces under natural viewing conditions. Yet, categorization of facial expressions is typically measured in experimental contexts with homogenous sets of face stimuli. Here we evaluated how the 6 basic facial emotions (Fear, Disgust, Happiness, Anger, Surprise or Sadness) can be rapidly and automatically categorized with faces varying in head orientation, lighting condition, identity, gender, age, ethnic origin and background context. High-density electroencephalography was recorded in 17 participants viewing 50 s sequences with natural variable images of neutral-expression faces alternating at a 6 Hz rate. Every five stimuli (1.2 Hz), variable natural images of one of the six basic expressions were presented. Despite the wide physical variability across images, a significant F/5 = 1.2 Hz response and its harmonics (e.g., 2F/5 = 2.4 Hz, etc.) was observed for all expression changes at the group-level and in every individual participant. Facial categorization responses were found mainly over occipito-temporal sites, with distinct hemispheric lateralization and cortical topographies according to the different expressions. Specifically, a stronger response was found to Sadness categorization, especially over the left hemisphere, as compared to Fear and Happiness, together with a right hemispheric dominance for categorization of Fearful faces. Importantly, these differences were specific to upright faces, ruling out the contribution of low-level visual cues. Overall, these observations point to robust rapid and automatic facial expression categorization processes in the human brain.