Exemplar variance supports robust learning of facial identity

Prior familiarity enhances recognition memory of faces, not just images of faces, when accompanied by conceptual information

A recent paper reported that recognition discriminability was improved for faces that were familiarized prior to study, but only if that familiarization protocol included conceptual information, like a name (Akan & Benjamin, Journal of Memory and Language, 131, 104,433, 2023). In those experiments, familiarity with each facial identity was gained through exposures to the same facial image prior to study, and memory for each facial identity was tested using the same images across study and test. That design characteristic has a serious constraint on generality, since it is possible that prior conceptual information enhances memory for images (of faces), but not for the representation of the face itself. Here we evaluated whether this finding generalizes to a paradigm in which each exposure of a face is a novel image. In two experiments, faces were familiarized with orienting tasks that induced more perceptual or more conceptual processing prior to study and test phases. Results from recognition tests replicated the results from Akan and Benjamin (2023): (1) Discriminability was enhanced when prior familiarity involved conceptual processing but not when it involved perceptual processing, and (2) familiarity gained through either perceptual or conceptual processing led to an increase in both correct and false identifications. This successful replication in a design with exclusively novel images indicates that the discriminability advantage provided by conceptual familiarity goes beyond memory for facial images and applies to memory for faces. These findings have implications in practical contexts, such as eyewitness identification situations involving suspects who are previously known or familiar to the witness.

The importance of conceptual knowledge when becoming familiar with faces during naturalistic viewing

Although the ability to recognise familiar faces is a critical part of everyday life, the process by which a face becomes familiar in the real world is not fully understood. Previous research has focussed on the importance of perceptual experience. However, in natural viewing, perceptual experience with faces is accompanied by increased knowledge about the person and the context in which they are encountered. Although conceptual information is known to be crucial for the formation of new episodic memories, it requires a period of consolidation. It is unclear, however, whether a similar process occurs when we learn new faces. Using a natural viewing paradigm, we investigated how the context in which events are presented influences our understanding of those events and whether, after a period of consolidation, this has a subsequent effect on face recognition. The context was manipulated by presenting events in 1) the original sequence, or 2) a scrambled sequence. Although this manipulation was predicted to have a significant effect on conceptual understanding of events, it had no effect on overall visual experience with the faces. Our prediction was that this contextual manipulation would affect face recognition after the information has been consolidated into memory. We found that understanding of the narrative was greater for participants who viewed the movie in the original sequence compared to those that viewed the movie in a scrambled order. To determine if the context in which the movie was viewed had an effect on face recognition, we compared recognition in the original and scrambled condition. We found an overall effect of conceptual knowledge on face recognition. That is, participants who viewed the original sequence had higher face recognition compared to participants who viewed the scrambled sequence. However, our planned comparisons did not reveal a greater effect of conceptual knowledge on face recognition after consolidation. In an exploratory analysis, we found that overlap in conceptual knowledge between participants was significantly correlated with the overlap in face recognition. We also found that this relationship was greater after a period of consolidation. Together, these findings provide new insights into the role of non-visual, conceptual knowledge for face recognition during natural viewing.

Multiple images captured from a single encounter do not promote face learning

Viewing multiple images of a newly encountered face improves recognition of that identity in new instances. Studies examining face learning have presented high-variability (HV) images that incorporate changes that occur from moment-to-moment (e.g., head orientation and expression) and over time (e.g., lighting, hairstyle, and health). We examined whether low-variability (LV) images (i.e., images that incorporate only moment-to-moment changes) also promote generalisation of learning such that novel instances are recognised. Participants viewed a single image, six LV images, or six HV images of a target identity before being asked to recognise novel images of that identity in a face matching task (training stimuli remained visible) or a memory task (training stimuli were removed). In Experiment 1 (n = 71), participants indicated which image(s) in 8-image arrays belonged to the target identity. In Experiment 2 (n = 73), participants indicated whether sequentially presented images belonged to the target identity. Relative to the single-image condition, sensitivity to identity improved and response biases were less conservative in the HV condition; we found no evidence of generalisation of learning in the LV condition regardless of testing protocol. Our findings suggest that day-to-day variability in appearance plays an essential role in acquiring expertise with a novel face.

Acoustic compression in Zoom audio does not compromise voice recognition performance

Human voice recognition over telephone channels typically yields lower accuracy when compared to audio recorded in a studio environment with higher quality. Here, we investigated the extent to which audio in video conferencing, subject to various lossy compression mechanisms, affects human voice recognition performance. Voice recognition performance was tested in an old-new recognition task under three audio conditions (telephone, Zoom, studio) across all matched (familiarization and test with same audio condition) and mismatched combinations (familiarization and test with different audio conditions). Participants were familiarized with female voices presented in either studio-quality (N = 22), Zoom-quality (N = 21), or telephone-quality (N = 20) stimuli. Subsequently, all listeners performed an identical voice recognition test containing a balanced stimulus set from all three conditions. Results revealed that voice recognition performance (d') in Zoom audio was not significantly different to studio audio but both in Zoom and studio audio listeners performed significantly better compared to telephone audio. This suggests that signal processing of the speech codec used by Zoom provides equally relevant information in terms of voice recognition compared to studio audio. Interestingly, listeners familiarized with voices via Zoom audio showed a trend towards a better recognition performance in the test (p = 0.056) compared to listeners familiarized with studio audio. We discuss future directions according to which a possible advantage of Zoom audio for voice recognition might be related to some of the speech coding mechanisms used by Zoom.

The impact of stability in appearance on the development of facial representations

The way faces become familiar and what information is represented as familiarity develops has puzzled researchers in the field of human face recognition for decades. In this paper, we present three experiments serving as proof of concept for a cost-efficient mechanism of face learning describing how facial representations form over time and accounting for recognition errors. We propose that the encoding of facial information is dynamic and modulated by the intrinsic stability in individual faces' appearance. We drew on a robust and ecological method using a proxy of exposure to famous faces in the real world and manipulated test images to assess the prediction that recognition of famous faces is affected by their relative stability in appearance. We consistently show that stable facial appearances (like Tom Cruise's) facilitate recognition in early stages of familiarisation but that performance does not improve much over time. In contrast, variations in appearance (like Jared Leto's) hinder recognition at first but improve performance with further media exposure. This pattern of results is consistent with the proposed cost-efficient face learning mechanism whereby facial representations build on a foundation of large-scale diagnostic information and refine over time if needed. When coarse information loses its diagnostic value through the experience of variations in appearance across encounters, diagnostic facial details and/or their spatial relationships must receive more weights, leading to refined representations that are more discriminative and reliable than representations of stable faces.

Not the norm: Face likeness is not the same as similarity to familiar face prototypes

Face images depicting the same individual can differ substantially from one another. Ecological variation in pose, expression, lighting, and other sources of appearance variability complicates the recognition and matching of unfamiliar faces, but acquired familiarity leads to the ability to cope with these challenges. Among the many ways that face of the same individual can vary, some images are judged to be better likenesses of familiar individuals than others. Simply put, these images look more like the individual under consideration than others. But what does it mean for an image to be a better likeness than another? Does likeness entail typicality, or is it something distinct from this? We examined the relationship between the likeness of face images and the similarity of those images to average images of target individuals using a set of famous faces selected for reciprocal familiarity/unfamiliarity across US and UK participants. We found that though likeness judgments are correlated with similarity-to-prototype judgments made by both familiar and unfamiliar participants, this correlation was smaller than the correlation between similarity judgments made by different participant groups. This implies that while familiarity weakens the relationship between likeness and similarity-to-prototype judgments, it does not change similarity-to-prototype judgments to the same degree.

Provoked overt recognition in acquired prosopagnosia using multiple different images of famous faces

Provoked overt recognition refers to the fact that patients with acquired prosopagnosia can sometimes recognize faces when presented in arrays of individuals from the same category (e.g., actors or politicians). We ask whether a prosopagnosic patient might experience recognition when presented with multiple different images of the same face simultaneously. Over two sessions, patient Herschel, a 66-year-old British man with acquired prosopagnosia, viewed face images individually or in arrays. On several occasions he failed to recognize single photos of an individual but successfully identified that person when the same photos were presented together. For example, Herschel failed to recognize any individual images of King Charles or Paul McCartney but recognised both in arrays of the same photos. Like reports based on category membership, overt recognition was transient and inconsistent. These findings are discussed in terms of models of covert recognition, alongside more recent research on within-person variability for face perception.

Unfamiliar face matching ability predicts the slope of face learning

We provide the first examination of individual differences in the efficiency of face learning. Investigating individual differences in face learning can illuminate potential mechanisms and provide greater understanding of why certain individuals might be more efficient face learners. Participants completed two unfamiliar face matching tasks and a learning task in which learning was assessed after viewing 1, 3, 6, and 9 images of to-be-learned identities. Individual differences in the slope of face learning (i.e., increases in sensitivity to identity) were predicted by the ability to discriminate between matched (same-identity) vs. mismatched (different-identity) pairs of wholly unfamiliar faces. A Dual Process Signal Detection model showed that three parameters increased with learning: Familiarity (an unconscious type of memory that varies in strength), recollection-old (conscious recognition of a learned identity), and recollection-new (conscious/confident rejection of novel identities). Good (vs. poor) matchers had higher Recollection-Old scores throughout learning and showed a steeper increase in Recollection-New. We conclude that good matchers are better able to capitalize on exposure to within-person variability in appearance, an effect that is attributable to their conscious memory for both learned and novel faces. These results have applied implications and will inform contemporary and traditional models of face identification.

From Pictures to the People in Them: Averaging Within-Person Variability Leads to Face Familiarization

Familiar faces can be confidently recognized despite sometimes radical changes in their appearance. Exposure to within-person variability-differences in facial characteristics over successive encounters-contributes to face familiarization. Research also suggests that viewers create mental averages of the different views of faces they encounter while learning them. Averaging over within-person variability is thus a promising mechanism for face familiarization. In Experiment 1, 153 Canadian undergraduates (88 female; age: M = 21 years, SD = 5.24) learned six target identities from eight different photos of each target interspersed among 32 distractor identities. Face-matching accuracy improved similarly irrespective of awareness of the target's identity, confirming that target faces presented among distractors can be learned incidentally. In Experiment 2, 170 Canadian undergraduates (125 female; age: M = 22.6 years, SD = 6.02) were tested using a novel indirect measure of learning. The results show that viewers update a mental average of a person's face as it becomes learned. Our findings are the first to show how averaging within-person variability over time leads to face familiarization.

Perceptual similarity modulates effects of learning from variability on face recognition

Face recognition is a challenging classification task that humans perform effortlessly for familiar faces. Recent studies have emphasized the importance of exposure to high variability appearances of the same identity to perform this task. However, these studies did not explicitly measure the perceptual similarity between the learned images and the images presented at test, which may account for the advantage of learning from high variability. Particularly, randomly selected test images are more likely to be perceptually similar to learned high variability images, and dissimilar to learned low variability images. Here we dissociated effects of learning from variability and study-test perceptual similarity, by collecting human similarity ratings for the study and test images. Using these measures, we independently manipulated the variability between the learning images and their perceptual similarity to the test images. Different groups of participants learned face identities from a low or high variability set of images. The learning phase was followed by a face matching test (Experiment 1) or a face recognition task (Experiment 2) that presented novel images of the learned identities that were perceptually dissimilar or similar to the learned images. Results of both experiments show that perceptual similarity between study and test, rather than image variability at learning per se, predicts face recognition. We conclude that learning from high variability improves face recognition for perceptually similar but not for perceptually dissimilar images. These findings may not be specific to faces and should be similarly evaluated for other domains.

What happens to our representation of identity as familiar faces age? Evidence from priming and identity aftereffects

Matching identity in images of unfamiliar faces is error prone, but we can easily recognize highly variable images of familiar faces - even images taken decades apart. Recent theoretical development based on computational modelling can account for how we recognize extremely variable instances of the same identity. We provide complementary behavioural data by examining older adults' representation of older celebrities who were also famous when young. In Experiment 1, participants completed a long-lag repetition priming task in which primes and test stimuli were the same age or different ages. In Experiment 2, participants completed an identity after effects task in which the adapting stimulus was an older or young photograph of one celebrity and the test stimulus was a morph between the adapting identity and a different celebrity; the adapting stimulus was the same age as the test stimulus on some trials (e.g., both old) or a different age (e.g., adapter young, test stimulus old). The magnitude of priming and identity after effects were not influenced by whether the prime and adapting stimulus were the same age or different age as the test face. Collectively, our findings suggest that humans have one common mental representation for a familiar face (e.g., Paul McCartney) that incorporates visual changes across decades, rather than multiple age-specific representations. These findings make novel predictions for state-of-the-art algorithms (e.g., Deep Convolutional Neural Networks).

The cultural learning account of first impressions

Humans spontaneously attribute character traits to strangers based on their facial appearance. Although these 'first impressions' typically have no basis in reality, some authors have assumed that they have an innate origin. By contrast, the Trait Inference Mapping (TIM) account proposes that first impressions are products of culturally acquired associative mappings that allow activation to spread from representations of facial appearance to representations of trait profiles. According to TIM, cultural instruments, including propaganda, illustrated storybooks, art and iconography, ritual, film, and TV, expose many individuals within a community to common sources of correlated face-trait experience, yielding first impressions that are shared by many, but typically inaccurate. Here, we review emerging empirical findings, many of which accord with TIM, and argue that future work must distinguish first impressions based on invariant facial features (e.g., shape) from those based on facial behaviours (e.g., expressions).

The pairs training effect in unfamiliar face matching

A wealth of studies have shown that humans are remarkably poor at determining whether two face images show the same person or not (face matching). Given the prevalence of photo-ID, and the fact that people employed to check photo-ID are typically unfamiliar with the person pictured, there is a need to improve unfamiliar face matching accuracy. One method of improvement is to have participants complete the task in a pair, which results in subsequent improvements in the low performer (“the pairs training effect”). Here, we sought to replicate the original finding, to test the longevity of the pairs training effect, and to shed light on the potential underlying mechanisms. In two experiments, we replicated the pairs training effect and showed it is maintained after a delay (Experiment 1). We found no differences between high and low performers in confidence (Experiment 1) or response times (Experiment 2), and the content of the pairs’ discussions (Experiment 2) did not explain the results. The pairs training effect in unfamiliar face matching is robust, but the mechanisms underlying the effects remain as yet unexplained.

Impaired grouping of ambient facial images in autism

Ambient facial images depict individuals from a variety of viewing angles, with a range of poses and expressions, under different lighting conditions. Exposure to ambient images is thought to help observers form robust representations of the individuals depicted. Previous results suggest that autistic people may derive less benefit from exposure to this exemplar variation than non-autistic people. To date, however, it remains unclear why. One possibility is that autistic individuals possess atypical perceptual learning mechanisms. Alternatively, however, the learning mechanisms may be intact, but receive low-quality perceptual input from face encoding processes. To examine this second possibility, we investigated whether autistic people are less able to group ambient images of unfamiliar individuals based on their identity. Participants were asked to identify which of four ambient images depicted an oddball identity. Each trial assessed the grouping of different facial identities, thereby preventing face learning across trials. As such, the task assessed participants’ ability to group ambient images of unfamiliar people. In two experiments we found that matched non-autistic controls correctly identified the oddball identities more often than our autistic participants. These results imply that poor face learning from variation by autistic individuals may well be attributable to low-quality perceptual input, not aberrant learning mechanisms.

The Episodic Prototypes Model (EPM): On the nature and genesis of facial representations

Faces undergo massive changes over time and life events. We need a mental representation which is flexible enough to cope with the existing visual varieties, but which is also stable enough to be the basis for valid recognition. Two main theoretical frameworks exist to describe facial representations: prototype models assuming one central item comprising all visual experiences of a face, and exemplar models assuming single representations of each visual experience of a face. We introduce a much more ecological valid model dealing with episodic prototypes (the Episodic Prototypes Model—EPM), where faces are represented by a low number of prototypes that refer to specific Episodes of Life (EoL, e.g., early adulthood, mature age) during which the facial appearance shows only moderate variation. Such an episodic view of mental representation allows for efficient storage, as the number of needed prototypes is relatively low, and it allows for the needed variation within a prototype that keeps the everyday and steadily ongoing changes across a certain period of time. Studies 1–3 provide evidence that facial representations are highly dependent on temporal aspects which is in accord with EoL, and that individual learning history generates the structure and content of respective prototypes. In Study 4, we used implicit measures (RT) in a face verification task to investigate the postulated power of the EPM. We could demonstrate that episodic prototypes clearly outperformed visual depictions of exhaustive prototypes, supporting the general idea of our approach.

Face Recognition by Humans and Machines: Three Fundamental Advances from Deep Learning

Deep learning models currently achieve human levels of performance on real-world face recognition tasks. We review scientific progress in understanding human face processing using computational approaches based on deep learning. This review is organized around three fundamental advances. First, deep networks trained for face identification generate a representation that retains structured information about the face (e.g., identity, demographics, appearance, social traits, expression) and the input image (e.g., viewpoint, illumination). This forces us to rethink the universe of possible solutions to the problem of inverse optics in vision. Second, deep learning models indicate that high-level visual representations of faces cannot be understood in terms of interpretable features. This has implications for understanding neural tuning and population coding in the high-level visual cortex. Third, learning in deep networks is a multistep process that forces theoretical consideration of diverse categories of learning that can overlap, accumulate over time, and interact. Diverse learning types are needed to model the development of human face processing skills, cross-race effects, and familiarity with individual faces.

Unimodal and cross-modal identity judgements using an audio-visual sorting task: Evidence for independent processing of faces and voices

Unimodal and cross-modal information provided by faces and voices contribute to identity percepts. To examine how these sources of information interact, we devised a novel audio-visual sorting task in which participants were required to group video-only and audio-only clips into two identities. In a series of three experiments, we show that unimodal face and voice sorting were more accurate than cross-modal sorting: While face sorting was consistently most accurate followed by voice sorting, cross-modal sorting was at chancel level or below. In Experiment 1, we compared performance in our novel audio-visual sorting task to a traditional identity matching task, showing that unimodal and cross-modal identity perception were overall moderately more accurate than the traditional identity matching task. In Experiment 2, separating unimodal from cross-modal sorting led to small improvements in accuracy for unimodal sorting, but no change in cross-modal sorting performance. In Experiment 3, we explored the effect of minimal audio-visual training: Participants were shown a clip of the two identities in conversation prior to completing the sorting task. This led to small, nonsignificant improvements in accuracy for unimodal and cross-modal sorting. Our results indicate that unfamiliar face and voice perception operate relatively independently with no evidence of mutual benefit, suggesting that extracting reliable cross-modal identity information is challenging.

Seeing through disguise: Getting to know you with a deep convolutional neural network

People use disguise to look unlike themselves (evasion) or to look like someone else (impersonation). Evasion disguise challenges human ability to see an identity across variable images; Impersonation challenges human ability to tell people apart. Personal familiarity with an individual face helps humans to see through disguise. Here we propose a model of familiarity based on high-level visual learning mechanisms that we tested using a deep convolutional neural network (DCNN) trained for face identification. DCNNs generate a face space in which identities and images co-exist in a unified computational framework, that is categorically structured around identity, rather than retinotopy. This allows for simultaneous manipulation of mechanisms that contrast identities and cluster images. In Experiment 1, we measured the DCNN's baseline accuracy (unfamiliar condition) for identification of faces in no disguise and disguise conditions. Disguise affected DCNN performance in much the same way it affects human performance for unfamiliar faces in disguise (cf. Noyes & Jenkins, 2019). In Experiment 2, we simulated familiarity for individual identities by averaging the DCNN-generated representations from multiple images of each identity. Averaging improved DCNN recognition of faces in evasion disguise, but reduced the ability of the DCNN to differentiate identities of similar appearance. In Experiment 3, we implemented a contrast learning technique to simultaneously teach the DCNN appearance variation and identity contrasts between different individuals. This facilitated identification with both evasion and impersonation disguise. Familiar face recognition requires an ability to group images of the same identity together and separate different identities. The deep network provides a high-level visual representation for face recognition that supports both of these mechanisms of face learning simultaneously.

Understanding face identification through within-person variability in appearance: Introduction to a virtual special issue

In the effort to determine the cognitive processes underlying the identification of faces, the dissimilarities between images of different people have long been studied. In contrast, the inherent variability between different images of the same face has either been treated as a nuisance variable that should be eliminated from psychological experiments or it has not been considered at all. Over the past decade, research efforts have increased substantially to demonstrate that this within-person variation is meaningful and can give insight into various processes of face identification, such as identity matching, face learning, and familiar face recognition. In this virtual special issue of the Quarterly Journal of Experimental Psychology, we explain the importance of within-person variability for face identification and bring together recent relevant articles published in the journal.

Comparing unfamiliar voice and face identity perception using identity sorting tasks

Identity sorting tasks, in which participants sort multiple naturally varying stimuli of usually two identities into perceived identities, have recently gained popularity in voice and face processing research. In both modalities, participants who are unfamiliar with the identities tend to perceive multiple stimuli of the same identity as different people and thus fail to "tell people together." These similarities across modalities suggest that modality-general mechanisms may underpin sorting behaviour. In this study, participants completed a voice sorting and a face sorting task. Taking an individual differences approach, we asked whether participants' performance on voice and face sorting of unfamiliar identities is correlated. Participants additionally completed a voice discrimination (Bangor Voice Matching Test) and a face discrimination task (Glasgow Face Matching Test). Using these tasks, we tested whether performance on sorting related to explicit identity discrimination. Performance on voice sorting and face sorting tasks was correlated, suggesting that common modality-general processes underpin these tasks. However, no significant correlations were found between sorting and discrimination performance, with the exception of significant relationships for performance on "same identity" trials with "telling people together" for voices and faces. Overall, any reported relationships were however relatively weak, suggesting the presence of additional modality-specific and task-specific processes.

Face search in CCTV surveillance

Background

We present a series of experiments on visual search in a highly complex environment, security closed-circuit television (CCTV). Using real surveillance footage from a large city transport hub, we ask viewers to search for target individuals. Search targets are presented in a number of ways, using naturally occurring images including their passports and photo ID, social media and custody images/videos. Our aim is to establish general principles for search efficiency within this realistic context.

Results

Across four studies we find that providing multiple photos of the search target consistently improves performance. Three different photos of the target, taken at different times, give substantial performance improvements by comparison to a single target. By contrast, providing targets in moving videos or with biographical context does not lead to improvements in search accuracy.

Conclusions

We discuss the multiple-image advantage in relation to a growing understanding of the importance of within-person variability in face recognition.

Electronic supplementary material

The online version of this article (10.1186/s41235-019-0193-0) contains supplementary material, which is available to authorized users.

Two Sides of Face Learning: Improving Between-Identity Discrimination While Tolerating More Within-Person Variability in Appearance

Two photos of an unfamiliar face are often perceived as belonging to different people—an error that disappears when a face is familiar. Face learning has been characterized as increased tolerance of within-person variability in appearance and is facilitated by exposure to such variability (e.g., differences in expression, lighting, and aesthetics). We hypothesized that increased tolerance of variability in appearance might lead to reduced discrimination and that misidentifications would be reduced if a face was learned in the context of a similar-looking identity. After validating our stimuli (Experiments 1a and 1b), we conducted three experiments investigating face learning. In two of these, participants learned three faces (Experiment 2: 15 images/identity and Experiment 3: 5 images/identity), two of which were similar. In a recognition task, misidentifications did not change as a function of similarity, although participants recognized more images of the target in Experiment 2 (i.e., after learning 15 images). In Experiment 4, participants learned one identity and the number of images studied varied across groups. Recognition of new images increased with the number of images studied, with no changes in false alarms; sensitivity (A′) marginally increased. The results suggest that recognition and discrimination reflect separable processes with minimal influence of between-person similarity on discrimination.

Acoustic voice variation within and between speakers

Little is known about the nature or extent of everyday variability in voice quality. This paper describes a series of principal component analyses to explore within- and between-talker acoustic variation and the extent to which they conform to expectations derived from current models of voice perception. Based on studies of faces and cognitive models of speaker recognition, the authors hypothesized that a few measures would be important across speakers, but that much of within-speaker variability would be idiosyncratic. Analyses used multiple sentence productions from 50 female and 50 male speakers of English, recorded over three days. Twenty-six acoustic variables from a psychoacoustic model of voice quality were measured every 5 ms on vowels and approximants. Across speakers the balance between higher harmonic amplitudes and inharmonic energy in the voice accounted for the most variance (females = 20%, males = 22%). Formant frequencies and their variability accounted for an additional 12% of variance across speakers. Remaining variance appeared largely idiosyncratic, suggesting that the speaker-specific voice space is different for different people. Results further showed that voice spaces for individuals and for the population of talkers have very similar acoustic structures. Implications for prototype models of voice perception and recognition are discussed.

Flexible voices: Identity perception from variable vocal signals

Human voices are extremely variable: The same person can sound very different depending on whether they are speaking, laughing, shouting or whispering. In order to successfully recognise someone from their voice, a listener needs to be able to generalize across these different vocal signals (‘telling people together’). However, in most studies of voice-identity processing to date, the substantial within-person variability has been eliminated through the use of highly controlled stimuli, thus focussing on how we tell people apart. We argue that this obscures our understanding of voice-identity processing by controlling away an essential feature of vocal stimuli that may include diagnostic information. In this paper, we propose that we need to extend the focus of voice-identity research to account for both “telling people together” as well as “telling people apart.” That is, we must account for whether, and to what extent, listeners can overcome within-person variability to obtain a stable percept of person identity from vocal cues. To do this, our theoretical and methodological frameworks need to be adjusted to explicitly include the study of within-person variability.

Face averages and multiple images in a live matching task

We know from previous research that unfamiliar face matching (determining whether two simultaneously presented images show the same person or not) is very error-prone. A small number of studies in laboratory settings have shown that the use of multiple images or a face average, rather than a single image, can improve face matching performance. Here, we tested 1,999 participants using four-image arrays and face averages in two separate live matching tasks. Matching a single image to a live person resulted in numerous errors (79.9% accuracy across both experiments), and neither multiple images (82.4% accuracy) nor face averages (76.9% accuracy) improved performance. These results are important when considering possible alterations which could be made to photo-ID. Although multiple images and face averages have produced measurable improvements in performance in recent laboratory studies, they do not produce benefits in a real-world live face matching context.

Individual differences in eyewitness accuracy across multiple lineups of faces

Theories of face recognition in cognitive psychology stipulate that the hallmark of accurate identification is the ability to recognize a person consistently, across different encounters. In this study, we apply this reasoning to eyewitness identification by assessing the recognition of the same target person repeatedly, over six successive lineups. Such repeat identifications are challenging and can be performed only by a proportion of individuals, both when a target exhibits limited and more substantial variability in appearance across lineups (Experiments 1 and 2). The ability to do so correlates with individual differences in identification accuracy on two established tests of unfamiliar face recognition (Experiment 3). This indicates that most observers have limited facial representations of target persons in eyewitness scenarios, which do not allow for robust identification in most individuals, partly due to limitations in their ability to recognize unfamiliar faces. In turn, these findings suggest that consistency of responses across multiple lineups of faces could be applied to assess which individuals are accurate eyewitnesses.

Search templates that incorporate within-face variation improve visual search for faces

Searching for unfamiliar faces in crowds is an important task in modern society. In surveillance and security settings, it is sometimes critical to locate a target individual quickly and accurately. In this study, we examine whether we can improve search efficiency in these visual search tasks by changing the face information that is provided to participants. In Experiment 1, we compare speed and accuracy of visual search when searching for unfamiliar and familiar faces after being exposed to either a single exemplar image or a face average created from multiple images of the target face. In Experiment 2, we compare search efficiency when single exemplars and multiple exemplars are provided. Consistent with studies of unfamiliar face matching tasks, we find that, relative to a single image, having multiple images of the target improves the accuracy of visual search. In Experiment 3, we compared search performance for face averages and multiple exemplars while also varying crowd size. Multiple exemplars conferred an additional advantage over face averages, suggesting that exposure to within-face variability results in the best search performance. We discuss the implications of these findings for face-in-a-crowd search and visual search tasks more generally.

How many voices did you hear? Natural variability disrupts identity perception from unfamiliar voices

Our voices sound different depending on the context (laughing vs. talking to a child vs. giving a speech), making within‐person variability an inherent feature of human voices. When perceiving speaker identities, listeners therefore need to not only ‘tell people apart’ (perceiving exemplars from two different speakers as separate identities) but also ‘tell people together’ (perceiving different exemplars from the same speaker as a single identity). In the current study, we investigated how such natural within‐person variability affects voice identity perception. Using voices from a popular TV show, listeners, who were either familiar or unfamiliar with this show, sorted naturally varying voice clips from two speakers into clusters to represent perceived identities. Across three independent participant samples, unfamiliar listeners perceived more identities than familiar listeners and frequently mistook exemplars from the same speaker to be different identities. These findings point towards a selective failure in ‘telling people together’. Our study highlights within‐person variability as a key feature of voices that has striking effects on (unfamiliar) voice identity perception. Our findings not only open up a new line of enquiry in the field of voice perception but also call for a re‐evaluation of theoretical models to account for natural variability during identity perception.

Improving face identification with specialist teams

People vary in their ability to identify faces, and this variability is relatively stable across repeated testing. This suggests that recruiting high performers can improve identity verification accuracy in applied settings. Here, we report the first systematic study to evaluate real-world benefits of selecting high performers based on performance in standardized face identification tests. We simulated a recruitment process for a specialist team tasked with detecting fraudulent passport applications. University students (n = 114) completed a battery of screening tests followed by a real-world face identification task that is performed routinely when issuing identity documents. Consistent with previous work, individual differences in the real-world task were relatively stable across repeated tests taken 1 week apart (r = 0.6), and accuracy scores on screening tests and the real-world task were moderately correlated. Nevertheless, performance gains achieved by selecting groups based on screening tests were surprisingly small, leading to a 7% improvement in accuracy. Statistically aggregating decisions across individuals-using a 'wisdom of crowds' approach-led to more substantial gains than selection alone. Finally, controlling for individual accuracy of team members, the performance of a team in one test predicted their performance in a subsequent test, suggesting that a 'good team' is not only defined by the individual accuracy of team members. Overall, these results underline the need to use a combination of approaches to improve face identification performance in professional settings.

More than a sum of parts: robust face recognition by integrating variation

Familiarity incrementally improves our ability to identify faces. It has been hypothesized that this improvement reflects the refinement of memory representations which incorporate variation in appearance across encounters. Although it is established that exposure to variation improves face identification accuracy, it is not clear how variation is assimilated into internal face representations. To address this, we used a novel approach to isolate the effect of integrating separate exposures into a single-identity representation. Participants (n = 113) were exposed to either a single video clip or a pair of video clips of target identities. Pairs of video clips were presented as either a single identity (associated with a single name, e.g. Betty-Sue) or dual identities (associated with two names, e.g. Betty and Sue). Results show that participants exposed to pairs of video clips showed better matching performance compared with participants trained with a single clip. More importantly, identification accuracy was higher for faces presented as single identities compared to faces presented as dual identities. This provides the first direct evidence that the integration of information across separate exposures benefits face matching, thereby establishing a mechanism that may explain people's impressive ability to recognize familiar faces.

Expressive Faces Confuse Identity

We used highly variable, so-called 'ambient' images to test whether expressions affect the identity recognition of real-world facial images. Using movie segments of two actors unknown to our participants, we created image pairs - each image within a pair being captured from the same film segment. This ensured that, within pairs, variables such as lighting were constant whilst expressiveness differed. We created two packs of cards, one containing neutral face images, the other, their expressive counterparts. Participants sorted the card packs into piles, one for each perceived identity. As with previous studies, the perceived number of identities was higher than the veridical number of two. Interestingly, when looking within piles, we found a strong difference between the expressive and neutral sorting tasks. With expressive faces, identity piles were significantly more likely to contain cards of both identities. This finding demonstrates that, over and above other image variables, expressiveness variability can cause identity confusion; evidently, expression is not disregarded or factored out when we classify facial identity in real-world images. Our results provide clear support for a face processing architecture in which both invariant and changeable facial information may be drawn upon to drive our decisions of identity.

Expression Dependence in the Perception of Facial Identity

We recognise familiar faces irrespective of their expression. This ability, crucial for social interactions, is a fundamental feature of face perception. We ask whether this constancy of facial identity may be compromised by changes in expression. This, in turn, addresses the issue of whether facial identity and expression are processed separately or interact. Using an identification task, participants learned the identities of two actors from naturalistic (so-called ambient) face images taken from movies. Training was either with neutral images or their expressive counterparts, perceived expressiveness having been determined experimentally. Expressive training responses were slower and more erroneous than neutral training responses. When tested with novel images of the actors that varied in expressiveness, neutrally trained participants gave slower and less accurate responses to images of high compared with low expressiveness. These findings clearly demonstrate that facial expressions impede the processing and learning of facial identity. Because this expression dependence is consistent with a late bifurcation model of face processing, in which changeable facial aspects and identity are coded in a common framework, it suggests that expressions are a part of facial identity representation.

The utility of multiple synthesized views in the recognition of unfamiliar faces

The ability to recognize an unfamiliar individual on the basis of prior exposure to a photograph is notoriously poor and prone to errors, but recognition accuracy is improved when multiple photographs are available. In applied situations, when only limited real images are available (e.g., from a mugshot or CCTV image), the generation of new images might provide a technological prosthesis for otherwise fallible human recognition. We report two experiments examining the effects of providing computer-generated additional views of a target face. In Experiment 1, provision of computer-generated views supported better target face recognition than exposure to the target image alone and equivalent performance to that for exposure of multiple photograph views. Experiment 2 replicated the advantage of providing generated views, but also indicated an advantage for multiple viewings of the single target photograph. These results strengthen the claim that identifying a target face can be improved by providing multiple synthesized views based on a single target image. In addition, our results suggest that the degree of advantage provided by synthesized views may be affected by the quality of synthesized material.

Modulation of the composite face effect by unintended emotion cues

When upper and lower regions from different emotionless faces are aligned to form a facial composite, observers 'fuse' the two halves together, perceptually. The illusory distortion induced by task-irrelevant ('distractor') halves hinders participants' judgements about task-relevant ('target') halves. This composite-face effect reveals a tendency to integrate feature information from disparate regions of intact upright faces, consistent with theories of holistic face processing. However, observers frequently perceive emotion in ostensibly neutral faces, contrary to the intentions of experimenters. This study sought to determine whether this 'perceived emotion' influences the composite-face effect. In our first experiment, we confirmed that the composite effect grows stronger as the strength of distractor emotion increased. Critically, effects of distractor emotion were induced by weak emotion intensities, and were incidental insofar as emotion cues hindered image matching, not emotion labelling per se. In Experiment 2, we found a correlation between the presence of perceived emotion in a set of ostensibly neutral distractor regions sourced from commonly used face databases, and the strength of illusory distortion they induced. In Experiment 3, participants completed a sequential matching composite task in which half of the distractor regions were rated high and low for perceived emotion, respectively. Significantly stronger composite effects were induced by the high-emotion distractor halves. These convergent results suggest that perceived emotion increases the strength of the composite-face effect induced by supposedly emotionless faces. These findings have important implications for the study of holistic face processing in typical and atypical populations.

Forming impressions of facial attractiveness is mandatory

First impressions of social traits, such as attractiveness, from faces are often claimed to be made automatically, given their speed and reliability. However, speed of processing is only one aspect of automaticity. Here we address a further aspect, asking whether impression formation is mandatory. Mandatory formation requires that impressions are formed about social traits even when this is task-irrelevant, and that once formed, these impressions are difficult to inhibit. In two experiments, participants learned what new people looked like for the purpose of future identification, from sets of images high or low in attractiveness. They then rated middle-attractiveness images of each person, for attractiveness. Even though instructed to rate the specific images, not the people, their ratings were biased by the attractiveness of the learned images. A third control experiment, with participants rating names, demonstrated that participants in Experiments 1 and 2 were not simply rating the people, rather than the specific images as instructed. These results show that the formation of attractiveness impressions from faces is mandatory, thus broadening the evidence for automaticity of facial impressions. The mandatory formation of impressions is likely to have an important impact in real-world situations such as online dating sites.

The Transfer of Object Learning after Training with Multiple Exemplars

Object recognition improves with training. This training effect only partially generalizes to untrained images of the trained objects (new exemplars, orientation,…). The aim of this study is to investigate whether and to what extent the learning transfer improves when participants are trained with more exemplars of an object. Participants were trained to recognize two sets of stimuli using a backward masking paradigm. During training with the first set, only one exemplar of each object was presented. The second set was trained using four exemplars of each object. After 3 days of training, participants were tested on all the trained exemplars and a completely new exemplar of the same objects. In addition, recognition performance was compared to a set of completely new objects. For the objects of which four exemplars were used during training, participants showed more generalization toward new exemplars compared to when they were only trained with one exemplar. Part of the generalization effect extended to completely new objects. In conclusion, more variation during training leads to more generalization toward new visual stimuli.

Dynamic Emotional Faces Generalise Better to a New Expression but not to a New View

Prior research based on static images has found limited improvement for recognising previously learnt faces in a new expression after several different facial expressions of these faces had been shown during the learning session. We investigated whether non-rigid motion of facial expression facilitates the learning process. In Experiment 1, participants remembered faces that were either presented in short video clips or still images. To assess the effect of exposure to expression variation, each face was either learnt through a single expression or three different expressions. Experiment 2 examined whether learning faces in video clips could generalise more effectively to a new view. The results show that faces learnt from video clips generalised effectively to a new expression with exposure to a single expression, whereas faces learnt from stills showed poorer generalisation with exposure to either single or three expressions. However, although superior recognition performance was demonstrated for faces learnt through video clips, dynamic facial expression did not create better transfer of learning to faces tested in a new view. The data thus fail to support the hypothesis that non-rigid motion enhances viewpoint invariance. These findings reveal both benefits and limitations of exposures to moving expressions for expression-invariant face recognition.

Learning faces from variability

Research on face learning has tended to use sets of images that vary systematically on dimensions such as pose and illumination. In contrast, we have proposed that exposure to naturally varying images of a person may be a critical part of the familiarization process. Here, we present two experiments investigating face learning with "ambient images"-relatively unconstrained photos taken from internet searches. Participants learned name and face associations for unfamiliar identities presented in high or low within-person variability-that is, images of the same person returned by internet search on their name (high variability) versus different images of the same person taken from the same event (low variability). In Experiment 1 we show more accurate performance on a speeded name verification task for identities learned in high than in low variability, when the test images are completely novel photos. In Experiment 2 we show more accurate performance on a face matching task for identities previously learned in high than in low variability. The results show that exposure to a large range of within-person variability leads to enhanced learning of new identities.

Variation in Photos of the Same Face Drives Improvements in Identity Verification

People are poor at matching the identity of unfamiliar faces, but very good at identifying familiar faces. Theoretical accounts suggest that representations derived from exposure to variation are instrumental in driving this familiarity based improvement. In support of this, recent work shows that providing multiple photographs of an unfamiliar face improves identity verification accuracy. Here, we test whether the extent of variation is critical to this improvement, by manipulating the degree of within-identity variation that participants are exposed to in a sequential matching test. Participants were more accurate and adopted more liberal response criteria, when matching high-variability pairs to probe images, compared with either low-variability pairs or single images. Importantly, benefits of variation are not explained by independent contributions of single images, suggesting that people extrapolate information across images to produce gains in identification accuracy. These results suggest that photo-ID can be improved by incorporating broader ranges of variation in facial appearance.