Ensemble coding of facial identity is robust, but may not contribute to face learning

Ensemble coding - the rapid extraction of a perceptual average - has been proposed as a potential mechanism underlying face learning. We tested this proposal across five pre-registered experiments in which four ambient images of an identity were presented in the study phase. In Experiments 1 and 2a-c, participants were asked whether a test image was in the study array; these experiments examined the robustness of ensemble coding. Experiment 1 replicated ensemble coding in an online sample; participants recognize images from the study array and the average of those images. Experiments 2a-c provide evidence that ensemble coding meets several criteria of a possible learning mechanism: It is robust to changes in head orientation (± 60o), survives a short (30s) delay, and persists when images of two identities are interleaved during the study phase. Experiment 3 examined whether ensemble coding is sufficient for face learning (i.e., facilitates recognition of novel images of a target identity). Each study array comprised four ambient images (variability + average), a single image, or an average of four images (average only). Participants were asked whether a novel test image showed the identity from a study array. Performance was best in the four-image condition, with no difference between the single-image and average-only conditions. We conclude that ensemble coding of facial identity is robust but that the perceptual average per se is not sufficient for face learning.

Modeling face recognition in the predictive coding framework: A combined computational modeling and functional imaging study

The learning of new facial identities and the recognition of familiar faces are crucial processes for social interactions. Recently, a combined computational modeling and functional magnetic resonance imaging (fMRI) study used predictive coding as a biologically plausible framework to model face identity learning and to relate specific model parameters with brain activity (Apps and Tsakiris, Nat Commun 4, 2698, 2013). On the one hand, it was shown that behavioral responses on a two-option face recognition task could be predicted by the level of contextual and facial familiarity in a computational model derived from predictive-coding principles. On the other hand, brain activity in specific brain regions was associated with these parameters. More specifically, brain activity in the superior temporal sulcus (STS) varied with contextual familiarity, whereas activity in the fusiform face area (FFA) covaried with the prediction error parameter that updated facial familiarity. Literature combining fMRI assessments and computational modeling in humans still needs to be expanded. Furthermore, prior results are largely not replicated. The present study was, therefore, specifically set up to replicate these previous findings. Our results support the original findings in two critical aspects. First, on a group level, the behavioral responses were modeled best by the same computational model reported by the original authors. Second, we showed that estimates of these model parameters covary with brain activity in specific, face-sensitive brain regions. Our results thus provide further evidence that the functional properties of the face perception network conform to central principles of predictive coding. However, our study yielded diverging findings on specific computational model parameters reflected in brain activity. On the one hand, we did not find any evidence of a computational involvement of the STS. On the other hand, our results showed that activity in the right FFA was associated with multiple computational model parameters. Our data do not provide evidence for functional segregation between particular face-sensitive brain regions, as previously proposed.

Why can people with developmental prosopagnosia recognise some familiar faces? Insights from subjective experience

Developmental prosopagnosia is a relatively common visuo-cognitive condition, characterised by impaired facial identity recognition. Impairment severity appears to reside on a continuum, however, it is unknown whether instances of milder deficits reflect the successful use of spontaneous (typical) face recognition strategies, or the application of extraneous compensatory cues to recognition. Here, we explore this issue in two studies. First, 23 adults with developmental prosopagnosia were asked about their use of spontaneous versus compensatory face recognition techniques in everyday life, using a series of closed- and open-ended questions. Second, the same participants performed a computerised famous face recognition task where they were asked to provide reasons why they could make any successful identifications. Findings from both studies suggest that people with developmental prosopagnosia can successfully, and quite frequently, use compensatory strategies to recognition, and that these cues support the majority of instances of preserved familiar face recognition. In contrast, 16 of the 23 participants were able to spontaneously recognise familiar faces on at least some occasions, but there were vast individual differences in frequencies of success. These findings have important implications for our conceptualisation of the condition, as well as for diagnostic practice.

Developing familiarity during the first eight months of knowing a person: A longitudinal EEG study on face and identity learning

It is well-established that familiar and unfamiliar faces are processed differently, but surprisingly little is known about how familiarity builds up over time and how novel faces gradually become represented in the brain. Here, we used event-related brain potentials (ERPs) in a pre-registered, longitudinal study to examine the neural processes accompanying face and identity learning during the first eight months of knowing a person. Specifically, we examined how increasing real-life familiarity affects visual recognition (N250 Familiarity Effect) and the integration of person-related knowledge (Sustained Familiarity Effect, SFE). Sixteen first-year undergraduates were tested in three sessions, approximately one, five, and eight months after the start of the academic year, with highly variable "ambient" images of a new friend they had met at university and of an unfamiliar person. We observed clear ERP familiarity effects for the new friend after one month of familiarity. While there was an increase in the N250 effect over the course of the study, no change in the SFE was observed. These results suggest that visual face representations develop faster relative to the integration of identity-specific knowledge.

How quickly do we learn new faces in everyday life? Neurophysiological evidence for face identity learning after a brief real-life encounter

Faces learnt in a single experimental session elicit a familiarity effect in event-related brain potentials (ERPs), with more negative amplitudes for newly learnt relative to unfamiliar faces in the N250 component. However, no ERP study has examined face learning following a brief real-life encounter, and it is not clear how long it takes to learn new faces in such ecologically more valid conditions. To investigate these questions, the present study examined whether robust image-independent representations, as reflected in the N250 familiarity effect, could be established after a brief unconstrained social interaction by analysing the ERPs elicited by highly variable images of the newly learnt identity and an unfamiliar person. Significant N250 familiarity effects were observed after a 30-min (Experiment 1) and a 10-min (Experiment 2) encounter, and a trend was observed after 5 min of learning (Experiment 3), demonstrating that 5-10 min of exposure were sufficient for the initial establishment of image-independent representations. Additionally, the magnitude of the effects reported after 10 and 30 min was comparable suggesting that the first 10 min of a social encounter might be crucial, with extra 20 min from the same encounter not adding further benefit for the initial formation of robust face representations.

Learning faces from variability: Four- and five-year-olds differ from older children and adults

Children under 6 years of age have difficulty recognizing a familiar face across changes in appearance and telling the face apart from similar-looking people. Understanding the process by which newly encountered faces become familiar can provide insights into these difficulties. Exposure to the ways in which a person varies in appearance is one mechanism by which adults and older children (≥6 years) learn new faces. We provide the first investigation of whether this mechanism for face learning functions in younger children. Children aged 4 and 5 years were read two storybooks featuring an unfamiliar character. Participants viewed six images of the character in one story and one image of the character in the other story. After each story, children were asked to identify novel images of the character that were intermixed with images of a similar-looking distractor. Like older children, 4- and 5-year-olds were more sensitive to identity in the 6-image condition, but they also adapted a less conservative criterion. Young children identified more images of the character after viewing six images versus one image. However, many also incorrectly identified more images of the distractor after viewing six images versus one image, an effect not previously found for older children and adults. These results suggest that this mechanism for face learning is not fully refined before 6 years of age.

Multiple-image arrays in face matching tasks with and without memory

Previous research has shown that exposure to within-person variability facilitates face learning. A different body of work has examined potential benefits of providing multiple images in face matching tasks. Viewers are asked to judge whether a target face matches a single face image (as when checking photo-ID) or multiple face images of the same person. The evidence here is less clear, with some studies finding a small multiple-image benefit, and others finding no advantage. In four experiments, we address this discrepancy in the benefits of multiple images from learning and matching studies. We show that multiple-image arrays only facilitate face matching when arrays precede targets. Unlike simultaneous face matching tasks, sequential matching and learning tasks involve memory and require abstraction of a stable representation of the face from the array, for subsequent comparison with a target. Our results show that benefits from multiple-image arrays occur only when this abstraction is required, and not when array and target images are available at once. These studies reconcile apparent differences between face learning and face matching and provide a theoretical framework for the study of within-person variability in face perception.

Is having similar eye movement patterns during face learning and recognition beneficial for recognition performance? Evidence from hidden Markov modeling

The hidden Markov model (HMM)-based approach for eye movement analysis is able to reflect individual differences in both spatial and temporal aspects of eye movements. Here we used this approach to understand the relationship between eye movements during face learning and recognition, and its association with recognition performance. We discovered holistic (i.e., mainly looking at the face center) and analytic (i.e., specifically looking at the two eyes in addition to the face center) patterns during both learning and recognition. Although for both learning and recognition, participants who adopted analytic patterns had better recognition performance than those with holistic patterns, a significant positive correlation between the likelihood of participants' patterns being classified as analytic and their recognition performance was only observed during recognition. Significantly more participants adopted holistic patterns during learning than recognition. Interestingly, about 40% of the participants used different patterns between learning and recognition, and among them 90% switched their patterns from holistic at learning to analytic at recognition. In contrast to the scan path theory, which posits that eye movements during learning have to be recapitulated during recognition for the recognition to be successful, participants who used the same or different patterns during learning and recognition did not differ in recognition performance. The similarity between their learning and recognition eye movement patterns also did not correlate with their recognition performance. These findings suggested that perceptuomotor memory elicited by eye movement patterns during learning does not play an important role in recognition. In contrast, the retrieval of diagnostic information for recognition, such as the eyes for face recognition, is a better predictor for recognition performance.

Fixations to the eyes aids in facial encoding; covertly attending to the eyes does not

When looking at images of faces, people will often focus their fixations on the eyes. It has previously been demonstrated that the eyes convey important information that may improve later facial recognition. Whether this advantage requires that the eyes be fixated, or merely attended to covertly (i.e. while looking elsewhere), is unclear from previous work. While attending to the eyes covertly without fixating them may be sufficient, the act of using overt attention to fixate the eyes may improve the processing of important details used for later recognition. In the present study, participants were shown a series of faces and, in Experiment 1, asked to attend to them normally while avoiding looking at either the eyes or, as a control, the mouth (overt attentional avoidance condition); or in Experiment 2 fixate the center of the face while covertly attending to either the eyes or the mouth (covert attention condition). After the first phase, participants were asked to perform an old/new face recognition task. We demonstrate that a) when fixations to the eyes are avoided during initial viewing then subsequent face discrimination suffers, and b) covert attention to the eyes alone is insufficient to improve face discrimination performance. Together, these findings demonstrate that fixating the eyes provides an encoding advantage that is not availed by covert attention alone.

Evidence for view-invariant face recognition units in unfamiliar face learning

Many models of face recognition incorporate the idea of a face recognition unit (FRU), an abstracted representation formed from each experience of a face which aids recognition under novel viewing conditions. Some previous studies have failed to find evidence of this FRU representation. Here, we report three experiments which investigated this theoretical construct by modifying the face learning procedure from that in previous work. During learning, one or two views of previously unfamiliar faces were shown to participants in a serial matching task. Later, participants attempted to recognize both seen and novel views of the learned faces (recognition phase). Experiment 1 tested participants' recognition of a novel view, a day after learning. Experiment 2 was identical, but tested participants on the same day as learning. Experiment 3 repeated Experiment 1, but tested participants on a novel view that was outside the rotation of those views learned. Results revealed a significant advantage, across all experiments, for recognizing a novel view when two views had been learned compared to single view learning. The observed view invariance supports the notion that an FRU representation is established during multi-view face learning under particular learning conditions.

Face perception and learning in autism spectrum disorders

Autism Spectrum Disorder (ASD) is characterized by impairment in social communication and restricted and repetitive interests. While not included in the diagnostic characterization, aspects of face processing and learning have shown disruptions at all stages of development in ASD, although the exact nature and extent of the impairment vary by age and level of functioning of the ASD sample as well as by task demands. In this review, we examine the nature of face attention, perception, and learning in individuals with ASD focusing on three broad age ranges (early development, middle childhood, and adolescence/adulthood). We propose that early delays in basic face processing contribute to the atypical trajectory of social communicative skills in individuals with ASD and contribute to poor social learning throughout development. Face learning is a life-long necessity, as the social world of individual only broadens with age, and thus addressing both the source of the impairment in ASD as well as the trajectory of ability throughout the lifespan, through targeted treatments, may serve to positively impact the lives of individuals who struggle with social information and understanding.

The other-race effect in face learning: Using naturalistic images to investigate face ethnicity effects in a learning paradigm

The other-race effect in face identification has been reported in many situations and by many different ethnicities, yet it remains poorly understood. One reason for this lack of clarity may be a limitation in the methodologies that have been used to test it. Experiments typically use an old-new recognition task to demonstrate the existence of the other-race effect, but such tasks are susceptible to different social and perceptual influences, particularly in terms of the extent to which all faces are equally individuated at study. In this paper we report an experiment in which we used a face learning methodology to measure the other-race effect. We obtained naturalistic photographs of Chinese and Caucasian individuals, which allowed us to test the ability of participants to generalize their learning to new ecologically valid exemplars of a face identity. We show a strong own-race advantage in face learning, such that participants required many fewer trials to learn names of own-race individuals than those of other-race individuals and were better able to identify learned own-race individuals in novel naturalistic stimuli. Since our methodology requires individuation of all faces, and generalization over large image changes, our finding of an other-race effect can be attributed to a specific deficit in the sensitivity of perceptual and memory processes to other-race faces.

Learning faces from variability

Research on face learning has tended to use sets of images that vary systematically on dimensions such as pose and illumination. In contrast, we have proposed that exposure to naturally varying images of a person may be a critical part of the familiarization process. Here, we present two experiments investigating face learning with "ambient images"-relatively unconstrained photos taken from internet searches. Participants learned name and face associations for unfamiliar identities presented in high or low within-person variability-that is, images of the same person returned by internet search on their name (high variability) versus different images of the same person taken from the same event (low variability). In Experiment 1 we show more accurate performance on a speeded name verification task for identities learned in high than in low variability, when the test images are completely novel photos. In Experiment 2 we show more accurate performance on a face matching task for identities previously learned in high than in low variability. The results show that exposure to a large range of within-person variability leads to enhanced learning of new identities.

Event-related potentials in response to cheating and cooperation in a social dilemma game

A sequential prisoner's dilemma game was combined with psychophysiological measures to examine the cognitive underpinnings of reciprocal exchange. Participants played four rounds of the game with partners who either cooperated or cheated. In a control condition, the partners' faces were shown, but no interaction took place. The partners' behaviors were consistent in the first three rounds of the game, but in the last round some of the partners unexpectedly changed strategies. In the first round of the game, the feedback about a partner's decision elicited a feedback P300, which was more pronounced for cooperation and cheating in comparison to the control condition, but did not vary as a function of feedback valence. In the last round, both the feedback negativity and the feedback P300 were sensitive to expectancy violations. There was no consistent evidence for a negativity bias, that is, enhanced allocation of attention to feedback about another person's cheating in comparison to feedback about another person's cooperation. Instead, participants focused on both positive and negative information, and flexibly adjusted their processing biases to the diagnosticity of the information. This conclusion was corroborated by the ERP correlates of memory retrieval. Successful retrieval of a partner's reputation was associated with an anterior positivity between 400 and 600 ms after face onset. This anterior positivity was more pronounced for both cooperator and cheater faces in comparison to control faces. The results suggest that it is not the negativity of social information, but rather its motivational and behavioral relevance that determines its processing.

Telling faces together: Learning new faces through exposure to multiple instances

We are usually able to recognize novel instances of familiar faces with little difficulty, yet recognition of unfamiliar faces can be dramatically impaired by natural within-person variability in appearance. In a card-sorting task for facial identity, different photos of the same unfamiliar face are often seen as different people. Here we report two card-sorting experiments in which we manipulate whether participants know the number of identities present. Without constraints, participants sort faces into many identities. However, when told the number of identities present, they are highly accurate. This minimal contextual information appears to support viewers in "telling faces together". In Experiment 2 we show that exposure to within-person variability in the sorting task improves performance in a subsequent face-matching task. This appears to offer a fast route to learning generalizable representations of new faces.

Implicit learning of geometric eigenfaces

The human visual system can implicitly extract a prototype of encountered visual objects (Posner & Keele, 1968). While learning a prototype provides an efficient way of encoding objects at the category level, discrimination among individual objects requires encoding of variations among them as well. Here we show that in addition to the prototype, human adults also implicitly learn the feature correlations that capture the most significant geometric variations among faces. After studying a group of synthetic faces, observers mistook as seen previously unseen faces representing the first two principal components (eigenfaces, Turk & Pentland, 1991) of the studied faces at significantly higher rates than the correct recognition of the faces actually studied. Implicit learning of the most significant eigenfaces provides an optimal way for encoding variations among faces. The data thus extend the types of summary statistics that can be implicitly extracted by the visual system to include several principal components.