Face identity and facial expression representations with adaptation paradigms: New directions for potential applications

Adaptation and aftereffect are well-known procedures for exploring our neural representation of visual stimuli. It has been reported that they occur in face identity, facial expressions, and low-level visual features. This method has two primary advantages. One is to reveal the common or shared process of faces, that is, the overlapped or discrete representation of face identities or facial expressions. The other is to investigate the coding system or theory of face processing that underlies the ability to recognize faces. This study aims to organize recent research to guide the reader into the field of face adaptation and its aftereffect and to suggest possible future expansions in the use of this paradigm. To achieve this, we reviewed the behavioral short-term aftereffect studies on face identity (i.e., who it is) and facial expressions (i.e., what expressions such as happiness and anger are expressed), and summarized their findings about the neural representation of faces. First, we summarize the basic characteristics of face aftereffects compared to simple visual features to clarify that facial aftereffects occur at a different stage and are not inherited or combinations of low-level visual features. Next, we introduce the norm-based coding hypothesis, which is one of the theories used to represent face identity and facial expressions, and adaptation is a commonly used procedure to examine this. Subsequently, we reviewed studies that applied this paradigm to immature or impaired face recognition (i.e., children and individuals with autism spectrum disorder or prosopagnosia) and examined the relationships between their poor recognition performance and representations. Moreover, we reviewed studies dealing with the representation of non-presented faces and social signals conveyed via faces and discussed that the face adaptation paradigm is also appropriate for these types of examinations. Finally, we summarize the research conducted to date and propose a new direction for the face adaptation paradigm.

Self-tuition as an essential design feature of the brain

We are curious by nature, particularly when young. Evolution has endowed our brain with an inbuilt obligation to educate itself. In this perspectives article, we posit that self-tuition is an evolved principle of vertebrate brain design that is reflected in its basic architecture and critical for its normal development. Self-tuition involves coordination between functionally distinct components of the brain, with one set of areas motivating exploration that leads to the experiences that train another set. We review key hypothalamic and telencephalic structures involved in this interplay, including their anatomical connections and placement within the segmental architecture of conserved forebrain circuits. We discuss the nature of educative behaviours motivated by the hypothalamus, innate stimulus biases, the relationship to survival in early life, and mechanisms by which telencephalic areas gradually accumulate knowledge. We argue that this aspect of brain function is of paramount importance for systems neuroscience, as it confers neural specialization and allows animals to attain far more sophisticated behaviours than would be possible through genetic mechanisms alone. Self-tuition is of particular importance in humans and other primates, whose large brains and complex social cognition rely critically on experience-based learning during a protracted childhood period.

This article is part of the theme issue ‘Systems neuroscience through the lens of evolutionary theory’.

Personalized quantification of facial normality: a machine learning approach

What is a normal face? A fundamental task for the facial reconstructive surgeon is to answer that question as it pertains to any given individual. Accordingly, it would be important to be able to place the facial appearance of a patient with congenital or acquired deformity numerically along their own continuum of normality, and to measure any surgical changes against such a personalized benchmark. This has not previously been possible. We have solved this problem by designing a computerized model that produces realistic, normalized versions of any given facial image, and objectively measures the perceptual distance between the raw and normalized facial image pair. The model is able to faithfully predict human scoring of facial normality. We believe this work represents a paradigm shift in the assessment of the human face, holding great promise for development as an objective tool for surgical planning, patient education, and as a means for clinical outcome measurement.

Distraction biases working memory for faces

Working memory persists in the face of distraction, yet not without consequence. Previous research has shown that memory for low-level visual features is systematically influenced by the maintenance or presentation of a similar distractor stimulus. Responses are frequently biased in stimulus space towards a perceptual distractor, though this has yet to be determined for high-level stimuli. We investigated whether these influences are shared for complex visual stimuli such as faces. To quantify response accuracies for these stimuli, we used a delayed-estimation task with a computer-generated "face space" consisting of 80 faces that varied continuously as a function of age and sex. In a set of three experiments, we found that responses for a target face held in working memory were biased towards a distractor face presented during the maintenance period. The amount of response bias did not vary as a function of distance between target and distractor. Our data suggest that, similar to low-level visual features, high-level face representations in working memory are biased by the processing of related but task-irrelevant information.

Understanding individual differences in theory of mind via representation of minds, not mental states

The human ability to make inferences about the minds of conspecifics is remarkable. The majority of work in this area focuses on mental state representation ('theory of mind'), but has had limited success in explaining individual differences in this ability, and is characterized by the lack of a theoretical framework that can account for the effect of variability in the population of minds to which individuals are exposed. We draw analogies between faces and minds as complex social stimuli, and suggest that theoretical and empirical progress on understanding the mechanisms underlying mind representation can be achieved by adopting a 'Mind-space' framework; that minds, like faces, are represented within a multidimensional psychological space. This Mind-space framework can accommodate the representation of whole cognitive systems, and may help to explain individual differences in the consistency and accuracy with which the mental states of others are inferred. Mind-space may also have relevance for understanding human development, intergroup relations, and the atypical social cognition seen in several clinical conditions.

Color Afterimages in Autistic Adults

It has been suggested that attenuated adaptation to visual stimuli in autism is the result of atypical perceptual priors (e.g., Pellicano and Burr in Trends Cogn Sci 16(10):504-510, 2012. doi: 10.1016/j.tics.2012.08.009 ). This study investigated adaptation to color in autistic adults, measuring both strength of afterimage and the influence of top-down knowledge. We found no difference in color afterimage strength between autistic and typical adults. Effects of top-down knowledge on afterimage intensity shown by Lupyan (Acta Psychol 161:117-130, 2015. doi: 10.1016/j.actpsy.2015.08.006 ) were not replicated for either group. This study finds intact color adaptation in autistic adults. This is in contrast to findings of attenuated adaptation to faces and numerosity in autistic children. Future research should investigate the possibility of developmental differences in adaptation and further examine top-down effects on adaptation.

Restricted attention to social cues in schizophrenia patients

Deficits of psychosocial functioning are a robust finding in schizophrenia. Research on social cognition may open a new avenue for the development of effective interventions. As a correlate of social perceptive information processing deficits, schizophrenia patients (SZP) show deviant gaze behavior (GB) while viewing emotional faces. As understanding of a social environment requires gathering complex social information, our study aimed at investigating the gaze behavior of SZP related to social interactions and its impact on the level of social and role functioning. GB of 32 SZP and 37 healthy control individuals (HCI) was investigated with a high-resolution eye tracker during an unguided viewing of 12 complex pictures of social interaction scenes. Regarding whole pictures, SZP showed a shorter scanpath length, fewer fixations and a shorter mean distance between fixations. Furthermore, SZP exhibited fewer and shorter fixations on faces, but not on the socially informative bodies nor on the background, suggesting a cue-specific abnormality. Logistic regression with bootstrapping yielded a model including two GB parameters; a subsequent ROC curve analysis indicated an excellent ability of group discrimination (AUC .85). Face-related GB aberrations correlated with lower social and role functioning and with delusional thinking, but not with negative symptoms. Training of spontaneous integration of face-related social information seems promising to enable a holistic perception of social information, which may in turn improve social and role functioning. The observed ability to discriminate SZP from HCI warrants further research on the predictive validity of GB in psychosis risk prediction.

Face-Space: A Unifying Concept in Face Recognition Research

The concept of a multidimensional psychological space, in which faces can be represented according to their perceived properties, is fundamental to the modern theorist in face processing. Yet the idea was not clearly expressed until 1991. The background that led to the development of face-space is explained, and its continuing influence on theories of face processing is discussed. Research that has explored the properties of the face-space and sought to understand caricature, including facial adaptation paradigms, is reviewed. Face-space as a theoretical framework for understanding the effect of ethnicity and the development of face recognition is evaluated. Finally, two applications of face-space in the forensic setting are discussed. From initially being presented as a model to explain distinctiveness, inversion, and the effect of ethnicity, face-space has become a central pillar in many aspects of face processing. It is currently being developed to help us understand adaptation effects with faces. While being in principle a simple concept, face-space has shaped, and continues to shape, our understanding of face perception.

Face processing in the brains of pre-school aged children measured with MEG

There are two competing theories concerning the development of face perception: a late maturation account and an early maturation account. Magnetoencephalography (MEG) neuroimaging holds promise for adjudicating between the two opposing accounts by providing objective neurophysiological measures of face processing, with sufficient temporal resolution to isolate face-specific brain responses from those associated with other sensory, cognitive and motor processes. The current study used a customized child MEG system to measure M100 and M170 brain responses in 15 children aged three to six years while they viewed faces, cars and their phase-scrambled counterparts. Compared to adults tested using the same stimuli in a conventional MEG system, children showed significantly larger and later M100 responses. Children's M170 responses, derived by subtracting the responses to phase-scrambled images from the corresponding images (faces or cars) were delayed in latency but otherwise resembled the adult M170. This component has not been obtained in previous studies of young children tested using conventional adult MEG systems. However children did show a markedly reduced M170 response to cars in comparison to adults. This may reflect children's lack of expertise with cars relative to faces. Taken together, these data are in accord with recent behavioural and neuroimaging data that support early maturation of the basic face processing functions.

Category-specific face prototypes are emerging, but not yet mature, in 5-year-old children

Adults' expertise in face recognition has been attributed to norm-based coding. Moreover, adults possess separable norms for a variety of face categories (e.g., race, sex, age) that appear to enhance recognition by reducing redundancy in the information shared by faces and ensuring that only relevant dimensions are used to encode faces from a given category. Although 5-year-old children process own-race faces using norm-based coding, little is known about the organization and refinement of their face space. The current study investigated whether 5-year-olds rely on category-specific norms and whether experience facilitates the development of dissociable face prototypes. In Experiment 1, we examined whether Chinese 5-year-olds show race-contingent opposing aftereffects and the extent to which aftereffects transfer across face race among Caucasian and Chinese 5-year-olds. Both participant races showed partial transfer of aftereffects across face race; however, there was no evidence for race-contingent opposing aftereffects. To examine whether experience facilitates the development of category-specific prototypes, we investigated whether race-contingent aftereffects are present among Caucasian 5-year-olds with abundant exposure to Chinese faces (Experiment 2) and then tested separate groups of 5-year-olds with two other categories with which they have considerable experience: sex (male/female faces) and age (adult/child faces) (Experiment 3). Across all three categories, 5-year-olds showed no category-contingent opposing aftereffects. These results demonstrate that 5 years of age is a stage characterized by minimal separation in the norms and associated coding dimensions used for faces from different categories and suggest that refinement of the mechanisms that underlie expert face processing occurs throughout childhood.

The left perceptual bias for adult and infant faces in adults and 5-year-old children: face age matters

A large number of studies have shown that adults rely more heavily on information conveyed by the left side of the face in judging emotional state, gender and identity. This phenomenon, called left perceptual bias (LPB), suggests a right hemisphere lateralization of face processing mechanisms. Although specialization of neural mechanisms for processing over-experienced face categories begins during the first year of life, little is known about the developmental trajectory of the LPB and whether or when the bias becomes selective for specific face categories as a result of experience. To address these questions we tested adults (Experiment 1) and 5-year-old children (Experiment 2) with null or limited experience with infants in an identity matching-to-sample task with chimeric adult and infant faces, for which both adults and children have been shown to manifest differential processing abilities. Results showed that 5-year-olds manifest a leftward bias selective for adult faces, and the magnitude of the bias is larger for adult compared to infant faces in adults. This evidence is in line with earlier demonstrations of a perceptual processing advantage for adult faces in adults and children and points to the role of experience in shaping neurocognitive specialization for face processing.

Not just the norm: exemplar-based models also predict face aftereffects

The face recognition literature has considered two competing accounts of how faces are represented within the visual system: Exemplar-based models assume that faces are represented via their similarity to exemplars of previously experienced faces, while norm-based models assume that faces are represented with respect to their deviation from an average face, or norm. Face identity aftereffects have been taken as compelling evidence in favor of a norm-based account over an exemplar-based account. After a relatively brief period of adaptation to an adaptor face, the perceived identity of a test face is shifted toward a face with attributes opposite to those of the adaptor, suggesting an explicit psychological representation of the norm. Surprisingly, despite near universal recognition that face identity aftereffects imply norm-based coding, there have been no published attempts to simulate the predictions of norm- and exemplar-based models in face adaptation paradigms. Here, we implemented and tested variations of norm and exemplar models. Contrary to common claims, our simulations revealed that both an exemplar-based model and a version of a two-pool norm-based model, but not a traditional norm-based model, predict face identity aftereffects following face adaptation.

Reduced face aftereffects in autism are not due to poor attention

This study aimed to determine why face identity aftereffects are diminished in children with autism, relative to typical children. To address the possibility that reduced face aftereffects might reflect reduced attention to adapting stimuli, we investigated the consequence of controlling attention to adapting faces during a face identity aftereffect task in children with autism and typical children. We also included a size-change between adaptation and test stimuli to determine whether the reduced aftereffects reflect atypical adaptation to low- or higher-level stimulus properties. Results indicated that when attention was controlled and directed towards adapting stimuli, face identity aftereffects in children with autism were significantly reduced relative to typical children. This finding challenges the notion that atypicalities in the quality and/or quantity of children's attention during adaptation might account for group differences previously observed in this paradigm. Additionally, evidence of diminished face identity aftereffects despite a stimulus size change supports an adaptive processing atypicality in autism that extends beyond low-level, retinotopically coded stimulus properties. These findings support the notion that diminished face aftereffects in autism reflect atypicalities in adaptive norm-based coding, which could also contribute to face processing difficulties in this group.

Perceiving vocal age and gender: an adaptation approach

Aftereffects of adaptation have revealed both independent and interactive coding of facial signals including identity and expression or gender and age. By contrast, interactive processing of non-linguistic features in voices has rarely been investigated. Here we studied bidirectional cross-categorical aftereffects of adaptation to vocal age and gender. Prolonged exposure to young (~20yrs) or old (~70yrs) male or female voices biased perception of subsequent test voices away from the adapting age (Exp. 1) and the adapting gender (Exp. 2). Relative to gender-congruent adaptor-test pairings, vocal age aftereffects (VAAEs) were reduced but remained significant when voice gender changed between adaptation and test. This suggests that the VAAE relies on both gender-specific and gender-independent age representations for male and female voices. By contrast, voice gender aftereffects (VGAEs) were not modulated by age-congruency of adaptor and test voices (Exp. 2). Instead, young voice adaptors generally induced larger VGAEs than old voice adaptors. This suggests that young voices are particularly efficient gender adaptors, likely reflecting more pronounced sexual dimorphism in these voices. In sum, our findings demonstrate how high-level processing of vocal age and gender is partially intertwined.

Face adaptation effects: reviewing the impact of adapting information, time, and transfer

The ability to adapt is essential to live and survive in an ever-changing environment such as the human ecosystem. Here we review the literature on adaptation effects of face stimuli to give an overview of existing findings in this area, highlight gaps in its research literature, initiate new directions in face adaptation research, and help to design future adaptation studies. Furthermore, this review should lead to better understanding of the processing characteristics as well as the mental representations of face-relevant information. The review systematizes studies at a behavioral level in respect of a framework which includes three dimensions representing the major characteristics of studies in this field of research. These dimensions comprise (1) the specificity of adapting face information, e.g., identity, gender, or age aspects of the material to be adapted to (2) aspects of timing (e.g., the sustainability of adaptation effects) and (3) transfer relations between face images presented during adaptation and adaptation tests (e.g., images of the same or different identities). The review concludes with options for how to combine findings across different dimensions to demonstrate the relevance of our framework for future studies.