Children’s representations of facial expression and identity: Identity-contingent expression aftereffects

Face identity and facial expression representations with adaptation paradigms: New directions for potential applications

Adaptation and aftereffect are well-known procedures for exploring our neural representation of visual stimuli. It has been reported that they occur in face identity, facial expressions, and low-level visual features. This method has two primary advantages. One is to reveal the common or shared process of faces, that is, the overlapped or discrete representation of face identities or facial expressions. The other is to investigate the coding system or theory of face processing that underlies the ability to recognize faces. This study aims to organize recent research to guide the reader into the field of face adaptation and its aftereffect and to suggest possible future expansions in the use of this paradigm. To achieve this, we reviewed the behavioral short-term aftereffect studies on face identity (i.e., who it is) and facial expressions (i.e., what expressions such as happiness and anger are expressed), and summarized their findings about the neural representation of faces. First, we summarize the basic characteristics of face aftereffects compared to simple visual features to clarify that facial aftereffects occur at a different stage and are not inherited or combinations of low-level visual features. Next, we introduce the norm-based coding hypothesis, which is one of the theories used to represent face identity and facial expressions, and adaptation is a commonly used procedure to examine this. Subsequently, we reviewed studies that applied this paradigm to immature or impaired face recognition (i.e., children and individuals with autism spectrum disorder or prosopagnosia) and examined the relationships between their poor recognition performance and representations. Moreover, we reviewed studies dealing with the representation of non-presented faces and social signals conveyed via faces and discussed that the face adaptation paradigm is also appropriate for these types of examinations. Finally, we summarize the research conducted to date and propose a new direction for the face adaptation paradigm.

Vocal emotion adaptation aftereffects within and across speaker genders: Roles of timbre and fundamental frequency

While the human perceptual system constantly adapts to the environment, some of the underlying mechanisms are still poorly understood. For instance, although previous research demonstrated perceptual aftereffects in emotional voice adaptation, the contribution of different vocal cues to these effects is unclear. In two experiments, we used parameter-specific morphing of adaptor voices to investigate the relative roles of fundamental frequency (F0) and timbre in vocal emotion adaptation, using angry and fearful utterances. Participants adapted to voices containing emotion-specific information in either F0 or timbre, with all other parameters kept constant at an intermediate 50% morph level. Full emotional voices and ambiguous voices were used as reference conditions. All adaptor stimuli were either of the same (Experiment 1) or opposite speaker gender (Experiment 2) of subsequently presented target voices. In Experiment 1, we found consistent aftereffects in all adaptation conditions. Crucially, aftereffects following timbre adaptation were much larger than following F0 adaptation and were only marginally smaller than those following full adaptation. In Experiment 2, adaptation aftereffects appeared massively and proportionally reduced, with differences between morph types being no longer significant. These results suggest that timbre plays a larger role than F0 in vocal emotion adaptation, and that vocal emotion adaptation is compromised by eliminating gender-correspondence between adaptor and target stimuli. Our findings also add to mounting evidence suggesting a major role of timbre in auditory adaptation.

The Role of Racial and Developmental Experience on Emotional Adaptive Coding in Autism Spectrum Disorder

Sensitivity to emotional face aids in rapid detection and evaluation of others, such that by school-age, children and youth exhibit adult-like patterns when the prolonged viewing of an emotional face distorts the perception of a subsequent face. However, the developmental considerations of this phenomenon (known as emotional adaptive coding) are unclear given ongoing maturational and experiential changes, including the influence of own-race experiences or the lack of face expertise, as is evident in autism spectrum disorder (ASD). This study addressed whether emotional adaptive coding is sensitive to factors of face perception expertise, specifically self-race and developmental experience, in adults (age 19-28 years) and youth (age 10-16 years). Emotional adaptive coding was not influenced by race expertise (i.e., other versus same race identity) in White and Asian adults. Emotional adaptation coding during childhood and adolescence is consistent with adults, though youth with ASD exhibited stronger adaptor after-effects in response to other-race faces, relative to TD youth and adults. By extending prior work to examine the integration of race and emotional adaptive coding in ASD, we discovered that the strength of response in ASD is atypical when viewing other-race faces, which clarifies the role of racial and facial experience on emotional face adaption.

The Change of Expression Configuration Affects Identity-Dependent Expression Aftereffect but Not Identity-Independent Expression Aftereffect

The present study examined the influence of expression configuration on cross-identity expression aftereffect. The expression configuration refers to the spatial arrangement of facial features in a face for conveying an emotion, e.g., an open-mouth smile vs. a closed-mouth smile. In the first of two experiments, the expression aftereffect is measured using a cross-identity/cross-expression configuration factorial design. The facial identities of test faces were the same or different from the adaptor, while orthogonally, the expression configurations of those facial identities were also the same or different. The results show that the change of expression configuration impaired the expression aftereffect when the facial identities of adaptor and tests were the same; however, the impairment effect disappears when facial identities were different, indicating the identity-independent expression representation is more robust to the change of the expression configuration in comparison with the identity-dependent expression representation. In the second experiment, we used schematic line faces as adaptors and real faces as tests to minimize the similarity between the adaptor and tests, which is expected to exclude the contribution from the identity-dependent expression representation to expression aftereffect. The second experiment yields a similar result as the identity-independent expression aftereffect observed in Experiment 1. The findings indicate the different neural sensitivities to expression configuration for identity-dependent and identity-independent expression systems.

Interactions between the perception of age and ethnicity in faces: an event-related potential study

Face perception models propose that different facial attributes are processed by anatomically distinct neural pathways that partially overlap. Whether these attributes interact functionally is an open question. Our goal was to determine if there are interactions between age and ethnicity processing and, if so, at what temporal epoch these interactions are evident. We monitored event-related potentials on electroencephalography while subjects categorized faces by age or ethnicity in two conditions: a baseline in which the other of these two properties not being categorized was held constant and an interference condition in which it also varied, as modelled after the Garner interference paradigm. We found that, when participants were categorizing faces by age, variations in ethnicity increased the amplitude of the right face-selective N170 component. When subjects were categorizing faces by ethnicity, variations in age did not alter the N170. We concluded that there is an asymmetric pattern of influence between age and ethnicity on early face-specific stages of visual processing, which has parallels with behavioural evidence of asymmetric interactions between identity and expression processing of faces.

Independence of face identity and expression processing: exploring the role of motion

According to the classic Bruce and Young (1986) model of face recognition, identity and emotional expression information from the face are processed in parallel and independently. Since this functional model was published, a growing body of research has challenged this viewpoint and instead support an interdependence view. In addition, neural models of face processing emphasize differences in terms of the processing of changeable and invariant aspects of faces. This article provides a critical appraisal of this literature and discusses the role of motion in both expression and identity recognition and the intertwined nature of identity, expression and motion processing. We conclude by discussing recent advancements in this area and research questions that still need to be addressed.

Perceiving vocal age and gender: an adaptation approach

Aftereffects of adaptation have revealed both independent and interactive coding of facial signals including identity and expression or gender and age. By contrast, interactive processing of non-linguistic features in voices has rarely been investigated. Here we studied bidirectional cross-categorical aftereffects of adaptation to vocal age and gender. Prolonged exposure to young (~20yrs) or old (~70yrs) male or female voices biased perception of subsequent test voices away from the adapting age (Exp. 1) and the adapting gender (Exp. 2). Relative to gender-congruent adaptor-test pairings, vocal age aftereffects (VAAEs) were reduced but remained significant when voice gender changed between adaptation and test. This suggests that the VAAE relies on both gender-specific and gender-independent age representations for male and female voices. By contrast, voice gender aftereffects (VGAEs) were not modulated by age-congruency of adaptor and test voices (Exp. 2). Instead, young voice adaptors generally induced larger VGAEs than old voice adaptors. This suggests that young voices are particularly efficient gender adaptors, likely reflecting more pronounced sexual dimorphism in these voices. In sum, our findings demonstrate how high-level processing of vocal age and gender is partially intertwined.

Psychophysical measures of sensitivity to facial expression of emotion

We report the development of two simple, objective, psychophysical measures of the ability to discriminate facial expressions of emotion that vary in intensity from a neutral facial expression and to discriminate between varying intensities of emotional facial expression. The stimuli were created by morphing photographs of models expressing four basic emotions, anger, disgust, happiness, and sadness with neutral expressions. Psychometric functions were obtained for 15 healthy young adults using the Method of Constant Stimuli with a two-interval forced-choice procedure. Individual data points were fitted by Quick functions for each task and each emotion, allowing estimates of absolute thresholds and slopes. The tasks give objective and sensitive measures of the basic perceptual abilities required for perceiving and interpreting emotional facial expressions.

Withholding response in the face of a smile: age-related differences in prefrontal sensitivity to Nogo cues following happy and angry faces

The modulation of control processes by stimulus salience, as well as associated neural activation, changes over development. We investigated age-related differences in the influence of facial emotion on brain activation when an action had to be withheld, focusing on a developmental period characterized by rapid social-emotional and cognitive change. Groups of kindergarten and young school-aged children and a group of young adults performed a modified Go/Nogo task. Response cues were preceded by happy or angry faces. After controlling for task performance, left orbitofrontal regions discriminated trials with happy vs. angry faces in children but not in adults when a response was withheld, and this effect decreased parametrically with age group. Age-related changes in prefrontal responsiveness to facial expression were not observed when an action was required, nor did this region show age-related activation changes with the demand to withhold a response in general. Such results reveal age-related differences in prefrontal activation that are specific to stimulus valence and depend on the action required.

The role of skin texture and facial shape in representations of age and identity

Faces have both shape and skin texture, but the relative importance of the two in face representations is unclear. Our goals were first, to determine the contribution of shape versus texture to aftereffects for facial age and identity and second, to assess whether adaptation transferred between shape and texture, suggesting integration in a single representation. In our first experiment we examined age aftereffects. We obtained young and old images of two celebrities and created hybrid images, one combining the structure of the old face with the skin texture of the young face, the other combining the young structure with the old skin texture. This allowed us to create adaptation contrasts where the two adapting faces had the same facial structure but different skin texture, and vice versa. In the second experiment, we performed a similar study but this time examining identity aftereffects between two people of a similar age. We found that both skin texture and facial shape generated significant age aftereffects, but the contribution was greater from texture than from shape. Both texture and shape also generated significant identity aftereffects, but the contribution was greater from shape than from texture. In the last experiment, we used the normal and hybrid images to determine if adaptation to one property (i.e., texture) could create aftereffects in the perception of age in the other property (i.e., shape). While there was significant within-component adaptation for texture and shape, there was no evidence of cross-component adaptation. We conclude that shape and texture contribute differently to different face representations, with texture dominating for age. The lack of cross-component adaptation transfer suggests independent encoding of shape and texture, at least for age representations.

Adaptive face space coding in congenital prosopagnosia: typical figural aftereffects but abnormal identity aftereffects

People with congenital prosopagnosia (CP) report difficulty recognising faces in everyday life and perform poorly on face recognition tests. Here, we investigate whether impaired adaptive face space coding might contribute to poor face recognition in CP. To pinpoint how adaptation may affect face processing, a group of CPs and matched controls completed two complementary face adaptation tasks: the figural aftereffect, which reflects adaptation to general distortions of shape, and the identity aftereffect, which directly taps the mechanisms involved in the discrimination of different face identities. CPs displayed a typical figural aftereffect, consistent with evidence that they are able to process some shape-based information from faces, e.g., cues to discriminate sex. CPs also demonstrated a significant identity aftereffect. However, unlike controls, CPs impression of the identity of the neutral average face was not significantly shifted by adaptation, suggesting that adaptive coding of identity is abnormal in CP. In sum, CPs show reduced aftereffects but only when the task directly taps the use of face norms used to code individual identity. This finding of a reduced face identity aftereffect in individuals with severe face recognition problems is consistent with suggestions that adaptive coding may have a functional role in face recognition.

Cross-emotion facial expression aftereffects

Increasing evidence suggests that the visual representations of different emotional facial expressions overlap. Here we used an adaptation paradigm to investigate overlap of anger, disgust and fear expressions. In Experiment 1, participants categorized faces morphed from neutral to anger or neutral to disgust after adaptation to expressions of anger, disgust, and fear. Adaptation to expressions of both anger and disgust was found to bias perception of anger expressions away from anger. For disgust expressions, adaptation to disgust biased perception away from disgust, whereas fear adaptation biased perception towards disgust. Adaptation to anger had no measurable effect. In Experiment 2, covering the mouth-region of the disgust adaptation face was found to severely diminish the effect of disgust adaptation on perception of anger targets whereas covering the nose- or eye-region had no effect. In Experiment 3, adaptation to anger had a substantial effect on perception of anger targets when the mouth-region of the anger face was covered; indicating that the results of Experiment 2 are not an artefact of the stimuli and procedures used. These results indicate that the visual representations of anger, disgust and fear expressions overlap to a considerable degree. Furthermore, the nature of this overlap appears related to the communicative functions of these expressions.

The changing face of emotion: age-related patterns of amygdala activation to salient faces

The present study investigated age-related differences in the amygdala and other nodes of face-processing networks in response to facial expression and familiarity. fMRI data were analyzed from 31 children (3.5-8.5 years) and 14 young adults (18-33 years) who viewed pictures of familiar (mothers) and unfamiliar emotional faces. Results showed that amygdala activation for faces over a scrambled image baseline increased with age. Children, but not adults, showed greater amygdala activation to happy than angry faces; in addition, amygdala activation for angry faces increased with age. In keeping with growing evidence of a positivity bias in young children, our data suggest that children find happy faces to be more salient or meaningful than angry faces. Both children and adults showed preferential activation to mothers' over strangers' faces in a region of rostral anterior cingulate cortex associated with self-evaluation, suggesting that some nodes in frontal evaluative networks are active early in development. This study presents novel data on neural correlates of face processing in childhood and indicates that preferential amygdala activation for emotional expressions changes with age.