Looking beyond the static face in emotion recognition: The informative case of interest

Do People Agree on How Positive Emotions Are Expressed? A Survey of Four Emotions and Five Modalities Across 11 Cultures

While much is known about how negative emotions are expressed in different modalities, our understanding of the nonverbal expressions of positive emotions remains limited. In the present research, we draw upon disparate lines of theoretical and empirical work on positive emotions, and systematically examine which channels are thought to be used for expressing four positive emotions: feeling moved, gratitude, interest, and triumph. Employing the intersubjective approach, an established method in cross-cultural psychology, we first explored how the four positive emotions were reported to be expressed in two North American community samples (Studies 1a and 1b: n = 1466). We next confirmed the cross-cultural generalizability of our findings by surveying respondents from ten countries that diverged on cultural values (Study 2: n = 1826). Feeling moved was thought to be signaled with facial expressions, gratitude with the use of words, interest with words, face and voice, and triumph with body posture, vocal cues, facial expressions, and words. These findings provide cross-culturally consistent findings of differential expressions across positive emotions. Notably, positive emotions were thought to be expressed via modalities that go beyond the face.

Perceiving dynamic emotions expressed simultaneously in the face and body minimizes perceptual differences between young and older adults

OBJECTIVES

It is commonly argued that older adults show difficulties in standardized tasks of emotional expression perception, yet most previous works relied on classic sets of static, decontextualized, and stereotypical facial expressions. In real-life, facial expressions are dynamic and embedded in a rich context, two key factors that may aid emotion perception. Specifically, body language provides important affective cues that may disambiguate facial movements.

METHOD

We compared emotion perception of dynamic faces, bodies, and their combination, in a sample of older (age 60-83, n=126) and young (age 18-30, n=124) adults. We used the Geneva Multimodal Emotion Portrayals (GEMEP) set, which includes a full view of expressers' faces and bodies, displaying a diverse range of positive and negative emotions, portrayed dynamically and holistically in a non-stereotypical, unconstrained manner. Critically, we digitally manipulated the dynamic cue such that perceivers viewed isolated faces (without bodies), isolated bodies (without faces), or faces with bodies.

RESULTS

Older adults showed better perception of positive and negative dynamic facial expressions, while young adults showed better perception of positive isolated dynamic bodily expressions. Importantly, emotion perception of faces with bodies was comparable across ages.

DISCUSSION

Dynamic emotion perception in young and older adults may be more similar than previously assumed, especially when the task is more realistic and ecological. Our results emphasize the importance of contextualized and ecological tasks in emotion perception across ages.

Children's perception of emotions in the context of live interactions: Eye movements and emotion judgements

Research examining children's understanding of emotional expressions has generally used static, isolated facial expressions presented in a non-interactive context. However, these tasks do not resemble children's experiences with expressions in daily life, where they must attend to a range of information, including others' facial expressions, movements, and the situation surrounding the expression. In this research, we examine the development of visual attention to another's emotional expressions during a live interaction. Via an eye-tracker, children (4-11 years old) and adults viewed an experimenter open a series of opaque boxes and make an expression (happiness, sadness, fear, or disgust) based on the object inside. Participants determined which of four possible objects (stickers, a broken toy, a spider, or dog poop) was in the box. We examined the proportion of the trial in which participants looked to three areas of the face (the eyes, mouth, and nose area), and the available contextual information (the box held by the experimenter, the four objects). Although children spent less time looking to the face than adults did, their pattern of visual attention within the face and to object AOIs did not differ from that of adults. Finally, for adults, increased accuracy was linked to spending less time looking to the objects whereas increased accuracy for children was not strongly linked to any emotion cue. These data indicate that although children spend less time looking to the face during live interactions than adults do, the proportion of time spent looking to areas of the face and context are generally adult-like.

Surprise: unfolding of facial expressions

Responses to surprising events are dynamic. We argue that initial responses are primarily driven by the unexpectedness of the surprising event and reflect an interrupted and surprised state in which the outcome does not make sense yet. Later responses, after sense-making, are more likely to incorporate the valence of the outcome itself. To identify initial and later responses to surprising stimuli, we conducted two repetition-change studies and coded the general valence of facial expressions using computerised facial coding and specific facial action using the Facial Action Coding System (FACS). Results partly supported our unfolding logic. The computerised coding showed that initial expressions to positive surprises were less positive than later expressions. Moreover, expressions to positive and negative surprises were initially similar, but after some time differentiated depending on the valence of the event. Importantly, these patterns were particularly pronounced in a subset of facially expressive participants, who also showed facial action in the FACS coding. The FACS data showed that the initial phase was characterised by limited facial action, whereas the later increase in positivity seems to be explained by smiling. Conceptual as well as methodological implications are discussed.

Use and Usefulness of Dynamic Face Stimuli for Face Perception Studies-a Review of Behavioral Findings and Methodology

Faces that move contain rich information about facial form, such as facial features and their configuration, alongside the motion of those features. During social interactions, humans constantly decode and integrate these cues. To fully understand human face perception, it is important to investigate what information dynamic faces convey and how the human visual system extracts and processes information from this visual input. However, partly due to the difficulty of designing well-controlled dynamic face stimuli, many face perception studies still rely on static faces as stimuli. Here, we focus on evidence demonstrating the usefulness of dynamic faces as stimuli, and evaluate different types of dynamic face stimuli to study face perception. Studies based on dynamic face stimuli revealed a high sensitivity of the human visual system to natural facial motion and consistently reported dynamic advantages when static face information is insufficient for the task. These findings support the hypothesis that the human perceptual system integrates sensory cues for robust perception. In the present paper, we review the different types of dynamic face stimuli used in these studies, and assess their usefulness for several research questions. Natural videos of faces are ecological stimuli but provide limited control of facial form and motion. Point-light faces allow for good control of facial motion but are highly unnatural. Image-based morphing is a way to achieve control over facial motion while preserving the natural facial form. Synthetic facial animations allow separation of facial form and motion to study aspects such as identity-from-motion. While synthetic faces are less natural than videos of faces, recent advances in photo-realistic rendering may close this gap and provide naturalistic stimuli with full control over facial motion. We believe that many open questions, such as what dynamic advantages exist beyond emotion and identity recognition and which dynamic aspects drive these advantages, can be addressed adequately with different types of stimuli and will improve our understanding of face perception in more ecological settings.

Jumping for Joy: The Importance of the Body and of Dynamics in the Expression and Recognition of Positive Emotions

The majority of research on emotion expression has focused on static facial prototypes of a few selected, mostly negative emotions. Implicitly, most researchers seem to have considered all positive emotions as sharing one common signal (namely, the smile), and consequently as being largely indistinguishable from each other in terms of expression. Recently, a new wave of studies has started to challenge the traditional assumption by considering the role of multiple modalities and the dynamics in the expression and recognition of positive emotions. Based on these recent studies, we suggest that positive emotions are better expressed and correctly perceived when (a) they are communicated simultaneously through the face and body and (b) perceivers have access to dynamic stimuli. Notably, we argue that this improvement is comparatively more important for positive emotions than for negative emotions. Our view is that the misperception of positive emotions has fewer immediate and potentially life-threatening consequences than the misperception of negative emotions; therefore, from an evolutionary perspective, there was only limited benefit in the development of clear, quick signals that allow observers to draw fine distinctions between them. Consequently, we suggest that the successful communication of positive emotions requires a stronger signal than that of negative emotions, and that this signal is provided by the use of the body and the way those movements unfold. We hope our contribution to this growing field provides a new direction and a theoretical grounding for the many lines of empirical research on the expression and recognition of positive emotions.