Human and machine recognition of dynamic and static facial expressions: prototypicality, ambiguity, and complexity

A growing body of research suggests that movement aids facial expression recognition. However, less is known about the conditions under which the dynamic advantage occurs. The aim of this research was to test emotion recognition in static and dynamic facial expressions, thereby exploring the role of three featural parameters (prototypicality, ambiguity, and complexity) in human and machine analysis. In two studies, facial expression videos and corresponding images depicting the peak of the target and non-target emotion were presented to human observers and the machine classifier (FACET). Results revealed higher recognition rates for dynamic stimuli compared to non-target images. Such benefit disappeared in the context of target-emotion images which were similarly well (or even better) recognised than videos, and more prototypical, less ambiguous, and more complex in appearance than non-target images. While prototypicality and ambiguity exerted more predictive power in machine performance, complexity was more indicative of human emotion recognition. Interestingly, recognition performance by the machine was found to be superior to humans for both target and non-target images. Together, the findings point towards a compensatory role of dynamic information, particularly when static-based stimuli lack relevant features of the target emotion. Implications for research using automatic facial expression analysis (AFEA) are discussed.

Contextual effects on smile perception and recognition memory

Most past research has focused on the role played by social context information in emotion classification, such as whether a display is perceived as belonging to one emotion category or another. The current study aims to investigate whether the effect of context extends to the interpretation of emotion displays, i.e. smiles that could be judged either as posed or spontaneous readouts of underlying positive emotion. A between-subjects design (N = 93) was used to investigate the perception and recall of posed smiles, presented together with a happy or polite social context scenario. Results showed that smiles seen in a happy context were judged as more spontaneous than the same smiles presented in a polite context. Also, smiles were misremembered as having more of the physical attributes (i.e., Duchenne marker) associated with spontaneous enjoyment when they appeared in the happy than polite context condition. Together, these findings indicate that social context information is routinely encoded during emotion perception, thereby shaping the interpretation and recognition memory of facial expressions.

The Human Face as a Dynamic Tool for Social Communication

As a highly social species, humans frequently exchange social information to support almost all facets of life. One of the richest and most powerful tools in social communication is the face, from which observers can quickly and easily make a number of inferences - about identity, gender, sex, age, race, ethnicity, sexual orientation, physical health, attractiveness, emotional state, personality traits, pain or physical pleasure, deception, and even social status. With the advent of the digital economy, increasing globalization and cultural integration, understanding precisely which face information supports social communication and which produces misunderstanding is central to the evolving needs of modern society (for example, in the design of socially interactive digital avatars and companion robots). Doing so is challenging, however, because the face can be thought of as comprising a high-dimensional, dynamic information space, and this impacts cognitive science and neuroimaging, and their broader applications in the digital economy. New opportunities to address this challenge are arising from the development of new methods and technologies, coupled with the emergence of a modern scientific culture that embraces cross-disciplinary approaches. Here, we briefly review one such approach that combines state-of-the-art computer graphics, psychophysics and vision science, cultural psychology and social cognition, and highlight the main knowledge advances it has generated. In the light of current developments, we provide a vision of the future directions in the field of human facial communication within and across cultures.

Perceptual and affective mechanisms in facial expression recognition: An integrative review

Facial expressions of emotion involve a physical component of morphological changes in a face and an affective component conveying information about the expresser's internal feelings. It remains unresolved how much recognition and discrimination of expressions rely on the perception of morphological patterns or the processing of affective content. This review of research on the role of visual and emotional factors in expression recognition reached three major conclusions. First, behavioral, neurophysiological, and computational measures indicate that basic expressions are reliably recognized and discriminated from one another, albeit the effect may be inflated by the use of prototypical expression stimuli and forced-choice responses. Second, affective content along the dimensions of valence and arousal is extracted early from facial expressions, although this coarse affective representation contributes minimally to categorical recognition of specific expressions. Third, the physical configuration and visual saliency of facial features contribute significantly to expression recognition, with "emotionless" computational models being able to reproduce some of the basic phenomena demonstrated in human observers. We conclude that facial expression recognition, as it has been investigated in conventional laboratory tasks, depends to a greater extent on perceptual than affective information and mechanisms.

Current Emotion Research in Social Psychology: Thinking About Emotions and Other People

This article discusses contemporary social psychological approaches to (a) the social relations and appraisals associated with specific emotions; (b) other people’s impact on appraisal processes; (c) effects of emotion on other people; and (d) interpersonal emotion regulation. We argue that single-minded cognitive perspectives restrict our understanding of interpersonal and group-related emotional processes, and that new methodologies addressing real-time interpersonal and group processes present promising opportunities for future progress.

Judgments of social awkwardness from brief exposure to children with and without high-functioning autism

We form first impressions of many traits based on very short interactions. This study examines whether typical adults judge children with high-functioning autism to be more socially awkward than their typically developing peers based on very brief exposure to still images, audio-visual, video-only, or audio-only information. We used video and audio recordings of children with and without high-functioning autism captured during a story-retelling task. Typically developing adults were presented with 1 s and 3 s clips of these children, as well as still images, and asked to judge whether the person in the clip was socially awkward. Our findings show that participants who are naïve to diagnostic differences between the children in the clips judged children with high-functioning autism to be socially awkward at a significantly higher rate than their typically developing peers. These results remain consistent for exposures as short as 1 s to visual and/or auditory information, as well as for still images. These data suggest that typical adults use subtle nonverbal and non-linguistic cues produced by children with high-functioning autism to form rapid judgments of social awkwardness with the potential for significant repercussions in social interactions.

The MPI emotional body expressions database for narrative scenarios

Emotion expression in human-human interaction takes place via various types of information, including body motion. Research on the perceptual-cognitive mechanisms underlying the processing of natural emotional body language can benefit greatly from datasets of natural emotional body expressions that facilitate stimulus manipulation and analysis. The existing databases have so far focused on few emotion categories which display predominantly prototypical, exaggerated emotion expressions. Moreover, many of these databases consist of video recordings which limit the ability to manipulate and analyse the physical properties of these stimuli. We present a new database consisting of a large set (over 1400) of natural emotional body expressions typical of monologues. To achieve close-to-natural emotional body expressions, amateur actors were narrating coherent stories while their body movements were recorded with motion capture technology. The resulting 3-dimensional motion data recorded at a high frame rate (120 frames per second) provides fine-grained information about body movements and allows the manipulation of movement on a body joint basis. For each expression it gives the positions and orientations in space of 23 body joints for every frame. We report the results of physical motion properties analysis and of an emotion categorisation study. The reactions of observers from the emotion categorisation study are included in the database. Moreover, we recorded the intended emotion expression for each motion sequence from the actor to allow for investigations regarding the link between intended and perceived emotions. The motion sequences along with the accompanying information are made available in a searchable MPI Emotional Body Expression Database. We hope that this database will enable researchers to study expression and perception of naturally occurring emotional body expressions in greater depth.

Children, but not chimpanzees, have facial correlates of determination

Facial expressions have long been proposed to be important agents in forming and maintaining cooperative interactions in social groups. Human beings are inordinately cooperative when compared with their closest-living relatives, the great apes, and hence one might expect species differences in facial expressivity in contexts in which cooperation could be advantageous. Here, human children and chimpanzees were given an identical task designed to induce an element of frustration (it was impossible to solve). In children, but not chimpanzees, facial expressions associated with effort and determination positively correlated with persistence at the task. By contrast, bodily indicators of stress (self-directed behaviour) negatively correlated with task persistence in chimpanzees. Thus, children exhibited more behaviour as they persisted, and chimpanzees exhibited less. The facial expressions produced by children, could, therefore, function to solicit prosocial assistance from others.

Coherence between Emotion and Facial Expression: Evidence from Laboratory Experiments

Evidence on the coherence between emotion and facial expression in adults from laboratory experiments is reviewed. High coherence has been found in several studies between amusement and smiling; low to moderate coherence between other positive emotions and smiling. The available evidence for surprise and disgust suggests that these emotions are accompanied by their “traditional” facial expressions, and even components of these expressions, only in a minority of cases. Evidence concerning sadness, anger, and fear is very limited. For sadness, one study suggests that high emotion–expression coherence may exist in specific situations, whereas for anger and fear, the evidence points to low coherence. Insufficient emotion intensity and inhibition of facial expressions seem unable to account for the observed dissociations between emotion and facial expression.