Being facially expressive is socially advantageous

Individuals vary in how they move their faces in everyday social interactions. In a first large-scale study, we measured variation in dynamic facial behaviour during social interaction and examined dyadic outcomes and impression formation. In Study 1, we recorded semi-structured video calls with 52 participants interacting with a confederate across various everyday contexts. Video clips were rated by 176 independent participants. In Study 2, we examined video calls of 1315 participants engaging in unstructured video-call interactions. Facial expressivity indices were extracted using automated Facial Action Coding Scheme analysis and measures of personality and partner impressions were obtained by self-report. Facial expressivity varied considerably across participants, but little across contexts, social partners or time. In Study 1, more facially expressive participants were more well-liked, agreeable, and successful at negotiating (if also more agreeable). Participants who were more facially competent, readable, and perceived as readable were also more well-liked. In Study 2, we replicated the findings that facial expressivity was associated with agreeableness and liking by their social partner, and additionally found it to be associated with extraversion and neuroticism. Findings suggest that facial behaviour is a stable individual difference that proffers social advantages, pointing towards an affiliative, adaptive function.

The convergence of naturalistic paradigms and cognitive neuroscience methods to investigate memory and its development

Studies that involve lab-based stimuli (e.g., words, pictures) are fundamental in the memory literature. At the same time, there is growing acknowledgment that memory processes assessed in the lab may not be analogous to how memory operates in the real world. Naturalistic paradigms can bridge this gap and over the decades a growing proportion of memory research has involved more naturalistic events. However, there is significant variation in the types of naturalistic studies used to study memory and its development, each with various advantages and limitations. Further, there are notable gaps in how often different types of naturalistic approaches have been combined with cognitive neuroscience methods (e.g., fMRI, EEG) to elucidate the neural processes and substrates involved in memory encoding and retrieval in the real world. Here we summarize and discuss what we identify as progressively more naturalistic methodologies used in the memory literature (movie, virtual reality, staged-events inside and outside of the lab, photo-taking, and naturally occurring event studies). Our goal is to describe each approach's benefits (e.g., naturalistic quality, feasibility), limitations (e.g., viability of neuroimaging method for event encoding versus event retrieval), and discuss possible future directions with each approach. We focus on child studies, when available, but also highlight past adult studies. Although there is a growing body of child memory research, naturalistic approaches combined with cognitive neuroscience methodologies in this domain remain sparse. Overall, this viewpoint article reviews how we can study memory through the lens of developmental cognitive neuroscience, while utilizing naturalistic and real-world events.

Reduced stereotypicality and spared use of facial expression predictions for social evaluation in autism

Background/Objective

Autism has been investigated through traditional emotion recognition paradigms, merely investigating accuracy, thereby constraining how potential differences across autistic and control individuals may be observed, identified, and described. Moreover, the use of emotional facial expression information for social functioning in autism is of relevance to provide a deeper understanding of the condition.

Method

Adult autistic individuals (n = 34) and adult control individuals (n = 34) were assessed with a social perception behavioral paradigm exploring facial expression predictions and their impact on social evaluation.

Results

Autistic individuals held less stereotypical predictions than controls. Importantly, despite such differences in predictions, the use of such predictions for social evaluation did not differ significantly between groups, as autistic individuals relied on their predictions to evaluate others to the same extent as controls.

Conclusions

These results help to understand how autistic individuals perceive social stimuli and evaluate others, revealing a deviation from stereotypicality beyond which social evaluation strategies may be intact.

An investigation of the modulatory effects of empathic and autistic traits on emotional and facial motor responses during live social interactions

A close relationship between emotional contagion and spontaneous facial mimicry has been theoretically proposed and is supported by empirical data. Facial expressions are essential in terms of both emotional and motor synchrony. Previous studies have demonstrated that trait emotional empathy enhanced spontaneous facial mimicry, but the relationship between autistic traits and spontaneous mimicry remained controversial. Moreover, previous studies presented faces that were static or videotaped, which may lack the "liveliness" of real-life social interactions. We addressed this limitation by using an image relay system to present live performances and pre-recorded videos of smiling or frowning dynamic facial expressions to 94 healthy female participants. We assessed their subjective experiential valence and arousal ratings to infer the amplitude of emotional contagion. We measured the electromyographic activities of the zygomaticus major and corrugator supercilii muscles to estimate spontaneous facial mimicry. Individual differences measures included trait emotional empathy (empathic concern) and the autism-spectrum quotient. We did not find that live performances enhanced the modulatory effect of trait differences on emotional contagion or spontaneous facial mimicry. However, we found that a high trait empathic concern was associated with stronger emotional contagion and corrugator mimicry. We found no two-way interaction between the autism spectrum quotient and emotional condition, suggesting that autistic traits did not modulate emotional contagion or spontaneous facial mimicry. Our findings imply that previous findings regarding the relationship between emotional empathy and emotional contagion/spontaneous facial mimicry using videos and photos could be generalized to real-life interactions.

Electromyographic Validation of Spontaneous Facial Mimicry Detection Using Automated Facial Action Coding

Although electromyography (EMG) remains the standard, researchers have begun using automated facial action coding system (FACS) software to evaluate spontaneous facial mimicry despite the lack of evidence of its validity. Using the facial EMG of the zygomaticus major (ZM) as a standard, we confirmed the detection of spontaneous facial mimicry in action unit 12 (AU12, lip corner puller) via an automated FACS. Participants were alternately presented with real-time model performance and prerecorded videos of dynamic facial expressions, while simultaneous ZM signal and frontal facial videos were acquired. Facial videos were estimated for AU12 using FaceReader, Py-Feat, and OpenFace. The automated FACS is less sensitive and less accurate than facial EMG, but AU12 mimicking responses were significantly correlated with ZM responses. All three software programs detected enhanced facial mimicry by live performances. The AU12 time series showed a roughly 100 to 300 ms latency relative to the ZM. Our results suggested that while the automated FACS could not replace facial EMG in mimicry detection, it could serve a purpose for large effect sizes. Researchers should be cautious with the automated FACS outputs, especially when studying clinical populations. In addition, developers should consider the EMG validation of AU estimation as a benchmark.

Arousal modulates the amygdala-insula reciprocal connectivity during naturalistic emotional movie watching

Emotional arousal is a complex state recruiting distributed cortical and subcortical structures, in which the amygdala and insula play an important role. Although previous neuroimaging studies have showed that the amygdala and insula manifest reciprocal connectivity, the effective connectivities and modulatory patterns on the amygdala-insula interactions underpinning arousal are still largely unknown. One of the reasons may be attributed to static and discrete laboratory brain imaging paradigms used in most existing studies. In this study, by integrating naturalistic-paradigm (i.e., movie watching) functional magnetic resonance imaging (fMRI) with a computational affective model that predicts dynamic arousal for the movie stimuli, we investigated the effective amygdala-insula interactions and the modulatory effect of the input arousal on the effective connections. Specifically, the predicted dynamic arousal of the movie served as regressors in general linear model (GLM) analysis and brain activations were identified accordingly. The regions of interest (i.e., the bilateral amygdala and insula) were localized according to the GLM activation map. The effective connectivity and modulatory effect were then inferred by using dynamic causal modeling (DCM). Our experimental results demonstrated that amygdala was the site of driving arousal input and arousal had a modulatory effect on the reciprocal connections between amygdala and insula. Our study provides novel evidence to the underlying neural mechanisms of arousal in a dynamical naturalistic setting.

Facial affect recognition in context in adults with and without TBI

Introduction

Several studies have reported impaired emotion recognition in adults with traumatic brain injury (TBI), but studies have two major design features that limit application of results to real-world contexts: (1) participants choose from among lists of basic emotions, rather than generating emotion labels, and (2) images are typically presented in isolation rather than in context. To address these limitations, we created an open-labeling task with faces shown alone or in real-life scenes, to more closely approximate how adults with TBI label facial emotions beyond the lab.

Methods

Participants were 55 adults (29 female) with moderate to severe TBI and 55 uninjured comparison peers, individually matched for race, sex, and age. Participants viewed 60 photographs of faces, either alone or in the pictured person's real-life context, and were asked what that person was feeling. We calculated the percent of responses that were standard forced-choice-task options, and also used sentiment intensity analysis to compare verbal responses between the two groups. We tracked eye movements for a subset of participants, to explore whether gaze duration or number of fixations helped explain any group differences in labels.

Results

Over 50% of responses in both groups were words other than basic emotions on standard affect tasks, highlighting the importance of eliciting open-ended responses. Valence of labels by participants with TBI was attenuated relative to valence of Comparison group labels, i.e., TBI group responses were less positive to positive images and the same was true for negative images, although the TBI group responses had higher lexical diversity. There were no significant differences in gaze duration or number of fixations between groups.

Discussion

Results revealed qualitative differences in affect labels between adults with and without TBI that would not have emerged on standard forced-choice tasks. Verbal differences did not appear to be attributable to differences in gaze patterns, leaving open the question of mechanisms of atypical affect processing in adults with TBI.

Crosstalk in Facial EMG and Its Reduction Using ICA

There is ample evidence that electromyography (EMG) signals from the corrugator supercilii and zygomatic major muscles can provide valuable information for the assessment of subjective emotional experiences. Although previous research suggested that facial EMG data could be affected by crosstalk from adjacent facial muscles, it remains unproven whether such crosstalk occurs and, if so, how it can be reduced. To investigate this, we instructed participants (n = 29) to perform the facial actions of frowning, smiling, chewing, and speaking, in isolation and combination. During these actions, we measured facial EMG signals from the corrugator supercilii, zygomatic major, masseter, and suprahyoid muscles. We performed an independent component analysis (ICA) of the EMG data and removed crosstalk components. Speaking and chewing induced EMG activity in the masseter and suprahyoid muscles, as well as the zygomatic major muscle. The ICA-reconstructed EMG signals reduced the effects of speaking and chewing on zygomatic major activity, compared with the original signals. These data suggest that: (1) mouth actions could induce crosstalk in zygomatic major EMG signals, and (2) ICA can reduce the effects of such crosstalk.

Physiological arousal guides situational appraisals and metacognitive recall for naturalistic experiences

As individuals navigate the world, they are bound to have emotionally intense experiences. These events not only influence momentary physiological and affective responses, but may also have a powerful impact on one's memory for their emotional experience. In this research, we used the naturalistic context of a haunted house to examine how physiological arousal is associated with metacognitive emotional memory (i.e., the extent to which an individual remembers having experienced a certain emotion). Participants first navigated the haunted house while heart rate and explicit situational appraisals were recorded, and then recalled specific events from the haunted house and the intensity of these affective events approximately one week later. We found that heart rate predicted both the intensity of reported scariness in the haunted house and meta-cognitive memory of affect during recall. Critically, we found evidence for malleability in metacognitive emotional memory based on how the event was initially labeled. Individuals tended to recall events that they explicitly labeled as fear-evoking as being more intense than they reported at the time of the event. We found the opposite relationship for events that they labeled as not fear-evoking. Taken together, this indicates that there are strong relationships between physiological arousal and emotional experiences in naturalistic contexts, but that affective labeling can modulate the relationship between these features when reflecting on the emotionality of that experience in memory.

Six facial prosodic expressions caregivers similarly display to infants and dogs

Parents tend to use a specific communication style, including specific facial expressions, when speaking to their preverbal infants which has important implications for children's healthy development. In the present study, we investigated these facial prosodic features of caregivers with a novel method that compares infant-, dog- and adult-directed communication. We identified three novel facial displays in addition to the already described three facial expressions (i.e. the 'prosodic faces') that mothers and fathers are typically displaying when interacting with their 1-18 month-old infants and family dogs, but not when interacting with another adult. The so-called Special Happy expression proved to be the most frequent face type during infant- and dog-directed communication which always includes a Duchenne marker to convey an honest and intense happy emotion of the speaker. These results suggest that the 'prosodic faces' play an important role in both adult-infant and human-dog interactions and fulfil specific functions: to call and maintain the partner's attention, to foster emotionally positive interactions, and to strengthen social bonds. Our study highlights the relevance of future comparative studies on facial prosody and its potential contribution to healthy emotional and cognitive development of infants.

Exploring effects of response biases in affect induction procedures

This study examined whether self-reports or ratings of experienced affect, often used as manipulation checks on the efficacy of affect induction procedures (AIPs), reflect genuine changes in affective states rather than response biases arising from demand characteristics or social desirability effects. In a between-participants design, participants were exposed to positive, negative and neutral images with valence-congruent music or sound to induce happy, sad and neutral mood. Half of the participants had to actively appraise each image whereas the other half viewed images passively. We hypothesised that if ratings of affective valence are subject to response biases then they should reflect the target mood in the same way for active appraisal and passive exposure as participants encountered the same affective stimuli in both conditions. We also tested whether the AIP resulted in mood-congruent changes in facial expressions analysed by FaceReader to see whether behavioural indicators corroborate the self-reports. The results showed that while participants' ratings reflected the induced target valence, the difference between positive and negative AIP was significantly attenuated in the active appraisal condition, suggesting that self-reports of mood experienced after the AIP are not entirely a reflection of response biases. However, there were no effects of the AIP on FaceReader valence scores, in line with theories questioning the existence of cross-culturally and inter-individually universal behavioural indicators of affective states. Efficacy of AIPs is therefore best checked using self-reports.

More What Duchenne Smiles Do, Less What They Express

We comment on an article by Sheldon et al. from a previous issue of Perspectives (May 2021). They argued that the presence of positive emotion (Hypothesis 1), the intensity of positive emotion (Hypothesis 2), and chronic positive mood (Hypothesis 3) are reliably signaled by the Duchenne smile (DS). We reexamined the cited literature in support of each hypothesis and show that the study findings were mostly inconclusive, irrelevant, incomplete, and/or misread. In fact, there is no single (empirical) article that would unanimously support the idea that DSs function solely as indicators of felt positive affect. Additional evidence is reviewed, suggesting that DSs can be-and often are-displayed deliberately and in the absence of positive feelings. Although DSs may lead to favorable interpersonal perceptions and positive emotional responses in the observer, we propose a functional view that focuses on what facial actions-here specifically DSs-do rather than what they express.

Impairment of emotional expression detection after unilateral medial temporal structure resection

Detecting emotional facial expressions is an initial and indispensable component of face-to-face communication. Neuropsychological studies on the neural substrates of this process have shown that bilateral amygdala lesions impaired the detection of emotional facial expressions. However, the findings were inconsistent, possibly due to the limited number of patients examined. Furthermore, whether this processing is based on emotional or visual factors of facial expressions remains unknown. To investigate this issue, we tested a group of patients (n = 23) with unilateral resection of medial temporal lobe structures, including the amygdala, and compared their performance under resected- and intact-hemisphere stimulation conditions. The participants were asked to detect normal facial expressions of anger and happiness, and artificially created anti-expressions, among a crowd with neutral expressions. Reaction times for the detection of normal expressions versus anti-expressions were shorter when the target faces were presented to the visual field contralateral to the intact hemisphere (i.e., stimulation of the intact hemisphere; e.g., right visual field for patients with right hemispheric resection) compared with the visual field contralateral to the resected hemisphere (i.e., stimulation of the resected hemisphere). Our findings imply that the medial temporal lobe structures, including the amygdala, play an essential role in the detection of emotional facial expressions, according to the emotional significance of the expressions.

The Emotional Experiences of Paralympic Swimming Medalists: Not All Wins and Losses Are Equal

The goal of this study was to determine if emotional expressions at the end of swimmers' 2016 Paralympic races varied according to medal won and if their race wins and losses were close or not close. Using FaceReader software, videos of 46 races of medal-winning Paralympic (M age = 24.6; SD = 5.4) swimmers' faces (78 males and 60 females) from 22 countries were analyzed. Silver medalists were angrier and sadder than gold medalists and angrier and more disgusted than bronze medalists. Swimmers who swam slower than their 2015 best time were angrier than Paralympians who swam faster. Paralympians who finished lower than their 2015 world ranking had more neutral emotions and were less happy than Paralympians who finished higher. Gold medalists who narrowly defeated silver medalists were less happy and more fearful than gold medalists who won easily. Bronze medalists with close wins had fewer neutral emotions and were happier, less angry, and more surprised than bronze medalists with not-close wins. All medalists with close wins were more surprised than medalists with easier wins. Bronze medalists with close losses to silver medalists were happier and less angry than bronze medalists who lost more easily. Effect sizes ranged from d = 0.27 to 1.01. These results provide theoretical support to basic emotion theory and confirm the anecdotal observations that Paralympic competition generates wide-ranging and diverse emotions.

Sty in the Mind's Eye: A Meta-Analytic Investigation of the Nomological Network and Internal Consistency of the "Reading the Mind in the Eyes" Test

The Reading the Mind in the Eyes Test (RMET) is the most popular adult measure of individual differences in theory of mind. We present a meta-analytic investigation of the test's psychometric properties (k = 119 effect sizes, 61 studies, n_total = 8,611 persons). Using random effects models, we found the internal consistency of the test was acceptable (α = .73). However, the RMET was more strongly related with emotion perception (r = .33, ρ = .48) relative to alternative theory of mind measures (r = .29, ρ = .39), and weakly to moderately related with vocabulary (r = .25, ρ = .32), cognitive empathy (r = .14, ρ = .20), and affective empathy (r = .13, ρ = .19). Overall, we conclude that the RMET operates rather as emotion perception measure than as theory of mind measure, challenging the interpretation of RMET results.

Expression Authenticity: The Role of Genuine and Deliberate Displays in Emotion Perception

People dedicate significant attention to others' facial expressions and to deciphering their meaning. Hence, knowing whether such expressions are genuine or deliberate is important. Early research proposed that authenticity could be discerned based on reliable facial muscle activations unique to genuine emotional experiences that are impossible to produce voluntarily. With an increasing body of research, such claims may no longer hold up to empirical scrutiny. In this article, expression authenticity is considered within the context of senders' ability to produce convincing facial displays that resemble genuine affect and human decoders' judgments of expression authenticity. This includes a discussion of spontaneous vs. posed expressions, as well as appearance- vs. elicitation-based approaches for defining emotion recognition accuracy. We further expand on the functional role of facial displays as neurophysiological states and communicative signals, thereby drawing upon the encoding-decoding and affect-induction perspectives of emotion expressions. Theoretical and methodological issues are addressed with the aim to instigate greater conceptual and operational clarity in future investigations of expression authenticity.

Anger and Sadness Expressions Situated in Both Positive and Negative Contexts: An Investigation in South Korea and the United States

A formidable challenge to the research of non-verbal behavior can be in the assumptions that we sometimes make, and the subsequent questions that arise from those assumptions. In this article, we proceed with an investigation that would have been precluded by the assumption of a 1:1 correspondence between facial expressions and discrete emotional experiences. We investigated two expressions that in the normative sense are considered negative expressions. One expression, "anger" could be described as clenched fists, furrowed brows, tense jaws and lips, the showing of teeth, and flared nostrils, and the other "sadness" could be described as downward turned mouths, tears, drooping eyes, and wrinkled foreheads. Here, we investigated the prevalence, understanding, and use of these expressions in both positive and negative contexts in South Korea and the United States. We found evidence in both cultures, that anger and sadness displays are used to express positive emotions, a notion relevant to Dimorphous Theory. Moreover, we found that anger and sadness expressions communicated appetitive feelings of wanting to "go!" and consummatory feelings of wanting to "pause," respectively. There were moderations of our effects consistent with past work in Affect Valuation Theory and Display Rule Theory. We discuss our findings, their theoretical relevance, and how the assumptions that are made can narrow the questions that we ask in the field on non-verbal behavior.

The dynamic process of ambiguous emotion perception

Everyday social interactions hinge on our ability to resolve uncertainty in nonverbal cues. For example, although some facial expressions (e.g. happy, angry) convey a clear affective meaning, others (e.g. surprise) are ambiguous, in that their meaning is determined by the context. Here, we used mouse-tracking to examine the underlying process of resolving uncertainty. Previous work has suggested an initial negativity, in part via faster response times for negative than positive ratings of surprise. We examined valence categorizations of filtered images in order to compare faster (low spatial frequencies; LSF) versus more deliberate processing (high spatial frequencies; HSF). When participants categorised faces as "positive", they first exhibited a partial attraction toward the competing ("negative") response option, and this effect was exacerbated for HSF than LSF faces. Thus, the effect of response conflict due to an initial negativity bias was exaggerated for HSF faces, likely because these images allow for greater deliberation than the LSFs. These results are consistent with the notion that more positive categorizations are characterised by an initial attraction to a default, negative response.

The emotion-facial expression link: evidence from human and automatic expression recognition

While it has been taken for granted in the development of several automatic facial expression recognition tools, the question of the coherence between subjective feelings and facial expressions is still a subject of debate. On one hand, the "Basic Emotion View" conceives emotions as genetically hardwired and, therefore, being genuinely displayed through facial expressions. Consequently, emotion recognition is perceiver independent. On the other hand, the constructivist approach conceives emotions as socially constructed, the emotional meaning of a facial expression being inferred by the perceiver. Hence, emotion recognition is perceiver dependent. In order (1) to evaluate the coherence between the subjective feeling of emotions and their spontaneous facial displays, and (2) to compare the recognition of such displays by human perceivers and by an automatic facial expression classifier, 232 videos of expressers recruited to carry out an emotion elicitation task were annotated by 1383 human perceivers as well as by Affdex, an automatic classifier. Results show a weak consistency between self-reported emotional states by expressers and their facial emotional displays. They also show low accuracy both of human perceivers and of the automatic classifier to infer the subjective feeling from the spontaneous facial expressions displayed by expressers. However, the results are more in favor of a perceiver-dependent view. Based on these results, the hypothesis of genetically hardwired emotion genuinely displayed is difficult to support, whereas the idea of emotion and facial expression as being socially constructed appears to be more likely. Accordingly, automatic emotion recognition tools based on facial expressions should be questioned.

Comment: A role of Language in Infant Emotion Concept Acquisition

Ruba and Repacholi (2020) review an important debate in the emotion development literature: whether infants can perceive and understand facial configurations as instances of discrete emotion categories. Consistent with a psychological constructionist account (Lindquist & Gendron, 2013; Shablack & Lindquist, 2019), they conclude that infants can perceive valence on faces, but argue the evidence is far from clear that infants perceive and understand discrete emotions. Ruba and Repacholi outline a novel developmental trajectory of emotion perception and understanding in which early emotion concept learning may be language-independent. In this comment, we argue that language may play a role in emotion concept acquisition even prior to children’s ability to produce emotion labels. We look forward to future research addressing this hypothesis.

EmoCo: Visual Analysis of Emotion Coherence in Presentation Videos

Emotions play a key role in human communication and public presentations. Human emotions are usually expressed through multiple modalities. Therefore, exploring multimodal emotions and their coherence is of great value for understanding emotional expressions in presentations and improving presentation skills. However, manually watching and studying presentation videos is often tedious and time-consuming. There is a lack of tool support to help conduct an efficient and in-depth multi-level analysis. Thus, in this paper, we introduce EmoCo, an interactive visual analytics system to facilitate efficient analysis of emotion coherence across facial, text, and audio modalities in presentation videos. Our visualization system features a channel coherence view and a sentence clustering view that together enable users to obtain a quick overview of emotion coherence and its temporal evolution. In addition, a detail view and word view enable detailed exploration and comparison from the sentence level and word level, respectively. We thoroughly evaluate the proposed system and visualization techniques through two usage scenarios based on TED Talk videos and interviews with two domain experts. The results demonstrate the effectiveness of our system in gaining insights into emotion coherence in presentations.

Affective and Semantic Representations of Valence: A Conceptual Framework

The current article discusses the distinction between affective valence—the degree to which an affective response represents pleasure or displeasure—and semantic valence, the degree to which an object or event is considered positive or negative. To date, measures that reflect positivity and negativity are usually placed under the same conceptual umbrella (e.g., valence, affective, emotional), with minimal distinction between the modes of valence they reflect. Recent work suggests that what might seem to reflect a monolithic structure of valence has at least two different, confounding underlying sources, affective and semantic, that are fundamentally distinct, dissociable, and that obey different, recognizable rules. The current work discusses this distinction and provides implications for affective science from both the theoretical and the empirical perspective.

An unfamiliar social presence reduces facial disgust responses to food stimuli

Consumers' emotional responses complement sensory and hedonic ratings in the prediction of food choice and consumption behaviour. The challenge with the measurement of consumption emotions is that emotions are highly context dependent. For emotion evaluations to bring greater insight to food research and development, it is essential that the influence of contextual variables on emotion are quantified. The present study contributes to the discussion with an investigation of the effect of an unfamiliar social presence on affective facial responses to visual food stimuli. Seventy participants (52 female and 18 male) viewed food images of varying acceptability either alone, or in the presence of the researcher. Subjective liking ratings were measured using a labelled affective magnitude scale, and facial muscle activity from zygomaticus major (contracted during smiling), corrugator supercilii (contracted during frowning) and levator labii superioris (contracted during nose wrinkling) were measured with an EMG recording system. Controlling for individual differences in facial expressivity and food image acceptability using linear mixed models, it was found that social context did not predict smiling or frowning muscle activity. Social context did predict the intensity of muscle activity indicative of a disgust response, with participants in the observed condition exhibiting less levator activity than participants in the alone condition. Regardless of social context, each muscle was found to have a relationship with subjective liking, with the direction of effects as expected. The results indicate that emotional stimuli and social context both influence food-evoked facial expression and provides support for the utility of facial EMG in measuring food-evoked emotion.

Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements

It is commonly assumed that a person's emotional state can be readily inferred from his or her facial movements, typically called emotional expressions or facial expressions. This assumption influences legal judgments, policy decisions, national security protocols, and educational practices; guides the diagnosis and treatment of psychiatric illness, as well as the development of commercial applications; and pervades everyday social interactions as well as research in other scientific fields such as artificial intelligence, neuroscience, and computer vision. In this article, we survey examples of this widespread assumption, which we refer to as the common view, and we then examine the scientific evidence that tests this view, focusing on the six most popular emotion categories used by consumers of emotion research: anger, disgust, fear, happiness, sadness, and surprise. The available scientific evidence suggests that people do sometimes smile when happy, frown when sad, scowl when angry, and so on, as proposed by the common view, more than what would be expected by chance. Yet how people communicate anger, disgust, fear, happiness, sadness, and surprise varies substantially across cultures, situations, and even across people within a single situation. Furthermore, similar configurations of facial movements variably express instances of more than one emotion category. In fact, a given configuration of facial movements, such as a scowl, often communicates something other than an emotional state. Scientists agree that facial movements convey a range of information and are important for social communication, emotional or otherwise. But our review suggests an urgent need for research that examines how people actually move their faces to express emotions and other social information in the variety of contexts that make up everyday life, as well as careful study of the mechanisms by which people perceive instances of emotion in one another. We make specific research recommendations that will yield a more valid picture of how people move their faces to express emotions and how they infer emotional meaning from facial movements in situations of everyday life. This research is crucial to provide consumers of emotion research with the translational information they require.

Tracking the affective state of unseen persons

Emotion recognition is widely assumed to be determined by face and body features, and measures of emotion perception typically use unnatural, static, or decontextualized face stimuli. Using our method called affective tracking, we show that observers can infer, recognize, and track over time the affect of an invisible person based solely on visual spatial context. We further show that visual context provides a substantial and unique contribution to the perception of human affect, beyond the information available from face and body. This method reveals that emotion recognition is, at its heart, a context-based process.

Emotion recognition is an essential human ability critical for social functioning. It is widely assumed that identifying facial expression is the key to this, and models of emotion recognition have mainly focused on facial and bodily features in static, unnatural conditions. We developed a method called affective tracking to reveal and quantify the enormous contribution of visual context to affect (valence and arousal) perception. When characters’ faces and bodies were masked in silent videos, viewers inferred the affect of the invisible characters successfully and in high agreement based solely on visual context. We further show that the context is not only sufficient but also necessary to accurately perceive human affect over time, as it provides a substantial and unique contribution beyond the information available from face and body. Our method (which we have made publicly available) reveals that emotion recognition is, at its heart, an issue of context as much as it is about faces.

Facial Expressions of Basic Emotions in Japanese Laypeople

Facial expressions that show emotion play an important role in human social interactions. In previous theoretical studies, researchers have suggested that there are universal, prototypical facial expressions specific to basic emotions. However, the results of some empirical studies that tested the production of emotional facial expressions based on particular scenarios only partially supported the theoretical predictions. In addition, all of the previous studies were conducted in Western cultures. We investigated Japanese laypeople (n = 65) to provide further empirical evidence regarding the production of emotional facial expressions. The participants produced facial expressions for six basic emotions (anger, disgust, fear, happiness, sadness, and surprise) in specific scenarios. Under the baseline condition, the participants imitated photographs of prototypical facial expressions. The produced facial expressions were automatically coded using FaceReader in terms of the intensities of emotions and facial action units. In contrast to the photograph condition, where all target emotions were shown clearly, the scenario condition elicited the target emotions clearly only for happy and surprised expressions. The photograph and scenario conditions yielded different profiles for the intensities of emotions and facial action units associated with all of the facial expressions tested. These results provide partial support for the theory of universal, prototypical facial expressions for basic emotions but suggest the possibility that the theory may need to be modified based on empirical evidence.

In the face of stress: Interpreting individual differences in stress-induced facial expressions

Stress is an inevitable part of life that can profoundly impact social and emotional functioning, contributing to the development of psychiatric disease. One key component of emotion and social processing is facial expressions, which humans can readily detect and react to even without conscious awareness. Facial expressions have been the focus of philosophic and scientific interest for centuries. Historically, facial expressions have been relegated to peripheral indices of fixed emotion states. More recently, affective neuroscience has undergone a conceptual revolution, resulting in novel interpretations of these muscle movements. Here, we review the role of facial expressions according to the leading affective neuroscience theories, including constructed emotion and social-motivation accounts. We specifically highlight recent data (Mayo et al, 2018) demonstrating the way in which stress shapes facial expressions and how this is influenced by individual factors. In particular, we focus on the consequence of genetic variation within the endocannabinoid system, a neuromodulatory system implicated in stress and emotion, and its impact on stress-induced facial muscle activity. In a re-analysis of this dataset, we highlight how gender may also influence these processes, conceptualized as variation in the "fight-or-flight" or "tend-and-befriend" behavioral responses to stress. We speculate on how these interpretations may contribute to a broader understanding of facial expressions, discuss the potential use of facial expressions as a trans-diagnostic marker of psychiatric disease, and suggest future work necessary to resolve outstanding questions.

Differences in Emotion Recognition From Body and Face Cues Between Deaf and Hearing Individuals

Deaf individuals may compensate for the lack of the auditory input by showing enhanced capacities in certain visual tasks. Here we assessed whether this also applies to recognition of emotions expressed by bodily and facial cues. In Experiment 1, we compared deaf participants and hearing controls in a task measuring recognition of the six basic emotions expressed by actors in a series of video-clips in which either the face, the body, or both the face and body were visible. In Experiment 2, we measured the weight of body and face cues in conveying emotional information when intense genuine emotions are expressed, a situation in which face expressions alone may have ambiguous valence. We found that deaf individuals were better at identifying disgust and fear from body cues (Experiment 1) and in integrating face and body cues in case of intense negative genuine emotions (Experiment 2). Our findings support the capacity of deaf individuals to compensate for the lack of the auditory input enhancing perceptual and attentional capacities in the spared modalities, showing that this capacity extends to the affective domain.

Distinct facial expressions represent pain and pleasure across cultures

Real-world studies show that the facial expressions produced during pain and orgasm-two different and intense affective experiences-are virtually indistinguishable. However, this finding is counterintuitive, because facial expressions are widely considered to be a powerful tool for social interaction. Consequently, debate continues as to whether the facial expressions of these extreme positive and negative affective states serve a communicative function. Here, we address this debate from a novel angle by modeling the mental representations of dynamic facial expressions of pain and orgasm in 40 observers in each of two cultures (Western, East Asian) using a data-driven method. Using a complementary approach of machine learning, an information-theoretic analysis, and a human perceptual discrimination task, we show that mental representations of pain and orgasm are physically and perceptually distinct in each culture. Cross-cultural comparisons also revealed that pain is represented by similar face movements across cultures, whereas orgasm showed distinct cultural accents. Together, our data show that mental representations of the facial expressions of pain and orgasm are distinct, which questions their nondiagnosticity and instead suggests they could be used for communicative purposes. Our results also highlight the potential role of cultural and perceptual factors in shaping the mental representation of these facial expressions. We discuss new research directions to further explore their relationship to the production of facial expressions.

Surprise: unfolding of facial expressions

Responses to surprising events are dynamic. We argue that initial responses are primarily driven by the unexpectedness of the surprising event and reflect an interrupted and surprised state in which the outcome does not make sense yet. Later responses, after sense-making, are more likely to incorporate the valence of the outcome itself. To identify initial and later responses to surprising stimuli, we conducted two repetition-change studies and coded the general valence of facial expressions using computerised facial coding and specific facial action using the Facial Action Coding System (FACS). Results partly supported our unfolding logic. The computerised coding showed that initial expressions to positive surprises were less positive than later expressions. Moreover, expressions to positive and negative surprises were initially similar, but after some time differentiated depending on the valence of the event. Importantly, these patterns were particularly pronounced in a subset of facially expressive participants, who also showed facial action in the FACS coding. The FACS data showed that the initial phase was characterised by limited facial action, whereas the later increase in positivity seems to be explained by smiling. Conceptual as well as methodological implications are discussed.

Functional Networks for Social Communication in the Macaque Monkey

All primates communicate. To dissect the neural circuits of social communication, we used fMRI to map non-human primate brain regions for social perception, second-person (interactive) social cognition, and orofacial movement generation. Face perception, second-person cognition, and face motor networks were largely non-overlapping and acted as distinct functional units rather than an integrated feedforward-processing pipeline. Whereas second-person context selectively engaged a region of medial prefrontal cortex, production of orofacial movements recruited distributed subcortical and cortical areas in medial and lateral frontal and insular cortex. These areas exhibited some specialization, but not dissociation, of function along the medio-lateral axis. Production of lipsmack movements recruited areas including putative homologs of Broca's area. These findings provide a new view of the neural architecture for social communication and suggest expressive orofacial movements generated by lateral premotor cortex as a putative evolutionary precursor to human speech.

Discrimination between smiling faces: Human observers vs. automated face analysis

This study investigated (a) how prototypical happy faces (with happy eyes and a smile) can be discriminated from blended expressions with a smile but non-happy eyes, depending on type and intensity of the eye expression; and (b) how smile discrimination differs for human perceivers versus automated face analysis, depending on affective valence and morphological facial features. Human observers categorized faces as happy or non-happy, or rated their valence. Automated analysis (FACET software) computed seven expressions (including joy/happiness) and 20 facial action units (AUs). Physical properties (low-level image statistics and visual saliency) of the face stimuli were controlled. Results revealed, first, that some blended expressions (especially, with angry eyes) had lower discrimination thresholds (i.e., they were identified as "non-happy" at lower non-happy eye intensities) than others (especially, with neutral eyes). Second, discrimination sensitivity was better for human perceivers than for automated FACET analysis. As an additional finding, affective valence predicted human discrimination performance, whereas morphological AUs predicted FACET discrimination. FACET can be a valid tool for categorizing prototypical expressions, but is currently more limited than human observers for discrimination of blended expressions. Configural processing facilitates detection of in/congruence(s) across regions, and thus detection of non-genuine smiling faces (due to non-happy eyes).

Emotion Perception as Conceptual Synchrony

Psychological research on emotion perception anchors heavily on an object perception analogy. We present static “cues,” such as facial expressions, as objects for perceivers to categorize. Yet in the real world, emotions play out as dynamic multidimensional events. Current theoretical approaches and research methods are limited in their ability to capture this complexity. We draw on insights from a predictive coding account of neural activity and a grounded cognition account of concept representation to conceive of emotion perception as a stream of synchronized conceptualizations between two individuals, which is supported and shaped by language. We articulate how this framework can illuminate the fundamental need to study culture, as well as other sources of conceptual variation, in unpacking conceptual synchrony in emotion. We close by suggesting that the conceptual system provides the necessary flexibility to overcome gaps in emotional synchrony.

Facial color is an efficient mechanism to visually transmit emotion

Emotions correspond to the execution of a number of computations by the central nervous system. Previous research has studied the hypothesis that some of these computations yield visually identifiable facial muscle movements. Here, we study the supplemental hypothesis that some of these computations yield facial blood flow changes unique to the category and valence of each emotion. These blood flow changes are visible as specific facial color patterns to observers, who can then successfully decode the emotion. We present converging computational and behavioral evidence in favor of this hypothesis. Our studies demonstrate that people identify the correct emotion category and valence from these facial colors, even in the absence of any facial muscle movement.

Facial expressions of emotion in humans are believed to be produced by contracting one’s facial muscles, generally called action units. However, the surface of the face is also innervated with a large network of blood vessels. Blood flow variations in these vessels yield visible color changes on the face. Here, we study the hypothesis that these visible facial colors allow observers to successfully transmit and visually interpret emotion even in the absence of facial muscle activation. To study this hypothesis, we address the following two questions. Are observable facial colors consistent within and differential between emotion categories and positive vs. negative valence? And does the human visual system use these facial colors to decode emotion from faces? These questions suggest the existence of an important, unexplored mechanism of the production of facial expressions of emotion by a sender and their visual interpretation by an observer. The results of our studies provide evidence in favor of our hypothesis. We show that people successfully decode emotion using these color features, even in the absence of any facial muscle activation. We also demonstrate that this color signal is independent from that provided by facial muscle movements. These results support a revised model of the production and perception of facial expressions of emotion where facial color is an effective mechanism to visually transmit and decode emotion.

Facial Displays Are Tools for Social Influence

Based on modern theories of signal evolution and animal communication, the behavioral ecology view of facial displays (BECV) reconceives our 'facial expressions of emotion' as social tools that serve as lead signs to contingent action in social negotiation. BECV offers an externalist, functionalist view of facial displays that is not bound to Western conceptions about either expressions or emotions. It easily accommodates recent findings of diversity in facial displays, their public context-dependency, and the curious but common occurrence of solitary facial behavior. Finally, BECV restores continuity of human facial behavior research with modern functional accounts of non-human communication, and provides a non-mentalistic account of facial displays well-suited to new developments in artificial intelligence and social robotics.

Ausência de Prejuízo no Reconhecimento de Expressões Faciais entre Indivíduos com Parkinson

RESUMO: Entre os sintomas não motores da doença de Parkinson, dificuldades no reconhecimento de expressões faciais emocionais vêm sendo amplamente discutidas, pois as áreas cerebrais relacionadas a tal habilidade podem estar afetadas na doença. Este estudo investigou, em idosos, o reconhecimento das seis expressões emocionais faciais consideradas universais por meio do instrumento Teste de Percepção Emocional de Faces, em que participantes executam uma tarefa de reconhecimento de expressões emocionais faciais. Participaram 41 indivíduos com idade média de 64,9 anos, 27 homens e 14 mulheres. Não foi observada significativa dificuldade no reconhecimento de nenhuma das expressões emocionais por parte dos parkinsonianos. Tal resultado pode indicar a necessidade do desenvolvimento de instrumentos e técnicas mais adequadas para esse tipo de investigação na população brasileira.

Dogs and humans respond to emotionally competent stimuli by producing different facial actions

The commonality of facial expressions of emotion has been studied in different species since Darwin, with most of the research focusing on closely related primate species. However, it is unclear to what extent there exists common facial expression in species more phylogenetically distant, but sharing a need for common interspecific emotional understanding. Here we used the objective, anatomically-based tools, FACS and DogFACS (Facial Action Coding Systems), to quantify and compare human and domestic dog facial expressions in response to emotionally-competent stimuli associated with different categories of emotional arousal. We sought to answer two questions: Firstly, do dogs display specific discriminatory facial movements in response to different categories of emotional stimuli? Secondly, do dogs display similar facial movements to humans when reacting in emotionally comparable contexts? We found that dogs displayed distinctive facial actions depending on the category of stimuli. However, dogs produced different facial movements to humans in comparable states of emotional arousal. These results refute the commonality of emotional expression across mammals, since dogs do not display human-like facial expressions. Given the unique interspecific relationship between dogs and humans, two highly social but evolutionarily distant species sharing a common environment, these findings give new insight into the origin of emotion expression.

Postural Communication of Emotion: Perception of Distinct Poses of Five Discrete Emotions

Emotion can be communicated through multiple distinct modalities. However, an often-ignored channel of communication is posture. Recent research indicates that bodily posture plays an important role in the perception of emotion. However, research examining postural communication of emotion is limited by the variety of validated emotion poses and unknown cohesion of categorical and dimensional ratings. The present study addressed these limitations. Specifically, we examined individuals' (1) categorization of emotion postures depicting 5 discrete emotions (joy, sadness, fear, anger, and disgust), (2) categorization of different poses depicting the same discrete emotion, and (3) ratings of valence and arousal for each emotion pose. Findings revealed that participants successfully categorized each posture as the target emotion, including disgust. Moreover, participants accurately identified multiple distinct poses within each emotion category. In addition to the categorical responses, dimensional ratings of valence and arousal revealed interesting overlap and distinctions between and within emotion categories. These findings provide the first evidence of an identifiable posture for disgust and instantiate the principle of equifinality of emotional communication through the inclusion of distinct poses within emotion categories. Additionally, the dimensional ratings corroborated the categorical data and provide further granularity for future researchers to consider in examining how distinct emotion poses are perceived.

Can perceivers recognise emotions from spontaneous expressions?

Posed stimuli dominate the study of nonverbal communication of emotion, but concerns have been raised that the use of posed stimuli may inflate recognition accuracy relative to spontaneous expressions. Here, we compare recognition of emotions from spontaneous expressions with that of matched posed stimuli. Participants made forced-choice judgments about the expressed emotion and whether the expression was spontaneous, and rated expressions on intensity (Experiments 1 and 2) and prototypicality (Experiment 2). Listeners were able to accurately infer emotions from both posed and spontaneous expressions, from auditory, visual, and audiovisual cues. Furthermore, perceived intensity and prototypicality were found to play a role in the accurate recognition of emotion, particularly from spontaneous expressions. Our findings demonstrate that perceivers can reliably recognise emotions from spontaneous expressions, and that depending on the comparison set, recognition levels can even be equivalent to that of posed stimulus sets.