Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time

Many morphs: Parsing gesture signals from the noise

Parsing signals from noise is a general problem for signallers and recipients, and for researchers studying communicative systems. Substantial efforts have been invested in comparing how other species encode information and meaning, and how signalling is structured. However, research depends on identifying and discriminating signals that represent meaningful units of analysis. Early approaches to defining signal repertoires applied top-down approaches, classifying cases into predefined signal types. Recently, more labour-intensive methods have taken a bottom-up approach describing detailed features of each signal and clustering cases based on patterns of similarity in multi-dimensional feature-space that were previously undetectable. Nevertheless, it remains essential to assess whether the resulting repertoires are composed of relevant units from the perspective of the species using them, and redefining repertoires when additional data become available. In this paper we provide a framework that takes data from the largest set of wild chimpanzee (Pan troglodytes) gestures currently available, splitting gesture types at a fine scale based on modifying features of gesture expression using latent class analysis (a model-based cluster detection algorithm for categorical variables), and then determining whether this splitting process reduces uncertainty about the goal or community of the gesture. Our method allows different features of interest to be incorporated into the splitting process, providing substantial future flexibility across, for example, species, populations, and levels of signal granularity. Doing so, we provide a powerful tool allowing researchers interested in gestural communication to establish repertoires of relevant units for subsequent analyses within and between systems of communication.

Facial representations of complex affective states combining pain and a negative emotion

Pain is rarely communicated alone, as it is often accompanied by emotions such as anger or sadness. Communicating these affective states involves shared representations. However, how an individual conceptually represents these combined states must first be tested. The objective of this study was to measure the interaction between pain and negative emotions on two types of facial representations of these states, namely visual (i.e., interactive virtual agents; VAs) and sensorimotor (i.e., one's production of facial configurations). Twenty-eight participants (15 women) read short written scenarios involving only pain or a combined experience of pain and a negative emotion (anger, disgust, fear, or sadness). They produced facial configurations representing these experiences on the faces of the VAs and on their face (own production or imitation of VAs). The results suggest that affective states related to a direct threat to the body (i.e., anger, disgust, and pain) share a similar facial representation, while those that present no immediate danger (i.e., fear and sadness) differ. Although visual and sensorimotor representations of these states provide congruent affective information, they are differently influenced by factors associated with the communication cycle. These findings contribute to our understanding of pain communication in different affective contexts.

Congenital Anosmia and Facial Emotion Recognition

Major functions of the olfactory system include guiding ingestion and avoidance of environmental hazards. People with anosmia report reliance on others, for example to check the edibility of food, as their primary coping strategy. Facial expressions are a major source of non-verbal social information that can be used to guide approach and avoidance behaviour. Thus, it is of interest to explore whether a life-long absence of the sense of smell heightens sensitivity to others' facial emotions, particularly those depicting threat. In the present, online study 28 people with congenital anosmia (mean age 43.46) and 24 people reporting no olfactory dysfunction (mean age 42.75) completed a facial emotion recognition task whereby emotionally neutral faces (6 different identities) morphed, over 40 stages, to express one of 5 basic emotions: anger, disgust, fear, happiness, or sadness. Results showed that, while the groups did not differ in their ability to identify the final, full-strength emotional expressions, nor in the accuracy of their first response, the congenital anosmia group successfully identified the emotions at significantly lower intensity (i.e. an earlier stage of the morph) than the control group. Exploratory analysis showed this main effect was primarily driven by an advantage in detecting anger and disgust. These findings indicate the absence of a functioning sense of smell during development leads to compensatory changes in visual, social cognition. Future work should explore the neural and behavioural basis for this advantage.

Interoceptive sensitivity and perception of others' emotions: an investigation based on a two-stage model

Recent research shows that sensitivity to interoceptive sensitivity is associated with a more granular experience of emotions. These studies suggest that individuals sensitive to their interoceptive signals can better perceive somatic physiological changes as compared to their counterparts. Therefore, they discriminate among a wide and subtle range of emotions. Further, the perception of others' emotions could be based on our own emotional experiences. However, whether interoceptive sensitivity is related to the perception of others' emotions remains unclear. Therefore, this study examined the relationship between interoceptive sensitivity and emotional perception. Considering the model that emotion perception comprises two processes, categorization of facial expressions and approach-avoidance responses, this study examined both categorizations of facial expressions and approach-avoidance responses. The results showed no relationship between interoceptive sensitivity and the perception of emotion, which suggests that interoceptive sensitivity is related to the experience of emotion but does not affect the granularity of emotional perception. Future studies should diversely and empirically examine the role of the body in emotional perception from the perspective of interoceptive sensitivity.

Visual Field Restriction in the Recognition of Basic Facial Expressions: A Combined Eye Tracking and Gaze Contingency Study

Uncertainties and discrepant results in identifying crucial areas for emotional facial expression recognition may stem from the eye tracking data analysis methods used. Many studies employ parameters of analysis that predominantly prioritize the examination of the foveal vision angle, ignoring the potential influences of simultaneous parafoveal and peripheral information. To explore the possible underlying causes of these discrepancies, we investigated the role of the visual field aperture in emotional facial expression recognition with 163 volunteers randomly assigned to three groups: no visual restriction (NVR), parafoveal and foveal vision (PFFV), and foveal vision (FV). Employing eye tracking and gaze contingency, we collected visual inspection and judgment data over 30 frontal face images, equally distributed among five emotions. Raw eye tracking data underwent Eye Movements Metrics and Visualizations (EyeMMV) processing. Accordingly, the visual inspection time, number of fixations, and fixation duration increased with the visual field restriction. Nevertheless, the accuracy showed significant differences among the NVR/FV and PFFV/FV groups, despite there being no difference in NVR/PFFV. The findings underscore the impact of specific visual field areas on facial expression recognition, highlighting the importance of parafoveal vision. The results suggest that eye tracking data analysis methods should incorporate projection angles extending to at least the parafoveal level.

The Predictive Role of the Posterior Cerebellum in the Processing of Dynamic Emotions

Recent studies have bolstered the important role of the cerebellum in high-level socio-affective functions. In particular, neuroscientific evidence shows that the posterior cerebellum is involved in social cognition and emotion processing, presumably through its involvement in temporal processing and in predicting the outcomes of social sequences. We used cerebellar transcranial random noise stimulation (ctRNS) targeting the posterior cerebellum to affect the performance of 32 healthy participants during an emotion discrimination task, including both static and dynamic facial expressions (i.e., transitioning from a static neutral image to a happy/sad emotion). ctRNS, compared to the sham condition, significantly reduced the participants' accuracy to discriminate static sad facial expressions, but it increased participants' accuracy to discriminate dynamic sad facial expressions. No effects emerged with happy faces. These findings may suggest the existence of two different circuits in the posterior cerebellum for the processing of negative emotional stimuli: a first-time-independent mechanism which can be selectively disrupted by ctRNS, and a second time-dependent mechanism of predictive "sequence detection" which can be selectively enhanced by ctRNS. This latter mechanism might be included among the cerebellar operational models constantly engaged in the rapid adjustment of social predictions based on dynamic behavioral information inherent to others' actions. We speculate that it might be one of the basic principles underlying the understanding of other individuals' social and emotional behaviors during interactions.

Perception and memory-based representations of facial emotions: Associations with personality functioning, affective states and recognition abilities

Personality traits and affective states are associated with biases in facial emotion perception. However, the precise personality impairments and affective states that underlie these biases remain largely unknown. To investigate how relevant factors influence facial emotion perception and recollection, Experiment 1 employed an image reconstruction approach in which community-dwelling adults (N = 89) rated the similarity of pairs of facial expressions, including those recalled from memory. Subsequently, perception- and memory-based expression representations derived from such ratings were assessed across participants and related to measures of personality impairment, state affect, and visual recognition abilities. Impairment in self-direction and level of positive affect accounted for the largest components of individual variability in perception and memory representations, respectively. Additionally, individual differences in these representations were impacted by face recognition ability. In Experiment 2, adult participants (N = 81) rated facial image reconstructions derived in Experiment 1, revealing that individual variability was associated with specific visual face properties, such as expressiveness, representation accuracy, and positivity/negativity. These findings highlight and clarify the influence of personality, affective state, and recognition abilities on individual differences in the perception and recollection of facial expressions.

The role of the differential outcomes procedure and schizotypy in the recognition of dynamic facial expressions of emotions

Emotional facial expression recognition is a key ability for adequate social functioning. The current study aims to test if the differential outcomes procedure (DOP) may improve the recognition of dynamic facial expressions of emotions and to further explore whether schizotypal personality traits may have any effect on performance. 183 undergraduate students completed a task where a face morphed from a neutral expression to one of the six basic emotions at full intensity over 10 s. Participants had to press spacebar as soon as they identified the emotion and choose which had appeared. In the first block, participants received no outcomes. In the second block, a group received specific outcomes associated to each emotion (DOP group), while another group received non-differential outcomes after correctly responding (NOP group). Employing generalized linear models (GLMs) and Bayesian inference we estimated different parameters to answer our research goals. Schizotypal personality traits did not seem to affect dynamic emotional facial expression recognition. Participants of the DOP group were less likely to respond incorrectly to faces showing Fear and Surprise at fewer intensity levels. This may suggest that the DOP could lead to better identification of the main features that differentiate each facial expression of emotion.

Cultural facial expressions dynamically convey emotion category and intensity information

Communicating emotional intensity plays a vital ecological role because it provides valuable information about the nature and likelihood of the sender's behavior.1,2,3 For example, attack often follows signals of intense aggression if receivers fail to retreat.4,5 Humans regularly use facial expressions to communicate such information.6,7,8,9,10,11 Yet how this complex signaling task is achieved remains unknown. We addressed this question using a perception-based, data-driven method to mathematically model the specific facial movements that receivers use to classify the six basic emotions-"happy," "surprise," "fear," "disgust," "anger," and "sad"-and judge their intensity in two distinct cultures (East Asian, Western European; total n = 120). In both cultures, receivers expected facial expressions to dynamically represent emotion category and intensity information over time, using a multi-component compositional signaling structure. Specifically, emotion intensifiers peaked earlier or later than emotion classifiers and represented intensity using amplitude variations. Emotion intensifiers are also more similar across emotions than classifiers are, suggesting a latent broad-plus-specific signaling structure. Cross-cultural analysis further revealed similarities and differences in expectations that could impact cross-cultural communication. Specifically, East Asian and Western European receivers have similar expectations about which facial movements represent high intensity for threat-related emotions, such as "anger," "disgust," and "fear," but differ on those that represent low threat emotions, such as happiness and sadness. Together, our results provide new insights into the intricate processes by which facial expressions can achieve complex dynamic signaling tasks by revealing the rich information embedded in facial expressions.

Recognizing facial expressions of emotion amid noise: A dynamic advantage

Humans communicate internal states through complex facial movements shaped by biological and evolutionary constraints. Although real-life social interactions are flooded with dynamic signals, current knowledge on facial expression recognition mainly arises from studies using static face images. This experimental bias might stem from previous studies consistently reporting that young adults minimally benefit from the richer dynamic over static information, whereas children, the elderly, and clinical populations very strongly do (Richoz, Jack, Garrod, Schyns, & Caldara, 2015, Richoz, Jack, Garrod, Schyns, & Caldara, 2018b). These observations point to a near-optimal facial expression decoding system in young adults, almost insensitive to the advantage of dynamic over static cues. Surprisingly, no study has yet tested the idea that such evidence might be rooted in a ceiling effect. To this aim, we asked 70 healthy young adults to perform static and dynamic facial expression recognition of the six basic expressions while parametrically and randomly varying the low-level normalized phase and contrast signal (0%-100%) of the faces. As predicted, when 100% face signals were presented, static and dynamic expressions were recognized with equal efficiency with the exception of those with the most informative dynamics (i.e., happiness and surprise). However, when less signal was available, dynamic expressions were all better recognized than their static counterpart (peaking at ∼20%). Our data show that facial movements increase our ability to efficiently identify emotional states of others under the suboptimal visual conditions that can occur in everyday life. Dynamic signals are more effective and sensitive than static ones for decoding all facial expressions of emotion for all human observers.

Multimodal investigations of emotional face processing and social trait judgment of faces

Faces are among the most important visual stimuli that humans perceive in everyday life. While extensive literature has examined emotional processing and social evaluations of faces, most studies have examined either topic using unimodal approaches. In this review, we promote the use of multimodal cognitive neuroscience approaches to study these processes, using two lines of research as examples: ambiguity in facial expressions of emotion and social trait judgment of faces. In the first set of studies, we identified an event-related potential that signals emotion ambiguity using electroencephalography and we found convergent neural responses to emotion ambiguity using functional neuroimaging and single-neuron recordings. In the second set of studies, we discuss how different neuroimaging and personality-dimensional approaches together provide new insights into social trait judgments of faces. In both sets of studies, we provide an in-depth comparison between neurotypicals and people with autism spectrum disorder. We offer a computational account for the behavioral and neural markers of the different facial processing between the two groups. Finally, we suggest new practices for studying the emotional processing and social evaluations of faces. All data discussed in the case studies of this review are publicly available.

The inside out model of emotion recognition: how the shape of one's internal emotional landscape influences the recognition of others' emotions

Some people are exceptional at reading emotional expressions, while others struggle. Here we ask whether the way we experience emotion "on the inside" influences the way we expect emotions to be expressed in the "outside world" and subsequently our ability to read others' emotional expressions. Across multiple experiments, incorporating discovery and replication samples, we develop EmoMap (N = 20; N = 271) and ExpressionMap (N = 98; replication N = 193) to map adults' experiences of emotions and visual representations of others' emotions. Some individuals have modular maps, wherein emotional experiences and visual representations are consistent and distinct-anger looks and feels different from happiness, which looks and feels different from sadness. In contrast, others have experiences and representations that are variable and overlapping-anger, happiness, and sadness look and feel similar and are easily confused for one another. Here we illustrate an association between these maps: those with consistent and distinct experiences of emotion also have consistent and distinct visual representations of emotion. Finally (N = 193), we construct the Inside Out Model of Emotion Recognition, which explains 60.8% of the variance in emotion recognition and illuminates multiple pathways to emotion recognition difficulties. These findings have important implications for understanding the emotion recognition difficulties documented in numerous clinical populations.

Py-Feat: Python Facial Expression Analysis Toolbox

Studying facial expressions is a notoriously difficult endeavor. Recent advances in the field of affective computing have yielded impressive progress in automatically detecting facial expressions from pictures and videos. However, much of this work has yet to be widely disseminated in social science domains such as psychology. Current state-of-the-art models require considerable domain expertise that is not traditionally incorporated into social science training programs. Furthermore, there is a notable absence of user-friendly and open-source software that provides a comprehensive set of tools and functions that support facial expression research. In this paper, we introduce Py-Feat, an open-source Python toolbox that provides support for detecting, preprocessing, analyzing, and visualizing facial expression data. Py-Feat makes it easy for domain experts to disseminate and benchmark computer vision models and also for end users to quickly process, analyze, and visualize face expression data. We hope this platform will facilitate increased use of facial expression data in human behavior research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s42761-023-00191-4.

The Future of Affective Science: Introduction to the Special Issue

Modern affective science-the empirical study of emotional responding and affective experience-has been active for a half-century. The Future of Affective Science special issue considers the history of this field and proposes new directions for the decades ahead. Contributors represent diverse theoretical perspectives, methodological expertise, and domains of study, and the special issue includes both literature reviews and new empirical studies as illustrations. This introductory article synthesizes the contributions, articulating the broader context of the current status of our field and highlighting common themes across articles as well as gaps notable even in this special issue. Sections of the article address theoretical and conceptual issues, research methodology, the questions we ask, and translation of basic affective science to applied domains. We conclude that much has been learned from the first 50 years of affective science, and it is now time for new theories, new research questions, and innovative methods for the decades ahead.

Nonlocal contrast calculated by the second order visual mechanisms and its significance in identifying facial emotions

Background: Previously obtained results indicate that faces are / preattentively/ detected in the visual scene very fast, and information on facial expression is rapidly extracted at the lower levels of the visual system. At the same time different facial attributes make different contributions in facial expression recognition. However, it is known, among the preattentive mechanisms there are none that would be selective for certain facial features, such as eyes or mouth. The aim of our study was to identify a candidate for the role of such a mechanism. Our assumption was that the most informative areas of the image are those characterized by spatial heterogeneity, particularly with nonlocal contrast changes. These areas may be identified / in the human visual system/ by the second-order visual / mechanisms/ filters selective to contrast modulations of brightness gradients. Methods: We developed a software program imitating the operation of these / mechanisms/ filters and finding areas of contrast heterogeneity in the image. Using this program, we extracted areas with maximum, minimum and medium contrast modulation amplitudes from the initial face images, then we used these to make three variants of one and the same face. The faces were demonstrated to the observers along with other objects synthesized the same way. The participants had to identify faces and define facial emotional expressions. Results: It was found that the greater is the contrast modulation amplitude of the areas shaping the face, the more precisely the emotion is identified. Conclusions: The results suggest that areas with a greater increase in nonlocal contrast are more informative in facial images, and the second-order visual / mechanisms/ filters can claim the role of /filters/ elements that detect areas of interest, attract visual attention and are windows through which subsequent levels of visual processing receive valuable information.

Facial affect recognition in context in adults with and without TBI

Introduction

Several studies have reported impaired emotion recognition in adults with traumatic brain injury (TBI), but studies have two major design features that limit application of results to real-world contexts: (1) participants choose from among lists of basic emotions, rather than generating emotion labels, and (2) images are typically presented in isolation rather than in context. To address these limitations, we created an open-labeling task with faces shown alone or in real-life scenes, to more closely approximate how adults with TBI label facial emotions beyond the lab.

Methods

Participants were 55 adults (29 female) with moderate to severe TBI and 55 uninjured comparison peers, individually matched for race, sex, and age. Participants viewed 60 photographs of faces, either alone or in the pictured person's real-life context, and were asked what that person was feeling. We calculated the percent of responses that were standard forced-choice-task options, and also used sentiment intensity analysis to compare verbal responses between the two groups. We tracked eye movements for a subset of participants, to explore whether gaze duration or number of fixations helped explain any group differences in labels.

Results

Over 50% of responses in both groups were words other than basic emotions on standard affect tasks, highlighting the importance of eliciting open-ended responses. Valence of labels by participants with TBI was attenuated relative to valence of Comparison group labels, i.e., TBI group responses were less positive to positive images and the same was true for negative images, although the TBI group responses had higher lexical diversity. There were no significant differences in gaze duration or number of fixations between groups.

Discussion

Results revealed qualitative differences in affect labels between adults with and without TBI that would not have emerged on standard forced-choice tasks. Verbal differences did not appear to be attributable to differences in gaze patterns, leaving open the question of mechanisms of atypical affect processing in adults with TBI.

How nervous am I? How computer vision succeeds and humans fail in interpreting state anxiety from dynamic facial behaviour

For human interaction, it is important to understand what emotional state others are in. Especially the observation of faces aids us in putting behaviours into context and gives insight into emotions and mental states of others. Detecting whether someone is nervous, a form of state anxiety, is such an example as it reveals a person's familiarity and contentment with the circumstances. With recent developments in computer vision we developed behavioural nervousness models to show which time-varying facial cues reveal whether someone is nervous in an interview setting. The facial changes, reflecting a state of anxiety, led to more visual exposure and less chemosensory (taste and olfaction) exposure. However, experienced observers had difficulty picking up these changes and failed to detect nervousness levels accurately therewith. This study highlights humans' limited capacity in determining complex emotional states but at the same time provides an automated model that can assist us in achieving fair assessments of so far unexplored emotional states.

Autistic adults exhibit highly precise representations of others' emotions but a reduced influence of emotion representations on emotion recognition accuracy

To date, studies have not yet established the mechanisms underpinning differences in autistic and non-autistic emotion recognition. The current study first investigated whether autistic and non-autistic adults differed in terms of the precision and/or differentiation of their visual emotion representations and their general matching abilities, and second, explored whether differences therein were related to challenges in accurately recognizing emotional expressions. To fulfil these aims, 45 autistic and 45 non-autistic individuals completed three tasks employing dynamic point light displays of emotional facial expressions. We identified that autistic individuals had more precise visual emotion representations than their non-autistic counterparts, however, this did not confer any benefit for their emotion recognition. Whilst for non-autistic people, non-verbal reasoning and the interaction between precision of emotion representations and matching ability predicted emotion recognition, no variables contributed to autistic emotion recognition. These findings raise the possibility that autistic individuals are less guided by their emotion representations, thus lending support to Bayesian accounts of autism.

Emotion recognition of faces and emoji in individuals with moderate-severe traumatic brain injury

BACKGROUND

Facial emotion recognition deficits are common after moderate-severe traumatic brain injury (TBI) and linked to poor social outcomes. We examine whether emotion recognition deficits extend to facial expressions depicted by emoji.

METHODS

Fifty-one individuals with moderate-severe TBI (25 female) and fifty-one neurotypical peers (26 female) viewed photos of human faces and emoji. Participants selected the best-fitting label from a set of basic emotions (anger, disgust, fear, sadness, neutral, surprise, happy) or social emotions (embarrassed, remorseful, anxious, neutral, flirting, confident, proud).

RESULTS

We analyzed the likelihood of correctly labeling an emotion by group (neurotypical, TBI), stimulus condition (basic faces, basic emoji, social emoji), sex (female, male), and their interactions. Participants with TBI did not significantly differ from neurotypical peers in overall emotion labeling accuracy. Both groups had poorer labeling accuracy for emoji compared to faces. Participants with TBI (but not neurotypical peers) had poorer accuracy for labeling social emotions depicted by emoji compared to basic emotions depicted by emoji. There were no effects of participant sex.

DISCUSSION

Because emotion representation is more ambiguous in emoji than human faces, studying emoji use and perception in TBI is an important consideration for understanding functional communication and social participation after brain injury.

Approach-avoidance behavioural patterns towards an affective voice

Facial expressions are evolutionarily acquired social signals that are processed rapidly by the receiver when deciding to either approach or avoid the person sending the signal. The predominance of the approach or avoidance responses has been examined for various facial expressions. Similar to facial expressions, the approach-avoidance response to affective voice may have been acquired evolutionarily. However, studies examining approach-avoidance to an affective voice are limited. This study used the approach-avoidance task framework developed for facial expressions to determine the dominant behavioural responses to affective voices that indicate either happiness or anger. The results show that the approach response was dominant for the happy voice, while the avoidance response was dominant for the angry voice. This tendency was observed even without the presence of any facial stimuli. These results suggest that an affective voice may be an evolutionarily acquired approach-avoidance-inducing social signal, similar to facial expressions.

Visual perception of emotion cues in dogs: a critical review of methodologies

Comparative studies of human-dog cognition have grown exponentially since the 2000's, but the focus on how dogs look at us (as well as other dogs) as social partners is a more recent phenomenon despite its importance to human-dog interactions. Here, we briefly summarise the current state of research in visual perception of emotion cues in dogs and why this area is important; we then critically review its most commonly used methods, by discussing conceptual and methodological challenges and associated limitations in depth; finally, we suggest some possible solutions and recommend best practice for future research. Typically, most studies in this field have concentrated on facial emotional cues, with full body information rarely considered. There are many challenges in the way studies are conceptually designed (e.g., use of non-naturalistic stimuli) and the way researchers incorporate biases (e.g., anthropomorphism) into experimental designs, which may lead to problematic conclusions. However, technological and scientific advances offer the opportunity to gather much more valid, objective, and systematic data in this rapidly expanding field of study. Solving conceptual and methodological challenges in the field of emotion perception research in dogs will not only be beneficial in improving research in dog-human interactions, but also within the comparative psychology area, in which dogs are an important model species to study evolutionary processes.

A narrative review of the interconnection between pilot acute stress, startle, and surprise effects in the aviation context: Contribution of physiological measurements

Aviation remains one of the safest modes of transportation. However, an inappropriate response to an unexpected event can lead to flight incidents and accidents. Among several contributory factors, startle and surprise, which can lead to or exacerbate the pilot's state of stress, are often cited. Unlike stress, which has been the subject of much study in the context of driving and piloting, studies on startle and surprise are less numerous and these concepts are sometimes used interchangeably. Thus, the definitions of stress, startle, and surprise are reviewed, and related differences are put in evidence. Furthermore, it is proposed to distinguish these notions in the evaluation and to add physiological measures to subjective measures in their study. Indeed, Landman's theoretical model makes it possible to show the links between these concepts and studies using physiological parameters show that they would make it possible to disentangle the links between stress, startle and surprise in the context of aviation. Finally, we draw some perspectives to set up further studies focusing specifically on these concepts and their measurement.

Testing, explaining, and exploring models of facial expressions of emotions

Models are the hallmark of mature scientific inquiry. In psychology, this maturity has been reached in a pervasive question-what models best represent facial expressions of emotion? Several hypotheses propose different combinations of facial movements [action units (AUs)] as best representing the six basic emotions and four conversational signals across cultures. We developed a new framework to formalize such hypotheses as predictive models, compare their ability to predict human emotion categorizations in Western and East Asian cultures, explain the causal role of individual AUs, and explore updated, culture-accented models that improve performance by reducing a prevalent Western bias. Our predictive models also provide a noise ceiling to inform the explanatory power and limitations of different factors (e.g., AUs and individual differences). Thus, our framework provides a new approach to test models of social signals, explain their predictive power, and explore their optimization, with direct implications for theory development.

The roles of caregivers and friends in adolescent daily emotion dynamics

Daily emotion dynamics provide valuable information about individuals' emotion processes as they go about their lives. Emotion dynamics such as emotion levels (mean), emotion variability (degree of fluctuation), and emotion network density (strength of temporal connections among emotions) are associated with risks for various psychopathology in youth and adults. Prior work has shown that caregivers and friends play crucial socializing roles in adolescent emotional well-being, but less is known about their roles in daily emotion dynamics. This study examined whether caregiver emotion coaching, caregiver-adolescent closeness, and friendship quality were associated with adolescents' emotion levels, emotion variability, and emotion network density. Further, we examined whether caregiver-adolescent closeness moderated the associations between coaching and emotion dynamics. Participants were 150 adolescents (61% girls; Mage = 14.75) and one of their caregivers (95% female; Mage = 43.35) who completed a baseline survey and 21 daily surveys. Results showed that caregiver emotion coaching interacted with caregiver-adolescent closeness in predicting emotion levels and variability. Specifically, when closeness was higher, emotion coaching was significantly associated with lower sadness and anger levels, higher happiness levels, and lower happiness variability. Caregiver emotion coaching, independent of closeness, was also associated with lower anxiety levels, lower sadness variability, and lower emotion network density. Friendship quality was significantly associated with lower levels of sadness, anxiety, and anger, higher levels of happiness, and lower variability in anxiety and anger. These findings suggest that caregivers and friends are central to everyday emotion levels and variability and a more flexible emotion system in adolescents.

Enhanced recognition of disgusted expressions occurs in spite of attentional avoidance at encoding

Introduction

Negative emotional content is prioritized in memory. Prioritized attention to negative stimuli has been suggested to mediate this valence-memory association. However, research suggests only a limited role for attention in this observed memory advantage. We tested the role of attention in memory for disgusted facial expressions, a powerful social-emotional stimulus.

Methods

We measured attention using an incidental, free-viewing encoding task and memory using a surprise memory test for the viewed expressions.

Results and Discussion

Replicating prior studies, we found increased attentional dwell-time for neutral over disgusted expressions at encoding. However, contrary to the attention-memory link hypothesis, disgusted faces were better remembered than neutral faces. Although dwell-time was found to partially mediate the association between valence and memory, this effect was much weaker than the opposite direct effect. These findings point to independence of memory for disgusted faces from attention during encoding.

Amygdala connectivity and aggression

Neurobiological models propose that reactive aggression is predicated on impairments in amygdala-prefrontal connectivity that subserves moral decision-making and emotion regulation. The amygdala is a key component within this neural network that modulates reactive aggression. We provide a review of amygdala dysfunctional brain networks leading to reactive aggressive behavior. We elaborate on key concepts, focusing on moral decision-making and emotion regulation in a developmental context, and brain network connectivity factors relating to amygdala (dys)function-factors which we suggest predispose to reactive aggression. We additionally discuss insights into the latest treatment interventions, providing the utilization of the scientific findings for practice.

The McNorm library: creating and validating a new library of emotionally expressive whole body dance movements

The ability to exchange affective cues with others plays a key role in our ability to create and maintain meaningful social relationships. We express our emotions through a variety of socially salient cues, including facial expressions, the voice, and body movement. While significant advances have been made in our understanding of verbal and facial communication, to date, understanding of the role played by human body movement in our social interactions remains incomplete. To this end, here we describe the creation and validation of a new set of emotionally expressive whole-body dance movement stimuli, named the Motion Capture Norming (McNorm) Library, which was designed to reconcile a number of limitations associated with previous movement stimuli. This library comprises a series of point-light representations of a dancer's movements, which were performed to communicate to observers neutrality, happiness, sadness, anger, and fear. Based on results from two validation experiments, participants could reliably discriminate the intended emotion expressed in the clips in this stimulus set, with accuracy rates up to 60% (chance = 20%). We further explored the impact of dance experience and trait empathy on emotion recognition and found that neither significantly impacted emotion discrimination. As all materials for presenting and analysing this movement library are openly available, we hope this resource will aid other researchers in further exploration of affective communication expressed by human bodily movement.

Automatic recognition of schizophrenia from facial videos using 3D convolutional neural network

Schizophrenia affects patients and their families and society because of chronic impairments in cognition, behavior, and emotion. However, its clinical diagnosis mainly depends on the clinicians' knowledge of the patients' symptoms. Other auxiliary diagnostic methods such as MRI and EEG are cumbersome and time-consuming. Recently, the convolutional neural network (CNN) has been applied to the auxiliary diagnosis of psychiatry. Hence, in this study, a method based on deep learning and facial videos is proposed for the rapid detection of schizophrenia. Herein, 125 videos from 125 schizophrenic patients and 75 videos from 75 healthy controls based on emotional stimulation tasks were obtained. The video preprocessing included the experiment clips extraction, face detection, facial region cropping, resizing to 500 × 500 pixel size, and uniform sampling of 100 frames. The preprocessed facial videos were used to train the Resnet18_3D. We utilized ten-fold cross-validation, and held-out testing set to evaluate the model with the accuracy, the precision, the sensitivity, the specificity, the balanced accuracy, and the AUC. The Resnet18_3D trained on Film_order achieved the best performance with accuracy, sensitivity, specificity, balanced accuracy, and AUC of 89.00%, 96.80%, 76.00%, 86.40% and 0.9397. The neural network model indeed recognizes healthy controls and schizophrenic patients through the changes in the area of the face. The results show that facial video under emotional stimulation can be used to classify schizophrenic patients and help clinicians with diagnosis in the clinical environment. Among the different types of stimuli, the video stimuli with fixed emotional order showed the best classification performance.

How Are You Feeling? Interpretation of Emotions through Facial Expressions of People Wearing Different Personal Protective Equipment: An Observational Study

(1) Background: The perception of others’ emotions based on non-verbal cues, such as facial expressions, is fundamental for interpersonal communication and mutual support. Using personal protection equipment (PPE) in a work environment during the SAR-CoV-2 pandemic challenged health professionals’ ability to recognise emotions and expressions while wearing PPE. The working hypothesis of this study was that the increased limitation of facial visibility, due to the use of a personal protective device, would interfere with the perception of basic emotions in the participants. (2) Methods: Through a cross-sectional descriptive study, the present research aimed to analyse the identification of four basic emotions (happiness; sadness; fear/surprise; and disgust/anger) through three types of PPE (FFP2 respirator, protective overall and powered air-purifying respirator (PAPR)), by using 32 photographs. The study was conducted using volunteer participants who met the inclusion criteria (individuals older than 13 without cognitive limitations). Participants had to recognise the emotions of actors in photographs that were randomly displayed in an online form. (3) Results: In general, the 690 participants better recognised happiness and fear, independently of the PPE utilised. Women could better identify different emotions, along with university graduates and young and middle-aged adults. Emotional identification was at its worst when the participants wore protective overalls (5.42 ± 1.22), followed by the PAPR (5.83 ± 1.38); the best scores were obtained using the FFP2 masks (6.57 ± 1.20). Sadness was the least recognised emotion, regardless of age. (4) Conclusions: The personal protective devices interfere in the recognition of emotions, with the protective overalls having the greatest impact, and the FFP2 mask the least. The emotions that were best recognised were happiness and fear/surprise, while the least recognised emotion was sadness. Women were better at identifying emotions, as well as participants with higher education, and young and middle-aged adults.

Face masks inhibit facial cues for approachability and trustworthiness: an eyetracking study

Wearing face masks during the Covid-19 pandemic has undeniable benefits from our health perspective. However, the interpersonal costs on social interactions may have been underappreciated. Because masks obscure critical facial regions signaling approach/avoidance intent and social trust, this implies that facial inference of approachability and trustworthiness may be severely discounted. Here, in our eyetracking experiment, we show that people judged masked faces as less approachable and trustworthy. Further analyses showed that the attention directed towards the eye region relative to the mouth region mediated the effect on approachability, but not on trustworthiness. This is because for masked faces, with the mouth region obscured, visual attention is then automatically diverted away from the mouth and towards the eye region, which is an undiagnostic cue for judging a target's approachability. Together, these findings support that mask-wearing inhibits the critical facial cues needed for social judgements.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12144-022-03705-8.

Global diabetes burden: analysis of regional differences to improve diabetes care

INTRODUCTION

The current evaluation processes of the burden of diabetes are incomplete and subject to bias. This study aimed to identify regional differences in the diabetes burden on a universal level from the perspective of people with diabetes.

RESEARCH DESIGN AND METHODS

We developed a worldwide online diabetes observatory based on 34 million diabetes-related tweets from 172 countries covering 41 languages, spanning from 2017 to 2021. After translating all tweets to English, we used machine learning algorithms to remove institutional tweets and jokes, geolocate users, identify topics of interest and quantify associated sentiments and emotions across the seven World Bank regions.

RESULTS

We identified four topics of interest for people with diabetes (PWD) in the Middle East and North Africa and another 18 topics in North America. Topics related to glycemic control and food are shared among six regions of the world. These topics were mainly associated with sadness (35% and 39% on average compared with levels of sadness in other topics). We also revealed several region-specific concerns (eg, insulin pricing in North America or the burden of daily diabetes management in Europe and Central Asia).

CONCLUSIONS

The needs and concerns of PWD vary significantly worldwide, and the burden of diabetes is perceived differently. Our results will support better integration of these regional differences into diabetes programs to improve patient-centric diabetes research and care, focused on the most relevant concerns to enhance personalized medicine and self-management of PWD.

Enhanced Mirror Neuron Network Activity and Effective Connectivity during Live Interaction Among Female Subjects

Facial expressions are indispensable in daily human communication. Previous neuroimaging studies investigating facial expression processing have presented pre-recorded stimuli and lacked live face-to-face interaction. Our paradigm alternated between presentations of real-time model performance and pre-recorded videos of dynamic facial expressions to participants. Simultaneous functional magnetic resonance imaging (fMRI) and facial electromyography activity recordings, as well as post-scan valence and arousal ratings were acquired from 44 female participants. Live facial expressions enhanced the subjective valence and arousal ratings as well as facial muscular responses. Live performances showed greater engagement of the right posterior superior temporal sulcus (pSTS), right inferior frontal gyrus (IFG), right amygdala and right fusiform gyrus, and modulated the effective connectivity within the right mirror neuron system (IFG, pSTS, and right inferior parietal lobule). A support vector machine algorithm could classify multivoxel activation patterns in brain regions involved in dynamic facial expression processing in the mentalizing networks (anterior and posterior cingulate cortex). These results indicate that live social interaction modulates the activity and connectivity of the right mirror neuron system and enhances spontaneous mimicry, further facilitating emotional contagion.

Small Pupils Lead to Lower Judgements of a Person’s Characteristics for Exaggerated, but Not for Realistic Pupils

Our eyes convey information about a person. The pupils may provide information regarding our emotional states when presented along with different emotional expressions. We examined the effects of pupil size and vergence on inferring other people’s characteristics in neutral expression eyes. Pupil sizes were manipulated by overlaying black disks onto the pupils of the original eye images. The disk area was then changed to create small, medium, and large pupils. Vergence was simulated by shifting the medium-sized disks nasally in one eye. Pupil sizes were exaggerated for Experiment 1 and followed values from the literature for Experiment 2. The first Purkinje image from the eye photos in Experiment 2 was kept to preserve image realism. The characteristics measured were sex, age, attractiveness, trustworthiness, intelligence, valence, and arousal. Participants completed one of two online experiments and rated eight eye pictures with differently sized pupils and with vergence eyes. Both experiments were identical except for the stimuli designs. Results from Experiment 1 revealed rating differences between pupil sizes for all characteristics except sex, age, and arousal. Specifically, eyes with extremely small pupil sizes and artificial vergence received the lowest ratings compared to medium and large pupil sizes. Results from Experiment 2 only indicated weak effects of pupil size and vergence, particularly for intelligence ratings. We conclude that the pupils can influence how characteristics of another person are perceived and may be regarded as important social signals in subconscious social interaction processes. However, the effects may be rather small for neutral expressions.

The emotions effect on a virtual characters design–A student perspective analysis

Interaction between people and virtual characters through digital and electronic devices is a reality. In this context, the design of virtual characters must incorporate emotional expression at a nonverbal level looking for effective communication with the user. This exploratory study investigates the design features of an avatar functioning as a virtual assistant in educational contexts. From a multidisciplinary approach, the user's research was elaborated by a semi-open questionnaire of self-perception of emotional characteristics: likeability, attractiveness, and applicability of a set of six 2D and 3D characters. The results extracted from a sample of 69 university students provide a relevant information on design features and open new lines for future research. Aspects such as Ekman's basic emotion discrimination and the design of facial expression are analyzed. The incorporation of other body parts, their spatial orientation and contextual elements, seems to contribute to effective emotional communication. The results also highlight how the design of a virtual character should take into consideration the complexity involved in facial gestures and changes in relation to the vertical axis and planes of movement. Finally, this article discusses the complexity involved in expressing a given emotion in a virtual character.

The spatio-temporal features of perceived-as-genuine and deliberate expressions

Reading the genuineness of facial expressions is important for increasing the credibility of information conveyed by faces. However, it remains unclear which spatio-temporal characteristics of facial movements serve as critical cues to the perceived genuineness of facial expressions. This study focused on observable spatio-temporal differences between perceived-as-genuine and deliberate expressions of happiness and anger expressions. In this experiment, 89 Japanese participants were asked to judge the perceived genuineness of faces in videos showing happiness or anger expressions. To identify diagnostic facial cues to the perceived genuineness of the facial expressions, we analyzed a total of 128 face videos using an automated facial action detection system; thereby, moment-to-moment activations in facial action units were annotated, and nonnegative matrix factorization extracted sparse and meaningful components from all action units data. The results showed that genuineness judgments reduced when more spatial patterns were observed in facial expressions. As for the temporal features, the perceived-as-deliberate expressions of happiness generally had faster onsets to the peak than the perceived-as-genuine expressions of happiness. Moreover, opening the mouth negatively contributed to the perceived-as-genuine expressions, irrespective of the type of facial expressions. These findings provide the first evidence for dynamic facial cues to the perceived genuineness of happiness and anger expressions.

Transcranial Direct Current Stimulation Over the Right Temporal Parietal Junction Facilitates Spontaneous Micro-Expression Recognition

Micro-expressions are fleeting and subtle emotional expressions. As they are spontaneous and uncontrollable by one's mind, micro-expressions are considered an indicator of genuine emotions. Their accurate recognition and interpretation promote interpersonal interaction and social communication. Therefore, enhancing the ability to recognize micro-expressions has captured much attention. In the current study, we investigated the effects of training on micro-expression recognition with a Chinese version of the Micro-Expression Training Tool (METT). Our goal was to confirm whether the recognition accuracy of spontaneous micro-expressions could be improved through training and brain stimulation. Since the right temporal parietal junction (rTPJ) has been shown to be involved in the explicit process of facial emotion recognition, we hypothesized that the rTPJ would play a role in facilitating the recognition of micro-expressions. The results showed that anodal transcranial direct-current stimulation (tDCS) of the rTPJ indeed improved the recognition of spontaneous micro-expressions, especially for those associated with fear. The improved accuracy of recognizing fear spontaneous micro-expressions was positively correlated with personal distress in the anodal group but not in the sham group. Our study supports that the combined use of tDCS and METT can be a viable way to train and enhance micro-expression recognition.

Spatio-Temporal Properties of Amused, Embarrassed, and Pained Smiles

Smiles are universal but nuanced facial expressions that are most frequently used in face-to-face communications, typically indicating amusement but sometimes conveying negative emotions such as embarrassment and pain. Although previous studies have suggested that spatial and temporal properties could differ among these various types of smiles, no study has thoroughly analyzed these properties. This study aimed to clarify the spatiotemporal properties of smiles conveying amusement, embarrassment, and pain using a spontaneous facial behavior database. The results regarding spatial patterns revealed that pained smiles showed less eye constriction and more overall facial tension than amused smiles; no spatial differences were identified between embarrassed and amused smiles. Regarding temporal properties, embarrassed and pained smiles remained in a state of higher facial tension than amused smiles. Moreover, embarrassed smiles showed a more gradual change from tension states to the smile state than amused smiles, and pained smiles had lower probabilities of staying in or transitioning to the smile state compared to amused smiles. By comparing the spatiotemporal properties of these three smile types, this study revealed that the probability of transitioning between discrete states could help distinguish amused, embarrassed, and pained smiles.

The Effects of Separate Facial Areas on Emotion Recognition in Different Adult Age Groups: A Laboratory and a Naturalistic Study

The identification of facial expressions is critical for social interaction. The ability to recognize facial emotional expressions declines with age. These age effects have been associated with differential age-related looking patterns. The present research project set out to systematically test the role of specific facial areas for emotion recognition across the adult lifespan. Study 1 investigated the impact of displaying only separate facial areas versus the full face on emotion recognition in 62 younger (20-24 years) and 65 middle-aged adults (40-65 years). Study 2 examined if wearing face masks differentially compromises younger (18-33 years, N = 71) versus middle-aged to older adults' (51-83 years, N = 73) ability to identify different emotional expressions. Results of Study 1 suggested no general decrease in emotion recognition across the lifespan; instead, age-related performance seems to depend on the specific emotion and presented face area. Similarly, Study 2 observed only deficits in the identification of angry, fearful, and neutral expressions in older adults, but no age-related differences with regards to happy, sad, and disgusted expressions. Overall, face masks reduced participants' emotion recognition; however, there were no differential age effects. Results are discussed in light of current models of age-related changes in emotion recognition.

Measuring ethical behavior with AI and natural language processing to assess business success

Everybody claims to be ethical. However, there is a huge difference between declaring ethical behavior and living up to high ethical standards. In this paper, we demonstrate that “hidden honest signals” in the language and the use of “small words” can show true moral values and behavior of individuals and organizations and that this ethical behavior is correlated to real-world success; however not always in the direction we might expect. Leveraging the latest advances of AI in natural language processing (NLP), we construct three different “tribes” of ethical, moral, and non-ethical people, based on Twitter feeds of people of known high and low ethics and morals: fair and modest collaborators codified as ethical “bees”; hard-working competitive workers as moral “ants”; and selfish, arrogant people as non-ethical “leeches”. Results from three studies involving a total of 49 workgroups and 281 individuals within three different industries (healthcare, business consulting, and higher education) confirm the validity of our model. Associating membership in ethical or unethical tribes with performance, we find that being ethical correlates positively or negatively with success depending on the context.

Synthesizing a Talking Child Avatar to Train Interviewers Working with Maltreated Children

When responding to allegations of child sexual, physical, and psychological abuse, Child Protection Service (CPS) workers and police personnel need to elicit detailed and accurate accounts of the abuse to assist in decision-making and prosecution. Current research emphasizes the importance of the interviewer’s ability to follow empirically based guidelines. In doing so, it is essential to implement economical and scientific training courses for interviewers. Due to recent advances in artificial intelligence, we propose to generate a realistic and interactive child avatar, aiming to mimic a child. Our ongoing research involves the integration and interaction of different components with each other, including how to handle the language, auditory, emotional, and visual components of the avatar. This paper presents three subjective studies that investigate and compare various state-of-the-art methods for implementing multiple aspects of the child avatar. The first user study evaluates the whole system and shows that the system is well received by the expert and highlights the importance of its realism. The second user study investigates the emotional component and how it can be integrated with video and audio, and the third user study investigates realism in the auditory and visual components of the avatar created by different methods. The insights and feedback from these studies have contributed to the refined and improved architecture of the child avatar system which we present here.

Computational Process of Sharing Emotion: An Authentic Information Perspective

Although results of many psychology studies have shown that sharing emotion achieves dyadic interaction, no report has explained a study of the transmission of authentic information from emotional expressions that can strengthen perceivers. For this study, we used computational modeling, which is a multinomial processing tree, for formal quantification of the process of sharing emotion that emphasizes the perception of authentic information for expressers’ feeling states from facial expressions. Results indicated that the ability to perceive authentic information of feeling states from a happy expression has a higher probability than the probability of judging authentic information from anger expressions. Next, happy facial expressions can activate both emotional elicitation and sharing emotion in perceivers, where emotional elicitation alone is working rather than sharing emotion for angry facial expressions. Third, parameters to detect anger experiences were found to be correlated positively with those of happiness. No robust correlation was found between the parameters extracted from this experiment task and questionnaire-measured emotional contagion, empathy, and social anxiety. Results of this study revealed the possibility that a new computational approach contributes to description of emotion sharing processes.

Features and Extra-Striate Body Area Representations of Diagnostic Body Parts in Anger and Fear Perception

Social species perceive emotion via extracting diagnostic features of body movements. Although extensive studies have contributed to knowledge on how the entire body is used as context for decoding bodily expression, we know little about whether specific body parts (e.g., arms and legs) transmit enough information for body understanding. In this study, we performed behavioral experiments using the Bubbles paradigm on static body images to directly explore diagnostic body parts for categorizing angry, fearful and neutral expressions. Results showed that subjects recognized emotional bodies through diagnostic features from the torso with arms. We then conducted a follow-up functional magnetic resonance imaging (fMRI) experiment on body part images to examine whether diagnostic parts modulated body-related brain activity and corresponding neural representations. We found greater activations of the extra-striate body area (EBA) in response to both anger and fear than neutral for the torso and arms. Representational similarity analysis showed that neural patterns of the EBA distinguished different bodily expressions. Furthermore, the torso with arms and whole body had higher similarities in EBA representations relative to the legs and whole body, and to the head and whole body. Taken together, these results indicate that diagnostic body parts (i.e., torso with arms) can communicate bodily expression in a detectable manner.

The Compositionality of Facial Expressions

The principle of compositionality, an important postulation in language and cognition research, posits that the meaning of a complex expression is determined by the meaning of its constituting parts and the operation performed on those parts. Here, we provide strong evidence that this principle plays a significant role also in interpreting facial expressions. In three studies in which perceivers interpreted sequences of two emotional facial expression images, we show that the composite meaning of facial expressions results from the meaning of its constituting expressions and an algebraic operation performed on those expressions. Our study offers a systematic account as to how the meaning of facial expressions (single and sequences) are being formed and perceived. In a broader context, our results raise the possibility that the principle of compositionality may apply to human communication modalities beyond spoken language, whereby a minimal number of components are expanded to a much greater number of meanings.