Body cues, not facial expressions, discriminate between intense positive and negative emotions

Neuroanatomical markers of social cognition in neglected adolescents

Growing up in neglectful households can impact multiple aspects of social cognition. However, research on neglect's effects on social cognition processes and their neuroanatomical correlates during adolescence is scarce. Here, we aimed to comprehensively assess social cognition processes (recognition of basic and contextual emotions, theory of mind, the experience of envy and Schadenfreude and empathy for pain) and their structural brain correlates in adolescents with legal neglect records within family-based care. First, we compared neglected adolescents (n = 27) with control participants (n = 25) on context-sensitive social cognition tasks while controlling for physical and emotional abuse and executive and intellectual functioning. Additionally, we explored the grey matter correlates of these domains through voxel-based morphometry. Compared to controls, neglected adolescents exhibited lower performance in contextual emotional recognition and theory of mind, higher levels of envy and Schadenfreude and diminished empathy. Physical and emotional abuse and executive or intellectual functioning did not explain these effects. Moreover, social cognition scores correlated with brain volumes in regions subserving social cognition and emotional processing. Our results underscore the potential impact of neglect on different aspects of social cognition during adolescence, emphasizing the necessity for preventive and intervention strategies to address these deficits in this population.

EMOKINE: A software package and computational framework for scaling up the creation of highly controlled emotional full-body movement datasets

EMOKINE is a software package and dataset creation suite for emotional full-body movement research in experimental psychology, affective neuroscience, and computer vision. A computational framework, comprehensive instructions, a pilot dataset, observer ratings, and kinematic feature extraction code are provided to facilitate future dataset creations at scale. In addition, the EMOKINE framework outlines how complex sequences of movements may advance emotion research. Traditionally, often emotional-'action'-based stimuli are used in such research, like hand-waving or walking motions. Here instead, a pilot dataset is provided with short dance choreographies, repeated several times by a dancer who expressed different emotional intentions at each repetition: anger, contentment, fear, joy, neutrality, and sadness. The dataset was simultaneously filmed professionally, and recorded using XSENS® motion capture technology (17 sensors, 240 frames/second). Thirty-two statistics from 12 kinematic features were extracted offline, for the first time in one single dataset: speed, acceleration, angular speed, angular acceleration, limb contraction, distance to center of mass, quantity of motion, dimensionless jerk (integral), head angle (with regards to vertical axis and to back), and space (convex hull 2D and 3D). Average, median absolute deviation (MAD), and maximum value were computed as applicable. The EMOKINE software is appliable to other motion-capture systems and is openly available on the Zenodo Repository. Releases on GitHub include: (i) the code to extract the 32 statistics, (ii) a rigging plugin for Python for MVNX file-conversion to Blender format (MVNX=output file XSENS® system), and (iii) a Python-script-powered custom software to assist with blurring faces; latter two under GPLv3 licenses.

The elephant in the zoom: will psychoanalysis survive the screen?

While screen-mediated analysis long predated the pandemic, it was largely seen as non-equivalent to in-person treatment by analysts and patients alike. When COVID forced us to move our entire practices to the screen, our concerns about its limitations were replaced by relief; we could continue doing analytic work during a terrifying and challenging time. Three years later, many have chosen to continue practicing remotely for reasons that are no longer driven by fears of exposure. We mostly minimize or deny our earlier concerns about the limitations of screen work. Have we chosen convenience, ease, and a personal sense of safety over togetherness, while ignoring the underbelly of remote work? This paper identifies the convergence of several forces underlying our decision to stay remote, including guilt and anxiety about privileging our own self-interest, unmourned losses and collective PTSD, fear of the future and existential anxiety about living in a techno-culture that threatens to replace us. Our denial of these powerful forces makes it easy to rationalize a decision to embrace remote work and disavow the threat it poses to our field.

The priming effect of emotional words on body expressions: Two ERP studies

The impact of emotional words on the recognition of body expression and the underlying neurodynamic mechanisms remain poorly understood. This study used a classic supraliminal priming paradigm and event related potential (ERP) to investigate the effect of emotion-label words (experiment 1) and emotional verbs (experiment 2) on the recognition of body expressions. The behavioral results revealed that individuals exhibited a higher accuracy in recognizing happy expressions when presented with a happy-label word condition, in contrast to neutral expressions. Furthermore, it was observed that the accuracy of recognizing happy body expressions was reduced when preceded by angry verb priming, compared to happy and neutral priming conditions. Conversely, the accuracy of recognizing angry body expressions was higher in response to angry verb priming than happy and neutral primings. The ERP results showed that, in the recognition of happy body expressions, the P300 amplitude elicited by angry-label words was more positive, while a congruent verb-expression condition elicited more positive P300 amplitude than an incongruent condition in the left hemisphere and midline. However, in the recognition of angry body expressions, the N400 amplitude elicited by a congruent verb-expression condition was smaller than that elicited by an incongruent condition. These results suggest that both abstract emotion-label words and specific emotional verbs influence the recognition of body expressions. In addition, integrating happy semantic context and body expression might occur at the P300 stage, whereas integrating angry semantic context and body expression might occur at the N400 stage. These findings present novel evidence regarding the criticality of emotional context in the recognition of emotions.

Perception of emotionally incongruent cues: evidence for overreliance on body vs. face expressions in Parkinson's disease

Individuals with Parkinson's disease (PD) may exhibit impaired emotion perception. However, research demonstrating this decline has been based almost entirely on the recognition of isolated emotional cues. In real life, emotional cues such as expressive faces are typically encountered alongside expressive bodies. The current study investigated emotion perception in individuals with PD (n = 37) using emotionally incongruent composite displays of facial and body expressions, as well as isolated face and body expressions, and congruent composite displays as a baseline. In addition to a group of healthy controls (HC) (n = 50), we also included control individuals with schizophrenia (SZ) (n = 30), who display, as in PD, similar motor symptomology and decreased emotion perception abilities. The results show that individuals with PD showed an increased tendency to categorize incongruent face-body combinations in line with the body emotion, whereas those with HC showed a tendency to classify them in line with the facial emotion. No consistent pattern for prioritizing the face or body was found in individuals with SZ. These results were not explained by the emotional recognition of the isolated cues, cognitive status, depression, or motor symptoms of individuals with PD and SZ. As real-life expressions may include inconsistent cues in the body and face, these findings may have implications for the way individuals with PD and SZ interpret the emotions of others.

The Chinese customers and service staff interactive affective system (CCSIAS): introduction to a multimodal stimulus dataset

To research the emotional interaction between customers and service staff, single-modal stimuli are being used to activate subjects' emotions while multimodal emotion stimuli with better efficiency are often neglected. This study aims to construct a multimodal emotion stimuli database (CCSIAS) with video records of real work status of 29 service staff and audio clips of interactions between customers and service staff by setting up wide-angle cameras and searching in company's Ocean Engine for 15 consecutive days. First, we developed a tool to assess the emotional statuses of customers and service staff in Study 1. Second, 40 Masters and PhD students were invited to assess the audio and video data to evaluate the emotional states of customers and service staff in Study 2, using the tools developed in Study 1. Third, 118 participants were recruited to test the results from Study 2 to ensure the stability of the derived data. The results showed that 139 sets of stable emotional audio & video data were constructed (26 sets were high, 59 sets were medium and 54 sets were low). The amount of emotional information is important for the effective activation of participants' emotional states, and the degree of emotional activation of video data is significantly higher than that of the audio data. Overall, it was shown that the study of emotional interaction phenomena requires a multimodal dataset. The CCSIAS (https://osf.io/muc86/) can extend the depth and breadth of emotional interaction research and can be applied to different emotional states between customers and service staff activation in the fields of organizational behavior and psychology.

Face and context integration in emotion inference is limited and variable across categories and individuals

The ability to make nuanced inferences about other people's emotional states is central to social functioning. While emotion inferences can be sensitive to both facial movements and the situational context that they occur in, relatively little is understood about when these two sources of information are integrated across emotion categories and individuals. In a series of studies, we use one archival and five empirical datasets to demonstrate that people could be integrating, but that emotion inferences are just as well (and sometimes better) captured by knowledge of the situation alone, while isolated facial cues are insufficient. Further, people integrate facial cues more for categories for which they most frequently encounter facial expressions in everyday life (e.g., happiness). People are also moderately stable over time in their reliance on situational cues and integration of cues and those who reliably utilize situation cues more also have better situated emotion knowledge. These findings underscore the importance of studying variability in reliance on and integration of cues.

Facial feedback manipulation influences the automatic detection of unexpected emotional body expressions

Unexpected or changing facial expressions are known to be able to engage more automatic processing than frequently occurring facial expressions, thereby inducing a neural differential wave response known as expression mismatch negativity (EMMN). Recent studies have shown that EMMN can be modulated by the observer's facial feedback (i.e., feedback from their own facial movements). A similar EMMN activity has been discovered for body expressions, but thus far only a few emotion types have been investigated. It is unknown whether the EMMNs evoked by body expressions can be influenced by facial feedback. To explore this question, we recorded EEG activity of 29 participants in the reverse oddball paradigm. Here two unexamined categories of body expressions were presented, happy and sad, placed in two paired stimulus sequences: in one the happy body was presented with a probability of 80% (standards) while the sad body was presented with a probability of 20% (deviants), and in the other the probabilities were reversed. The facial feedback was manipulated by different pen holding conditions (i.e., participants holding the pen with the teeth, lips, or nondominant hand). The nonparametric cluster permutation test revealed significant happy and sad body-related EMMN (bEMMN) activities. The happy-bEMMN were more negative than sad-bEMMN within the range of 100-150 ms. Additionally, the bEMMN amplitude of both emotions is modulated by the facial feedback conditions. These results expand the range of emotional types applicable to bEMMN and provide evidence for the validity of the facial feedback hypothesis across emotional carriers.

Hierarchical Context-Based Emotion Recognition With Scene Graphs

For a better intention inference, we often try to figure out the emotional states of other people in social communications. Many studies on affective computing have been carried out to infer emotions through perceiving human states, i.e., facial expression and body posture. Such methods are skillful in a controlled environment. However, it often leads to misestimation due to the deficiency of effective inputs in unconstrained circumstances, that is, where context-aware emotion recognition appeared. We take inspiration from the advanced reasoning pattern of humans in perceived emotion recognition and propose the hierarchical context-based emotion recognition method with scene graphs. We propose to extract three contexts from the image, i.e., the entity context, the global context, and the scene context. The scene context contains abstract information about entity labels and their relationships. It is similar to the information processing of the human visual sensing mechanism. After that, these contexts are further fused to perform emotion recognition. We carried out a bunch of experiments on the widely used context-aware emotion datasets, i.e., CAER-S, EMOTIC, and BOdy Language Dataset (BoLD). We demonstrate that the hierarchical contexts can benefit emotion recognition by improving the accuracy of the SOTA score from 84.82% to 90.83% on CAER-S. The ablation experiments show that hierarchical contexts provide complementary information. Our method improves the F1 score of the SOTA result from 29.33% to 30.24% (C-F1) on EMOTIC. We also build the image-based emotion recognition task with BoLD-Img from BoLD and obtain a better emotion recognition score (ERS) score of 0.2153.

Guarding and flow in the movements of people with chronic pain: A qualitative study of physiotherapists' observations

BACKGROUND

Among the adaptations of movement consistently associated with disability in chronic pain, guarding is common. Based on previous work, we sought to understand better the constituents of guarding; we also used the concept of flow to explore the description of un/naturalness that emerged from physiotherapists' descriptions of movement in chronic pain. The aim was to inform the design of technical systems to support people with chronic pain in everyday activities.

METHODS

Sixteen physiotherapists, experts in chronic pain, were interviewed while repeatedly watching short video clips of people with chronic low back pain doing simple movements; physiotherapists described the movements, particularly in relation to guarding and flow. The transcribed interviews were analysed thematically to elaborate these constructs.

RESULTS

Moderate agreement emerged on the extent of guarding in the videos, with good agreement that guarding conveyed caution about movement, distinct from biomechanical variables of stiffness or slow speed. Physiotherapists' comments on flow showed slightly better agreement, and described the overall movement in terms of restriction (where there was no flow or only some flow), of tempo of the entire movement, and as naturalness (distinguished from normality of movement).

CONCLUSIONS

These qualities of movement may be useful in designing technical systems to support self-management of chronic pain.

SIGNIFICANCE

Drawing on the descriptions of movements of people with chronic low back pain provided by expert physiotherapists to standard stimuli, two key concepts were elaborated. Guarding was distinguished from stiffness (a physical limitation) or slowness as motivated by fear or worry about movement. Flow served to describe harmonious and continuous movement, even when adapted around restrictions of pain. Movement behaviours associated with pain are better understood in terms of their particular function than aggregated without reference to function.

Four fundamental dimensions underlie the perception of human actions

We evaluate the actions of other individuals based upon a variety of movements that reveal critical information to guide decision making and behavioural responses. These signals convey a range of information about the actor, including their goals, intentions and internal mental states. Although progress has been made to identify cortical regions involved in action processing, the organising principles underlying our representation of actions still remains unclear. In this paper we investigated the conceptual space that underlies action perception by assessing which qualities are fundamental to the perception of human actions. We recorded 240 different actions using motion-capture and used these data to animate a volumetric avatar that performed the different actions. 230 participants then viewed these actions and rated the extent to which each action demonstrated 23 different action characteristics (e.g., avoiding-approaching, pulling-pushing, weak-powerful). We analysed these data using Exploratory Factor Analysis to examine the latent factors underlying visual action perception. The best fitting model was a four-dimensional model with oblique rotation. We named the factors: friendly-unfriendly, formidable-feeble, planned-unplanned, and abduction-adduction. The first two factors of friendliness and formidableness explained approximately 22% of the variance each, compared to planned and abduction, which explained approximately 7-8% of the variance each; as such we interpret this representation of action space as having 2 + 2 dimensions. A closer examination of the first two factors suggests a similarity to the principal factors underlying our evaluation of facial traits and emotions, whilst the last two factors of planning and abduction appear unique to actions.

A multi-modality and multi-dyad approach to measuring flexibility in psychotherapy

INTRODUCTION

Flexibility, the ability of an individual to adapt to environmental changes in ways that facilitate goal attainment, has been proposed as a potential mechanism underlying psychopathology and psychotherapy. In psychotherapy, most findings are based on self-report measures that have important limitations. We propose a multimodal, multi-dyad approach based on a nonlinear dynamical systems framework to capture the complexity of this concept.

METHOD

A new research paradigm was designed to explore the validity of the proposed conceptual model. The paradigm includes a psychotherapy-like social interaction, during which body movement and facial expressiveness data were collected. We analyzed the data using Hankel Alternative View of Koopmann analysis to reconstruct attractors of the observed behaviors and compare them.

RESULTS

The patterns of behavior in the two cases differ, and differences in the reconstructed attractors correspond with differences in self-report measures and behavior in the interactions.

CONCLUSIONS

The case studies show that information provided by a single modality is not enough to provide the full picture, and multiple modalities are needed. These observations can serve as an initial support for our claims that a multi-modal and multi-dyad approach to flexibility can address some of the issues of measurement in the field.

The Relative Contribution of Facial and Body Information to the Perception of Cuteness

Faces and bodies both provide cues to age and cuteness, but little work has explored their interaction in cuteness perception. This study examines the interplay of facial and bodily cues in the perception of cuteness, particularly when these cues convey conflicting age information. Participants rated the cuteness of face-body composites that combined either a child or adult face with an age-congruent or incongruent body alongside manipulations of the head-to-body height ratio (HBR). The findings from two experiments indicated that child-like facial features enhanced the perceived cuteness of adult bodies, while child-like bodily features generally had negative impacts. Furthermore, the results showed that an increased head size significantly boosted the perceived cuteness for child faces more than for adult faces. Lastly, the influence of the HBR was more pronounced when the outline of a body's silhouette was the only available information compared to when detailed facial and bodily features were presented. This study suggests that body proportion information, derived from the body's outline, and facial and bodily features, derived from the interior surface, are integrated to form a unitary representation of a whole person in cuteness perception. Our findings highlight the dominance of facial features over bodily information in cuteness perception, with facial attributes serving as key references for evaluating face-body relationships and body proportions. This research offers significant insights into social cognition and character design, particularly in how people perceive entities with mixed features of different social categories, underlining the importance of congruency in perceptual elements.

Development, refinement, and validation of an equine musculoskeletal pain scale

Musculoskeletal disease is a common cause of chronic pain that is often overlooked and inadequately treated, impacting the quality of life of humans and horses alike. Lameness due to musculoskeletal pain is prevalent in horses, but the perception of pain by owners is low compared with veterinary diagnosis. Therefore, this study aims to establish and validate a pain scale for chronic equine orthopaedic pain that is user-friendly for horse owners and veterinarians to facilitate the identification and monitoring of pain in horses. The newly developed musculoskeletal pain scale (MPS) was applied to 154 horses (mean age 20 ± 6.4 years SD) housed at an equine sanctuary, of which 128 (83%) suffered from chronic orthopaedic disease. To complete the MPS, the horses were observed and videotaped from a distance while at rest in their box or enclosure. In addition, they received a complete clinical and orthopaedic exam. The need for veterinary intervention to address pain (assessed and executed by the sanctuary independent from this study) was used as a longitudinal health outcome to determine the MPS's predictive validity. To determine the interrater agreement, the MPS was scored for a randomly selected subset of 30 horses by six additional blinded raters, three equine veterinary practitioners, and three experienced equestrians. An iterative process was used to refine the tool based on improvements in the MPS's correlation with lameness evaluated at the walk and trot, predictive validity for longitudinal health outcomes, and interrater agreement. The intraclass correlation improved from 0.77 of the original MPS to 0.88 of the refined version (95% confidence interval: 0.8-0.94). The refined MPS correlated significantly with lameness at the walk (r = 0.44, p = 0.001) and trot (r = 0.5, p < 0.0001). The refined MPS significantly differed between horses that needed veterinary intervention (mean MPS = 8.6) and those that did not (mean MPS = 5.0, p = 0.0007). In summary, the MPS showed good interrater repeatability between expert and lay scorers, significant correlation with lameness at the walk and trot, and good predictive validity for longitudinal health outcomes, confirming its ability to identify horses with orthopaedic health problems.

Cultural facial expressions dynamically convey emotion category and intensity information

Communicating emotional intensity plays a vital ecological role because it provides valuable information about the nature and likelihood of the sender's behavior.1,2,3 For example, attack often follows signals of intense aggression if receivers fail to retreat.4,5 Humans regularly use facial expressions to communicate such information.6,7,8,9,10,11 Yet how this complex signaling task is achieved remains unknown. We addressed this question using a perception-based, data-driven method to mathematically model the specific facial movements that receivers use to classify the six basic emotions-"happy," "surprise," "fear," "disgust," "anger," and "sad"-and judge their intensity in two distinct cultures (East Asian, Western European; total n = 120). In both cultures, receivers expected facial expressions to dynamically represent emotion category and intensity information over time, using a multi-component compositional signaling structure. Specifically, emotion intensifiers peaked earlier or later than emotion classifiers and represented intensity using amplitude variations. Emotion intensifiers are also more similar across emotions than classifiers are, suggesting a latent broad-plus-specific signaling structure. Cross-cultural analysis further revealed similarities and differences in expectations that could impact cross-cultural communication. Specifically, East Asian and Western European receivers have similar expectations about which facial movements represent high intensity for threat-related emotions, such as "anger," "disgust," and "fear," but differ on those that represent low threat emotions, such as happiness and sadness. Together, our results provide new insights into the intricate processes by which facial expressions can achieve complex dynamic signaling tasks by revealing the rich information embedded in facial expressions.

Multimodal investigations of emotional face processing and social trait judgment of faces

Faces are among the most important visual stimuli that humans perceive in everyday life. While extensive literature has examined emotional processing and social evaluations of faces, most studies have examined either topic using unimodal approaches. In this review, we promote the use of multimodal cognitive neuroscience approaches to study these processes, using two lines of research as examples: ambiguity in facial expressions of emotion and social trait judgment of faces. In the first set of studies, we identified an event-related potential that signals emotion ambiguity using electroencephalography and we found convergent neural responses to emotion ambiguity using functional neuroimaging and single-neuron recordings. In the second set of studies, we discuss how different neuroimaging and personality-dimensional approaches together provide new insights into social trait judgments of faces. In both sets of studies, we provide an in-depth comparison between neurotypicals and people with autism spectrum disorder. We offer a computational account for the behavioral and neural markers of the different facial processing between the two groups. Finally, we suggest new practices for studying the emotional processing and social evaluations of faces. All data discussed in the case studies of this review are publicly available.

Infrequent facial expressions of emotion do not bias attention

Despite the obvious importance of facial expressions of emotion, most studies have found that they do not bias attention. A critical limitation, however, is that these studies generally present face distractors on all trials of the experiment. For other kinds of emotional stimuli, such as emotional scenes, infrequently presented stimuli elicit greater attentional bias than frequently presented stimuli, perhaps due to suppression or habituation. The goal of the current study then was to test whether such modulation of attentional bias by distractor frequency generalizes to facial expressions of emotion. In Experiment 1, both angry and happy faces were unable to bias attention, despite being infrequently presented. Even when the location of these face cues were more unpredictable-presented in one of two possible locations-still no attentional bias was observed (Experiment 2). Moreover, there was no bottom-up influence for angry and happy faces shown under high or low perceptual load (Experiment 3). We conclude that task-irrelevant posed facial expressions of emotion cannot bias attention even when presented infrequently.

Bodily expressed emotion understanding through integrating Laban movement analysis

Bodily expressed emotion understanding (BEEU) aims to automatically recognize human emotional expressions from body movements. Psychological research has demonstrated that people often move using specific motor elements to convey emotions. This work takes three steps to integrate human motor elements to study BEEU. First, we introduce BoME (body motor elements), a highly precise dataset for human motor elements. Second, we apply baseline models to estimate these elements on BoME, showing that deep learning methods are capable of learning effective representations of human movement. Finally, we propose a dual-source solution to enhance the BEEU model with the BoME dataset, which trains with both motor element and emotion labels and simultaneously produces predictions for both. Through experiments on the BoLD in-the-wild emotion understanding benchmark, we showcase the significant benefit of our approach. These results may inspire further research utilizing human motor elements for emotion understanding and mental health analysis.

Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion: Drawing Insights From Psychology, Engineering, and the Arts, This Article Provides a Comprehensive Overview of the Field of Emotion Analysis in Visual Media and Discusses the Latest Research, Systems, Challenges, Ethical Implications, and Potential Impact of Artificial Emotional Intelligence on Society

The emergence of artificial emotional intelligence technology is revolutionizing the fields of computers and robotics, allowing for a new level of communication and understanding of human behavior that was once thought impossible. While recent advancements in deep learning have transformed the field of computer vision, automated understanding of evoked or expressed emotions in visual media remains in its infancy. This foundering stems from the absence of a universally accepted definition of "emotion," coupled with the inherently subjective nature of emotions and their intricate nuances. In this article, we provide a comprehensive, multidisciplinary overview of the field of emotion analysis in visual media, drawing on insights from psychology, engineering, and the arts. We begin by exploring the psychological foundations of emotion and the computational principles that underpin the understanding of emotions from images and videos. We then review the latest research and systems within the field, accentuating the most promising approaches. We also discuss the current technological challenges and limitations of emotion analysis, underscoring the necessity for continued investigation and innovation. We contend that this represents a "Holy Grail" research problem in computing and delineate pivotal directions for future inquiry. Finally, we examine the ethical ramifications of emotion-understanding technologies and contemplate their potential societal impacts. Overall, this article endeavors to equip readers with a deeper understanding of the domain of emotion analysis in visual media and to inspire further research and development in this captivating and rapidly evolving field.

Deepfake smiles matter less-the psychological and neural impact of presumed AI-generated faces

High-quality AI-generated portraits ("deepfakes") are becoming increasingly prevalent. Understanding the responses they evoke in perceivers is crucial in assessing their societal implications. Here we investigate the impact of the belief that depicted persons are real or deepfakes on psychological and neural measures of human face perception. Using EEG, we tracked participants' (N = 30) brain responses to real faces showing positive, neutral, and negative expressions, after being informed that they are either real or fake. Smiling faces marked as fake appeared less positive, as reflected in expression ratings, and induced slower evaluations. Whereas presumed real smiles elicited canonical emotion effects with differences relative to neutral faces in the P1 and N170 components (markers of early visual perception) and in the EPN component (indicative of reflexive emotional processing), presumed deepfake smiles showed none of these effects. Additionally, only smiles presumed as fake showed enhanced LPP activity compared to neutral faces, suggesting more effortful evaluation. Negative expressions induced typical emotion effects, whether considered real or fake. Our findings demonstrate a dampening effect on perceptual, emotional, and evaluative processing of presumed deepfake smiles, but not angry expressions, adding new specificity to the debate on the societal impact of AI-generated content.

Advancing Naturalistic Affective Science with Deep Learning

People express their own emotions and perceive others' emotions via a variety of channels, including facial movements, body gestures, vocal prosody, and language. Studying these channels of affective behavior offers insight into both the experience and perception of emotion. Prior research has predominantly focused on studying individual channels of affective behavior in isolation using tightly controlled, non-naturalistic experiments. This approach limits our understanding of emotion in more naturalistic contexts where different channels of information tend to interact. Traditional methods struggle to address this limitation: manually annotating behavior is time-consuming, making it infeasible to do at large scale; manually selecting and manipulating stimuli based on hypotheses may neglect unanticipated features, potentially generating biased conclusions; and common linear modeling approaches cannot fully capture the complex, nonlinear, and interactive nature of real-life affective processes. In this methodology review, we describe how deep learning can be applied to address these challenges to advance a more naturalistic affective science. First, we describe current practices in affective research and explain why existing methods face challenges in revealing a more naturalistic understanding of emotion. Second, we introduce deep learning approaches and explain how they can be applied to tackle three main challenges: quantifying naturalistic behaviors, selecting and manipulating naturalistic stimuli, and modeling naturalistic affective processes. Finally, we describe the limitations of these deep learning methods, and how these limitations might be avoided or mitigated. By detailing the promise and the peril of deep learning, this review aims to pave the way for a more naturalistic affective science.

Incongruent gestures slow the processing of facial expressions in university students with social anxiety

In recent years, an increasing number of studies have examined the mechanisms underlying nonverbal emotional information processing in people with high social anxiety (HSA). However, most of these studies have focused on the processing of facial expressions, and there has been scarce research on gesture or even face-gesture combined processing in HSA individuals. The present study explored the processing characteristics and mechanism of the interaction between gestures and facial expressions in people with HSA and low social anxiety (LSA). The present study recruited university students as participants and used the Liebowitz Social Anxiety Scale scores to distinguish the HSA and LSA groups. We used a 2 (group: HSA and LSA) × 2 (emotion valence: positive, negative) × 2 (task: face, gesture) multifactor mixed design, and videos of a single face or gesture and combined face-gesture cues were used as stimuli. We found that (1) there is a distinction in the processing of faces and gestures, with individuals recognizing gestures faster than faces; (2) there is an attentional enhancement in the processing of gestures, particularly for negative gestures; and (3) when the emotional valence of faces and gestures align, it facilitates the recognition of both. However, incongruent gestures have a stronger impact on the processing of facial expressions compared to facial expressions themselves, suggesting that the processing of facial emotions is more influenced by environmental cues provided by gestures. These findings indicated that gestures played an important role in emotional processing, and facial emotional processing was more dependent on the environmental cues derived from gestures, which helps to clarify the reasons for biases in the interpretation of emotional information in people with HSA.

The background assimilation effect: Facial emotional perception is affected by surrounding stimuli

The perception of facial emotion is not only determined by the physical features of the face itself but also be influenced by the emotional information of the background or surrounding information. However, the details of such effect are not fully understood. Here, the authors tested the perceived emotion of a target face surrounded by stimuli with different levels of emotional valence. In Experiment 1, four types of objects were divided into three groups (negative, unpleasant flowers and unpleasant animals; mildly negative (neutral), houses; positive, pleasant flowers). In Experiment 2, three groups of surrounding faces with different social-emotional valence (negative, neutral, and positive) were formed with the memory of affective personal knowledge. The data from two experiments showed that the perception of facial emotion can be influenced and modulated by the emotional valence of the surrounding stimuli, which can be explained by assimilation: the positive stimuli increased the valence of a target face, while the negative stimuli comparatively decreased it. Furthermore, the neutral stimuli also increased the valence of the target, which could be explained by the social positive effect. Therefore, the process of assimilation is likely to be a high-level emotional cognition rather than a low-level visual perception. The results of this study may help us better understand face perception in realistic scenarios.

Functionally analogous body- and animacy-responsive areas are present in the dog (Canis familiaris) and human occipito-temporal lobe

Comparing the neural correlates of socio-cognitive skills across species provides insights into the evolution of the social brain and has revealed face- and body-sensitive regions in the primate temporal lobe. Although from a different lineage, dogs share convergent visuo-cognitive skills with humans and a temporal lobe which evolved independently in carnivorans. We investigated the neural correlates of face and body perception in dogs (N = 15) and humans (N = 40) using functional MRI. Combining univariate and multivariate analysis approaches, we found functionally analogous occipito-temporal regions involved in the perception of animate entities and bodies in both species and face-sensitive regions in humans. Though unpredicted, we also observed neural representations of faces compared to inanimate objects, and dog compared to human bodies in dog olfactory regions. These findings shed light on the evolutionary foundations of human and dog social cognition and the predominant role of the temporal lobe.

Recognition of emotional body language from dyadic and monadic point-light displays in 5-year-old children and adults

Most child studies on emotion perception used faces and speech as emotion stimuli, but little is known about children's perception of emotions conveyed by body movements, that is, emotional body language (EBL). This study aimed to investigate whether processing advantages for positive emotions in children and negative emotions in adults found in studies on emotional face and term perception also occur in EBL perception. We also aimed to uncover which specific movement features of EBL contribute to emotion perception from interactive dyads compared with noninteractive monads in children and adults. We asked 5-year-old children and adults to categorize happy and angry point-light displays (PLDs), presented as pairs (dyads) and single actors (monads), in a button-press task. By applying representational similarity analyses, we determined intra- and interpersonal movement features of the PLDs and their relation to the participants' emotional categorizations. Results showed significantly higher recognition of happy PLDs in 5-year-olds and of angry PLDs in adults in monads but not in dyads. In both age groups, emotion recognition depended significantly on kinematic and postural movement features such as limb contraction and vertical movement in monads and dyads, whereas in dyads recognition also relied on interpersonal proximity measures such as interpersonal distance. Thus, EBL processing in monads seems to undergo a similar developmental shift from a positivity bias to a negativity bias, as was previously found for emotional faces and terms. Despite these age-specific processing biases, children and adults seem to use similar movement features in EBL processing.

A 5-emotions stimuli set for emotion perception research with full-body dance movements

Ekman famously contended that there are different channels of emotional expression (face, voice, body), and that emotion recognition ability confers an adaptive advantage to the individual. Yet, still today, much emotion perception research is focussed on emotion recognition from the face, and few validated emotionally expressive full-body stimuli sets are available. Based on research on emotional speech perception, we created a new, highly controlled full-body stimuli set. We used the same-sequence approach, and not emotional actions (e.g., jumping of joy, recoiling in fear): One professional dancer danced 30 sequences of (dance) movements five times each, expressing joy, anger, fear, sadness or a neutral state, one at each repetition. We outline the creation of a total of 150, 6-s-long such video stimuli, that show the dancer as a white silhouette on a black background. Ratings from 90 participants (emotion recognition, aesthetic judgment) showed that intended emotion was recognized above chance (chance: 20%; joy: 45%, anger: 48%, fear: 37%, sadness: 50%, neutral state: 51%), and that aesthetic judgment was sensitive to the intended emotion (beauty ratings: joy > anger > fear > neutral state, and sad > fear > neutral state). The stimuli set, normative values and code are available for download.

First impressions from faces in the real world: Commentary on Sutherland and Young (2022)

The study of first impressions from faces now emphasizes the need to understand trait inferences made to naturalistic face images (British Journal of Psychology, 113, 2022, 1056). Face recognition algorithms based on deep convolutional neural networks simultaneously represent invariant, changeable and environmental variables in face images. Therefore, we suggest them as a comprehensive 'face space' model of first impressions of naturalistic faces. We also suggest that to understand trait inferences in the real world, a logical next step is to consider trait inferences made to whole people (faces and bodies). On the role of cultural contributions to trait perception, we think it is important for the field to begin to consider the way in which trait inferences motivate (or not) behaviour in independent and interdependent cultures.

Exploring the Emotional Functions of Co-Speech Hand Gesture in Language and Communication

Research over the past four decades has built a convincing case that co-speech hand gestures play a powerful role in human cognition . However, this recent focus on the cognitive function of gesture has, to a large extent, overlooked its emotional role-a role that was once central to research on bodily expression. In the present review, we first give a brief summary of the wealth of research demonstrating the cognitive function of co-speech gestures in language acquisition, learning, and thinking. Building on this foundation, we revisit the emotional function of gesture across a wide range of communicative contexts, from clinical to artistic to educational, and spanning diverse fields, from cognitive neuroscience to linguistics to affective science. Bridging the cognitive and emotional functions of gesture highlights promising avenues of research that have varied practical and theoretical implications for human-machine interactions, therapeutic interventions, language evolution, embodied cognition, and more.

White matter microstructure in face and body networks predicts facial expression and body posture perception across development

Facial expression and body posture recognition have protracted developmental trajectories. Interactions between face and body perception, such as the influence of body posture on facial expression perception, also change with development. While the brain regions underpinning face and body processing are well-defined, little is known about how white-matter tracts linking these regions relate to perceptual development. Here, we obtained complementary diffusion magnetic resonance imaging (MRI) measures (fractional anisotropy [FA], spherical mean Ṧ_μ ), and a quantitative MRI myelin-proxy measure (R1), within white-matter tracts of face- and body-selective networks in children and adolescents and related these to perceptual development. In tracts linking occipital and fusiform face areas, facial expression perception was predicted by age-related maturation, as measured by Ṧ_μ and R1, as well as age-independent individual differences in microstructure, captured by FA and R1. Tract microstructure measures linking posterior superior temporal sulcus body region with anterior temporal lobe (ATL) were related to the influence of body on facial expression perception, supporting ATL as a site of face and body network convergence. Overall, our results highlight age-dependent and age-independent constraints that white-matter microstructure poses on perceptual abilities during development and the importance of complementary microstructural measures in linking brain structure and behaviour.

A database of heterogeneous faces for studying naturalistic expressions

Facial expressions are thought to be complex visual signals, critical for communication between social agents. Most prior work aimed at understanding how facial expressions are recognized has relied on stimulus databases featuring posed facial expressions, designed to represent putative emotional categories (such as 'happy' and 'angry'). Here we use an alternative selection strategy to develop the Wild Faces Database (WFD); a set of one thousand images capturing a diverse range of ambient facial behaviors from outside of the laboratory. We characterized the perceived emotional content in these images using a standard categorization task in which participants were asked to classify the apparent facial expression in each image. In addition, participants were asked to indicate the intensity and genuineness of each expression. While modal scores indicate that the WFD captures a range of different emotional expressions, in comparing the WFD to images taken from other, more conventional databases, we found that participants responded more variably and less specifically to the wild-type faces, perhaps indicating that natural expressions are more multiplexed than a categorical model would predict. We argue that this variability can be employed to explore latent dimensions in our mental representation of facial expressions. Further, images in the WFD were rated as less intense and more genuine than images taken from other databases, suggesting a greater degree of authenticity among WFD images. The strong positive correlation between intensity and genuineness scores demonstrating that even the high arousal states captured in the WFD were perceived as authentic. Collectively, these findings highlight the potential utility of the WFD as a new resource for bridging the gap between the laboratory and real world in studies of expression recognition.

How Pain-Related Facial Expressions Are Evaluated in Relation to Gender, Race, and Emotion

Inequities in pain assessment are well-documented; however, the psychological mechanisms underlying such biases are poorly understood. We investigated potential perceptual biases in the judgments of faces displaying pain-related movements. Across five online studies, 956 adult participants viewed images of computer-generated faces (“targets”) that varied in features related to race (Black and White) and gender (women and men). Target identity was manipulated across participants, and each target had equivalent facial movements that displayed varying intensities of movement in facial action-units related to pain (Studies 1–4) or pain and emotion (Study 5). On each trial, participants provided categorical judgments as to whether a target was in pain (Studies 1–4) or which expression the target displayed (Study 5) and then rated the perceived intensity of the expression. Meta-analyses of Studies 1–4 revealed that movement intensity was positively associated with both categorizing a trial as painful and perceived pain intensity. Target race and gender did not consistently affect pain-related judgments, contrary to well-documented clinical inequities. In Study 5, in which pain was equally likely relative to other emotions, pain was the least frequently selected emotion (5%). Our results suggest that perceivers can utilize facial movements to evaluate pain in other individuals, but perceiving pain may depend on contextual factors. Furthermore, assessments of computer-generated, pain-related facial movements online do not replicate sociocultural biases observed in the clinic. These findings provide a foundation for future studies comparing CGI and real images of pain and emphasize the need for further work on the relationship between pain and emotion.

Words matter when inferring emotions: a conceptual replication and extension

It is long known that facial configurations play a critical role when inferring mental and emotional states from others. Nevertheless, there is still a scientific debate on how we infer emotions from facial configurations. The theory of constructed emotion (TCE) suggests that we may infer different emotions from the same facial configuration, depending on the context (e.g. provided by visual and lexical cues) in which they are perceived. For instance, a recent study found that participants were more accurate in inferring mental and emotional states across three different datasets (i.e. RMET, static and dynamic emojis) when words were provided (i.e. forced-choice task), compared to when they were not (i.e. free-labelling task), suggesting that words serve as contexts that modulate the inference from facial configurations. The goal of the current within-subject study was to replicate and extend these findings by adding a fourth dataset (KDEF-dyn), consisting of morphed human faces (to increase the ecological validity). Replicating previous findings, we observed that words increased accuracy across the three (previously used) datasets, an effect that was also observed for the facial morphed stimuli. Our findings are in line with the TCE, providing support for the importance of contextual verbal cues in emotion perception.

SMG: A Micro-gesture Dataset Towards Spontaneous Body Gestures for Emotional Stress State Analysis

We explore using body gestures for hidden emotional state analysis. As an important non-verbal communicative fashion, human body gestures are capable of conveying emotional information during social communication. In previous works, efforts have been made mainly on facial expressions, speech, or expressive body gestures to interpret classical expressive emotions. Differently, we focus on a specific group of body gestures, called micro-gestures (MGs), used in the psychology research field to interpret inner human feelings. MGs are subtle and spontaneous body movements that are proven, together with micro-expressions, to be more reliable than normal facial expressions for conveying hidden emotional information. In this work, a comprehensive study of MGs is presented from the computer vision aspect, including a novel spontaneous micro-gesture (SMG) dataset with two emotional stress states and a comprehensive statistical analysis indicating the correlations between MGs and emotional states. Novel frameworks are further presented together with various state-of-the-art methods as benchmarks for automatic classification, online recognition of MGs, and emotional stress state recognition. The dataset and methods presented could inspire a new way of utilizing body gestures for human emotion understanding and bring a new direction to the emotion AI community. The source code and dataset are made available: https://github.com/mikecheninoulu/SMG.

First impressions: Integrating faces and bodies in personality trait perception

Faces and bodies spontaneously elicit personality trait judgments (e.g., trustworthy, dominant, lazy). We examined how trait information from the face and body combine to form first impressions of the whole person and whether trait judgments from the face and body are affected by seeing the whole person. Consistent with the trait-dependence hypothesis, Experiment 1 showed that the relative contribution of the face and body to whole-person perception varied with the trait judged. Agreeableness traits (e.g., warm, aggressive, sympathetic, trustworthy) were inferred primarily from the face, conscientiousness traits (e.g., dependable, careless) from the body, and extraversion traits (e.g., dominant, quiet, confident) from the whole person. A control experiment showed that both clothing and body shape contributed to whole-person judgments. In Experiment 2, we found that a face (body) rated in the whole person elicited a different rating than when it was rated in isolation. Specifically, when trait ratings differed for an isolated face and body of the same identity, the whole-person context biased in-context ratings of the faces and bodies towards the ratings of the context. These results showed that face and body trait perception interact more than previously assumed. We combine current and established findings to propose a novel framework to account for face-body integration in trait perception. This framework incorporates basic elements such as perceptual determinants, nonperceptual determinants, trait formation, and integration, as well as predictive factors such as the rater, the person rated, and the situation.

Emotion is perceived accurately from isolated body parts, especially hands

Body posture and configuration provide important visual cues about the emotion states of other people. We know that bodily form is processed holistically, however, emotion recognition may depend on different mechanisms; certain body parts, such as the hands, may be especially important for perceiving emotion. This study therefore compared participants' emotion recognition performance when shown images of full bodies, or of isolated hands, arms, heads and torsos. Across three experiments, emotion recognition accuracy was above chance for all body parts. While emotions were recognized most accurately from full bodies, recognition performance from the hands was more accurate than for other body parts. Representational similarity analysis further showed that the pattern of errors for the hands was related to that for full bodies. Performance was reduced when stimuli were inverted, showing a clear body inversion effect. The high performance for hands was not due only to the fact that there are two hands, as performance remained well above chance even when just one hand was shown. These results demonstrate that emotions can be decoded from body parts. Furthermore, certain features, such as the hands, are more important to emotion perception than others. STATEMENT OF RELEVANCE: Successful social interaction relies on accurately perceiving emotional information from others. Bodies provide an abundance of emotion cues; however, the way in which emotional bodies and body parts are perceived is unclear. We investigated this perceptual process by comparing emotion recognition for body parts with that for full bodies. Crucially, we found that while emotions were most accurately recognized from full bodies, emotions were also classified accurately when images of isolated hands, arms, heads and torsos were seen. Of the body parts shown, emotion recognition from the hands was most accurate. Furthermore, shared patterns of emotion classification for hands and full bodies suggested that emotion recognition mechanisms are shared for full bodies and body parts. That the hands are key to emotion perception is important evidence in its own right. It could also be applied to interventions for individuals who find it difficult to read emotions from faces and bodies.

The McNorm library: creating and validating a new library of emotionally expressive whole body dance movements

The ability to exchange affective cues with others plays a key role in our ability to create and maintain meaningful social relationships. We express our emotions through a variety of socially salient cues, including facial expressions, the voice, and body movement. While significant advances have been made in our understanding of verbal and facial communication, to date, understanding of the role played by human body movement in our social interactions remains incomplete. To this end, here we describe the creation and validation of a new set of emotionally expressive whole-body dance movement stimuli, named the Motion Capture Norming (McNorm) Library, which was designed to reconcile a number of limitations associated with previous movement stimuli. This library comprises a series of point-light representations of a dancer's movements, which were performed to communicate to observers neutrality, happiness, sadness, anger, and fear. Based on results from two validation experiments, participants could reliably discriminate the intended emotion expressed in the clips in this stimulus set, with accuracy rates up to 60% (chance = 20%). We further explored the impact of dance experience and trait empathy on emotion recognition and found that neither significantly impacted emotion discrimination. As all materials for presenting and analysing this movement library are openly available, we hope this resource will aid other researchers in further exploration of affective communication expressed by human bodily movement.

What is missing in the study of emotion expression?

While approaching celebrations for the 150 years of "The Expression of the Emotions in Man and Animals", scientists' conclusions on emotion expression are still debated. Emotion expression has been traditionally anchored to prototypical and mutually exclusive facial expressions (e.g., anger, disgust, fear, happiness, sadness, and surprise). However, people express emotions in nuanced patterns and - crucially - not everything is in the face. In recent decades considerable work has critiqued this classical view, calling for a more fluid and flexible approach that considers how humans dynamically perform genuine expressions with their bodies in context. A growing body of evidence suggests that each emotional display is a complex, multi-component, motoric event. The human face is never static, but continuously acts and reacts to internal and environmental stimuli, with the coordinated action of muscles throughout the body. Moreover, two anatomically and functionally different neural pathways sub-serve voluntary and involuntary expressions. An interesting implication is that we have distinct and independent pathways for genuine and posed facial expressions, and different combinations may occur across the vertical facial axis. Investigating the time course of these facial blends, which can be controlled consciously only in part, is recently providing a useful operational test for comparing the different predictions of various models on the lateralization of emotions. This concise review will identify shortcomings and new challenges regarding the study of emotion expressions at face, body, and contextual levels, eventually resulting in a theoretical and methodological shift in the study of emotions. We contend that the most feasible solution to address the complex world of emotion expression is defining a completely new and more complete approach to emotional investigation. This approach can potentially lead us to the roots of emotional display, and to the individual mechanisms underlying their expression (i.e., individual emotional signatures).

A broadly tuned network for affective body language in the macaque brain

Body language is a powerful tool that we use to communicate how we feel, but it is unclear whether other primates also communicate in this way. Here, we use functional magnetic resonance imaging to show that the body-selective patches in macaques are activated by affective body language. Unexpectedly, we found these regions to be tolerant of naturalistic variation in posture as well as species; the bodies of macaques, humans, and domestic cats all evoked a stronger response when they conveyed fear than when they conveyed no affect. Multivariate analyses confirmed that the neural representation of fear-related body expressions was species-invariant. Collectively, these findings demonstrate that, like humans, macaques have body-selective brain regions in the ventral visual pathway for processing affective body language. These data also indicate that representations of body stimuli in these regions are built on the basis of emergent properties, such as socio-affective meaning, and not just putative image properties.

Emotional body expressions facilitate working memory: Evidence from the n‐back task

In daily life, individuals need to recognize and update emotional information from others' changing body expressions. However, whether emotional bodies can enhance working memory (WM) remains unknown. In the present study, participants completed a modified n-back task, in which they were required to indicate whether a presented image of an emotional body matched that of an item displayed before each block (0-back) or two positions previously (2-back). Each block comprised only fear, happiness, or neutral. We found that in the 0-back trials, when compared with neutral body expressions, the participants took less time and showed comparable ceiling effects for accuracy in happy bodies followed by fearful bodies. When WM load increased to 2-back, both fearful and happy bodies significantly facilitated WM performance (i.e., faster reaction time and higher accuracy) relative to neutral conditions. In summary, the current findings reveal the enhancement effect of emotional body expressions on WM and highlight the importance of emotional action information in WM.

Judging the emotional states of customer service staff in the workplace: A multimodal dataset analysis

Background

Emotions play a decisive and central role in the workplace, especially in the service-oriented enterprises. Due to the highly participatory and interactive nature of the service process, employees' emotions are usually highly volatile during the service delivery process, which can have a negative impact on business performance. Therefore, it is important to effectively judge the emotional states of customer service staff.

Methods

We collected data on real-life work situations of call center employees in a large company. Three consecutive studies were conducted: first, the emotional states of 29 customer service staff were videotaped by wide-angle cameras. In Study 1, we constructed scoring criteria and auxiliary tools of picture-type scales through a free association test. In Study 2, two groups of experts were invited to evaluate the emotional states of customer service staff. In Study 3, based on the results in Study 2 and a multimodal emotional recognition method, a multimodal dataset was constructed to explore how each modality conveys the emotions of customer service staff in workplace.

Results

Through the scoring by 2 groups of experts and 1 group of volunteers, we first developed a set of scoring criteria and picture-type scales with the combination of SAM scale for judging the emotional state of customer service staff. Then we constructed 99 (out of 297) sets of stable multimodal emotion datasets. Based on the comparison among the datasets, we found that voice conveys emotional valence in the workplace more significantly, and that facial expressions have more prominant connection with emotional arousal.

Conclusion

Theoretically, this study enriches the way in which emotion data is collected and can provide a basis for the subsequent development of multimodal emotional datasets. Practically, it can provide guidance for the effective judgment of employee emotions in the workplace.

Emotional body postures affect inhibitory control only when task-relevant

A classical theoretical frame to interpret motor reactions to emotional stimuli is that such stimuli, particularly those threat-related, are processed preferentially, i.e., they are capable of capturing and grabbing attention automatically. Research has recently challenged this view, showing that the task relevance of emotional stimuli is crucial to having a reliable behavioral effect. Such evidence indicated that emotional facial expressions do not automatically influence motor responses in healthy young adults, but they do so only when intrinsically pertinent to the ongoing subject's goals. Given the theoretical relevance of these findings, it is essential to assess their generalizability to different, socially relevant emotional stimuli such as emotional body postures. To address this issue, we compared the performance of 36 right-handed participants in two different versions of a Go/No-go task. In the Emotional Discrimination task, participants were required to withhold their responses at the display of emotional body postures (fearful or happy) and to move at the presentation of neutral postures. Differently, in the control task, the same images were shown, but participants had to respond according to the color of the actor/actress' t-shirt, disregarding the emotional content. Results showed that participants made more commission errors (instances in which they moved even though the No-go signal was presented) for happy than fearful body postures in the Emotional Discrimination task. However, this difference disappeared in the control task. Such evidence indicates that, like facial emotion, emotional body expressions do not influence motor control automatically, but only when they are task-relevant.

Decoding spatiotemporal features of emotional body language in social interactions

How are emotions perceived through human body language in social interactions? This study used point-light displays of human interactions portraying emotional scenes (1) to examine quantitative intrapersonal kinematic and postural body configurations, (2) to calculate interaction-specific parameters of these interactions, and (3) to analyze how far both contribute to the perception of an emotion category (i.e. anger, sadness, happiness or affection) as well as to the perception of emotional valence. By using ANOVA and classification trees, we investigated emotion-specific differences in the calculated parameters. We further applied representational similarity analyses to determine how perceptual ratings relate to intra- and interpersonal features of the observed scene. Results showed that within an interaction, intrapersonal kinematic cues corresponded to emotion category ratings, whereas postural cues reflected valence ratings. Perception of emotion category was also driven by interpersonal orientation, proxemics, the time spent in the personal space of the counterpart, and the motion–energy balance between interacting people. Furthermore, motion–energy balance and orientation relate to valence ratings. Thus, features of emotional body language are connected with the emotional content of an observed scene and people make use of the observed emotionally expressive body language and interpersonal coordination to infer emotional content of interactions.

The Role of Contextual Information in Classifying Spontaneous Social Laughter

Laughter is a ubiquitous and important social signal, but its nature is yet to be fully explored. One of the open empirical questions is about the role of context in the interpretation of laughter. Can laughs presented on their own convey specific feelings and social motives? How influential is social context when a person tries to understand the meaning of a laugh? Here we test the extent to which the classification of laughs produced in different situations is guided by knowing the context within which these laughs were produced. In the current study, stimuli were spontaneous laughs recorded in social situations engineered to elicit amusement, embarrassment, and schadenfreude. In a between-subjects design, participants classified these laughs being assigned to one of the four experimental conditions: audio only, audio-visual, side-by-side videos of two interactants, and side-by-side videos accompanied by a brief vignette. Participants’ task was to label each laugh as an instance of amusement, embarrassment, or schadenfreude laugh, or “other.” Laughs produced in situations inducing embarrassment were classified more accurately than laughs produced in other situations. Most importantly, eliminating information about the social settings in which laughs were produced decreased participants’ classification accuracy such that accuracy was no better than chance in the experimental conditions providing minimal contextual information. Our findings demonstrate the importance of context in the interpretation of laughter and highlight the complexity of experimental investigations of schadenfreude displays.

EmBody/EmFace as a new open tool to assess emotion recognition from body and face expressions

Nonverbal expressions contribute substantially to social interaction by providing information on another person’s intentions and feelings. While emotion recognition from dynamic facial expressions has been widely studied, dynamic body expressions and the interplay of emotion recognition from facial and body expressions have attracted less attention, as suitable diagnostic tools are scarce. Here, we provide validation data on a new open source paradigm enabling the assessment of emotion recognition from both 3D-animated emotional body expressions (Task 1: EmBody) and emotionally corresponding dynamic faces (Task 2: EmFace). Both tasks use visually standardized items depicting three emotional states (angry, happy, neutral), and can be used alone or together. We here demonstrate successful psychometric matching of the EmBody/EmFace items in a sample of 217 healthy subjects with excellent retest reliability and validity (correlations with the Reading-the-Mind-in-the-Eyes-Test and Autism-Spectrum Quotient, no correlations with intelligence, and given factorial validity). Taken together, the EmBody/EmFace is a novel, effective (< 5 min per task), highly standardized and reliably precise tool to sensitively assess and compare emotion recognition from body and face stimuli. The EmBody/EmFace has a wide range of potential applications in affective, cognitive and social neuroscience, and in clinical research studying face- and body-specific emotion recognition in patient populations suffering from social interaction deficits such as autism, schizophrenia, or social anxiety.

A systematic survey of face stimuli used in psychological research 2000-2020

For decades, psychology has relied on highly standardized images to understand how people respond to faces. Many of these stimuli are rigorously generated and supported by excellent normative data; as such, they have played an important role in the development of face science. However, there is now clear evidence that testing with ambient images (i.e., naturalistic images "in the wild") and including expressions that are spontaneous can lead to new and important insights. To precisely quantify the extent to which our current knowledge base has relied on standardized and posed stimuli, we systematically surveyed the face stimuli used in 12 key journals in this field across 2000-2020 (N = 3374 articles). Although a small number of posed expression databases continue to dominate the literature, the use of spontaneous expressions seems to be increasing. However, there has been no increase in the use of ambient or dynamic stimuli over time. The vast majority of articles have used highly standardized and nonmoving pictures of faces. An emerging trend is that virtual faces are being used as stand-ins for human faces in research. Overall, the results of the present survey highlight that there has been a significant imbalance in favor of standardized face stimuli. We argue that psychology would benefit from a more balanced approach because ambient and spontaneous stimuli have much to offer. We advocate a cognitive ethological approach that involves studying face processing in natural settings as well as the lab, incorporating more stimuli from "the wild".

Perceived positive social interdependence in online versus face-to-face team-based learning styles of collaborative learning: a randomized, controlled, mixed-methods study

BACKGROUND

Collaborative learning is a group learning approach in which positive social interdependence within a group is key to better learning performance and future attitudes toward team practice. Recent attempts to replace a face-to-face environment with an online one have been developed using information communication technology. However, this raises the concern that online collaborative learning (OCL) may reduce positive social interdependence. Therefore, this study aimed to compare the degree of social interdependence in OCL with face-to-face environments and clarify aspects that affect social interdependence in OCL.

METHODS

We conducted a crossover study comparing online and face-to-face collaborative learning environments in a clinical reasoning class using team-based learning for medical students (n = 124) in 2021. The participants were randomly assigned to two cohorts: Cohort A began in an online environment, while Cohort B began in a face-to-face environment. At the study's midpoint, the two cohorts exchanged the environments as a washout. The participants completed surveys using the social interdependence in collaborative learning scale (SOCS) to measure their perceived positive social interdependence before and after the class. Changes in the mean SOCS scores were compared using paired t-tests. Qualitative data related to the characteristics of the online environment were obtained from the focus groups and coded using thematic analysis.

RESULTS

The matched-pair tests of SOCS showed significant progression between pre- and post-program scores in the online and face-to-face groups. There were no significant differences in overall SOCS scores between the two groups. Sub-analysis by subcategory showed significant improvement in boundary (discontinuities among individuals) and means interdependence (resources, roles, and tasks) in both groups, but outcome interdependence (goals and rewards) improved significantly only in the online group. Qualitative analysis revealed four major themes affecting social interdependence in OCL: communication, task-sharing process, perception of other groups, and working facilities.

CONCLUSIONS

There is a difference in the communication styles of students in face-to-face and online environments, and these various influences equalize the social interdependence in a face-to-face and online environment.

An examination of autonomic and facial responses to prototypical facial emotion expressions in psychopathy

Meta-analyses have found that people high in psychopathy categorize (or “recognize”) others’ prototypical facial emotion expressions with reduced accuracy. However, these have been contested with remaining questions regarding the strength, specificity, and mechanisms of this ability in psychopathy. In addition, few studies have tested holistically whether psychopathy is related to reduced facial mimicry or autonomic arousal in response to others’ dynamic facial expressions. Therefore, the current study presented 6 s videos of a target person making prototypical emotion expressions (anger, fear, disgust, sadness, joy, and neutral) to N = 88 incarcerated adult males while recording facial electromyography, skin conductance response (SCR), and heart rate. Participants identified the emotion category and rated the valence and intensity of the target person’s emotion. Psychopathy was assessed via the Psychopathy Checklist-Revised (PCL-R). We predicted that overall PCL-R scores and scores for the interpersonal/affective traits, in particular, would be related to reduced emotion categorization accuracy, valence ratings, intensity ratings, facial mimicry, SCR amplitude, and cardiac deceleration in response to the prototypical facial emotion expressions. In contrast to our hypotheses, PCL-R scores were unrelated to emotion categorization accuracy, valence ratings, and intensity ratings. Stimuli failed to elicit facial mimicry from the full sample, which does not allow drawing conclusions about the relationship between psychopathy and facial mimicry. However, participants displayed general autonomic arousal responses, but not to prototypical emotion expressions per se. PCL-R scores were also unrelated to SCR and cardiac deceleration. These findings failed to identify aberrant behavioral and physiological responses to prototypical facial emotion expressions in relation to psychopathy.

Emotional context effect on recognition of varying facial emotion expression intensities in depression

BACKGROUND

Previous research has indicated that Major Depressive Disorder (MDD) patients have deficits in the process of facial emotion recognition. In most of these studies, isolated emotional faces were used, and the effect of the surrounding context of the face was neglected. We aimed to investigate how context emotion (sad or happy) affects facial emotion recognition and whether this effect is different in depressive individuals compared to the control group.

METHODS

Happy, sad, neutral facial expressions with congruent and incongruent visual contexts were presented to 51 MDD patients and 42 matched healthy controls. Emotional facial expressions are presented as morphs gradually expressing happiness or sadness with 40% and 80% intensity levels. Mean reaction time, mean accuracy rate, and mean emotion intensity rating score was calculated for each condition.

RESULTS

The performances on facial emotion intensity rating and accuracy rate were similar between MDD patients and controls. MDD patients were slower to recognize all facial emotions and to recognize facial emotions with emotionally incongruent backgrounds compared to congruent ones.

LIMITATIONS

Antidepressant therapy of patients might have affected our results.

CONCLUSIONS

Emotional contextual features have an important role in facial emotion recognition but this effect is independent of depression. Longer reaction time in depression may be related to some cognitive impairments.