Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements

Multiracial Reading the Mind in the Eyes Test (MRMET): An inclusive version of an influential measure

Can an inclusive test of face cognition meet or exceed the psychometric properties of a prominent less inclusive test? Here, we norm and validate an updated version of the influential Reading the Mind in the Eyes Test (RMET), a clinically significant neuropsychiatric paradigm that has long been used to assess theory of mind and social cognition. Unlike the RMET, our Multiracial Reading the Mind in the Eyes Test (MRMET) incorporates racially inclusive stimuli, nongendered answer choices, ground-truth referenced answers, and more accessible vocabulary. We show, via a series of large datasets, that the MRMET meets or exceeds RMET across major psychometric indices. Moreover, the reliable signal captured by the two tests is statistically indistinguishable, evidence for full interchangeability. We thus present the MRMET as a high-quality, inclusive, normed and validated alternative to the RMET, and as a case in point that inclusivity in psychometric tests of face cognition is an achievable aim. The MRMET test and our normative and validation data sets are openly available under a CC-BY-SA 4.0 license at osf.io/ahq6n.

Lesson learned from the COVID-19 pandemic: toddlers learn earlier to read emotions with face masks

In a prior study we demonstrated that the presence of face masks impairs the human capability of accurately inferring emotions conveyed through facial expressions, at all ages. The degree of impairment posed by face covering was notably more pronounced in children aged between three and five years old. In the current study, we conducted the same test as a follow-up after one year from the onset of the COVID-19 pandemic, when the requirement of wearing face masks was holding in almost all circumstances of everyday life when social interactions occur. The results indicate a noteworthy improvement in recognizing facial expressions with face masks among children aged three to five, compared to the pre-pandemic settings. These findings hold a significant importance, suggesting that toddlers effectively mitigated the social challenges associated with masks use: they overcame initial environmental limitations, improving their capability to interpret facial expressions even in the absence of visual cues from the lower part of the face.

Sensorimotor regulation of facial expression - An untouched frontier

Facial expression is a critical form of nonverbal social communication which promotes emotional exchange and affiliation among humans. Facial expressions are generated via precise contraction of the facial muscles, guided by sensory feedback. While the neural pathways underlying facial motor control are well characterized in humans and primates, it remains unknown how tactile and proprioceptive information reaches these pathways to guide facial muscle contraction. Thus, despite the importance of facial expressions for social functioning, little is known about how they are generated as a unique sensorimotor behavior. In this review, we highlight current knowledge about sensory feedback from the face and how it is distinct from other body regions. We describe connectivity between the facial sensory and motor brain systems, and call attention to the other brain systems which influence facial expression behavior, including vision, gustation, emotion, and interoception. Finally, we petition for more research on the sensory basis of facial expressions, asserting that incomplete understanding of sensorimotor mechanisms is a barrier to addressing atypical facial expressivity in clinical populations.

How emotion is experienced and expressed in multiple cultures: a large-scale experiment across North America, Europe, and Japan

Core to understanding emotion are subjective experiences and their expression in facial behavior. Past studies have largely focused on six emotions and prototypical facial poses, reflecting limitations in scale and narrow assumptions about the variety of emotions and their patterns of expression. We examine 45,231 facial reactions to 2,185 evocative videos, largely in North America, Europe, and Japan, collecting participants' self-reported experiences in English or Japanese and manual and automated annotations of facial movement. Guided by Semantic Space Theory, we uncover 21 dimensions of emotion in the self-reported experiences of participants in Japan, the United States, and Western Europe, and considerable cross-cultural similarities in experience. Facial expressions predict at least 12 dimensions of experience, despite massive individual differences in experience. We find considerable cross-cultural convergence in the facial actions involved in the expression of emotion, and culture-specific display tendencies-many facial movements differ in intensity in Japan compared to the U.S./Canada and Europe but represent similar experiences. These results quantitatively detail that people in dramatically different cultures experience and express emotion in a high-dimensional, categorical, and similar but complex fashion.

Masked emotions: does children's affective state influence emotion recognition?

Introduction

Facial emotion recognition abilities of children have been the focus of attention across various fields, with implications for communication, social interaction, and human behavior. In response to the COVID-19 pandemic, wearing a face mask in public became mandatory in many countries, hindering social information perception and emotion recognition. Given the importance of visual communication for children's social-emotional development, concerns have been raised on whether face masks could impair their ability to recognize emotions and thereby possibly impact their social-emotional development.

Methods

To this extent, a quasiexperimental study was designed with a two-fold objective: firstly, to identify children's accuracy in recognizing basic emotions (anger, happiness, fear, disgust, sadness) and emotional neutrality when presented with faces under two conditions: one with no-masks and another with faces partially covered by various types of masks (medical, nonmedical, surgical, or cloth); secondly, to explore any correlation between children's emotion recognition accuracy and their affective state. Sixty-nine (69) elementary school students aged 6-7 years old from Greece were recruited for this purpose. Following specific requirements of the second phase of the experiment students were assigned to one of three (3) distinct affective condition groups: Group A-Happiness, Group B-Sadness, and Group C-Emotional Neutrality. Image stimuli were drawn from the FACES Dataset, and students' affective state was registered using the self-reporting emotions-registration tool, AffectLecture app.

Results

The study's findings indicate that children can accurately recognize emotions even with masks, although recognizing disgust is more challenging. Additionally, following both positive and negative affective state priming promoted systematic inaccuracies in emotion recognition. Most significantly, results showed a negative bias for children in negative affective state and a positive bias for those in positive affective state.

Discussion

Children's affective state significantly influenced their emotion recognition abilities; sad affective states led to lower recognition overall and a bias toward recognizing sad expressions, while happy affective states resulted in a positive bias, improving recognition of happiness, and affecting how emotional neutrality and sadness were actually perceived. In conclusion, this study sheds light on the intriguing dynamics of how face masks affect children's emotion recognition, but also underlines the profound influence of their affective state.

Being facially expressive is socially advantageous

Individuals vary in how they move their faces in everyday social interactions. In a first large-scale study, we measured variation in dynamic facial behaviour during social interaction and examined dyadic outcomes and impression formation. In Study 1, we recorded semi-structured video calls with 52 participants interacting with a confederate across various everyday contexts. Video clips were rated by 176 independent participants. In Study 2, we examined video calls of 1315 participants engaging in unstructured video-call interactions. Facial expressivity indices were extracted using automated Facial Action Coding Scheme analysis and measures of personality and partner impressions were obtained by self-report. Facial expressivity varied considerably across participants, but little across contexts, social partners or time. In Study 1, more facially expressive participants were more well-liked, agreeable, and successful at negotiating (if also more agreeable). Participants who were more facially competent, readable, and perceived as readable were also more well-liked. In Study 2, we replicated the findings that facial expressivity was associated with agreeableness and liking by their social partner, and additionally found it to be associated with extraversion and neuroticism. Findings suggest that facial behaviour is a stable individual difference that proffers social advantages, pointing towards an affiliative, adaptive function.

On manipulation by emotional AI: UK adults' views and governance implications

With growing commercial, regulatory and scholarly interest in use of Artificial Intelligence (AI) to profile and interact with human emotion ("emotional AI"), attention is turning to its capacity for manipulating people, relating to factors impacting on a person's decisions and behavior. Given prior social disquiet about AI and profiling technologies, surprisingly little is known on people's views on the benefits and harms of emotional AI technologies, especially their capacity for manipulation. This matters because regulators of AI (such as in the European Union and the UK) wish to stimulate AI innovation, minimize harms and build public trust in these systems, but to do so they should understand the public's expectations. Addressing this, we ascertain UK adults' perspectives on the potential of emotional AI technologies for manipulating people through a two-stage study. Stage One (the qualitative phase) uses design fiction principles to generate adequate understanding and informed discussion in 10 focus groups with diverse participants (n = 46) on how emotional AI technologies may be used in a range of mundane, everyday settings. The focus groups primarily flagged concerns about manipulation in two settings: emotion profiling in social media (involving deepfakes, false information and conspiracy theories), and emotion profiling in child oriented "emotoys" (where the toy responds to the child's facial and verbal expressions). In both these settings, participants express concerns that emotion profiling covertly exploits users' cognitive or affective weaknesses and vulnerabilities; additionally, in the social media setting, participants express concerns that emotion profiling damages people's capacity for rational thought and action. To explore these insights at a larger scale, Stage Two (the quantitative phase), conducts a UK-wide, demographically representative national survey (n = 2,068) on attitudes toward emotional AI. Taking care to avoid leading and dystopian framings of emotional AI, we find that large majorities express concern about the potential for being manipulated through social media and emotoys. In addition to signaling need for civic protections and practical means of ensuring trust in emerging technologies, the research also leads us to provide a policy-friendly subdivision of what is meant by manipulation through emotional AI and related technologies.

Cultural challenges for adapting behavioral intervention frameworks: A critical examination from a cultural psychology perspective

We introduce the bias and equivalence framework to highlight how concepts, methods, and tools from cultural psychology can contribute to successful cultural adaptation and implementation of behavioral interventions. To situate our contribution, we provide a review of recent cultural adaptation research and existing frameworks. We identified 68 different frameworks that have been cited when reporting cultural adaptations and highlight three major adaptation dimensions that can be used to differentiate adaptations. Regarding effectiveness, we found an average effect size of zr = 0.24 (95%CI 0.20, 0.29) in 24 meta-analyses published since 2014, but also substantive differences across domains and unclear effects of the extent of cultural adaptations. To advance cultural adaptation efforts, we outline a framework that integrates key steps from previous cultural adaptation frameworks and highlight how cultural bias and equivalence considerations in conjunction with community engagement help a) in the diagnosis of behavioral or psychological problems, b) identification of possible interventions, c) the selection of specific mechanisms of behavior change, d) the specification and documentation of dose effects and thresholds for diagnosis, e) entry and exit points within intervention programs, and f) cost-benefit-sustainability discussions. We provide guiding questions that may help researchers when adapting interventions to novel cultural contexts.

Zoti's Social Toolkit: Developing and piloting novel animated tasks to assess emotional understanding and conflict resolution skills in childhood

Current methods used to investigate emotional inference and conflict resolution knowledge are limited in their suitability for use with children with language disorders due to a reliance on language processing. This is problematic, as nearly 8% of the population are estimated to have developmental language disorder (DLD). In this paper, we present 'Zoti's Social Toolkit', a set of animated scenarios that can be used to assess emotion inferencing and conflict resolution knowledge. All animated scenarios contain interpersonal situations centred around a gender-neutral alien named Zoti. Four studies investigated the face and construct validity of the stimuli. The final stimulus set can be used with children, who may or may not have language difficulties and is openly available for use in research.

Emotion in motion: perceiving fear in the behaviour of individuals from minimal motion capture displays

The ability to quickly and accurately recognise emotional states is adaptive for numerous social functions. Although body movements are a potentially crucial cue for inferring emotions, few studies have studied the perception of body movements made in naturalistic emotional states. The current research focuses on the use of body movement information in the perception of fear expressed by targets in a virtual heights paradigm. Across three studies, participants made judgments about the emotional states of others based on motion-capture body movement recordings of those individuals actively engaged in walking a virtual plank at ground-level or 80 stories above a city street. Results indicated that participants were reliably able to differentiate between height and non-height conditions (Studies 1 & 2), were more likely to spontaneously describe target behaviour in the height condition as fearful (Study 2) and their fear estimates were highly calibrated with the fear ratings from the targets (Studies 1-3). Findings show that VR height scenarios can induce fearful behaviour and that people can perceive fear in minimal representations of body movement.

Uncertain world: How children's curiosity and intolerance of uncertainty relate to their behaviour and emotion under uncertainty

Curiosity and intolerance of uncertainty (IU) are both thought to drive information seeking but may have different affective profiles; curiosity is often associated with positive affective responses to uncertainty and improved learning outcomes, whereas IU is associated with negative affective responses and anxiety. Curiosity and IU have not previously been examined together in children but may both play an important role in understanding how children respond to uncertainty. Our research aimed to examine how individual differences in parent-reported curiosity and IU were associated with behavioural and emotional responses to uncertainty. Children aged 8 to 12 (n = 133) completed a game in which they were presented with an array of buttons on the screen that, when clicked, played neutral or aversive sounds. Children pressed buttons (information seeking) and rated their emotions and worry under conditions of high and low uncertainty. Facial expressions were also monitored for affective responses. Analyses revealed that children sought more information under high uncertainty than low uncertainty trials and that more curious children reported feeling happier. Contrary to expectations, IU and curiosity were not related to the number of buttons children pressed, nor to their self-reported emotion or worry. However, exploratory analyses suggest that children who are high in IU may engage in more information seeking that reflects checking or safety-seeking than those who are low in IU. In addition, our findings suggest that there may be age-related change in the effects of IU on worry, with IU more strongly related to worry in uncertain situations for older children than younger children.

Facial representations of complex affective states combining pain and a negative emotion

Pain is rarely communicated alone, as it is often accompanied by emotions such as anger or sadness. Communicating these affective states involves shared representations. However, how an individual conceptually represents these combined states must first be tested. The objective of this study was to measure the interaction between pain and negative emotions on two types of facial representations of these states, namely visual (i.e., interactive virtual agents; VAs) and sensorimotor (i.e., one's production of facial configurations). Twenty-eight participants (15 women) read short written scenarios involving only pain or a combined experience of pain and a negative emotion (anger, disgust, fear, or sadness). They produced facial configurations representing these experiences on the faces of the VAs and on their face (own production or imitation of VAs). The results suggest that affective states related to a direct threat to the body (i.e., anger, disgust, and pain) share a similar facial representation, while those that present no immediate danger (i.e., fear and sadness) differ. Although visual and sensorimotor representations of these states provide congruent affective information, they are differently influenced by factors associated with the communication cycle. These findings contribute to our understanding of pain communication in different affective contexts.

Social threat processing in adults and children: Faster orienting to, but shorter dwell time on, angry faces during visual search

Attention to emotional signals conveyed by others is critical for gleaning information about potential social partners and the larger social context. Children appear to detect social threats (e.g., angry faces) faster than non-threatening social signals (e.g., neutral faces). However, methods that rely on behavioral responses alone are limited in identifying different attentional processes involved in threat detection or responding. To address this question, we used a visual search paradigm to assess behavioral (i.e., reaction time to select a target image) and attentional (i.e., eye-tracking fixations, saccadic shifts, and dwell time) responses in children (ages 7-10 years old, N = 42) and adults (ages 18-23 years old, N = 46). In doing so, we compared behavioral responding and attentional detection and engagement with threatening (i.e., angry and fearful faces) and non-threatening (i.e., happy faces) social signals. Overall, children and adults were faster to detect social threats (i.e., angry faces), but spent a smaller proportion of time dwelling on them and had slower behavioral responses. Findings underscore the importance of combining different measures to parse differences between processing versus responding to social signals across development. RESEARCH HIGHLIGHTS: Children and adults are slower to select angry faces when measured by time to mouse-click but faster to detect angry faces when measured by time to first eye fixation. The use of eye-tracking addresses some limitations of prior visual search tasks with children that rely on behavioral responses alone. Results suggest shorter time to first fixation, but subsequently, shorter duration of dwell on social threat in children and adults.

Development of a tool for infant facial emotion recognition (InFER) for postpartum mothers with mental illnesses

Understanding deficits in recognition of infant emotions in mothers with mental illnesses is limited by the lack of validated instruments. We present the development and content validation of the infant facial emotion recognition tool (InFER) in India to examine the ability of mothers to detect the infants' emotions. A total of 164 images of infant faces in various emotional states were gathered from the parents of four infants (two male and two female: up to 12 months old). Infant emotion in each image was identified by the respective mother. Content validation was carried out by 21 experts. Images with ≥70% concordance among experts were selected. The newly developed tool, InFER, consists of a total 39 infant images representing the six basic emotions. This tool was then administered among mothers during their postpartum period-10 healthy mothers and 10 mothers who had remitted from any schizophrenia spectrum disorder, bipolar affective disorder or major depressive disorder. The mean age and mean years of education for both groups were comparable (age∼25 years, education ∼15 years). A significant difference was found between the two groups in their ability to recognize infant emotions (Mann-Whitney U = 12.5; p = 0.004). InFER is a promising tool in Indian settings for understanding maternal recognition of infant emotions.

Spatio-Temporal Image-Based Encoded Atlases for EEG Emotion Recognition

Emotion recognition plays an essential role in human-human interaction since it is a key to understanding the emotional states and reactions of human beings when they are subject to events and engagements in everyday life. Moving towards human-computer interaction, the study of emotions becomes fundamental because it is at the basis of the design of advanced systems to support a broad spectrum of application areas, including forensic, rehabilitative, educational, and many others. An effective method for discriminating emotions is based on ElectroEncephaloGraphy (EEG) data analysis, which is used as input for classification systems. Collecting brain signals on several channels and for a wide range of emotions produces cumbersome datasets that are hard to manage, transmit, and use in varied applications. In this context, the paper introduces the Empátheia system, which explores a different EEG representation by encoding EEG signals into images prior to their classification. In particular, the proposed system extracts spatio-temporal image encodings, or atlases, from EEG data through the Processing and transfeR of Interaction States and Mappings through Image-based eNcoding (PRISMIN) framework, thus obtaining a compact representation of the input signals. The atlases are then classified through the Empátheia architecture, which comprises branches based on convolutional, recurrent, and transformer models designed and tuned to capture the spatial and temporal aspects of emotions. Extensive experiments were conducted on the Shanghai Jiao Tong University (SJTU) Emotion EEG Dataset (SEED) public dataset, where the proposed system significantly reduced its size while retaining high performance. The results obtained highlight the effectiveness of the proposed approach and suggest new avenues for data representation in emotion recognition from EEG signals.

Application of Stereo Digital Image Correlation on Facial Expressions Sensing

Facial expression is an important way to reflect human emotions and it represents a dynamic deformation process. Analyzing facial movements is an effective means of understanding expressions. However, there is currently a lack of methods capable of analyzing the dynamic details of full-field deformation in expressions. In this paper, in order to enable effective dynamic analysis of expressions, a classic optical measuring method called stereo digital image correlation (stereo-DIC or 3D-DIC) is employed to analyze the deformation fields of facial expressions. The forming processes of six basic facial expressions of certain experimental subjects are analyzed through the displacement and strain fields calculated by 3D-DIC. The displacement fields of each expression exhibit strong consistency with the action units (AUs) defined by the classical Facial Action Coding System (FACS). Moreover, it is shown that the gradient of the displacement, i.e., the strain fields, offers special advantages in characterizing facial expressions due to their localized nature, effectively sensing the nuanced dynamics of facial movements. By processing extensive data, this study demonstrates two featured regions in six basic expressions, one where deformation begins and the other where deformation is most severe. Based on these two regions, the temporal evolutions of the six basic expressions are discussed. The presented investigations demonstrate the superior performance of 3D-DIC in the quantitative analysis of facial expressions. The proposed analytical strategy might have potential value in objectively characterizing human expressions based on quantitative measurement.

Facial Expression Recognition for Measuring Jurors' Attention in Acoustic Jury Tests

The perception of sound greatly impacts users' emotional states, expectations, affective relationships with products, and purchase decisions. Consequently, assessing the perceived quality of sounds through jury testing is crucial in product design. However, the subjective nature of jurors' responses may limit the accuracy and reliability of jury test outcomes. This research explores the utility of facial expression analysis in jury testing to enhance response reliability and mitigate subjectivity. Some quantitative indicators allow the research hypothesis to be validated, such as the correlation between jurors' emotional responses and valence values, the accuracy of jury tests, and the disparities between jurors' questionnaire responses and the emotions measured by FER (facial expression recognition). Specifically, analysis of attention levels during different statuses reveals a discernible decrease in attention levels, with 70 percent of jurors exhibiting reduced attention levels in the 'distracted' state and 62 percent in the 'heavy-eyed' state. On the other hand, regression analysis shows that the correlation between jurors' valence and their choices in the jury test increases when considering the data where the jurors are attentive. The correlation highlights the potential of facial expression analysis as a reliable tool for assessing juror engagement. The findings suggest that integrating facial expression recognition can enhance the accuracy of jury testing in product design by providing a more dependable assessment of user responses and deeper insights into participants' reactions to auditory stimuli.

Reduced stereotypicality and spared use of facial expression predictions for social evaluation in autism

Background/Objective

Autism has been investigated through traditional emotion recognition paradigms, merely investigating accuracy, thereby constraining how potential differences across autistic and control individuals may be observed, identified, and described. Moreover, the use of emotional facial expression information for social functioning in autism is of relevance to provide a deeper understanding of the condition.

Method

Adult autistic individuals (n = 34) and adult control individuals (n = 34) were assessed with a social perception behavioral paradigm exploring facial expression predictions and their impact on social evaluation.

Results

Autistic individuals held less stereotypical predictions than controls. Importantly, despite such differences in predictions, the use of such predictions for social evaluation did not differ significantly between groups, as autistic individuals relied on their predictions to evaluate others to the same extent as controls.

Conclusions

These results help to understand how autistic individuals perceive social stimuli and evaluate others, revealing a deviation from stereotypicality beyond which social evaluation strategies may be intact.

Differential Brain Activation for Four Emotions in VR-2D and VR-3D Modes

Similar to traditional imaging, virtual reality (VR) imagery encompasses nonstereoscopic (VR-2D) and stereoscopic (VR-3D) modes. Currently, Russell's emotional model has been extensively studied in traditional 2D and VR-3D modes, but there is limited comparative research between VR-2D and VR-3D modes. In this study, we investigate whether Russell's emotional model exhibits stronger brain activation states in VR-3D mode compared to VR-2D mode. By designing an experiment covering four emotional categories (high arousal-high pleasure (HAHV), high arousal-low pleasure (HALV), low arousal-low pleasure (LALV), and low arousal-high pleasure (LAHV)), EEG signals were collected from 30 healthy undergraduate and graduate students while watching videos in both VR modes. Initially, power spectral density (PSD) computations revealed distinct brain activation patterns in different emotional states across the two modes, with VR-3D videos inducing significantly higher brainwave energy, primarily in the frontal, temporal, and occipital regions. Subsequently, Differential entropy (DE) feature sets, selected via a dual ten-fold cross-validation Support Vector Machine (SVM) classifier, demonstrate satisfactory classification accuracy, particularly superior in the VR-3D mode. The paper subsequently presents a deep learning-based EEG emotion recognition framework, adeptly utilizing the frequency, spatial, and temporal information of EEG data to improve recognition accuracy. The contribution of each individual feature to the prediction probabilities is discussed through machine-learning interpretability based on Shapley values. The study reveals notable differences in brain activation states for identical emotions between the two modes, with VR-3D mode showing more pronounced activation.

Face and context integration in emotion inference is limited and variable across categories and individuals

The ability to make nuanced inferences about other people's emotional states is central to social functioning. While emotion inferences can be sensitive to both facial movements and the situational context that they occur in, relatively little is understood about when these two sources of information are integrated across emotion categories and individuals. In a series of studies, we use one archival and five empirical datasets to demonstrate that people could be integrating, but that emotion inferences are just as well (and sometimes better) captured by knowledge of the situation alone, while isolated facial cues are insufficient. Further, people integrate facial cues more for categories for which they most frequently encounter facial expressions in everyday life (e.g., happiness). People are also moderately stable over time in their reliance on situational cues and integration of cues and those who reliably utilize situation cues more also have better situated emotion knowledge. These findings underscore the importance of studying variability in reliance on and integration of cues.

Deep learning reveals what facial expressions mean to people in different cultures

Cross-cultural studies of the meaning of facial expressions have largely focused on judgments of small sets of stereotypical images by small numbers of people. Here, we used large-scale data collection and machine learning to map what facial expressions convey in six countries. Using a mimicry paradigm, 5,833 participants formed facial expressions found in 4,659 naturalistic images, resulting in 423,193 participant-generated facial expressions. In their own language, participants also rated each expression in terms of 48 emotions and mental states. A deep neural network tasked with predicting the culture-specific meanings people attributed to facial movements while ignoring physical appearance and context discovered 28 distinct dimensions of facial expression, with 21 dimensions showing strong evidence of universality and the remainder showing varying degrees of cultural specificity. These results capture the underlying dimensions of the meanings of facial expressions within and across cultures in unprecedented detail.

Detection of arousal and valence from facial expressions and physiological responses evoked by different types of stressors

Automatically detecting mental state such as stress from video images of the face could support evaluating stress responses in applicants for high risk jobs or contribute to timely stress detection in challenging operational settings (e.g., aircrew, command center operators). Challenges in automatically estimating mental state include the generalization of models across contexts and across participants. We here aim to create robust models by training them using data from different contexts and including physiological features. Fifty-one participants were exposed to different types of stressors (cognitive, social evaluative and startle) and baseline variants of the stressors. Video, electrocardiogram (ECG), electrodermal activity (EDA) and self-reports (arousal and valence) were recorded. Logistic regression models aimed to classify between high and low arousal and valence across participants, where "high" and "low" were defined relative to the center of the rating scale. Accuracy scores of different models were evaluated: models trained and tested within a specific context (either a baseline or stressor variant of a task), intermediate context (baseline and stressor variant of a task), or general context (all conditions together). Furthermore, for these different model variants, only the video data was included, only the physiological data, or both video and physiological data. We found that all (video, physiological and video-physio) models could successfully distinguish between high- and low-rated arousal and valence, though performance tended to be better for (1) arousal than valence, (2) specific context than intermediate and general contexts, (3) video-physio data than video or physiological data alone. Automatic feature selection resulted in inclusion of 3-20 features, where the models based on video-physio data usually included features from video, ECG and EDA. Still, performance of video-only models approached the performance of video-physio models. Arousal and valence ratings by three experienced human observers scores based on part of the video data did not match with self-reports. In sum, we showed that it is possible to automatically monitor arousal and valence even in relatively general contexts and better than humans can (in the given circumstances), and that non-contact video images of faces capture an important part of the information, which has practical advantages.

See no evil: Attentional bias toward threat is diminished in aged monkeys

Prior evidence demonstrates that relative to younger adults, older human adults exhibit attentional biases toward positive and/or away from negative socioaffective stimuli (i.e., the age-related positivity effect). Whether or not the effect is phylogenetically conserved is currently unknown and its biopsychosocial origins are debated. To address this gap, we evaluated how visual processing of socioaffective stimuli differs in aged, compared to middle-aged, rhesus monkeys (Macaca mulatta) using eye tracking in two experimental designs that are directly comparable to those historically used for evaluating attentional biases in humans. Results of our study demonstrate that while younger rhesus possesses robust attentional biases toward threatening pictures of conspecifics' faces, aged animals evidence no such bias. Critically, these biases emerged only when threatening faces were paired with neutral and not ostensibly "positive" faces, suggesting social context modifies the effect. Results of our study suggest that the evolutionarily shared mechanisms drive age-related decline in visual biases toward negative stimuli in aging across primate species. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Language access differentially alters functional connectivity during emotion perception across cultures

Introduction

It is often assumed that the ability to recognize the emotions of others is reflexive and automatic, driven only by observable facial muscle configurations. However, research suggests that accumulated emotion concept knowledge shapes the way people perceive the emotional meaning of others' facial muscle movements. Cultural upbringing can shape an individual's concept knowledge, such as expectations about which facial muscle configurations convey anger, disgust, or sadness. Additionally, growing evidence suggests that access to emotion category words, such as "anger," facilitates access to such emotion concept knowledge and in turn facilitates emotion perception.

Methods

To investigate the impact of cultural influence and emotion concept accessibility on emotion perception, participants from two cultural groups (Chinese and White Americans) completed a functional magnetic resonance imaging scanning session to assess functional connectivity between brain regions during emotion perception. Across four blocks, participants were primed with either English emotion category words ("anger," "disgust") or control text (XXXXXX) before viewing images of White American actors posing facial muscle configurations that are stereotypical of anger and disgust in the United States.

Results

We found that when primed with "disgust" versus control text prior to seeing disgusted facial expressions, Chinese participants showed a significant decrease in functional connectivity between a region associated with semantic retrieval (the inferior frontal gyrus) and regions associated with semantic processing, visual perception, and social cognition. Priming the word "anger" did not impact functional connectivity for Chinese participants relative to control text, and priming neither "disgust" nor "anger" impacted functional connectivity for White American participants.

Discussion

These findings provide preliminary evidence that emotion concept accessibility differentially impacts perception based on participants' cultural background.

No influence of emotional expression on size underestimation of upright faces

Faces are a primary means of conveying social information between humans. One important factor modulating the perception of human faces is emotional expression. Face inversion also affects perception, including judgments of emotional expression, possibly through the disruption of configural processing. One intriguing inversion effect is an illusion whereby faces appear to be physically smaller when upright than when inverted. This illusion appears to be highly selective for faces. In this study, we investigated whether the emotional expression of a face (neutral, happy, afraid, and angry) modulates the magnitude of this size illusion. Results showed that for all four expressions, there was a clear bias for inverted stimuli to be judged as larger than upright ones. This demonstrates that there is no influence of emotional expression on the size underestimation of upright faces, a surprising result given that recognition of different emotional expressions is known to be affected unevenly by inversion. Results are discussed considering recent neuroimaging research which used population receptive field (pRF) mapping to investigate the neural mechanisms underlying face perception features and which may provide an explanation for how an upright face appears smaller than an inverted one. Elucidation of this effect would lead to a greater understanding of how humans communicate.

Remember walking in their shoes? The relation of self-referential source memory and emotion recognition

Deficits in the ability to read the emotions of others have been demonstrated in mental disorders, such as dissociation and schizophrenia, which involve a distorted sense of self. This study examined whether weakened self-referential source memory, being unable to remember whether a piece of information has been processed with reference to oneself, is linked to ineffective emotion recognition. In two samples from a college and community, we quantified the participants' ability to remember the self-generated versus non-self-generated origins of sentences they had previously read or partially generated. We also measured their ability to read others' emotions accurately when viewing photos of people in affect-charged situations. Multinomial processing tree modelling was applied to obtain a measure of self-referential source memory that was not biased by non-mnemonic factors. Our first experiment with college participants revealed a positive correlation between correctly remembering the origins of sentences and accurately recognising the emotions of others. This correlation was successfully replicated in the second experiment with community participants. The current study offers evidence of a link between self-referential source memory and emotion recognition.

Differential effects of attachment security on visual fixation to facial expressions of emotion in 14-month-old infants: an eye-tracking study

Introduction

Models of attachment and information processing suggest that the attention infants allocate to social information might occur in a schema-driven processing manner according to their attachment pattern. A major source of social information for infants consists of facial expressions of emotion. We tested for differences in attention to facial expressions and emotional discrimination between infants classified as securely attached (B), insecure-avoidant (A), and insecure-resistant (C).

Methods

Sixty-one 14-month-old infants participated in the Strange Situation Procedure and an experimental task of Visual Habituation and Visual Paired-Comparison Task (VPC). In the Habituation phase, a Low-Arousal Happy face (habituation face) was presented followed by a VPC task of 6 trials composed of two contrasting emotional faces always involving the same actress: the one used in habituation (trial old face) and a new one (trial new face) portraying changes in valence (Low-Arousal Angry face), arousal (High-Arousal Happy face), or valence + arousal (High-Arousal Angry face). Measures of fixation time (FT) and number of fixations (FC) were obtained for the habituation face, the trial old face, the trial new face, and the difference between the trial old face and the trial new face using an eye-tracking system.

Results

We found a higher FT and FC for the trial new face when compared with the trial old face, regardless of the emotional condition (valence, arousal, valence + arousal contrasts), suggesting that 14-month-old infants were able to discriminate different emotional faces. However, this effect differed according to attachment pattern: resistant-attached infants (C) had significantly higher FT and FC for the new face than patterns B and A, indicating they may remain hypervigilant toward emotional change. On the contrary, avoidant infants (A) revealed significantly longer looking times to the trial old face, suggesting overall avoidance of novel expressions and thus less sensitivity to emotional change.

Discussion

Overall, these findings corroborate that attachment is associated with infants' social information processing.

Healthy and Happy? An Ethical Investigation of Emotion Recognition and Regulation Technologies (ERR) within Ambient Assisted Living (AAL)

Ambient Assisted Living (AAL) refers to technologies that track daily activities of persons in need of care to enhance their autonomy and minimise their need for assistance. New technological developments show an increasing effort to integrate automated emotion recognition and regulation (ERR) into AAL systems. These technologies aim to recognise emotions via different sensors and, eventually, to regulate emotions defined as "negative" via different forms of intervention. Although these technologies are already implemented in other areas, AAL stands out by its tendency to enable an inconspicuous 24-hour surveillance in the private living space of users who rely on the technology to maintain a certain degree of independence in their daily activities. The combination of both technologies represents a new dimension of emotion recognition in a potentially vulnerable group of users. Our paper aims to provide an ethical contextualisation of the novel combination of both technologies. We discuss different concepts of emotions, namely Basic Emotion Theory (BET) and the Circumplex Model of Affect (CMA), that form the basis of ERR and provide an overview over the current technological developments in AAL. We highlight four ethical issues that specifically arise in the context of ERR in AAL systems, namely concerns regarding (1) the reductionist view of emotions, (2) solutionism as an underlying assumption of these technologies, (3) the privacy and autonomy of users and their emotions, (4) the tendency of machine learning techniques to normalise and generalise human behaviour and emotional reactions.

The Sound of Emotional Prosody: Nearly 3 Decades of Research and Future Directions

Emotional voices attract considerable attention. A search on any browser using "emotional prosody" as a key phrase leads to more than a million entries. Such interest is evident in the scientific literature as well; readers are reminded in the introductory paragraphs of countless articles of the great importance of prosody and that listeners easily infer the emotional state of speakers through acoustic information. However, despite decades of research on this topic and important achievements, the mapping between acoustics and emotional states is still unclear. In this article, we chart the rich literature on emotional prosody for both newcomers to the field and researchers seeking updates. We also summarize problems revealed by a sample of the literature of the last decades and propose concrete research directions for addressing them, ultimately to satisfy the need for more mechanistic knowledge of emotional prosody.

Cultural facial expressions dynamically convey emotion category and intensity information

Communicating emotional intensity plays a vital ecological role because it provides valuable information about the nature and likelihood of the sender's behavior.1,2,3 For example, attack often follows signals of intense aggression if receivers fail to retreat.4,5 Humans regularly use facial expressions to communicate such information.6,7,8,9,10,11 Yet how this complex signaling task is achieved remains unknown. We addressed this question using a perception-based, data-driven method to mathematically model the specific facial movements that receivers use to classify the six basic emotions-"happy," "surprise," "fear," "disgust," "anger," and "sad"-and judge their intensity in two distinct cultures (East Asian, Western European; total n = 120). In both cultures, receivers expected facial expressions to dynamically represent emotion category and intensity information over time, using a multi-component compositional signaling structure. Specifically, emotion intensifiers peaked earlier or later than emotion classifiers and represented intensity using amplitude variations. Emotion intensifiers are also more similar across emotions than classifiers are, suggesting a latent broad-plus-specific signaling structure. Cross-cultural analysis further revealed similarities and differences in expectations that could impact cross-cultural communication. Specifically, East Asian and Western European receivers have similar expectations about which facial movements represent high intensity for threat-related emotions, such as "anger," "disgust," and "fear," but differ on those that represent low threat emotions, such as happiness and sadness. Together, our results provide new insights into the intricate processes by which facial expressions can achieve complex dynamic signaling tasks by revealing the rich information embedded in facial expressions.

State-Aware Deep Item Response Theory using student facial features

This paper introduces a novel approach to Item Response Theory (IRT) by incorporating deep learning to analyze student facial expressions to enhance the prediction and understanding of student responses to test items. This research is based on the assertion that students' facial expressions offer crucial insights into their cognitive and affective states during testing, subsequently influencing their item responses. The proposed State-Aware Deep Item Response Theory (SAD-IRT) model introduces a new parameter, the student state parameter, which can be viewed as a relative subjective difficulty parameter. It is latent-regressed from students' facial features while solving test items using state-of-the-art deep learning techniques. In an experiment with 20 students, SAD-IRT boosted prediction performance in students' responses compared to prior models without the student state parameter, including standard IRT and its deep neural network implementation, while maintaining consistent predictions of student ability and item difficulty parameters. The research further illustrates the model's early prediction ability in predicting the student's response result before the student answered. This study holds substantial implications for educational assessment, laying the groundwork for more personalized and effective learning and assessment strategies that consider students' emotional and cognitive states.

Multimodal investigations of emotional face processing and social trait judgment of faces

Faces are among the most important visual stimuli that humans perceive in everyday life. While extensive literature has examined emotional processing and social evaluations of faces, most studies have examined either topic using unimodal approaches. In this review, we promote the use of multimodal cognitive neuroscience approaches to study these processes, using two lines of research as examples: ambiguity in facial expressions of emotion and social trait judgment of faces. In the first set of studies, we identified an event-related potential that signals emotion ambiguity using electroencephalography and we found convergent neural responses to emotion ambiguity using functional neuroimaging and single-neuron recordings. In the second set of studies, we discuss how different neuroimaging and personality-dimensional approaches together provide new insights into social trait judgments of faces. In both sets of studies, we provide an in-depth comparison between neurotypicals and people with autism spectrum disorder. We offer a computational account for the behavioral and neural markers of the different facial processing between the two groups. Finally, we suggest new practices for studying the emotional processing and social evaluations of faces. All data discussed in the case studies of this review are publicly available.

Understanding Naturalistic Facial Expressions with Deep Learning and Multimodal Large Language Models

This paper provides a comprehensive overview of affective computing systems for facial expression recognition (FER) research in naturalistic contexts. The first section presents an updated account of user-friendly FER toolboxes incorporating state-of-the-art deep learning models and elaborates on their neural architectures, datasets, and performances across domains. These sophisticated FER toolboxes can robustly address a variety of challenges encountered in the wild such as variations in illumination and head pose, which may otherwise impact recognition accuracy. The second section of this paper discusses multimodal large language models (MLLMs) and their potential applications in affective science. MLLMs exhibit human-level capabilities for FER and enable the quantification of various contextual variables to provide context-aware emotion inferences. These advancements have the potential to revolutionize current methodological approaches for studying the contextual influences on emotions, leading to the development of contextualized emotion models.

Development of the RIKEN database for dynamic facial expressions with multiple angles

The development of facial expressions with sensing information is progressing in multidisciplinary fields, such as psychology, affective computing, and cognitive science. Previous facial datasets have not simultaneously dealt with multiple theoretical views of emotion, individualized context, or multi-angle/depth information. We developed a new facial database (RIKEN facial expression database) that includes multiple theoretical views of emotions and expressers' individualized events with multi-angle and depth information. The RIKEN facial expression database contains recordings of 48 Japanese participants captured using ten Kinect cameras at 25 events. This study identified several valence-related facial patterns and found them consistent with previous research investigating the coherence between facial movements and internal states. This database represents an advancement in developing a new sensing system, conducting psychological experiments, and understanding the complexity of emotional events.

The cross-race effect in automatic facial expression recognition violates measurement invariance

Emotion has been a subject undergoing intensive research in psychology and cognitive neuroscience over several decades. Recently, more and more studies of emotion have adopted automatic rather than manual methods of facial emotion recognition to analyze images or videos of human faces. Compared to manual methods, these computer-vision-based, automatic methods can help objectively and rapidly analyze a large amount of data. These automatic methods have also been validated and believed to be accurate in their judgments. However, these automatic methods often rely on statistical learning models (e.g., deep neural networks), which are intrinsically inductive and thus suffer from problems of induction. Specifically, the models that were trained primarily on Western faces may not generalize well to accurately judge Eastern faces, which can then jeopardize the measurement invariance of emotions in cross-cultural studies. To demonstrate such a possibility, the present study carries out a cross-racial validation of two popular facial emotion recognition systems-FaceReader and DeepFace-using two Western and two Eastern face datasets. Although both systems could achieve overall high accuracies in the judgments of emotion category on the Western datasets, they performed relatively poorly on the Eastern datasets, especially in recognition of negative emotions. While these results caution the use of these automatic methods of emotion recognition on non-Western faces, the results also suggest that the measurements of happiness outputted by these automatic methods are accurate and invariant across races and hence can still be utilized for cross-cultural studies of positive psychology.

Category Flexibility in Emotion Learning

Learners flexibly update category boundaries to adjust to the range of experiences they encounter. However, little is known about whether the degree of flexibility is consistent across domains. We examined whether categorization of social input, specifically emotions, is afforded more flexibility as compared to other biological input. To address this question, children (6-12 years; 32 female, 37 male; 7 Hispanic or Latino, 62 not Hispanic or Latino; 8 Black or African American, 14 multiracial, 46 White, 1 selected "other") categorized faces morphed from calm to upset and animals morphed from a horse to a cow across task phases that differed in the distribution of stimuli presented. Learners flexibly adjusted both emotion and animal category boundaries according to distributional information, yet children showed more flexibility when updating their category boundaries for emotions. These results provide support for the idea that children-who must adjust to the vast and varied emotional signals of their social partners-respond to social signals dynamically in order to make predictions about the internal states and future behaviors of others.

Py-Feat: Python Facial Expression Analysis Toolbox

Studying facial expressions is a notoriously difficult endeavor. Recent advances in the field of affective computing have yielded impressive progress in automatically detecting facial expressions from pictures and videos. However, much of this work has yet to be widely disseminated in social science domains such as psychology. Current state-of-the-art models require considerable domain expertise that is not traditionally incorporated into social science training programs. Furthermore, there is a notable absence of user-friendly and open-source software that provides a comprehensive set of tools and functions that support facial expression research. In this paper, we introduce Py-Feat, an open-source Python toolbox that provides support for detecting, preprocessing, analyzing, and visualizing facial expression data. Py-Feat makes it easy for domain experts to disseminate and benchmark computer vision models and also for end users to quickly process, analyze, and visualize face expression data. We hope this platform will facilitate increased use of facial expression data in human behavior research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s42761-023-00191-4.

Existential threat and responses to emotional displays of ingroup and outgroup members

The present research investigates how emotional displays shape reactions to ingroup and outgroup members when people are reminded of death. We hypothesized that under mortality salience, emotions that signal social distance promote worldview defense (i.e., increased ingroup favoritism and outgroup derogation), whereas emotions that signal affiliation promote affiliation need (i.e., reduced ingroup favoritism and outgroup derogation). In three studies, participants viewed emotional displays of ingroup and/or outgroup members after a mortality salience or control manipulation. Results revealed that under mortality salience, anger increased ingroup favoritism and outgroup derogation (Study 1), enhanced perceived overlap with the ingroup (Study 3), and increased positive facial behavior to ingroup displays-measured via the Facial Action Coding System (Studies 1 and 2) and electromyography of the zygomaticus major muscle (Study 3). In contrast, happiness decreased ingroup favoritism and outgroup derogation (Study 2), and increased positive facial behavior towards outgroup members (Study 3). The findings suggest that, in times of threat, emotional displays can determine whether people move away from unfamiliar others or try to form as many friendly relations as possible.

High-resolution surface electromyographic activities of facial muscles during the six basic emotional expressions in healthy adults: a prospective observational study

High-resolution facial surface electromyography (HR-sEMG) is suited to discriminate between different facial movements. Whether HR-sEMG also allows a discrimination among the six basic emotions of facial expression is unclear. 36 healthy participants (53% female, 18-67 years) were included for four sessions. Electromyograms were recorded from both sides of the face using a muscle-position oriented electrode application (Fridlund scheme) and by a landmark-oriented, muscle unrelated symmetrical electrode arrangement (Kuramoto scheme) simultaneously on the face. In each session, participants expressed the six basic emotions in response to standardized facial images expressing the corresponding emotions. This was repeated once on the same day. Both sessions were repeated two weeks later to assess repetition effects. HR-sEMG characteristics showed systematic regional distribution patterns of emotional muscle activation for both schemes with very low interindividual variability. Statistical discrimination between the different HR-sEMG patterns was good for both schemes for most but not all basic emotions (ranging from p > 0.05 to mostly p < 0.001) when using HR-sEMG of the entire face. When using information only from the lower face, the Kuramoto scheme allowed a more reliable discrimination of all six emotions (all p < 0.001). A landmark-oriented HR-sEMG recording allows specific discrimination of facial muscle activity patterns during basic emotional expressions.

The hole-board apparatus in the study of anxiety

Anxiety disorders pose a significant challenge in contemporary society, and their impact in terms of social and economic burden is overwhelming. Behavioral research conducted on animal subjects is crucial for comprehending these disorders and, from a translational standpoint, for introducing innovative therapeutic approaches. In this context, the Hole-Board apparatus has emerged as a widely utilized test for studying anxiety-related behaviors in rodents. Although a substantial body of literature underscores the utility and reliability of the Hole-Board in anxiety research, recent decades have witnessed a range of studies that have led to uncertainties and misinterpretations regarding the validity of this behavioral assay. The objective of this review is twofold: firstly, to underscore the utility and reliability of the Hole-Board assay, and concurrently, to examine the underlying factors contributing to potential misconceptions surrounding its utilization in the study of anxiety and anxiety-related behaviors. We will present results from both conventional quantitative analyses and multivariate approaches, while referencing a comprehensive collection of studies conducted using the Hole-Board.

Four Misconceptions About Nonverbal Communication

Research and theory in nonverbal communication have made great advances toward understanding the patterns and functions of nonverbal behavior in social settings. Progress has been hindered, we argue, by presumptions about nonverbal behavior that follow from both received wisdom and faulty evidence. In this article, we document four persistent misconceptions about nonverbal communication-namely, that people communicate using decodable body language; that they have a stable personal space by which they regulate contact with others; that they express emotion using universal, evolved, iconic, categorical facial expressions; and that they can deceive and detect deception, using dependable telltale clues. We show how these misconceptions permeate research as well as the practices of popular behavior experts, with consequences that extend from intimate relationships to the boardroom and courtroom and even to the arena of international security. Notwithstanding these misconceptions, existing frameworks of nonverbal communication are being challenged by more comprehensive systems approaches and by virtual technologies that ambiguate the roles and identities of interactants and the contexts of interaction.

Synchronized affect in shared experiences strengthens social connection

People structure their days to experience events with others. We gather to eat meals, watch TV, and attend concerts together. What constitutes a shared experience and how does it manifest in dyadic behavior? The present study investigates how shared experiences-measured through emotional, motoric, physiological, and cognitive alignment-promote social bonding. We recorded the facial expressions and electrodermal activity (EDA) of participants as they watched four episodes of a TV show for a total of 4 h with another participant. Participants displayed temporally synchronized and spatially aligned emotional facial expressions and the degree of synchronization predicted the self-reported social connection ratings between viewing partners. We observed a similar pattern of results for dyadic physiological synchrony measured via EDA and their cognitive impressions of the characters. All four of these factors, temporal synchrony of positive facial expressions, spatial alignment of expressions, EDA synchrony, and character impression similarity, contributed to a latent factor of a shared experience that predicted social connection. Our findings suggest that the development of interpersonal affiliations in shared experiences emerges from shared affective experiences comprising synchronous processes and demonstrate that these complex interpersonal processes can be studied in a holistic and multi-modal framework leveraging naturalistic experimental designs.

Higher social tolerance is associated with more complex facial behavior in macaques

The social complexity hypothesis for communicative complexity posits that animal societies with more complex social systems require more complex communication systems. We tested the social complexity hypothesis on three macaque species that vary in their degree of social tolerance and complexity. We coded facial behavior in >3000 social interactions across three social contexts (aggressive, submissive, affiliative) in 389 animals, using the Facial Action Coding System for macaques (MaqFACS). We quantified communicative complexity using three measures of uncertainty: entropy, specificity, and prediction error. We found that the relative entropy of facial behavior was higher for the more tolerant crested macaques as compared to the less tolerant Barbary and rhesus macaques across all social contexts, indicating that crested macaques more frequently use a higher diversity of facial behavior. The context specificity of facial behavior was higher in rhesus as compared to Barbary and crested macaques, demonstrating that Barbary and crested macaques used facial behavior more flexibly across different social contexts. Finally, a random forest classifier predicted social context from facial behavior with highest accuracy for rhesus and lowest for crested, indicating there is higher uncertainty and complexity in the facial behavior of crested macaques. Overall, our results support the social complexity hypothesis.

Brain mechanisms associated with facial encoding of affective states

Affective states are typically accompanied by facial expressions, but these behavioral manifestations are highly variable. Even highly arousing and negative valent experiences, such as pain, show great instability in facial affect encoding. The present study investigated which neural mechanisms are associated with variations in facial affect encoding by focusing on facial encoding of sustained pain experiences. Facial expressions, pain ratings, and brain activity (BOLD-fMRI) during tonic heat pain were recorded in 27 healthy participants. We analyzed facial expressions by using the Facial Action Coding System (FACS) and examined brain activations during epochs of painful stimulation that were accompanied by facial expressions of pain. Epochs of facial expressions of pain were coupled with activity increase in motor areas (M1, premotor and SMA) as well as in areas involved in nociceptive processing, including primary and secondary somatosensory cortex, posterior and anterior insula, and the anterior part of the mid-cingulate cortex. In contrast, prefrontal structures (ventrolateral and medial prefrontal) were less activated during incidences of facial expressions, consistent with a role in down-regulating facial displays. These results indicate that incidences of facial encoding of pain reflect activity within nociceptive pathways interacting or possibly competing with prefrontal inhibitory systems that gate the level of expressiveness.

Transition dynamics shape mental state concepts

People have a unique ability to represent other people's internal thoughts and feelings-their mental states. Mental state knowledge has a rich conceptual structure, organized along key dimensions, such as valence. People use this conceptual structure to guide social interactions. How do people acquire their understanding of this structure? Here we investigate an underexplored contributor to this process: observation of mental state dynamics. Mental states-including both emotions and cognitive states-are not static. Rather, the transitions from one state to another are systematic and predictable. Drawing on prior cognitive science, we hypothesize that these transition dynamics may shape the conceptual structure that people learn to apply to mental states. Across nine behavioral experiments (N = 1,439), we tested whether the transition probabilities between mental states causally shape people's conceptual judgments of those states. In each study, we found that observing frequent transitions between mental states caused people to judge them to be conceptually similar. Computational modeling indicated that people translated mental state dynamics into concepts by embedding the states as points within a geometric space. The closer two states are within this space, the greater the likelihood of transitions between them. In three neural network experiments, we trained artificial neural networks to predict real human mental state dynamics. The networks spontaneously learned the same conceptual dimensions that people use to understand mental states. Together these results indicate that mental state dynamics-and the goal of predicting them-shape the structure of mental state concepts. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Facial expression of patients with Graves' orbitopathy

PURPOSE

Patients with Graves' orbitopathy (GO) have characteristic facial expressions that are different from those of healthy individuals due to the combination of somatic and psychiatric symptoms. However, the facial expressions of GO patients have not yet been described and analyzed systematically. Thus, the present study aimed to present the facial expressions of GO patients and explore their applications in clinical practice.

METHODS

Facial image and clinical data of 943 GO patients were included, and 126 patients answered quality of life (GO-QOL) questionnaires. Each patient was labeled for one facial expression. Then, a portrait was drawn for every facial expression. Logistic and linear regression was performed to analyze the correlation between facial expression and clinical indicators, including QOL, disease activity and severity. The VGG-19 network model was utilized to discriminate facial expressions automatically.

RESULTS

Two groups, i.e., the non-negative emotion (neutral, happy) and the negative emotion (disgust, angry, fear, sadness, surprise), and seven expressions of GO patients were systematically analyzed. Facial expression was statistically associated with GO activity (P = 0.002), severity (P < 0.001), QOL visual functioning subscale scores (P = 0.001), and QOL appearance subscale score (P = 0.012). The deep learning model achieved satisfactory results (accuracy 0.851, sensitivity 0.899, precision 0.899, specificity 0.720, F1 score 0.899, and AUC 0.847).

CONCLUSIONS

As a novel clinical sign, facial expression holds the potential to be incorporated into GO assessment system in the future. The discrimination model may assist clinicians in real-life patient care.

The Ethics of Emotional Artificial Intelligence: A Mixed Method Analysis

Emotions play a significant role in human relations, decision-making, and the motivation to act on those decisions. There are ongoing attempts to use artificial intelligence (AI) to read human emotions, and to predict human behavior or actions that may follow those emotions. However, a person's emotions cannot be easily identified, measured, and evaluated by others, including automated machines and algorithms run by AI. The ethics of emotional AI is under research and this study has examined the emotional variables as well as the perception of emotional AI in two large random groups of college students in an international university in Japan, with a heavy representation of Japanese, Indonesian, Korean, Chinese, Thai, Vietnamese, and other Asian nationalities. Surveys with multiple close-ended questions and an open-ended essay question regarding emotional AI were administered for quantitative and qualitative analysis, respectively. The results demonstrate how ethically questionable results may be obtained through affective computing and by searching for correlations in a variety of factors in collected data to classify individuals into certain categories and thus aggravate bias and discrimination. Nevertheless, the qualitative study of students' essays shows a rather optimistic view over the use of emotional AI, which helps underscore the need to increase awareness about the ethical pitfalls of AI technologies in the complex field of human emotions.

Emotional content and sharing on Facebook: A theory cage match

While emotional content predicts social media post sharing, competing theories of emotion imply different predictions about how emotional content will influence the virality of social media posts. We tested and compared these theoretical frameworks. Teams of annotators assessed more than 4000 multimedia posts from Polish and Lithuanian Facebook for more than 20 emotions. We found that, drawing on semantic space theory, modeling discrete emotions independently was superior to models examining valence (positive or negative), activation/arousal (high or low), or clusters of emotions and was on par with but had more explanatory power than a seven basic emotion model. Certain discrete emotions were associated with post sharing, including both positive and negative and relatively lower and higher activation/arousal emotions (e.g., amusement, cute/kama muta, anger, and sadness) even when controlling for number of followers, time up, topic, and Facebook angry reactions. These results provide key insights into better understanding of social media post virality.

Human and machine recognition of dynamic and static facial expressions: prototypicality, ambiguity, and complexity

A growing body of research suggests that movement aids facial expression recognition. However, less is known about the conditions under which the dynamic advantage occurs. The aim of this research was to test emotion recognition in static and dynamic facial expressions, thereby exploring the role of three featural parameters (prototypicality, ambiguity, and complexity) in human and machine analysis. In two studies, facial expression videos and corresponding images depicting the peak of the target and non-target emotion were presented to human observers and the machine classifier (FACET). Results revealed higher recognition rates for dynamic stimuli compared to non-target images. Such benefit disappeared in the context of target-emotion images which were similarly well (or even better) recognised than videos, and more prototypical, less ambiguous, and more complex in appearance than non-target images. While prototypicality and ambiguity exerted more predictive power in machine performance, complexity was more indicative of human emotion recognition. Interestingly, recognition performance by the machine was found to be superior to humans for both target and non-target images. Together, the findings point towards a compensatory role of dynamic information, particularly when static-based stimuli lack relevant features of the target emotion. Implications for research using automatic facial expression analysis (AFEA) are discussed.

Social media emotions annotation guide (SMEmo): Development and initial validity

The proper measurement of emotion is vital to understanding the relationship between emotional expression in social media and other factors, such as online information sharing. This work develops a standardized annotation scheme for quantifying emotions in social media using recent emotion theory and research. Human annotators assessed both social media posts and their own reactions to the posts' content on scales of 0 to 100 for each of 20 (Study 1) and 23 (Study 2) emotions. For Study 1, we analyzed English-language posts from Twitter (N = 244) and YouTube (N = 50). Associations between emotion ratings and text-based measures (LIWC, VADER, EmoLex, NRC-EIL, Emotionality) demonstrated convergent and discriminant validity. In Study 2, we tested an expanded version of the scheme in-country, in-language, on Polish (N = 3648) and Lithuanian (N = 1934) multimedia Facebook posts. While the correlations were lower than with English, patterns of convergent and discriminant validity with EmoLex and NRC-EIL still held. Coder reliability was strong across samples, with intraclass correlations of .80 or higher for 10 different emotions in Study 1 and 16 different emotions in Study 2. This research improves the measurement of emotions in social media to include more dimensions, multimedia, and context compared to prior schemes.