Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements

Relationships between Nursing Students' Skill Mastery, Test Anxiety, Self-Efficacy, and Facial Expressions: A Preliminary Observational Study

Test anxiety and self-efficacy significantly influence the mastery of nursing skills. Facial expression recognition tools are central components to recognising these elements. This study investigated the frequent facial expressions conveyed by nursing students and examined the relationships between nursing skill mastery, test anxiety, self-efficacy, and facial expressions in a test-taking situation. Thirty-three second-year nursing students who were attending a university in a Korean metropolitan city participated. Test anxiety, self-efficacy, and facial expressions were collected while the students inserted indwelling catheters. Using Microsoft Azure software, the researchers examined the students' facial expressions. Negative facial expressions, such as anger, disgust, sadness, and surprise, were more common during the test-taking situation than the practice trial. Fear was positively correlated with anxiety. None of the facial expressions had significant relationships with self-efficacy; however, disgust was positively associated with nursing skill mastery. The facial expressions during the practice and test-taking situations were similar; however, fear and disgust may have been indicators of test anxiety and skill mastery. To create a screening tool for detecting and caring for students' emotions, further studies should explore students' facial expressions that were not evaluated in this study.

Emotion words link faces to emotional scenarios in early childhood

Recent theories have suggested that emotion words may facilitate the development of emotion concepts. The present study investigates whether emotion words affect children's performance on an emotion category learning task. Across two experiments, 72 three-year-old children (49 female) were asked to identify which emotional face best matched particular emotional scenarios during nine pretest and nine posttest trials. The scenarios in the present studies aligned with emotions typically learned among older age groups (annoyed, disgusted, and nervous). Between pretest and posttest, children participated in training in which a facial configuration (annoyed, disgusted, or nervous) was paired with an associated scenario while they heard the emotion labeled explicitly or heard irrelevant information (Experiment 1) or heard a broad emotion label versus irrelevant information (Experiment 2). Aside from the labels presented, all other information was kept the same across conditions, including the specific faces and scenarios heard during learning trials. In Experiment 1, children's emotion understanding increased more from pretest to posttest in the explicit label versus irrelevant condition, t(34) = 2.26, p = .030, d = .75, but in Experiment 2 the broad emotion labels did not provide an advantage over irrelevant information, t(34) = .72, p = .474, d = .24. These results suggest that emotion labels may be particularly helpful for young children learning about unfamiliar emotions, because specific labels may help children to aggregate disparate emotional information into meaningful categories. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Children track probabilistic distributions of facial cues across individuals

Children face a difficult task in learning how to reason about other people's emotions. How intensely facial configurations are displayed can vary not only according to what and how much emotion people are experiencing, but also across individuals based on differences in personality, gender, and culture. To navigate these sources of variability, children may use statistical information about other's facial cues to make interpretations about perceived emotions in others. We examined this possibility by testing children's ability to adjust to differences in the intensity of facial cues across different individuals. In the present study, children (6- to 10-year-olds) categorized the information communicated by facial configurations of emotion varying continuously from "calm" to "upset," with differences in the intensity of each actor's facial movements. We found that children's threshold for categorizing a facial configuration as "upset" shifted depending on the statistical information encountered about each of the different individuals. These results suggest that children are able to track individual differences in facial behavior and use these differences to flexibly update their interpretations of facial cues associated with emotion. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

AI as customer

Purpose

The purpose of the paper is to note that customers are not necessarily human and to figure out how best to serve artificial intelligence (AI) customers. The authors also propose several major research streams, as examples, to help launch research on AI customers and how to serve them.

Design/methodology/approach

The current paper is a conceptual one that draws upon research from many areas to support the ideas proposed.

Findings

AI customer are proliferating. AI as customers can augment or replace human customers and can be the customer itself. Service providers may also be AI, which means that both humans serving AI customers and AI serving AI customers are relevant here. The authors show that even truly autonomous AI customers are likely to be more common in the future. The authors conclude that reverse engineering will probably not be successful in understanding AI customers and that an approach similar to how we research human consumer behavior is likely to be more useful.

Originality/value

Virtually, the entire literature on customers and how to serve them assumes that customers are human. With the rapid advancement of AI, purchase decisions are increasingly made by AI, suggesting that it is now important and necessary to consider the possibility of AI customers and how best to serve them. This paper opens the door for such research.

Spotting When Algorithms Are Wrong

Users of sociotechnical systems often have no way to independently verify whether the system output which they use to make decisions is correct; they are epistemically dependent on the system. We argue that this leads to problems when the system is wrong, namely to bad decisions and violations of the norm of practical reasoning. To prevent this from occurring we suggest the implementation of defeaters: information that a system is unreliable in a specific case (undercutting defeat) or independent information that the output is wrong (rebutting defeat). Practically, we suggest to design defeaters based on the different ways in which a system might produce erroneous outputs, and analyse this suggestion with a case study of the risk classification algorithm used by the Dutch tax agency.

The face is central to primate multicomponent signals

A wealth of experimental and observational evidence suggests that faces have become increasingly important in the communication system of primates over evolutionary time and that both the static and moveable aspects of faces convey considerable information. Therefore, whenever there is a visual component to any multicomponent signal the face is potentially relevant. However, the role of the face is not always considered in primate multicomponent communication research. We review the literature and make a case for greater focus on the face going forward. We propose that the face can be overlooked for two main reasons: first, due to methodological difficulty. Examination of multicomponent signals in primates is difficult, so scientists tend to examine a limited number of signals in combination. Detailed examination of the subtle and dynamic components of facial signals is particularly hard to achieve in studies of primates. Second, due to a common assumption that the face contains “emotional” content. A priori categorisation of facial behavior as “emotional” ignores the potentially communicative and predictive information present in the face that might contribute to signals. In short, we argue that the face is central to multicomponent signals (and also many multimodal signals) and suggest future directions for investigating this phenomenon.

Automatic Coding of Facial Expressions of Pain: Are We There Yet?

Introduction

The experience of pain is regularly accompanied by facial expressions. The gold standard for analyzing these facial expressions is the Facial Action Coding System (FACS), which provides so-called action units (AUs) as parametrical indicators of facial muscular activity. Particular combinations of AUs have appeared to be pain-indicative. The manual coding of AUs is, however, too time- and labor-intensive in clinical practice. New developments in automatic facial expression analysis have promised to enable automatic detection of AUs, which might be used for pain detection.

Objective

Our aim is to compare manual with automatic AU coding of facial expressions of pain.

Methods

FaceReader7 was used for automatic AU detection. We compared the performance of FaceReader7 using videos of 40 participants (20 younger with a mean age of 25.7 years and 20 older with a mean age of 52.1 years) undergoing experimentally induced heat pain to manually coded AUs as gold standard labeling. Percentages of correctly and falsely classified AUs were calculated, and we computed as indicators of congruency, “sensitivity/recall,” “precision,” and “overall agreement (F1).”

Results

The automatic coding of AUs only showed poor to moderate outcomes regarding sensitivity/recall, precision, and F1. The congruency was better for younger compared to older faces and was better for pain-indicative AUs compared to other AUs.

Conclusion

At the moment, automatic analyses of genuine facial expressions of pain may qualify at best as semiautomatic systems, which require further validation by human observers before they can be used to validly assess facial expressions of pain.

Facial expressions elicit multiplexed perceptions of emotion categories and dimensions

Human facial expressions are complex, multi-component signals that can communicate rich information about emotions,1-5 including specific categories, such as "anger," and broader dimensions, such as "negative valence, high arousal."6-8 An enduring question is how this complex signaling is achieved. Communication theory predicts that multi-component signals could transmit each type of emotion information-i.e., specific categories and broader dimensions-via the same or different facial signal components, with implications for elucidating the system and ontology of facial expression communication.9 We addressed this question using a communication-systems-based method that agnostically generates facial expressions and uses the receiver's perceptions to model the specific facial signal components that represent emotion category and dimensional information to them.10-12 First, we derived the facial expressions that elicit the perception of emotion categories (i.e., the six classic emotions13 plus 19 complex emotions3) and dimensions (i.e., valence and arousal) separately, in 60 individual participants. Comparison of these facial signals showed that they share subsets of components, suggesting that specific latent signals jointly represent-i.e., multiplex-categorical and dimensional information. Further examination revealed these specific latent signals and the joint information they represent. Our results-based on white Western participants, same-ethnicity face stimuli, and commonly used English emotion terms-show that facial expressions can jointly represent specific emotion categories and broad dimensions to perceivers via multiplexed facial signal components. Our results provide insights into the ontology and system of facial expression communication and a new information-theoretic framework that can characterize its complexities.

Unsupervised Learning of Brain State Dynamics during Emotion Imagination using High-Density EEG

This study applies adaptive mixture independent component analysis (AMICA) to learn a set of ICA models, each optimized by fitting a distributional model for each identified component process while maximizing component process independence within some subsets of time points of a multi-channel EEG dataset. Here, we applied 20-model AMICA decomposition to long-duration (1-2 hr), high-density (128-channel) EEG data recorded while participants used guided imagination to imagine situations stimulating the experience of 15 specified emotions. These decompositions tended to return models identifying spatiotemporal EEG patterns or states within single emotion imagination periods. Model probability transitions reflected time-courses of EEG dynamics during emotion imagination, which varied across emotions. Transitions between models accounting for imagined "grief" and "happiness" were more abrupt and better aligned with participant reports, while transitions for imagined "contentment" extended into adjoining "relaxation" periods. The spatial distributions of brain-localizable independent component processes (ICs) were more similar within participants (across emotions) than emotions (across participants). Across participants, brain regions with differences in IC spatial distributions (i.e., dipole density) between emotion imagination versus relaxation were identified in or near the left rostrolateral prefrontal, posterior cingulate cortex, right insula, bilateral sensorimotor, premotor, and associative visual cortex. No difference in dipole density was found between positive versus negative emotions. AMICA models of changes in high-density EEG dynamics may allow data-driven insights into brain dynamics during emotional experience, possibly enabling the improved performance of EEG-based emotion decoding and advancing our understanding of emotion.

AffectiveTDA: Using Topological Data Analysis to Improve Analysis and Explainability in Affective Computing

We present an approach utilizing Topological Data Analysis to study the structure of face poses used in affective computing, i.e., the process of recognizing human emotion. The approach uses a conditional comparison of different emotions, both respective and irrespective of time, with multiple topological distance metrics, dimension reduction techniques, and face subsections (e.g., eyes, nose, mouth, etc.). The results confirm that our topology-based approach captures known patterns, distinctions between emotions, and distinctions between individuals, which is an important step towards more robust and explainable emotion recognition by machines.

Digital Technologies for Emotion-Regulation Assessment and Intervention: A Conceptual Review

The ability to regulate emotions in response to stress is central to healthy development. While early research in emotion regulation predominantly employed static, self-report measurement, the past decade has seen a shift in focus toward understanding the dynamic nature of regulation processes. This is reflected in recent refinements in the definition of emotion regulation, which emphasize the importance of the ability to flexibly adapt regulation efforts across contexts. The latest proliferation of digital technologies employed in mental health research offers the opportunity to capture the state- and context-sensitive nature of emotion regulation. In this conceptual review, we examine the use of digital technologies (ecological momentary assessment; wearable and smartphone technology, physical activity, acoustic data, visual data, and geo-location; smart home technology; virtual reality; social media) in the assessment of emotion regulation and describe their application to interventions. We also discuss challenges and ethical considerations, and outline areas for future research.

Investigating the Role of Culture on Negative Emotion Expressions in the Wild

Even though culture has been found to play some role in negative emotion expression, affective computing research primarily takes on a basic emotion approach when analyzing social signals for automatic emotion recognition technologies. Furthermore, automatic negative emotion recognition systems still train data that originates primarily from North America and contains a majority of Caucasian training samples. As such, the current study aims to address this problem by analyzing what the differences are of the underlying social signals by leveraging machine learning models to classify 3 negative emotions, contempt, anger and disgust (CAD) amongst 3 different cultures: North American, Persian, and Filipino. Using a curated data set compiled from YouTube videos, a support vector machine (SVM) was used to predict negative emotions amongst differing cultures. In addition a one-way ANOVA was used to analyse the differences that exist between each culture group in-terms of level of activation of underlying social signal. Our results not only highlighted the significant differences in the associated social signals that were activated for each culture, but also indicated the specific underlying social signals that differ in our cross-cultural data sets. Furthermore, the automatic classification methods showed North American expressions of CAD to be well-recognized, while Filipino and Persian expressions were recognized at near chance levels.

Comparing supervised and unsupervised approaches to multimodal emotion recognition

We investigated emotion classification from brief video recordings from the GEMEP database wherein actors portrayed 18 emotions. Vocal features consisted of acoustic parameters related to frequency, intensity, spectral distribution, and durations. Facial features consisted of facial action units. We first performed a series of person-independent supervised classification experiments. Best performance (AUC = 0.88) was obtained by merging the output from the best unimodal vocal (Elastic Net, AUC = 0.82) and facial (Random Forest, AUC = 0.80) classifiers using a late fusion approach and the product rule method. All 18 emotions were recognized with above-chance recall, although recognition rates varied widely across emotions (e.g., high for amusement, anger, and disgust; and low for shame). Multimodal feature patterns for each emotion are described in terms of the vocal and facial features that contributed most to classifier performance. Next, a series of exploratory unsupervised classification experiments were performed to gain more insight into how emotion expressions are organized. Solutions from traditional clustering techniques were interpreted using decision trees in order to explore which features underlie clustering. Another approach utilized various dimensionality reduction techniques paired with inspection of data visualizations. Unsupervised methods did not cluster stimuli in terms of emotion categories, but several explanatory patterns were observed. Some could be interpreted in terms of valence and arousal, but actor and gender specific aspects also contributed to clustering. Identifying explanatory patterns holds great potential as a meta-heuristic when unsupervised methods are used in complex classification tasks.

Toddler Word Learning is Robust to Changes in Emotional Context

Word learning is a crucial aspect of early social and cognitive development, and previous research indicates that children's word learning is influenced by the context in which the word is spoken. However, the role of emotions as contextual cues to word learning remains less clear. The present study investigated word learning among 2.5-year-old children in angry, happy, sad, and variable emotional contexts. Fifty-six children (30 female; Mean age=2.49 years) participated in a novel noun generalization task in which children observed an experimenter labeling objects in either a consistently angry, consistently happy, consistently sad, or variable (one exemplar per emotion) context. Children were then asked to identify the label-object association. Results revealed that children's performance was above chance levels for all four conditions (all t's>3.68, all p's<.01), but performance did not significantly differ by condition (F(3,52)=0.51, p=.677). These results provide valuable information regarding potential boundaries for when contextual information may versus may not influence children's word learning.

Toward a Theory of Emotions in Competitive Sports

In this article, we introduce a theory on the dynamic development of affective processes, affect regulation, and the relationship between emotions and sport performance. The theory focusses on how affective processes emerge and develop during competitive sport involvement. Based on Scherer's component process model, we postulate six components of emotion that interact with each other in a circular fashion: (I) triggering processes, (II) physiological reactions, (III) action tendencies, (IV) expressive behaviors, (V) subjective experience, and (VI) higher cognitive processes. The theory stresses the dynamics of affective processes and describes the consequences for performance in competitive sports. It assumes that the peculiarities of different sports must be taken into account in order to understand the affective processes, and offers starting points on which strategies can be used to effectively regulate affective states. Consequences for research and practice are derived and discussed. To study the development of affective processes, future research should test the assumptions in ecologically valid contexts, such as real competitions or competition-like situations, using multi-component measures of emotions.

A Systematic Review of the Role of Oxytocin, Cortisol, and Testosterone in Facial Emotional Processing

A topic of interest is the way decoding and interpreting facial emotional expressions can lead to mutual understanding. Facial emotional expression is a basic source of information that guarantees the functioning of other higher cognitive processes (e.g., empathy, cooperativity, prosociality, or decision-making, among others). In this regard, hormones such as oxytocin, cortisol, and/or testosterone have been found to be important in modifying facial emotion processing. In fact, brain structures that participate in facial emotion processing have been shown to be rich in receptors for these hormones. Nonetheless, much of this research has been based on correlational designs. In recent years, a growing number of researchers have tried to carry out controlled laboratory manipulation of these hormones by administering synthetic forms of these hormones. The main objective of this study was to carry out a systematic review of studies that assess whether manipulation of these three hormones effectively promotes significant alterations in facial emotional processing. To carry out this review, PRISMA quality criteria for reviews were followed, using the following digital databases: PsycINFO, PubMed, Dialnet, Psicodoc, Web of Knowledge, and the Cochrane Library, and focusing on manuscripts with a robust research design (e.g., randomized, single- or double-blind, and/or placebo-controlled) to increase the value of this systematic review. An initial identification of 6340 abstracts and retrieval of 910 full texts led to the final inclusion of 101 papers that met all the inclusion criteria. Only about 18% of the manuscripts included reported a direct effect of hormone manipulation. In fact, emotional accuracy seemed to be enhanced after oxytocin increases, but it diminished when cortisol and/or testosterone increased. Nonetheless, when emotional valence and participants' gender were included, hormonal manipulation reached significance (in around 53% of the articles). In fact, these studies offered a heterogeneous pattern in the way these hormones altered speed processing, attention, and memory. This study reinforces the idea that these hormones are important, but not the main modulators of facial emotion processing. As our comprehension of hormonal effects on emotional processing improves, the potential to design good treatments to improve this ability will be greater.

Interpersonal Consequences of Deceptive Expressions of Sadness

Emotional expressions evoke predictable responses from observers; displays of sadness are commonly met with sympathy and help from others. Accordingly, people may be motivated to feign emotions to elicit a desired response. In the absence of suspicion, we predicted that emotional and behavioral responses to genuine (vs. deceptive) expressers would be guided by empirically valid cues of sadness authenticity. Consistent with this hypothesis, untrained observers (total N = 1,300) reported less sympathy and offered less help to deceptive (vs. genuine) expressers of sadness. This effect was replicated using both posed, low-stakes, laboratory-created stimuli, and spontaneous, real, high-stakes emotional appeals to the public. Furthermore, lens models suggest that sympathy reactions were guided by difficult-to-fake facial actions associated with sadness. Results suggest that naive observers use empirically valid cues to deception to coordinate social interactions, providing novel evidence that people are sensitive to subtle cues to deception.

Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks

Facial expressions are one of the most powerful ways to depict specific patterns in human behavior and describe the human emotional state. However, despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot correctly handle variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task, even for humans. This paper investigates the suitability of state-of-the-art deep learning architectures based on convolutional neural networks (CNNs) to deal with long video sequences captured in the wild for continuous emotion recognition. For such an aim, several 2D CNN models that were designed to model spatial information are extended to allow spatiotemporal representation learning from videos, considering a complex and multi-dimensional emotion space, where continuous values of valence and arousal must be predicted. We have developed and evaluated convolutional recurrent neural networks, combining 2D CNNs and long short term-memory units and inflated 3D CNN models, which are built by inflating the weights of a pre-trained 2D CNN model during fine-tuning, using application-specific videos. Experimental results on the challenging SEWA-DB dataset have shown that these architectures can effectively be fine-tuned to encode spatiotemporal information from successive raw pixel images and achieve state-of-the-art results on such a dataset.

A Cautionary Note on Predicting Social Judgments from Faces with Deep Neural Networks

People spontaneously infer other people’s psychology from faces, encompassing inferences of their affective states, cognitive states, and stable traits such as personality. These judgments are known to be often invalid, but nonetheless bias many social decisions. Their importance and ubiquity have made them popular targets for automated prediction using deep convolutional neural networks (DCNNs). Here, we investigated the applicability of this approach: how well does it generalize, and what biases does it introduce? We compared three distinct sets of features (from a face identification DCNN, an object recognition DCNN, and using facial geometry), and tested their prediction across multiple out-of-sample datasets. Across judgments and datasets, features from both pre-trained DCNNs provided better predictions than did facial geometry. However, predictions using object recognition DCNN features were not robust to superficial cues (e.g., color and hair style). Importantly, predictions using face identification DCNN features were not specific: models trained to predict one social judgment (e.g., trustworthiness) also significantly predicted other social judgments (e.g., femininity and criminal), and at an even higher accuracy in some cases than predicting the judgment of interest (e.g., trustworthiness). Models trained to predict affective states (e.g., happy) also significantly predicted judgments of stable traits (e.g., sociable), and vice versa. Our analysis pipeline not only provides a flexible and efficient framework for predicting affective and social judgments from faces but also highlights the dangers of such automated predictions: correlated but unintended judgments can drive the predictions of the intended judgments.

Lie to my face: An electromyography approach to the study of deceptive behavior

BACKGROUND

Deception is present in all walks of life, from social interactions to matters of homeland security. Nevertheless, reliable indicators of deceptive behavior in real-life scenarios remain elusive.

METHODS

By integrating electrophysiological and communicative approaches, we demonstrate a new and objective detection approach to identify participant-specific indicators of deceptive behavior in an interactive scenario of a two-person deception task. We recorded participants' facial muscle activity using novel dry screen-printed electrode arrays and applied machine-learning algorithms to identify lies based on brief facial responses.

RESULTS

With an average accuracy of 73%, we identified two groups of participants: Those who revealed their lies by activating their cheek muscles and those who activated their eyebrows. We found that the participants lied more often with time, with some switching their telltale muscle groups. Moreover, while the automated classifier, reported here, outperformed untrained human detectors, their performance was correlated, suggesting reliance on shared features.

CONCLUSIONS

Our findings demonstrate the feasibility of using wearable electrode arrays in detecting human lies in a social setting and set the stage for future research on individual differences in deception expression.

The representation of emotion knowledge across development

The present study examined how children spontaneously represent facial cues associated with emotion. 106 three‐ to six‐year‐old children (48 male, 58 female; 9.4% Asian, 84.0% White, 6.6% more than one race) and 40 adults (10 male, 30 female; 10% Hispanic, 30% Asian, 2.5% Black, 57.5% White) were recruited from a Midwestern city (2019–2020), and sorted emotion cues in a spatial arrangement method that assesses emotion knowledge without reliance on emotion vocabulary. Using supervised and unsupervised analyses, the study found evidence for continuities and gradual changes in children's emotion knowledge compared to adults. Emotion knowledge develops through an incremental learning process in which children change their representations using combinations of factors—particularly valence—that are weighted differently across development.

Automatic Recognition of Macaque Facial Expressions for Detection of Affective States

Internal affective states produce external manifestations such as facial expressions. In humans, the Facial Action Coding System (FACS) is widely used to objectively quantify the elemental facial action units (AUs) that build complex facial expressions. A similar system has been developed for macaque monkeys-the Macaque FACS (MaqFACS); yet, unlike the human counterpart, which is already partially replaced by automatic algorithms, this system still requires labor-intensive coding. Here, we developed and implemented the first prototype for automatic MaqFACS coding. We applied the approach to the analysis of behavioral and neural data recorded from freely interacting macaque monkeys. The method achieved high performance in the recognition of six dominant AUs, generalizing between conspecific individuals (Macaca mulatta) and even between species (Macaca fascicularis). The study lays the foundation for fully automated detection of facial expressions in animals, which is crucial for investigating the neural substrates of social and affective states.

Nothing to Be Happy about: Consumer Emotions and AI

Advancements in artificial intelligence and Big Data allow for a range of goods and services to determine and respond to a consumer’s emotional state of mind. Considerable potential surrounds the technological ability to detect and respond to an individual’s emotions, yet such technology is also controversial and raises questions surrounding the legal protection of emotions. Despite their highly sensitive and private nature, this article highlights the inadequate protection of emotions in aspects of data protection and consumer protection law, arguing that the contribution by recent proposal for an Artificial Intelligence Act is not only unsuitable to overcome such deficits but does little to support the assertion that emotions are highly sensitive.

NetFACS: Using network science to understand facial communication systems

Understanding facial signals in humans and other species is crucial for understanding the evolution, complexity, and function of the face as a communication tool. The Facial Action Coding System (FACS) enables researchers to measure facial movements accurately, but we currently lack tools to reliably analyse data and efficiently communicate results. Network analysis can provide a way to use the information encoded in FACS datasets: by treating individual AUs (the smallest units of facial movements) as nodes in a network and their co-occurrence as connections, we can analyse and visualise differences in the use of combinations of AUs in different conditions. Here, we present ‘NetFACS’, a statistical package that uses occurrence probabilities and resampling methods to answer questions about the use of AUs, AU combinations, and the facial communication system as a whole in humans and non-human animals. Using highly stereotyped facial signals as an example, we illustrate some of the current functionalities of NetFACS. We show that very few AUs are specific to certain stereotypical contexts; that AUs are not used independently from each other; that graph-level properties of stereotypical signals differ; and that clusters of AUs allow us to reconstruct facial signals, even when blind to the underlying conditions. The flexibility and widespread use of network analysis allows us to move away from studying facial signals as stereotyped expressions, and towards a dynamic and differentiated approach to facial communication.

Physiological linkage during shared positive and shared negative emotion

Physiological linkage refers to the degree to which peoples' physiological responses change in coordinated ways. Here, we examine whether and how physiological linkage relates to incidents of shared emotion, distinguished by valence. Past research has used an "overall average" approach and characterized how physiological linkage over relatively long time periods (e.g., 10-15 min) reflects psychological and social processes (e.g., marital satisfaction, empathy). Here, we used a "momentary" approach and characterized whether physiological linkage over relatively short time periods (i.e., 15 s) reflects shared positive emotion, shared negative emotion, or both, and whether linkage during shared emotions relates to relational functioning. Married couples (156 dyads) had a 15-min conflict conversation in the laboratory. Using behavioral coding, each second of conversation was classified into 1 of 4 emotion categories: shared positive emotion, shared negative emotion, shared neutral emotion, or unshared emotion. Using a composite of 3 peripheral physiological measures (i.e., heart rate, skin conductance, finger pulse amplitude), we computed momentary in-phase and antiphase linkage to represent coordinated changes in the same or opposite direction, respectively. We found that shared positive emotion was associated with higher in-phase and lower antiphase linkage, relative to the other 3 emotion categories. Greater in-phase physiological linkage during shared positive emotion was also consistently associated with higher-quality interactions and relationships, both concurrently and longitudinally (i.e., 5 to 6 years later). These findings advance our understanding of the nature of physiological linkage, the emotional conditions under which it occurs, and its possible associations with relational functioning. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Linguistic and developmental influences on superordinate facial configuration categorization in infancy

Humans perceive emotions in terms of categories, such as "happiness," "sadness," and "anger." To learn these complex conceptual emotion categories, humans must first be able to perceive regularities in expressive behaviors (e.g., facial configurations) across individuals. Recent research suggests that infants spontaneously form "basic-level" categories of facial configurations (e.g., happy vs. fear), but not "superordinate" categories of facial configurations (e.g., positive vs. negative). The current studies further explore how infant age and language impact superordinate categorization of facial configurations associated with different negative emotions. Across all experiments, infants were habituated to one person displaying facial configurations associated with anger and disgust. While 10-month-olds formed a category of person identity (Experiment 1), 14-month-olds formed a category that included negative facial configurations displayed by the same person (Experiment 2). However, neither age formed the hypothesized superordinate category of negative valence. When a verbal label ("toma") was added to each of the habituation events (Experiment 3), 10-month-olds formed a category similar to 14-month-olds in Experiment 2. These findings intersect a larger conversation about the nature and development of children's emotion categories and highlight the importance of considering developmental processes, such as language learning and attentional/memory development, in the design and interpretation of infant categorization studies.

Theories of Conversation for Conversational IR

Conversational information retrieval is a relatively new and fast-developing research area, but conversation itself has been well studied for decades. Researchers have analysed linguistic phenomena such as structure and semantics but also paralinguistic features such as tone, body language, and even the physiological states of interlocutors. We tend to treat computers as social agents—especially if they have some humanlike features in their design—and so work from human-to-human conversation is highly relevant to how we think about the design of human-to-computer applications. In this article, we summarise some salient past work, focusing on social norms; structures; and affect, prosody, and style. We examine social communication theories briefly as a review to see what we have learned about how humans interact with each other and how that might pertain to agents and robots. We also discuss some implications for research and design of conversational IR systems.

Elements of Neuroanthropology

Neuroanthropology is the integration of neuroscience into anthropology and aims to understand “brains in the wild.” This interdisciplinary field examines patterns of human variation in field settings and provides empirical research that complements work done in clinical and laboratory settings. Neuroanthropology often uses ethnography in combination with theories and methods from cognitive science as a way to capture how culture, mind, and brain interact. This article describes nine elements that outline how to do neuroanthropology research: (1) integrating biology and culture through neuroscience and biocultural anthropology; (2) extending focus of anthropology on what people say and do to include what people process; (3) sizing culture appropriately, from broad patterns of culture to culture in small-scale settings; (4) understanding patterns of cultural variation, in particular how culture produces patterns of shared variation; (5) considering individuals in interaction with culture, with levels of analysis that can go from biology to social structures; (6) focusing on interactive elements that bring together biological and cultural processes; (7) conceptual triangulation, which draws on anthropology, psychology, and neuroscience in conjunction with field, clinic, and laboratory; (8) critical complementarity as a way to integrate the strengths of critical scholarship with interdisciplinary work; and (9) using methodological triangulation as a way to advance interdisciplinary research. These elements are illustrated through three case studies: research on US combat veterans and how they use Brazilian Jiu Jitsu as a way to manage the transition to becoming civilians, work on human-raptor interactions to understand how and why these interactions can prove beneficial for human handlers, and adapting cue reactivity research on addiction to a field-based approach to understand how people interact with cues in naturalistic settings.

Emotion as Information in Early Social Learning

The majority of research on infants’ and children’s understanding of emotional expressions has focused on their abilities to use emotional expressions to infer how other people feel. However, an emerging body of work suggests that emotional expressions support rich, powerful inferences not just about emotional states but also about other unobserved states, such as hidden events in the physical world and mental states of other people (e.g., beliefs and desires). Here we argue that infants and children harness others’ emotional expressions as a source of information for learning about the physical and social world broadly. This “emotion as information” framework integrates affective, developmental, and computational cognitive sciences, extending the scope of signals that count as “information” in early learning.

Profiles of Naturalistic Attentional Trajectories Associated with Internalizing Behaviors in School-Age Children: A Mobile Eye Tracking Study

The temperament profile Behavioral Inhibition (BI) is a strong predictor of internalizing behaviors in childhood. Patterns of attention towards or away from threat are a commonality of both BI and internalizing behaviors. Attention biases are traditionally measured with computer tasks presenting affective stimuli, which can lack ecological validity. Recent studies suggest that naturalistic visual attention need not mirror findings from computer tasks, and, more specifically, children high in BI may attend less to threats in naturalistic tasks. Here, we characterized latent trajectories of naturalistic visual attention over time to a female stranger, measured with mobile eye tracking, among kindergarteners oversampled for BI. Group-based trajectory modeling (GBTM) revealed two latent trajectories: 1) high initial orienting to the stranger, gradual decay, and recovery, and 2) low initial orienting and continued avoidance. Higher probability of membership to the "avoidant" group was linked to greater report of internalizing behaviors. We demonstrate the efficacy of mobile eye tracking in quantifying naturalistic patterns of visual attention to social novelty, as well as the importance of naturalistic measures of attention in characterizing socioemotional risk factors.

Building and validation of a set of facial expression images to detect emotions: a transcultural study

The automatic emotion recognition from facial expressions has become an exceptional tool in research involving human subjects and has made it possible to obtain objective measurements of the emotional state of research subjects. Different software and commercial solutions are offered to perform this task. However, the adaptation to cultural context and the recognition of complex expressions and/or emotions are two of the main challenges faced by these solutions. Here, we describe the construction and validation of a set of facial expression images suitable for training a recognition system. Our datasets consist of images of people with no experience in acting who were recorded with a webcam as they performed a computer-assisted task in a room with a light background and overhead illumination. The six basic emotions and mockery were included and a combination of OpenCV, Dlib and Scikit-learn Python libraries were used to develop a support vector machine classifier. The code is available at GitHub and the images will be provided upon request. Since transcultural facial expressions to evaluate complex emotions and open-source solutions were used in this study, we strongly believe that our dataset will be useful in different research contexts.

Infants show pupil dilatory responses to happy and angry facial expressions

Facial expressions are one way in which infants and adults communicate emotion. Infants scan expressions similarly to adults, yet it remains unclear whether they are receptive to the affective information they convey. The current study investigates 6-, 9- and 12-month infants' (N = 146) pupillary responses to the six "basic" emotional expressions (happy, sad, surprise, fear, anger, and disgust). To do this we use dynamic stimuli and gaze-contingent eye-tracking to simulate brief interactive exchanges, alongside a static control condition. Infants' arousal responses were stronger for dynamic compared to static stimuli. And for dynamic stimuli we found that, compared to neutral, infants showed dilatory responses for happy and angry expressions only. Although previous work has shown infants can discriminate perceptually between facial expressions, our data suggest that sensitivity to the affective content of all six basic emotional expressions may not fully emerge until later in ontogeny.

Age and Cultural Differences in Recognitions of Emotions from Masked Faces among Koreans and Americans

This study investigates age and cultural differences in the negative effects of senders’ wearing masks on receivers’ readabilities of senders’ facially expressed emotions in interpersonal interactions. An online experiment was thus conducted with Koreans and Americans aged over 20 years. Based on sampling quotas by nationality, age group and gender, Korean (n = 240) and American (n = 273) participants were recruited from panel members of a Korean research company and Amazon’s Mechanical Turk via email and the website, respectively. The participants played receiver roles to infer senders’ facially expressed emotions presented in photos in the experiment. They judged emotions facially expressed by the senders without masks and with masks are shown in photos. The results revealed that the senders’ wearing masks reduced the readabilities of the senders’ facially expressed anger among participants aged 30–49 years more than among participants aged 20–29 years. The senders’ wearing masks decreased the readabilities of the senders’ facially expressed fear for participants in their 50’s more than for participants in their 20’s. When the senders wore masks, the readabilities of the senders’ facially expressed happiness dropped among participants aged over 60 years more than among participants aged 20–49 years. When senders wore masks, American participants’ readabilities of disgust, fear, sadness and happiness expressed in the senders’ faces declined more than Korean participants’ readabilities of those emotions. The implications and limitations of these findings are discussed.

Interoceptive sensibility predicts the ability to infer others' emotional states

Emotional sensations and inferring another's emotional states have been suggested to depend on predictive models of the causes of bodily sensations, so-called interoceptive inferences. In this framework, higher sensibility for interoceptive changes (IS) reflects higher precision of interoceptive signals. The present study examined the link between IS and emotion recognition, testing whether individuals with higher IS recognize others' emotions more easily and are more sensitive to learn from biased probabilities of emotional expressions. We recorded skin conductance responses (SCRs) from forty-six healthy volunteers performing a speeded-response task, which required them to indicate whether a neutral facial expression dynamically turned into a happy or fearful expression. Moreover, varying probabilities of emotional expressions by their block-wise base rate aimed to generate a bias for the more frequently encountered emotion. As a result, we found that individuals with higher IS showed lower thresholds for emotion recognition, reflected in decreased reaction times for emotional expressions especially of high intensity. Moreover, individuals with increased IS benefited more from a biased probability of an emotion, reflected in decreased reaction times for expected emotions. Lastly, weak evidence supporting a differential modulation of SCR by IS as a function of varying probabilities was found. Our results indicate that higher interoceptive sensibility facilitates the recognition of emotional changes and is accompanied by a more precise adaptation to emotion probabilities.

Olfaction in the Multisensory Processing of Faces: A Narrative Review of the Influence of Human Body Odors

A recent body of research has emerged regarding the interactions between olfaction and other sensory channels to process social information. The current review examines the influence of body odors on face perception, a core component of human social cognition. First, we review studies reporting how body odors interact with the perception of invariant facial information (i.e., identity, sex, attractiveness, trustworthiness, and dominance). Although we mainly focus on the influence of body odors based on axillary odor, we also review findings about specific steroids present in axillary sweat (i.e., androstenone, androstenol, androstadienone, and estratetraenol). We next survey the literature showing body odor influences on the perception of transient face properties, notably in discussing the role of body odors in facilitating or hindering the perception of emotional facial expression, in relation to competing frameworks of emotions. Finally, we discuss the developmental origins of these olfaction-to-vision influences, as an emerging literature indicates that odor cues strongly influence face perception in infants. Body odors with a high social relevance such as the odor emanating from the mother have a widespread influence on various aspects of face perception in infancy, including categorization of faces among other objects, face scanning behavior, or facial expression perception. We conclude by suggesting that the weight of olfaction might be especially strong in infancy, shaping social perception, especially in slow-maturing senses such as vision, and that this early tutoring function of olfaction spans all developmental stages to disambiguate a complex social environment by conveying key information for social interactions until adulthood.

The dynamic mask: Facial correlates of character portrayal in professional actors

Actors make modifications to their face, voice, and body to match standard gestural conceptions of the fictional characters they are portraying during stage performances. However, the gestural manifestations of acting have not been quantified experimentally, least of all in group-level analyses. To quantify the facial correlates of character portrayal in professional actors for the first time, we had 24 actors portray a contrastive series of nine stock characters (e.g., king, bully, lover) that were organised according to a predictive scheme based on the two statistically independent personality dimensions of assertiveness (i.e., the tendency to satisfy personal concerns) and cooperativeness (i.e., the tendency to satisfy others’ concerns). We used three-dimensional motion capture to examine changes in facial dimensions, with an emphasis on the relative expansion/contraction of four facial segments related to the brow, eyebrows, lips, and jaw, respectively. The results demonstrated that expansions in both upper- and lower-facial segments were related to increases in the levels of character cooperativeness, but not assertiveness. These findings demonstrate that actors reliably manipulate their facial features in a contrastive manner to differentiate characters based on their underlying personality traits.

Emotion Words in Early Childhood: A Language Transcript Analysis

Children learn the abstract, challenging categories of emotions from young ages, and it has recently been suggested that language (and more specifically emotion words) may aid this learning. To examine the language that young children hear and produce as they're learning emotion categories, the present study examined nearly 2,000 transcripts from 179 children ranging from 15- to 47-months from the Child Language Data Exchange System (CHILDES). Results provide key descriptive, developmental, and predictive information regarding child emotion language production, including the finding that child emotion word production was predicted by mothers' emotion word production (β=.21, p<.001), but not by child or mother language complexity (β=.01, p=.690; β=.00, p=.872). Frequency of specific emotion words are presented, as are developmental trends in early emotion language production and input. These results improve the understanding of children's daily emotional language environments and may inform theories of emotional development.

Does facial redness really affect emotion perception? Evidence for limited generalisability of effects of facial redness on emotion perception in a large sample

We conducted a preregistered study (N = 609) to conceptually replicate and extend prior research regarding the effects of facial redness on emotion perception. In a within-subjects design, participants saw emotion faces (anger, happiness, fear, neutral) of a random female and a random male target with default facial colouration and increased facial redness and were asked to simultaneously rate the intensity of six emotions (happiness, surprise, sadness, fear, disgust, anger) for each emotion face. The emotion intensity was rated higher, when the emotion face and the rated emotion matched than when the emotion face and the rated emotion did not match. However, increased facial redness did not influence the intensity of the rated emotion. The results of this conceptual replication limit the generalisability of previous findings, challenge the assumption that facial redness is used as a cue to infer emotions, and point to the necessity to develop a more nuanced theoretical account of contextual boundaries.

Using genetic algorithms to uncover individual differences in how humans represent facial emotion

Emotional facial expressions critically impact social interactions and cognition. However, emotion research to date has generally relied on the assumption that people represent categorical emotions in the same way, using standardized stimulus sets and overlooking important individual differences. To resolve this problem, we developed and tested a task using genetic algorithms to derive assumption-free, participant-generated emotional expressions. One hundred and five participants generated a subjective representation of happy, angry, fearful and sad faces. Population-level consistency was observed for happy faces, but fearful and sad faces showed a high degree of variability. High test-retest reliability was observed across all emotions. A separate group of 108 individuals accurately identified happy and angry faces from the first study, while fearful and sad faces were commonly misidentified. These findings are an important first step towards understanding individual differences in emotion representation, with the potential to reconceptualize the way we study atypical emotion processing in future research.

Infants are sensitive to cultural differences in emotions at 11 months

A myriad of emotion perception studies has shown infants' ability to discriminate different emotional categories, yet there has been little investigation of infants' perception of cultural differences in emotions. Hence little is known about the extent to which culture-specific emotion information is recognised in the beginning of life. Caucasian Australian infants of 10-12 months participated in a visual-paired comparison task where their preferential looking patterns to three types of infant-directed emotions (anger, happiness, surprise) from two different cultures (Australian, Japanese) were examined. Differences in racial appearances were controlled. Infants exhibited preferential looking to Japanese over Caucasian Australian mothers' angry and surprised expressions, whereas no difference was observed in trials involving East-Asian Australian mothers. In addition, infants preferred Caucasian Australian mothers' happy expressions. These findings suggest that 11-month-olds are sensitive to cultural differences in spontaneous infant-directed emotional expressions when they are combined with a difference in racial appearance.

Measurement of Behavioral Emotion Regulation Strategies in Early Childhood: The Early Emotion Regulation Behavior Questionnaire (EERBQ)

The Early Emotion Regulation Behavior Questionnaire (EERBQ) assesses children's emotion regulation (ER) behavioral strategies in both positive and negative emotional contexts. Psychometric properties and factor structure were tested in a sample of caregivers across the United States (N = 362) with children ages 2-6 years-old (56% male; 73% White). Findings suggest that the EERBQ is psychometrically sound and correlates with other well-established measures of children's socioemotional functioning. Previously, researchers have only been able to assess children's emotional behavioral regulatory strategies in a laboratory setting. Thus, use of the EERBQ addresses a critical gap in the current literature by providing researchers and practitioners with an instrument to measure young children's early emotional functioning outside of a laboratory context. This is particularly salient because early difficulty regulating emotions is often a precursor to persistent adverse developmental outcomes. Thus, the ability to easily to collect rich and predictive behavioral regulation data is imperative for early identification and treatment of youths' emotional and behavioral problems.

Face-to-Face Clinical Practice Under COVID-19 Pandemic: How Psychotherapists Describe Their Experiences

Driven by the theory-building around the role of the non-verbal components to communication, we aimed to understand how therapists experience the therapeutic process using a facial mask. The empirical evidence of the power of non-verbal communication to engage therapists and clients in therapeutic work, develop a positive and collaborative relationship between them, and display empathy is quite large. A mixed-methods approach was adopted, drawing from the therapists’ participation in an online survey. A sample of 137 psychotherapists with different therapy orientations and years of clinical practice participated in the study. Therapists conducted face-to-face therapy wearing face masks with existing and/or new clients. We performed an exploratory analysis, using descriptive statistics, to explore the psychotherapists’ evaluations regarding perceived impact of face masks on different therapy quality dimensions. In a complementary rationale, we analyzed the therapists’ perspectives on their experience wearing face masks using the thematic analysis methodology. Results show that among 137 psychotherapists, 114 were attending both existing and new clients, whereas only 13 were seeing exclusively existing clients and 10 were working exclusively with new clients. Despite no major differences were found between conditions regarding the perceived impact of face masks on different therapy quality dimensions and strategies adopted, the qualitative analysis allowed us to expand the quantitative results and deepen understanding of psychotherapists’ experience. Based on general and typical patterns, we propose two distinct models to describe the therapist’s experiences narrative when working with existing or new clients wearing face masks. Based on the results, we propose some recommendations to clinical practice in similar conditions.

Adversity and Emotional Functioning

Exposure to early adversity has been linked to variations in emotional functioning. To date, however, the precise nature of these variations has been difficult to pinpoint given widespread differences in the ways in which aspects of emotional functioning are defined and measured. Here, more consistent with models of emotional functioning in typically developing populations (e.g., Halberstadt et al., 2001), we propose defining emotional functioning as consisting of distinct domains of emotion expression, perception, knowledge, reactivity, and regulation. We argue that this framework is useful for guiding hypothesis generation about the specific impact of early adversity on children's emotional functioning. We operationalize the construct of emotional functioning, highlight what is currently known about the association between adversity exposure and each domain of emotional functioning, propose potential mechanisms for these associations, and set the stage for future research examining the development of emotional functioning in the context of early adversity.