Large-Scale Observational Evidence of Cross-Cultural Differences in Facial Behavior

A Systematic Review and New Analyses of the Gender-Equality Paradox

Some studies show that living conditions, such as economy, gender equality, and education, are associated with the magnitude of psychological sex differences. We systematically and quantitatively reviewed 54 articles and conducted new analyses on 27 meta-analyses and large-scale studies to investigate the association between living conditions and psychological sex differences. We found that sex differences in personality, verbal abilities, episodic memory, and negative emotions are more pronounced in countries with higher living conditions. In contrast, sex differences in sexual behavior, partner preferences, and math are smaller in countries with higher living conditions. We also observed that economic indicators of living conditions, such as gross domestic product, are most sensitive in predicting the magnitude of sex differences. Taken together, results indicate that more sex differences are larger, rather than smaller, in countries with higher living conditions. It should therefore be expected that the magnitude of most psychological sex differences will remain unchanged or become more pronounced with improvements in living conditions, such as economy, gender equality, and education.

How Male and Female Literary Authors Write About Affect Across Cultures and Over Historical Periods

A wealth of literature suggests the existence of sex differences in how emotions are experienced, recognized, expressed, and regulated. However, to what extent these differences result from the put in place of stereotypes and social rules is still a matter of debate. Literature is an essential cultural institution, a transposition of the social life of people but also of their intimate affective experiences, which can serve to address questions of psychological relevance. Here, we created a large corpus of literary fiction enriched by authors' metadata to measure the extent to which culture influences how men and women write about emotion. Our results show that even though before the twenty-first century and across 116 countries women more than men have written about affect, starting from 2000, this difference has diminished substantially. Also, in the past, women's narratives were more positively laden and less arousing. While the difference in arousal is ubiquitous and still present nowadays, sex differences in valence vary as a function of culture and have dissolved in recent years. Altogether, these findings suggest that historic evolution is associated with men and women writing similarly about emotions and reveal a sizable impact of culture on the affective characteristics of the lexicon.

Supplementary Information

The online version contains supplementary material available at 10.1007/s42761-023-00219-9.

Synchronized affect in shared experiences strengthens social connection

People structure their days to experience events with others. We gather to eat meals, watch TV, and attend concerts together. What constitutes a shared experience and how does it manifest in dyadic behavior? The present study investigates how shared experiences-measured through emotional, motoric, physiological, and cognitive alignment-promote social bonding. We recorded the facial expressions and electrodermal activity (EDA) of participants as they watched four episodes of a TV show for a total of 4 h with another participant. Participants displayed temporally synchronized and spatially aligned emotional facial expressions and the degree of synchronization predicted the self-reported social connection ratings between viewing partners. We observed a similar pattern of results for dyadic physiological synchrony measured via EDA and their cognitive impressions of the characters. All four of these factors, temporal synchrony of positive facial expressions, spatial alignment of expressions, EDA synchrony, and character impression similarity, contributed to a latent factor of a shared experience that predicted social connection. Our findings suggest that the development of interpersonal affiliations in shared experiences emerges from shared affective experiences comprising synchronous processes and demonstrate that these complex interpersonal processes can be studied in a holistic and multi-modal framework leveraging naturalistic experimental designs.

These Aren't The Droids You Are Looking for: Promises and Challenges for the Intersection of Affective Science and Robotics/AI

AI research focused on interactions with humans, particularly in the form of robots or virtual agents, has expanded in the last two decades to include concepts related to affective processes. Affective computing is an emerging field that deals with issues such as how the diagnosis of affective states of users can be used to improve such interactions, also with a view to demonstrate affective behavior towards the user. This type of research often is based on two beliefs: (1) artificial emotional intelligence will improve human computer interaction (or more specifically human robot interaction), and (2) we understand the role of affective behavior in human interaction sufficiently to tell artificial systems what to do. However, within affective science the focus of research is often to test a particular assumption, such as "smiles affect liking." Such focus does not provide the information necessary to synthesize affective behavior in long dynamic and real-time interactions. In consequence, theories do not play a large role in the development of artificial affective systems by engineers, but self-learning systems develop their behavior out of large corpora of recorded interactions. The status quo is characterized by measurement issues, theoretical lacunae regarding prevalence and functions of affective behavior in interaction, and underpowered studies that cannot provide the solid empirical foundation for further theoretical developments. This contribution will highlight some of these challenges and point towards next steps to create a rapprochement between engineers and affective scientists with a view to improving theory and solid applications.

Spatio-Temporal Properties of Amused, Embarrassed, and Pained Smiles

Smiles are universal but nuanced facial expressions that are most frequently used in face-to-face communications, typically indicating amusement but sometimes conveying negative emotions such as embarrassment and pain. Although previous studies have suggested that spatial and temporal properties could differ among these various types of smiles, no study has thoroughly analyzed these properties. This study aimed to clarify the spatiotemporal properties of smiles conveying amusement, embarrassment, and pain using a spontaneous facial behavior database. The results regarding spatial patterns revealed that pained smiles showed less eye constriction and more overall facial tension than amused smiles; no spatial differences were identified between embarrassed and amused smiles. Regarding temporal properties, embarrassed and pained smiles remained in a state of higher facial tension than amused smiles. Moreover, embarrassed smiles showed a more gradual change from tension states to the smile state than amused smiles, and pained smiles had lower probabilities of staying in or transitioning to the smile state compared to amused smiles. By comparing the spatiotemporal properties of these three smile types, this study revealed that the probability of transitioning between discrete states could help distinguish amused, embarrassed, and pained smiles.

Self-Protection as an Adaptive Female Strategy

Many male traits are well explained by sexual selection theory as adaptations to mating competition and mate choice, whereas no unifying theory explains traits expressed more in females. Anne Campbell's "staying alive" theory proposed that human females produce stronger self-protective reactions than males to aggressive threats because self-protection tends to have higher fitness value for females than males. We examined whether Campbell's theory has more general applicability by considering whether human females respond with greater self-protectiveness than males to other threats beyond aggression. We searched the literature for physiological, behavioral, and emotional responses to major physical and social threats, and found consistent support for females' responding with greater self-protectiveness than males. Females mount stronger immune responses to many pathogens; experience a lower threshold to detect, and lesser tolerance of, pain; awaken more frequently at night; express greater concern about physically dangerous stimuli; exert more effort to avoid social conflicts; exhibit a personality style more focused on life's dangers; react to threats with greater fear, disgust and sadness; and develop more threat-based clinical conditions than males. Our findings suggest that in relation to threat human females have relatively heightened protective reactions compared to males. The pervasiveness of this result across multiple domains suggests that general mechanisms might exist underlying females' unique adaptations. An understanding of such processes would enhance knowledge of female health and well-being.

Observing Pain in Individuals with Cognitive Impairment: A Pilot Comparison Attempt across Countries and across Different Types of Cognitive Impairment

Facial expression is a key aspect in observational scales developed to improve pain assessment in individuals with cognitive impairments. Although these scales are used internationally in individuals with different types of cognitive impairments, it is not known whether observing facial expressions of pain might differ between regions or between different types of cognitive impairments. In a pilot study, facial responses to standardized experimental pressure pain were assessed among individuals with different types of cognitive impairments (dementia, mild cognitive impairment, Huntington’s disease, and intellectual disability) from different countries (Denmark, Germany, Italy, Israel, and Spain) and were analyzed using facial descriptors from the PAIC scale (Pain Assessment in Impaired Cognition). We found high inter-rater reliability between observers from different countries. Moreover, facial responses to pain did not differ between individuals with dementia from different countries (Denmark, Germany, and Spain). However, the type of cognitive impairment had a significant impact; with individuals with intellectual disability (all being from Israel) showing the strongest facial responses. Our pilot data suggest that the country of origin does not strongly affect how pain is facially expressed or how facial responses are being scored. However, the type of cognitive impairment showed a clear effect in our pilot study, with elevated facial responses in individuals with intellectual disability.

Nonverbal Social Sensing: What Social Sensing Can and Cannot Do for the Study of Nonverbal Behavior From Video

The study of nonverbal behavior (NVB), and in particular kinesics (i.e., face and body motions), is typically seen as cost-intensive. However, the development of new technologies (e.g., ubiquitous sensing, computer vision, and algorithms) and approaches to study social behavior [i.e., social signal processing (SSP)] makes it possible to train algorithms to automatically code NVB, from action/motion units to inferences. Nonverbal social sensing refers to the use of these technologies and approaches for the study of kinesics based on video recordings. Nonverbal social sensing appears as an inspiring and encouraging approach to study NVB at reduced costs, making it a more attractive research field. However, does this promise hold? After presenting what nonverbal social sensing is and can do, we discussed the key challenges that researchers face when using nonverbal social sensing on video data. Although nonverbal social sensing is a promising tool, researchers need to be aware of the fact that algorithms might be as biased as humans when extracting NVB or that the automated NVB coding might remain context-dependent. We provided study examples to discuss these challenges and point to potential solutions.

FaceSync: Open source framework for recording facial expressions with head-mounted cameras

Advances in computer vision and machine learning algorithms have enabled researchers to extract facial expression data from face video recordings with greater ease and speed than standard manual coding methods, which has led to a dramatic increase in the pace of facial expression research. However, there are many limitations in recording facial expressions in laboratory settings.  Conventional video recording setups using webcams, tripod-mounted cameras, or pan-tilt-zoom cameras require making compromises between cost, reliability, and flexibility. As an alternative, we propose the use of a mobile head-mounted camera that can be easily constructed from our open-source instructions and blueprints at a fraction of the cost of conventional setups. The head-mounted camera framework is supported by the open source Python toolbox FaceSync, which provides an automated method for synchronizing videos. We provide four proof-of-concept studies demonstrating the benefits of this recording system in reliably measuring and analyzing facial expressions in diverse experimental setups, including group interaction experiments.