The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures

Emotion topology: extracting fundamental components of emotions from text using word embeddings

This exploratory study examined the potential of word embeddings, an automated numerical representation of written text, as a novel method for emotion decomposition analysis. Drawing from a substantial dataset scraped from a Social Media site, we constructed emotion vectors to extract the dimensions of emotions, as annotated by the readers of the texts, directly from human language. Our findings demonstrated that word embeddings yield emotional components akin to those found in previous literature, offering an alternative perspective not bounded by theoretical presuppositions, as well as showing that the dimensional structure of emotions is reflected in the semantic structure of their text-based expressions. Our study highlights word embeddings as a promising tool for uncovering the nuances of human emotions and comments on the potential of this approach for other psychological domains, providing a basis for future studies. The exploratory nature of this research paves the way for further development and refinement of this method, promising to enrich our understanding of emotional constructs and psychological phenomena in a more ecologically valid and data-driven manner.

Commonalities and variations in emotion representation across modalities and brain regions

Humans express emotions through various modalities such as facial expressions and natural language. However, the relationships between emotions expressed through different modalities and their correlations with neural activities remain uncertain. Here, we aimed to unveil some of these uncertainties by investigating the similarity of emotion representations across modalities and brain regions. First, we represented various emotion categories as multi-dimensional vectors derived from visual (face), linguistic, and visio-linguistic data, and used representational similarity analysis to compare these modalities. Second, we examined the linear transferability of emotion representation from other modalities to the visual modality. Third, we compared the representational structure derived in the first step with those from brain activities across 360 regions. Our findings revealed that emotion representations share commonalities across modalities with modality-type dependent variations, and they can be linearly mapped from other modalities to the visual modality. Additionally, emotion representations in uni-modalities showed relatively higher similarity with specific brain regions, while multi-modal emotion representation was most similar to representations across the entire brain region. These findings suggest that emotional experiences are represented differently across various brain regions with varying degrees of similarity to different modality types, and that they may be multi-modally conveyable in visual and linguistic domains.

Social media emotions annotation guide (SMEmo): Development and initial validity

The proper measurement of emotion is vital to understanding the relationship between emotional expression in social media and other factors, such as online information sharing. This work develops a standardized annotation scheme for quantifying emotions in social media using recent emotion theory and research. Human annotators assessed both social media posts and their own reactions to the posts' content on scales of 0 to 100 for each of 20 (Study 1) and 23 (Study 2) emotions. For Study 1, we analyzed English-language posts from Twitter (N = 244) and YouTube (N = 50). Associations between emotion ratings and text-based measures (LIWC, VADER, EmoLex, NRC-EIL, Emotionality) demonstrated convergent and discriminant validity. In Study 2, we tested an expanded version of the scheme in-country, in-language, on Polish (N = 3648) and Lithuanian (N = 1934) multimedia Facebook posts. While the correlations were lower than with English, patterns of convergent and discriminant validity with EmoLex and NRC-EIL still held. Coder reliability was strong across samples, with intraclass correlations of .80 or higher for 10 different emotions in Study 1 and 16 different emotions in Study 2. This research improves the measurement of emotions in social media to include more dimensions, multimedia, and context compared to prior schemes.

A shared structure for emotion experiences from narratives, videos, and everyday life

Our knowledge of the diversity and psychological organization of emotion experiences is based primarily on studies that used a single type of stimulus with an often limited set of rating scales and analyses. Here we take a comprehensive data-driven approach. We surveyed 1,000+ participants on a diverse set of ratings of emotion experiences to a validated set of ca. 150 text narratives, a validated set of ca. 1,000 videos, and over 10,000 personal experiences sampled longitudinally in everyday life, permitting a unique comparison. All three types of emotion experiences were characterized by similar dimensional spaces that included valence and arousal, as well as dimensions related to generalizability. Emotion experiences were distributed along continuous gradients, with no clear clusters even for the so-called basic emotions. Individual differences in personality traits were associated with differences in everyday emotion experiences but not with emotions evoked by narratives or videos.

Semantic effects on the perception of emotional prosody in native and non-native Chinese speakers

While previous research has found an in-group advantage (IGA) favouring native speakers in emotional prosody perception over non-native speakers, the effects of semantics on emotional prosody perception remain unclear. This study investigated the effects of semantics on emotional prosody perception in Chinese words and sentences for native and non-native Chinese speakers. The critical manipulation was the congruence of prosodic (positive, negative) and semantic (positive, negative, and neutral) valence. Participants listened to a series of audio clips and judged whether the emotional prosody was positive or negative for each utterance. The results revealed an IGA effect: native speakers perceived emotional prosody more accurately and quickly than non-native speakers in Chinese words and sentences. Furthermore, a semantic congruence effect was observed in Chinese words, where both native and non-native speakers recognised emotional prosody more accurately in the semantic-prosody congruent condition than in the incongruent condition. However, in Chinese sentences, this congruence effect was only present for non-native speakers. Additionally, the IGA effect and semantic congruence effect on emotional prosody perception were influenced by prosody valence. These findings illuminate the role of semantics in emotional prosody perception, highlighting perceptual differences between native and non-native Chinese speakers.

How emotion is experienced and expressed in multiple cultures: a large-scale experiment across North America, Europe, and Japan

Core to understanding emotion are subjective experiences and their expression in facial behavior. Past studies have largely focused on six emotions and prototypical facial poses, reflecting limitations in scale and narrow assumptions about the variety of emotions and their patterns of expression. We examine 45,231 facial reactions to 2,185 evocative videos, largely in North America, Europe, and Japan, collecting participants' self-reported experiences in English or Japanese and manual and automated annotations of facial movement. Guided by Semantic Space Theory, we uncover 21 dimensions of emotion in the self-reported experiences of participants in Japan, the United States, and Western Europe, and considerable cross-cultural similarities in experience. Facial expressions predict at least 12 dimensions of experience, despite massive individual differences in experience. We find considerable cross-cultural convergence in the facial actions involved in the expression of emotion, and culture-specific display tendencies-many facial movements differ in intensity in Japan compared to the U.S./Canada and Europe but represent similar experiences. These results quantitatively detail that people in dramatically different cultures experience and express emotion in a high-dimensional, categorical, and similar but complex fashion.

Deep learning reveals what facial expressions mean to people in different cultures

Cross-cultural studies of the meaning of facial expressions have largely focused on judgments of small sets of stereotypical images by small numbers of people. Here, we used large-scale data collection and machine learning to map what facial expressions convey in six countries. Using a mimicry paradigm, 5,833 participants formed facial expressions found in 4,659 naturalistic images, resulting in 423,193 participant-generated facial expressions. In their own language, participants also rated each expression in terms of 48 emotions and mental states. A deep neural network tasked with predicting the culture-specific meanings people attributed to facial movements while ignoring physical appearance and context discovered 28 distinct dimensions of facial expression, with 21 dimensions showing strong evidence of universality and the remainder showing varying degrees of cultural specificity. These results capture the underlying dimensions of the meanings of facial expressions within and across cultures in unprecedented detail.

The Sound of Emotional Prosody: Nearly 3 Decades of Research and Future Directions

Emotional voices attract considerable attention. A search on any browser using "emotional prosody" as a key phrase leads to more than a million entries. Such interest is evident in the scientific literature as well; readers are reminded in the introductory paragraphs of countless articles of the great importance of prosody and that listeners easily infer the emotional state of speakers through acoustic information. However, despite decades of research on this topic and important achievements, the mapping between acoustics and emotional states is still unclear. In this article, we chart the rich literature on emotional prosody for both newcomers to the field and researchers seeking updates. We also summarize problems revealed by a sample of the literature of the last decades and propose concrete research directions for addressing them, ultimately to satisfy the need for more mechanistic knowledge of emotional prosody.

Learning emotional dialects: A British population study of cross-cultural communication

The aim of the current research was to explore whether we can improve the recognition of cross-cultural freely-expressed emotional faces in British participants. We tested several methods for improving the recognition of freely-expressed emotional faces, such as different methods for presenting other-culture expressions of emotion from individuals from Chile, New Zealand and Singapore in two experimental stages. In the first experimental stage, in phase one, participants were asked to identify the emotion of cross-cultural freely-expressed faces. In the second phase, different cohorts were presented with interactive side-by-side, back-to-back and dynamic morphing of cross-cultural freely-expressed emotional faces, and control conditions. In the final phase, we repeated phase one using novel stimuli. We found that all non-control conditions led to recognition improvements. Morphing was the most effective condition for improving the recognition of cross-cultural emotional faces. In the second experimental stage, we presented morphing to different cohorts including own-to-other and other-to-own freely-expressed cross-cultural emotional faces and neutral-to-emotional and emotional-to-neutral other-culture freely-expressed emotional faces. All conditions led to recognition improvements and the presentation of freely-expressed own-to-other cultural-emotional faces provided the most effective learning. These findings suggest that training can improve the recognition of cross-cultural freely-expressed emotional expressions.

The nature and neurobiology of fear and anxiety: State of the science and opportunities for accelerating discovery

Fear and anxiety play a central role in mammalian life, and there is considerable interest in clarifying their nature, identifying their biological underpinnings, and determining their consequences for health and disease. Here we provide a roundtable discussion on the nature and biological bases of fear- and anxiety-related states, traits, and disorders. The discussants include scientists familiar with a wide variety of populations and a broad spectrum of techniques. The goal of the roundtable was to take stock of the state of the science and provide a roadmap to the next generation of fear and anxiety research. Much of the discussion centered on the key challenges facing the field, the most fruitful avenues for future research, and emerging opportunities for accelerating discovery, with implications for scientists, funders, and other stakeholders. Understanding fear and anxiety is a matter of practical importance. Anxiety disorders are a leading burden on public health and existing treatments are far from curative, underscoring the urgency of developing a deeper understanding of the factors governing threat-related emotions.

The background assimilation effect: Facial emotional perception is affected by surrounding stimuli

The perception of facial emotion is not only determined by the physical features of the face itself but also be influenced by the emotional information of the background or surrounding information. However, the details of such effect are not fully understood. Here, the authors tested the perceived emotion of a target face surrounded by stimuli with different levels of emotional valence. In Experiment 1, four types of objects were divided into three groups (negative, unpleasant flowers and unpleasant animals; mildly negative (neutral), houses; positive, pleasant flowers). In Experiment 2, three groups of surrounding faces with different social-emotional valence (negative, neutral, and positive) were formed with the memory of affective personal knowledge. The data from two experiments showed that the perception of facial emotion can be influenced and modulated by the emotional valence of the surrounding stimuli, which can be explained by assimilation: the positive stimuli increased the valence of a target face, while the negative stimuli comparatively decreased it. Furthermore, the neutral stimuli also increased the valence of the target, which could be explained by the social positive effect. Therefore, the process of assimilation is likely to be a high-level emotional cognition rather than a low-level visual perception. The results of this study may help us better understand face perception in realistic scenarios.

Language experience predicts music processing in a half-million speakers of fifty-four languages

Tonal languages differ from other languages in their use of pitch (tones) to distinguish words. Lifelong experience speaking and hearing tonal languages has been argued to shape auditory processing in ways that generalize beyond the perception of linguistic pitch to the perception of pitch in other domains like music. We conducted a meta-analysis of prior studies testing this idea, finding moderate evidence supporting it. But prior studies were limited by mostly small sample sizes representing a small number of languages and countries, making it challenging to disentangle the effects of linguistic experience from variability in music training, cultural differences, and other potential confounds. To address these issues, we used web-based citizen science to assess music perception skill on a global scale in 34,034 native speakers of 19 tonal languages (e.g., Mandarin, Yoruba). We compared their performance to 459,066 native speakers of other languages, including 6 pitch-accented (e.g., Japanese) and 29 non-tonal languages (e.g., Hungarian). Whether or not participants had taken music lessons, native speakers of all 19 tonal languages had an improved ability to discriminate musical melodies on average, relative to speakers of non-tonal languages. But this improvement came with a trade-off: tonal language speakers were also worse at processing the musical beat. The results, which held across native speakers of many diverse languages and were robust to geographic and demographic variation, demonstrate that linguistic experience shapes music perception, with implications for relations between music, language, and culture in the human mind.

Modelling individual and cross-cultural variation in the mapping of emotions to speech prosody

The existence of a mapping between emotions and speech prosody is commonly assumed. We propose a Bayesian modelling framework to analyse this mapping. Our models are fitted to a large collection of intended emotional prosody, yielding more than 3,000 minutes of recordings. Our descriptive study reveals that the mapping within corpora is relatively constant, whereas the mapping varies across corpora. To account for this heterogeneity, we fit a series of increasingly complex models. Model comparison reveals that models taking into account mapping differences across countries, languages, sexes and individuals outperform models that only assume a global mapping. Further analysis shows that differences across individuals, cultures and sexes contribute more to the model prediction than a shared global mapping. Our models, which can be explored in an online interactive visualization, offer a description of the mapping between acoustic features and emotions in prosody.

Children infer the behavioral contexts of unfamiliar foreign songs

Music commonly appears in behavioral contexts in which it can be seen as playing a functional role, as when a parent sings a lullaby with the goal of soothing a baby. Humans readily make inferences, based on the sounds they hear, regarding the behavioral contexts associated with music. These inferences tend to be accurate, even if the songs are in foreign languages or unfamiliar musical idioms; upon hearing a Blackfoot lullaby, a Korean listener with no experience of Blackfoot music, language, or broader culture is far more likely to judge the music's function as "used to soothe a baby" than "used for dancing". Are such inferences shaped by musical exposure or does the human mind naturally detect links between musical form and function of these kinds? Children's developing experience of music provides a clear test of this question. We studied musical inferences in a large sample of children recruited online (N = 5,033), who heard dance, lullaby, and healing songs from 70 world cultures and who were tasked with guessing the original behavioral context in which each was performed. Children reliably inferred the original behavioral contexts with only minimal improvement in performance from the youngest (age 4) to the oldest (age 16), providing little evidence for an effect of experience. Children's inferences tightly correlated with those of adults for the same songs, as collected from a similar online experiment (N = 98,150). Moreover, similar acoustical features were predictive of the inferences of both samples. These findings suggest that accurate inferences about the behavioral contexts of music, driven by universal links between form and function in music across cultures, do not always require extensive musical experience. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Deep learning reveals what vocal bursts express in different cultures

Human social life is rich with sighs, chuckles, shrieks and other emotional vocalizations, called 'vocal bursts'. Nevertheless, the meaning of vocal bursts across cultures is only beginning to be understood. Here, we combined large-scale experimental data collection with deep learning to reveal the shared and culture-specific meanings of vocal bursts. A total of n = 4,031 participants in China, India, South Africa, the USA and Venezuela mimicked vocal bursts drawn from 2,756 seed recordings. Participants also judged the emotional meaning of each vocal burst. A deep neural network tasked with predicting the culture-specific meanings people attributed to vocal bursts while disregarding context and speaker identity discovered 24 acoustic dimensions, or kinds, of vocal expression with distinct emotion-related meanings. The meanings attributed to these complex vocal modulations were 79% preserved across the five countries and three languages. These results reveal the underlying dimensions of human emotional vocalization in remarkable detail.

Perception and classification of emotions in nonsense speech: Humans versus machines

This article contributes to a more adequate modelling of emotions encoded in speech, by addressing four fallacies prevalent in traditional affective computing: First, studies concentrate on few emotions and disregard all other ones ('closed world'). Second, studies use clean (lab) data or real-life ones but do not compare clean and noisy data in a comparable setting ('clean world'). Third, machine learning approaches need large amounts of data; however, their performance has not yet been assessed by systematically comparing different approaches and different sizes of databases ('small world'). Fourth, although human annotations of emotion constitute the basis for automatic classification, human perception and machine classification have not yet been compared on a strict basis ('one world'). Finally, we deal with the intrinsic ambiguities of emotions by interpreting the confusions between categories ('fuzzy world'). We use acted nonsense speech from the GEMEP corpus, emotional 'distractors' as categories not entailed in the test set, real-life noises that mask the clear recordings, and different sizes of the training set for machine learning. We show that machine learning based on state-of-the-art feature representations (wav2vec2) is able to mirror the main emotional categories ('pillars') present in perceptual emotional constellations even in degradated acoustic conditions.

Searching, Navigating, and Recommending Movies through Emotions: A Scoping Review

Movies offer viewers a broad range of emotional experiences, providing entertainment, and meaning. Following the PRISMA-ScR guidelines, we reviewed the literature on digital systems designed to help users search and browse movie libraries and offer recommendations based on emotional content. Our search yielded 83 eligible documents (published between 2000 and 2021). We identified 22 case studies, 34 empirical studies, 26 proof of concept, and one theoretical paper. User transactions (e.g., ratings, tags) were the preferred source of information. The documents examined approached emotions from both a categorical ( $n=35$ ) and dimensional ( $n=18$ ) perspectives, and nine documents offer a combination of both approaches. Although there are several authors mentioned, the references used are frequently dated, and 12 documents do not mention author or model used. We identified 61 words related to emotion or affect. Documents presented on average 1.36 positive terms and 2.64 negative terms. Sentiment analysis ( $n=31$ ) is frequently used for emotion identification, followed by subjective evaluations ( $n=15$ ), movie low-level audio and visual features (n = 11), and face recognition technologies ( $n=8$ ). We discuss limitations and offer a brief review of current emotion models and research.

Context reconsidered: Complex signal ensembles, relational meaning, and population thinking in psychological science

This article considers the status and study of "context" in psychological science through the lens of research on emotional expressions. The article begins by updating three well-trod methodological debates on the role of context in emotional expressions to reconsider several fundamental assumptions lurking within the field's dominant methodological tradition: namely, that certain expressive movements have biologically prepared, inherent emotional meanings that issue from singular, universal processes which are independent of but interact with contextual influences. The second part of this article considers the scientific opportunities that await if we set aside this traditional understanding of "context" as a moderator of signals with inherent psychological meaning and instead consider the possibility that psychological events emerge in ecosystems of signal ensembles, such that the psychological meaning of any individual signal is entirely relational. Such a fundamental shift has radical implications not only for the science of emotion but for psychological science more generally. It offers opportunities to improve the validity and trustworthiness of psychological science beyond what can be achieved with improvements to methodological rigor alone. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Processing emotional prosody in a foreign language: the case of German and Hebrew

This study investigated the universality of emotional prosody in perception of discrete emotions when semantics is not available. In two experiments the perception of emotional prosody in Hebrew and German by listeners who speak one of the languages but not the other was investigated. Having a parallel tool in both languages allowed to conduct controlled comparisons. In Experiment 1, 39 native German speakers with no knowledge of Hebrew and 80 native Israeli speakers rated Hebrew sentences spoken with four different emotional prosodies (anger, fear, happiness, sadness) or neutral. The Hebrew version of the Test for Rating of Emotions in Speech (T-RES) was used for this purpose. Ratings indicated participants’ agreement on how much the sentence conveyed each of four discrete emotions (anger, fear, happiness and sadness). In Experient 2, 30 native speakers of German, and 24 Israeli native speakers of Hebrew who had no knowledge of German rated sentences of the German version of the T-RES. Based only on the prosody, German-speaking participants were able to accurately identify the emotions in the Hebrew sentences and Hebrew-speaking participants were able to identify the emotions in the German sentences. In both experiments ratings between the groups were similar. These findings show that individuals are able to identify emotions in a foreign language even if they do not have access to semantics. This ability goes beyond identification of target emotion; similarities between languages exist even for “wrong” perception. This adds to accumulating evidence in the literature on the universality of emotional prosody.

An analysis of awe evoked by COVID-19 on green purchasing behavior: A dual-path effect of approach-avoidance motivation

The spread of the COVID-19 virus shows that it is time to re-emphasize the ethical attitude of “awe of others, awe of nature, and awe of life.” It once again reveals the importance of green development. In this study, we introduce awe into the context of COVID-19 and construct an “emotion-motivation-behavior” framework, aiming to explore the relationship between the epidemic and green purchasing behavior from a psychological perspective. Study 1 demonstrates the effect of awe on green purchasing and examines the mediating role of the motivation perspective, to reveal the potential different path. Specifically, prosocial motivation mediates the effect of positive awe evoked by COVID-19 on green purchasing; risk avoidance motivation mediates the effect of negative awe evoked by COVID-19 on green purchasing. Study 2 examined the moderating effect of self-construal. These findings have important management implications for enterprises to correctly use emotional guidance strategies and promote green marketing practices during the COVID-19.

Cross-linguistic patterns of speech prosodic differences in autism: A machine learning study

Differences in speech prosody are a widely observed feature of Autism Spectrum Disorder (ASD). However, it is unclear how prosodic differences in ASD manifest across different languages that demonstrate cross-linguistic variability in prosody. Using a supervised machine-learning analytic approach, we examined acoustic features relevant to rhythmic and intonational aspects of prosody derived from narrative samples elicited in English and Cantonese, two typologically and prosodically distinct languages. Our models revealed successful classification of ASD diagnosis using rhythm-relative features within and across both languages. Classification with intonation-relevant features was significant for English but not Cantonese. Results highlight differences in rhythm as a key prosodic feature impacted in ASD, and also demonstrate important variability in other prosodic properties that appear to be modulated by language-specific differences, such as intonation.

Hybrid deep convolutional model-based emotion recognition using multiple physiological signals

Emotion recognition has become increasingly utilized in the medical, advertising, and military domains. Recognizing the cues of emotion from human behaviors or physiological responses is encouraging for the research community. However, extracting true characteristics from sensor data to understand emotions can be challenging due to the complex nature of these signals. Therefore, advanced feature engineering techniques are required for accurate signal recognition. This study presents a hybrid affective model that employs a transfer learning approach for emotion classification using large-frame sensor signals which employ a genuine dataset of signal fusion gathered from 30 participants using wearable sensor systems interconnected with mobile devices. The proposed approach implements several learning algorithms such as Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), Recurrent Neural Network (RNN), and several other shallow methods on the sensor input to handle the requirements for the traditional feature extraction process. The findings reveal that the use of deep learning methods is satisfactory in affect recognition when a great number of frames is employed, and the proposed hybrid deep model outperforms traditional neural network (overall accuracy of 54%) and deep learning approaches (overall accuracy of 76%), with an average classification accuracy of 93%. This hybrid deep model also has a higher accuracy than our previously proposed statistical autoregressive hidden Markov model (AR-HMM) approach, with 88.6% accuracy. Accuracy assessment was performed by means of several statistics measures (accuracy, precision, recall, F-measure, and RMSE).

Comparing supervised and unsupervised approaches to multimodal emotion recognition

We investigated emotion classification from brief video recordings from the GEMEP database wherein actors portrayed 18 emotions. Vocal features consisted of acoustic parameters related to frequency, intensity, spectral distribution, and durations. Facial features consisted of facial action units. We first performed a series of person-independent supervised classification experiments. Best performance (AUC = 0.88) was obtained by merging the output from the best unimodal vocal (Elastic Net, AUC = 0.82) and facial (Random Forest, AUC = 0.80) classifiers using a late fusion approach and the product rule method. All 18 emotions were recognized with above-chance recall, although recognition rates varied widely across emotions (e.g., high for amusement, anger, and disgust; and low for shame). Multimodal feature patterns for each emotion are described in terms of the vocal and facial features that contributed most to classifier performance. Next, a series of exploratory unsupervised classification experiments were performed to gain more insight into how emotion expressions are organized. Solutions from traditional clustering techniques were interpreted using decision trees in order to explore which features underlie clustering. Another approach utilized various dimensionality reduction techniques paired with inspection of data visualizations. Unsupervised methods did not cluster stimuli in terms of emotion categories, but several explanatory patterns were observed. Some could be interpreted in terms of valence and arousal, but actor and gender specific aspects also contributed to clustering. Identifying explanatory patterns holds great potential as a meta-heuristic when unsupervised methods are used in complex classification tasks.

Mexican Emotional Speech Database Based on Semantic, Frequency, Familiarity, Concreteness, and Cultural Shaping of Affective Prosody

In this paper, the Mexican Emotional Speech Database (MESD) that contains single-word emotional utterances for anger, disgust, fear, happiness, neutral and sadness with adult (male and female) and child voices is described. To validate the emotional prosody of the uttered words, a cubic Support Vector Machines classifier was trained on the basis of prosodic, spectral and voice quality features for each case study: (1) male adult, (2) female adult and (3) child. In addition, cultural, semantic, and linguistic shaping of emotional expression was assessed by statistical analysis. This study was registered at BioMed Central and is part of the implementation of a published study protocol. Mean emotional classification accuracies yielded 93.3%, 89.4% and 83.3% for male, female and child utterances respectively. Statistical analysis emphasized the shaping of emotional prosodies by semantic and linguistic features. A cultural variation in emotional expression was highlighted by comparing the MESD with the INTERFACE for Castilian Spanish database. The MESD provides reliable content for linguistic emotional prosody shaped by the Mexican cultural environment. In order to facilitate further investigations, a corpus controlled for linguistic features and emotional semantics, as well as one containing words repeated across voices and emotions are provided. The MESD is made freely available.

Nonverbal Auditory Cues Allow Relationship Quality to be Inferred During Conversations

The claim that nonverbal cues provide more information than the linguistic content of a conversational exchange (the Mehrabian Conjecture) has been widely cited and equally widely disputed, mainly on methodological grounds. Most studies that have tested the Conjecture have used individual words or short phrases spoken by actors imitating emotions. While cue recognition is certainly important, speech evolved to manage interactions and relationships rather than simple information exchange. In a cross-cultural design, we tested participants’ ability to identify the quality of the interaction (rapport) in naturalistic third party conversations in their own and a less familiar language, using full auditory content versus audio clips whose verbal content has been digitally altered to differing extents. We found that, using nonverbal content alone, people are 75–90% as accurate as they are with full audio cues in identifying positive vs negative relationships, and 45–53% as accurate in identifying eight different relationship types. The results broadly support Mehrabian’s claim that a significant amount of information about others’ social relationships is conveyed in the nonverbal component of speech.

The representational dynamics of perceived voice emotions evolve from categories to dimensions

Long-standing affective science theories conceive the perception of emotional stimuli either as discrete categories (for example, an angry voice) or continuous dimensional attributes (for example, an intense and negative vocal emotion). Which position provides a better account is still widely debated. Here we contrast the positions to account for acoustics-independent perceptual and cerebral representational geometry of perceived voice emotions. We combined multimodal imaging of the cerebral response to heard vocal stimuli (using functional magnetic resonance imaging and magneto-encephalography) with post-scanning behavioural assessment of voice emotion perception. By using representational similarity analysis, we find that categories prevail in perceptual and early (less than 200 ms) frontotemporal cerebral representational geometries and that dimensions impinge predominantly on a later limbic-temporal network (at 240 ms and after 500 ms). These results reconcile the two opposing views by reframing the perception of emotions as the interplay of cerebral networks with different representational dynamics that emphasize either categories or dimensions.

Gender Differences in Identifying Facial, Prosodic, and Semantic Emotions Show Category- and Channel-Specific Effects Mediated by Encoder's Gender

Purpose The nature of gender differences in emotion processing has remained unclear due to the discrepancies in existing literature. This study examined the modulatory effects of emotion categories and communication channels on gender differences in verbal and nonverbal emotion perception. Method Eighty-eight participants (43 females and 45 males) were asked to identify three basic emotions (i.e., happiness, sadness, and anger) and neutrality encoded by female or male actors from verbal (i.e., semantic) or nonverbal (i.e., facial and prosodic) channels. Results While women showed an overall advantage in performance, their superiority was dependent on specific types of emotion and channel. Specifically, women outperformed men in regard to two basic emotions (happiness and sadness) in the nonverbal channels and only the anger category with verbal content. Conversely, men did better for the anger category in the nonverbal channels and for the other two emotions (happiness and sadness) in verbal content. There was an emotion- and channel-specific interaction effect between the two types of gender differences, with male subjects showing higher sensitivity to sad faces and prosody portrayed by the female encoders. Conclusion These findings reveal explicit emotion processing as a highly dynamic complex process with significant gender differences tied to specific emotion categories and communication channels. Supplemental Material https://doi.org/10.23641/asha.15032583.

What Do We Hear in the Voice? An Open-Ended Judgment Study of Emotional Speech Prosody

The current study investigated what can be understood from another person's tone of voice. Participants from five English-speaking nations (Australia, India, Kenya, Singapore, and the United States) listened to vocal expressions of nine positive and nine negative affective states recorded by actors from their own nation. In response, they wrote open-ended judgments of what they believed the actor was trying to express. Responses cut across the chronological emotion process and included descriptions of situations, cognitive appraisals, feeling states, physiological arousal, expressive behaviors, emotion regulation, and attempts at social influence. Accuracy in terms of emotion categories was overall modest, whereas accuracy in terms of valence and arousal was more substantial. Coding participants' 57,380 responses yielded a taxonomy of 56 categories, which included affective states as well as person descriptors, communication behaviors, and abnormal states. Open-ended responses thus reveal a wide range of ways in which people spontaneously perceive the intent behind emotional speech prosody.

Cultural values shape the expression of self-evaluative social emotions

Social emotions are key to everyday social life and therefore shaped by cultural values in their expression. Prior research has focused on facial expressions of emotions. What is less clear, however, is the extent to which cultural values shape other modalities of emotional expression. In the present study, we applied a novel paradigm using depth sensor imaging technology to capture changes in participants’ body posture in real time. We aimed to (1) identify the nuances in the postural expression that are thought to characterize social emotions and (2) assess how individual differences in cultural values impact the postural expression of emotions. Participants in two separate studies were 132 undergraduate college students whose upper-body postural expansion was recorded after they recalled emotion episodes. Positive emotions elevated participants’ upper-body posture whereas negative emotions resulted in lowered upper-body posture. The effects on changes in upper-body posture were moderated by participants’ self-ratings of the vertical and horizontal dimensions of individualism and collectivism. The findings provide initial evidence of the nuances in the way cultural values influence the postural expression of emotions.

Are Emotions Natural Kinds After All? Rethinking the Issue of Response Coherence

The synchronized co-activation of multiple responses-motivational, behavioral, and physiological-has been taken as a defining feature of emotion. Such response coherence has been observed inconsistently however, and this has led some to view emotion programs as lacking biological reality. Yet, response coherence is not always expected or desirable if an emotion program is to carry out its adaptive function. Rather, the hallmark of emotion is the capacity to orchestrate multiple mechanisms adaptively-responses will co-activate in stereotypical fashion or not depending on how the emotion orchestrator interacts with the situation. Nevertheless, might responses cohere in the general case where input variables are specified minimally? Here we focus on shame as a case study. We measure participants' responses regarding each of 27 socially devalued actions and personal characteristics. We observe internal and external coherence: The intensities of felt shame and of various motivations of shame (hiding, lying, destroying evidence, and threatening witnesses) vary in proportion (i) to one another, and (ii) to the degree to which audiences devalue the disgraced individual-the threat shame defends against. These responses cohere both within and between the United States and India. Further, alternative explanations involving the low-level variable of arousal do not seem to account for these results, suggesting that coherence is imparted by a shame system. These findings indicate that coherence can be observed at multiple levels and raise the possibility that emotion programs orchestrate responses, even in those situations where coherence is low.

The emotional canvas of human screams: patterns and acoustic cues in the perceptual categorization of a basic call type

Screams occur across taxonomically widespread species, typically in antipredator situations, and are strikingly similar acoustically, but in nonhuman primates, they have taken on acoustically varied forms in association with more contextually complex functions related to agonistic recruitment. Humans scream in an even broader range of contexts, but the extent to which acoustic variation allows listeners to perceive different emotional meanings remains unknown. We investigated how listeners responded to 30 contextually diverse human screams on six different emotion prompts as well as how selected acoustic cues predicted these responses. We found that acoustic variation in screams was associated with the perception of different emotions from these calls. Emotion ratings generally fell along two dimensions: one contrasting perceived anger, frustration, and pain with surprise and happiness, roughly associated with call duration and roughness, and one related to perceived fear, associated with call fundamental frequency. Listeners were more likely to rate screams highly in emotion prompts matching the source context, suggesting that some screams conveyed information about emotional context, but it is noteworthy that the analysis of screams from happiness contexts (n = 11 screams) revealed that they more often yielded higher ratings of fear. We discuss the implications of these findings for the role and evolution of nonlinguistic vocalizations in human communication, including consideration of how the expanded diversity in calls such as human screams might represent a derived function of language.

When affect overlaps with concept: emotion recognition in semantic variant of primary progressive aphasia

The most recent theories of emotions have postulated that their expression and recognition depend on acquired conceptual knowledge. In other words, the conceptual knowledge derived from prior experiences guide our ability to make sense of such emotions. However, clear evidence is still lacking to contradict more traditional theories, considering emotions as innate, distinct and universal physiological states. In addition, whether valence processing (i.e. recognition of the pleasant/unpleasant character of emotions) also relies on semantic knowledge is yet to be determined. To investigate the contribution of semantic knowledge to facial emotion recognition and valence processing, we conducted a behavioural and neuroimaging study in 20 controls and 16 patients with the semantic variant of primary progressive aphasia, a neurodegenerative disease that is prototypical of semantic memory impairment, and in which an emotion recognition deficit has already been described. We assessed participants’ knowledge of emotion concepts and recognition of 10 basic (e.g. anger) or self-conscious (e.g. embarrassment) facial emotional expressions presented both statically (images) and dynamically (videos). All participants also underwent a brain MRI. Group comparisons revealed deficits in both emotion concept knowledge and emotion recognition in patients, independently of type of emotion and presentation. These measures were significantly correlated with each other in patients and with semantic fluency in patients and controls. Neuroimaging analyses showed that both emotion recognition and emotion conceptual knowledge were correlated with reduced grey matter density in similar areas within frontal ventral, temporal, insular and striatal regions, together with white fibre degeneration in tracts connecting frontal regions with each other as well as with temporal regions. We then performed a qualitative analysis of responses made during the facial emotion recognition task, by delineating valence errors (when one emotion was mistaken for another of a different valence), from other errors made during the emotion recognition test. We found that patients made more valence errors. The number of valence errors correlated with emotion conceptual knowledge as well as with reduced grey matter volume in brain regions already retrieved to correlate with this score. Specificity analyses allowed us to conclude that this cognitive relationship and anatomical overlap were not mediated by a general effect of disease severity. Our findings suggest that semantic knowledge guides the recognition of emotions and is also involved in valence processing. Our study supports a constructionist view of emotion recognition and valence processing, and could help to refine current theories on the interweaving of semantic knowledge and emotion processing.

Sixteen facial expressions occur in similar contexts worldwide

Understanding the degree to which human facial expressions co-vary with specific social contexts across cultures is central to the theory that emotions enable adaptive responses to important challenges and opportunities^1-6. Concrete evidence linking social context to specific facial expressions is sparse and is largely based on survey-based approaches, which are often constrained by language and small sample sizes^7-13. Here, by applying machine-learning methods to real-world, dynamic behaviour, we ascertain whether naturalistic social contexts (for example, weddings or sporting competitions) are associated with specific facial expressions¹⁴ across different cultures. In two experiments using deep neural networks, we examined the extent to which 16 types of facial expression occurred systematically in thousands of contexts in 6 million videos from 144 countries. We found that each kind of facial expression had distinct associations with a set of contexts that were 70% preserved across 12 world regions. Consistent with these associations, regions varied in how frequently different facial expressions were produced as a function of which contexts were most salient. Our results reveal fine-grained patterns in human facial expressions that are preserved across the modern world.

Inferential affective tracking reveals the remarkable speed of context-based emotion perception

Understanding the emotional states of others is important for social functioning. Recent studies show that context plays an essential role in emotion recognition. However, it remains unclear whether emotion inference from visual scene context is as efficient as emotion recognition from faces. Here, we measured the speed of context-based emotion perception, using Inferential Affective Tracking (IAT) with naturalistic and dynamic videos. Using cross-correlation analyses, we found that inferring affect based on visual context alone is just as fast as tracking affect with all available information including face and body. We further demonstrated that this approach has high precision and sensitivity to sub-second lags. Our results suggest that emotion recognition from dynamic contextual information might be automatic and immediate. Seemingly complex context-based emotion perception is far more efficient than previously assumed.

Semantic Space Theory: A Computational Approach to Emotion

Within affective science, the central line of inquiry, animated by basic emotion theory and constructivist accounts, has been the search for one-to-one mappings between six emotions and their subjective experiences, prototypical expressions, and underlying brain states. We offer an alternative perspective: semantic space theory. This computational approach uses wide-ranging naturalistic stimuli and open-ended statistical techniques to capture systematic variation in emotion-related behaviors. Upwards of 25 distinct varieties of emotional experience have distinct profiles of associated antecedents and expressions. These emotions are high-dimensional, categorical, and often blended. This approach also reveals that specific emotions, more than valence, organize emotional experience, expression, and neural processing. Overall, moving beyond traditional models to study broader semantic spaces of emotion can enrich our understanding of human experience.

Universal facial expressions uncovered in art of the ancient Americas: A computational approach

Central to the study of emotion is evidence concerning its universality, particularly the degree to which emotional expressions are similar across cultures. Here, we present an approach to studying the universality of emotional expression that rules out cultural contact and circumvents potential biases in survey-based methods: A computational analysis of apparent facial expressions portrayed in artwork created by members of cultures isolated from Western civilization. Using data-driven methods, we find that facial expressions depicted in 63 sculptures from the ancient Americas tend to accord with Western expectations for emotions that unfold in specific social contexts. Ancient American sculptures tend to portray at least five facial expressions in contexts predicted by Westerners, including "pain" in torture, "determination"/"strain" in heavy lifting, "anger" in combat, "elation" in social touch, and "sadness" in defeat-supporting the universality of these expressions.

The Neural Representation of Visually Evoked Emotion Is High-Dimensional, Categorical, and Distributed across Transmodal Brain Regions

Central to our subjective lives is the experience of different emotions. Recent behavioral work mapping emotional responses to 2,185 videos found that people experience upward of 27 distinct emotions occupying a high-dimensional space, and that emotion categories, more so than affective dimensions (e.g., valence), organize self-reports of subjective experience. Here, we sought to identify the neural substrates of this high-dimensional space of emotional experience using fMRI responses to all 2,185 videos. Our analyses demonstrated that (1) dozens of video-evoked emotions were accurately predicted from fMRI patterns in multiple brain regions with different regional configurations for individual emotions; (2) emotion categories better predicted cortical and subcortical responses than affective dimensions, outperforming visual and semantic covariates in transmodal regions; and (3) emotion-related fMRI responses had a cluster-like organization efficiently characterized by distinct categories. These results support an emerging theory of the high-dimensional emotion space, illuminating its neural foundations distributed across transmodal regions.

What music makes us feel: At least 13 dimensions organize subjective experiences associated with music across different cultures

What is the nature of the feelings evoked by music? We investigated how people represent the subjective experiences associated with Western and Chinese music and the form in which these representational processes are preserved across different cultural groups. US (n = 1,591) and Chinese (n = 1,258) participants listened to 2,168 music samples and reported on the specific feelings (e.g., "angry," "dreamy") or broad affective features (e.g., valence, arousal) that they made individuals feel. Using large-scale statistical tools, we uncovered 13 distinct types of subjective experience associated with music in both cultures. Specific feelings such as "triumphant" were better preserved across the 2 cultures than levels of valence and arousal, contrasting with theoretical claims that valence and arousal are building blocks of subjective experience. This held true even for music selected on the basis of its valence and arousal levels and for traditional Chinese music. Furthermore, the feelings associated with music were found to occupy continuous gradients, contradicting discrete emotion theories. Our findings, visualized within an interactive map (https://www.ocf.berkeley.edu/∼acowen/music.html) reveal a complex, high-dimensional space of subjective experience associated with music in multiple cultures. These findings can inform inquiries ranging from the etiology of affective disorders to the neurological basis of emotion.

What Is Complex/Emotional About Emotional Complexity?

Affective experiences can fluctuate, be combined, and fused, resulting in various phenomena labeled as being emotionally complex. Despite the lack of a common theoretical framework, several phenomena including mixed emotions, emodiversity, meta-emotions, awe, among several others, have been defined as being emotionally complex. In this conceptual analysis, I aim to integrate the diversity of emotional complexity by describing various phenomena associated with this construct. This integration offers a more comprehensive panorama of the current usage of the concept of emotional complexity compared to previous attempts to consolidate the field. Furthermore, this conceptual analysis intends to disentangle the emotional fingerprints of emotional complexity. In particular, I present evidence and arguments showing that complex emotions can be characterized as having specific facial expressions, appraisals, and functional significance. Finally, I suggest that it is possible to describe emotional complexity using concepts and properties from the complex systems theory. Concepts such as the hierarchical organization of the affect system and emergent self-organization are used to explain current evidence on emotional complexity. I explain that applying complex systems theory to emotional complexity is not only theoretically convenient, but that complex systems theory also serves to advance new forms to conceptualize the affect system. The current conceptual analysis can help to organize current research and theory in order to encourage new research endeavors in the field of emotional complexity and acknowledge the importance of emotional complexity in models of affect, for which I suggest some specific guidelines.

Mapping the Passions: Toward a High-Dimensional Taxonomy of Emotional Experience and Expression

What would a comprehensive atlas of human emotions include? For 50 years, scientists have sought to map emotion-related experience, expression, physiology, and recognition in terms of the "basic six"-anger, disgust, fear, happiness, sadness, and surprise. Claims about the relationships between these six emotions and prototypical facial configurations have provided the basis for a long-standing debate over the diagnostic value of expression (for review and latest installment in this debate, see Barrett et al., p. 1). Building on recent empirical findings and methodologies, we offer an alternative conceptual and methodological approach that reveals a richer taxonomy of emotion. Dozens of distinct varieties of emotion are reliably distinguished by language, evoked in distinct circumstances, and perceived in distinct expressions of the face, body, and voice. Traditional models-both the basic six and affective-circumplex model (valence and arousal)-capture a fraction of the systematic variability in emotional response. In contrast, emotion-related responses (e.g., the smile of embarrassment, triumphant postures, sympathetic vocalizations, blends of distinct expressions) can be explained by richer models of emotion. Given these developments, we discuss why tests of a basic-six model of emotion are not tests of the diagnostic value of facial expression more generally. Determining the full extent of what facial expressions can tell us, marginally and in conjunction with other behavioral and contextual cues, will require mapping the high-dimensional, continuous space of facial, bodily, and vocal signals onto richly multifaceted experiences using large-scale statistical modeling and machine-learning methods.

What the face displays: Mapping 28 emotions conveyed by naturalistic expression

What emotions do the face and body express? Guided by new conceptual and quantitative approaches (Cowen, Elfenbein, Laukka, & Keltner, 2018; Cowen & Keltner, 2017, 2018), we explore the taxonomy of emotion recognized in facial-bodily expression. Participants (N = 1,794; 940 female, ages 18-76 years) judged the emotions captured in 1,500 photographs of facial-bodily expression in terms of emotion categories, appraisals, free response, and ecological validity. We find that facial-bodily expressions can reliably signal at least 28 distinct categories of emotion that occur in everyday life. Emotion categories, more so than appraisals such as valence and arousal, organize emotion recognition. However, categories of emotion recognized in naturalistic facial and bodily behavior are not discrete but bridged by smooth gradients that correspond to continuous variations in meaning. Our results support a novel view that emotions occupy a high-dimensional space of categories bridged by smooth gradients of meaning. They offer an approximation of a taxonomy of facial-bodily expressions, visualized within an online interactive map. (PsycInfo Database Record (c) 2020 APA, all rights reserved).