Understanding Naturalistic Facial Expressions with Deep Learning and Multimodal Large Language Models

This paper provides a comprehensive overview of affective computing systems for facial expression recognition (FER) research in naturalistic contexts. The first section presents an updated account of user-friendly FER toolboxes incorporating state-of-the-art deep learning models and elaborates on their neural architectures, datasets, and performances across domains. These sophisticated FER toolboxes can robustly address a variety of challenges encountered in the wild such as variations in illumination and head pose, which may otherwise impact recognition accuracy. The second section of this paper discusses multimodal large language models (MLLMs) and their potential applications in affective science. MLLMs exhibit human-level capabilities for FER and enable the quantification of various contextual variables to provide context-aware emotion inferences. These advancements have the potential to revolutionize current methodological approaches for studying the contextual influences on emotions, leading to the development of contextualized emotion models.

The increasing instance of negative emotion reduce the performance of emotion recognition

Introduction

Emotion recognition plays a crucial role in affective computing. Recent studies have demonstrated that the fuzzy boundaries among negative emotions make recognition difficult. However, to the best of our knowledge, no formal study has been conducted thus far to explore the effects of increased negative emotion categories on emotion recognition.

Methods

A dataset of three sessions containing consistent non-negative emotions and increased types of negative emotions was designed and built which consisted the electroencephalogram (EEG) and the electrocardiogram (ECG) recording of 45 participants.

Results

The results revealed that as negative emotion categories increased, the recognition rates decreased by more than 9%. Further analysis depicted that the discriminative features gradually reduced with an increase in the negative emotion types, particularly in the θ, α, and β frequency bands.

Discussion

This study provided new insight into the balance of emotion-inducing stimuli materials.

Modeling Subjective Affect Annotations with Multi-Task Learning

In supervised learning, the generalization capabilities of trained models are based on the available annotations. Usually, multiple annotators are asked to annotate the dataset samples and, then, the common practice is to aggregate the different annotations by computing average scores or majority voting, and train and test models on these aggregated annotations. However, this practice is not suitable for all types of problems, especially when the subjective information of each annotator matters for the task modeling. For example, emotions experienced while watching a video or evoked by other sources of content, such as news headlines, are subjective: different individuals might perceive or experience different emotions. The aggregated annotations in emotion modeling may lose the subjective information and actually represent an annotation bias. In this paper, we highlight the weaknesses of models that are trained on aggregated annotations for modeling tasks related to affect. More concretely, we compare two generic Deep Learning architectures: a Single-Task (ST) architecture and a Multi-Task (MT) architecture. While the ST architecture models single emotional perception each time, the MT architecture jointly models every single annotation and the aggregated annotations at once. Our results show that the MT approach can more accurately model every single annotation and the aggregated annotations when compared to methods that are directly trained on the aggregated annotations. Furthermore, the MT approach achieves state-of-the-art results on the COGNIMUSE, IEMOCAP, and SemEval_2007 benchmarks.

The Multimodal Emotion Information Analysis of E-Commerce Online Pricing in Electronic Word of Mouth

E-commerce has developed rapidly, and product promotion refers to how e-commerce promotes consumers' consumption activities. The demand and computational complexity in the decision-making process are urgent problems to be solved to optimize dynamic pricing decisions of the e-commerce product lines. Therefore, a Q-learning algorithm model based on the neural network is proposed on the premise of multimodal emotion information recognition and analysis, and the dynamic pricing problem of the product line is studied. The results show that a multi-modal fusion model is established through the multi-modal fusion of speech emotion recognition and image emotion recognition to classify consumers' emotions. Then, they are used as auxiliary materials for understanding and analyzing the market demand. The long short-term memory (LSTM) classifier performs excellent image feature extraction. The accuracy rate is 3.92%-6.74% higher than that of other similar classifiers, and the accuracy rate of the image single-feature optimal model is 9.32% higher than that of the speech single-feature model.