Hierarchical Attention-Based Multimodal Fusion Network for Video Emotion Recognition

Cross-subject EEG-based emotion recognition through dynamic optimization of random forest with sparrow search algorithm

The objective of EEG-based emotion recognition is to classify emotions by decoding signals, with potential applications in the fields of artificial intelligence and bioinformatics. Cross-subject emotion recognition is more difficult than intra-subject emotion recognition. The poor adaptability of classification model parameters is a significant factor of low accuracy in cross-subject emotion recognition. We propose a model of a dynamically optimized Random Forest based on the Sparrow Search Algorithm (SSA-RF). The decision trees number (DTN) and the leave minimum number (LMN) of the RF are dynamically optimized by the SSA. 12 features are used to construct feature combinations for selecting the optimal feature combination. DEAP and SEED datasets are employed for testing the performance of SSA-RF. The experimental results show that the accuracy of binary classification is 76.81% on DEAP, and the accuracy of triple classification is 75.96% on SEED based on SSA-RF, which are both higher than that of traditional RF. This study provides new insights for the development of cross-subject emotion recognition, and has significant theoretical value.

Sparse Spatial-Temporal Emotion Graph Convolutional Network for Video Emotion Recognition

Video emotion recognition has attracted increasing attention. Most existing approaches are based on the spatial features extracted from video frames. The context information and their relationships in videos are often ignored. Thus, the performance of existing approaches is restricted. In this study, we propose a sparse spatial-temporal emotion graph convolutional network-based video emotion recognition method (SE-GCN). For the spatial graph, the emotional relationship between any two emotion proposal regions is first calculated and the sparse spatial graph is constructed according to the emotional relationship. For the temporal graph, the emotional information contained in each emotion proposal region is first analyzed and the sparse temporal graph is constructed by using the emotion proposal regions with rich emotional cues. Then, the reasoning features of the emotional relationship are obtained by the spatial-temporal GCN. Finally, the features of the emotion proposal regions and the spatial-temporal relationship features are fused to recognize the video emotion. Extensive experiments are conducted on four challenging benchmark datasets, that is, MHED, HEIV, VideoEmotion-8, and Ekman-6. The experimental results demonstrate that the proposed method achieves state-of-the-art performance.

Region Dual Attention-Based Video Emotion Recognition

To solve the emotional differences between different regions of the video frame and make use of the interrelationship between different regions, a region dual attention-based video emotion recognition method (RDAM) is proposed. RDAM takes as input video frame sequences and learns a discriminatory video emotion representation that can make full use of the emotional differences of different regions and the interrelationship between regions. Specifically, we construct two parallel attention modules: one is the regional location attention module, which generates a weight value for each feature region to identify the relative importance of different regions. Based on the weight, the emotion feature that can perceive the emotional sensitive region is generated. The other is the regional relationship attention module, which generates a region relation matrix that represents the interrelationship of different regions of a video frame. Based on the region relation matrix, the emotion feature that can perceive interrelationship between different regions is generated. The outputs of these two attention modules are fused to produce the emotional features of video frames. Then, the features of video frame sequences are fused by attention-based fusion network, and the final emotion feature of the video is produced. The experimental results on the video emotion recognition data sets show that the proposed method outperforms the other related works.