An Investigation of Olfactory-Enhanced Video on EEG-Based Emotion Recognition

Lightweight attention mechanisms for EEG emotion recognition for brain computer interface

BACKGROUND

In the realm of brain-computer interfaces (BCI), identifying emotions from electroencephalogram (EEG) data is a difficult endeavor because of the volume of data, the intricacy of the signals, and the several channels that make up the signals.

NEW METHODS

Using dual-stream structure scaling and multiple attention mechanisms (LDMGEEG), a lightweight network is provided to maximize the accuracy and performance of EEG-based emotion identification. Reducing the number of computational parameters while maintaining the current level of classification accuracy is the aim. This network employs a symmetric dual-stream architecture to assess separately time-domain and frequency-domain spatio-temporal maps constructed using differential entropy features of EEG signals as inputs.

RESULT

The experimental results show that after significantly lowering the number of parameters, the model achieved the best possible performance in the field, with a 95.18 % accuracy on the SEED dataset.

COMPARISON WITH EXISTING METHODS

Moreover, it reduced the number of parameters by 98 % when compared to existing models.

CONCLUSION

The proposed method distinct channel-time/frequency-space multiple attention and post-attention methods enhance the model's ability to aggregate features and result in lightweight performance.

Analysis and recognition of a novel experimental paradigm for musical emotion brain-computer interface

Musical emotions have received increasing attention over the years. To better recognize the emotions by brain-computer interface (BCI), the random music-playing and sequential music-playing experimental paradigms are proposed and compared in this paper. Two experimental paradigms consist of three positive pieces, three neutral pieces and three negative pieces of music. Ten subjects participate in two experimental paradigms. The features of electroencephalography (EEG) signals are firstly analyzed in the time, frequency and spatial domains. To improve the effect of emotion recognition, a recognition model is proposed with the optimal channels selecting by Pearson's correlation coefficient, and the feature fusion combining differential entropy and wavelet packet energy. According to the analysis results, the features of sequential music-playing experimental paradigm are more different among three emotions. The classification results of sequential music-playing experimental paradigm are also better, and its average results of positive, neutral and negative emotions are 78.53%, 72.81% and 77.35%, respectively. The more obvious the changes of EEG induced by the emotions, the higher the classification accuracy will be. After analyzing two experimental paradigms, a better way for music to induce the emotions can be explored. Therefore, our research offers a novel perspective on affective BCIs.

A novel method for modeling effective connections between brain regions based on EEG signals and graph neural networks for motor imagery detection

Classified as biomedical signal processing, cerebral signal processing plays a key role in human-computer interaction (HCI) and medical diagnosis. The motor imagery (MI) problem is an important research area in this field. Accurate solutions to this problem will greatly affect real-world applications. Most of the proposed methods are based on raw signal processing techniques. Known as prior knowledge, the structural-functional information and interregional connections can improve signal processing accuracy. It is possible to correctly perceive the generated signals by considering the brain structure (i.e. anatomical units), the source of signals, and the structural-functional dependence of different brain regions (i.e. effective connection) that are the semantic generators of signals. This study employed electroencephalograph (EEG) signals based on the activity of brain regions (cortex) and effective connections between brain regions based on dynamic causal modeling to solve the MI problem. EEG signals, as well as effective connections between brain regions to improve the interpretability of MI action, were fed into the architecture of Graph Convolutional Neural Network (GCN). The proposed model allowed GCN to extract more discriminative features. The results indicated that the proposed method was successful in developing a model with a MI detection accuracy of 93.73%.

TASA: Temporal Attention With Spatial Autoencoder Network for Odor-Induced Emotion Classification Using EEG

The olfactory system enables humans to smell different odors, which are closely related to emotions. The high temporal resolution and non-invasiveness of Electroencephalogram (EEG) make it suitable to objectively study human preferences for odors. Effectively learning the temporal dynamics and spatial information from EEG is crucial for detecting odor-induced emotional valence. In this paper, we propose a deep learning architecture called Temporal Attention with Spatial Autoencoder Network (TASA) for predicting odor-induced emotions using EEG. TASA consists of a filter-bank layer, a spatial encoder, a time segmentation layer, a Long Short-Term Memory (LSTM) module, a multi-head self-attention (MSA) layer, and a fully connected layer. We improve upon the previous work by utilizing a two-phase learning framework, using the autoencoder module to learn the spatial information among electrodes by reconstructing the given input with a latent representation in the spatial dimension, which aims to minimize information loss compared to spatial filtering with CNN. The second improvement is inspired by the continuous nature of the olfactory process; we propose to use LSTM-MSA in TASA to capture its temporal dynamics by learning the intercorrelation among the time segments of the EEG. TASA is evaluated on an existing olfactory EEG dataset and compared with several existing deep learning architectures to demonstrate its effectiveness in predicting olfactory-triggered emotional responses. Interpretability analyses with DeepLIFT also suggest that TASA learns spatial-spectral features that are relevant to olfactory-induced emotion recognition.

EESCN: A novel spiking neural network method for EEG-based emotion recognition

BACKGROUND AND OBJECTIVE

Although existing artificial neural networks have achieved good results in electroencephalograph (EEG) emotion recognition, further improvements are needed in terms of bio-interpretability and robustness. In this research, we aim to develop a highly efficient and high-performance method for emotion recognition based on EEG.

METHODS

We propose an Emo-EEGSpikeConvNet (EESCN), a novel emotion recognition method based on spiking neural network (SNN). It consists of a neuromorphic data generation module and a NeuroSpiking framework. The neuromorphic data generation module converts EEG data into 2D frame format as input to the NeuroSpiking framework, while the NeuroSpiking framework is used to extract spatio-temporal features of EEG for classification.

RESULTS

EESCN achieves high emotion recognition accuracies on DEAP and SEED-IV datasets, ranging from 94.56% to 94.81% on DEAP and a mean accuracy of 79.65% on SEED-IV. Compared to existing SNN methods, EESCN significantly improves EEG emotion recognition performance. In addition, it also has the advantages of faster running speed and less memory footprint.

CONCLUSIONS

EESCN has shown excellent performance and efficiency in EEG-based emotion recognition with potential for practical applications requiring portability and resource constraints.

BiTCAN: A emotion recognition network based on saliency in brain cognition

In recent years, with the continuous development of artificial intelligence and brain-computer interfaces, emotion recognition based on electroencephalogram (EEG) signals has become a prosperous research direction. Due to saliency in brain cognition, we construct a new spatio-temporal convolutional attention network for emotion recognition named BiTCAN. First, in the proposed method, the original EEG signals are de-baselined, and the two-dimensional mapping matrix sequence of EEG signals is constructed by combining the electrode position. Second, on the basis of the two-dimensional mapping matrix sequence, the features of saliency in brain cognition are extracted by using the Bi-hemisphere discrepancy module, and the spatio-temporal features of EEG signals are captured by using the 3-D convolution module. Finally, the saliency features and spatio-temporal features are fused into the attention module to further obtain the internal spatial relationships between brain regions, and which are input into the classifier for emotion recognition. Many experiments on DEAP and SEED (two public datasets) show that the accuracies of the proposed algorithm on both are higher than 97%, which is superior to most existing emotion recognition algorithms.

Joint domain symmetry and predictive balance for cross-dataset EEG emotion recognition

BACKGROUND

Cross-dataset EEG emotion recognition is an extremely challenging task, since data distributions of EEG from different datasets are greatly different, which makes the universal models yield unsatisfactory results. Although there are many methods have been proposed to reduce cross-dataset distribution discrepancies, they still neglected the following two problems. (1) Label space inconsistency: emotional label spaces of subjects from different datasets are different; (2) Uncertainty propagation: the uncertainty of misclassified emotion samples will propagate between datasets.

NEW METHOD

To solve these problems, we propose a novel method called domain symmetry and predictive balance (DSPB). For the problem of label space inconsistency, a domain symmetry module is designed to make label spaces of source and target domain to be the same, which randomly selects samples from the source domain and put into the target domain. For the problem of uncertainty propagation, a predictive balance module is proposed to reduce the prediction score of incorrect samples and then effectively reduce distribution differences between EEG from different datasets.

RESULTS

Experimental results show that our method achieve 61.48% average accuracies on the three cross-dataset tasks. Moreover, we find that gamma is the most relevant to emotion recognition among the five frequency bands, and the prefrontal and temporal brain regions are the channels carrying the most emotional information among the 62 brain channels.

COMPARISON WITH EXISTING METHODS

Compared with the partial domain adaptation method (SPDA) and the unsupervised domain adaptation (MS-MDA), our method improves average accuracies by 15.60% and 23.11%, respectively.

CONCLUSION

Besides, data distributions of EEG from different datasets but with the same emotional labels have been well aligned, which demonstrates the effectiveness of DSPB.

Application of cluster repeated mini-batch training method to classify electroencephalography for grab and lift tasks

Modern deep neural network training is based on mini-batch stochastic gradient optimization. While using extensive mini-batches improves the computational parallelism, the small batch training proved that it delivers improved generalization performance and allows a significantly smaller memory, which might also improve machine throughput. However, mini-batch size and characteristics, a key factor for training deep neural networks, has not been sufficiently investigated in training correlated group features and looping with highly complex ones. In addition, the unsupervised learning method clusters the data into different groups with similar properties to make the training process more stable and faster. Then, the supervised learning algorithm was applied with the cluster repeated mini-batch training (CRMT) methods. The CRMT algorithm changed the random minibatch characteristics in the training step into training in order of clusters. Specifically, the self-organizing maps (SOM) were used to cluster the information into n groups based on the dataset's labels Then, neural network models (ANN) were trained with each cluster using the cluster repeated mini-batch training method. Experiments conducted on EEG datasets demonstrate the survey of the proposed method and optimize it. In addition, the results in our research outperform other state-of-the-art methods.

Multi-scale fusion visual attention network for facial micro-expression recognition

Introduction

Micro-expressions are facial muscle movements that hide genuine emotions. In response to the challenge of micro-expression low-intensity, recent studies have attempted to locate localized areas of facial muscle movement. However, this ignores the feature redundancy caused by the inaccurate locating of the regions of interest.

Methods

This paper proposes a novel multi-scale fusion visual attention network (MFVAN), which learns multi-scale local attention weights to mask regions of redundancy features. Specifically, this model extracts the multi-scale features of the apex frame in the micro-expression video clips by convolutional neural networks. The attention mechanism focuses on the weights of local region features in the multi-scale feature maps. Then, we mask operate redundancy regions in multi-scale features and fuse local features with high attention weights for micro-expression recognition. The self-supervision and transfer learning reduce the influence of individual identity attributes and increase the robustness of multi-scale feature maps. Finally, the multi-scale classification loss, mask loss, and removing individual identity attributes loss joint to optimize the model.

Results

The proposed MFVAN method is evaluated on SMIC, CASME II, SAMM, and 3DB-Combined datasets that achieve state-of-the-art performance. The experimental results show that focusing on local at the multi-scale contributes to micro-expression recognition.

Discussion

This paper proposed MFVAN model is the first to combine image generation with visual attention mechanisms to solve the combination challenge problem of individual identity attribute interference and low-intensity facial muscle movements. Meanwhile, the MFVAN model reveal the impact of individual attributes on the localization of local ROIs. The experimental results show that a multi-scale fusion visual attention network contributes to micro-expression recognition.

An innovative EEG-based emotion recognition using a single channel-specific feature from the brain rhythm code method

Introduction

Efficiently recognizing emotions is a critical pursuit in brain-computer interface (BCI), as it has many applications for intelligent healthcare services. In this work, an innovative approach inspired by the genetic code in bioinformatics, which utilizes brain rhythm code features consisting of δ, θ, α, β, or γ, is proposed for electroencephalography (EEG)-based emotion recognition.

Methods

These features are first extracted from the sequencing technique. After evaluating them using four conventional machine learning classifiers, an optimal channel-specific feature that produces the highest accuracy in each emotional case is identified, so emotion recognition through minimal data is realized. By doing so, the complexity of emotion recognition can be significantly reduced, making it more achievable for practical hardware setups.

Results

The best classification accuracies achieved for the DEAP and MAHNOB datasets range from 83-92%, and for the SEED dataset, it is 78%. The experimental results are impressive, considering the minimal data employed. Further investigation of the optimal features shows that their representative channels are primarily on the frontal region, and associated rhythmic characteristics are typical of multiple kinds. Additionally, individual differences are found, as the optimal feature varies with subjects.

Discussion

Compared to previous studies, this work provides insights into designing portable devices, as only one electrode is appropriate to generate satisfactory performances. Consequently, it would advance the understanding of brain rhythms, which offers an innovative solution for classifying EEG signals in diverse BCI applications, including emotion recognition.

Recognizing emotions induced by wearable haptic vibration using noninvasive electroencephalogram

The integration of haptic technology into affective computing has led to a new field known as affective haptics. Nonetheless, the mechanism underlying the interaction between haptics and emotions remains unclear. In this paper, we proposed a novel haptic pattern with adaptive vibration intensity and rhythm according to the volume, and applied it into the emotional experiment paradigm. To verify its superiority, the proposed haptic pattern was compared with an existing haptic pattern by combining them with conventional visual-auditory stimuli to induce emotions (joy, sadness, fear, and neutral), and the subjects' EEG signals were collected simultaneously. The features of power spectral density (PSD), differential entropy (DE), differential asymmetry (DASM), and differential caudality (DCAU) were extracted, and the support vector machine (SVM) was utilized to recognize four target emotions. The results demonstrated that haptic stimuli enhanced the activity of the lateral temporal and prefrontal areas of the emotion-related brain regions. Moreover, the classification accuracy of the existing constant haptic pattern and the proposed adaptive haptic pattern increased by 7.71 and 8.60%, respectively. These findings indicate that flexible and varied haptic patterns can enhance immersion and fully stimulate target emotions, which are of great importance for wearable haptic interfaces and emotion communication through haptics.

SIFIAE: An adaptive emotion recognition model with EEG feature-label inconsistency consideration

BACKGROUND

A common but easily overlooked affective overlap problem has not been received enough attention in electroencephalogram (EEG)-based emotion recognition research. In real life, affective overlap refers to the current emotional state of human being is sometimes influenced easily by his/her historical mood. In stimulus-evoked EEG collection experiment, due to the short rest interval in consecutive trials, the inner mechanisms of neural responses make subjects cannot switch their emotion state easily and quickly, which might lead to the affective overlap. For example, we might be still in sad state to some extent even if we are watching a comedy because we just saw a tragedy before. In pattern recognition, affective overlap usually means that there exists the feature-label inconsistency in EEG data.

NEW METHODS

To alleviate the impact of inconsistent EEG data, we introduce a variable to adaptively explore the sample inconsistency in emotion recognition model development. Then, we propose a semi-supervised emotion recognition model for joint sample inconsistency and feature importance exploration (SIFIAE). Accordingly, an efficient optimization method to SIFIAE model is proposed.

RESULTS

Extensive experiments on the SEED-V dataset demonstrate the effectiveness of SIFIAE. Specifically, SIFIAE achieves 69.10%, 67.01%, 71.5%, 73.26%, 72.07% and 71.35% average accuracies in six cross-session emotion recognition tasks.

CONCLUSION

The results illustrated that the sample weights have a rising trend in the beginning of most trials, which coincides with the affective overlap hypothesis. The feature importance factor indicated the critical bands and channels are more obvious compared with some models without considering EEG feature-label inconsistency.

GCF2-Net: global-aware cross-modal feature fusion network for speech emotion recognition

Emotion recognition plays an essential role in interpersonal communication. However, existing recognition systems use only features of a single modality for emotion recognition, ignoring the interaction of information from the different modalities. Therefore, in our study, we propose a global-aware Cross-modal feature Fusion Network (GCF²-Net) for recognizing emotion. We construct a residual cross-modal fusion attention module (ResCMFA) to fuse information from multiple modalities and design a global-aware module to capture global details. More specifically, we first use transfer learning to extract wav2vec 2.0 features and text features fused by the ResCMFA module. Then, cross-modal fusion features are fed into the global-aware module to capture the most essential emotional information globally. Finally, the experiment results have shown that our proposed method has significant advantages than state-of-the-art methods on the IEMOCAP and MELD datasets, respectively.

eSEE-d: Emotional State Estimation Based on Eye-Tracking Dataset

Affective state estimation is a research field that has gained increased attention from the research community in the last decade. Two of the main catalysts for this are the advancement in the data analysis using artificial intelligence and the availability of high-quality video. Unfortunately, benchmarks and public datasets are limited, thus making the development of new methodologies and the implementation of comparative studies essential. The current work presents the eSEE-d database, which is a resource to be used for emotional State Estimation based on Eye-tracking data. Eye movements of 48 participants were recorded as they watched 10 emotion-evoking videos, each of them followed by a neutral video. Participants rated four emotions (tenderness, anger, disgust, sadness) on a scale from 0 to 10, which was later translated in terms of emotional arousal and valence levels. Furthermore, each participant filled three self-assessment questionnaires. An extensive analysis of the participants’ answers to the questionnaires’ self-assessment scores as well as their ratings during the experiments is presented. Moreover, eye and gaze features were extracted from the low-level eye-recorded metrics, and their correlations with the participants’ ratings are investigated. Finally, we take on the challenge to classify arousal and valence levels based solely on eye and gaze features, leading to promising results. In particular, the Deep Multilayer Perceptron (DMLP) network we developed achieved an accuracy of 92% in distinguishing positive valence from non-positive and 81% in distinguishing low arousal from medium arousal. The dataset is made publicly available.