A Fine-grained Hemispheric Asymmetry Network for accurate and interpretable EEG-based emotion classification

In this work, we propose a Fine-grained Hemispheric Asymmetry Network (FG-HANet), an end-to-end deep learning model that leverages hemispheric asymmetry features within 2-Hz narrow frequency bands for accurate and interpretable emotion classification over raw EEG data. In particular, the FG-HANet extracts features not only from original inputs but also from their mirrored versions, and applies Finite Impulse Response (FIR) filters at a granularity as fine as 2-Hz to acquire fine-grained spectral information. Furthermore, to guarantee sufficient attention to hemispheric asymmetry features, we tailor a three-stage training pipeline for the FG-HANet to further boost its performance. We conduct extensive evaluations on two public datasets, SEED and SEED-IV, and experimental results well demonstrate the superior performance of the proposed FG-HANet, i.e. 97.11% and 85.70% accuracy, respectively, building a new state-of-the-art. Our results also reveal the hemispheric dominance under different emotional states and the hemisphere asymmetry within 2-Hz frequency bands in individuals. These not only align with previous findings in neuroscience but also provide new insights into underlying emotion generation mechanisms.

Emotion recognition using multi-scale EEG features through graph convolutional attention network

Emotion recognition via electroencephalogram (EEG) signals holds significant promise across various domains, including the detection of emotions in patients with consciousness disorders, assisting in the diagnosis of depression, and assessing cognitive load. This process is critically important in the development and research of brain-computer interfaces, where precise and efficient recognition of emotions is paramount. In this work, we introduce a novel approach for emotion recognition employing multi-scale EEG features, denominated as the Dynamic Spatial-Spectral-Temporal Network (DSSTNet). DSSTNet includes three main parts, the first is spatial features extractor, which converts EEG signal into graph structure data, and uses graph convolutional network (GCN) to dynamically optimize the adjacency matrix during the training process to obtain the spatial features between the channels. Next, band attention module is composed of semi-global pooling, localized cross-band interaction and adaptive weighting, which further extracts frequency information. Finally, through the temporal features extractor, the deep temporal information is extracted by stacking several one-dimensional convolutional layers. In addition, in order to improve the performance of emotion recognition and filter valid channels, we add a ℓ2,1-norm regularization term to the loss function to make the adjacency matrix constraint sparse. This makes it easier to preserve emotionally relevant channels and eliminate noise in irrelevant channel. Combined with the channel selection method of graph theory, a small number of optimal channels are selected. We used a self-constructed dataset TJU-EmoEEG and a publicly available SEED dataset to evaluate DSSTNet. The experimental results demonstrate that DSSTNet outperforms current state-of-the-art (SOTA) methods in emotional recognition tasks.

Multifunctional Conductive Hydrogel for Sensing Underwater Applications and Wearable Electroencephalogram Recording

Flexible electronics have been rapidly advancing and have garnered significant interest in monitoring physiological activities and health conditions. However, flexible electronics are prone to detachment in humid environments, so developing human-friendly flexible electronic devices that can effectively monitor human movement under various aquatic conditions and function as flexible electrodes remains a significant challenge. Here, we report a strongly adherent, self-healing, and swelling-resistant conductive hydrogel formed by combining the dual synergistic effects of hydrogen bonding and dipole-dipole interactions. The hydrogel has a commendable linear operating range (∼200% strain, GF = 1.44), stability of electrical signals for 200 cycles, excellent conductivity (2.18 S m-1), self-healing properties (∼30 min), and durable underwater adhesion stability. The conductive hydrogel can be developed into a flexible electronic sensor for detecting motion signals, such as joint flexion and swallowing, as well as for real-time underwater communication using Morse code. Additionally, the integration of this polymer with a low contact impedance facilitates real-time, high-fidelity detection of electroencephalogram (EEG) signals, serving as a flexible electrode. It is believed that our hydrogel will have good prospects in future wearable electronics.

Multi-branch convolutional neural network with cross-attention mechanism for emotion recognition

Research on emotion recognition is an interesting area because of its wide-ranging applications in education, marketing, and medical fields. This study proposes a multi-branch convolutional neural network model based on cross-attention mechanism (MCNN-CA) for accurate recognition of different emotions. The proposed model provides automated extraction of relevant features from multimodal data and fusion of feature maps from diverse sources as modules for the subsequent emotion recognition. In the feature extraction stage, various convolutional neural networks were designed to extract critical information from multiple dimensional features. The feature fusion module was used to enhance the inter-correlation between features based on channel-efficient attention mechanism. This innovation proves effective in fusing distinctive features within a single mode and across different modes. The model was assessed based on EEG emotion recognition experiments on the SEED and SEED-IV datasets. Furthermore, the efficiency of the proposed model was evaluated via multimodal emotion experiments using EEG and text data from the ZuCo dataset. Comparative analysis alongside contemporary studies shows that our model excels in terms of accuracy, precision, recall, and F1-score.

A fine-tuning deep residual convolutional neural network for emotion recognition based on frequency-channel matrices representation of one-dimensional electroencephalography

Emotion recognition (ER) plays a crucial role in enabling machines to perceive human emotional and psychological states, thus enhancing human-machine interaction. Recently, there has been a growing interest in ER based on electroencephalogram (EEG) signals. However, due to the noisy, nonlinear, and nonstationary properties of electroencephalography signals, developing an automatic and high-accuracy ER system is still a challenging task. In this study, a pretrained deep residual convolutional neural network model, including 17 convolutional layers and one fully connected layer with transfer learning technique in combination frequency-channel matrices (FCM) of two-dimensional data based on Welch power spectral density estimate from the one-dimensional EEG data has been proposed for improving the ER by automatically learning the underlying intrinsic features of multi-channel EEG data. The experiment result shows a mean accuracy of 93.61 ± 0.84%, a mean precision of 94.70 ± 0.60%, a mean sensitivity of 95.13 ± 1.02%, a mean specificity of 91.04 ± 1.02%, and a mean F1-score of 94.91 ± 0.68%, respectively using 5-fold cross-validation on the DEAP dataset. Meanwhile, to better explore and understand how the proposed model works, we noted that the ranking of clustering effect of FCM for the same category by employing the t-distributed stochastic neighbor embedding strategy is: softmax layer activation is the best, the middle convolutional layer activation is the second, and the early max pooling layer activation is the worst. These findings confirm the promising potential of combining deep learning approaches with transfer learning techniques and FCM for effective ER tasks.

ST-SHAP: A hierarchical and explainable attention network for emotional EEG representation learning and decoding

BACKGROUND

Emotion recognition using electroencephalogram (EEG) has become a research hotspot in the field of human-computer interaction, how to sufficiently learn complex spatial-temporal representations of emotional EEG data and obtain explainable model prediction results are still great challenges.

NEW METHOD

In this study, a novel hierarchical and explainable attention network ST-SHAP which combines the Swin Transformer (ST) and SHapley Additive exPlanations (SHAP) technique is proposed for automatic emotional EEG classification. Firstly, a 3D spatial-temporal feature of emotional EEG data is generated via frequency band filtering, temporal segmentation, spatial mapping, and interpolation to fully preserve important spatial-temporal-frequency characteristics. Secondly, a hierarchical attention network is devised to sufficiently learn an abstract spatial-temporal representation of emotional EEG and perform classification. Concretely, in this decoding model, the W-MSA module is used for modeling correlations within local windows, the SW-MSA module allows for information interactions between different local windows, and the patch merging module further facilitates local-to-global multiscale modeling. Finally, the SHAP method is utilized to discover important brain regions for emotion processing and improve the explainability of the Swin Transformer model.

RESULTS

Two benchmark datasets, namely SEED and DREAMER, are used for classification performance evaluation. In the subject-dependent experiments, for SEED dataset, ST-SHAP achieves an average accuracy of 97.18%, while for DREAMER dataset, the average accuracy is 96.06% and 95.98% on arousal and valence dimension respectively. In addition, important brain regions that conform to prior knowledge of neurophysiology are discovered via a data-driven approach for both datasets.

COMPARISON WITH EXISTING METHODS

In terms of subject-dependent and subject-independent emotional EEG decoding accuracies, our method outperforms several closely related existing methods.

CONCLUSION

These experimental results fully prove the effectiveness and superiority of our proposed algorithm.

Develop an emotion recognition system using jointly connectivity between electroencephalogram and electrocardiogram signals

This study pioneers an innovative approach to improve the accuracy and dependability of emotion recognition (ER) systems by integrating electroencephalogram (EEG) with electrocardiogram (ECG) data. We propose a novel method of estimating effective connectivity (EC) to capture the dynamic interplay between the heart and brain during emotions of happiness, disgust, fear, and sadness. Leveraging three EC estimation techniques (Granger causality (GC), partial directed coherence (PDC) and directed transfer function (DTF)), we feed the resulting EC representations as inputs into convolutional neural networks (CNNs), namely ResNet-18 and MobileNetV2, known for their swift and superior performance. To evaluate this approach, we have used EEG and ECG data from the public MAHNOB-HCI database through 5-fold cross-validation criterion. Remarkably, our approach achieves an average accuracy of 97.34 ± 1.19 and 96.53 ± 3.54 for DTF images within the alpha frequency band using ResNet-18 and MobileNetV2, respectively. Comparative analyses in comparison to cutting-edge research unequivocally prove the efficacy of augmenting ECG with EEG, showcasing substantial improvements in ER performance across the spectrum of emotions studied.

Graph convolution network-based eeg signal analysis: a review

With the advancement of artificial intelligence technology, more and more effective methods are being used to identify and classify Electroencephalography (EEG) signals to address challenges in healthcare and brain-computer interface fields. The applications and major achievements of Graph Convolution Network (GCN) techniques in EEG signal analysis are reviewed in this paper. Through an exhaustive search of the published literature, a module-by-module discussion is carried out for the first time to address the current research status of GCN. An exhaustive classification of methods and a systematic analysis of key modules, such as brain map construction, node feature extraction, and GCN architecture design, are presented. In addition, we pay special attention to several key research issues related to GCN. This review enhances the understanding of the future potential of GCN in the field of EEG signal analysis. At the same time, several valuable development directions are sorted out for researchers in related fields, such as analysing the applicability of different GCN layers, building task-oriented GCN models, and improving adaptation to limited data.

Improved BCI calibration in multimodal emotion recognition using heterogeneous adversarial transfer learning

The use of brain-computer interface (BCI) technology to identify emotional states has gained significant interest, especially with the rise of virtual reality (VR) applications. However, the extensive calibration required for precise emotion recognition models presents a significant challenge, particularly for sensitive groups such as children, elderly, and patients. This study presents a novel approach that utilizes heterogeneous adversarial transfer learning (HATL) to synthesize electroencephalography (EEG) data from various other signal modalities, reducing the need for lengthy calibration phases. We benchmark the efficacy of three generative adversarial network (GAN) architectures, such as conditional GAN (CGAN), conditional Wasserstein GAN (CWGAN), and CWGAN with gradient penalty (CWGAN-GP) within this framework. The proposed framework is rigorously tested on two conventional open sourced datasets, SEED-V and DEAP. Additionally, the framework was applied to an immersive three-dimensional (3D) dataset named GraffitiVR, which we collected to capture the emotional and behavioral reactions of individuals experiencing urban graffiti in a VR environment. This expanded application provides insights into emotion recognition frameworks in VR settings, providing a wider range of contexts for assessing our methodology. When the accuracy of emotion recognition classifiers trained with CWGAN-GP-generated EEG data combined with non-EEG sensory data was compared against those trained using a combination of real EEG and non-EEG sensory data, the accuracy ratios were 93% on the SEED-V dataset, 99% on the DEAP dataset, and 97% on the GraffitiVR dataset. Moreover, in the GraffitiVR dataset, using CWGAN-GP-generated EEG data with non-EEG sensory data for emotion recognition models resulted in up to a 30% reduction in calibration time compared to classifiers trained on real EEG data with non-EEG sensory data. These results underscore the robustness and versatility of the proposed approach, significantly enhancing emotion recognition processes across a variety of environmental settings.

Cross-modality fusion with EEG and text for enhanced emotion detection in English writing

Introduction

Emotion detection in written text is critical for applications in human-computer interaction, affective computing, and personalized content recommendation. Traditional approaches to emotion detection primarily leverage textual features, using natural language processing techniques such as sentiment analysis, which, while effective, may miss subtle nuances of emotions. These methods often fall short in recognizing the complex, multimodal nature of human emotions, as they ignore physiological cues that could provide richer emotional insights.

Methods

To address these limitations, this paper proposes Emotion Fusion-Transformer, a cross-modality fusion model that integrates EEG signals and textual data to enhance emotion detection in English writing. By utilizing the Transformer architecture, our model effectively captures contextual relationships within the text while concurrently processing EEG signals to extract underlying emotional states. Specifically, the Emotion Fusion-Transformer first preprocesses EEG data through signal transformation and filtering, followed by feature extraction that complements the textual embeddings. These modalities are fused within a unified Transformer framework, allowing for a holistic view of both the cognitive and physiological dimensions of emotion.

Results and discussion

Experimental results demonstrate that the proposed model significantly outperforms text-only and EEG-only approaches, with improvements in both accuracy and F1-score across diverse emotional categories. This model shows promise for enhancing affective computing applications by bridging the gap between physiological and textual emotion detection, enabling more nuanced and accurate emotion analysis in English writing.

Directional Spatial and Spectral Attention Network (DSSA Net) for EEG-based emotion recognition

Significant strides have been made in emotion recognition from Electroencephalography (EEG) signals. However, effectively modeling the diverse spatial, spectral, and temporal features of multi-channel brain signals remains a challenge. This paper proposes a novel framework, the Directional Spatial and Spectral Attention Network (DSSA Net), which enhances emotion recognition accuracy by capturing critical spatial-spectral-temporal features from EEG signals. The framework consists of three modules: Positional Attention (PA), Spectral Attention (SA), and Temporal Attention (TA). The PA module includes Vertical Attention (VA) and Horizontal Attention (HA) branches, designed to detect active brain regions from different orientations. Experimental results on three benchmark EEG datasets demonstrate that DSSA Net outperforms most competitive methods. On the SEED and SEED-IV datasets, it achieves accuracies of 96.61% and 85.07% for subject-dependent emotion recognition, respectively, and 87.03% and 75.86% for subject-independent recognition. On the DEAP dataset, it attains accuracies of 94.97% for valence and 94.73% for arousal. These results showcase the framework's ability to leverage both spatial and spectral differences across brain hemispheres and regions, enhancing classification accuracy for emotion recognition.

Relationships between feeding behaviors and emotions: an electroencephalogram (EEG) frequency analysis study

Feeding behaviors may be easily affected by emotions, both being based on brain activity; however, the relationships between them have not been explicitly defined. In this study, we investigated how emotional environments modulate subjective feelings, brain activity, and feeding behaviors. Electroencephalogram (EEG) recordings were obtained from healthy participants in conditions of virtual comfortable space (CS) and uncomfortable space (UCS) while eating chocolate, and the times required for eating it were measured. We found that the more participants tended to feel comfortable under the CS, the more it took time to eat in the UCS. However, the EEG emergence patterns in the two virtual spaces varied across the individuals. Upon focusing on the theta and low-beta bands, the strength of the mental condition and eating times were found to be guided by these frequency bands. The results determined that the theta and low-beta bands are likely important and relevant waves for feeding behaviors under emotional circumstances, following alterations in mental conditions.

Enhanced cross-dataset electroencephalogram-based emotion recognition using unsupervised domain adaptation

Emotion recognition holds great promise in healthcare and in the development of affect-sensitive systems such as brain-computer interfaces (BCIs). However, the high cost of labeled data and significant differences in electroencephalogram (EEG) signals among individuals limit the cross-domain application of EEG-based emotion recognition models. Addressing cross-dataset scenarios poses greater challenges due to changes in subject demographics, recording devices, and stimuli presented. To tackle these challenges, we propose an improved method for classifying EEG-based emotions across domains with different distributions. We propose a Gradual Proximity-guided Target Data Selection (GPTDS) technique, which gradually selects reliable target domain samples for training based on their proximity to the source clusters and the model's confidence in predicting them. This approach avoids negative transfer caused by diverse and unreliable samples. Additionally, we introduce a cost-effective test-time augmentation (TTA) technique named Prediction Confidence-aware Test-Time Augmentation (PC-TTA). Traditional TTA methods often face substantial computational burden, limiting their practical utility. By applying TTA only when necessary, based on the model's predictive confidence, our approach improves the model's performance during inference while minimizing computational costs compared to traditional TTA approaches. Experiments on the DEAP and SEED datasets demonstrate that our method outperforms state-of-the-art approaches, achieving accuracies of 67.44% when trained on DEAP and tested on SEED, and 59.68% vice versa, with improvements of 7.09% and 6.07% over the baseline. It excels in detecting both positive and negative emotions, highlighting its effectiveness for practical emotion recognition in healthcare applications. Moreover, our proposed PC-TTA technique reduces computational time by a factor of 15 compared to traditional full TTA approaches.

Dynamic domain adaptive EEG emotion recognition based on multi-source selection

Emotion recognition based on electroencephalogram (EEG) has always been a research hotspot. However, due to significant individual variations in EEG signals, cross-subject emotion recognition based on EEG remains a challenging issue to address. In this article, we propose a dynamic domain-adaptive EEG emotion recognition method based on multi-source selection. The method considers each subject as a separate domain, filters suitable source domains from multiple subjects by assessing their resemblance, then further extracts the common and domain-specific features of the source and target domains, and then employs dynamic domain adaptation to mitigate inter-domain discrepancies. Global domain differences and local subdomain differences are also considered, and a dynamic factor is added so that the model training process first focuses on global distribution differences and gradually switches to local subdomain distributions. We conducted cross-subject and cross-session experiments on the SEED and SEED-IV datasets, respectively, and the cross-subject accuracies were 89.76% and 65.28%; the cross-session experiments were 91.63% and 67.83%. The experimental outcomes affirm the efficacy of the EEG emotion recognition approach put forward in this paper.

Emotion regulation strategies explain associations of theta and Beta with positive affect

Maladaptive emotion regulation (ER) strategies are a transdiagnostic construct in psychopathology. ER depends on cognitive control, so brain activity associated with cognitive control, such as frontal theta and beta, may be a factor in ER. This study investigated the association of theta and beta power with positive affect and whether emotion regulation strategies explain this association. One hundred and twenty-one undergraduate students (mean age = 20.74, SD = 5.29; 73% women) completed self-report questionnaires, including the Emotion Regulation Questionnaire and the Positive and Negative Affect Schedule. Spectral analysis was performed on resting state frontal electroencephalogram activity that was collected for eight 1-min periods of alternating open and closed eyes. Relative beta and theta band power were extracted relative to global field power at frontal channels. Regression analysis revealed that positive affect is significantly predicted by theta power (β = 0.24, p = .007) and beta power (β = -0.33, p < .0001). There was an indirect effect of beta power on positive affect via reappraisal, but not suppression. Additionally, theta power significantly predicted suppression, but no indirect effect was observed between theta power and positive affect. These findings are consistent with a prior study reporting a positive and negative relationship between theta and beta power, respectively, and positive affect induction. This study elucidates how modulation of theta and beta bands link to ER strategies.

EEG-based emotion recognition using multi-scale dynamic CNN and gated transformer

Emotions play a crucial role in human thoughts, cognitive processes, and decision-making. EEG has become a widely utilized tool in emotion recognition due to its high temporal resolution, real-time monitoring capabilities, portability, and cost-effectiveness. In this paper, we propose a novel end-to-end emotion recognition method from EEG signals, called MSDCGTNet, which is based on the Multi-Scale Dynamic 1D CNN and the Gated Transformer. First, the Multi-Scale Dynamic CNN is used to extract complex spatial and spectral features from raw EEG signals, which not only avoids information loss but also reduces computational costs associated with the time-frequency conversion of signals. Then, the Gated Transformer Encoder is utilized to capture global dependencies of EEG signals. This encoder focuses on specific regions of the input sequence while reducing computational resources through parallel processing with the improved multi-head self-attention mechanisms. Third, the Temporal Convolution Network is used to extract temporal features from the EEG signals. Finally, the extracted abstract features are fed into a classification module for emotion recognition. The proposed method was evaluated on three publicly available datasets: DEAP, SEED, and SEED_IV. Experimental results demonstrate the high accuracy and efficiency of the proposed method for emotion recognition. This approach proves to be robust and suitable for various practical applications. By addressing challenges posed by existing methods, the proposed method provides a valuable and effective solution for the field of Brain-Computer Interface (BCI).

CIT-EmotionNet: convolution interactive transformer network for EEG emotion recognition

Emotion recognition is a significant research problem in affective computing as it has a lot of potential areas of application. One of the approaches in emotion recognition uses electroencephalogram (EEG) signals to identify the emotion of a person. However, effectively using the global and local features of EEG signals to improve the performance of emotion recognition is still a challenge. In this study, we propose a novel Convolution Interactive Transformer Network for EEG Emotion Recognition, known as CIT-EmotionNet, which efficiently integrates the global and local features of EEG signals. We convert the raw EEG signals into spatial-spectral representations, which serve as the inputs into the model. The model integrates convolutional neural network (CNN) and Transformer within a single framework in a parallel manner. We propose a Convolution Interactive Transformer module, which facilitates the interaction and fusion of local and global features extracted by CNN and Transformer respectively, thereby improving the average accuracy of emotion recognition. The proposed CIT-EmotionNet outperforms state-of-the-art methods, achieving an average recognition accuracy of 98.57% and 92.09% on two publicly available datasets, SEED and SEED-IV, respectively.

Attention-Based PSO-LSTM for Emotion Estimation Using EEG

Recent advances in emotion recognition through Artificial Intelligence (AI) have demonstrated potential applications in various fields (e.g., healthcare, advertising, and driving technology), with electroencephalogram (EEG)-based approaches demonstrating superior accuracy compared to facial or vocal methods due to their resistance to intentional manipulation. This study presents a novel approach to enhance EEG-based emotion estimation accuracy by emphasizing temporal features and efficient parameter space exploration. We propose a model combining Long Short-Term Memory (LSTM) with an attention mechanism to highlight temporal features in EEG data while optimizing LSTM parameters through Particle Swarm Optimization (PSO). The attention mechanism assigned weights to LSTM hidden states, and PSO dynamically optimizes the vital parameters, including units, batch size, and dropout rate. Using the DEAP and SEED datasets, which serve as benchmark datasets for emotion estimation research using EEG, we evaluate the model's performance. For the DEAP dataset, we conduct a four-class classification of combinations of high and low valence and arousal states. We perform a three-class classification of negative, neutral, and positive emotions for the SEED dataset. The proposed model achieves an accuracy of 0.9409 on the DEAP dataset, surpassing the previous state-of-the-art accuracy of 0.9100 reported by Lin et al. The model attains an accuracy of 0.9732 on the SEED dataset, recording one of the highest accuracies among the related research. These results demonstrate that integrating the attention mechanism with PSO significantly improves the accuracy of EEG-based emotion estimation, contributing to the advancement of emotion recognition technology.

Emotion Recognition Model of EEG Signals Based on Double Attention Mechanism

BACKGROUND

Emotions play a crucial role in people's lives, profoundly affecting their cognition, decision-making, and interpersonal communication. Emotion recognition based on brain signals has become a significant challenge in the fields of affective computing and human-computer interaction.

METHODS

Addressing the issue of inaccurate feature extraction and low accuracy of existing deep learning models in emotion recognition, this paper proposes a multi-channel automatic classification model for emotion EEG signals named DACB, which is based on dual attention mechanisms, convolutional neural networks, and bidirectional long short-term memory networks. DACB extracts features in both temporal and spatial dimensions, incorporating not only convolutional neural networks but also SE attention mechanism modules for learning the importance of different channel features, thereby enhancing the network's performance. DACB also introduces dot product attention mechanisms to learn the importance of spatial and temporal features, effectively improving the model's accuracy.

RESULTS

The accuracy of this method in single-shot validation tests on the SEED-IV and DREAMER (Valence-Arousal-Dominance three-classification) datasets is 99.96% and 87.52%, 90.06%, and 89.05%, respectively. In 10-fold cross-validation tests, the accuracy is 99.73% and 84.26%, 85.40%, and 85.02%, outperforming other models.

CONCLUSIONS

This demonstrates that the DACB model achieves high accuracy in emotion classification tasks, demonstrating outstanding performance and generalization ability and providing new directions for future research in EEG signal recognition.

hvEEGNet: a novel deep learning model for high-fidelity EEG reconstruction

Introduction

Modeling multi-channel electroencephalographic (EEG) time-series is a challenging tasks, even for the most recent deep learning approaches. Particularly, in this work, we targeted our efforts to the high-fidelity reconstruction of this type of data, as this is of key relevance for several applications such as classification, anomaly detection, automatic labeling, and brain-computer interfaces.

Methods

We analyzed the most recent works finding that high-fidelity reconstruction is seriously challenged by the complex dynamics of the EEG signals and the large inter-subject variability. So far, previous works provided good results in either high-fidelity reconstruction of single-channel signals, or poor-quality reconstruction of multi-channel datasets. Therefore, in this paper, we present a novel deep learning model, called hvEEGNet, designed as a hierarchical variational autoencoder and trained with a new loss function. We tested it on the benchmark Dataset 2a (including 22-channel EEG data from 9 subjects).

Results

We show that it is able to reconstruct all EEG channels with high-fidelity, fastly (in a few tens of epochs), and with high consistency across different subjects. We also investigated the relationship between reconstruction fidelity and the training duration and, using hvEEGNet as an anomaly detector, we spotted some data in the benchmark dataset that are corrupted and never highlighted before.

Discussion

Thus, hvEEGNet could be very useful in several applications where automatic labeling of large EEG dataset is needed and time-consuming. At the same time, this work opens new fundamental research questions about (1) the effectiveness of deep learning models training (for EEG data) and (2) the need for a systematic characterization of the input EEG data to ensure robust modeling.

Enhancing cross-subject emotion recognition precision through unimodal EEG: a novel emotion preceptor model

Affective computing is a key research area in computer science, neuroscience, and psychology, aimed at enabling computers to recognize, understand, and respond to human emotional states. As the demand for affective computing technology grows, emotion recognition methods based on physiological signals have become research hotspots. Among these, electroencephalogram (EEG) signals, which reflect brain activity, are highly promising. However, due to individual physiological and anatomical differences, EEG signals introduce noise, reducing emotion recognition performance. Additionally, the synchronous collection of multimodal data in practical applications requires high equipment and environmental standards, limiting the practical use of EEG signals. To address these issues, this study proposes the Emotion Preceptor, a cross-subject emotion recognition model based on unimodal EEG signals. This model introduces a Static Spatial Adapter to integrate spatial information in EEG signals, reducing individual differences and extracting robust encoding information. The Temporal Causal Network then leverages temporal information to extract beneficial features for emotion recognition, achieving precise recognition based on unimodal EEG signals. Extensive experiments on the SEED and SEED-V datasets demonstrate the superior performance of the Emotion Preceptor and validate the effectiveness of the new data processing method that combines DE features in a temporal sequence. Additionally, we analyzed the model's data flow and encoding methods from a biological interpretability perspective and validated it with neuroscience research related to emotion generation and regulation, promoting further development in emotion recognition research based on EEG signals.

A temporal-spectral graph convolutional neural network model for EEG emotion recognition within and across subjects

EEG-based emotion recognition uses high-level information from neural activities to predict emotional responses in subjects. However, this information is sparsely distributed in frequency, time, and spatial domains and varied across subjects. To address these challenges in emotion recognition, we propose a novel neural network model named Temporal-Spectral Graph Convolutional Network (TSGCN). To capture high-level information distributed in time, spatial, and frequency domains, TSGCN considers both neural oscillation changes in different time windows and topological structures between different brain regions. Specifically, a Minimum Category Confusion (MCC) loss is used in TSGCN to reduce the inconsistencies between subjective ratings and predefined labels. In addition, to improve the generalization of TSGCN on cross-subject variation, we propose Deep and Shallow feature Dynamic Adversarial Learning (DSDAL) to calculate the distance between the source domain and the target domain. Extensive experiments were conducted on public datasets to demonstrate that TSGCN outperforms state-of-the-art methods in EEG-based emotion recognition. Ablation studies show that the mixed neural networks and our proposed methods in TSGCN significantly contribute to its high performance and robustness. Detailed investigations further provide the effectiveness of TSGCN in addressing the challenges in emotion recognition.

Domain adaptation spatial feature perception neural network for cross-subject EEG emotion recognition

Emotion recognition is a critical research topic within affective computing, with potential applications across various domains. Currently, EEG-based emotion recognition, utilizing deep learning frameworks, has been effectively applied and achieved commendable performance. However, existing deep learning-based models face challenges in capturing both the spatial activity features and spatial topology features of EEG signals simultaneously. To address this challenge, a domain-adaptation spatial-feature perception-network has been proposed for cross-subject EEG emotion recognition tasks, named DSP-EmotionNet. Firstly, a spatial activity topological feature extractor module has been designed to capture spatial activity features and spatial topology features of EEG signals, named SATFEM. Then, using SATFEM as the feature extractor, DSP-EmotionNet has been designed, significantly improving the accuracy of the model in cross-subject EEG emotion recognition tasks. The proposed model surpasses state-of-the-art methods in cross-subject EEG emotion recognition tasks, achieving an average recognition accuracy of 82.5% on the SEED dataset and 65.9% on the SEED-IV dataset.

An Ensemble Deep Learning Approach for EEG-Based Emotion Recognition Using Multi-Class CSP

In recent years, significant advancements have been made in the field of brain-computer interfaces (BCIs), particularly in the area of emotion recognition using EEG signals. The majority of earlier research in this field has missed the spatial-temporal characteristics of EEG signals, which are critical for accurate emotion recognition. In this study, a novel approach is presented for classifying emotions into three categories, positive, negative, and neutral, using a custom-collected dataset. The dataset used in this study was specifically collected for this purpose from 16 participants, comprising EEG recordings corresponding to the three emotional states induced by musical stimuli. A multi-class Common Spatial Pattern (MCCSP) technique was employed for the processing stage of the EEG signals. These processed signals were then fed into an ensemble model comprising three autoencoders with Convolutional Neural Network (CNN) layers. A classification accuracy of 99.44 ± 0.39% for the three emotional classes was achieved by the proposed method. This performance surpasses previous studies, demonstrating the effectiveness of the approach. The high accuracy indicates that the method could be a promising candidate for future BCI applications, providing a reliable means of emotion detection.

STAFNet: an adaptive multi-feature learning network via spatiotemporal fusion for EEG-based emotion recognition

Introduction

Emotion recognition using electroencephalography (EEG) is a key aspect of brain-computer interface research. Achieving precision requires effectively extracting and integrating both spatial and temporal features. However, many studies focus on a single dimension, neglecting the interplay and complementarity of multi-feature information, and the importance of fully integrating spatial and temporal dynamics to enhance performance.

Methods

We propose the Spatiotemporal Adaptive Fusion Network (STAFNet), a novel framework combining adaptive graph convolution and temporal transformers to enhance the accuracy and robustness of EEG-based emotion recognition. The model includes an adaptive graph convolutional module to capture brain connectivity patterns through spatial dynamic evolution and a multi-structured transformer fusion module to integrate latent correlations between spatial and temporal features for emotion classification.

Results

Extensive experiments were conducted on the SEED and SEED-IV datasets to evaluate the performance of STAFNet. The model achieved accuracies of 97.89% and 93.64%, respectively, outperforming state-of-the-art methods. Interpretability analyses, including confusion matrices and t-SNE visualizations, were employed to examine the influence of different emotions on the model's recognition performance. Furthermore, an investigation of varying GCN layer depths demonstrated that STAFNet effectively mitigates the over-smoothing issue in deeper GCN architectures.

Discussion

In summary, the findings validate the effectiveness of STAFNet in EEG-based emotion recognition. The results emphasize the critical role of spatiotemporal feature extraction and introduce an innovative framework for feature fusion, advancing the state of the art in emotion recognition.

Eeg based smart emotion recognition using meta heuristic optimization and hybrid deep learning techniques

In the domain of passive brain-computer interface applications, the identification of emotions is both essential and formidable. Significant research has recently been undertaken on emotion identification with electroencephalogram (EEG) data. The aim of this project is to develop a system that can analyse an individual's EEG and differentiate among positive, neutral, and negative emotional states. The suggested methodology use Independent Component Analysis (ICA) to remove artefacts from Electromyogram (EMG) and Electrooculogram (EOG) in EEG channel recordings. Filtering techniques are employed to improve the quality of EEG data by segmenting it into alpha, beta, gamma, and theta frequency bands. Feature extraction is performed with a hybrid meta-heuristic optimisation technique, such as ABC-GWO. The Hybrid Artificial Bee Colony and Grey Wolf Optimiser are employed to extract optimised features from the selected dataset. Finally, comprehensive evaluations are conducted utilising DEAP and SEED, two publically accessible datasets. The CNN model attains an accuracy of approximately 97% on the SEED dataset and 98% on the DEAP dataset. The hybrid CNN-ABC-GWO model achieves an accuracy of approximately 99% on both datasets, with ABC-GWO employed for hyperparameter tuning and classification. The proposed model demonstrates an accuracy of around 99% on the SEED dataset and 100% on the DEAP dataset. The experimental findings are contrasted utilising a singular technique, a widely employed hybrid learning method, or the cutting-edge method; the proposed method enhances recognition performance.

Crucial rhythms and subnetworks for emotion processing extracted by an interpretable deep learning framework from EEG networks

Electroencephalogram (EEG) brain networks describe the driving and synchronous relationships among multiple brain regions and can be used to identify different emotional states. However, methods for extracting interpretable structural features from brain networks are still lacking. In the current study, a novel deep learning structure comprising both an attention mechanism and a domain adversarial strategy is proposed to extract discriminant and interpretable features from brain networks. Specifically, the attention mechanism enhances the contribution of crucial rhythms and subnetworks for emotion recognition, whereas the domain-adversarial module improves the generalization performance of our proposed model for cross-subject tasks. We validated the effectiveness of the proposed method for subject-independent emotion recognition tasks with the SJTU Emotion EEG Dataset (SEED) and the EEGs recorded in our laboratory. The experimental results showed that the proposed method can effectively improve the classification accuracy of different emotions compared with commonly used methods such as domain adversarial neural networks. On the basis of the extracted network features, we also revealed crucial rhythms and subnetwork structures for emotion processing, which are consistent with those found in previous studies. Our proposed method not only improves the classification performance of brain networks but also provides a novel tool for revealing emotion processing mechanisms.

EEG-based responses of patients with disorders of consciousness and healthy controls to familiar and non-familiar emotional videos

OBJECTIVE

To investigate the differences in the brain responses of healthy controls (HC) and patients with disorders of consciousness (DOC) to familiar and non-familiar audiovisual stimuli and their consistency with the clinical progress.

METHODS

EEG responses of 19 HC and 19 patients with DOC were recorded while watching emotionally-valenced familiar and non-familiar videos. Differential entropy of the EEG recordings was used to train machine learning models aimed to distinguish brain responses to stimuli type. The consistency of brain responses with the clinical progress of the patients was also evaluated.

RESULTS

Models trained using data from HC outperformed those for patients. However, the performance of the models for patients was not influenced by their clinical condition. The models were successfully trained for over 75% of participants, regardless of their clinical condition. More than 75% of patients whose CRS-R scores increased post-study displayed distinguishable brain responses to both stimuli.

CONCLUSIONS

Responses to emotionally-valenced stimuli enabled modelling classifiers that were sensitive to the familiarity of the stimuli, regardless of the clinical condition of the participants and were consistent with their clinical progress in most cases.

SIGNIFICANCE

EEG responses are sensitive to familiarity of emotionally-valenced stimuli in HC and patients with DOC.

Multi-source Selective Graph Domain Adaptation Network for cross-subject EEG emotion recognition

Affective brain-computer interface is an important part of realizing emotional human-computer interaction. However, existing objective individual differences among subjects significantly hinder the application of electroencephalography (EEG) emotion recognition. Existing methods still lack the complete extraction of subject-invariant representations for EEG and the ability to fuse valuable information from multiple subjects to facilitate the emotion recognition of the target subject. To address the above challenges, we propose a Multi-source Selective Graph Domain Adaptation Network (MSGDAN), which can better utilize data from different source subjects and perform more robust emotion recognition on the target subject. The proposed network extracts and selects the individual information specific to each subject, where public information refers to subject-invariant components from multi-source subjects. Moreover, the graph domain adaptation network captures both functional connectivity and regional states of the brain via a dynamic graph network and then integrates graph domain adaptation to ensure the invariance of both functional connectivity and regional states. To evaluate our method, we conduct cross-subject emotion recognition experiments on the SEED, SEED-IV, and DEAP datasets. The results demonstrate that the MSGDAN has superior classification performance.

Set-pMAE: spatial-spEctral-temporal based parallel masked autoEncoder for EEG emotion recognition

The utilization of Electroencephalography (EEG) for emotion recognition has emerged as the primary tool in the field of affective computing. Traditional supervised learning methods are typically constrained by the availability of labeled data, which can result in weak generalizability of learned features. Additionally, EEG signals are highly correlated with human emotional states across temporal, spatial, and spectral dimensions. In this paper, we propose a Spatial-spEctral-Temporal based parallel Masked Autoencoder (SET-pMAE) model for EEG emotion recognition. SET-pMAE learns generic representations of spatial-temporal features and spatial-spectral features through a dual-branch self-supervised task. The reconstruction task of the spatial-temporal branch aims to capture the spatial-temporal contextual dependencies of EEG signals, while the reconstruction task of the spatial-spectral branch focuses on capturing the intrinsic spatial associations of the spectral domain across different brain regions. By learning from both tasks simultaneously, SET-pMAE can capture the generalized representations of features from the both tasks, thereby reducing the risk of overfitting. In order to verify the effectiveness of the proposed model, a series of experiments are conducted on the DEAP and DREAMER datasets. Results from experiments reveal that by employing self-supervised learning, the proposed model effectively captures more discriminative and generalized features, thereby attaining excellent performance.

Rebuttal to "Comments on 'Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features' "

Bharadwaj et al. (2023) present a comments paper evaluating the classification accuracy of several state-of-the-art methods using EEG data averaged over random class samples. According to the results, some of the methods achieve above-chance accuracy, while the method proposed in (Palazzo et al. 2020), that is the target of their analysis, does not. In this rebuttal, we address these claims and explain why they are not grounded in the cognitive neuroscience literature, and why the evaluation procedure is ineffective and unfair.

SFT-SGAT: A semi-supervised fine-tuning self-supervised graph attention network for emotion recognition and consciousness detection

Emotional recognition is highly important in the field of brain-computer interfaces (BCIs). However, due to the individual variability in electroencephalogram (EEG) signals and the challenges in obtaining accurate emotional labels, traditional methods have shown poor performance in cross-subject emotion recognition. In this study, we propose a cross-subject EEG emotion recognition method based on a semi-supervised fine-tuning self-supervised graph attention network (SFT-SGAT). First, we model multi-channel EEG signals by constructing a graph structure that dynamically captures the spatiotemporal topological features of EEG signals. Second, we employ a self-supervised graph attention neural network to facilitate model training, mitigating the impact of signal noise on the model. Finally, a semi-supervised approach is used to fine-tune the model, enhancing its generalization ability in cross-subject classification. By combining supervised and unsupervised learning techniques, the SFT-SGAT maximizes the utility of limited labeled data in EEG emotion recognition tasks, thereby enhancing the model's performance. Experiments based on leave-one-subject-out cross-validation demonstrate that SFT-SGAT achieves state-of-the-art cross-subject emotion recognition performance on the SEED and SEED-IV datasets, with accuracies of 92.04% and 82.76%, respectively. Furthermore, experiments conducted on a self-collected dataset comprising ten healthy subjects and eight patients with disorders of consciousness (DOCs) revealed that the SFT-SGAT attains high classification performance in healthy subjects (maximum accuracy of 95.84%) and was successfully applied to DOC patients, with four patients achieving emotion recognition accuracies exceeding 60%. The experiments demonstrate the effectiveness of the proposed SFT-SGAT model in cross-subject EEG emotion recognition and its potential for assessing levels of consciousness in patients with DOC.

Emotion Recognition Based on a EEG-fNIRS Hybrid Brain Network in the Source Space

Background/Objectives: Studies have shown that emotion recognition based on electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) multimodal physiological signals exhibits superior performance compared to that of unimodal approaches. Nonetheless, there remains a paucity of in-depth investigations analyzing the inherent relationship between EEG and fNIRS and constructing brain networks to improve the performance of emotion recognition. Methods: In this study, we introduce an innovative method to construct hybrid brain networks in the source space based on simultaneous EEG-fNIRS signals for emotion recognition. Specifically, we perform source localization on EEG signals to derive the EEG source signals. Subsequently, causal brain networks are established in the source space by analyzing the Granger causality between the EEG source signals, while coupled brain networks in the source space are formed by assessing the coupling strength between the EEG source signals and the fNIRS signals. The resultant causal brain networks and coupled brain networks are integrated to create hybrid brain networks in the source space, which serve as features for emotion recognition. Results: The effectiveness of our proposed method is validated on multiple emotion datasets. The experimental results indicate that the recognition performance of our approach significantly surpasses that of the baseline method. Conclusions: This work offers a novel perspective on the fusion of EEG and fNIRS signals in an emotion-evoked experimental paradigm and provides a feasible solution for enhancing emotion recognition performance.

A Method for Detecting Depression in Adolescence Based on an Affective Brain-Computer Interface and Resting-State Electroencephalogram Signals

Depression is increasingly prevalent among adolescents and can profoundly impact their lives. However, the early detection of depression is often hindered by the time-consuming diagnostic process and the absence of objective biomarkers. In this study, we propose a novel approach for depression detection based on an affective brain-computer interface (aBCI) and the resting-state electroencephalogram (EEG). By fusing EEG features associated with both emotional and resting states, our method captures comprehensive depression-related information. The final depression detection model, derived through decision fusion with multiple independent models, further enhances detection efficacy. Our experiments involved 40 adolescents with depression and 40 matched controls. The proposed model achieved an accuracy of 86.54% on cross-validation and 88.20% on the independent test set, demonstrating the efficiency of multimodal fusion. In addition, further analysis revealed distinct brain activity patterns between the two groups across different modalities. These findings hold promise for new directions in depression detection and intervention.

SATEER: Subject-Aware Transformer for EEG-Based Emotion Recognition

This study presents a Subject-Aware Transformer-based neural network designed for the Electroencephalogram (EEG) Emotion Recognition task (SATEER), which entails the analysis of EEG signals to classify and interpret human emotional states. SATEER processes the EEG waveforms by transforming them into Mel spectrograms, which can be seen as particular cases of images with the number of channels equal to the number of electrodes used during the recording process; this type of data can thus be processed using a Computer Vision pipeline. Distinct from preceding approaches, this model addresses the variability in individual responses to identical stimuli by incorporating a User Embedder module. This module enables the association of individual profiles with their EEGs, thereby enhancing classification accuracy. The efficacy of the model was rigorously evaluated using four publicly available datasets, demonstrating superior performance over existing methods in all conducted benchmarks. For instance, on the AMIGOS dataset (A dataset for Multimodal research of affect, personality traits, and mood on Individuals and GrOupS), SATEER's accuracy exceeds 99.8% accuracy across all labels and showcases an improvement of 0.47% over the state of the art. Furthermore, an exhaustive ablation study underscores the pivotal role of the User Embedder module and each other component of the presented model in achieving these advancements.

A Novel and Powerful Dual-Stream Multi-Level Graph Convolution Network for Emotion Recognition

Emotion recognition enables machines to more acutely perceive and understand users' emotional states, thereby offering more personalized and natural interactive experiences. Given the regularity of the responses of brain activity to human cognitive processes, we propose a powerful and novel dual-stream multi-level graph convolution network (DMGCN) with the ability to capture the hierarchies of connectivity between cerebral cortex neurons and improve computational efficiency. This consists of a hierarchical dynamic geometric interaction neural network (HDGIL) and multi-level feature fusion classifier (M2FC). First, the HDGIL diversifies representations by learning emotion-related representations in multi-level graphs. Subsequently, M2FC integrates advantages from methods for early and late feature fusion and enables the addition of more details to final representations from EEG samples. We conducted extensive experiments to validate the superiority of our model over numerous state-of-the-art (SOTA) baselines in terms of classification accuracy, the efficiency of graph embedding and information propagation, achieving accuracies of 98.73%, 95.97%, 72.74% and 94.89% for our model as well as increases of up to 0.59%, 0.32%, 2.24% and 3.17% over baselines on the DEAP-Arousal, DEAP-Valence, DEAP and SEED datasets, respectively. Additionally, these experiments demonstrated the effectiveness of each module for emotion recognition tasks.

MP: A steady-state visual evoked potential dataset based on multiple paradigms

In the field of steady-state visual evoked potential (SSVEP), stimulus paradigms are regularly arranged or mimic the style of a keyboard with the same size. However, stimulation paradigms have important effects on the performance of SSVEP systems, which correlate with the electroencephalogram (EEG) signal amplitude and recognition accuracy. This paper provides MP dataset that was acquired using a 12-target BCI speller. MP dataset contains 9-channel EEG signals from the occipital region of 24 subjects under 5 stimulation paradigms with different stimulus sizes and arrangements. Stimuli were encoded using joint frequency and phase modulation (JFPM) method. Subjects completed an offline prompted spelling task using a speller under 5 paradigms. Each experiment contains 8 blocks, and each block contains 12 trials. Designers can use this dataset to test the performance of algorithms considering "stimulus size" and "stimulus arrangement". EEG data showed SSVEP features through amplitude-frequency analysis. FBCCA and TRCA confirmed its suitability.

Frontal EEG correlation based human emotion identification and classification

Humans express their feelings and intentions of their actions or communication through emotions. Recent advancements in technology involve machines in human communication in day-to-day life. Thus, understanding of human emotions by machines will be very helpful in assisting the user in a far better way. Various physiological and non-physiological signals can be used to make the machines to recognize the emotion of a person. The identification of emotional content in the signals is crucial to understand emotion and the machines act with emotional intelligence at appropriate times, thus providing a better human machine interaction with emotion identification system and mental health monitoring for psychiatric patients. This work includes the creation of an emotion EEG dataset, the development of an algorithm for identifying the emotion elicitation segments in the EEG signal, and the classification of emotions from EEG signals. The EEG signals are divided into 3s segments, and the segments with emotional content are selected based on the decrease in correlation between the frontal electrodes. The selected segments are validated with the facial expressions of the subjects in the appropriate time segments of the face video. EEGNet is used to classify the emotion from the EEG signal. The classification accuracy with the selected emotional EEG segments is higher compared to the accuracy using all the EEG segments. In subject-specific classification, an average accuracy of 80.87% is obtained from the network trained with selected EEG segments, and 70.5% is obtained from training with all EEG segments. In subject-independent classification, the accuracy of classification is 67% and 63.8% with and without segment selection, respectively. The proposed method of selection of EEG segments is validated using the DEAP dataset, and classification accuracies and F1-scores of subject dependent and subject-independent methods are presented.

Discriminative possibilistic clustering promoting cross-domain emotion recognition

The affective Brain-Computer Interface (aBCI) systems strive to enhance prediction accuracy for individual subjects by leveraging data from multiple subjects. However, significant differences in EEG (Electroencephalogram) feature patterns among subjects often hinder these systems from achieving the desired outcomes. Although studies have attempted to address this challenge using subject-specific classifier strategies, the scarcity of labeled data remains a major hurdle. In light of this, Domain Adaptation (DA) technology has gradually emerged as a prominent approach in the field of EEG-based emotion recognition, attracting widespread research interest. The crux of DA learning lies in resolving the issue of distribution mismatch between training and testing datasets, which has become a focal point of academic attention. Currently, mainstream DA methods primarily focus on mitigating domain distribution discrepancies by minimizing the Maximum Mean Discrepancy (MMD) or its variants. Nevertheless, the presence of noisy samples in datasets can lead to pronounced shifts in domain means, thereby impairing the adaptive performance of DA methods based on MMD and its variants in practical applications to some extent. Research has revealed that the traditional MMD metric can be transformed into a 1-center clustering problem, and the possibility clustering model is adept at mitigating noise interference during the data clustering process. Consequently, the conventional MMD metric can be further relaxed into a possibilistic clustering model. Therefore, we construct a distributed distance measure with Discriminative Possibilistic Clustering criterion (DPC), which aims to achieve two objectives: (1) ensuring the discriminative effectiveness of domain distribution alignment by finding a shared subspace that minimizes the overall distribution distance between domains while maximizing the semantic distribution distance according to the principle of "sames attract and opposites repel"; and (2) enhancing the robustness of distribution distance measure by introducing a fuzzy entropy regularization term. Theoretical analysis confirms that the proposed DPC is an upper bound of the existing MMD metric under certain conditions. Therefore, the MMD objective can be effectively optimized by minimizing the DPC. Finally, we propose a domain adaptation in Emotion recognition based on DPC (EDPC) that introduces a graph Laplacian matrix to preserve the geometric structural consistency between data within the source and target domains, thereby enhancing label propagation performance. Simultaneously, by maximizing the use of source domain discriminative information to minimize domain discrimination errors, the generalization performance of the DA model is further improved. Comparative experiments on several representative domain adaptation learning methods using multiple EEG datasets (i.e., SEED and SEED-IV) show that, in most cases, the proposed method exhibits better or comparable consistent generalization performance.

DGSD: Dynamical graph self-distillation for EEG-based auditory spatial attention detection

Auditory Attention Detection (AAD) aims to detect the target speaker from brain signals in a multi-speaker environment. Although EEG-based AAD methods have shown promising results in recent years, current approaches primarily rely on traditional convolutional neural networks designed for processing Euclidean data like images. This makes it challenging to handle EEG signals, which possess non-Euclidean characteristics. In order to address this problem, this paper proposes a dynamical graph self-distillation (DGSD) approach for AAD, which does not require speech stimuli as input. Specifically, to effectively represent the non-Euclidean properties of EEG signals, dynamical graph convolutional networks are applied to represent the graph structure of EEG signals, which can also extract crucial features related to auditory spatial attention in EEG signals. In addition, to further improve AAD detection performance, self-distillation, consisting of feature distillation and hierarchical distillation strategies at each layer, is integrated. These strategies leverage features and classification results from the deepest network layers to guide the learning of shallow layers. Our experiments are conducted on two publicly available datasets, KUL and DTU. Under a 1-second time window, we achieve results of 90.0% and 79.6% accuracy on KUL and DTU, respectively. We compare our DGSD method with competitive baselines, and the experimental results indicate that the detection performance of our proposed DGSD method is not only superior to the best reproducible baseline but also significantly reduces the number of trainable parameters by approximately 100 times.

Emotion recognition using hierarchical spatial-temporal learning transformer from regional to global brain

Emotion recognition is an essential but challenging task in human-computer interaction systems due to the distinctive spatial structures and dynamic temporal dependencies associated with each emotion. However, current approaches fail to accurately capture the intricate effects of electroencephalogram (EEG) signals across different brain regions on emotion recognition. Therefore, this paper designs a transformer-based method, denoted by R2G-STLT, which relies on a spatial-temporal transformer encoder with regional to global hierarchical learning that learns the representative spatiotemporal features from the electrode level to the brain-region level. The regional spatial-temporal transformer (RST-Trans) encoder is designed to obtain spatial information and context dependence at the electrode level aiming to learn the regional spatiotemporal features. Then, the global spatial-temporal transformer (GST-Trans) encoder is utilized to extract reliable global spatiotemporal features, reflecting the impact of various brain regions on emotion recognition tasks. Moreover, the multi-head attention mechanism is placed into the GST-Trans encoder to empower it to capture the long-range spatial-temporal information among the brain regions. Finally, subject-independent experiments are conducted on each frequency band of the DEAP, SEED, and SEED-IV datasets to assess the performance of the proposed model. Results indicate that the R2G-STLT model surpasses several state-of-the-art approaches.

Cross-subject emotion recognition in brain-computer interface based on frequency band attention graph convolutional adversarial neural networks

BACKGROUND

Emotion is an important area in neuroscience. Cross-subject emotion recognition based on electroencephalogram (EEG) data is challenging due to physiological differences between subjects. Domain gap, which refers to the different distributions of EEG data at different subjects, has attracted great attention for cross-subject emotion recognition.

COMPARISON WITH EXISTING METHODS

This study focuses on narrowing the domain gap between subjects through the emotional frequency bands and the relationship information between EEG channels. Emotional frequency band features represent the energy distribution of EEG data in different frequency ranges, while relationship information between EEG channels provides spatial distribution information about EEG data.

NEW METHOD

To achieve this, this paper proposes a model called the Frequency Band Attention Graph convolutional Adversarial neural Network (FBAGAN). This model includes three components: a feature extractor, a classifier, and a discriminator. The feature extractor consists of a layer with a frequency band attention mechanism and a graph convolutional neural network. The mechanism effectively extracts frequency band information by assigning weights and Graph Convolutional Networks can extract relationship information between EEG channels by modeling the graph structure. The discriminator then helps minimize the gap in the frequency information and relationship information between the source and target domains, improving the model's ability to generalize.

RESULTS

The FBAGAN model is extensively tested on the SEED, SEED-IV, and DEAP datasets. The accuracy and standard deviation scores are 88.17% and 4.88, respectively, on the SEED dataset, and 77.35% and 3.72 on the SEED-IV dataset. On the DEAP dataset, the model achieves 69.64% for Arousal and 65.18% for Valence. These results outperform most existing models.

CONCLUSIONS

The experiments indicate that FBAGAN effectively addresses the challenges of transferring EEG channel domain and frequency band domain, leading to improved performance.

Multi-source domain adaptation for EEG emotion recognition based on inter-domain sample hybridization

Background

Electroencephalogram (EEG) is widely used in emotion recognition due to its precision and reliability. However, the nonstationarity of EEG signals causes significant differences between individuals or sessions, making it challenging to construct a robust model. Recently, domain adaptation (DA) methods have shown excellent results in cross-subject EEG emotion recognition by aligning marginal distributions. Nevertheless, these methods do not consider emotion category labels, which can lead to label confusion during alignment. Our study aims to alleviate this problem by promoting conditional distribution alignment during domain adaptation to improve cross-subject and cross-session emotion recognition performance.

Method

This study introduces a multi-source domain adaptation common-branch network for EEG emotion recognition and proposes a novel sample hybridization method. This method enables the introduction of target domain data information by directionally hybridizing source and target domain samples without increasing the overall sample size, thereby enhancing the effectiveness of conditional distribution alignment in domain adaptation. Cross-subject and cross-session experiments were conducted on two publicly available datasets, SEED and SEED-IV, to validate the proposed model.

Result

In cross-subject emotion recognition, our method achieved an average accuracy of 90.27% on the SEED dataset, with eight out of 15 subjects attaining a recognition accuracy higher than 90%. For the SEED-IV dataset, the recognition accuracy also reached 73.21%. Additionally, in the cross-session experiment, we sequentially used two out of the three session data as source domains and the remaining session as the target domain for emotion recognition. The proposed model yielded average accuracies of 94.16 and 75.05% on the two datasets, respectively.

Conclusion

Our proposed method aims to alleviate the difficulties of emotion recognition from the limited generalization ability of EEG features across subjects and sessions. Though adapting the multi-source domain adaptation and the sample hybridization method, the proposed method can effectively transfer the emotion-related knowledge of known subjects and achieve accurate emotion recognition on unlabeled subjects.

MASER: Enhancing EEG Spatial Resolution With State Space Modeling

Consumer-grade Electroencephalography (EEG) devices equipped with few electrodes often suffer from low spatial resolution, hindering the accurate capture of intricate brain activity patterns. To address this issue, we propose MASER, a novel super-resolution approach for EEG recording. In MASER, we design the eMamba block for extracting EEG features based on the principles of state space models (SSMs). We further stack eMamba blocks to form a low-resolution feature extractor and a high-resolution signal predictor, which enhances the feature representation. During the training of MASER, we fully consider the characteristics of multidimensional biological series signals, incorporating a smoothness constraint loss to achieve more consistent high-resolution reconstructions. MASER pioneers EEG-oriented state space modeling, effectively capturing the temporal dynamics and latent states, thereby revealing complex neural interactions over time. Extensive experiments show that the proposed MASER outperforms the state-of-the-art methods in super-resolution quality on two public EEG datasets, with normalized mean square error reduced by 16.25% and Pearson correlation improved by 1.13%. Moreover, a case study of motor imagery recognition highlights the advantages conferred by high-resolution EEG signals. With a 4x increase in spatial resolution by MASER, the recognition accuracy improves by 5.74%, implying a significant performance elevation in brain-computer interface (BCI) command mapping. By enhancing the spatial resolution of EEG signals, MASER makes EEG-based applications more accessible, reducing cost and setup time while maintaining high performance across various domains such as gaming, education, and healthcare.

Virtual reality-enabled high-performance emotion estimation with the most significant channel pairs

Human-computer interface (HCI) and electroencephalogram (EEG) signals are widely used in user experience (UX) interface designs to provide immersive interactions with the user. In the context of UX, EEG signals can be used within a metaverse system to assess user engagement, attention, emotional responses, or mental workload. By analyzing EEG signals, system designers can tailor the virtual environment, content, or interactions in real time to optimize UX, improve immersion, and personalize interactions. However, in this case, in addition to the signals' processing cost and classification accuracy, cybersickness in Virtual Reality (VR) systems needs to be resolved. At this point, channel selection methods can perform better for HCI and UX applications by reducing noisy and redundant information in generally unrelated EEG channels. For this purpose, a new method for EEG channel selection based on phase-locking value (PLV) analysis is proposed. We hypothesized that there are interactions between EEG channels in terms of PLV in repeated tasks in different trials of the emotion estimation experiment. Subsequently, frequency-based features were extracted. The features were classified by dividing them into bags using the Multiple-Instance Learning (MIL) variant. This study provides higher classification performance using fewer EEG channels for emotion prediction. The performance rate obtained in binary classification with the Random Forests (RF) algorithm is at a promising level of 99%. The proposed method achieved an accuracy of 99.38% for valence using all channels on the new dataset (VREMO) and 98.13% with channel selection. The benchmark dataset (DEAP) achieved accuracies of 98.16% using all channels and 98.13% with selected channels.

Multi-source domain transfer network based on subdomain adaptation and minimum class confusion for EEG emotion recognition

Electroencephalogram (EEG) signals, which objectively reflect the state of the brain, are widely favored in emotion recognition research. However, the presence of cross-session and cross-subject variation in EEG signals has hindered the practical implementation of EEG-based emotion recognition technologies. In this article, we propose a multi-source domain transfer method based on subdomain adaptation and minimum class confusion (MS-SAMCC) in response to the addressed issue. First, we introduce the mix-up data augmentation technique to generate augmented samples. Next, we propose a minimum class confusion subdomain adaptation method (MCCSA) as a sub-module of the multi-source domain adaptation module. This approach enables global alignment between each source domain and the target domain, while also achieving alignment among individual subdomains within them. Additionally, we employ minimum class confusion (MCC) as a regularizer for this sub-module. We performed experiments on SEED, SEED IV, and FACED datasets. In the cross-subject experiments, our method achieved mean classification accuracies of 87.14% on SEED, 63.24% on SEED IV, and 42.07% on FACED. In the cross-session experiments, our approach obtained average classification accuracies of 94.20% on SEED and 71.66% on SEED IV. These results demonstrate that the MS-SAMCC approach proposed in this study can effectively address EEG-based emotion recognition tasks.

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition

Introduction

EEG-based emotion recognition has gradually become a new research direction, known as affective Brain-Computer Interface (aBCI), which has huge application potential in human-computer interaction and neuroscience. However, how to extract spatio-temporal fusion features from complex EEG signals and build learning method with high recognition accuracy and strong interpretability is still challenging.

Methods

In this paper, we propose a hybrid attention spatio-temporal feature fusion network for EEG-based emotion recognition. First, we designed a spatial attention feature extractor capable of merging shallow and deep features to extract spatial information and adaptively select crucial features under different emotional states. Then, the temporal feature extractor based on the multi-head attention mechanism is integrated to perform spatio-temporal feature fusion to achieve emotion recognition. Finally, we visualize the extracted spatial attention features using feature maps, further analyzing key channels corresponding to different emotions and subjects.

Results

Our method outperforms the current state-of-the-art methods on two public datasets, SEED and DEAP. The recognition accuracy are 99.12% ± 1.25% (SEED), 98.93% ± 1.45% (DEAP-arousal), and 98.57% ± 2.60% (DEAP-valence). We also conduct ablation experiments, using statistical methods to analyze the impact of each module on the final result. The spatial attention features reveal that emotion-related neural patterns indeed exist, which is consistent with conclusions in the field of neurology.

Discussion

The experimental results show that our method can effectively extract and fuse spatial and temporal information. It has excellent recognition performance, and also possesses strong robustness, performing stably across different datasets and experimental environments for emotion recognition.

Emotion Recognition Using EEG Signals and Audiovisual Features with Contrastive Learning

Multimodal emotion recognition has emerged as a promising approach to capture the complex nature of human emotions by integrating information from various sources such as physiological signals, visual behavioral cues, and audio-visual content. However, current methods often struggle with effectively processing redundant or conflicting information across modalities and may overlook implicit inter-modal correlations. To address these challenges, this paper presents a novel multimodal emotion recognition framework which integrates audio-visual features with viewers' EEG data to enhance emotion classification accuracy. The proposed approach employs modality-specific encoders to extract spatiotemporal features, which are then aligned through contrastive learning to capture inter-modal relationships. Additionally, cross-modal attention mechanisms are incorporated for effective feature fusion across modalities. The framework, comprising pre-training, fine-tuning, and testing phases, is evaluated on multiple datasets of emotional responses. The experimental results demonstrate that the proposed multimodal approach, which combines audio-visual features with EEG data, is highly effective in recognizing emotions, highlighting its potential for advancing emotion recognition systems.