Enhanced performance of EEG-based brain-computer interfaces by joint sample and feature importance assessment

Electroencephalograph (EEG) has been a reliable data source for building brain-computer interface (BCI) systems; however, it is not reasonable to use the feature vector extracted from multiple EEG channels and frequency bands to perform recognition directly due to the two deficiencies. One is that EEG data is weak and non-stationary, which easily causes different EEG samples to have different quality. The other is that different feature dimensions corresponding to different brain regions and frequency bands have different correlations to a certain mental task, which is not sufficiently investigated. To this end, a Joint Sample and Feature importance Assessment (JSFA) model was proposed to simultaneously explore the different impacts of EEG samples and features in mental state recognition, in which the former is based on the self-paced learning technique while the latter is completed by the feature self-weighting technique. The efficacy of JSFA is extensively evaluated on two EEG data sets, i.e., SEED-IV and SEED-VIG. One is a classification task for emotion recognition and the other is a regression task for driving fatigue detection. Experimental results demonstrate that JSFA can effectively identify the importance of different EEG samples and features, leading to enhanced recognition performance of corresponding BCI systems.

Lightweight attention mechanisms for EEG emotion recognition for brain computer interface

BACKGROUND

In the realm of brain-computer interfaces (BCI), identifying emotions from electroencephalogram (EEG) data is a difficult endeavor because of the volume of data, the intricacy of the signals, and the several channels that make up the signals.

NEW METHODS

Using dual-stream structure scaling and multiple attention mechanisms (LDMGEEG), a lightweight network is provided to maximize the accuracy and performance of EEG-based emotion identification. Reducing the number of computational parameters while maintaining the current level of classification accuracy is the aim. This network employs a symmetric dual-stream architecture to assess separately time-domain and frequency-domain spatio-temporal maps constructed using differential entropy features of EEG signals as inputs.

RESULT

The experimental results show that after significantly lowering the number of parameters, the model achieved the best possible performance in the field, with a 95.18 % accuracy on the SEED dataset.

COMPARISON WITH EXISTING METHODS

Moreover, it reduced the number of parameters by 98 % when compared to existing models.

CONCLUSION

The proposed method distinct channel-time/frequency-space multiple attention and post-attention methods enhance the model's ability to aggregate features and result in lightweight performance.

Semi-supervised bipartite graph construction with active EEG sample selection for emotion recognition

Electroencephalogram (EEG) signals are derived from the central nervous system and inherently difficult to camouflage, leading to the recent popularity of EEG-based emotion recognition. However, due to the non-stationary nature of EEG, inter-subject variabilities become obstacles for recognition models to well adapt to different subjects. In this paper, we propose a novel approach called semi-supervised bipartite graph construction with active EEG sample selection (SBGASS) for cross-subject emotion recognition, which offers two significant advantages. Firstly, SBGASS adaptively learns a bipartite graph to characterize the underlying relationships between labeled and unlabeled EEG samples, effectively implementing the semantic connection for samples from different subjects. Secondly, we employ active sample selection technique in this paper to reduce the impact of negative samples (outliers or noise in the data) on bipartite graph construction. Drawing from the experimental results with the SEED-IV data set, we have gained the following three insights. (1) SBGASS actively rejects negative labeled samples, which helps mitigate the impact of negative samples when constructing the optimal bipartite graph and improves the model performance. (2) Through the learned optimal bipartite graph in SBGASS, the transferability of labeled EEG samples is quantitatively analyzed, which exhibits a decreasing tendency as the distance between each labeled sample and the corresponding class centroid increases. (3) Besides the improved recognition accuracy, the spatial-frequency patterns in emotion recognition are investigated by the acquired projection matrix.

Automatically Extracting and Utilizing EEG Channel Importance Based on Graph Convolutional Network for Emotion Recognition

Graph convolutional network (GCN) based on the brain network has been widely used for EEG emotion recognition. However, most studies train their models directly without considering network dimensionality reduction beforehand. In fact, some nodes and edges are invalid information or even interference information for the current task. It is necessary to reduce the network dimension and extract the core network. To address the problem of extracting and utilizing the core network, a core network extraction model (CWGCN) based on channel weighting and graph convolutional network and a graph convolutional network model (CCSR-GCN) based on channel convolution and style-based recalibration for emotion recognition have been proposed. The CWGCN model automatically extracts the core network and the channel importance parameter in a data-driven manner. The CCSR-GCN model innovatively uses the output information of the CWGCN model to identify the emotion state. The experimental results on SEED show that: 1) the core network extraction can help improve the performance of the GCN model; 2) the models of CWGCN and CCSR-GCN achieve better results than the currently popular methods. The idea and its implementation in this paper provide a novel and successful perspective for the application of GCN in brain network analysis of other specific tasks.

TSFAN: tensorized spatial-frequency attention network with domain adaptation for cross-session EEG-based biometric recognition

Objective.Electroencephalogram (EEG) signals are promising biometrics owning to their invisibility, adapting to the application scenarios with high-security requirements. However, It is challenging to explore EEG identity features without the interference of device and state differences of the subject across sessions. Existing methods treat training sessions as a single domain, affected by the different data distribution among sessions. Although most multi-source unsupervised domain adaptation (MUDA) methods bridge the domain gap between multiple source and target domains individually, relationships among the domain-invariant features of each distribution alignment are neglected.Approach.In this paper, we propose a MUDA method, Tensorized Spatial-Frequency Attention Network (TSFAN), to assist the performance of the target domain for EEG-based biometric recognition. Specifically, significant relationships of domain-invariant features are modeled via a tensorized attention mechanism. It jointly incorporates appropriate common spatial-frequency representations of pairwise source and target but also cross-source domains, without the effect of distribution discrepancy among source domains. Additionally, considering the curse of dimensionality, our TSFAN is approximately represented in Tucker format. Benefiting the low-rank Tucker Network, the TSFAN can scale linearly in the number of domains, providing us the great flexibility to extend TSFAN to the case associated with an arbitrary number of sessions.Main results.Extensive experiments on the representative benchmarks demonstrate the effectiveness of TSFAN in EEG-based biometric recognition, outperforming state-of-the-art approaches, as verified by cross-session validation.Significance.The proposed TSFAN aims to investigate the presence of consistent EEG identity features across sessions. It is achieved by utilizing a novel tensorized attention mechanism that collaborates intra-source transferable information with inter-source interactions, while remaining unaffected by domain shifts in multiple source domains. Furthermore, the electrode selection shows that EEG-based identity features across sessions are distributed across brain regions, and 20 electrodes based on 10-20 standard system are able to extract stable identity information.

Joint domain symmetry and predictive balance for cross-dataset EEG emotion recognition

BACKGROUND

Cross-dataset EEG emotion recognition is an extremely challenging task, since data distributions of EEG from different datasets are greatly different, which makes the universal models yield unsatisfactory results. Although there are many methods have been proposed to reduce cross-dataset distribution discrepancies, they still neglected the following two problems. (1) Label space inconsistency: emotional label spaces of subjects from different datasets are different; (2) Uncertainty propagation: the uncertainty of misclassified emotion samples will propagate between datasets.

NEW METHOD

To solve these problems, we propose a novel method called domain symmetry and predictive balance (DSPB). For the problem of label space inconsistency, a domain symmetry module is designed to make label spaces of source and target domain to be the same, which randomly selects samples from the source domain and put into the target domain. For the problem of uncertainty propagation, a predictive balance module is proposed to reduce the prediction score of incorrect samples and then effectively reduce distribution differences between EEG from different datasets.

RESULTS

Experimental results show that our method achieve 61.48% average accuracies on the three cross-dataset tasks. Moreover, we find that gamma is the most relevant to emotion recognition among the five frequency bands, and the prefrontal and temporal brain regions are the channels carrying the most emotional information among the 62 brain channels.

COMPARISON WITH EXISTING METHODS

Compared with the partial domain adaptation method (SPDA) and the unsupervised domain adaptation (MS-MDA), our method improves average accuracies by 15.60% and 23.11%, respectively.

CONCLUSION

Besides, data distributions of EEG from different datasets but with the same emotional labels have been well aligned, which demonstrates the effectiveness of DSPB.

Decoding Multi-Brain Motor Imagery From EEG Using Coupling Feature Extraction and Few-Shot Learning

Electroencephalography (EEG)-based motor imagery (MI) is one of brain computer interface (BCI) paradigms, which aims to build a direct communication pathway between human brain and external devices by decoding the brain activities. In a traditional way, MI BCI replies on a single brain, which suffers from the limitations, such as low accuracy and weak stability. To alleviate these limitations, multi-brain BCI has emerged based on the integration of multiple individuals' intelligence. Nevertheless, the existing decoding methods mainly use linear averaging or feature integration learning from multi-brain EEG data, and do not effectively utilize coupling relationship features, resulting in undesired decoding accuracy. To overcome these challenges, we proposed an EEG-based multi-brain MI decoding method, which utilizes coupling feature extraction and few-shot learning to capture coupling relationship features among multi-brains with only limited EEG data. We performed an experiment to collect EEG data from multiple persons who engaged in the same task simultaneously and compared the methods on the collected data. The comparison results showed that our proposed method improved the performance by 14.23% compared to the single-brain mode in the 10-shot three-class decoding task. It demonstrated the effectiveness of the proposed method and usability of the method in the context of only small amount of EEG data available.

Cross-Sensory EEG Emotion Recognition with Filter Bank Riemannian Feature and Adversarial Domain Adaptation

Emotion recognition is crucial in understanding human affective states with various applications. Electroencephalography (EEG)-a non-invasive neuroimaging technique that captures brain activity-has gained attention in emotion recognition. However, existing EEG-based emotion recognition systems are limited to specific sensory modalities, hindering their applicability. Our study innovates EEG emotion recognition, offering a comprehensive framework for overcoming sensory-focused limits and cross-sensory challenges. We collected cross-sensory emotion EEG data using multimodal emotion simulations (three sensory modalities: audio/visual/audio-visual with two emotion states: pleasure or unpleasure). The proposed framework-filter bank adversarial domain adaptation Riemann method (FBADR)-leverages filter bank techniques and Riemannian tangent space methods for feature extraction from cross-sensory EEG data. Compared with Riemannian methods, filter bank and adversarial domain adaptation could improve average accuracy by 13.68% and 8.36%, respectively. Comparative analysis of classification results proved that the proposed FBADR framework achieved a state-of-the-art cross-sensory emotion recognition performance and reached an average accuracy of 89.01% ± 5.06%. Moreover, the robustness of the proposed methods could ensure high cross-sensory recognition performance under a signal-to-noise ratio (SNR) ≥ 1 dB. Overall, our study contributes to the EEG-based emotion recognition field by providing a comprehensive framework that overcomes limitations of sensory-oriented approaches and successfully tackles the difficulties of cross-sensory situations.

Detecting slow eye movements using multi-scale one-dimensional convolutional neural network for driver sleepiness detection

BACKGROUND

Slow eye movements (SEMs), which occurs during eye-closed periods with high time coverage rate during simulated driving process, indicate drivers' sleep onset.

NEW METHOD

For the multi-scale characteristics of slow eye movement waveforms, we propose a multi-scale one-dimensional convolutional neural network (MS-1D-CNN) for classification. The MS-1D-CNN performs multiple down-sampling processing branches on the original signal and uses the local convolutional layer to extract the features for each branch.

RESULTS

We evaluate the classification performance of this model on ten subjects' standard train-test datasets and continuous test datasets by means of subject-subject evaluation and leave-one-subject-out cross validation, respectively. For the standard train-test datasets, the overall average classification accuracies are about 99.1% and 98.6%, in subject-subject evaluation and leave-one-subject-out cross validation, respectively. For the continuous test datasets, the overall average values of accuracy, precision, recall and F1-score are 99.3%, 98.9%, 99.5% and 99.1% in subject-subject evaluation, are 99.2%, 98.8%, 99.3% and 99.0% in leave-one-subject-out cross validation.

COMPARISON WITH EXISTING METHOD

Results of the standard train-test datasets show that the overall average classification accuracy of the MS-1D-CNN is quite higher than the baseline method based on hand-designed features by 3.5% and 3.5%, in subject-subject evaluation and leave-one-subject-out cross validation, respectively.

CONCLUSIONS

These results suggest that multi-scale transformation in the MS-1D-CNN model can enhance the representation ability of features, thereby improving classification accuracy. Experimental results verify the good performance of the MS-1D-CNN model, even in leave-one-subject-out cross validation, thus promoting the application of SEMs detection technology for driver sleepiness detection.

SIFIAE: An adaptive emotion recognition model with EEG feature-label inconsistency consideration

BACKGROUND

A common but easily overlooked affective overlap problem has not been received enough attention in electroencephalogram (EEG)-based emotion recognition research. In real life, affective overlap refers to the current emotional state of human being is sometimes influenced easily by his/her historical mood. In stimulus-evoked EEG collection experiment, due to the short rest interval in consecutive trials, the inner mechanisms of neural responses make subjects cannot switch their emotion state easily and quickly, which might lead to the affective overlap. For example, we might be still in sad state to some extent even if we are watching a comedy because we just saw a tragedy before. In pattern recognition, affective overlap usually means that there exists the feature-label inconsistency in EEG data.

NEW METHODS

To alleviate the impact of inconsistent EEG data, we introduce a variable to adaptively explore the sample inconsistency in emotion recognition model development. Then, we propose a semi-supervised emotion recognition model for joint sample inconsistency and feature importance exploration (SIFIAE). Accordingly, an efficient optimization method to SIFIAE model is proposed.

RESULTS

Extensive experiments on the SEED-V dataset demonstrate the effectiveness of SIFIAE. Specifically, SIFIAE achieves 69.10%, 67.01%, 71.5%, 73.26%, 72.07% and 71.35% average accuracies in six cross-session emotion recognition tasks.

CONCLUSION

The results illustrated that the sample weights have a rising trend in the beginning of most trials, which coincides with the affective overlap hypothesis. The feature importance factor indicated the critical bands and channels are more obvious compared with some models without considering EEG feature-label inconsistency.

Semi-supervised regression with adaptive graph learning for EEG-based emotion recognition

Electroencephalogram (EEG) signals are widely used in the field of emotion recognition since it is resistant to camouflage and contains abundant physiological information. However, EEG signals are non-stationary and have low signal-noise-ratio, making it more difficult to decode in comparison with data modalities such as facial expression and text. In this paper, we propose a model termed semi-supervised regression with adaptive graph learning (SRAGL) for cross-session EEG emotion recognition, which has two merits. On one hand, the emotional label information of unlabeled samples is jointly estimated with the other model variables by a semi-supervised regression in SRAGL. On the other hand, SRAGL adaptively learns a graph to depict the connections among EEG data samples which further facilitates the emotional label estimation process. From the experimental results on the SEED-IV data set, we have the following insights. 1) SRAGL achieves superior performance compared to some state-of-the-art algorithms. To be specific, the average accuracies are 78.18%, 80.55%, and 81.90% in the three cross-session emotion recognition tasks. 2) As the iteration number increases, SRAGL converges quickly and optimizes the emotion metric of EEG samples gradually, leading to a reliable similarity matrix finally. 3) Based on the learned regression projection matrix, we obtain the contribution of each EEG feature, which enables us to automatically identify critical frequency bands and brain regions in emotion recognition.

An Investigation of Olfactory-Enhanced Video on EEG-Based Emotion Recognition

Collecting emotional physiological signals is significant in building affective Human-Computer Interactions (HCI). However, how to evoke subjects' emotions efficiently in EEG-related emotional experiments is still a challenge. In this work, we developed a novel experimental paradigm that allows odors dynamically participate in different stages of video-evoked emotions, to investigate the efficiency of olfactory-enhanced videos in inducing subjects' emotions; According to the period that the odors participated in, the stimuli were divided into four patterns, i.e., the olfactory-enhanced video in early/later stimulus periods (OVEP/OVLP), and the traditional videos in early/later stimulus periods (TVEP/TVLP). The differential entropy (DE) feature and four classifiers were employed to test the efficiency of emotion recognition. The best average accuracies of the OVEP, OVLP, TVEP, and TVLP were 50.54%, 51.49%, 40.22%, and 57.55%, respectively. The experimental results indicated that the OVEP significantly outperformed the TVEP on classification performance, while there was no significant difference between the OVLP and TVLP. Besides, olfactory-enhanced videos achieved higher efficiency in evoking negative emotions than traditional videos. Moreover, we found that the neural patterns in response to emotions under different stimulus methods were stable, and for Fp1, FP2, and F7, there existed significant differences in whether adopt the odors.

Classifying oscillatory brain activity associated with Indian Rasas using network metrics

Neural signatures for the western classification of emotions have been widely discussed in the literature. The ancient Indian treatise on performing arts known as Natyashastra categorizes emotions into nine classes, known as Rasas. Rasa-as opposed to a pure emotion-is defined as a superposition of certain transitory, dominant, and temperamental emotional states. Although Rasas have been widely discussed in the text, dedicated brain imaging studies have not been conducted in their research. Our study examines the neural oscillations, recorded through electroencephalography (EEG) imaging, that are elicited while experiencing emotional states corresponding to Rasas. We identify differences among them using network-based functional connectivity metrics in five different frequency bands. Further, Random Forest models are trained on the extracted network features, and we present our findings based on classifier predictions. We observe slow (delta) and fast brain waves (beta and gamma) exhibited the maximum discriminating features between Rasas, whereas alpha and theta bands showed fewer distinguishable pairs. Out of nine Rasas, Sringaram (love), Bibhatsam (odious), and Bhayanakam (terror) were distinguishable from other Rasas the most across frequency bands. On the scale of most network metrics, Raudram (rage) and Sringaram are on the extremes, which also resulted in their good classification accuracy of 95%. This is reminiscent of the circumplex model where anger and contentment/happiness are on extremes on the pleasant scale. Interestingly, our results are consistent with the previous studies which highlight the significant role of higher frequency oscillations in the classification of emotions, in contrast to the alpha band that has shows non-significant differences across emotions. This research contributes to one of the first attempts to investigate the neural correlates of Rasas. Therefore, the results of this study can potentially guide the explorations into the entrainment of brain oscillations between performers and viewers, which can further lead to better performances and viewer experience.

Electroencephalograph-Based Emotion Recognition Using Brain Connectivity Feature and Domain Adaptive Residual Convolution Model

In electroencephalograph (EEG) emotion recognition research, obtaining high-level emotional features with more discriminative information has become the key to improving the classification performance. This study proposes a new end-to-end emotion recognition method based on brain connectivity (BC) features and domain adaptive residual convolutional network (short for BC-DA-RCNN), which could effectively extract the spatial connectivity information related to the emotional state of the human brain and introduce domain adaptation to achieve accurate emotion recognition within and across the subject’s EEG signals. The BC information is represented by the global brain network connectivity matrix. The DA-RCNN is used to extract high-level emotional features between different dimensions of EEG signals, reduce the domain offset between different subjects, and strengthen the common features between different subjects. The experimental results on the large public DEAP data set show that the accuracy of the subject-dependent and subject-independent binary emotion classification in valence reaches 95.15 and 88.28%, respectively, which outperforms all the benchmark methods. The proposed method is proven to have lower complexity, better generalization ability, and domain robustness that help to lay a solid foundation for the development of high-performance affective brain-computer interface applications.

OGSSL: A Semi-Supervised Classification Model Coupled With Optimal Graph Learning for EEG Emotion Recognition

Electroencephalogram(EEG) signals are generated from central nervous system which are difficult to disguise, leading to its popularity in emotion recognition. Recently,semi-supervisedlearning exhibits promisingemotion recognition performance by involving unlabeled EEG data into model training. However, if we first build a graph to characterize the sample similarities and then perform label propagation on this graph, these two steps cannotwell collaborate with each other. In this paper, we propose an OptimalGraph coupledSemi-Supervised Learning (OGSSL) model for EEG emotion recognition by unifying the adaptive graph learning and emotion recognition into a single objective. Besides, we improve the label indicator matrix of unlabeledsamples in order to directly obtain theiremotional states. Moreover, the key EEG frequency bands and brain regions in emotion expression are automatically recognized by the projectionmatrix of OGSSL. Experimental results on the SEED-IV data set demonstrate that 1) OGSSL achieves excellent average accuracies of 76.51%, 77.08% and 81.29% in three cross-sessionemotion recognition tasks, 2) OGSSL is competent for discriminative EEG feature selection in emotion recognition, and 3) the Gamma frequency band, the left/righttemporal, prefrontal,and (central) parietal lobes are identified to be more correlated with the occurrence of emotions.

Coupled Projection Transfer Metric Learning for Cross-Session Emotion Recognition from EEG

Distribution discrepancies between different sessions greatly degenerate the performance of video-evoked electroencephalogram (EEG) emotion recognition. There are discrepancies since the EEG signal is weak and non-stationary and these discrepancies are manifested in different trails in each session and even in some trails which belong to the same emotion. To this end, we propose a Coupled Projection Transfer Metric Learning (CPTML) model to jointly complete domain alignment and graph-based metric learning, which is a unified framework to simultaneously minimize cross-session and cross-trial divergences. By experimenting on the SEED_IV emotional dataset, we show that (1) CPTML exhibits a significantly better performance than several other approaches; (2) the cross-session distribution discrepancies are minimized and emotion metric graph across different trials are optimized in the CPTML-induced subspace, indicating the effectiveness of data alignment and metric exploration; and (3) critical EEG frequency bands and channels for emotion recognition are automatically identified from the learned projection matrices, providing more insights into the occurrence of the effect.