SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG

Cross-subject emotion recognition in brain-computer interface based on frequency band attention graph convolutional adversarial neural networks

BACKGROUND

Emotion is an important area in neuroscience. Cross-subject emotion recognition based on electroencephalogram (EEG) data is challenging due to physiological differences between subjects. Domain gap, which refers to the different distributions of EEG data at different subjects, has attracted great attention for cross-subject emotion recognition.

COMPARISON WITH EXISTING METHODS

This study focuses on narrowing the domain gap between subjects through the emotional frequency bands and the relationship information between EEG channels. Emotional frequency band features represent the energy distribution of EEG data in different frequency ranges, while relationship information between EEG channels provides spatial distribution information about EEG data.

NEW METHOD

To achieve this, this paper proposes a model called the Frequency Band Attention Graph convolutional Adversarial neural Network (FBAGAN). This model includes three components: a feature extractor, a classifier, and a discriminator. The feature extractor consists of a layer with a frequency band attention mechanism and a graph convolutional neural network. The mechanism effectively extracts frequency band information by assigning weights and Graph Convolutional Networks can extract relationship information between EEG channels by modeling the graph structure. The discriminator then helps minimize the gap in the frequency information and relationship information between the source and target domains, improving the model's ability to generalize.

RESULTS

The FBAGAN model is extensively tested on the SEED, SEED-IV, and DEAP datasets. The accuracy and standard deviation scores are 88.17% and 4.88, respectively, on the SEED dataset, and 77.35% and 3.72 on the SEED-IV dataset. On the DEAP dataset, the model achieves 69.64% for Arousal and 65.18% for Valence. These results outperform most existing models.

CONCLUSIONS

The experiments indicate that FBAGAN effectively addresses the challenges of transferring EEG channel domain and frequency band domain, leading to improved performance.

Multi-source Selective Graph Domain Adaptation Network for cross-subject EEG emotion recognition

Affective brain-computer interface is an important part of realizing emotional human-computer interaction. However, existing objective individual differences among subjects significantly hinder the application of electroencephalography (EEG) emotion recognition. Existing methods still lack the complete extraction of subject-invariant representations for EEG and the ability to fuse valuable information from multiple subjects to facilitate the emotion recognition of the target subject. To address the above challenges, we propose a Multi-source Selective Graph Domain Adaptation Network (MSGDAN), which can better utilize data from different source subjects and perform more robust emotion recognition on the target subject. The proposed network extracts and selects the individual information specific to each subject, where public information refers to subject-invariant components from multi-source subjects. Moreover, the graph domain adaptation network captures both functional connectivity and regional states of the brain via a dynamic graph network and then integrates graph domain adaptation to ensure the invariance of both functional connectivity and regional states. To evaluate our method, we conduct cross-subject emotion recognition experiments on the SEED, SEED-IV, and DEAP datasets. The results demonstrate that the MSGDAN has superior classification performance.

The influence of EEG channels and features significance on automatic detection of epileptic waves in MECT

Modified Electric Convulsive Therapy (MECT) is an efficacious physical therapy in treating mental disorders. The occurrence of epilepsy is a crucial benchmark for evaluating therapeutic effectiveness. However, the medical field still lacks relevant research on automatically detecting epileptic waves in MECT. Therefore, this article proposes a novel automatic detection method of epileptic waves in MECT. In this article, EEG local features (time, frequency, and time-frequency domains) and global features (Pearson correlation coefficient) are combined for epileptic wave detection with SVM (Support Vector Machine). We researched the system with 15 EEG detection channels. The dataset under investigation contains EEG data from 22 patients who received MECT and presented with epileptic seizures. The results revealed that LA (Logarithm of Activity) feature exhibits the best classification significance. When epileptic waves appear, there is a decrease in the power ratio of delta waves and an increase in the power ratio of theta waves. Additionally, the complexity of EEG decreases while the correlation between EEG channels increases. The Cz, F4, and P3 channels exhibit the highest classification significance among all EEG channels. Furthermore, based on the channel classification significance, the EEG detection channels number can be reduced to 8. Similarly, based on the feature classification significance, the local feature number can be reduced from 9 to 3. These conclusions can improve detection efficiency and reduce the cost for MECT. Moreover, the method we proposed can effectively detect epileptic waves in MECT. This work can provide physicians with a reference for evaluating the effectiveness of MECT.

Emotion recognition of EEG signals based on contrastive learning graph convolutional model

Objective.Electroencephalogram (EEG) signals offer invaluable insights into the complexities of emotion generation within the brain. Yet, the variability in EEG signals across individuals presents a formidable obstacle for empirical implementations. Our research addresses these challenges innovatively, focusing on the commonalities within distinct subjects' EEG data.Approach.We introduce a novel approach named Contrastive Learning Graph Convolutional Network (CLGCN). This method captures the distinctive features and crucial channel nodes related to individuals' emotional states. Specifically, CLGCN merges the dual benefits of CL's synchronous multisubject data learning and the GCN's proficiency in deciphering brain connectivity matrices. Understanding multifaceted brain functions and their information interchange processes is realized as CLGCN generates a standardized brain network learning matrix during a dataset's learning process.Main results.Our model underwent rigorous testing on the Database for Emotion Analysis using Physiological Signals (DEAP) and SEED datasets. In the five-fold cross-validation used for dependent subject experimental setting, it achieved an accuracy of 97.13% on the DEAP dataset and surpassed 99% on the SEED and SEED_IV datasets. In the incremental learning experiments with the SEED dataset, merely 5% of the data was sufficient to fine-tune the model, resulting in an accuracy of 92.8% for the new subject. These findings validate the model's efficacy.Significance.This work combines CL with GCN, improving the accuracy of decoding emotional states from EEG signals and offering valuable insights into uncovering the underlying mechanisms of emotional processes in the brain.

EEG-based emotion recognition using graph convolutional neural network with dual attention mechanism

EEG-based emotion recognition is becoming crucial in brain-computer interfaces (BCI). Currently, most researches focus on improving accuracy, while neglecting further research on the interpretability of models, we are committed to analyzing the impact of different brain regions and signal frequency bands on emotion generation based on graph structure. Therefore, this paper proposes a method named Dual Attention Mechanism Graph Convolutional Neural Network (DAMGCN). Specifically, we utilize graph convolutional neural networks to model the brain network as a graph to extract representative spatial features. Furthermore, we employ the self-attention mechanism of the Transformer model which allocates more electrode channel weights and signal frequency band weights to important brain regions and frequency bands. The visualization of attention mechanism clearly demonstrates the weight allocation learned by DAMGCN. During the performance evaluation of our model on the DEAP, SEED, and SEED-IV datasets, we achieved the best results on the SEED dataset, showing subject-dependent experiments' accuracy of 99.42% and subject-independent experiments' accuracy of 73.21%. The results are demonstrably superior to the accuracies of most existing models in the realm of EEG-based emotion recognition.

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Objective. Physiological signals based emotion recognition is a prominent research domain in the field of human-computer interaction. Previous studies predominantly focused on unimodal data, giving limited attention to the interplay among multiple modalities. Within the scope of multimodal emotion recognition, integrating the information from diverse modalities and leveraging the complementary information are the two essential issues to obtain the robust representations.Approach. Thus, we propose a intermediate fusion strategy for combining low-rank tensor fusion with the cross-modal attention to enhance the fusion of electroencephalogram, electrooculogram, electromyography, and galvanic skin response. Firstly, handcrafted features from distinct modalities are individually fed to corresponding feature extractors to obtain latent features. Subsequently, low-rank tensor is fused to integrate the information by the modality interaction representation. Finally, a cross-modal attention module is employed to explore the potential relationships between the distinct latent features and modality interaction representation, and recalibrate the weights of different modalities. And the resultant representation is adopted for emotion recognition.Main results. Furthermore, to validate the effectiveness of the proposed method, we execute subject-independent experiments within the DEAP dataset. The proposed method has achieved the accuracies of 73.82% and 74.55% for valence and arousal classification.Significance. The results of extensive experiments verify the outstanding performance of the proposed method.

Effects of tDCS on emotion recognition and brain oscillations

INTRODUCTION

Emotion recognition, the ability to interpret the emotional state of individuals by looking at their facial expressions, is essential for healthy social interactions and communication. There is limited research on the effects of tDCS on emotion recognition in the literature. This study aimed to investigate the effects of anodal stimulation of the ventromedial prefrontal cortex (vmPFC), a key region for emotion recognition from facial expressions, on emotion recognition and brain oscillations.

METHOD

A single-blind randomized-controlled study was conducted with 54 healthy participants. Before and after brain stimulation emotion recognition tasks were administered and resting-state EEG were recorded. The changes in task performances and brain oscillations were analyzed using repeated-measures two-way ANOVA analysis.

RESULTS

There was no significant difference in the emotion recognition tasks between groups in pre-post measurements. The changes in delta, theta, alpha, beta and gamma frequency bands in the frontal, temporal, and posterio-occipital regions, which were determined as regions of interest in resting state EEG data before and after tDCS, were compared between groups. The results showed that there was a significant difference between groups only in delta frequency before and after tDCS in the frontal and temporal regions. While an increase in delta activity was observed in the experimental group in the frontal and temporal regions, a decrease was observed in the control group.

CONCLUSIONS

The tDCS may not have improved emotion recognition because it may not have had the desired effect on the vmPFC, which is in the lower part of the prefrontal lobe. The changes in EEG frequencies observed section tDCS may be similar to those seen in some pathological processes, which could explain the lack of improvement in emotion recognition. Future studies to be carried out for better understand this effect are important.

A comprehensive review of deep learning in EEG-based emotion recognition: classifications, trends, and practical implications

Emotion recognition utilizing EEG signals has emerged as a pivotal component of human-computer interaction. In recent years, with the relentless advancement of deep learning techniques, using deep learning for analyzing EEG signals has assumed a prominent role in emotion recognition. Applying deep learning in the context of EEG-based emotion recognition carries profound practical implications. Although many model approaches and some review articles have scrutinized this domain, they have yet to undergo a comprehensive and precise classification and summarization process. The existing classifications are somewhat coarse, with insufficient attention given to the potential applications within this domain. Therefore, this article systematically classifies recent developments in EEG-based emotion recognition, providing researchers with a lucid understanding of this field's various trajectories and methodologies. Additionally, it elucidates why distinct directions necessitate distinct modeling approaches. In conclusion, this article synthesizes and dissects the practical significance of EEG signals in emotion recognition, emphasizing its promising avenues for future application.

Optimal channel dynamic selection for Constructing lightweight Data EEG-based emotion recognition

At present, most methods to improve the accuracy of emotion recognition based on electroencephalogram (EEG) are achieved by means of increasing the number of channels and feature types. This is to use the big data to train the classification model but it also increases the code complexity and consumes a large amount of computer time. We propose a method of Ant Colony Optimization with Convolutional Neural Networks and Long Short-Term Memory (ACO-CNN-LSTM) which can attain the dynamic optimal channels for lightweight data. First, transform the time-domain EEG signal to the frequency domain by Fast Fourier Transform (FFT), and the Differential Entropy (DE) of the three frequency bands (α , β and γ ) are extracted as the feature data; Then, based on the DE feature dataset, ACO is employed to plan the path where the electrodes are located in the brain map. The classification accuracy of CNN-LSTM is used as the objective function for path determination, and the electrodes on the optimal path are used as the optimal channels; Next, the initial learning rate and batchsize parameters are exactly matched the data characteristics, which can obtain the best initial learning rate and batchsize; Finally, the SJTU Emotion EEG Dataset (SEED) dataset is used for emotion recognition based on the ACO-CNN-LSTM. From the experimental results, it can be seen that: the average accuracy of three-classification (positive, neutral, negative) can achieve 96.59 %, which is based on the lightweight data by means of ACO-CNN-LSTM proposed in the paper. Meanwhile, the computer time consumed is reduced. The computational efficiency is increased by 15.85 % compared with the traditional CNN-LSTM method. The accuracy can achieve more than 90 % when the data volume is reduced to 50 %. In summary, the proposed method of ACO-CNN-LSTM in the paper can get higher efficiency and accuracy.

Detecting emotions through EEG signals based on modified convolutional fuzzy neural network

Emotion is a human sense that can influence an individual's life quality in both positive and negative ways. The ability to distinguish different types of emotion can lead researchers to estimate the current situation of patients or the probability of future disease. Recognizing emotions from images have problems concealing their feeling by modifying their facial expressions. This led researchers to consider Electroencephalography (EEG) signals for more accurate emotion detection. However, the complexity of EEG recordings and data analysis using conventional machine learning algorithms caused inconsistent emotion recognition. Therefore, utilizing hybrid deep learning models and other techniques has become common due to their ability to analyze complicated data and achieve higher performance by integrating diverse features of the models. However, researchers prioritize models with fewer parameters to achieve the highest average accuracy. This study improves the Convolutional Fuzzy Neural Network (CFNN) for emotion recognition using EEG signals to achieve a reliable detection system. Initially, the pre-processing and feature extraction phases are implemented to obtain noiseless and informative data. Then, the CFNN with modified architecture is trained to classify emotions. Several parametric and comparative experiments are performed. The proposed model achieved reliable performance for emotion recognition with an average accuracy of 98.21% and 98.08% for valence (pleasantness) and arousal (intensity), respectively, and outperformed state-of-the-art methods.

An enhanced Coati Optimization Algorithm for global optimization and feature selection in EEG emotion recognition

Emotion recognition based on Electroencephalography (EEG) signals has garnered significant attention across diverse domains including healthcare, education, information sharing, and gaming, among others. Despite its potential, the absence of a standardized feature set poses a challenge in efficiently classifying various emotions. Addressing the issue of high dimensionality, this paper introduces an advanced variant of the Coati Optimization Algorithm (COA), called eCOA for global optimization and selecting the best subset of EEG features for emotion recognition. Specifically, COA suffers from local optima and imbalanced exploitation abilities as other metaheuristic methods. The proposed eCOA incorporates the COA and RUNge Kutta Optimizer (RUN) algorithms. The Scale Factor (SF) and Enhanced Solution Quality (ESQ) mechanism from RUN are applied to resolve the raised shortcomings of COA. The proposed eCOA algorithm has been extensively evaluated using the CEC'22 test suite and two EEG emotion recognition datasets, DEAP and DREAMER. Furthermore, the eCOA is applied for binary and multi-class classification of emotions in the dimensions of valence, arousal, and dominance using a multi-layer perceptron neural network (MLPNN). The experimental results revealed that the eCOA algorithm has more powerful search capabilities than the original COA and seven well-known counterpart methods related to statistical, convergence, and diversity measures. Furthermore, eCOA can efficiently support feature selection to find the best EEG features to maximize performance on four quadratic emotion classification problems compared to the methods of its counterparts. The suggested method obtains a classification accuracy of 85.17% and 95.21% in the binary classification of low and high arousal emotions in two public datasets: DEAP and DREAMER, respectively, which are 5.58% and 8.98% superior to existing approaches working on the same datasets for different subjects, respectively.

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

Emotional recognition is a pivotal research domain in computer and cognitive science. Recent advancements have led to various emotion recognition methods, leveraging data from diverse sources like speech, facial expressions, electroencephalogram (EEG), electrocardiogram, and eye tracking (ET). This article introduces a novel emotion recognition framework, primarily targeting the analysis of users' psychological reactions and stimuli. It is important to note that the stimuli eliciting emotional responses are as critical as the responses themselves. Hence, our approach synergizes stimulus data with physical and physiological signals, pioneering a multimodal method for emotional cognition. Our proposed framework unites stimulus source data with physiological signals, aiming to enhance the accuracy and robustness of emotion recognition through data integration. We initiated an emotional cognition experiment to gather EEG and ET data alongside recording emotional responses. Building on this, we developed the Emotion-Multimodal Fusion Neural Network (E-MFNN), optimized for multimodal data fusion to process both stimulus and physiological data. We conducted extensive comparisons between our framework's outcomes and those from existing models, also assessing various algorithmic approaches within our framework. This comparison underscores our framework's efficacy in multimodal emotion recognition. The source code is publicly available at https://figshare.com/s/8833d837871c78542b29.

Music-evoked emotions classification using vision transformer in EEG signals

Introduction

The field of electroencephalogram (EEG)-based emotion identification has received significant attention and has been widely utilized in both human-computer interaction and therapeutic settings. The process of manually analyzing electroencephalogram signals is characterized by a significant investment of time and work. While machine learning methods have shown promising results in classifying emotions based on EEG data, the task of extracting distinct characteristics from these signals still poses a considerable difficulty.

Methods

In this study, we provide a unique deep learning model that incorporates an attention mechanism to effectively extract spatial and temporal information from emotion EEG recordings. The purpose of this model is to address the existing gap in the field. The implementation of emotion EEG classification involves the utilization of a global average pooling layer and a fully linked layer, which are employed to leverage the discernible characteristics. In order to assess the effectiveness of the suggested methodology, we initially gathered a dataset of EEG recordings related to music-induced emotions.

Experiments

Subsequently, we ran comparative tests between the state-of-the-art algorithms and the method given in this study, utilizing this proprietary dataset. Furthermore, a publicly accessible dataset was included in the subsequent comparative trials.

Discussion

The experimental findings provide evidence that the suggested methodology outperforms existing approaches in the categorization of emotion EEG signals, both in binary (positive and negative) and ternary (positive, negative, and neutral) scenarios.

ERTNet: an interpretable transformer-based framework for EEG emotion recognition

Background

Emotion recognition using EEG signals enables clinicians to assess patients' emotional states with precision and immediacy. However, the complexity of EEG signal data poses challenges for traditional recognition methods. Deep learning techniques effectively capture the nuanced emotional cues within these signals by leveraging extensive data. Nonetheless, most deep learning techniques lack interpretability while maintaining accuracy.

Methods

We developed an interpretable end-to-end EEG emotion recognition framework rooted in the hybrid CNN and transformer architecture. Specifically, temporal convolution isolates salient information from EEG signals while filtering out potential high-frequency noise. Spatial convolution discerns the topological connections between channels. Subsequently, the transformer module processes the feature maps to integrate high-level spatiotemporal features, enabling the identification of the prevailing emotional state.

Results

Experiments' results demonstrated that our model excels in diverse emotion classification, achieving an accuracy of 74.23% ± 2.59% on the dimensional model (DEAP) and 67.17% ± 1.70% on the discrete model (SEED-V). These results surpass the performances of both CNN and LSTM-based counterparts. Through interpretive analysis, we ascertained that the beta and gamma bands in the EEG signals exert the most significant impact on emotion recognition performance. Notably, our model can independently tailor a Gaussian-like convolution kernel, effectively filtering high-frequency noise from the input EEG data.

Discussion

Given its robust performance and interpretative capabilities, our proposed framework is a promising tool for EEG-driven emotion brain-computer interface.

PS-VTS: particle swarm with visit table strategy for automated emotion recognition with EEG signals

Recognizing emotions accurately in real life is crucial in human-computer interaction (HCI) systems. Electroencephalogram (EEG) signals have been extensively employed to identify emotions. The researchers have used several EEG-based emotion identification datasets to validate their proposed models. In this paper, we have employed a novel metaheuristic optimization approach for accurate emotion classification by applying it to select both channel and rhythm of EEG data. In this work, we have proposed the particle swarm with visit table strategy (PS-VTS) metaheuristic technique to improve the effectiveness of EEG-based human emotion identification. First, the EEG signals are denoised using a low pass filter, and then rhythm extraction is done using discrete wavelet transform (DWT). The continuous wavelet transform (CWT) approach transforms each rhythm signal into a rhythm image. The pre-trained MobilNetv2 model has been pre-trained for deep feature extraction, and a support vector machine (SVM) is used to classify the emotions. Two models are developed for optimal channels and rhythm sets. In Model 1, optimal channels are selected separately for each rhythm, and global optima are determined in the optimization process according to the best channel sets of the rhythms. The best rhythms are first determined for each channel, and then the optimal channel-rhythm set is selected in Model 2. Our proposed model obtained an accuracy of 99.2871% and 97.8571% for the classification of HA (high arousal)-LA (low arousal) and HV (high valence)-LV (low valence), respectively with the DEAP dataset. Our generated model obtained the highest classification accuracy compared to the previously reported methods.

Hybrid transfer learning strategy for cross-subject EEG emotion recognition

Emotion recognition constitutes a pivotal research topic within affective computing, owing to its potential applications across various domains. Currently, emotion recognition methods based on deep learning frameworks utilizing electroencephalogram (EEG) signals have demonstrated effective application and achieved impressive performance. However, in EEG-based emotion recognition, there exists a significant performance drop in cross-subject EEG Emotion recognition due to inter-individual differences among subjects. In order to address this challenge, a hybrid transfer learning strategy is proposed, and the Domain Adaptation with a Few-shot Fine-tuning Network (DFF-Net) is designed for cross-subject EEG emotion recognition. The first step involves the design of a domain adaptive learning module specialized for EEG emotion recognition, known as the Emo-DA module. Following this, the Emo-DA module is utilized to pre-train a model on both the source and target domains. Subsequently, fine-tuning is performed on the target domain specifically for the purpose of cross-subject EEG emotion recognition testing. This comprehensive approach effectively harnesses the attributes of domain adaptation and fine-tuning, resulting in a noteworthy improvement in the accuracy of the model for the challenging task of cross-subject EEG emotion recognition. The proposed DFF-Net surpasses the state-of-the-art methods in the cross-subject EEG emotion recognition task, achieving an average recognition accuracy of 93.37% on the SEED dataset and 82.32% on the SEED-IV dataset.

Exploring Adaptive Graph Topologies and Temporal Graph Networks for EEG-Based Depression Detection

In recent years, Graph Neural Networks (GNNs) based on deep learning techniques have achieved promising results in EEG-based depression detection tasks but still have some limitations. Firstly, most existing GNN-based methods use pre-computed graph adjacency matrices, which ignore the differences in brain networks between individuals. Additionally, methods based on graph-structured data do not consider the temporal dependency information of brain networks. To address these issues, we propose a deep learning algorithm that explores adaptive graph topologies and temporal graph networks for EEG-based depression detection. Specifically, we designed an Adaptive Graph Topology Generation (AGTG) module that can adaptively model the real-time connectivity of the brain networks, revealing differences between individuals. In addition, we designed a Graph Convolutional Gated Recurrent Unit (GCGRU) module to capture the temporal dynamical changes of brain networks. To further explore the differential features between depressed and healthy individuals, we adopt Graph Topology-based Max-Pooling (GTMP) module to extract graph representation vectors accurately. We conduct a comparative analysis with several advanced algorithms on both public and our own datasets. The results reveal that our final model achieves the highest Area Under the Receiver Operating Characteristic Curve (AUROC) on both datasets, with values of 83% and 99%, respectively. Furthermore, we perform extensive validation experiments demonstrating our proposed method's effectiveness and advantages. Finally, we present a comprehensive discussion on the differences in brain networks between healthy and depressed individuals based on the outputs of our final model's AGTG and GTMP modules.

Enhancing the accuracy of electroencephalogram-based emotion recognition through Long Short-Term Memory recurrent deep neural networks

Introduction

Emotions play a critical role in human communication, exerting a significant influence on brain function and behavior. One effective method of observing and analyzing these emotions is through electroencephalography (EEG) signals. Although numerous studies have been dedicated to emotion recognition (ER) using EEG signals, achieving improved accuracy in recognition remains a challenging task. To address this challenge, this paper presents a deep-learning approach for ER using EEG signals.

Background

ER is a dynamic field of research with diverse practical applications in healthcare, human-computer interaction, and affective computing. In ER studies, EEG signals are frequently employed as they offer a non-invasive and cost-effective means of measuring brain activity. Nevertheless, accurately identifying emotions from EEG signals poses a significant challenge due to the intricate and non-linear nature of these signals.

Methods

The present study proposes a novel approach for ER that encompasses multiple stages, including feature extraction, feature selection (FS) employing clustering, and classification using Dual-LSTM. To conduct the experiments, the DEAP dataset was employed, wherein a clustering technique was applied to Hurst's view and statistical features during the FS phase. Ultimately, Dual-LSTM was employed for accurate ER.

Results

The proposed method achieved a remarkable accuracy of 97.5% in accurately classifying emotions across four categories: arousal, valence, liking/disliking, dominance, and familiarity. This high level of accuracy serves as strong evidence for the effectiveness of the deep-learning approach to emotion recognition (ER) utilizing EEG signals.

Conclusion

The deep-learning approach proposed in this paper has shown promising results in emotion recognition using EEG signals. This method can be useful in various applications, such as developing more effective therapies for individuals with mood disorders or improving human-computer interaction by allowing machines to respond more intelligently to users' emotional states. However, further research is needed to validate the proposed method on larger datasets and to investigate its applicability to real-world scenarios.

Emotion recognition in EEG signals using deep learning methods: A review

Emotions are a critical aspect of daily life and serve a crucial role in human decision-making, planning, reasoning, and other mental states. As a result, they are considered a significant factor in human interactions. Human emotions can be identified through various sources, such as facial expressions, speech, behavior (gesture/position), or physiological signals. The use of physiological signals can enhance the objectivity and reliability of emotion detection. Compared with peripheral physiological signals, electroencephalogram (EEG) recordings are directly generated by the central nervous system and are closely related to human emotions. EEG signals have the great spatial resolution that facilitates the evaluation of brain functions, making them a popular modality in emotion recognition studies. Emotion recognition using EEG signals presents several challenges, including signal variability due to electrode positioning, individual differences in signal morphology, and lack of a universal standard for EEG signal processing. Moreover, identifying the appropriate features for emotion recognition from EEG data requires further research. Finally, there is a need to develop more robust artificial intelligence (AI) including conventional machine learning (ML) and deep learning (DL) methods to handle the complex and diverse EEG signals associated with emotional states. This paper examines the application of DL techniques in emotion recognition from EEG signals and provides a detailed discussion of relevant articles. The paper explores the significant challenges in emotion recognition using EEG signals, highlights the potential of DL techniques in addressing these challenges, and suggests the scope for future research in emotion recognition using DL techniques. The paper concludes with a summary of its findings.

A Review of Recurrent Neural Network-Based Methods in Computational Physiology

Artificial intelligence and machine learning techniques have progressed dramatically and become powerful tools required to solve complicated tasks, such as computer vision, speech recognition, and natural language processing. Since these techniques have provided promising and evident results in these fields, they emerged as valuable methods for applications in human physiology and healthcare. General physiological recordings are time-related expressions of bodily processes associated with health or morbidity. Sequence classification, anomaly detection, decision making, and future status prediction drive the learning algorithms to focus on the temporal pattern and model the nonstationary dynamics of the human body. These practical requirements give birth to the use of recurrent neural networks (RNNs), which offer a tractable solution in dealing with physiological time series and provide a way to understand complex time variations and dependencies. The primary objective of this article is to provide an overview of current applications of RNNs in the area of human physiology for automated prediction and diagnosis within different fields. Finally, we highlight some pathways of future RNN developments for human physiology.

Multi-channel EEG emotion recognition through residual graph attention neural network

In this paper, a novel EEG emotion recognition method based on residual graph attention neural network is proposed. The method constructs a three-dimensional sparse feature matrix according to the relative position of electrode channels, and inputs it into the residual network to extract high-level abstract features containing electrode spatial position information. At the same time, the adjacency matrix representing the connection relationship of electrode channels is constructed, and the time-domain features of multi-channel EEG are modeled using graph. Then, the graph attention neural network is utilized to learn the intrinsic connection relationship between EEG channels located in different brain regions from the adjacency matrix and the constructed graph structure data. Finally, the high-level abstract features extracted from the two networks are fused to judge the emotional state. The experiment is carried out on DEAP data set. The experimental results show that the spatial domain information of electrode channels and the intrinsic connection relationship between different channels contain salient information related to emotional state, and the proposed model can effectively fuse these information to improve the performance of multi-channel EEG emotion recognition.

Identification of emotions evoked by music via spatial-temporal transformer in multi-channel EEG signals

Introduction

Emotion plays a vital role in understanding activities and associations. Due to being non-invasive, many experts have employed EEG signals as a reliable technique for emotion recognition. Identifying emotions from multi-channel EEG signals is evolving into a crucial task for diagnosing emotional disorders in neuroscience. One challenge with automated emotion recognition in EEG signals is to extract and select the discriminating features to classify different emotions accurately.

Methods

In this study, we proposed a novel Transformer model for identifying emotions from multi-channel EEG signals. Note that we directly fed the raw EEG signal into the proposed Transformer, which aims at eliminating the issues caused by the local receptive fields in the convolutional neural networks. The presented deep learning model consists of two separate channels to address the spatial and temporal information in the EEG signals, respectively.

Results

In the experiments, we first collected the EEG recordings from 20 subjects during listening to music. Experimental results of the proposed approach for binary emotion classification (positive and negative) and ternary emotion classification (positive, negative, and neutral) indicated the accuracy of 97.3 and 97.1%, respectively. We conducted comparison experiments on the same dataset using the proposed method and state-of-the-art techniques. Moreover, we achieved a promising outcome in comparison with these approaches.

Discussion

Due to the performance of the proposed approach, it can be a potentially valuable instrument for human-computer interface system.

EEG rhythm based emotion recognition using multivariate decomposition and ensemble machine learning classifier

Recently, electroencephalogram (EEG) signals have shown great potential to recognize human emotions. The goal of effective computing is to assist computers in understanding various types of emotions via human-computer interaction (HCI). Multichannel EEG signals are used to measure the electrical activity of the brain in space and time. Automated emotion recognition using multichannel EEG signals is an interesting area of cognitive neuroscience and affective computing research. This research proposes EEG multichannel rhythmic features and ensemble machine learning (EML) classifiers with leave-one-subject-out cross-validation (LOSOCV) for automatic emotion classification from multichannel EEG recordings. Multivariate fast iterative filtering (MvFIF) is used to assess the EEG rhythm sequences. EEG rhythms delta(δ), theta(θ), alpha(α), beta(β), and gamma(γ) are separated based on the mean frequency of the EEG rhythm sequence. Three Hjorth parameters and nine entropy features were extracted from multichannel EEG rhythms. Extracted features are selected using the minimum redundancy maximum relevance (mRMR) approach. The experimental design was performed on two emotional datasets (GAMEEMO and DREAMER). The validation showed that gamma rhythm multichannel features with EML-based subspace K-nearest neighbor (SS KNN) were as high as 93.5%-99.8%, achieving high classification accuracy. The comparisons of δ, θ, α, β, and γ rhythms with EML, support vector machine (SVM), and artificial neural network (ANN) were performed. we also analyzed multi-class emotions (HVHA, HVLA, LVHA, LVLA) with an ensemble-based bagging tree on gamma rhythm. It provides a novel solution for multichannel rhythm-specific features in EEG data analysis.

Emotion recognition using spatial-temporal EEG features through convolutional graph attention network

Objective.Constructing an efficient human emotion recognition model based on electroencephalogram (EEG) signals is significant for realizing emotional brain-computer interaction and improving machine intelligence.Approach.In this paper, we present a spatial-temporal feature fused convolutional graph attention network (STFCGAT) model based on multi-channel EEG signals for human emotion recognition. First, we combined the single-channel differential entropy (DE) feature with the cross-channel functional connectivity (FC) feature to extract both the temporal variation and spatial topological information of EEG. After that, a novel convolutional graph attention network was used to fuse the DE and FC features and further extract higher-level graph structural information with sufficient expressive power for emotion recognition. Furthermore, we introduced a multi-headed attention mechanism in graph neural networks to improve the generalization ability of the model.Main results.We evaluated the emotion recognition performance of our proposed model on the public SEED and DEAP datasets, which achieved a classification accuracy of 99.11% ± 0.83% and 94.83% ± 3.41% in the subject-dependent and subject-independent experiments on the SEED dataset, and achieved an accuracy of 91.19% ± 1.24% and 92.03% ± 4.57% for discrimination of arousal and valence in subject-independent experiments on DEAP dataset. Notably, our model achieved state-of-the-art performance on cross-subject emotion recognition tasks for both datasets. In addition, we gained insight into the proposed frame through both the ablation experiments and the analysis of spatial patterns of FC and DE features.Significance.All these results prove the effectiveness of the STFCGAT architecture for emotion recognition and also indicate that there are significant differences in the spatial-temporal characteristics of the brain under different emotional states.

Feature Pyramid Networks and Long Short-Term Memory for EEG Feature Map-Based Emotion Recognition

The original EEG data collected are the 1D sequence, which ignores spatial topology information; Feature Pyramid Networks (FPN) is better at small dimension target detection and insufficient feature extraction in the scale transformation than CNN. We propose a method of FPN and Long Short-Term Memory (FPN-LSTM) for EEG feature map-based emotion recognition. According to the spatial arrangement of brain electrodes, the Azimuth Equidistant Projection (AEP) is employed to generate the 2D EEG map, which preserves the spatial topology information; then, the average power, variance power, and standard deviation power of three frequency bands (α, β, and γ) are extracted as the feature data for the EEG feature map. BiCubic interpolation is employed to interpolate the blank pixel among the electrodes; the three frequency bands EEG feature maps are used as the G, R, and B channels to generate EEG feature maps. Then, we put forward the idea of distributing the weight proportion for channels, assign large weight to strong emotion correlation channels (AF3, F3, F7, FC5, and T7), and assign small weight to the others; the proposed FPN-LSTM is used on EEG feature maps for emotion recognition. The experiment results show that the proposed method can achieve Value and Arousal recognition rates of 90.05% and 90.84%, respectively.

An Efficient Machine Learning-Based Emotional Valence Recognition Approach Towards Wearable EEG

Emotion artificial intelligence (AI) is being increasingly adopted in several industries such as healthcare and education. Facial expressions and tone of speech have been previously considered for emotion recognition, yet they have the drawback of being easily manipulated by subjects to mask their true emotions. Electroencephalography (EEG) has emerged as a reliable and cost-effective method to detect true human emotions. Recently, huge research effort has been put to develop efficient wearable EEG devices to be used by consumers in out of the lab scenarios. In this work, a subject-dependent emotional valence recognition method is implemented that is intended for utilization in emotion AI applications. Time and frequency features were computed from a single time series derived from the Fp1 and Fp2 channels. Several analyses were performed on the strongest valence emotions to determine the most relevant features, frequency bands, and EEG timeslots using the benchmark DEAP dataset. Binary classification experiments resulted in an accuracy of 97.42% using the alpha band, by that outperforming several approaches from literature by ~3-22%. Multiclass classification gave an accuracy of 95.0%. Feature computation and classification required less than 0.1 s. The proposed method thus has the advantage of reduced computational complexity as, unlike most methods in the literature, only two EEG channels were considered. In addition, minimal features concluded from the thorough analyses conducted in this study were used to achieve state-of-the-art performance. The implemented EEG emotion recognition method thus has the merits of being reliable and easily reproducible, making it well-suited for wearable EEG devices.

Subject-independent EEG emotion recognition with hybrid spatio-temporal GRU-Conv architecture

Recently, various deep learning frameworks have shown excellent performance in decoding electroencephalogram (EEG) signals, especially in human emotion recognition. However, most of them just focus on temporal features and ignore the features based on spatial dimensions. Traditional gated recurrent unit (GRU) model performs well in processing time series data, and convolutional neural network (CNN) can obtain spatial characteristics from input data. Therefore, this paper introduces a hybrid GRU and CNN deep learning framework named GRU-Conv to fully leverage the advantages of both. Nevertheless, contrary to most previous GRU architectures, we retain the output information of all GRU units. So, the GRU-Conv model could extract crucial spatio-temporal features from EEG data. And more especially, the proposed model acquires the multi-dimensional features of multi-units after temporal processing in GRU and then uses CNN to extract spatial information from the temporal features. In this way, the EEG signals with different characteristics could be classified more accurately. Finally, the subject-independent experiment shows that our model has good performance on SEED and DEAP databases. The average accuracy of the former is 87.04%. The mean accuracy of the latter is 70.07% for arousal and 67.36% for valence.

Emotion Detection Using Deep Normalized Attention-Based Neural Network and Modified-Random Forest

In the contemporary world, emotion detection of humans is procuring huge scope in extensive dimensions such as bio-metric security, HCI (human-computer interaction), etc. Such emotions could be detected from various means, such as information integration from facial expressions, gestures, speech, etc. Though such physical depictions contribute to emotion detection, EEG (electroencephalogram) signals have gained significant focus in emotion detection due to their sensitivity to alterations in emotional states. Hence, such signals could explore significant emotional state features. However, manual detection from EEG signals is a time-consuming process. With the evolution of artificial intelligence, researchers have attempted to use different data mining algorithms for emotion detection from EEG signals. Nevertheless, they have shown ineffective accuracy. To resolve this, the present study proposes a DNA-RCNN (Deep Normalized Attention-based Residual Convolutional Neural Network) to extract the appropriate features based on the discriminative representation of features. The proposed NN also explores alluring features with the proposed attention modules leading to consistent performance. Finally, classification is performed by the proposed M-RF (modified-random forest) with an empirical loss function. In this process, the learning weights on the data subset alleviate loss amongst the predicted value and ground truth, which assists in precise classification. Performance and comparative analysis are considered to explore the better performance of the proposed system in detecting emotions from EEG signals that confirms its effectiveness.

Identifying Complex Emotions in Alexithymia Affected Adolescents Using Machine Learning Techniques

Many scientific researchers' study focuses on enhancing automated systems to identify emotions and thus relies on brain signals. This study focuses on how brain wave signals can be used to classify many emotional states of humans. Electroencephalography (EEG)-based affective computing predominantly focuses on emotion classification based on facial expression, speech recognition, and text-based recognition through multimodality stimuli. The proposed work aims to implement a methodology to identify and codify discrete complex emotions such as pleasure and grief in a rare psychological disorder known as alexithymia. This type of disorder is highly elicited in unstable, fragile countries such as South Sudan, Lebanon, and Mauritius. These countries are continuously affected by civil wars and disaster and politically unstable, leading to a very poor economy and education system. This study focuses on an adolescent age group dataset by recording physiological data when emotion is exhibited in a multimodal virtual environment. We decocted time frequency analysis and amplitude time series correlates including frontal alpha symmetry using a complex Morlet wavelet. For data visualization, we used the UMAP technique to obtain a clear district view of emotions. We performed 5-fold cross validation along with 1 s window subjective classification on the dataset. We opted for traditional machine learning techniques to identify complex emotion labeling.

Graph Theoretical Analysis of EEG Functional Connectivity Patterns and Fusion with Physiological Signals for Emotion Recognition

Emotion recognition is a key attribute for realizing advances in human-computer interaction, especially when using non-intrusive physiological sensors, such as electroencephalograph (EEG) and electrocardiograph. Although functional connectivity of EEG has been utilized for emotion recognition, the graph theory analysis of EEG connectivity patterns has not been adequately explored. The exploitation of brain network characteristics could provide valuable information regarding emotions, while the combination of EEG and peripheral physiological signals can reveal correlation patterns of human internal state. In this work, a graph theoretical analysis of EEG functional connectivity patterns along with fusion between EEG and peripheral physiological signals for emotion recognition has been proposed. After extracting functional connectivity from EEG signals, both global and local graph theory features are extracted. Those features are concatenated with statistical features from peripheral physiological signals and fed to different classifiers and a Convolutional Neural Network (CNN) for emotion recognition. The average accuracy on the DEAP dataset using CNN was 55.62% and 57.38% for subject-independent valence and arousal classification, respectively, and 83.94% and 83.87% for subject-dependent classification. Those scores went up to 75.44% and 78.77% for subject-independent classification and 88.27% and 90.84% for subject-dependent classification using a feature selection algorithm, exceeding the current state-of-the-art results.

Use of Differential Entropy for Automated Emotion Recognition in a Virtual Reality Environment with EEG Signals

Emotion recognition is one of the most important issues in human–computer interaction (HCI), neuroscience, and psychology fields. It is generally accepted that emotion recognition with neural data such as electroencephalography (EEG) signals, functional magnetic resonance imaging (fMRI), and near-infrared spectroscopy (NIRS) is better than other emotion detection methods such as speech, mimics, body language, facial expressions, etc., in terms of reliability and accuracy. In particular, EEG signals are bioelectrical signals that are frequently used because of the many advantages they offer in the field of emotion recognition. This study proposes an improved approach for EEG-based emotion recognition on a publicly available newly published dataset, VREED. Differential entropy (DE) features were extracted from four wavebands (theta 4–8 Hz, alpha 8–13 Hz, beta 13–30 Hz, and gamma 30–49 Hz) to classify two emotional states (positive/negative). Five classifiers, namely Support Vector Machine (SVM), k-Nearest Neighbor (kNN), Naïve Bayesian (NB), Decision Tree (DT), and Logistic Regression (LR) were employed with DE features for the automated classification of two emotional states. In this work, we obtained the best average accuracy of 76.22% ± 2.06 with the SVM classifier in the classification of two states. Moreover, we observed from the results that the highest average accuracy score was produced with the gamma band, as previously reported in studies in EEG-based emotion recognition.

Entropy-Based Emotion Recognition from Multichannel EEG Signals Using Artificial Neural Network

Humans experience a variety of emotions throughout the course of their daily lives, including happiness, sadness, and rage. As a result, an effective emotion identification system is essential for electroencephalography (EEG) data to accurately reflect emotion in real-time. Although recent studies on this problem can provide acceptable performance measures, it is still not adequate for the implementation of a complete emotion recognition system. In this research work, we propose a new approach for an emotion recognition system, using multichannel EEG calculation with our developed entropy known as multivariate multiscale modified-distribution entropy (MM-mDistEn) which is combined with a model based on an artificial neural network (ANN) to attain a better outcome over existing methods. The proposed system has been tested with two different datasets and achieved better accuracy than existing methods. For the GAMEEMO dataset, we achieved an average accuracy ± standard deviation of 95.73% ± 0.67 for valence and 96.78% ± 0.25 for arousal. Moreover, the average accuracy percentage for the DEAP dataset reached 92.57% ± 1.51 in valence and 80.23% ± 1.83 in arousal.

Machine Learning Models for Classification of Human Emotions Using Multivariate Brain Signals

Humans can portray different expressions contrary to their emotional state of mind. Therefore, it is difficult to judge humans’ real emotional state simply by judging their physical appearance. Although researchers are working on facial expressions analysis, voice recognition, and gesture recognition; the accuracy levels of such analysis are much less and the results are not reliable. Hence, it becomes vital to have realistic emotion detector. Electroencephalogram (EEG) signals remain neutral to the external appearance and behavior of the human and help in ensuring accurate analysis of the state of mind. The EEG signals from various electrodes in different scalp regions are studied for performance. Hence, EEG has gained attention over time to obtain accurate results for the classification of emotional states in human beings for human–machine interaction as well as to design a program where an individual could perform a self-analysis of his emotional state. In the proposed scheme, we extract power spectral densities of multivariate EEG signals from different sections of the brain. From the extracted power spectral density (PSD), the features which provide a better feature for classification are selected and classified using long short-term memory (LSTM) and bi-directional long short-term memory (Bi-LSTM). The 2-D emotion model considered for the classification of frontal, parietal, temporal, and occipital is studied. The region-based classification is performed by considering positive and negative emotions. The performance accuracy of our previous model’s results of artificial neural network (ANN), support vector machine (SVM), K-nearest neighbor (K-NN), and LSTM was compared and 94.95% accuracy was received using Bi-LSTM considering four prefrontal electrodes.

An Approach to Emotion Recognition Using Brain Rhythm Sequencing and Asymmetric Features

Emotion can be influenced during self-isolation, and to avoid severe mood swings, emotional regulation is meaningful. To achieve this, efficiently recognizing emotion is a vital step, which can be realized by electroencephalography signals. Previously, inspired by the knowledge of sequencing in bioinformatics, a method termed brain rhythm sequencing that analyzes electroencephalography as the sequence consisting of the dominant rhythm has been proposed for seizure detection. In this work, with the help of similarity measure methods, the asymmetric features are extracted from the sequences generated by different channel data. After evaluating all asymmetric features for emotion recognition, the optimal feature that yields remarkable accuracy is identified. Therefore, the classification task can be accomplished through a small amount of channel data. From a music emotion recognition experiment and a public DEAP dataset, the classification accuracies of various test sets are approximately 80–85% when employing an optimal feature extracted from one pair of symmetrical channels. Such performances are impressive when using fewer resources is a concern. Further investigation revealed that emotion recognition shows strongly individual characteristics, so an appropriate solution is to include the subject-dependent properties. Compared to the existing works, this method benefits from the design of a portable emotion-aware device used during self-isolation, as fewer scalp sensors are needed. Hence, it would provide a novel way to realize emotional applications in the future.

Deep Sparse Autoencoder and Recursive Neural Network for EEG Emotion Recognition

Recently, emotional electroencephalography (EEG) has been of great importance in brain-computer interfaces, and it is more urgent to realize automatic emotion recognition. The EEG signal has the disadvantages of being non-smooth, non-linear, stochastic, and susceptible to background noise. Additionally, EEG signal processing network models have the disadvantages of a large number of parameters and long training time. To address the above issues, a novel model is presented in this paper. Initially, a deep sparse autoencoder network (DSAE) was used to remove redundant information from the EEG signal and reconstruct its underlying features. Further, combining a convolutional neural network (CNN) with long short-term memory (LSTM) can extract relevant features from task-related features, mine the correlation between the 32 channels of the EEG signal, and integrate contextual information from these frames. The proposed DSAE + CNN + LSTM (DCRNN) model was experimented with on the public dataset DEAP. The classification accuracies of valence and arousal reached 76.70% and 81.43%, respectively. Meanwhile, we conducted experiments with other comparative methods to further demonstrate the effectiveness of the DCRNN method.

The multiscale 3D convolutional network for emotion recognition based on electroencephalogram

Emotion recognition based on EEG (electroencephalogram) has become a research hotspot in the field of brain-computer interfaces (BCI). Compared with traditional machine learning, the convolutional neural network model has substantial advantages in automatic feature extraction in EEG-based emotion recognition. Motivated by the studies that multiple smaller scale kernels could increase non-linear expression than a larger scale, we propose a 3D convolutional neural network model with multiscale convolutional kernels to recognize emotional states based on EEG signals. We select more suitable time window data to carry out the emotion recognition of four classes (low valence vs. low arousal, low valence vs. high arousal, high valence vs. low arousal, and high valence vs. high arousal). The results using EEG signals in the DEAP and SEED-IV datasets show accuracies for our proposed emotion recognition network model (ERN) of 95.67 and 89.55%, respectively. The experimental results demonstrate that the proposed approach is potentially useful for enhancing emotional experience in BCI.

CR-GCN: Channel-Relationships-Based Graph Convolutional Network for EEG Emotion Recognition

Electroencephalography (EEG) is recorded by electrodes from different areas of the brain and is commonly used to measure neuronal activity. EEG-based methods have been widely used for emotion recognition recently. However, most current methods for EEG-based emotion recognition do not fully exploit the relationship of EEG channels, which affects the precision of emotion recognition. To address the issue, in this paper, we propose a novel method for EEG-based emotion recognition called CR-GCN: Channel-Relationships-based Graph Convolutional Network. Specifically, topological structure of EEG channels is distance-based and tends to capture local relationships, and brain functional connectivity tends to capture global relationships among EEG channels. Therefore, in this paper, we construct EEG channel relationships using an adjacency matrix in graph convolutional network where the adjacency matrix captures both local and global relationships among different EEG channels. Extensive experiments demonstrate that CR-GCN method significantly outperforms the state-of-the-art methods. In subject-dependent experiments, the average classification accuracies of 94.69% and 93.95% are achieved for valence and arousal. In subject-independent experiments, the average classification accuracies of 94.78% and 93.46% are obtained for valence and arousal.

Representation Learning and Pattern Recognition in Cognitive Biometrics: A Survey

Cognitive biometrics is an emerging branch of biometric technology. Recent research has demonstrated great potential for using cognitive biometrics in versatile applications, including biometric recognition and cognitive and emotional state recognition. There is a major need to summarize the latest developments in this field. Existing surveys have mainly focused on a small subset of cognitive biometric modalities, such as EEG and ECG. This article provides a comprehensive review of cognitive biometrics, covering all the major biosignal modalities and applications. A taxonomy is designed to structure the corresponding knowledge and guide the survey from signal acquisition and pre-processing to representation learning and pattern recognition. We provide a unified view of the methodological advances in these four aspects across various biosignals and applications, facilitating interdisciplinary research and knowledge transfer across fields. Furthermore, this article discusses open research directions in cognitive biometrics and proposes future prospects for developing reliable and secure cognitive biometric systems.

Correlating Vibration Patterns to Perception of Tactile Information for Long-Term Prosthetic Limb Use and Continued Rehabilitation of Neuropathic Pain

Prosthetic limbs (and orthotic devices) have been used as a paradigm for the treatment and rehabilitation of neuropathic pain, such as phantom limb pain. Long-term adoption of the devices for the continued use in rehabilitation remains low in part due to reduced embodiment and the high cognitive load associated with controlling the device. Previous research has shown that incorporating sensory feedback in prostheses can provide proprioceptive information, increase control and manipulation of objects, and improve embodiment. However, feedback experienced by the user varies daily and requires constant parameter adjustments to maintain accurate and intuitive sensory perception, further preventing long term adoption. Work therefore needs to be explored that correlate feedback modalities to perception of tactile information, such as texture and pressure. The work presented in this paper begins to explore this by utilizing a deep-learning algorithm to classify the dissipation of vibration artefacts found in the EMG signals of able-bodied individuals to specific texture patterns. Four texture patterns were applied to 7 participants using two vibration motors and repeated 3 times. In post processing, a RNN network identified the artefact features along equidistantly spaced EMG electrodes and correctly classified unseen data from each participant.

Multimodal Neurophysiological Transformer for Emotion Recognition

Understanding neural function often requires multiple modalities of data, including electrophysiogical data, imaging techniques, and demographic surveys. In this paper, we introduce a novel neurophysiological model to tackle major challenges in modeling multimodal data. First, we avoid non-alignment issues between raw signals and extracted, frequency-domain features by addressing the issue of variable sampling rates. Second, we encode modalities through "cross-attention" with other modalities. Lastly, we utilize properties of our parent transformer architecture to model long-range dependencies between segments across modalities and assess intermediary weights to better understand how source signals affect prediction. We apply our Multimodal Neurophysiological Transformer (MNT) to predict valence and arousal in an existing open-source dataset. Experiments on non-aligned multimodal time-series show that our model performs similarly and, in some cases, outperforms existing methods in classification tasks. In addition, qualitative analysis suggests that MNT is able to model neural influences on autonomic activity in predicting arousal. Our architecture has the potential to be fine-tuned to a variety of downstream tasks, including for BCI systems.

Effect of time windows in LSTM networks for EEG-based BCIs

People with impaired motor function could be helped by an effective brain–computer interface (BCI) based on a real-time electroencephalogram (EEG) and artificial intelligence algorithms. However, current methodologies for interpreting patient instructions from an EEG are not accurate enough to be completely safe in a real-world situation , where a poor decision would place their physical integrity at risk, such as when traveling in an electric wheelchair in a city. For various reasons, such as the low signal-to-noise ratio of portable EEGs or the effects of signal contamination (disturbances due to user movement, temporal variation of the features of EEG signals, etc.), a long short-term memory network (LSTM) (a type of recurrent neural network) that is able to learn data flow patterns from EEG signals could improve the classification of the actions taken by the user. In this paper, the effectiveness of using an LSTM with a low-cost wireless EEG device in real time is tested, and the time window that maximizes its classification accuracy is studied. The goal is to be able to implement it in the BCI of a smart wheelchair with a simple coded command protocol, such as opening or closing the eyes, which could be executed by patients with reduced mobility. Results show a higher resolution of the LSTM with an accuracy range between 77.61 and 92.14% compared to traditional classifiers (59.71%), and an optimal time window of around 7 s for the task done by users in this work. In addition, tests in real-life contexts show that a trade-off between accuracy and response times is necessary to ensure detection.