EEG-Based Emotion Classification Using a Deep Neural Network and Sparse Autoencoder

A Fine-grained Hemispheric Asymmetry Network for accurate and interpretable EEG-based emotion classification

In this work, we propose a Fine-grained Hemispheric Asymmetry Network (FG-HANet), an end-to-end deep learning model that leverages hemispheric asymmetry features within 2-Hz narrow frequency bands for accurate and interpretable emotion classification over raw EEG data. In particular, the FG-HANet extracts features not only from original inputs but also from their mirrored versions, and applies Finite Impulse Response (FIR) filters at a granularity as fine as 2-Hz to acquire fine-grained spectral information. Furthermore, to guarantee sufficient attention to hemispheric asymmetry features, we tailor a three-stage training pipeline for the FG-HANet to further boost its performance. We conduct extensive evaluations on two public datasets, SEED and SEED-IV, and experimental results well demonstrate the superior performance of the proposed FG-HANet, i.e. 97.11% and 85.70% accuracy, respectively, building a new state-of-the-art. Our results also reveal the hemispheric dominance under different emotional states and the hemisphere asymmetry within 2-Hz frequency bands in individuals. These not only align with previous findings in neuroscience but also provide new insights into underlying emotion generation mechanisms.

Emotion Recognition Model of EEG Signals Based on Double Attention Mechanism

BACKGROUND

Emotions play a crucial role in people's lives, profoundly affecting their cognition, decision-making, and interpersonal communication. Emotion recognition based on brain signals has become a significant challenge in the fields of affective computing and human-computer interaction.

METHODS

Addressing the issue of inaccurate feature extraction and low accuracy of existing deep learning models in emotion recognition, this paper proposes a multi-channel automatic classification model for emotion EEG signals named DACB, which is based on dual attention mechanisms, convolutional neural networks, and bidirectional long short-term memory networks. DACB extracts features in both temporal and spatial dimensions, incorporating not only convolutional neural networks but also SE attention mechanism modules for learning the importance of different channel features, thereby enhancing the network's performance. DACB also introduces dot product attention mechanisms to learn the importance of spatial and temporal features, effectively improving the model's accuracy.

RESULTS

The accuracy of this method in single-shot validation tests on the SEED-IV and DREAMER (Valence-Arousal-Dominance three-classification) datasets is 99.96% and 87.52%, 90.06%, and 89.05%, respectively. In 10-fold cross-validation tests, the accuracy is 99.73% and 84.26%, 85.40%, and 85.02%, outperforming other models.

CONCLUSIONS

This demonstrates that the DACB model achieves high accuracy in emotion classification tasks, demonstrating outstanding performance and generalization ability and providing new directions for future research in EEG signal recognition.

hvEEGNet: a novel deep learning model for high-fidelity EEG reconstruction

Introduction

Modeling multi-channel electroencephalographic (EEG) time-series is a challenging tasks, even for the most recent deep learning approaches. Particularly, in this work, we targeted our efforts to the high-fidelity reconstruction of this type of data, as this is of key relevance for several applications such as classification, anomaly detection, automatic labeling, and brain-computer interfaces.

Methods

We analyzed the most recent works finding that high-fidelity reconstruction is seriously challenged by the complex dynamics of the EEG signals and the large inter-subject variability. So far, previous works provided good results in either high-fidelity reconstruction of single-channel signals, or poor-quality reconstruction of multi-channel datasets. Therefore, in this paper, we present a novel deep learning model, called hvEEGNet, designed as a hierarchical variational autoencoder and trained with a new loss function. We tested it on the benchmark Dataset 2a (including 22-channel EEG data from 9 subjects).

Results

We show that it is able to reconstruct all EEG channels with high-fidelity, fastly (in a few tens of epochs), and with high consistency across different subjects. We also investigated the relationship between reconstruction fidelity and the training duration and, using hvEEGNet as an anomaly detector, we spotted some data in the benchmark dataset that are corrupted and never highlighted before.

Discussion

Thus, hvEEGNet could be very useful in several applications where automatic labeling of large EEG dataset is needed and time-consuming. At the same time, this work opens new fundamental research questions about (1) the effectiveness of deep learning models training (for EEG data) and (2) the need for a systematic characterization of the input EEG data to ensure robust modeling.

Frontal EEG correlation based human emotion identification and classification

Humans express their feelings and intentions of their actions or communication through emotions. Recent advancements in technology involve machines in human communication in day-to-day life. Thus, understanding of human emotions by machines will be very helpful in assisting the user in a far better way. Various physiological and non-physiological signals can be used to make the machines to recognize the emotion of a person. The identification of emotional content in the signals is crucial to understand emotion and the machines act with emotional intelligence at appropriate times, thus providing a better human machine interaction with emotion identification system and mental health monitoring for psychiatric patients. This work includes the creation of an emotion EEG dataset, the development of an algorithm for identifying the emotion elicitation segments in the EEG signal, and the classification of emotions from EEG signals. The EEG signals are divided into 3s segments, and the segments with emotional content are selected based on the decrease in correlation between the frontal electrodes. The selected segments are validated with the facial expressions of the subjects in the appropriate time segments of the face video. EEGNet is used to classify the emotion from the EEG signal. The classification accuracy with the selected emotional EEG segments is higher compared to the accuracy using all the EEG segments. In subject-specific classification, an average accuracy of 80.87% is obtained from the network trained with selected EEG segments, and 70.5% is obtained from training with all EEG segments. In subject-independent classification, the accuracy of classification is 67% and 63.8% with and without segment selection, respectively. The proposed method of selection of EEG segments is validated using the DEAP dataset, and classification accuracies and F1-scores of subject dependent and subject-independent methods are presented.

Mild focal cooling selectively impacts computations in dendritic trees

Focal cooling is a powerful technique to temporally scale neural dynamics. However, the underlying cellular mechanisms causing this scaling remain unresolved. Here, using targeted focal cooling (with a spatial resolution of 100 micrometers), dual somato-dendritic patch clamp recordings, two-photon calcium imaging, transmitter uncaging, and modeling we reveal that a 5°C drop can enhance synaptic transmission, plasticity, and input-output transformations in the distal apical tuft, but not in the basal dendrites of intrinsically bursting L5 pyramidal neurons. This enhancement depends on N-methyl-D-aspartate (NMDA) and Kv4.2, suggesting electrical structure modulation. Paradoxically, and despite the increase in tuft excitability, we observe a reduced rate of recovery from inactivation for apical Na+ channels, thereby regulating back-propagating action potential invasion, coincidence detection, and overall burst probability, resulting in an "apparent" slowing of somatic spike output. Our findings reveal a differential temperature sensitivity along the basal-tuft axis of L5 neurons analog modulates cortical output.

Review of deep representation learning techniques for brain-computer interfaces

In the field of brain-computer interfaces (BCIs), the potential for leveraging deep learning techniques for representing electroencephalogram (EEG) signals has gained substantial interest.Objective: This review synthesizes empirical findings from a collection of articles using deep representation learning techniques for BCI decoding, to provide a comprehensive analysis of the current state-of-the-art.Approach: Each article was scrutinized based on three criteria: (1) the deep representation learning technique employed, (2) the underlying motivation for its utilization, and (3) the approaches adopted for characterizing the learned representations.Main results: Among the 81 articles finally reviewed in depth, our analysis reveals a predominance of 31 articles using autoencoders. We identified 13 studies employing self-supervised learning (SSL) techniques, among which ten were published in 2022 or later, attesting to the relative youth of the field. However, at the time being, none of these have led to standard foundation models that are picked up by the BCI community. Likewise, only a few studies have introspected their learned representations. We observed that the motivation in most studies for using representation learning techniques is for solving transfer learning tasks, but we also found more specific motivations such as to learn robustness or invariances, as an algorithmic bridge, or finally to uncover the structure of the data.Significance: Given the potential of foundation models to effectively tackle these challenges, we advocate for a continued dedication to the advancement of foundation models specifically designed for EEG signal decoding by using SSL techniques. We also underline the imperative of establishing specialized benchmarks and datasets to facilitate the development and continuous improvement of such foundation models.

EEG emotion recognition based on an innovative information potential index

The recent exceptional demand for emotion recognition systems in clinical and non-medical applications has attracted the attention of many researchers. Since the brain is the primary object of understanding emotions and responding to them, electroencephalogram (EEG) signal analysis is one of the most popular approaches in affect classification. Previously, different approaches have been presented to benefit from brain connectivity information. We envisioned analyzing the interactions between brain electrodes with the information potential and providing a new index to quantify the connectivity matrix. The current study proposed a simple measure based on the cross-information potential between pairs of EEG electrodes to characterize emotions. This measure was tested for different EEG frequency bands to realize which EEG waves could be fruitful in recognizing emotions. Support vector machine and k-nearest neighbor (kNN) were implemented to classify four emotion categories based on two-dimensional valence and arousal space. Experimental results on the Database for Emotion Analysis using Physiological signals revealed a maximum accuracy of 90.14%, a sensitivity of 89.71%, and an F-score of 94.57% using kNN. The gamma frequency band obtained the highest recognition rates. Furthermore, low valence-low arousal was classified more effectively than other classes.

A review of artificial intelligence methods enabled music-evoked EEG emotion recognition and their applications

Music is an archaic form of emotional expression and arousal that can induce strong emotional experiences in listeners, which has important research and practical value in related fields such as emotion regulation. Among the various emotion recognition methods, the music-evoked emotion recognition method utilizing EEG signals provides real-time and direct brain response data, playing a crucial role in elucidating the neural mechanisms underlying music-induced emotions. Artificial intelligence technology has greatly facilitated the research on the recognition of music-evoked EEG emotions. AI algorithms have ushered in a new era for the extraction of characteristic frequency signals and the identification of novel feature signals. The robust computational capabilities of AI have provided fresh perspectives for the development of innovative quantitative models of emotions, tailored to various emotion recognition paradigms. The discourse surrounding AI algorithms in the context of emotional classification models is gaining momentum, with their applications in music therapy, neuroscience, and social activities increasingly coming under the spotlight. Through an in-depth analysis of the complete process of emotion recognition induced by music through electroencephalography (EEG) signals, we have systematically elucidated the influence of AI on pertinent research issues. This analysis offers a trove of innovative approaches that could pave the way for future research endeavors.

Emotion recognition of EEG signals based on contrastive learning graph convolutional model

Objective.Electroencephalogram (EEG) signals offer invaluable insights into the complexities of emotion generation within the brain. Yet, the variability in EEG signals across individuals presents a formidable obstacle for empirical implementations. Our research addresses these challenges innovatively, focusing on the commonalities within distinct subjects' EEG data.Approach.We introduce a novel approach named Contrastive Learning Graph Convolutional Network (CLGCN). This method captures the distinctive features and crucial channel nodes related to individuals' emotional states. Specifically, CLGCN merges the dual benefits of CL's synchronous multisubject data learning and the GCN's proficiency in deciphering brain connectivity matrices. Understanding multifaceted brain functions and their information interchange processes is realized as CLGCN generates a standardized brain network learning matrix during a dataset's learning process.Main results.Our model underwent rigorous testing on the Database for Emotion Analysis using Physiological Signals (DEAP) and SEED datasets. In the five-fold cross-validation used for dependent subject experimental setting, it achieved an accuracy of 97.13% on the DEAP dataset and surpassed 99% on the SEED and SEED_IV datasets. In the incremental learning experiments with the SEED dataset, merely 5% of the data was sufficient to fine-tune the model, resulting in an accuracy of 92.8% for the new subject. These findings validate the model's efficacy.Significance.This work combines CL with GCN, improving the accuracy of decoding emotional states from EEG signals and offering valuable insights into uncovering the underlying mechanisms of emotional processes in the brain.

Enhancing Arousal Level Detection in EEG Signals through Genetic Algorithm-based Feature Selection and Fast Bit Hopping

Background

This study explores a novel approach to detecting arousal levels through the analysis of electroencephalography (EEG) signals. Leveraging the Faller database with data from 18 healthy participants, we employ a 64-channel EEG system.

Methods

The approach we employ entails the extraction of ten frequency characteristics from every channel, culminating in a feature vector of 640 dimensions for each signal instance. To enhance classification accuracy, we employ a genetic algorithm for feature selection, treating it as a multiobjective optimization task. The approach utilizes fast bit hopping for efficiency, overcoming traditional bit-string limitations. A hybrid operator expedites algorithm convergence, and a solution selection strategy identifies the most suitable feature subset.

Results

Experimental results demonstrate the method's effectiveness in detecting arousal levels across diverse states, with improvements in accuracy, sensitivity, and specificity. In scenario one, the proposed method achieves an average accuracy, sensitivity, and specificity of 93.11%, 98.37%, and 99.14%, respectively. In scenario two, the averages stand at 81.35%, 88.65%, and 84.64%.

Conclusions

The obtained results indicate that the proposed method has a high capability of detecting arousal levels in different scenarios. In addition, the advantage of employing the proposed feature reduction method has been demonstrated.

Lemniscate of Bernoulli's map quantifiers: innovative measures for EEG emotion recognition

Thanks to the advent of affective computing, designing an automatic human emotion recognition system for clinical and non-clinical applications has attracted the attention of many researchers. Currently, multi-channel electroencephalogram (EEG)-based emotion recognition is a fundamental but challenging issue. This experiment envisioned developing a new scheme for automated EEG affect recognition. An innovative nonlinear feature engineering approach was presented based on Lemniscate of Bernoulli's Map (LBM), which belongs to the family of chaotic maps, in line with the EEG's nonlinear nature. As far as the authors know, LBM has not been utilized for biological signal analysis. Next, the map was characterized using several graphical indices. The feature vector was imposed on the feature selection algorithm while evaluating the role of the feature vector dimension on emotion recognition rates. Finally, the efficiency of the features on emotion recognition was appraised using two conventional classifiers and validated using the Database for Emotion Analysis using Physiological signals (DEAP) and SJTU Emotion EEG Dataset-IV (SEED-IV) benchmark databases. The experimental results showed a maximum accuracy of 92.16% for DEAP and 90.7% for SEED-IV. Achieving higher recognition rates compared to the state-of-art EEG emotion recognition systems suggest the proposed method based on LBM could have potential both in characterizing bio-signal dynamics and detecting affect-deficit disorders.

Spatio-Temporal Image-Based Encoded Atlases for EEG Emotion Recognition

Emotion recognition plays an essential role in human-human interaction since it is a key to understanding the emotional states and reactions of human beings when they are subject to events and engagements in everyday life. Moving towards human-computer interaction, the study of emotions becomes fundamental because it is at the basis of the design of advanced systems to support a broad spectrum of application areas, including forensic, rehabilitative, educational, and many others. An effective method for discriminating emotions is based on ElectroEncephaloGraphy (EEG) data analysis, which is used as input for classification systems. Collecting brain signals on several channels and for a wide range of emotions produces cumbersome datasets that are hard to manage, transmit, and use in varied applications. In this context, the paper introduces the Empátheia system, which explores a different EEG representation by encoding EEG signals into images prior to their classification. In particular, the proposed system extracts spatio-temporal image encodings, or atlases, from EEG data through the Processing and transfeR of Interaction States and Mappings through Image-based eNcoding (PRISMIN) framework, thus obtaining a compact representation of the input signals. The atlases are then classified through the Empátheia architecture, which comprises branches based on convolutional, recurrent, and transformer models designed and tuned to capture the spatial and temporal aspects of emotions. Extensive experiments were conducted on the Shanghai Jiao Tong University (SJTU) Emotion EEG Dataset (SEED) public dataset, where the proposed system significantly reduced its size while retaining high performance. The results obtained highlight the effectiveness of the proposed approach and suggest new avenues for data representation in emotion recognition from EEG signals.

LieWaves: dataset for lie detection based on EEG signals and wavelets

This study introduces an electroencephalography (EEG)-based dataset to analyze lie detection. Various analyses or detections can be performed using EEG signals. Lie detection using EEG data has recently become a significant topic. In every aspect of life, people find the need to tell lies to each other. While lies told daily may not have significant societal impacts, lie detection becomes crucial in legal, security, job interviews, or situations that could affect the community. This study aims to obtain EEG signals for lie detection, create a dataset, and analyze this dataset using signal processing techniques and deep learning methods. EEG signals were acquired from 27 individuals using a wearable EEG device called Emotiv Insight with 5 channels (AF3, T7, Pz, T8, AF4). Each person took part in two trials: one where they were honest and another where they were deceitful. During each experiment, participants evaluated beads they saw before the experiment and stole from them in front of a video clip. This study consisted of four stages. In the first stage, the LieWaves dataset was created with the EEG data obtained during these experiments. In the second stage, preprocessing was carried out. In this stage, the automatic and tunable artifact removal (ATAR) algorithm was applied to remove the artifacts from the EEG signals. Later, the overlapping sliding window (OSW) method was used for data augmentation. In the third stage, feature extraction was performed. To achieve this, EEG signals were analyzed by combining discrete wavelet transform (DWT) and fast Fourier transform (FFT) including statistical methods (SM). In the last stage, each obtained feature vector was classified separately using Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and CNNLSTM hybrid algorithms. At the study's conclusion, the most accurate result, achieving a 99.88% accuracy score, was produced using the LSTM and DWT techniques. With this study, a new data set was introduced to the literature, and it was aimed to eliminate the deficiencies in this field with this data set. Evaluation results obtained from the data set have shown that this data set can be effective in this field.

Positional multi-length and mutual-attention network for epileptic seizure classification

The automatic classification of epilepsy electroencephalogram (EEG) signals plays a crucial role in diagnosing neurological diseases. Although promising results have been achieved by deep learning methods in this task, capturing the minute abnormal characteristics, contextual information, and long dependencies of EEG signals remains a challenge. To address this challenge, a positional multi-length and mutual-attention (PMM) network is proposed for the automatic classification of epilepsy EEG signals. The PMM network incorporates a positional feature encoding process that extracts minute abnormal characteristics from the EEG signal and utilizes a multi-length feature learning process with a hierarchy residual dilated LSTM (RDLSTM) to capture long contextual dependencies. Furthermore, a mutual-attention feature reinforcement process is employed to learn the global and relative feature dependencies and enhance the discriminative abilities of the network. To validate the effectiveness PMM network, we conduct extensive experiments on the public dataset and the experimental results demonstrate the superior performance of the PMM network compared to state-of-the-art methods.

Enhancing EEG-based attachment style prediction: unveiling the impact of feature domains

Introduction

Attachment styles are crucial in human relationships and have been explored through neurophysiological responses and EEG data analysis. This study investigates the potential of EEG data in predicting and differentiating secure and insecure attachment styles, contributing to the understanding of the neural basis of interpersonal dynamics.

Methods

We engaged 27 participants in our study, employing an XGBoost classifier to analyze EEG data across various feature domains, including time-domain, complexity-based, and frequency-based attributes.

Results

The study found significant differences in the precision of attachment style prediction: a high precision rate of 96.18% for predicting insecure attachment, and a lower precision of 55.34% for secure attachment. Balanced accuracy metrics indicated an overall model accuracy of approximately 84.14%, taking into account dataset imbalances.

Discussion

These results highlight the challenges in using EEG patterns for attachment style prediction due to the complex nature of attachment insecurities. Individuals with heightened perceived insecurity predominantly aligned with the insecure attachment category, suggesting a link to their increased emotional reactivity and sensitivity to social cues. The study underscores the importance of time-domain features in prediction accuracy, followed by complexity-based features, while noting the lesser impact of frequency-based features. Our findings advance the understanding of the neural correlates of attachment and pave the way for future research, including expanding demographic diversity and integrating multimodal data to refine predictive models.

Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition

Introduction

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging.

Methods

In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples.

Results

To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches.

Discussion

The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

Development of depression detection algorithm using text scripts of routine psychiatric interview

Background

A psychiatric interview is one of the important procedures in diagnosing psychiatric disorders. Through this interview, psychiatrists listen to the patient's medical history and major complaints, check their emotional state, and obtain clues for clinical diagnosis. Although there have been attempts to diagnose a specific mental disorder from a short doctor-patient conversation, there has been no attempt to classify the patient's emotional state based on the text scripts from a formal interview of more than 30 min and use it to diagnose depression. This study aimed to utilize the existing machine learning algorithm in diagnosing depression using the transcripts of one-on-one interviews between psychiatrists and depressed patients.

Methods

Seventy-seven clinical patients [with depression (n = 60); without depression (n = 17)] with a prior psychiatric diagnosis history participated in this study. The study was conducted with 24 male and 53 female subjects with the mean age of 33.8 (± 3.0). Psychiatrists conducted a conversational interview with each patient that lasted at least 30 min. All interviews with the subjects between August 2021 and November 2022 were recorded and transcribed into text scripts, and a text emotion recognition module was used to indicate the subject's representative emotions of each sentence. A machine learning algorithm discriminates patients with depression and those without depression based on text scripts.

Results

A machine learning model classified text scripts from depressive patients with non-depressive ones with an acceptable accuracy rate (AUC of 0.85). The distribution of emotions (surprise, fear, anger, love, sadness, disgust, neutral, and happiness) was significantly different between patients with depression and those without depression (p < 0.001), and the most contributing emotion in classifying the two groups was disgust (p < 0.001).

Conclusion

This is a qualitative and retrospective study to develop a tool to detect depression against patients without depression based on the text scripts of psychiatric interview, suggesting a novel and practical approach to understand the emotional characteristics of depression patients and to use them to detect the diagnosis of depression based on machine learning methods. This model could assist psychiatrists in clinical settings who conduct routine conversations with patients using text transcripts of the interviews.

Automatic epileptic seizure detection based on persistent homology

Epilepsy is a prevalent brain disease, which is quite difficult-to-treat or cure. This study developed a novel automatic seizure detection method based on the persistent homology method. In this study, a Vietoris-Rips (VR) complex filtration model was constructed based on the EEG data. And the persistent homology method was applied to calculate the VR complex filtration barcodes to describe the topological changes of EEG recordings. Afterward, the barcodes as the topological characteristics of EEG signals were fed into the GoogLeNet for classification. The persistent homology is applicable for multi-channel EEG data analysis, where the global topological information is calculated and the features are extracted by considering the multi-channel EEG data as a whole, without the multiple calculations or the post-stitching. Three databases were used to evaluate the proposed approach and the results showed that the approach had high performances in the epilepsy detection. The results obtained from the CHB-MIT Database recordings revealed that the proposed approach can achieve a segment-based averaged accuracy, sensitivity and specificity values of 97.05%, 96.71% and 97.38%, and achieve an event-based averaged sensitivity value of 100% with 1.22 s average detection latency. In addition, on the Siena Scalp Database, the proposed method yields averaged accuracy, sensitivity and specificity values of 96.42%, 95.23% and 97.6%. Multiple tasks of the Bonn Database also showed achieved accuracy of 99.55%, 98.63%, 98.28% and 97.68%, respectively. The experimental results on these three EEG databases illustrate the efficiency and robustness of our approach for automatic detection of epileptic seizure.

BiTCAN: A emotion recognition network based on saliency in brain cognition

In recent years, with the continuous development of artificial intelligence and brain-computer interfaces, emotion recognition based on electroencephalogram (EEG) signals has become a prosperous research direction. Due to saliency in brain cognition, we construct a new spatio-temporal convolutional attention network for emotion recognition named BiTCAN. First, in the proposed method, the original EEG signals are de-baselined, and the two-dimensional mapping matrix sequence of EEG signals is constructed by combining the electrode position. Second, on the basis of the two-dimensional mapping matrix sequence, the features of saliency in brain cognition are extracted by using the Bi-hemisphere discrepancy module, and the spatio-temporal features of EEG signals are captured by using the 3-D convolution module. Finally, the saliency features and spatio-temporal features are fused into the attention module to further obtain the internal spatial relationships between brain regions, and which are input into the classifier for emotion recognition. Many experiments on DEAP and SEED (two public datasets) show that the accuracies of the proposed algorithm on both are higher than 97%, which is superior to most existing emotion recognition algorithms.

S-LSTM-ATT: a hybrid deep learning approach with optimized features for emotion recognition in electroencephalogram

Purpose

Human emotion recognition using electroencephalograms (EEG) is a critical area of research in human-machine interfaces. Furthermore, EEG data are convoluted and diverse; thus, acquiring consistent results from these signals remains challenging. As such, the authors felt compelled to investigate EEG signals to identify different emotions.

Methods

A novel deep learning (DL) model stacked long short-term memory with attention (S-LSTM-ATT) model is proposed for emotion recognition (ER) in EEG signals. Long Short-Term Memory (LSTM) and attention networks effectively handle time-series EEG data and recognise intrinsic connections and patterns. Therefore, the model combined the strengths of the LSTM model and incorporated an attention network to enhance its effectiveness. Optimal features were extracted from the metaheuristic-based firefly optimisation algorithm (FFOA) to identify different emotions efficiently.

Results

The proposed approach recognised emotions in two publicly available standard datasets: SEED and EEG Brainwave. An outstanding accuracy of 97.83% in the SEED and 98.36% in the EEG Brainwave datasets were obtained for three emotion indices: positive, neutral and negative. Aside from accuracy, a comprehensive comparison of the proposed model's precision, recall, F1 score and kappa score was performed to determine the model's applicability. When applied to the SEED and EEG Brainwave datasets, the proposed S-LSTM-ATT achieved superior results to baseline models such as Convolutional Neural Networks (CNN), Gated Recurrent Unit (GRU) and LSTM.

Conclusion

Combining an FFOA-based feature selection (FS) and an S-LSTM-ATT-based classification model demonstrated promising results with high accuracy. Other metrics like precision, recall, F1 score and kappa score proved the suitability of the proposed model for ER in EEG signals.

Mental Stress Classification Based on Selected Electroencephalography Channels Using Correlation Coefficient of Hjorth Parameters

Electroencephalography (EEG) signals offer invaluable insights into diverse activities of the human brain, including the intricate physiological and psychological responses associated with mental stress. A major challenge, however, is accurately identifying mental stress while mitigating the limitations associated with a large number of EEG channels. Such limitations encompass computational complexity, potential overfitting, and the prolonged setup time for electrode placement, all of which can hinder practical applications. To address these challenges, this study presents the novel CCHP method, aimed at identifying and ranking commonly optimal EEG channels based on their sensitivity to the mental stress state. This method's uniqueness lies in its ability not only to find common channels, but also to prioritize them according to their responsiveness to stress, ensuring consistency across subjects and making it potentially transformative for real-world applications. From our rigorous examinations, eight channels emerged as universally optimal in detecting stress variances across participants. Leveraging features from the time, frequency, and time-frequency domains of these channels, and employing machine learning algorithms, notably RLDA, SVM, and KNN, our approach achieved a remarkable accuracy of 81.56% with the SVM algorithm outperforming existing methodologies. The implications of this research are profound, offering a stepping stone toward the development of real-time stress detection devices, and consequently, enabling clinicians to make more informed therapeutic decisions based on comprehensive brain activity monitoring.

A multi-head residual connection GCN for EEG emotion recognition

Electroencephalography (EEG) emotion recognition is a crucial aspect of human-computer interaction. However, conventional neural networks have limitations in extracting profound EEG emotional features. This paper introduces a novel multi-head residual graph convolutional neural network (MRGCN) model that incorporates complex brain networks and graph convolution networks. The decomposition of multi-band differential entropy (DE) features exposes the temporal intricacy of emotion-linked brain activity, and the combination of short and long-distance brain networks can explore complex topological characteristics. Moreover, the residual-based architecture not only enhances performance but also augments classification stability across subjects. The visualization of brain network connectivity offers a practical technique for investigating emotional regulation mechanisms. The MRGCN model exhibits average classification accuracies of 95.8% and 98.9% for the DEAP and SEED datasets, respectively, highlighting its excellent performance and robustness.

Improved EEG-based emotion recognition through information enhancement in connectivity feature map

Electroencephalography (EEG), despite its inherited complexity, is a preferable brain signal for automatic human emotion recognition (ER), which is a challenging machine learning task with emerging applications. In any automatic ER, machine learning (ML) models classify emotions using the extracted features from the EEG signals, and therefore, such feature extraction is a crucial part of ER process. Recently, EEG channel connectivity features have been widely used in ER, where Pearson correlation coefficient (PCC), mutual information (MI), phase-locking value (PLV), and transfer entropy (TE) are well-known methods for connectivity feature map (CFM) construction. CFMs are typically formed in a two-dimensional configuration using the signals from two EEG channels, and such two-dimensional CFMs are usually symmetric and hold redundant information. This study proposes the construction of a more informative CFM that can lead to better ER. Specifically, the proposed innovative technique intelligently combines CFMs' measures of two different individual methods, and its outcomes are more informative as a fused CFM. Such CFM fusion does not incur additional computational costs in training the ML model. In this study, fused CFMs are constructed by combining every pair of methods from PCC, PLV, MI, and TE; and the resulting fused CFMs PCC + PLV, PCC + MI, PCC + TE, PLV + MI, PLV + TE, and MI + TE are used to classify emotion by convolutional neural network. Rigorous experiments on the DEAP benchmark EEG dataset show that the proposed CFMs deliver better ER performances than CFM with a single connectivity method (e.g., PCC). At a glance, PLV + MI-based ER is shown to be the most promising one as it outperforms the other methods.

Emotion detection from EEG using transfer learning

In this study, we employed transfer learning to overcome the challenge of limited data availability in EEG-based emotion detection. The base model used in this study was Resnet50. Additionally, we employed a novel feature combination in EEG-based emotion detection. The input to the model was in the form of an image matrix, which comprised Mean Phase Coherence (MPC) and Magnitude Squared Coherence (MSC) in the upper-triangular and lower-triangular matrices, respectively. We further improved the technique by incorporating features obtained from the Differential Entropy (DE) into the diagonal. The dataset used in this study, SEED EEG (62 channel EEG), comprises three classes (Positive, Neutral, and Negative). We calculated both subject-independent and subject-dependent accuracy. The subject-dependent accuracy was obtained using a 10-fold cross-validation method and was 93.1%, while the subject-independent classification was performed by employing the leave-one-subject-out (LOSO) strategy. The accuracy obtained in subject-independent classification was 71.6%. Both of these accuracies are at least twice better than the chance accuracy of classifying 3 classes. The study found the use of MSC and MPC in EEG-based emotion detection promising for emotion classification. The future scope of this work includes the use of data augmentation techniques, enhanced classifiers, and better features for emotion classification.

A Convolutional Autoencoder-based Explainable Clustering Approach for Resting-State EEG Analysis. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023;2023:1-4. [PMID: 38083554 DOI: 10.1109/embc40787.2023.10340375]

Machine learning methods have frequently been applied to electroencephalography (EEG) data. However, while supervised EEG classification is well-developed, relatively few studies have clustered EEG, which is problematic given the potential for clustering EEG to identify novel subtypes or patterns of dynamics that could improve our understanding of neuropsychiatric disorders. There are established methods for clustering EEG using manually extracted features that reduce the richness of the feature space for clustering, but only a couple studies have sought to use deep learning-based approaches with automated feature learning to cluster EEG. Those studies involve separately training an autoencoder and then performing clustering on the extracted features, and the separation of those steps can lead to poor quality clustering. In this study, we propose an explainable convolutional autoencoder-based approach that combines model training with clustering to yield high quality clusters. We apply the approach within the context of schizophrenia (SZ), identifying 8 EEG states characterized by varying levels of δ activity. We also find that individuals who spend more time outside of the dominant state tend to have increased negative symptom severity. Our approach represents a significant step forward for clustering resting-state EEG data and has the potential to lead to novel findings across a variety of neurological and neuropsychological disorders in future years.

Functional Connectivity Analysis in Multi-channel EEG for Emotion Detection with Phase Locking Value and 3D CNN

The noise-assisted multivariate Empirical mode decomposition (NA-MEMD) is applied to multi-channel EEG signals to obtain narrow-band scale-aligned intrinsic mode functions (IMFs) upon which functional connectivity analysis is performed. The connectivity pattern in relation to inherent functional activity of brain is estimated with the phase locking value (PLV). Instantaneous phase difference among different EEG channels gives PLV that is used to build the functional connectivity map. The connectivity map yields spatial-temporal feature representation which is taken as input of the proposed emotion detection system. The spatial-temporal features can be learned with a 3D convolutional neural network for classifying emotion states. The proposed system is evaluated on two publicly available DEAP and SEED dataset for binary and multi-class emotion classification. On detecting low versus high level in the valence and arousal dimensions, the attained accuracy values are 97.37% and 96.26% respectively. Meanwhile, this system yields 94.78% and 99.54% accuracy on multi-class task on DEAP and SEED, which outperform previously reported systems with other deep learning models and conventional EEG features.

Personality-Based Emotion Recognition Using EEG Signals with a CNN-LSTM Network

The accurate detection of emotions has significant implications in healthcare, psychology, and human-computer interaction. Integrating personality information into emotion recognition can enhance its utility in various applications. The present study introduces a novel deep learning approach to emotion recognition, which utilizes electroencephalography (EEG) signals and the Big Five personality traits. The study recruited 60 participants and recorded their EEG data while they viewed unique sequence stimuli designed to effectively capture the dynamic nature of human emotions and personality traits. A pre-trained convolutional neural network (CNN) was used to extract emotion-related features from the raw EEG data. Additionally, a long short-term memory (LSTM) network was used to extract features related to the Big Five personality traits. The network was able to accurately predict personality traits from EEG data. The extracted features were subsequently used in a novel network to predict emotional states within the arousal and valence dimensions. The experimental results showed that the proposed classifier outperformed common classifiers, with a high accuracy of 93.97%. The findings suggest that incorporating personality traits as features in the designed network, for emotion recognition, leads to higher accuracy, highlighting the significance of examining these traits in the analysis of emotions.

Emotion Recognition Using a Novel Granger Causality Quantifier and Combined Electrodes of EEG

Electroencephalogram (EEG) connectivity patterns can reflect neural correlates of emotion. However, the necessity of evaluating bulky data for multi-channel measurements increases the computational cost of the EEG network. To date, several approaches have been presented to pick the optimal cerebral channels, mainly depending on available data. Consequently, the risk of low data stability and reliability has increased by reducing the number of channels. Alternatively, this study suggests an electrode combination approach in which the brain is divided into six areas. After extracting EEG frequency bands, an innovative Granger causality-based measure was introduced to quantify brain connectivity patterns. The feature was subsequently subjected to a classification module to recognize valence-arousal dimensional emotions. A Database for Emotion Analysis Using Physiological Signals (DEAP) was used as a benchmark database to evaluate the scheme. The experimental results revealed a maximum accuracy of 89.55%. Additionally, EEG-based connectivity in the beta-frequency band was able to effectively classify dimensional emotions. In sum, combined EEG electrodes can efficiently replicate 32-channel EEG information.

Emotion Recognition from Spatio-Temporal Representation of EEG Signals via 3D-CNN with Ensemble Learning Techniques

The recognition of emotions is one of the most challenging issues in human-computer interaction (HCI). EEG signals are widely adopted as a method for recognizing emotions because of their ease of acquisition, mobility, and convenience. Deep neural networks (DNN) have provided excellent results in emotion recognition studies. Most studies, however, use other methods to extract handcrafted features, such as Pearson correlation coefficient (PCC), Principal Component Analysis, Higuchi Fractal Dimension (HFD), etc., even though DNN is capable of generating meaningful features. Furthermore, most earlier studies largely ignored spatial information between the different channels, focusing mainly on time domain and frequency domain representations. This study utilizes a pre-trained 3D-CNN MobileNet model with transfer learning on the spatio-temporal representation of EEG signals to extract features for emotion recognition. In addition to fully connected layers, hybrid models were explored using other decision layers such as multilayer perceptron (MLP), k-nearest neighbor (KNN), extreme learning machine (ELM), XGBoost (XGB), random forest (RF), and support vector machine (SVM). Additionally, this study investigates the effects of post-processing or filtering output labels. Extensive experiments were conducted on the SJTU Emotion EEG Dataset (SEED) (three classes) and SEED-IV (four classes) datasets, and the results obtained were comparable to the state-of-the-art. Based on the conventional 3D-CNN with ELM classifier, SEED and SEED-IV datasets showed a maximum accuracy of 89.18% and 81.60%, respectively. Post-filtering improved the emotional classification performance in the hybrid 3D-CNN with ELM model for SEED and SEED-IV datasets to 90.85% and 83.71%, respectively. Accordingly, spatial-temporal features extracted from the EEG, along with ensemble classifiers, were found to be the most effective in recognizing emotions compared to state-of-the-art methods.

An Efficient Machine Learning-Based Emotional Valence Recognition Approach Towards Wearable EEG

Emotion artificial intelligence (AI) is being increasingly adopted in several industries such as healthcare and education. Facial expressions and tone of speech have been previously considered for emotion recognition, yet they have the drawback of being easily manipulated by subjects to mask their true emotions. Electroencephalography (EEG) has emerged as a reliable and cost-effective method to detect true human emotions. Recently, huge research effort has been put to develop efficient wearable EEG devices to be used by consumers in out of the lab scenarios. In this work, a subject-dependent emotional valence recognition method is implemented that is intended for utilization in emotion AI applications. Time and frequency features were computed from a single time series derived from the Fp1 and Fp2 channels. Several analyses were performed on the strongest valence emotions to determine the most relevant features, frequency bands, and EEG timeslots using the benchmark DEAP dataset. Binary classification experiments resulted in an accuracy of 97.42% using the alpha band, by that outperforming several approaches from literature by ~3-22%. Multiclass classification gave an accuracy of 95.0%. Feature computation and classification required less than 0.1 s. The proposed method thus has the advantage of reduced computational complexity as, unlike most methods in the literature, only two EEG channels were considered. In addition, minimal features concluded from the thorough analyses conducted in this study were used to achieve state-of-the-art performance. The implemented EEG emotion recognition method thus has the merits of being reliable and easily reproducible, making it well-suited for wearable EEG devices.

Status of deep learning for EEG-based brain-computer interface applications

In the previous decade, breakthroughs in the central nervous system bioinformatics and computational innovation have prompted significant developments in brain-computer interface (BCI), elevating it to the forefront of applied science and research. BCI revitalization enables neurorehabilitation strategies for physically disabled patients (e.g., disabled patients and hemiplegia) and patients with brain injury (e.g., patients with stroke). Different methods have been developed for electroencephalogram (EEG)-based BCI applications. Due to the lack of a large set of EEG data, methods using matrix factorization and machine learning were the most popular. However, things have changed recently because a number of large, high-quality EEG datasets are now being made public and used in deep learning-based BCI applications. On the other hand, deep learning is demonstrating great prospects for solving complex relevant tasks such as motor imagery classification, epileptic seizure detection, and driver attention recognition using EEG data. Researchers are doing a lot of work on deep learning-based approaches in the BCI field right now. Moreover, there is a great demand for a study that emphasizes only deep learning models for EEG-based BCI applications. Therefore, we introduce this study to the recent proposed deep learning-based approaches in BCI using EEG data (from 2017 to 2022). The main differences, such as merits, drawbacks, and applications are introduced. Furthermore, we point out current challenges and the directions for future studies. We argue that this review study will help the EEG research community in their future research.

Extracting a Novel Emotional EEG Topographic Map Based on a Stacked Autoencoder Network

Emotion recognition based on brain signals has increasingly become attractive to evaluate human's internal emotional states. Conventional emotion recognition studies focus on developing machine learning and classifiers. However, most of these methods do not provide information on the involvement of different areas of the brain in emotions. Brain mapping is considered as one of the most distinguishing methods of showing the involvement of different areas of the brain in performing an activity. Most mapping techniques rely on projection and visualization of only one of the electroencephalogram (EEG) subband features onto brain regions. The present study aims to develop a new EEG-based brain mapping, which combines several features to provide more complete and useful information on a single map instead of common maps. In this study, the optimal combination of EEG features for each channel was extracted using a stacked autoencoder (SAE) network and visualizing a topographic map. Based on the research hypothesis, autoencoders can extract optimal features for quantitative EEG (QEEG) brain mapping. The DEAP EEG database was employed to extract topographic maps. The accuracy of image classifiers using the convolutional neural network (CNN) was used as a criterion for evaluating the distinction of the obtained maps from a stacked autoencoder topographic map (SAETM) method for different emotions. The average classification accuracy was obtained 0.8173 and 0.8037 in the valence and arousal dimensions, respectively. The extracted maps were also ranked by a team of experts compared to common maps. The results of quantitative and qualitative evaluation showed that the obtained map by SAETM has more information than conventional maps.

Musical Emotions Recognition Using Entropy Features and Channel Optimization Based on EEG

The dynamic of music is an important factor to arouse emotional experience, but current research mainly uses short-term artificial stimulus materials, which cannot effectively awaken complex emotions and reflect their dynamic brain response. In this paper, we used three long-term stimulus materials with many dynamic emotions inside: the "Waltz No. 2" containing pleasure and excitement, the "No. 14 Couplets" containing excitement, briskness, and nervousness, and the first movement of "Symphony No. 5 in C minor" containing passion, relaxation, cheerfulness, and nervousness. Approximate entropy (ApEn) and sample entropy (SampEn) were applied to extract the non-linear features of electroencephalogram (EEG) signals under long-term dynamic stimulation, and the K-Nearest Neighbor (KNN) method was used to recognize emotions. Further, a supervised feature vector dimensionality reduction method was proposed. Firstly, the optimal channel set for each subject was obtained by using a particle swarm optimization (PSO) algorithm, and then the number of times to select each channel in the optimal channel set of all subjects was counted. If the number was greater than or equal to the threshold, it was a common channel suitable for all subjects. The recognition results based on the optimal channel set demonstrated that each accuracy of two categories of emotions based on "Waltz No. 2" and three categories of emotions based on "No. 14 Couplets" was generally above 80%, respectively, and the recognition accuracy of four categories based on the first movement of "Symphony No. 5 in C minor" was about 70%. The recognition accuracy based on the common channel set was about 10% lower than that based on the optimal channel set, but not much different from that based on the whole channel set. This result suggested that the common channel could basically reflect the universal features of the whole subjects while realizing feature dimension reduction. The common channels were mainly distributed in the frontal lobe, central region, parietal lobe, occipital lobe, and temporal lobe. The channel number distributed in the frontal lobe was greater than the ones in other regions, indicating that the frontal lobe was the main emotional response region. Brain region topographic map based on the common channel set showed that there were differences in entropy intensity between different brain regions of the same emotion and the same brain region of different emotions. The number of times to select each channel in the optimal channel set of all 30 subjects showed that the principal component channels representing five brain regions were Fp1/F3 in the frontal lobe, CP5 in the central region, Pz in the parietal lobe, O2 in the occipital lobe, and T8 in the temporal lobe, respectively.

Emotion Recognition in a Health Continuum: Comparison of Healthy Adults of Advancing Age, Community Dwelling Adults Bearing Vascular Risk Factors and People Diagnosed with Mild Cognitive Impairment

The identification of basic emotions plays an important role in social relationships and behaviors linked to survival. In neurodegenerative conditions such as Alzheimer's disease (AD), the ability to recognize emotions may already be impaired at early stages of the disease, such as the stage of Mild Cognitive Impairment (MCI). However, as regards vascular pathologies related to cognitive impairment, very little is known about emotion recognition in people bearing vascular risk factors (VRF). Therefore, the aim of the present study was to examine emotion recognition ability in the health continuum "healthy advancing age-advancing age with VRF-MCI". The sample consisted of 106 adults divided in three diagnostic groups; 43 adults with MCI, 41 adults bearing one or more VRF, and 22 healthy controls of advancing age (HC). Since HC were more educated and younger than the other two groups, the age-group and level of educational were taken into account in the statistical analyses. A dynamic visual test was administered to examine recognition of basic emotions and emotionally neutral conditions. The results showed only a significant diagnostic group x educational level interaction as regards total emotion recognition ability, F (4, 28.910) = 4.117 p = 0.004 η2 = 0.166. High educational level seems to contribute to a high-level-emotion-recognition-performance both in healthy adults of advancing age and in adults bearing vascular risk factors. Medium educational level appears to play the same role only in healthy adults. Neither educational level can help MCI people to enhance their significantly lower emotion recognition ability.

EEG-based emotion recognition using hybrid CNN and LSTM classification

Emotions are a mental state that is accompanied by a distinct physiologic rhythm, as well as physical, behavioral, and mental changes. In the latest days, physiological activity has been used to study emotional reactions. This study describes the electroencephalography (EEG) signals, the brain wave pattern, and emotion analysis all of these are interrelated and based on the consequences of human behavior and Post-Traumatic Stress Disorder (PTSD). Post-traumatic stress disorder effects for long-term illness are associated with considerable suffering, impairment, and social/emotional impairment. PTSD is connected to subcortical responses to injury memories, thoughts, and emotions and alterations in brain circuitry. Predominantly EEG signals are the way of examining the electrical potential of the human feelings cum expression for every changing phenomenon that the individual faces. When going through literature there are some lacunae while analyzing emotions. There exist some reliability issues and also masking of real emotional behavior by the victims. Keeping this research gap and hindrance faced by the previous researchers the present study aims to fulfill the requirements, the efforts can be made to overcome this problem, and the proposed automated CNN-LSTM with ResNet-152 algorithm. Compared with the existing techniques, the proposed techniques achieved a higher level of accuracy of 98% by applying the hybrid deep learning algorithm.

Classification of Contrasting Discrete Emotional States Indicated by EEG Based Graph Theoretical Network Measures

The present study shows new findings that reveal the high association between emotional arousal and neuro-functional brain connectivity measures. For this purpose, contrasting discrete emotional states (happiness vs sadness, amusement vs disgust, calmness vs excitement, calmness vs anger, fear vs anger) are classified by using Support Vector Machines (SVMs) driven by Graph Theoretical segregation (clustering coefficients, transitivity, modularity) and integration (global efficiency, local efficiency) measures of the brain network. Emotional EEG data mediated by short duration video film clips is downloaded from publicly available database called DREAMER. Pearson Correlation (PC) and Spearman Correlation have been examined to estimate statistical dependencies between relatively shorter (6 sec) and longer (12 sec) non-overlapped EEG segments across the cortex. Then the corresponding brain connectivity encoded as a graph is transformed into binary numbers with respect to two different thresholds (60%max and mean). Statistical differences between contrasting emotions are obtained by using both one-way Anova tests and step-wise logistic regression modelling in accordance with variables (dependency estimation, segment length, threshold, network measure). Combined integration measures provided the highest classification accuracies (CAs) (75.00% 80.65%) when PC is applied to longer segments in accordance with particular threshold as the mean. The segregation measures also provided useful CAs (74.13% 80.00%), while the combination of both measures did not. The results reveal that discrete emotional states are characterized by balanced network measures even if both segregation and integration measures vary depending on arousal scores of audio-visual stimuli due to neurotransmitter release during video watching.

Emotion Recognition from EEG Signals Using Recurrent Neural Networks

The application of electroencephalogram (EEG)-based emotion recognition (ER) to the brain–computer interface (BCI) has become increasingly popular over the past decade. Emotion recognition systems involve pre-processing and feature extraction, followed by classification. Deep learning has recently been used to classify emotions in BCI systems, and the results have been improved when compared to classic classification approaches. The main objective of this study is to classify the emotions from electroencephalogram signals using variant recurrent neural network architectures. Three architectures are used in this work for the recognition of emotions using EEG signals: RNN (recurrent neural network), LSTM (long short-term memory network), and GRU (gated recurrent unit). The efficiency of these networks, in terms of performance measures was confirmed by experimental data. The experiment was conducted by using the EEG Brain Wave Dataset: Feeling Emotions, and achieved an average accuracy of 95% for RNN, 97% for LSTM, and 96% for GRU for emotion detection problems.

Applying Big Transfer-based classifiers to the DEAP dataset

Affective brain-computer interfaces are a fast-growing area of research. Accurate estimation of emotional states from physiological signals is of great interest to the fields of psychology and human-computer interaction. The DEAP dataset is one of the most popular datasets for emotional classification. In this study we generated heat maps from spectral data within the neurological signals found in the DEAP dataset. To account for the class imbalance within this dataset, we then discarded images belonging to the larger class. We used these images to fine-tune several Big Transfer neural networks for binary classification of arousal, valence, and dominance affective states. Our best classifier was able to achieve greater than 98% accuracy and 990% balanced accuracy in all three classification tasks. We also investigated the effects of this balancing method on our classifiers.

Representation Learning and Pattern Recognition in Cognitive Biometrics: A Survey

Cognitive biometrics is an emerging branch of biometric technology. Recent research has demonstrated great potential for using cognitive biometrics in versatile applications, including biometric recognition and cognitive and emotional state recognition. There is a major need to summarize the latest developments in this field. Existing surveys have mainly focused on a small subset of cognitive biometric modalities, such as EEG and ECG. This article provides a comprehensive review of cognitive biometrics, covering all the major biosignal modalities and applications. A taxonomy is designed to structure the corresponding knowledge and guide the survey from signal acquisition and pre-processing to representation learning and pattern recognition. We provide a unified view of the methodological advances in these four aspects across various biosignals and applications, facilitating interdisciplinary research and knowledge transfer across fields. Furthermore, this article discusses open research directions in cognitive biometrics and proposes future prospects for developing reliable and secure cognitive biometric systems.

EEG-Based Emotion Recognition Using a 2D CNN with Different Kernels

Emotion recognition is receiving significant attention in research on health care and Human-Computer Interaction (HCI). Due to the high correlation with emotion and the capability to affect deceptive external expressions such as voices and faces, Electroencephalogram (EEG) based emotion recognition methods have been globally accepted and widely applied. Recently, great improvements have been made in the development of machine learning for EEG-based emotion detection. However, there are still some major disadvantages in previous studies. Firstly, traditional machine learning methods require extracting features manually which is time-consuming and rely heavily on human experts. Secondly, to improve the model accuracies, many researchers used user-dependent models that lack generalization and universality. Moreover, there is still room for improvement in the recognition accuracies in most studies. Therefore, to overcome these shortcomings, an EEG-based novel deep neural network is proposed for emotion classification in this article. The proposed 2D CNN uses two convolutional kernels of different sizes to extract emotion-related features along both the time direction and the spatial direction. To verify the feasibility of the proposed model, the pubic emotion dataset DEAP is used in experiments. The results show accuracies of up to 99.99% and 99.98 for arousal and valence binary classification, respectively, which are encouraging for research and applications in the emotion recognition field.

Deep Learning Methods for Multi-Channel EEG-Based Emotion Recognition

Currently, Fourier-based, wavelet-based, and Hilbert-based time–frequency techniques have generated considerable interest in classification studies for emotion recognition in human–computer interface investigations. Empirical mode decomposition (EMD), one of the Hilbert-based time–frequency techniques, has been developed as a tool for adaptive signal processing. Additionally, the multi-variate version strongly influences designing the common oscillation structure of a multi-channel signal by utilizing the common instantaneous concepts of frequency and bandwidth. Additionally, electroencephalographic (EEG) signals are strongly preferred for comprehending emotion recognition perspectives in human–machine interactions. This study aims to herald an emotion detection design via EEG signal decomposition using multi-variate empirical mode decomposition (MEMD). For emotion recognition, the SJTU emotion EEG dataset (SEED) is classified using deep learning methods. Convolutional neural networks (AlexNet, DenseNet-201, ResNet-101, and ResNet50) and AutoKeras architectures are selected for image classification. The proposed framework reaches 99% and 100% classification accuracy when transfer learning methods and the AutoKeras method are used, respectively.

Emotion recognition from EEG based on multi-task learning with capsule network and attention mechanism

Deep learning (DL) technologies have recently shown great potential in emotion recognition based on electroencephalography (EEG). However, existing DL-based EEG emotion recognition methods are built on single-task learning, i.e., learning arousal, valence, and dominance individually, which may ignore the complementary information of different tasks. In addition, single-task learning involves a new round of training every time a new task appears, which is time consuming. To this end, we propose a novel method for EEG-based emotion recognition based on multi-task learning with capsule network (CapsNet) and attention mechanism. First, multi-task learning can learn multiple tasks simultaneously while exploiting commonalities and differences across tasks, it can also obtain more data from different tasks, which can improve generalization and robustness. Second, the innovative structure of the CapsNet enables it to effectively characterize the intrinsic relationship among various EEG channels. Finally, the attention mechanism can change the weight of different channels to extract important information. In the DEAP dataset, the average accuracy reached 97.25%, 97.41%, and 98.35% on arousal, valence, and dominance, respectively. In the DREAMER dataset, average accuracy reached 94.96%, 95.54%, and 95.52% on arousal, valence, and dominance, respectively. Experimental results demonstrate the efficiency of the proposed method for EEG emotion recognition.

Two-dimensional CNN-based distinction of human emotions from EEG channels selected by multi-objective evolutionary algorithm

In this study we explore how different levels of emotional intensity (Arousal) and pleasantness (Valence) are reflected in electroencephalographic (EEG) signals. We performed the experiments on EEG data of 32 subjects from the DEAP public dataset, where the subjects were stimulated using 60-s videos to elicitate different levels of Arousal/Valence and then self-reported the rating from 1 to 9 using the self-assessment Manikin (SAM). The EEG data was pre-processed and used as input to a convolutional neural network (CNN). First, the 32 EEG channels were used to compute the maximum accuracy level obtainable for each subject as well as for creating a single model using data from all the subjects. The experiment was repeated using one channel at a time, to see if specific channels contain more information to discriminate between low vs high arousal/valence. The results indicate than using one channel the accuracy is lower compared to using all the 32 channels. An optimization process for EEG channel selection is then designed with the Non-dominated Sorting Genetic Algorithm II (NSGA-II) with the objective to obtain optimal channel combinations with high accuracy recognition. The genetic algorithm evaluates all possible combinations using a chromosome representation for all the 32 channels, and the EEG data from each chromosome in the different populations are tested iteratively solving two unconstrained objectives; to maximize classification accuracy and to reduce the number of required EEG channels for the classification process. Best combinations obtained from a Pareto-front suggests that as few as 8–10 channels can fulfill this condition and provide the basis for a lighter design of EEG systems for emotion recognition. In the best case, the results show accuracies of up to 1.00 for low vs high arousal using eight EEG channels, and 1.00 for low vs high valence using only two EEG channels. These results are encouraging for research and healthcare applications that will require automatic emotion recognition with wearable EEG.