Recognition of Emotional States Using Multiscale Information Analysis of High Frequency EEG Oscillations

EEG Dataset for the Recognition of Different Emotions Induced in Voice-User Interaction

Electroencephalography (EEG)-based open-access datasets are available for emotion recognition studies, where external auditory/visual stimuli are used to artificially evoke pre-defined emotions. In this study, we provide a novel EEG dataset containing the emotional information induced during a realistic human-computer interaction (HCI) using a voice user interface system that mimics natural human-to-human communication. To validate our dataset via neurophysiological investigation and binary emotion classification, we applied a series of signal processing and machine learning methods to the EEG data. The maximum classification accuracy ranged from 43.3% to 90.8% over 38 subjects and classification features could be interpreted neurophysiologically. Our EEG data could be used to develop a reliable HCI system because they were acquired in a natural HCI environment. In addition, auxiliary physiological data measured simultaneously with the EEG data also showed plausible results, i.e., electrocardiogram, photoplethysmogram, galvanic skin response, and facial images, which could be utilized for automatic emotion discrimination independently from, as well as together with the EEG data via the fusion of multi-modal physiological datasets.

A novel methodology for emotion recognition through 62-lead EEG signals: multilevel heterogeneous recurrence analysis

Objective

Recognizing emotions from electroencephalography (EEG) signals is a challenging task due to the complex, nonlinear, and nonstationary characteristics of brain activity. Traditional methods often fail to capture these subtle dynamics, while deep learning approaches lack explainability. In this research, we introduce a novel three-phase methodology integrating manifold embedding, multilevel heterogeneous recurrence analysis (MHRA), and ensemble learning to address these limitations in EEG-based emotion recognition.

Approach

The proposed methodology was evaluated using the SJTU-SEED IV database. We first applied uniform manifold approximation and projection (UMAP) for manifold embedding of the 62-lead EEG signals into a lower-dimensional space. We then developed MHRA to characterize the complex recurrence dynamics of brain activity across multiple transition levels. Finally, we employed tree-based ensemble learning methods to classify four emotions (neutral, sad, fear, happy) based on the extracted MHRA features.

Main results

Our approach achieved high performance, with an accuracy of 0.7885 and an AUC of 0.7552, outperforming existing methods on the same dataset. Additionally, our methodology provided the most consistent recognition performance across different emotions. Sensitivity analysis revealed specific MHRA metrics that were strongly associated with each emotion, offering valuable insights into the underlying neural dynamics.

Significance

This study presents a novel framework for EEG-based emotion recognition that effectively captures the complex nonlinear and nonstationary dynamics of brain activity while maintaining explainability. The proposed methodology offers significant potential for advancing our understanding of emotional processing and developing more reliable emotion recognition systems with broad applications in healthcare and beyond.

Positional multi-length and mutual-attention network for epileptic seizure classification

The automatic classification of epilepsy electroencephalogram (EEG) signals plays a crucial role in diagnosing neurological diseases. Although promising results have been achieved by deep learning methods in this task, capturing the minute abnormal characteristics, contextual information, and long dependencies of EEG signals remains a challenge. To address this challenge, a positional multi-length and mutual-attention (PMM) network is proposed for the automatic classification of epilepsy EEG signals. The PMM network incorporates a positional feature encoding process that extracts minute abnormal characteristics from the EEG signal and utilizes a multi-length feature learning process with a hierarchy residual dilated LSTM (RDLSTM) to capture long contextual dependencies. Furthermore, a mutual-attention feature reinforcement process is employed to learn the global and relative feature dependencies and enhance the discriminative abilities of the network. To validate the effectiveness PMM network, we conduct extensive experiments on the public dataset and the experimental results demonstrate the superior performance of the PMM network compared to state-of-the-art methods.

Neural dynamics underlying the processing of implicit form-meaning connections: The dissociative roles of theta and alpha oscillations

Implicit learning plays an important role in the language acquisition. In addition to helping people acquire the form-level rules (e.g., the word order regularities), implicit learning can also facilitate the acquisition of word meanings (i.e., the establishment of connections between the word form and its meanings). Although some behavioral studies have explored the processing of implicit form-meaning connections, the neural dynamics underlying this processing remains unclear. Through examining whether participants could implicitly acquire the literal and metaphorical meanings of novel words, and applying the time-frequency analysis on the electroencephalogram (EEG) data collected in the testing phase, the neural oscillations corresponding to the processing of implicit form-literal and form-metaphorical meaning connections were explored. The results showed that participants in the experimental group could implicitly acquire the form-literal and form-metaphorical meaning connections after training, while participants in the control group who were not trained did not have access to such form-meaning connections. Meanwhile, during the processing of form-literal meaning connections, the greater suppression of alpha oscillations was induced by the testing items that follow the same rules as the training items (i.e., the regular testing items) in the experimental group, whereas the stronger enhancement of theta oscillations was elicited by the regular testing items in the experimental group during the processing of form-metaphorical meaning connections. Our study provides insights for understanding the processing of implicit form-literal and form-metaphorical meaning connections and the neural dynamics underlying the processing.

Classification of emotional stress and physical stress using a multispectral based deep feature extraction model

A classification model (Stress Classification-Net) of emotional stress and physical stress is proposed, which can extract classification features based on multispectral and tissue blood oxygen saturation (StO2) characteristics. Related features are extracted on this basis, and the learning model with frequency domain and signal amplification is proposed for the first time. Given that multispectral imaging signals are time series data, time series StO2 is extracted from spectral signals. The proper region of interest (ROI) is obtained by a composite criterion, and the ROI source is determined by the universality and robustness of the signal. The frequency-domain signals of ROI are further obtained by wavelet transform. To fully utilize the frequency-domain characteristics, the multi-neighbor vector of locally aggregated descriptors (MN-VLAD) model is proposed to extract useful features. The acquired time series features are finally put into the long short-term memory (LSTM) model to learn the classification characteristics. Through SC-NET model, the classification signals of emotional stress and physical stress are successfully obtained. Experiments show that the classification result is encouraging, and the accuracy of the proposed algorithm is over 90%.

EEG Emotion Recognition Applied to the Effect Analysis of Music on Emotion Changes in Psychological Healthcare

Music therapy is increasingly being used to promote physical health. Emotion semantic recognition is more objective and provides direct awareness of the real emotional state based on electroencephalogram (EEG) signals. Therefore, we proposed a music therapy method to carry out emotion semantic matching between the EEG signal and music audio signal, which can improve the reliability of emotional judgments, and, furthermore, deeply mine the potential influence correlations between music and emotions. Our proposed EER model (EEG-based Emotion Recognition Model) could identify 20 types of emotions based on 32 EEG channels, and the average recognition accuracy was above 90% and 80%, respectively. Our proposed music-based emotion classification model (MEC model) could classify eight typical emotion types of music based on nine music feature combinations, and the average classification accuracy was above 90%. In addition, the semantic mapping was analyzed according to the influence of different music types on emotional changes from different perspectives based on the two models, and the results showed that the joy type of music video could improve fear, disgust, mania, and trust emotions into surprise or intimacy emotions, while the sad type of music video could reduce intimacy to the fear emotion.

Multidimensional Feature in Emotion Recognition Based on Multi-Channel EEG Signals

As a major daily task for the popularization of artificial intelligence technology, more and more attention has been paid to the scientific research of mental state electroencephalogram (EEG) in recent years. To retain the spatial information of EEG signals and fully mine the EEG timing-related information, this paper proposes a novel EEG emotion recognition method. First, to obtain the frequency, spatial, and temporal information of multichannel EEG signals more comprehensively, we choose the multidimensional feature structure as the input of the artificial neural network. Then, a neural network model based on depthwise separable convolution is proposed, extracting the input structure's frequency and spatial features. The network can effectively reduce the computational parameters. Finally, we modeled using the ordered neuronal long short-term memory (ON-LSTM) network, which can automatically learn hierarchical information to extract deep emotional features hidden in EEG time series. The experimental results show that the proposed model can reasonably learn the correlation and temporal dimension information content between EEG multi-channel and improve emotion classification performance. We performed the experimental validation of this paper in two publicly available EEG emotional datasets. In the experiments on the DEAP dataset (a dataset for emotion analysis using EEG, physiological, and video signals), the mean accuracy of emotion recognition for arousal and valence is 95.02% and 94.61%, respectively. In the experiments on the SEED dataset (a dataset collection for various purposes using EEG signals), the average accuracy of emotion recognition is 95.49%.

Decoding the neural signatures of valence and arousal from portable EEG headset

Emotion classification using electroencephalography (EEG) data and machine learning techniques have been on the rise in the recent past. However, past studies use data from medical-grade EEG setups with long set-up times and environment constraints. This paper focuses on classifying emotions on the valence-arousal plane using various feature extraction, feature selection, and machine learning techniques. We evaluate different feature extraction and selection techniques and propose the optimal set of features and electrodes for emotion recognition. The images from the OASIS image dataset were used to elicit valence and arousal emotions, and the EEG data was recorded using the Emotiv Epoc X mobile EEG headset. The analysis is carried out on publicly available datasets: DEAP and DREAMER for benchmarking. We propose a novel feature ranking technique and incremental learning approach to analyze performance dependence on the number of participants. Leave-one-subject-out cross-validation was carried out to identify subject bias in emotion elicitation patterns. The importance of different electrode locations was calculated, which could be used for designing a headset for emotion recognition. The collected dataset and pipeline are also published. Our study achieved a root mean square score (RMSE) of 0.905 on DREAMER, 1.902 on DEAP, and 2.728 on our dataset for valence label and a score of 0.749 on DREAMER, 1.769 on DEAP, and 2.3 on our proposed dataset for arousal label.

A Review of AI Cloud and Edge Sensors, Methods, and Applications for the Recognition of Emotional, Affective and Physiological States

Affective, emotional, and physiological states (AFFECT) detection and recognition by capturing human signals is a fast-growing area, which has been applied across numerous domains. The research aim is to review publications on how techniques that use brain and biometric sensors can be used for AFFECT recognition, consolidate the findings, provide a rationale for the current methods, compare the effectiveness of existing methods, and quantify how likely they are to address the issues/challenges in the field. In efforts to achieve the key goals of Society 5.0, Industry 5.0, and human-centered design better, the recognition of emotional, affective, and physiological states is progressively becoming an important matter and offers tremendous growth of knowledge and progress in these and other related fields. In this research, a review of AFFECT recognition brain and biometric sensors, methods, and applications was performed, based on Plutchik's wheel of emotions. Due to the immense variety of existing sensors and sensing systems, this study aimed to provide an analysis of the available sensors that can be used to define human AFFECT, and to classify them based on the type of sensing area and their efficiency in real implementations. Based on statistical and multiple criteria analysis across 169 nations, our outcomes introduce a connection between a nation's success, its number of Web of Science articles published, and its frequency of citation on AFFECT recognition. The principal conclusions present how this research contributes to the big picture in the field under analysis and explore forthcoming study trends.

Evaluation of Brain Functional Connectivity from Electroencephalographic Signals Under Different Emotional States

The identification of the emotional states corresponding to the four quadrants of the valence/arousal space has been widely analyzed in the scientific literature by means of multiple techniques. Nevertheless, most of these methods were based on the assessment of each brain region separately, without considering the possible interactions among different areas. In order to study these interconnections, this study computes for the first time the functional connectivity metric called cross-sample entropy for the analysis of the brain synchronization in four groups of emotions from electroencephalographic signals. Outcomes reported a strong synchronization in the interconnections among central, parietal and occipital areas, while the interactions between left frontal and temporal structures with the rest of brain regions presented the lowest coordination. These differences were statistically significant for the four groups of emotions. All emotions were simultaneously classified with a 95.43% of accuracy, overcoming the results reported in previous studies. Moreover, the differences between high and low levels of valence and arousal, taking into account the state of the counterpart dimension, also provided notable findings about the degree of synchronization in the brain within different emotional conditions and the possible implications of these outcomes from a psychophysiological point of view.

Signal Quality Investigation of a New Wearable Frontal Lobe EEG Device

The demand for non-laboratory and long-term EEG acquisition in scientific and clinical applications has put forward new requirements for wearable EEG devices. In this paper, a new wearable frontal EEG device called Mindeep was proposed. A signal quality study was then conducted, which included simulated signal tests and signal quality comparison experiments. Simulated signals with different frequencies and amplitudes were used to test the stability of Mindeep’s circuit, and the high correlation coefficients (>0.9) proved that Mindeep has a stable and reliable hardware circuit. The signal quality comparison experiment, between Mindeep and the gold standard device, Neuroscan, included three tasks: (1) resting; (2) auditory oddball; and (3) attention. In the resting state, the average normalized cross-correlation coefficients between EEG signals recorded by the two devices was around 0.72 ± 0.02, Berger effect was observed (p < 0.01), and the comparison results in the time and frequency domain illustrated the ability of Mindeep to record high-quality EEG signals. The significant differences between high tone and low tone in auditory event-related potential collected by Mindeep was observed in N2 and P2. The attention recognition accuracy of Mindeep achieved 71.12% and 74.76% based on EEG features and the XGBoost model in the two attention tasks, respectively, which were higher than that of Neuroscan (70.19% and 72.80%). The results validated the performance of Mindeep as a prefrontal EEG recording device, which has a wide range of potential applications in audiology, cognitive neuroscience, and daily requirements.

A Review of Methods of Diagnosis and Complexity Analysis of Alzheimer's Disease Using EEG Signals

This study will concentrate on recent research on EEG signals for Alzheimer's diagnosis, identifying and comparing key steps of EEG-based Alzheimer's disease (AD) detection, such as EEG signal acquisition, preprocessing function extraction, and classification methods. Furthermore, highlighting general approaches, variations, and agreement in the use of EEG identified shortcomings and guidelines for multiple experimental stages ranging from demographic characteristics to outcomes monitoring for future research. Two main targets have been defined based on the article's purpose: (1) discriminative (or detection), i.e., look for differences in EEG-based features across groups, such as MCI, moderate Alzheimer's disease, extreme Alzheimer's disease, other forms of dementia, and stable normal elderly controls; and (2) progression determination, i.e., look for correlations between EEG-based features and clinical markers linked to MCI-to-AD conversion and Alzheimer's disease intensity progression. Limitations mentioned in the reviewed papers were also gathered and explored in this study, with the goal of gaining a better understanding of the problems that need to be addressed in order to advance the use of EEG in Alzheimer's disease science.

EEG electrodes selection for emotion recognition independent of stimulus presentation paradigms

Most of the electroencephalography (EEG) based emotion recognition systems rely on single stimulus to evoke emotions. EEG data is mostly recorded with higher number of electrodes that can lead to data redundancy and longer experimental setup time. The question “whether the configuration with lesser number of electrodes is common amongst different stimuli presentation paradigms” remains unanswered. There are publicly available datasets for EEG based human emotional states recognition. Since this work is focused towards classifying emotions while subjects are experiencing different stimuli, therefore we need to perform new experiments. Keeping aforementioned issues in consideration, this work presents a novel experimental study that records EEG data for three different human emotional states evoked with four different stimuli presentation paradigms. A methodology based on iterative Genetic Algorithm in combination with majority voting has been used to achieve configuration with reduced number of EEG electrodes keeping in consideration minimum loss of classification accuracy. The results obtained are comparable with recent studies. Stimulus independent configurations with lesser number of electrodes lead towards low computational complexity as well as reduced set up time for future EEG based smart systems for emotions recognition

EEG-Based Emotion Recognition by Exploiting Fused Network Entropy Measures of Complex Networks across Subjects

It is well known that there may be significant individual differences in physiological signal patterns for emotional responses. Emotion recognition based on electroencephalogram (EEG) signals is still a challenging task in the context of developing an individual-independent recognition method. In our paper, from the perspective of spatial topology and temporal information of brain emotional patterns in an EEG, we exploit complex networks to characterize EEG signals to effectively extract EEG information for emotion recognition. First, we exploit visibility graphs to construct complex networks from EEG signals. Then, two kinds of network entropy measures (nodal degree entropy and clustering coefficient entropy) are calculated. By applying the AUC method, the effective features are input into the SVM classifier to perform emotion recognition across subjects. The experiment results showed that, for the EEG signals of 62 channels, the features of 18 channels selected by AUC were significant (p < 0.005). For the classification of positive and negative emotions, the average recognition rate was 87.26%; for the classification of positive, negative, and neutral emotions, the average recognition rate was 68.44%. Our method improves mean accuracy by an average of 2.28% compared with other existing methods. Our results fully demonstrate that a more accurate recognition of emotional EEG signals can be achieved relative to the available relevant studies, indicating that our method can provide more generalizability in practical use.

Frontal EEG-Based Multi-Level Attention States Recognition Using Dynamical Complexity and Extreme Gradient Boosting

Measuring and identifying the specific level of sustained attention during continuous tasks is essential in many applications, especially for avoiding the terrible consequences caused by reduced attention of people with special tasks. To this end, we recorded EEG signals from 42 subjects during the performance of a sustained attention task and obtained resting state and three levels of attentional states using the calibrated response time. EEG-based dynamical complexity features and Extreme Gradient Boosting (XGBoost) classifier were proposed as the classification model, Complexity-XGBoost, to distinguish multi-level attention states with improved accuracy. The maximum average accuracy of Complexity-XGBoost were 81.39 ± 1.47% for four attention levels, 80.42 ± 0.84% for three attention levels, and 95.36 ± 2.31% for two attention levels in 5-fold cross-validation. The proposed method is compared with other models of traditional EEG features and different classification algorithms, the results confirmed the effectiveness of the proposed method. We also found that the frontal EEG dynamical complexity measures were related to the changing process of response during sustained attention task. The proposed dynamical complexity approach could be helpful to recognize attention status during important tasks to improve safety and efficiency, and be useful for further brain-computer interaction research in clinical research or daily practice, such as the cognitive assessment or neural feedback treatment of individuals with attention deficit hyperactivity disorders, Alzheimer’s disease, and other diseases which affect the sustained attention function.

Comparing Neural Correlates of Human Emotions across Multiple Stimulus Presentation Paradigms

Most electroencephalography (EEG)-based emotion recognition systems rely on a single stimulus to evoke emotions. These systems make use of videos, sounds, and images as stimuli. Few studies have been found for self-induced emotions. The question "if different stimulus presentation paradigms for same emotion, produce any subject and stimulus independent neural correlates" remains unanswered. Furthermore, we found that there are publicly available datasets that are used in a large number of studies targeting EEG-based human emotional state recognition. Since one of the major concerns and contributions of this work is towards classifying emotions while subjects experience different stimulus-presentation paradigms, we need to perform new experiments. This paper presents a novel experimental study that recorded EEG data for three different human emotional states evoked with four different stimuli presentation paradigms. Fear, neutral, and joy have been considered as three emotional states. In this work, features were extracted with common spatial pattern (CSP) from recorded EEG data and classified through linear discriminant analysis (LDA). The considered emotion-evoking paradigms included emotional imagery, pictures, sounds, and audio-video movie clips. Experiments were conducted with twenty-five participants. Classification performance in different paradigms was evaluated, considering different spectral bands. With a few exceptions, all paradigms showed the best emotion recognition for higher frequency spectral ranges. Interestingly, joy emotions were classified more strongly as compared to fear. The average neural patterns for fear vs. joy emotional states are presented with topographical maps based on spatial filters obtained with CSP for averaged band power changes for all four paradigms. With respect to the spectral bands, beta and alpha oscillation responses produced the highest number of significant results for the paradigms under consideration. With respect to brain region, the frontal lobe produced the most significant results irrespective of paradigms and spectral bands. The temporal site also played an effective role in generating statistically significant findings. To the best of our knowledge, no study has been conducted for EEG emotion recognition while considering four different stimuli paradigms. This work provides a good contribution towards designing EEG-based system for human emotion recognition that could work effectively in different real-time scenarios.

Dynamic Joint Domain Adaptation Network for Motor Imagery Classification

Electroencephalogram (EEG) has been widely used in brain computer interface (BCI) due to its convenience and reliability. The EEG-based BCI applications are majorly limited by the time-consuming calibration procedure for discriminative feature representation and classification. Existing EEG classification methods either heavily depend on the handcrafted features or require adequate annotated samples at each session for calibration. To address these issues, we propose a novel dynamic joint domain adaptation network based on adversarial learning strategy to learn domain-invariant feature representation, and thus improve EEG classification performance in the target domain by leveraging useful information from the source session. Specifically, we explore the global discriminator to align the marginal distribution across domains, and the local discriminator to reduce the conditional distribution discrepancy between sub-domains via conditioning on deep representation as well as the predicted labels from the classifier. In addition, we further investigate a dynamic adversarial factor to adaptively estimate the relative importance of alignment between the marginal and conditional distributions. To evaluate the efficacy of our method, extensive experiments are conducted on two public EEG datasets, namely, Datasets IIa and IIb of BCI Competition IV. The experimental results demonstrate that the proposed method achieves superior performance compared with the state-of-the-art methods.

The Role of Features Types and Personalized Assessment in Detecting Affective State Using Dry Electrode EEG

Assessing the human affective state using electroencephalography (EEG) have shown good potential but failed to demonstrate reliable performance in real-life applications. Especially if one applies a setup that might impact affective processing and relies on generalized models of affect. Additionally, using subjective assessment of ones affect as ground truth has often been disputed. To shed the light on the former challenge we explored the use of a convenient EEG system with 20 participants to capture their reaction to affective movie clips in a naturalistic setting. Employing state-of-the-art machine learning approach demonstrated that the highest performance is reached when combining linear features, namely symmetry features and single-channel features, with nonlinear ones derived by a multiscale entropy approach. Nevertheless, the best performance, reflected in the highest F1-score achieved in a binary classification task for valence was 0.71 and for arousal 0.62. The performance was 10–20% better compared to using ratings provided by 13 independent raters. We argue that affective self-assessment might be underrated and it is crucial to account for personal differences in both perception and physiological response to affective cues.

Human Emotion Recognition: Review of Sensors and Methods

Automated emotion recognition (AEE) is an important issue in various fields of activities which use human emotional reactions as a signal for marketing, technical equipment, or human-robot interaction. This paper analyzes scientific research and technical papers for sensor use analysis, among various methods implemented or researched. This paper covers a few classes of sensors, using contactless methods as well as contact and skin-penetrating electrodes for human emotion detection and the measurement of their intensity. The results of the analysis performed in this paper present applicable methods for each type of emotion and their intensity and propose their classification. The classification of emotion sensors is presented to reveal area of application and expected outcomes from each method, as well as their limitations. This paper should be relevant for researchers using human emotion evaluation and analysis, when there is a need to choose a proper method for their purposes or to find alternative decisions. Based on the analyzed human emotion recognition sensors and methods, we developed some practical applications for humanizing the Internet of Things (IoT) and affective computing systems.

Conditional Adversarial Domain Adaptation Neural Network for Motor Imagery EEG Decoding

Decoding motor imagery (MI) electroencephalogram (EEG) signals for brain-computer interfaces (BCIs) is a challenging task because of the severe non-stationarity of perceptual decision processes. Recently, deep learning techniques have had great success in EEG decoding because of their prominent ability to learn features from raw EEG signals automatically. However, the challenge that the deep learning method faces is that the shortage of labeled EEG signals and EEGs sampled from other subjects cannot be used directly to train a convolutional neural network (ConvNet) for a target subject. To solve this problem, in this paper, we present a novel conditional domain adaptation neural network (CDAN) framework for MI EEG signal decoding. Specifically, in the CDAN, a densely connected ConvNet is firstly applied to obtain high-level discriminative features from raw EEG time series. Then, a novel conditional domain discriminator is introduced to work as an adversarial with the label classifier to learn commonly shared intra-subjects EEG features. As a result, the CDAN model trained with sufficient EEG signals from other subjects can be used to classify the signals from the target subject efficiently. Competitive experimental results on a public EEG dataset (High Gamma Dataset) against the state-of-the-art methods demonstrate the efficacy of the proposed framework in recognizing MI EEG signals, indicating its effectiveness in automatic perceptual decision decoding.