EEG-Based BCI Emotion Recognition: A Survey

Two-dimensional CNN-based distinction of human emotions from EEG channels selected by multi-objective evolutionary algorithm

In this study we explore how different levels of emotional intensity (Arousal) and pleasantness (Valence) are reflected in electroencephalographic (EEG) signals. We performed the experiments on EEG data of 32 subjects from the DEAP public dataset, where the subjects were stimulated using 60-s videos to elicitate different levels of Arousal/Valence and then self-reported the rating from 1 to 9 using the self-assessment Manikin (SAM). The EEG data was pre-processed and used as input to a convolutional neural network (CNN). First, the 32 EEG channels were used to compute the maximum accuracy level obtainable for each subject as well as for creating a single model using data from all the subjects. The experiment was repeated using one channel at a time, to see if specific channels contain more information to discriminate between low vs high arousal/valence. The results indicate than using one channel the accuracy is lower compared to using all the 32 channels. An optimization process for EEG channel selection is then designed with the Non-dominated Sorting Genetic Algorithm II (NSGA-II) with the objective to obtain optimal channel combinations with high accuracy recognition. The genetic algorithm evaluates all possible combinations using a chromosome representation for all the 32 channels, and the EEG data from each chromosome in the different populations are tested iteratively solving two unconstrained objectives; to maximize classification accuracy and to reduce the number of required EEG channels for the classification process. Best combinations obtained from a Pareto-front suggests that as few as 8–10 channels can fulfill this condition and provide the basis for a lighter design of EEG systems for emotion recognition. In the best case, the results show accuracies of up to 1.00 for low vs high arousal using eight EEG channels, and 1.00 for low vs high valence using only two EEG channels. These results are encouraging for research and healthcare applications that will require automatic emotion recognition with wearable EEG.

Neural Decoding of EEG Signals with Machine Learning: A Systematic Review

Electroencephalography (EEG) is a non-invasive technique used to record the brain's evoked and induced electrical activity from the scalp. Artificial intelligence, particularly machine learning (ML) and deep learning (DL) algorithms, are increasingly being applied to EEG data for pattern analysis, group membership classification, and brain-computer interface purposes. This study aimed to systematically review recent advances in ML and DL supervised models for decoding and classifying EEG signals. Moreover, this article provides a comprehensive review of the state-of-the-art techniques used for EEG signal preprocessing and feature extraction. To this end, several academic databases were searched to explore relevant studies from the year 2000 to the present. Our results showed that the application of ML and DL in both mental workload and motor imagery tasks has received substantial attention in recent years. A total of 75% of DL studies applied convolutional neural networks with various learning algorithms, and 36% of ML studies achieved competitive accuracy by using a support vector machine algorithm. Wavelet transform was found to be the most common feature extraction method used for all types of tasks. We further examined the specific feature extraction methods and end classifier recommendations discovered in this systematic review.

EEG-Based Emotion Recognition for Modulating Social-Aware Robot Navigation

Companion robots play an important role to accompany humans and provide emotional support, such as reducing human social isolation and loneliness. Based on recognizing human partner's mental states, a companion robot is able to dynamically adjust its behaviors, and make human-robot interaction smoother and natural. Human emotion has been recognized by many modalities like facial expression and voice. Neurophysiological signals have shown promising results in emotion recognition, since it is an innate signal of human brain which cannot be faked. In this paper, emotional state recognition using a neurophysiology method is studied to guide and modulate companion-robot navigation to enhance its social capabilities. Electroencephalogram (EEG), a type of neurophysiological signals, is used to recognize human emotional state, and then feed into a navigation path planning algorithm for controlling a companion robot's routes. Simulation results show that mobile robot presents navigation behaviors modulated by dynamic human emotional states.

A Review of the Role of Machine Learning Techniques towards Brain–Computer Interface Applications

This review article provides a deep insight into the Brain–Computer Interface (BCI) and the application of Machine Learning (ML) technology in BCIs. It investigates the various types of research undertaken in this realm and discusses the role played by ML in performing different BCI tasks. It also reviews the ML methods used for mental state detection, mental task categorization, emotion classification, electroencephalogram (EEG) signal classification, event-related potential (ERP) signal classification, motor imagery categorization, and limb movement classification. This work explores the various methods employed in BCI mechanisms for feature extraction, selection, and classification and provides a comparative study of reviewed methods. This paper assists the readers to gain information regarding the developments made in BCI and ML domains and future improvements needed for improving and designing better BCI applications.

EEG Correlates of Middle Eastern Music Improvisations on the Ney Instrument

The cognitive sciences have witnessed a growing interest in cognitive and neural basis of human creativity. Music improvisations constitute an ideal paradigm to study creativity, but the underlying cognitive processes remain poorly understood. In addition, studies on music improvisations using scales other than the major and minor chords are scarce. Middle Eastern Music is characterized by the additional use of microtones, resulting in a tonal–spatial system called Maqam. No EEG correlates have been proposed yet for the eight most commonly used maqams. The Ney, an end-blown flute that is popular and widely used in the Middle East was used by a professional musician to perform 24 improvisations at low, medium, and high tempos. Using the EMOTIV EPOC+, a 14-channel wireless EEG headset, brainwaves were recorded and quantified before and during improvisations. Pairwise comparisons were calculated using IBM-SPSS and a principal component analysis was used to evaluate the variability between the maqams. A significant increase of low frequency bands theta power and alpha power were observed at the frontal left and temporal left area as well as a significant increase in higher frequency bands beta-high bands and gamma at the right temporal and left parietal area. This study reveals the first EEG observations of the eight most commonly used maqam and is proposing EEG signatures for various maqams.

Positive and Negative Emotion Classification Based on Multi-channel

The EEG features of different emotions were extracted based on multi-channel and forehead channels in this study. The EEG signals of 26 subjects were collected by the emotional video evoked method. The results show that the energy ratio and differential entropy of the frequency band can be used to classify positive and negative emotions effectively, and the best effect can be achieved by using an SVM classifier. When only the forehead and forehead signals are used, the highest classification accuracy can reach 66%. When the data of all channels are used, the highest accuracy of the model can reach 82%. After channel selection, the best model of this study can be obtained. The accuracy is more than 86%.

Brain-Computer Interface: Advancement and Challenges

Brain-Computer Interface (BCI) is an advanced and multidisciplinary active research domain based on neuroscience, signal processing, biomedical sensors, hardware, etc. Since the last decades, several groundbreaking research has been conducted in this domain. Still, no comprehensive review that covers the BCI domain completely has been conducted yet. Hence, a comprehensive overview of the BCI domain is presented in this study. This study covers several applications of BCI and upholds the significance of this domain. Then, each element of BCI systems, including techniques, datasets, feature extraction methods, evaluation measurement matrices, existing BCI algorithms, and classifiers, are explained concisely. In addition, a brief overview of the technologies or hardware, mostly sensors used in BCI, is appended. Finally, the paper investigates several unsolved challenges of the BCI and explains them with possible solutions.

Bridging the gap between emotion and joint action

Our daily human life is filled with a myriad of joint action moments, be it children playing, adults working together (i.e., team sports), or strangers navigating through a crowd. Joint action brings individuals (and embodiment of their emotions) together, in space and in time. Yet little is known about how individual emotions propagate through embodied presence in a group, and how joint action changes individual emotion. In fact, the multi-agent component is largely missing from neuroscience-based approaches to emotion, and reversely joint action research has not found a way yet to include emotion as one of the key parameters to model socio-motor interaction. In this review, we first identify the gap and then stockpile evidence showing strong entanglement between emotion and acting together from various branches of sciences. We propose an integrative approach to bridge the gap, highlight five research avenues to do so in behavioral neuroscience and digital sciences, and address some of the key challenges in the area faced by modern societies.

EEG-Based Emotion Recognition by Convolutional Neural Network with Multi-Scale Kernels

Besides facial or gesture-based emotion recognition, Electroencephalogram (EEG) data have been drawing attention thanks to their capability in countering the effect of deceptive external expressions of humans, like faces or speeches. Emotion recognition based on EEG signals heavily relies on the features and their delineation, which requires the selection of feature categories converted from the raw signals and types of expressions that could display the intrinsic properties of an individual signal or a group of them. Moreover, the correlation or interaction among channels and frequency bands also contain crucial information for emotional state prediction, and it is commonly disregarded in conventional approaches. Therefore, in our method, the correlation between 32 channels and frequency bands were put into use to enhance the emotion prediction performance. The extracted features chosen from the time domain were arranged into feature-homogeneous matrices, with their positions following the corresponding electrodes placed on the scalp. Based on this 3D representation of EEG signals, the model must have the ability to learn the local and global patterns that describe the short and long-range relations of EEG channels, along with the embedded features. To deal with this problem, we proposed the 2D CNN with different kernel-size of convolutional layers assembled into a convolution block, combining features that were distributed in small and large regions. Ten-fold cross validation was conducted on the DEAP dataset to prove the effectiveness of our approach. We achieved the average accuracies of 98.27% and 98.36% for arousal and valence binary classification, respectively.

Optimized Projection and Fisher Discriminative Dictionary Learning for EEG Emotion Recognition

Electroencephalogram (EEG)-based emotion recognition (ER) has drawn increasing attention in the brain–computer interface (BCI) due to its great potentials in human–machine interaction applications. According to the characteristics of rhythms, EEG signals usually can be divided into several different frequency bands. Most existing methods concatenate multiple frequency band features together and treat them as a single feature vector. However, it is often difficult to utilize band-specific information in this way. In this study, an optimized projection and Fisher discriminative dictionary learning (OPFDDL) model is proposed to efficiently exploit the specific discriminative information of each frequency band. Using subspace projection technology, EEG signals of all frequency bands are projected into a subspace. The shared dictionary is learned in the projection subspace such that the specific discriminative information of each frequency band can be utilized efficiently, and simultaneously, the shared discriminative information among multiple bands can be preserved. In particular, the Fisher discrimination criterion is imposed on the atoms to minimize within-class sparse reconstruction error and maximize between-class sparse reconstruction error. Then, an alternating optimization algorithm is developed to obtain the optimal solution for the projection matrix and the dictionary. Experimental results on two EEG-based ER datasets show that this model can achieve remarkable results and demonstrate its effectiveness.

WeDea: A New EEG-based Framework for Emotion Recognition

With the development of sensing technologies and machine learning, techniques that can identify emotions and inner states of a human through physiological signals, known as electroencephalography (EEG), have been actively developed and applied to various domains, such as automobiles, robotics, healthcare, and customer-support services. Thus, the demand for acquiring and analyzing EEG signals in real-time is increasing. In this paper, we aimed to acquire a new EEG dataset based on the discrete emotion theory, termed as WeDea (Wireless-based eeg Data for emotion analysis), and propose a new combination for WeDea analysis. For the collected WeDea dataset, we used video clips as emotional stimulants that were selected by 15 volunteers. Consequently, WeDea is a multi-way dataset measured while 30 subjects are watching the selected 79 video clips under five different emotional states using a convenient portable headset device. Furthermore, we designed a framework for recognizing human emotional state using this new database. The practical results for different types of emotions have proven that WeDea is a promising resource for emotion analysis and can be applied to the field of neuroscience.

A Novel User Emotional Interaction Design Model Using Long and Short-Term Memory Networks and Deep Learning

Emotional design is an important development trend of interaction design. Emotional design in products plays a key role in enhancing user experience and inducing user emotional resonance. In recent years, based on the user's emotional experience, the design concept of strengthening product emotional design has become a new direction for most designers to improve their design thinking. In the emotional interaction design, the machine needs to capture the user's key information in real time, recognize the user's emotional state, and use a variety of clues to finally determine the appropriate user model. Based on this background, this research uses a deep learning mechanism for more accurate and effective emotion recognition, thereby optimizing the design of the interactive system and improving the user experience. First of all, this research discusses how to use user characteristics such as speech, facial expression, video, heartbeat, etc., to make machines more accurately recognize human emotions. Through the analysis of various characteristics, the speech is selected as the experimental material. Second, a speech-based emotion recognition method is proposed. The mel-Frequency cepstral coefficient (MFCC) of the speech signal is used as the input of the improved long and short-term memory network (ILSTM). To ensure the integrity of the information and the accuracy of the output at the next moment, ILSTM makes peephole connections in the forget gate and input gate of LSTM, and adds the unit state as input data to the threshold layer. The emotional features obtained by ILSTM are input into the attention layer, and the self-attention mechanism is used to calculate the weight of each frame of speech signal. The speech features with higher weights are used to distinguish different emotions and complete the emotion recognition of the speech signal. Experiments on the EMO-DB and CASIA datasets verify the effectiveness of the model for emotion recognition. Finally, the feasibility of emotional interaction system design is discussed.

Learning Subject-Generalized Topographical EEG Embeddings Using Deep Variational Autoencoders and Domain-Adversarial Regularization

Two of the biggest challenges in building models for detecting emotions from electroencephalography (EEG) devices are the relatively small amount of labeled samples and the strong variability of signal feature distributions between different subjects. In this study, we propose a context-generalized model that tackles the data constraints and subject variability simultaneously using a deep neural network architecture optimized for normally distributed subject-independent feature embeddings. Variational autoencoders (VAEs) at the input level allow the lower feature layers of the model to be trained on both labeled and unlabeled samples, maximizing the use of the limited data resources. Meanwhile, variational regularization encourages the model to learn Gaussian-distributed feature embeddings, resulting in robustness to small dataset imbalances. Subject-adversarial regularization applied to the bi-lateral features further enforces subject-independence on the final feature embedding used for emotion classification. The results from subject-independent performance experiments on the SEED and DEAP EEG-emotion datasets show that our model generalizes better across subjects than other state-of-the-art feature embeddings when paired with deep learning classifiers. Furthermore, qualitative analysis of the embedding space reveals that our proposed subject-invariant bi-lateral variational domain adversarial neural network (BiVDANN) architecture may improve the subject-independent performance by discovering normally distributed features.

The thermodynamics of cognition: A mathematical treatment

There is a general expectation that the laws of classical physics must apply to biology, particularly the neural system. The evoked cycle represents the brain's energy/information exchange with the physical environment through stimulus. Therefore, the thermodynamics of emotions might elucidate the neurological origin of intellectual evolution, and explain the psychological and health consequences of positive and negative emotional states based on their energy profiles. We utilized the Carnot cycle and Landauer's principle to analyze the energetic consequences of the brain's resting and evoked states during and after various cognitive states. Namely, positive emotional states can be represented by the reversed Carnot cycle, whereas negative emotional reactions trigger the Carnot cycle. The two conditions have contrasting energetic and entropic aftereffects with consequences for mental energy. The mathematics of the Carnot and reversed Carnot cycles, which can explain recent findings in human psychology, might be constructive in the scientific endeavor in turning psychology into hard science.

Oscillatory EEG Signatures of Affective Processes during Interaction with Adaptive Computer Systems

Affect monitoring is being discussed as a novel strategy to make adaptive systems more user-oriented. Basic knowledge about oscillatory processes and functional connectivity underlying affect during naturalistic human-computer interactions (HCI) is, however, scarce. This study assessed local oscillatory power entrainment and distributed functional connectivity in a close-to-naturalistic HCI-paradigm. Sixteen participants interacted with a simulated assistance system which deliberately evoked positive (supporting goal-achievement) and negative (impeding goal-achievement) affective reactions. Electroencephalography (EEG) was used to examine the reactivity of the cortical system during the interaction by studying both event-related (de-)synchronization (ERD/ERS) and event-related functional coupling of cortical networks towards system-initiated assistance. Significantly higher α-band and β-band ERD in centro-parietal and parieto-occipital regions and β-band ERD in bi-lateral fronto-central regions were observed during impeding system behavior. Supportive system behavior activated significantly higher γ-band ERS in bi-hemispheric parietal-occipital regions. This was accompanied by functional coupling of remote β-band and γ-band activity in the medial frontal, left fronto-central and parietal regions, respectively. Our findings identify oscillatory signatures of positive and negative affective processes as reactions to system-initiated assistance. The findings contribute to the development of EEG-based neuroadaptive assistance loops by suggesting a non-obtrusive method for monitoring affect in HCI.

The Role of Features Types and Personalized Assessment in Detecting Affective State Using Dry Electrode EEG

Assessing the human affective state using electroencephalography (EEG) have shown good potential but failed to demonstrate reliable performance in real-life applications. Especially if one applies a setup that might impact affective processing and relies on generalized models of affect. Additionally, using subjective assessment of ones affect as ground truth has often been disputed. To shed the light on the former challenge we explored the use of a convenient EEG system with 20 participants to capture their reaction to affective movie clips in a naturalistic setting. Employing state-of-the-art machine learning approach demonstrated that the highest performance is reached when combining linear features, namely symmetry features and single-channel features, with nonlinear ones derived by a multiscale entropy approach. Nevertheless, the best performance, reflected in the highest F1-score achieved in a binary classification task for valence was 0.71 and for arousal 0.62. The performance was 10–20% better compared to using ratings provided by 13 independent raters. We argue that affective self-assessment might be underrated and it is crucial to account for personal differences in both perception and physiological response to affective cues.