EEG-Based Emotion Recognition Using Deep Learning and M3GP

Emotion recognition in EEG signals using deep learning methods: A review

Emotions are a critical aspect of daily life and serve a crucial role in human decision-making, planning, reasoning, and other mental states. As a result, they are considered a significant factor in human interactions. Human emotions can be identified through various sources, such as facial expressions, speech, behavior (gesture/position), or physiological signals. The use of physiological signals can enhance the objectivity and reliability of emotion detection. Compared with peripheral physiological signals, electroencephalogram (EEG) recordings are directly generated by the central nervous system and are closely related to human emotions. EEG signals have the great spatial resolution that facilitates the evaluation of brain functions, making them a popular modality in emotion recognition studies. Emotion recognition using EEG signals presents several challenges, including signal variability due to electrode positioning, individual differences in signal morphology, and lack of a universal standard for EEG signal processing. Moreover, identifying the appropriate features for emotion recognition from EEG data requires further research. Finally, there is a need to develop more robust artificial intelligence (AI) including conventional machine learning (ML) and deep learning (DL) methods to handle the complex and diverse EEG signals associated with emotional states. This paper examines the application of DL techniques in emotion recognition from EEG signals and provides a detailed discussion of relevant articles. The paper explores the significant challenges in emotion recognition using EEG signals, highlights the potential of DL techniques in addressing these challenges, and suggests the scope for future research in emotion recognition using DL techniques. The paper concludes with a summary of its findings.

An efficient deep learning framework for P300 evoked related potential detection in EEG signal

BACKGROUND

Incorporating the time-frequency localization properties of Gabor transform (GT), the complexity understandings of convolutional neural network (CNN), and histogram of oriented gradients (HOG) efficacy in distinguishing positive peaks can exhibit their characteristics to reveal an effective solution in the detection of P300 evoked related potential (ERP). By applying a drastic number of convolutional layers, the majority of deep networks elicit sufficient properties for the output determination, leading to gigantic and time-consuming structures. In this paper, we propose a novel deep learning framework by the combination of tuned GT, and modified HOG with the CNN as "TGT-MHOG-CNN" for detection of P300 ERP in EEG signal.

METHOD

In the proposed method, GT is tuned based on triangular function for EEG signals, and then spectrograms including time-frequency information are captured. The function's parameters are justified to differentiate the signals with the P300 component. Furthermore, HOG is modified (MHOG) for the 2-D EEG signal, and consequently, gradients patterns are extracted for the target potentials. MHOG is potent in distinguishing the positive peak in the general waveform; however, GT unravels time-frequency information, which is ignored in the gradient histogram. These outputs of GT and MHOG do not share the same nature in the images nor overlap. Therefore, more extensive information is reached without redundancy or excessive information by fusing them. Combining GT and MHOG provides different patterns which benefit CNN for more precise detection. Consequently, TGT-MHOG-CNN ends in a more straightforward structure than other networks, and therefore, the whole performance is acceptable with faster rates and very high accuracy.

RESULTS

BCI Competition II and III datasets are used to evaluate the performance of the proposed method. These datasets include a complete record for P300 ERP with BCI2000 using a paradigm, and it has numerous noises, including power and muscle-based noises. The objective is to predict the correct character in each provided character selection epochs. Compared to state-of-the-art methods, simulation results indicate striking abilities of the proposed framework for P300 ERP detection. Our best record reached the P300 ERP classification rates of over 98.7% accuracy and 98.7% precision for BCI Competition II and 99% accuracy and 100% precision for BCI Competition III datasets, with superiority in execution time for the mentioned datasets.

M1M2: Deep-Learning-Based Real-Time Emotion Recognition from Neural Activity

Emotion recognition, or the ability of computers to interpret people's emotional states, is a very active research area with vast applications to improve people's lives. However, most image-based emotion recognition techniques are flawed, as humans can intentionally hide their emotions by changing facial expressions. Consequently, brain signals are being used to detect human emotions with improved accuracy, but most proposed systems demonstrate poor performance as EEG signals are difficult to classify using standard machine learning and deep learning techniques. This paper proposes two convolutional neural network (CNN) models (M1: heavily parameterized CNN model and M2: lightly parameterized CNN model) coupled with elegant feature extraction methods for effective recognition. In this study, the most popular EEG benchmark dataset, the DEAP, is utilized with two of its labels, valence, and arousal, for binary classification. We use Fast Fourier Transformation to extract the frequency domain features, convolutional layers for deep features, and complementary features to represent the dataset. The M1 and M2 CNN models achieve nearly perfect accuracy of 99.89% and 99.22%, respectively, which outperform every previous state-of-the-art model. We empirically demonstrate that the M2 model requires only 2 seconds of EEG signal for 99.22% accuracy, and it can achieve over 96% accuracy with only 125 milliseconds of EEG data for valence classification. Moreover, the proposed M2 model achieves 96.8% accuracy on valence using only 10% of the training dataset, demonstrating our proposed system's effectiveness. Documented implementation codes for every experiment are published for reproducibility.

Use of Differential Entropy for Automated Emotion Recognition in a Virtual Reality Environment with EEG Signals

Emotion recognition is one of the most important issues in human–computer interaction (HCI), neuroscience, and psychology fields. It is generally accepted that emotion recognition with neural data such as electroencephalography (EEG) signals, functional magnetic resonance imaging (fMRI), and near-infrared spectroscopy (NIRS) is better than other emotion detection methods such as speech, mimics, body language, facial expressions, etc., in terms of reliability and accuracy. In particular, EEG signals are bioelectrical signals that are frequently used because of the many advantages they offer in the field of emotion recognition. This study proposes an improved approach for EEG-based emotion recognition on a publicly available newly published dataset, VREED. Differential entropy (DE) features were extracted from four wavebands (theta 4–8 Hz, alpha 8–13 Hz, beta 13–30 Hz, and gamma 30–49 Hz) to classify two emotional states (positive/negative). Five classifiers, namely Support Vector Machine (SVM), k-Nearest Neighbor (kNN), Naïve Bayesian (NB), Decision Tree (DT), and Logistic Regression (LR) were employed with DE features for the automated classification of two emotional states. In this work, we obtained the best average accuracy of 76.22% ± 2.06 with the SVM classifier in the classification of two states. Moreover, we observed from the results that the highest average accuracy score was produced with the gamma band, as previously reported in studies in EEG-based emotion recognition.

Evaluating Ensemble Learning Methods for Multi-Modal Emotion Recognition Using Sensor Data Fusion

Automatic recognition of human emotions is not a trivial process. There are many factors affecting emotions internally and externally. Expressing emotions could also be performed in many ways such as text, speech, body gestures or even physiologically by physiological body responses. Emotion detection enables many applications such as adaptive user interfaces, interactive games, and human robot interaction and many more. The availability of advanced technologies such as mobiles, sensors, and data analytics tools led to the ability to collect data from various sources, which enabled researchers to predict human emotions accurately. Most current research uses them in the lab experiments for data collection. In this work, we use direct and real time sensor data to construct a subject-independent (generic) multi-modal emotion prediction model. This research integrates both on-body physiological markers, surrounding sensory data, and emotion measurements to achieve the following goals: (1) Collecting a multi-modal data set including environmental, body responses, and emotions. (2) Creating subject-independent Predictive models of emotional states based on fusing environmental and physiological variables. (3) Assessing ensemble learning methods and comparing their performance for creating a generic subject-independent model for emotion recognition with high accuracy and comparing the results with previous similar research. To achieve that, we conducted a real-world study “in the wild” with physiological and mobile sensors. Collecting the data-set is coming from participants walking around Minia university campus to create accurate predictive models. Various ensemble learning models (Bagging, Boosting, and Stacking) have been used, combining the following base algorithms (K Nearest Neighbor KNN, Decision Tree DT, Random Forest RF, and Support Vector Machine SVM) as base learners and DT as a meta-classifier. The results showed that, the ensemble stacking learner technique gave the best accuracy of 98.2% compared with other variants of ensemble learning methods. On the contrary, bagging and boosting methods gave (96.4%) and (96.6%) accuracy levels respectively.

Research Directions of Applied Cognitive Sciences

Cognitive science is an interdisciplinary field of investigation of the mind and intelligence [...]

Collision Localization and Classification on the End-Effector of a Cable-Driven Manipulator Applied to EV Auto-Charging Based on DCNN-SVM

With the increasing popularity of electric vehicles, cable-driven serial manipulators have been applied in auto-charging processes for electric vehicles. To ensure the safety of the physical vehicle-robot interaction in this scenario, this paper presents a model-independent collision localization and classification method for cable-driven serial manipulators. First, based on the dynamic characteristics of the manipulator, data sets of terminal collision are constructed. In contrast to utilizing signals based on torque sensors, our data sets comprise the vibration signals of a specific compensator. Then, the collected data sets are applied to construct and train our collision localization and classification model, which consists of a double-layer CNN and an SVM. Compared to previous works, the proposed method can extract features without manual intervention and can deal with collision when the contact surface is irregular. Furthermore, the proposed method is able to generate the location and classification of the collision at the same time. The simulated experiment results show the validity of the proposed collision localization and classification method, with promising prediction accuracy.