Review on Emotion Recognition Based on Electroencephalography

Detecting emotions through EEG signals based on modified convolutional fuzzy neural network

Emotion is a human sense that can influence an individual's life quality in both positive and negative ways. The ability to distinguish different types of emotion can lead researchers to estimate the current situation of patients or the probability of future disease. Recognizing emotions from images have problems concealing their feeling by modifying their facial expressions. This led researchers to consider Electroencephalography (EEG) signals for more accurate emotion detection. However, the complexity of EEG recordings and data analysis using conventional machine learning algorithms caused inconsistent emotion recognition. Therefore, utilizing hybrid deep learning models and other techniques has become common due to their ability to analyze complicated data and achieve higher performance by integrating diverse features of the models. However, researchers prioritize models with fewer parameters to achieve the highest average accuracy. This study improves the Convolutional Fuzzy Neural Network (CFNN) for emotion recognition using EEG signals to achieve a reliable detection system. Initially, the pre-processing and feature extraction phases are implemented to obtain noiseless and informative data. Then, the CFNN with modified architecture is trained to classify emotions. Several parametric and comparative experiments are performed. The proposed model achieved reliable performance for emotion recognition with an average accuracy of 98.21% and 98.08% for valence (pleasantness) and arousal (intensity), respectively, and outperformed state-of-the-art methods.

Electroencephalogram-Based Facial Gesture Recognition Using Self-Organizing Map

Brain-computer interfaces (BCIs) allow information to be transmitted directly from the human brain to a computer, enhancing the ability of human brain activity to interact with the environment. In particular, BCI-based control systems are highly desirable because they can control equipment used by people with disabilities, such as wheelchairs and prosthetic legs. BCIs make use of electroencephalograms (EEGs) to decode the human brain's status. This paper presents an EEG-based facial gesture recognition method based on a self-organizing map (SOM). The proposed facial gesture recognition uses α, β, and θ power bands of the EEG signals as the features of the gesture. The SOM-Hebb classifier is utilized to classify the feature vectors. We utilized the proposed method to develop an online facial gesture recognition system. The facial gestures were defined by combining facial movements that are easy to detect in EEG signals. The recognition accuracy of the system was examined through experiments. The recognition accuracy of the system ranged from 76.90% to 97.57% depending on the number of gestures recognized. The lowest accuracy (76.90%) occurred when recognizing seven gestures, though this is still quite accurate when compared to other EEG-based recognition systems. The implemented online recognition system was developed using MATLAB, and the system took 5.7 s to complete the recognition flow.

Contribution of cosmetic ingredients and skin care textures to emotions

OBJECTIVE

Emotions play an important role in consumers' perception of a sensory experience. The objective of this work was to investigate the ability of basic skin care formulas (i.e. without interference of odour, colour and packaging) and pillar ingredients (i.e. emollients and rheology modifiers) to elicit emotions. Another objective was to track, as claimed by neurocosmetics, the possible effect of formulas to trigger emotions from their direct biochemical effects on the skin.

METHODS

Standard methodologies were mobilized, combining subjective and behavioural parameters (i.e. verbatim, prosody and gesture). Sense and Story methodology based on a collection of metaphoric verbatim was conducted after an induction phase. In addition, an experimental electrophysiological real-time visualization method was tried as a first experience in cosmetics. Finally, the ability of formulations with emotional benefits to modulate the release of neuropeptides by sensory neurons was evaluated on a 3D human model (epidermis co-cultured with sensory neurons).

RESULTS

Skin care formulas were shown to play a role in emotional potential and the types of emotion generated, while changing one ingredient mostly acted on the intensity of the emotions. Verbatim provided contrasted answers depending on the protocol, highlighting the interest of non-verbal approaches to detect subtle effects. The in vitro model substantiated physiological effects of skin care formulas with emotional potential on human skin sensory neuron activity.

CONCLUSION

Emotions were impacted by the change in ingredients and were better captured through non-verbal methods.

An advance in novel intelligent sensory technologies: From an implicit-tracking perspective of food perception

Food sensory evaluation mainly includes explicit and implicit measurement methods. Implicit measures of consumer perception are gaining significant attention in food sensory and consumer science as they provide effective, subconscious, objective analysis. A wide range of advanced technologies are now available for analyzing physiological and psychological responses, including facial analysis technology, neuroimaging technology, autonomic nervous system technology, and behavioral pattern measurement. However, researchers in the food field often lack systematic knowledge of these multidisciplinary technologies and struggle with interpreting their results. In order to bridge this gap, this review systematically describes the principles and highlights the applications in food sensory and consumer science of facial analysis technologies such as eye tracking, facial electromyography, and automatic facial expression analysis, as well as neuroimaging technologies like electroencephalography, magnetoencephalography, functional magnetic resonance imaging, and functional near-infrared spectroscopy. Furthermore, we critically compare and discuss these advanced implicit techniques in the context of food sensory research and then accordingly propose prospects. Ultimately, we conclude that implicit measures should be complemented by traditional explicit measures to capture responses beyond preference. Facial analysis technologies offer a more objective reflection of sensory perception and attitudes toward food, whereas neuroimaging techniques provide valuable insight into the implicit physiological responses during food consumption. To enhance the interpretability and generalizability of implicit measurement results, further sensory studies are needed. Looking ahead, the combination of different methodological techniques in real-life situations holds promise for consumer sensory science in the field of food research.

[Research on emotion recognition method based on IWOA-ELM algorithm for electroencephalogram]

Emotion is a crucial physiological attribute in humans, and emotion recognition technology can significantly assist individuals in self-awareness. Addressing the challenge of significant differences in electroencephalogram (EEG) signals among different subjects, we introduce a novel mechanism in the traditional whale optimization algorithm (WOA) to expedite the optimization and convergence of the algorithm. Furthermore, the improved whale optimization algorithm (IWOA) was applied to search for the optimal training solution in the extreme learning machine (ELM) model, encompassing the best feature set, training parameters, and EEG channels. By testing 24 common EEG emotion features, we concluded that optimal EEG emotion features exhibited a certain level of specificity while also demonstrating some commonality among subjects. The proposed method achieved an average recognition accuracy of 92.19% in EEG emotion recognition, significantly reducing the manual tuning workload and offering higher accuracy with shorter training times compared to the control method. It outperformed existing methods, providing a superior performance and introducing a novel perspective for decoding EEG signals, thereby contributing to the field of emotion research from EEG signal.

Method for Assessing the Influence of Phobic Stimuli in Virtual Simulators

In the organizing of professional training, the assessment of the trainee's reaction and state in stressful situations is of great importance. Phobic reactions are a specific type of stress reaction that, however, is rarely taken into account when developing virtual simulators, and are a risk factor in the workplace. A method for evaluating the impact of various phobic stimuli on the quality of training is considered, which takes into account the time, accuracy, and speed of performing professional tasks, as well as the characteristics of electroencephalograms (the amplitude, power, coherence, Hurst exponent, and degree of interhemispheric asymmetry). To evaluate the impact of phobias during experimental research, participants in the experimental group performed exercises in different environments: under normal conditions and under the influence of acrophobic and arachnophobic stimuli. The participants were divided into subgroups using clustering algorithms and an expert neurologist. After that, a comparison of the subgroup metrics was carried out. The research conducted makes it possible to partially confirm our hypotheses about the negative impact of phobic effects on some participants in the experimental group. The relationship between the reaction to a phobia and the characteristics of brain activity was revealed, and the characteristics of the electroencephalogram signal were considered as the metrics for detecting a phobic reaction.

Towards an objective theory of subjective liking: A first step in understanding the sense of beauty

The study of the electroencephalogram signals recorded from subjects during an experience is a way to understand the brain processes that underlie their physical and emotional involvement. Such signals have the form of time series, and their analysis could benefit from applying techniques that are specific to this kind of data. Neuroaesthetics, as defined by Zeki in 1999, is the scientific approach to the study of aesthetic perceptions of art, music, or any other experience that can give rise to aesthetic judgments, such as liking or disliking a painting. Starting from a proprietary dataset of 248 trials from 16 subjects exposed to art paintings, using a real ecological context, this paper analyses the application of a novel symbolic machine learning technique, specifically designed to extract information from unstructured data and to express it in form of logical rules. Our purpose is to extract qualitative and quantitative logical rules, to relate the voltage at specific frequencies and in specific electrodes, and that, within the limits of the experiment, may help to understand the brain process that drives liking or disliking experiences in human subjects.

FCAN-XGBoost: A Novel Hybrid Model for EEG Emotion Recognition

In recent years, artificial intelligence (AI) technology has promoted the development of electroencephalogram (EEG) emotion recognition. However, existing methods often overlook the computational cost of EEG emotion recognition, and there is still room for improvement in the accuracy of EEG emotion recognition. In this study, we propose a novel EEG emotion recognition algorithm called FCAN-XGBoost, which is a fusion of two algorithms, FCAN and XGBoost. The FCAN module is a feature attention network (FANet) that we have proposed for the first time, which processes the differential entropy (DE) and power spectral density (PSD) features extracted from the four frequency bands of the EEG signal and performs feature fusion and deep feature extraction. Finally, the deep features are fed into the eXtreme Gradient Boosting (XGBoost) algorithm to classify the four emotions. We evaluated the proposed method on the DEAP and DREAMER datasets and achieved a four-category emotion recognition accuracy of 95.26% and 94.05%, respectively. Additionally, our proposed method reduces the computational cost of EEG emotion recognition by at least 75.45% for computation time and 67.51% for memory occupation. The performance of FCAN-XGBoost outperforms the state-of-the-art four-category model and reduces computational costs without losing classification performance compared with other models.

Online Learning for Wearable EEG-Based Emotion Classification

Giving emotional intelligence to machines can facilitate the early detection and prediction of mental diseases and symptoms. Electroencephalography (EEG)-based emotion recognition is widely applied because it measures electrical correlates directly from the brain rather than indirect measurement of other physiological responses initiated by the brain. Therefore, we used non-invasive and portable EEG sensors to develop a real-time emotion classification pipeline. The pipeline trains different binary classifiers for Valence and Arousal dimensions from an incoming EEG data stream achieving a 23.9% (Arousal) and 25.8% (Valence) higher F1-Score on the state-of-art AMIGOS dataset than previous work. Afterward, the pipeline was applied to the curated dataset from 15 participants using two consumer-grade EEG devices while watching 16 short emotional videos in a controlled environment. Mean F1-Scores of 87% (Arousal) and 82% (Valence) were achieved for an immediate label setting. Additionally, the pipeline proved to be fast enough to achieve predictions in real-time in a live scenario with delayed labels while continuously being updated. The significant discrepancy from the readily available labels on the classification scores leads to future work to include more data. Thereafter, the pipeline is ready to be used for real-time applications of emotion classification.

Decision-making based on an improved visual analytics approach for emotion prediction

Visual Analytics approach allows driving informed and effective decision-making. It assists decision-makers to visually interact with large amount of data and to computationally learn valuable hidden patterns in that data, which improve the decision quality. In this article, we introduce an enhanced visual analytics model combining cognitive-based visual analysis to data mining-based automatic analysis. As emotions are strongly related to human behaviour and society, emotion prediction is widely considered by decision making activities. Unlike speech and facial expressions modalities, EEG (electroencephalogram) has the advantage of being able to record information about the internal emotional state that is not always translated by perceptible external manifestations. For this reason, we applied the proposed cognitive approach on EEG data to demonstrate its efficiency for predicting emotional reaction to films. For automatic analysis, we developed the Echo State Network (ESN) technique considered as an efficient machine learning solution due to its straightforward training procedure and high modelling ability for handling time-series problems. Finally, utility and usability tests were performed to evaluate the developed prototype.

Comprehensive Analysis of Feature Extraction Methods for Emotion Recognition from Multichannel EEG Recordings

Advances in signal processing and machine learning have expedited electroencephalogram (EEG)-based emotion recognition research, and numerous EEG signal features have been investigated to detect or characterize human emotions. However, most studies in this area have used relatively small monocentric data and focused on a limited range of EEG features, making it difficult to compare the utility of different sets of EEG features for emotion recognition. This study addressed that by comparing the classification accuracy (performance) of a comprehensive range of EEG feature sets for identifying emotional states, in terms of valence and arousal. The classification accuracy of five EEG feature sets were investigated, including statistical features, fractal dimension (FD), Hjorth parameters, higher order spectra (HOS), and those derived using wavelet analysis. Performance was evaluated using two classifier methods, support vector machine (SVM) and classification and regression tree (CART), across five independent and publicly available datasets linking EEG to emotional states: MAHNOB-HCI, DEAP, SEED, AMIGOS, and DREAMER. The FD-CART feature-classification method attained the best mean classification accuracy for valence (85.06%) and arousal (84.55%) across the five datasets. The stability of these findings across the five different datasets also indicate that FD features derived from EEG data are reliable for emotion recognition. The results may lead to the possible development of an online feature extraction framework, thereby enabling the development of an EEG-based emotion recognition system in real time.

EEG diagnosis of depression based on multi-channel data fusion and clipping augmentation and convolutional neural network

Depression is an undetectable mental disease. Most of the patients with depressive symptoms do not know that they are suffering from depression. Since the novel Coronavirus pandemic 2019, the number of patients with depression has increased rapidly. There are two kinds of traditional depression diagnosis. One is that professional psychiatrists make diagnosis results for patients, but it is not conducive to large-scale depression detection. Another is to use electroencephalography (EEG) to record neuronal activity. Then, the features of the EEG are extracted using manual or traditional machine learning methods to diagnose the state and type of depression. Although this method achieves good results, it does not fully utilize the multi-channel information of EEG. Aiming at this problem, an EEG diagnosis method for depression based on multi-channel data fusion cropping enhancement and convolutional neural network is proposed. First, the multi-channel EEG data are transformed into 2D images after multi-channel fusion (MCF) and multi-scale clipping (MSC) augmentation. Second, it is trained by a multi-channel convolutional neural network (MCNN). Finally, the trained model is loaded into the detection device to classify the input EEG signals. The experimental results show that the combination of MCF and MSC can make full use of the information contained in the single sensor records, and significantly improve the classification accuracy and clustering effect of depression diagnosis. The method has the advantages of low complexity and good robustness in signal processing and feature extraction, which is beneficial to the wide application of detection systems.

Emotion Recognition in a Health Continuum: Comparison of Healthy Adults of Advancing Age, Community Dwelling Adults Bearing Vascular Risk Factors and People Diagnosed with Mild Cognitive Impairment

The identification of basic emotions plays an important role in social relationships and behaviors linked to survival. In neurodegenerative conditions such as Alzheimer's disease (AD), the ability to recognize emotions may already be impaired at early stages of the disease, such as the stage of Mild Cognitive Impairment (MCI). However, as regards vascular pathologies related to cognitive impairment, very little is known about emotion recognition in people bearing vascular risk factors (VRF). Therefore, the aim of the present study was to examine emotion recognition ability in the health continuum "healthy advancing age-advancing age with VRF-MCI". The sample consisted of 106 adults divided in three diagnostic groups; 43 adults with MCI, 41 adults bearing one or more VRF, and 22 healthy controls of advancing age (HC). Since HC were more educated and younger than the other two groups, the age-group and level of educational were taken into account in the statistical analyses. A dynamic visual test was administered to examine recognition of basic emotions and emotionally neutral conditions. The results showed only a significant diagnostic group x educational level interaction as regards total emotion recognition ability, F (4, 28.910) = 4.117 p = 0.004 η² = 0.166. High educational level seems to contribute to a high-level-emotion-recognition-performance both in healthy adults of advancing age and in adults bearing vascular risk factors. Medium educational level appears to play the same role only in healthy adults. Neither educational level can help MCI people to enhance their significantly lower emotion recognition ability.

Machine Learning Models for Classification of Human Emotions Using Multivariate Brain Signals

Humans can portray different expressions contrary to their emotional state of mind. Therefore, it is difficult to judge humans’ real emotional state simply by judging their physical appearance. Although researchers are working on facial expressions analysis, voice recognition, and gesture recognition; the accuracy levels of such analysis are much less and the results are not reliable. Hence, it becomes vital to have realistic emotion detector. Electroencephalogram (EEG) signals remain neutral to the external appearance and behavior of the human and help in ensuring accurate analysis of the state of mind. The EEG signals from various electrodes in different scalp regions are studied for performance. Hence, EEG has gained attention over time to obtain accurate results for the classification of emotional states in human beings for human–machine interaction as well as to design a program where an individual could perform a self-analysis of his emotional state. In the proposed scheme, we extract power spectral densities of multivariate EEG signals from different sections of the brain. From the extracted power spectral density (PSD), the features which provide a better feature for classification are selected and classified using long short-term memory (LSTM) and bi-directional long short-term memory (Bi-LSTM). The 2-D emotion model considered for the classification of frontal, parietal, temporal, and occipital is studied. The region-based classification is performed by considering positive and negative emotions. The performance accuracy of our previous model’s results of artificial neural network (ANN), support vector machine (SVM), K-nearest neighbor (K-NN), and LSTM was compared and 94.95% accuracy was received using Bi-LSTM considering four prefrontal electrodes.

Empirical mode decomposition using deep learning model for financial market forecasting

Financial market forecasting is an essential component of financial systems; however, predicting financial market trends is a challenging job due to noisy and non-stationary information. Deep learning is renowned for bringing out excellent abstract features from the huge volume of raw data without depending on prior knowledge, which is potentially fascinating in forecasting financial transactions. This article aims to propose a deep learning model that autonomously mines the statistical rules of data and guides the financial market transactions based on empirical mode decomposition (EMD) with back-propagation neural networks (BPNN). Through the characteristic time scale of data, the intrinsic wave pattern was obtained and then decomposed. Financial market transaction data were analyzed, optimized using PSO, and predicted. Combining the nonlinear and non-stationary financial time series can improve prediction accuracy. The predictive model of deep learning, based on the analysis of the massive financial trading data, can forecast the future trend of financial market price, forming a trading signal when particular confidence is satisfied. The empirical results show that the EMD-based deep learning model has an excellent predicting performance.

Accelerating 3D Convolutional Neural Network with Channel Bottleneck Module for EEG-Based Emotion Recognition

Deep learning-based emotion recognition using EEG has received increasing attention in recent years. The existing studies on emotion recognition show great variability in their employed methods including the choice of deep learning approaches and the type of input features. Although deep learning models for EEG-based emotion recognition can deliver superior accuracy, it comes at the cost of high computational complexity. Here, we propose a novel 3D convolutional neural network with a channel bottleneck module (CNN-BN) model for EEG-based emotion recognition, with the aim of accelerating the CNN computation without a significant loss in classification accuracy. To this end, we constructed a 3D spatiotemporal representation of EEG signals as the input of our proposed model. Our CNN-BN model extracts spatiotemporal EEG features, which effectively utilize the spatial and temporal information in EEG. We evaluated the performance of the CNN-BN model in the valence and arousal classification tasks. Our proposed CNN-BN model achieved an average accuracy of 99.1% and 99.5% for valence and arousal, respectively, on the DEAP dataset, while significantly reducing the number of parameters by 93.08% and FLOPs by 94.94%. The CNN-BN model with fewer parameters based on 3D EEG spatiotemporal representation outperforms the state-of-the-art models. Our proposed CNN-BN model with a better parameter efficiency has excellent potential for accelerating CNN-based emotion recognition without losing classification performance.

A Preliminary Investigation on Frequency Dependant Cues for Human Emotions

The recent advances in Human-Computer Interaction and Artificial Intelligence have significantly increased the importance of identifying human emotions from different sensory cues. Hence, understanding the underlying relationships between emotions and sensory cues have become a subject of study in many fields including Acoustics, Psychology, Psychiatry, Neuroscience and Biochemistry. This work is a preliminary step towards investigating cues for human emotion on a fundamental level by aiming to establish relationships between tonal frequencies of sound and emotions. For that, an online perception test is conducted, in which participants are asked to rate the perceived emotions corresponding to each tone. The results show that a crossover point for four primary emotions lies in the frequency range of 417–440 Hz, thus consolidating the hypothesis that the frequency range of 432–440 Hz is neutral from human emotion perspective. It is also observed that the frequency dependant relationships between emotion pairs Happy—Sad, and Anger—Calm are approximately mirrored symmetric in nature.

Towards Knowledge-Based Tourism Chinese Question Answering System

With the rapid development of the tourism industry, various travel websites are emerging. The tourism question answering system explores a large amount of information from these travel websites to answer tourism questions, which is critical for providing a competitive travel experience. In this paper, we propose a framework that automatically constructs a tourism knowledge graph from a series of travel websites with regard to tourist attractions in Zhejiang province, China. Backed by this domain-specific knowledge base, we developed a tourism question answering system that also incorporates the underlying knowledge from a large-scale language model such as BERT. Experiments on real-world datasets demonstrate that the proposed method outperforms the baseline on various metrics. We also show the effectiveness of each of the question answering components in detail, including the query intent recognition and the answer generation.