Recognition of human emotions using EEG signals: A review

CATM: A Multi-Feature-Based Cross-Scale Attentional Convolutional EEG Emotion Recognition Model

Aiming at the problem that existing emotion recognition methods fail to make full use of the information in the time, frequency, and spatial domains in the EEG signals, which leads to the low accuracy of EEG emotion classification, this paper proposes a multi-feature, multi-frequency band-based cross-scale attention convolutional model (CATM). The model is mainly composed of a cross-scale attention module, a frequency-space attention module, a feature transition module, a temporal feature extraction module, and a depth classification module. First, the cross-scale attentional convolution module extracts spatial features at different scales for the preprocessed EEG signals; then, the frequency-space attention module assigns higher weights to important channels and spatial locations; next, the temporal feature extraction module extracts temporal features of the EEG signals; and, finally, the depth classification module categorizes the EEG signals into emotions. We evaluated the proposed method on the DEAP dataset with accuracies of 99.70% and 99.74% in the valence and arousal binary classification experiments, respectively; the accuracy in the valence-arousal four-classification experiment was 97.27%. In addition, considering the application of fewer channels, we also conducted 5-channel experiments, and the binary classification accuracies of valence and arousal were 97.96% and 98.11%, respectively. The valence-arousal four-classification accuracy was 92.86%. The experimental results show that the method proposed in this paper exhibits better results compared to other recent methods, and also achieves better results in few-channel experiments.

EEG-based emotion recognition using graph convolutional neural network with dual attention mechanism

EEG-based emotion recognition is becoming crucial in brain-computer interfaces (BCI). Currently, most researches focus on improving accuracy, while neglecting further research on the interpretability of models, we are committed to analyzing the impact of different brain regions and signal frequency bands on emotion generation based on graph structure. Therefore, this paper proposes a method named Dual Attention Mechanism Graph Convolutional Neural Network (DAMGCN). Specifically, we utilize graph convolutional neural networks to model the brain network as a graph to extract representative spatial features. Furthermore, we employ the self-attention mechanism of the Transformer model which allocates more electrode channel weights and signal frequency band weights to important brain regions and frequency bands. The visualization of attention mechanism clearly demonstrates the weight allocation learned by DAMGCN. During the performance evaluation of our model on the DEAP, SEED, and SEED-IV datasets, we achieved the best results on the SEED dataset, showing subject-dependent experiments' accuracy of 99.42% and subject-independent experiments' accuracy of 73.21%. The results are demonstrably superior to the accuracies of most existing models in the realm of EEG-based emotion recognition.

An improved empirical mode decomposition method with ensemble classifiers for analysis of multichannel EEG in BCI emotion recognition

Emotion recognition using EEG is a difficult study because the signals' unstable behavior, which is brought on by the brain's complex neuronal activity, makes it difficult to extract the underlying patterns inside it. Therefore, to analyse the signal more efficiently, in this article, a hybrid model based on IEMD-KW-Ens (Improved Empirical Mode Decomposition-Kruskal Wallis-Ensemble classifiers) technique is used. Here IEMD based technique is proposed to interpret EEG signals by adding an improved sifting stopping criterion with median filter to get the optimal decomposed EEG signals for further processing. A mixture of time, frequency and non-linear distinct features are extracted for constructing the feature vector. Afterward, we conducted feature selection using KW test to remove the insignificant ones from the feature set. Later the classification of emotions in three-dimensional model is performed in two categories i.e. machine learning based RUSBoosted trees and deep learning based convolutional neural network (CNN) for DEAP and DREAMER datasets and the outcomes are evaluated for valence, arousal, and dominance classes. The findings demonstrate that the hybrid model can successfully classify emotions in multichannel EEG signals. The decomposition approach is also instructive for improving the model's utility in emotional computing.

GC-STCL: A Granger Causality-Based Spatial-Temporal Contrastive Learning Framework for EEG Emotion Recognition

EEG signals capture information through multi-channel electrodes and hold promising prospects for human emotion recognition. However, the presence of high levels of noise and the diverse nature of EEG signals pose significant challenges, leading to potential overfitting issues that further complicate the extraction of meaningful information. To address this issue, we propose a Granger causal-based spatial-temporal contrastive learning framework, which significantly enhances the ability to capture EEG signal information by modeling rich spatial-temporal relationships. Specifically, in the spatial dimension, we employ a sampling strategy to select positive sample pairs from individuals watching the same video. Subsequently, a Granger causality test is utilized to enhance graph data and construct potential causality for each channel. Finally, a residual graph convolutional neural network is employed to extract features from EEG signals and compute spatial contrast loss. In the temporal dimension, we first apply a frequency domain noise reduction module for data enhancement on each time series. Then, we introduce the Granger-Former model to capture time domain representation and calculate the time contrast loss. We conduct extensive experiments on two publicly available sentiment recognition datasets (DEAP and SEED), achieving 1.65% improvement of the DEAP dataset and 1.55% improvement of the SEED dataset compared to state-of-the-art unsupervised models. Our method outperforms benchmark methods in terms of prediction accuracy as well as interpretability.

A comprehensive review of deep learning in EEG-based emotion recognition: classifications, trends, and practical implications

Emotion recognition utilizing EEG signals has emerged as a pivotal component of human-computer interaction. In recent years, with the relentless advancement of deep learning techniques, using deep learning for analyzing EEG signals has assumed a prominent role in emotion recognition. Applying deep learning in the context of EEG-based emotion recognition carries profound practical implications. Although many model approaches and some review articles have scrutinized this domain, they have yet to undergo a comprehensive and precise classification and summarization process. The existing classifications are somewhat coarse, with insufficient attention given to the potential applications within this domain. Therefore, this article systematically classifies recent developments in EEG-based emotion recognition, providing researchers with a lucid understanding of this field's various trajectories and methodologies. Additionally, it elucidates why distinct directions necessitate distinct modeling approaches. In conclusion, this article synthesizes and dissects the practical significance of EEG signals in emotion recognition, emphasizing its promising avenues for future application.

Instantaneous estimation of momentary affective responses using neurophysiological signals and a spatiotemporal emotional intensity regression network

Previous studies in affective computing often use a fixed emotional label to train an emotion classifier with electroencephalography (EEG) from individuals experiencing an affective stimulus. However, EEGs encode emotional dynamics that include varying intensities within a given emotional category. To investigate these variations in emotional intensity, we propose a framework that obtains momentary affective labels for fine-grained segments of EEGs with human feedback. We then model these labeled segments using a novel spatiotemporal emotional intensity regression network (STEIR-Net). It integrates temporal EEG patterns from nine predefined cortical regions to provide a continuous estimation of emotional intensity. We demonstrate that the STEIR-Net outperforms classical regression models by reducing the root mean square error (RMSE) by an average of 4∼9 % and 2∼4 % for the SEED and SEED-IV databases, respectively. We find that the frontal and temporal cortical regions contribute significantly to the affective intensity's variation. Higher absolute values of the Spearman correlation coefficient between the model estimation and momentary affective labels under happiness (0.2114) and fear (0.2072) compared to neutral (0.1694) and sad (0.1895) emotions were observed. Besides, increasing the input length of the EEG segments from 4 to 20 s further reduces the RMSE from 1.3548 to 1.3188.

An advance in novel intelligent sensory technologies: From an implicit-tracking perspective of food perception

Food sensory evaluation mainly includes explicit and implicit measurement methods. Implicit measures of consumer perception are gaining significant attention in food sensory and consumer science as they provide effective, subconscious, objective analysis. A wide range of advanced technologies are now available for analyzing physiological and psychological responses, including facial analysis technology, neuroimaging technology, autonomic nervous system technology, and behavioral pattern measurement. However, researchers in the food field often lack systematic knowledge of these multidisciplinary technologies and struggle with interpreting their results. In order to bridge this gap, this review systematically describes the principles and highlights the applications in food sensory and consumer science of facial analysis technologies such as eye tracking, facial electromyography, and automatic facial expression analysis, as well as neuroimaging technologies like electroencephalography, magnetoencephalography, functional magnetic resonance imaging, and functional near-infrared spectroscopy. Furthermore, we critically compare and discuss these advanced implicit techniques in the context of food sensory research and then accordingly propose prospects. Ultimately, we conclude that implicit measures should be complemented by traditional explicit measures to capture responses beyond preference. Facial analysis technologies offer a more objective reflection of sensory perception and attitudes toward food, whereas neuroimaging techniques provide valuable insight into the implicit physiological responses during food consumption. To enhance the interpretability and generalizability of implicit measurement results, further sensory studies are needed. Looking ahead, the combination of different methodological techniques in real-life situations holds promise for consumer sensory science in the field of food research.

Enhancing EEG-based attachment style prediction: unveiling the impact of feature domains

Introduction

Attachment styles are crucial in human relationships and have been explored through neurophysiological responses and EEG data analysis. This study investigates the potential of EEG data in predicting and differentiating secure and insecure attachment styles, contributing to the understanding of the neural basis of interpersonal dynamics.

Methods

We engaged 27 participants in our study, employing an XGBoost classifier to analyze EEG data across various feature domains, including time-domain, complexity-based, and frequency-based attributes.

Results

The study found significant differences in the precision of attachment style prediction: a high precision rate of 96.18% for predicting insecure attachment, and a lower precision of 55.34% for secure attachment. Balanced accuracy metrics indicated an overall model accuracy of approximately 84.14%, taking into account dataset imbalances.

Discussion

These results highlight the challenges in using EEG patterns for attachment style prediction due to the complex nature of attachment insecurities. Individuals with heightened perceived insecurity predominantly aligned with the insecure attachment category, suggesting a link to their increased emotional reactivity and sensitivity to social cues. The study underscores the importance of time-domain features in prediction accuracy, followed by complexity-based features, while noting the lesser impact of frequency-based features. Our findings advance the understanding of the neural correlates of attachment and pave the way for future research, including expanding demographic diversity and integrating multimodal data to refine predictive models.

Investigation of different ML approaches in classification of emotions induced by acute stress

Background

Machine learning is becoming a common tool in monitoring emotion. However, methodological studies of the processing pipeline are scarce, especially ones using subjective appraisals as ground truth.

New method

A novel protocol was used to induce cognitive load and physical discomfort, and emotional dimensions (arousal, valence, and dominance) were reported after each task. The performance of five common ML models with a versatile set of features (physiological features, task performance data, and personality trait) was compared in binary classification of subjectively assessed emotions.

Results

The psychophysiological responses proved the protocol was successful in changing the mental state from baseline, also the cognitive and physical tasks were different. The optimization and performance of ML models used for emotion detection were evaluated. Additionally, methods to account for imbalanced classes were applied and shown to improve the classification performance.

Comparison with existing methods

Classification of human emotional states often assumes the states are determined by the stimuli. However, individual appraisals vary. None of the past studies have classified subjective emotional dimensions with a set of features including biosignals, personality and behavior.

Conclusion

Our data represent a typical setup in affective computing utilizing psychophysiological monitoring: N is low compared to number of features, inter-individual variability is high, and class imbalance cannot be avoided. Our observations are a) if possible, include features representing physiology, behavior and personality, b) use simple models and limited number of features to improve interpretability, c) address the possible imbalance, d) if the data size allows, use nested cross-validation.

Understanding the role of emotion in decision making process: using machine learning to analyze physiological responses to visual, auditory, and combined stimulation

Emotions significantly shape decision-making, and targeted emotional elicitations represent an important factor in neuromarketing, where they impact advertising effectiveness by capturing potential customers' attention intricately associated with emotional triggers. Analyzing biometric parameters after stimulus exposure may help in understanding emotional states. This study investigates autonomic and central nervous system responses to emotional stimuli, including images, auditory cues, and their combination while recording physiological signals, namely the electrocardiogram, blood volume pulse, galvanic skin response, pupillometry, respiration, and the electroencephalogram. The primary goal of the proposed analysis is to compare emotional stimulation methods and to identify the most effective approach for distinct physiological patterns. A novel feature selection technique is applied to further optimize the separation of four emotional states. Basic machine learning approaches are used in order to discern emotions as elicited by different kinds of stimulation. Electroencephalographic signals, Galvanic skin response and cardio-respiratory coupling-derived features provided the most significant features in distinguishing the four emotional states. Further findings highlight how auditory stimuli play a crucial role in creating distinct physiological patterns that enhance classification within a four-class problem. When combining all three types of stimulation, a validation accuracy of 49% was achieved. The sound-only and the image-only phases resulted in 52% and 44% accuracy respectively, whereas the combined stimulation of images and sounds led to 51% accuracy. Isolated visual stimuli yield less distinct patterns, necessitating more signals for relatively inferior performance compared to other types of stimuli. This surprising significance arises from limited auditory exploration in emotional recognition literature, particularly contrasted with the pleathora of studies performed using visual stimulation. In marketing, auditory components might hold a more relevant potential to significantly influence consumer choices.

Cross-subject emotion recognition using hierarchical feature optimization and support vector machine with multi-kernel collaboration

Objective. Due to individual differences, it is greatly challenging to realize the multiple types of emotion identification across subjects.Approach. In this research, a hierarchical feature optimization method is proposed in order to represent emotional states effectively based on peripheral physiological signals. Firstly, sparse learning combined with binary search is employed to achieve feature selection of single signals. Then an improved fast correlation-based filter is proposed to implement fusion optimization of multi-channel signal features. Aiming at overcoming the limitations of the support vector machine (SVM), which uses a single kernel function to make decisions, the multi-kernel function collaboration strategy is proposed to improve the classification performance of SVM.Main results. The effectiveness of the proposed method is verified on the DEAP dataset. Experimental results show that the proposed method presents a competitive performance for four cross-subject  types of emotion identification with an accuracy of 84% (group 1) and 85.07% (group 2). Significance. The proposed model with hierarchical feature optimization and SVM with multi-kernel function collaboration demonstrates superior emotion recognition accuracy compared to state-of-the-art techniques. In addition, the analysis based on DEAP dataset composition characteristics presents a novel perspective to explore the emotion recognition issue more objectively and comprehensively.

Classifying human emotions in HRI: applying global optimization model to EEG brain signals

Significant efforts have been made in the past decade to humanize both the form and function of social robots to increase their acceptance among humans. To this end, social robots have recently been combined with brain-computer interface (BCI) systems in an attempt to give them an understanding of human mental states, particularly emotions. However, emotion recognition using BCIs poses several challenges, such as subjectivity of emotions, contextual dependency, and a lack of reliable neuro-metrics for real-time processing of emotions. Furthermore, the use of BCI systems introduces its own set of limitations, such as the bias-variance trade-off, dimensionality, and noise in the input data space. In this study, we sought to address some of these challenges by detecting human emotional states from EEG brain activity during human-robot interaction (HRI). EEG signals were collected from 10 participants who interacted with a Pepper robot that demonstrated either a positive or negative personality. Using emotion valence and arousal measures derived from frontal brain asymmetry (FBA), several machine learning models were trained to classify human's mental states in response to the robot personality. To improve classification accuracy, all proposed classifiers were subjected to a Global Optimization Model (GOM) based on feature selection and hyperparameter optimization techniques. The results showed that it is possible to classify a user's emotional responses to the robot's behavior from the EEG signals with an accuracy of up to 92%. The outcome of the current study contributes to the first level of the Theory of Mind (ToM) in Human-Robot Interaction, enabling robots to comprehend users' emotional responses and attribute mental states to them. Our work advances the field of social and assistive robotics by paving the way for the development of more empathetic and responsive HRI in the future.

Emotion recognition in EEG signals using deep learning methods: A review

Emotions are a critical aspect of daily life and serve a crucial role in human decision-making, planning, reasoning, and other mental states. As a result, they are considered a significant factor in human interactions. Human emotions can be identified through various sources, such as facial expressions, speech, behavior (gesture/position), or physiological signals. The use of physiological signals can enhance the objectivity and reliability of emotion detection. Compared with peripheral physiological signals, electroencephalogram (EEG) recordings are directly generated by the central nervous system and are closely related to human emotions. EEG signals have the great spatial resolution that facilitates the evaluation of brain functions, making them a popular modality in emotion recognition studies. Emotion recognition using EEG signals presents several challenges, including signal variability due to electrode positioning, individual differences in signal morphology, and lack of a universal standard for EEG signal processing. Moreover, identifying the appropriate features for emotion recognition from EEG data requires further research. Finally, there is a need to develop more robust artificial intelligence (AI) including conventional machine learning (ML) and deep learning (DL) methods to handle the complex and diverse EEG signals associated with emotional states. This paper examines the application of DL techniques in emotion recognition from EEG signals and provides a detailed discussion of relevant articles. The paper explores the significant challenges in emotion recognition using EEG signals, highlights the potential of DL techniques in addressing these challenges, and suggests the scope for future research in emotion recognition using DL techniques. The paper concludes with a summary of its findings.

Mental Stress Classification Based on Selected Electroencephalography Channels Using Correlation Coefficient of Hjorth Parameters

Electroencephalography (EEG) signals offer invaluable insights into diverse activities of the human brain, including the intricate physiological and psychological responses associated with mental stress. A major challenge, however, is accurately identifying mental stress while mitigating the limitations associated with a large number of EEG channels. Such limitations encompass computational complexity, potential overfitting, and the prolonged setup time for electrode placement, all of which can hinder practical applications. To address these challenges, this study presents the novel CCHP method, aimed at identifying and ranking commonly optimal EEG channels based on their sensitivity to the mental stress state. This method's uniqueness lies in its ability not only to find common channels, but also to prioritize them according to their responsiveness to stress, ensuring consistency across subjects and making it potentially transformative for real-world applications. From our rigorous examinations, eight channels emerged as universally optimal in detecting stress variances across participants. Leveraging features from the time, frequency, and time-frequency domains of these channels, and employing machine learning algorithms, notably RLDA, SVM, and KNN, our approach achieved a remarkable accuracy of 81.56% with the SVM algorithm outperforming existing methodologies. The implications of this research are profound, offering a stepping stone toward the development of real-time stress detection devices, and consequently, enabling clinicians to make more informed therapeutic decisions based on comprehensive brain activity monitoring.

Transformers for autonomous recognition of psychiatric dysfunction via raw and imbalanced EEG signals

Early identification of mental disorders, based on subjective interviews, is extremely challenging in the clinical setting. There is a growing interest in developing automated screening tools for potential mental health problems based on biological markers. Here, we demonstrate the feasibility of an AI-powered diagnosis of different mental disorders using EEG data. Specifically, this work aims to classify different mental disorders in the following ecological context accurately: (1) using raw EEG data, (2) collected during rest, (3) during both eye open, and eye closed conditions, (4) at short 2-min duration, (5) on participants with different psychiatric conditions, (6) with some overlapping symptoms, and (7) with strongly imbalanced classes. To tackle this challenge, we designed and optimized a transformer-based architecture, where class imbalance is addressed through focal loss and class weight balancing. Using the recently released TDBRAIN dataset (n= 1274 participants), our method classifies each participant as either a neurotypical or suffering from major depressive disorder (MDD), attention deficit hyperactivity disorder (ADHD), subjective memory complaints (SMC), or obsessive-compulsive disorder (OCD). We evaluate the performance of the proposed architecture on both the window-level and the patient-level. The classification of the 2-min raw EEG data into five classes achieved a window-level accuracy of 63.2% and 65.8% for open and closed eye conditions, respectively. When the classification is limited to three main classes (MDD, ADHD, SMC), window level accuracy improved to 75.1% and 69.9% for eye open and eye closed conditions, respectively. Our work paves the way for developing novel AI-based methods for accurately diagnosing mental disorders using raw resting-state EEG data.

Multiclass classification of imagined speech EEG using noise-assisted multivariate empirical mode decomposition and multireceptive field convolutional neural network

Introduction

In this study, we classified electroencephalography (EEG) data of imagined speech using signal decomposition and multireceptive convolutional neural network. The imagined speech EEG with five vowels /a/, /e/, /i/, /o/, and /u/, and mute (rest) sounds were obtained from ten study participants.

Materials and methods

First, two different signal decomposition methods were applied for comparison: noise-assisted multivariate empirical mode decomposition and wavelet packet decomposition. Six statistical features were calculated from the decomposed eight sub-frequency bands EEG. Next, all features obtained from each channel of the trial were vectorized and used as the input vector of classifiers. Lastly, EEG was classified using multireceptive field convolutional neural network and several other classifiers for comparison.

Results

We achieved an average classification rate of 73.09 and up to 80.41% in a multiclass (six classes) setup (Chance: 16.67%). In comparison with various other classifiers, significant improvements for other classifiers were achieved (p-value < 0.05). From the frequency sub-band analysis, high-frequency band regions and the lowest-frequency band region contain more information about imagined vowel EEG data. The misclassification and classification rate of each vowel imaginary EEG was analyzed through a confusion matrix.

Discussion

Imagined speech EEG can be classified successfully using the proposed signal decomposition method and a convolutional neural network. The proposed classification method for imagined speech EEG can contribute to developing a practical imagined speech-based brain-computer interfaces system.

Towards an objective theory of subjective liking: A first step in understanding the sense of beauty

The study of the electroencephalogram signals recorded from subjects during an experience is a way to understand the brain processes that underlie their physical and emotional involvement. Such signals have the form of time series, and their analysis could benefit from applying techniques that are specific to this kind of data. Neuroaesthetics, as defined by Zeki in 1999, is the scientific approach to the study of aesthetic perceptions of art, music, or any other experience that can give rise to aesthetic judgments, such as liking or disliking a painting. Starting from a proprietary dataset of 248 trials from 16 subjects exposed to art paintings, using a real ecological context, this paper analyses the application of a novel symbolic machine learning technique, specifically designed to extract information from unstructured data and to express it in form of logical rules. Our purpose is to extract qualitative and quantitative logical rules, to relate the voltage at specific frequencies and in specific electrodes, and that, within the limits of the experiment, may help to understand the brain process that drives liking or disliking experiences in human subjects.

FCAN-XGBoost: A Novel Hybrid Model for EEG Emotion Recognition

In recent years, artificial intelligence (AI) technology has promoted the development of electroencephalogram (EEG) emotion recognition. However, existing methods often overlook the computational cost of EEG emotion recognition, and there is still room for improvement in the accuracy of EEG emotion recognition. In this study, we propose a novel EEG emotion recognition algorithm called FCAN-XGBoost, which is a fusion of two algorithms, FCAN and XGBoost. The FCAN module is a feature attention network (FANet) that we have proposed for the first time, which processes the differential entropy (DE) and power spectral density (PSD) features extracted from the four frequency bands of the EEG signal and performs feature fusion and deep feature extraction. Finally, the deep features are fed into the eXtreme Gradient Boosting (XGBoost) algorithm to classify the four emotions. We evaluated the proposed method on the DEAP and DREAMER datasets and achieved a four-category emotion recognition accuracy of 95.26% and 94.05%, respectively. Additionally, our proposed method reduces the computational cost of EEG emotion recognition by at least 75.45% for computation time and 67.51% for memory occupation. The performance of FCAN-XGBoost outperforms the state-of-the-art four-category model and reduces computational costs without losing classification performance compared with other models.

Affective Neural Responses Sonified through Labeled Correlation Alignment

Sound synthesis refers to the creation of original acoustic signals with broad applications in artistic innovation, such as music creation for games and videos. Nonetheless, machine learning architectures face numerous challenges when learning musical structures from arbitrary corpora. This issue involves adapting patterns borrowed from other contexts to a concrete composition objective. Using Labeled Correlation Alignment (LCA), we propose an approach to sonify neural responses to affective music-listening data, identifying the brain features that are most congruent with the simultaneously extracted auditory features. For dealing with inter/intra-subject variability, a combination of Phase Locking Value and Gaussian Functional Connectivity is employed. The proposed two-step LCA approach embraces a separate coupling stage of input features to a set of emotion label sets using Centered Kernel Alignment. This step is followed by canonical correlation analysis to select multimodal representations with higher relationships. LCA enables physiological explanation by adding a backward transformation to estimate the matching contribution of each extracted brain neural feature set. Correlation estimates and partition quality represent performance measures. The evaluation uses a Vector Quantized Variational AutoEncoder to create an acoustic envelope from the tested Affective Music-Listening database. Validation results demonstrate the ability of the developed LCA approach to generate low-level music based on neural activity elicited by emotions while maintaining the ability to distinguish between the acoustic outputs.

EEG and peripheral markers of viewer ratings: a study of short films

Introduction

Cinema is an important part of modern culture, influencing millions of viewers. Research suggested many models for the prediction of film success, one of them being the use of neuroscientific tools. The aim of our study was to find physiological markers of viewer perception and correlate them to short film ratings given by our subjects. Short films are used as a test case for directors and screenwriters and can be created to raise funding for future projects; however, they have not been studied properly with physiological methods.

Methods

We recorded electroencephalography (18 sensors), facial electromyography (corrugator supercilii and zygomaticus major), photoplethysmography, and skin conductance in 21 participants while watching and evaluating 8 short films (4 dramas and 4 comedies). Also, we used machine learning (CatBoost, SVR) to predict the exact rating of each film (from 1 to 10), based on all physiological indicators. In addition, we classified each film as low or high rated by our subjects (with Logistic Regression, KNN, decision tree, CatBoost, and SVC).

Results

The results showed that ratings did not differ between genres. Corrugator supercilii activity ("frowning" muscle) was larger when watching dramas; whereas zygomaticus major ("smiling" muscle) activity was larger during the watching of comedies. Of all somatic and vegetative markers, only zygomaticus major activity, PNN50, SD1/SD2 (heart rate variability parameters) positively correlated to the film ratings. The EEG engagement indices, beta/(alpha+theta) and beta/alpha correlated positively with the film ratings in the majority of sensors. Arousal (beta_F3 + beta_F4)/(alpha_F3 + alpha_F4), and valence (alpha_F4/beta_F4) - (alpha_F3/beta_F3) indices also correlated positively to film ratings. When we attempted to predict exact ratings, MAPE was 0.55. As for the binary classification, logistic regression yielded the best values (area under the ROC curve = 0.62) than other methods (0.51-0.60).

Discussion

Overall, we revealed EEG and peripheral markers, which reflect viewer ratings and can predict them to a certain extent. In general, high film ratings can reflect a fusion of high arousal and different valence, positive valence being more important. These findings broaden our knowledge about the physiological basis of viewer perception and can be potentially used at the stage of film production.

The consumers' response to product design: a narrative review

This paper reviews the research ideas around consumer response to product design. From the product side, we discuss the most significant design features preferred by average consumers, such as aesthetics and utility. And from the consumer side, we investigate the human factors influencing consumer perceptions. We present the main approaches used to measure the consumer response to product design and summarize the multiple biases that occur during the evaluation. Finally, we present in detail the most commonly used methods to analyze consumer response data and their roles in the design evaluation context. Practitioner Summary: To answer the question: What causes differences in design response? We summarise the research findings related to product design features and human factors. We highlight the biases that can emerge from the measurement approach. And discuss the most common analysis methods used for product design response information.

Influence of Multimodal Emotional Stimulations on Brain Activity: An Electroencephalographic Study

This study aimed to reveal the influence of emotional valence and sensory modality on neural activity in response to multimodal emotional stimuli using scalp EEG. In this study, 20 healthy participants completed the emotional multimodal stimulation experiment for three stimulus modalities (audio, visual, and audio-visual), all of which are from the same video source with two emotional components (pleasure or unpleasure), and EEG data were collected using six experimental conditions and one resting state. We analyzed power spectral density (PSD) and event-related potential (ERP) components in response to multimodal emotional stimuli, for spectral and temporal analysis. PSD results showed that the single modality (audio only/visual only) emotional stimulation PSD differed from multi-modality (audio-visual) in a wide brain and band range due to the changes in modality and not from the changes in emotional degree. The most pronounced N200-to-P300 potential shifts occurred in monomodal rather than multimodal emotional stimulations. This study suggests that emotional saliency and sensory processing efficiency perform a significant role in shaping neural activity during multimodal emotional stimulation, with the sensory modality being more influential in PSD. These findings contribute to our understanding of the neural mechanisms involved in multimodal emotional stimulation.

A Greedy Optimized Intelligent Framework for Early Detection of Alzheimer's Disease Using EEG Signal

Recent researchers have been drawn to the analysis of electroencephalogram (EEG) signals in order to confirm the disease and severity range by viewing the EEG signal which has complicated the dataset. The conventional models such as machine learning, classifiers, and other mathematical models achieved the lowest classification score. The current study proposes to implement a novel deep feature with the best solution for EEG signal analysis and severity specification. A greedy sandpiper-based recurrent neural system (SbRNS) model for predicting Alzheimer's disease (AD) severity has been proposed. The filtered data are used as input for the feature analysis and the severity range is divided into three classes: low, medium, and high. The designed approach was then implemented in the matrix laboratory (MATLAB) system, and the effectiveness score was calculated using key metrics such as precision, recall, specificity, accuracy, and misclassification score. The validation results show that the proposed scheme achieved the best classification outcome.

An Efficient Machine Learning-Based Emotional Valence Recognition Approach Towards Wearable EEG

Emotion artificial intelligence (AI) is being increasingly adopted in several industries such as healthcare and education. Facial expressions and tone of speech have been previously considered for emotion recognition, yet they have the drawback of being easily manipulated by subjects to mask their true emotions. Electroencephalography (EEG) has emerged as a reliable and cost-effective method to detect true human emotions. Recently, huge research effort has been put to develop efficient wearable EEG devices to be used by consumers in out of the lab scenarios. In this work, a subject-dependent emotional valence recognition method is implemented that is intended for utilization in emotion AI applications. Time and frequency features were computed from a single time series derived from the Fp1 and Fp2 channels. Several analyses were performed on the strongest valence emotions to determine the most relevant features, frequency bands, and EEG timeslots using the benchmark DEAP dataset. Binary classification experiments resulted in an accuracy of 97.42% using the alpha band, by that outperforming several approaches from literature by ~3-22%. Multiclass classification gave an accuracy of 95.0%. Feature computation and classification required less than 0.1 s. The proposed method thus has the advantage of reduced computational complexity as, unlike most methods in the literature, only two EEG channels were considered. In addition, minimal features concluded from the thorough analyses conducted in this study were used to achieve state-of-the-art performance. The implemented EEG emotion recognition method thus has the merits of being reliable and easily reproducible, making it well-suited for wearable EEG devices.

Status of deep learning for EEG-based brain-computer interface applications

In the previous decade, breakthroughs in the central nervous system bioinformatics and computational innovation have prompted significant developments in brain-computer interface (BCI), elevating it to the forefront of applied science and research. BCI revitalization enables neurorehabilitation strategies for physically disabled patients (e.g., disabled patients and hemiplegia) and patients with brain injury (e.g., patients with stroke). Different methods have been developed for electroencephalogram (EEG)-based BCI applications. Due to the lack of a large set of EEG data, methods using matrix factorization and machine learning were the most popular. However, things have changed recently because a number of large, high-quality EEG datasets are now being made public and used in deep learning-based BCI applications. On the other hand, deep learning is demonstrating great prospects for solving complex relevant tasks such as motor imagery classification, epileptic seizure detection, and driver attention recognition using EEG data. Researchers are doing a lot of work on deep learning-based approaches in the BCI field right now. Moreover, there is a great demand for a study that emphasizes only deep learning models for EEG-based BCI applications. Therefore, we introduce this study to the recent proposed deep learning-based approaches in BCI using EEG data (from 2017 to 2022). The main differences, such as merits, drawbacks, and applications are introduced. Furthermore, we point out current challenges and the directions for future studies. We argue that this review study will help the EEG research community in their future research.

Stylized faces enhance ERP features used for the detection of emotional responses

For their ease of accessibility and low cost, current Brain-Computer Interfaces (BCI) used to detect subjective emotional and affective states rely largely on electroencephalographic (EEG) signals. Public datasets are available for researchers to design models for affect detection from EEG. However, few designs focus on optimally exploiting the nature of the stimulus elicitation to improve accuracy. The RSVP protocol is used in this experiment to present human faces of emotion to 28 participants while EEG was measured. We found that artificially enhanced human faces with exaggerated, cartoonish visual features significantly improve some commonly used neural correlates of emotion as measured by event-related potentials (ERPs). These images elicit an enhanced N170 component, well known to relate to the facial visual encoding process. Our findings suggest that the study of emotion elicitation could exploit consistent, high detail, AI generated stimuli transformations to study the characteristics of electrical brain activity related to visual affective stimuli. Furthermore, this specific result might be useful in the context of affective BCI design, where a higher accuracy in affect decoding from EEG can improve the experience of a user.

TC-Net: A Transformer Capsule Network for EEG-based emotion recognition

Deep learning has recently achieved remarkable success in emotion recognition based on Electroencephalogram (EEG), in which convolutional neural networks (CNNs) are the mostly used models. However, due to the local feature learning mechanism, CNNs have difficulty in capturing the global contextual information involving temporal domain, frequency domain, intra-channel and inter-channel. In this paper, we propose a Transformer Capsule Network (TC-Net), which mainly contains an EEG Transformer module to extract EEG features and an Emotion Capsule module to refine the features and classify the emotion states. In the EEG Transformer module, EEG signals are partitioned into non-overlapping windows. A Transformer block is adopted to capture global features among different windows, and we propose a novel patch merging strategy named EEG-PatchMerging (EEG-PM) to better extract local features. In the Emotion Capsule module, each channel of the EEG feature maps is encoded into a capsule to better characterize the spatial relationships among multiple features. Experimental results on two popular datasets (i.e., DEAP and DREAMER) demonstrate that the proposed method achieves the state-of-the-art performance in the subject-dependent scenario. Specifically, on DEAP (DREAMER), our TC-Net achieves the average accuracies of 98.76% (98.59%), 98.81% (98.61%) and 98.82% (98.67%) at valence, arousal and dominance dimensions, respectively. Moreover, the proposed TC-Net also shows high effectiveness in multi-state emotion recognition tasks using the popular VA and VAD models. The main limitation of the proposed model is that it tends to obtain relatively low performance in the cross-subject recognition task, which is worthy of further study in the future.

Automatic detection of abnormal EEG signals using multiscale features with ensemble learning

Electroencephalogram (EEG) is an economical and convenient auxiliary test to aid in the diagnosis and analysis of brain-related neurological diseases. In recent years, machine learning has shown great potential in clinical EEG abnormality detection. However, existing methods usually fail to consider the issue of feature redundancy when extracting the relevant EEG features. In addition, the importance of utilizing the patient age information in EEG detection is ignored. In this paper, a new framework is proposed for distinguishing an unknown EEG recording as either normal or abnormal by identifying different types of EEG-derived significant features. In the proposed framework, different hierarchical salient features are extracted using a time-wise multi-scale aggregation strategy, based on a selected group of statistical characteristics calculated from the optimum discrete wavelet transform coefficients. We also fuse the age information with multi-scale features for further improving discrimination. The integrated features are classified using three ensemble learning classifiers, CatBoost, LightGBM, and random forest. Experimental results show that our method with CatBoost classifier can yield superior performance vis-a-vis competing techniques, which indicates the great promise of our methodology in EEG pathology detection.

Deep learning-based self-induced emotion recognition using EEG

Emotion recognition from electroencephalogram (EEG) signals requires accurate and efficient signal processing and feature extraction. Deep learning technology has enabled the automatic extraction of raw EEG signal features that contribute to classifying emotions more accurately. Despite such advances, classification of emotions from EEG signals, especially recorded during recalling specific memories or imagining emotional situations has not yet been investigated. In addition, high-density EEG signal classification using deep neural networks faces challenges, such as high computational complexity, redundant channels, and low accuracy. To address these problems, we evaluate the effects of using a simple channel selection method for classifying self-induced emotions based on deep learning. The experiments demonstrate that selecting key channels based on signal statistics can reduce the computational complexity by 89% without decreasing the classification accuracy. The channel selection method with the highest accuracy was the kurtosis-based method, which achieved accuracies of 79.03% and 79.36% for the valence and arousal scales, respectively. The experimental results show that the proposed framework outperforms conventional methods, even though it uses fewer channels. Our proposed method can be beneficial for the effective use of EEG signals in practical applications.

EEG emotion recognition based on cross-frequency granger causality feature extraction and fusion in the left and right hemispheres

EEG emotion recognition based on Granger causality (GC) brain networks mainly focus on the EEG signal from the same-frequency bands, however, there are still some causality relationships between EEG signals in the cross-frequency bands. Considering the functional asymmetric of the left and right hemispheres to emotional response, this paper proposes an EEG emotion recognition scheme based on cross-frequency GC feature extraction and fusion in the left and right hemispheres. Firstly, we calculate the GC relationship of EEG signals according to the frequencies and hemispheres, and mainly focus on the causality of the cross-frequency EEG signals in left and right hemispheres. Then, to remove the redundant connections of the GC brain network, an adaptive two-stage decorrelation feature extraction scheme is proposed under the condition of maintaining the best emotion recognition performance. Finally, a multi-GC feature fusion scheme is designed to balance the recognition accuracy and feature number of each GC feature, which comprehensively considers the influence of the recognition accuracy and computational complexity. Experimental results on the DEAP emotion dataset show that the proposed scheme can achieve an average accuracy of 84.91% for four classifications, which improved the classification accuracy by up to 8.43% compared with that of the traditional same-frequency band GC features.

Application of Neuroengineering Based on EEG Features in the Industrial Design of Comfort

The smart wheelchair is a service robot that can be used as a means of transportation for the elderly and the disabled. The patients were given an intelligent wheelchair designed by electroencephalogram (EEG), which was used for more than 8 hours and tested continuously for 1 month. By ridit analysis, the difference between the two groups was statistically significant (U = 3.72, P < 0.01). The scores of visual analogue scale (VAS) and joint ground visuality (JGV) in the observation group were significantly better than those in the control group. The modules of physiological function (PF), physical pain (PP), overall health (OH), vitality (VT), social function (SF), emotional function (EF), and mental health (MH) in the SF-36 scores of the two groups were significantly improved (P < 0.05), and the improvement of each module in the observation group was significantly better than that in the control group (P < 0.05). The levels of serum IL-6, IL-10, and superoxide dismutase (SOD) in the two groups were significantly improved (P < 0.05), and the improvement of serum IL-6, IL-10, and SOD in the observation group was significantly better than that in the control group (P < 0.05). It is suggested that neural engineering based on EEG characteristics can be well applied in comfort industrial design.

Intelligent Extraction of Salient Feature From Electroencephalogram Using Redundant Discrete Wavelet Transform

At present, electroencephalogram (EEG) signals play an irreplaceable role in the diagnosis and treatment of human diseases and medical research. EEG signals need to be processed in order to reduce the adverse effects of irrelevant physiological process interference and measurement noise. Wavelet transform (WT) can provide a time-frequency representation of a dynamic process, and it has been widely utilized in salient feature analysis of EEG. In this paper, we investigate the problem of translation variability (TV) in discrete wavelet transform (DWT), which causes degradation of time-frequency localization. It will be verified through numerical simulations that TV is caused by downsampling operations in decomposition process of DWT. The presence of TV may cause severe distortions of features in wavelet subspaces. However, this phenomenon has not attracted much attention in the scientific community. Redundant discrete wavelet transform (RDWT) is derived by eliminating the downsampling operation. RDWT enjoys the attractive merit of translation invariance. RDWT shares the same time-frequency pattern with that of DWT. The discrete delta impulse function is used to test the time-frequency response of DWT and RDWT in wavelet subspaces. The results show that DWT is very sensitive to the translation of delta impulse function, while RDWT keeps the decomposition results unchanged. This conclusion has also been verified again in decomposition of actual EEG signals. In conclusion, to avoid possible distortions of features caused by translation sensitivity in DWT, we recommend the use of RDWT with more stable performance in BCI research and clinical applications.

Multi-Classifier Fusion Based on MI–SFFS for Cross-Subject Emotion Recognition

With the widespread use of emotion recognition, cross-subject emotion recognition based on EEG signals has become a hot topic in affective computing. Electroencephalography (EEG) can be used to detect the brain’s electrical activity associated with different emotions. The aim of this research is to improve the accuracy by enhancing the generalization of features. A Multi-Classifier Fusion method based on mutual information with sequential forward floating selection (MI_SFFS) is proposed. The dataset used in this paper is DEAP, which is a multi-modal open dataset containing 32 EEG channels and multiple other physiological signals. First, high-dimensional features are extracted from 15 EEG channels of DEAP after using a 10 s time window for data slicing. Second, MI and SFFS are integrated as a novel feature-selection method. Then, support vector machine (SVM), k-nearest neighbor (KNN) and random forest (RF) are employed to classify positive and negative emotions to obtain the output probabilities of classifiers as weighted features for further classification. To evaluate the model performance, leave-one-out cross-validation is adopted. Finally, cross-subject classification accuracies of 0.7089, 0.7106 and 0.7361 are achieved by the SVM, KNN and RF classifiers, respectively. The results demonstrate the feasibility of the model by splicing different classifiers’ output probabilities as a portion of the weighted features.

Spatial-temporal Network for Fine-grained-level Emotion EEG Recognition

Electroencephalogram (EEG)-based affective computing brain-computer interfaces provide the capability for machines to understand human intentions. In practice, people are more concerned with the strength of a certain emotional state over a short period of time, which was called as fine-grained-level emotion in this paper. In this study, we built a fine-grained-level emotion EEG dataset that contains two coarse-grained emotions and four corresponding fine-grained-level emotions. To fully extract the features of the EEG signals, we proposed a corresponding fine-grained emotion EEG network (FG-emotionNet) for spatial-temporal feature extraction. Each feature extraction layer is linked to raw EEG signals to alleviate overfitting and ensure that the spatial features of each scale can be extracted from the raw signals. Moreover, all previous scale features are fused before the current spatial-feature layer to enhance the scale features in the spatial block. Additionally, long short-term memory is adopted as the temporal block to extract the temporal features based on spatial features and classify the category of fine-grained emotions. Subject-dependent and cross-session experiments demonstrated that the performance of the proposed method is superior to that of the representative methods in emotion recognition and similar structure methods with proposed method.

Graphical representation learning-based approach for automatic classification of electroencephalogram signals in depression

Depression is a major depressive disorder characterized by persistent sadness and a sense of worthlessness, as well as a loss of interest in pleasurable activities, which leads to a variety of physical and emotional problems. It is a worldwide illness that affects millions of people and should be detected at an early stage to prevent negative effects on an individual's life. Electroencephalogram (EEG) is a non-invasive technique for detecting depression that analyses brain signals to determine the current mental state of depressed subjects. In this study, we propose a method for automatic feature extraction to detect depression by first constructing a graph from the dataset where the nodes represent the subjects in the dataset and where the edge weights obtained using the Euclidean distance reflect the relationship between them. The Node2vec algorithmic framework is then used to compute feature representations for nodes in a graph in the form of node embeddings ensuring that similar nodes in the graph remain near in the embedding. These node embeddings act as useful features which can be directly used by classification algorithms to determine whether a subject is depressed thus reducing the effort required for manual handcrafted feature extraction. To combine the features collected from the multiple channels of the EEG data, the method proposes three types of fusion methods: graph-level fusion, feature-level fusion, and decision-level fusion. The proposed method is tested on three publicly available datasets with 3, 20, and 128 channels, respectively, and compared to five state-of-the-art methods. The results show that the proposed method detects depression effectively with a peak accuracy of 0.933 in decision-level fusion, which is the highest among the state-of-the-art methods.

Bio-Signals in Medical Applications and Challenges Using Artificial Intelligence

Artificial Intelligence (AI) has broadly connected the medical field at various levels of diagnosis based on the congruous data generated. Different types of bio-signal can be used to monitor a patient’s condition and in decision making. Medical equipment uses signals to communicate information to care staff. AI algorithms and approaches will help to predict health problems and check the health status of organs, while AI prediction, classification, and regression algorithms are helping the medical industry to protect from health hazards. The early prediction and detection of health conditions will guide people to stay healthy. This paper represents the scope of bio-signals using AI in the medical area. It will illustrate possible case studies relevant to bio-signals generated through IoT sensors. The bio-signals that retrospectively occur are discussed, and the new challenges of medical diagnosis using bio-signals are identified.

EEG Channel Correlation Based Model for Emotion Recognition

Emotion recognition using Artificial Intelligence (AI) is a fundamental prerequisite to improve Human-Computer Interaction (HCI). Recognizing emotion from Electroencephalogram (EEG) has been globally accepted in many applications such as intelligent thinking, decision-making, social communication, feeling detection, affective computing, etc. Nevertheless, due to having too low amplitude variation related to time on EEG signal, the proper recognition of emotion from this signal has become too challenging. Usually, considerable effort is required to identify the proper feature or feature set for an effective feature-based emotion recognition system. To extenuate the manual human effort of feature extraction, we proposed a deep machine-learning-based model with Convolutional Neural Network (CNN). At first, the one-dimensional EEG data were converted to Pearson's Correlation Coefficient (PCC) featured images of channel correlation of EEG sub-bands. Then the images were fed into the CNN model to recognize emotion. Two protocols were conducted, namely, protocol-1 to identify two levels and protocol-2 to recognize three levels of valence and arousal that demonstrate emotion. We investigated that only the upper triangular portion of the PCC featured images reduced the computational complexity and size of memory without hampering the model accuracy. The maximum accuracy of 78.22% on valence and 74.92% on arousal were obtained using the internationally authorized DEAP dataset.