EEG-based emotion recognition in an immersive virtual reality environment: From local activity to brain network features

Fatigue Characterization of EEG Brain Networks Under Mixed Reality Stereo Vision

Mixed Reality (MR) technology possesses profound and extensive potential across a multitude of domains, including, but not limited to industry, healthcare, and education. However, prolonged use of MR devices to watch stereoscopic content may lead to visual fatigue. Since visual fatigue involves multiple brain regions, our study aims to explore the topological characteristics of brain networks derived from electroencephalogram (EEG) data. Because the Phase-Locked Value (PLV) is capable of effectively measuring the phase synchronization relationship between brain regions, it was calculated between all pairs of channels in both comfort and fatigue states. Subsequently, a sparse brain network was constructed based on PLV by applying an appropriate threshold. The node properties (betweenness centrality, clustering coefficient, node efficiency) and edge properties (characteristic path length) were calculated based on the corresponding brain network within specific frequency bands for both comfort and fatigue states. In analyzing the PLV of brain connectivity in comfort and fatigue states, a notable enhancement in brain connectivity is observed within the alpha, theta, and delta frequency bands during fatigue status. By analyzing the node and edge properties of brain networks, it is evident that the mean values of these properties in the fatigue state were higher than those in the comfort state. By analyzing the node and edge properties at a local level, the average difference in betweenness centrality, clustering coefficients, and nodal efficiency across the three EEG frequency bands was computed to find significant brain regions. The main findings are as follows: Betweenness centrality primarily differs in frontal and parietal regions, with minor involvement in temporal and central regions. The clustering Coefficient mainly varies in the frontal region, with slight differences being seen in the temporal and occipital regions. Nodal efficiency primarily varies in the frontal, temporal, and central regions, with minor differences being seen in the parietal and occipital regions. Edge property analysis indicates that there is a higher occurrence of long-distance connections among brain regions during the fatigue state, which reflects a loss of synaptic transmission efficiency on a global level. Our study plays a crucial role in understanding the neural mechanisms underlying visual fatigue, potentially providing insights that could be applied to high-demand cognitive fields where prolonged use of MR devices leads to visual fatigue.

The association between social rewards and anxiety: Links from neurophysiological analysis in virtual reality and social interaction game

Individuals' affective experience can be intricate, influenced by various factors including monetary rewards and social factors during social interaction. However, within this array of factors, divergent evidence has been considered as potential contributors to social anxiety. To gain a better understanding of the specific factors associated with anxiety during social interaction, we combined a social interaction task with neurophysiological recordings obtained through an anxiety-elicitation task conducted in a Virtual Reality (VR) environment. Employing inter-subject representational similarity analysis (ISRSA), we explored the potential linkage between individuals' anxiety neural patterns and their affective experiences during social interaction. Our findings suggest that, after controlling for other factors, the influence of the partner's emotional cues on individuals' affective experiences is specifically linked to their neural pattern of anxiety. This indicates that the emergence of anxiety during social interaction may be particularly associated with the emotional cues provided by the social partner, rather than individuals' own reward or prediction errors during social interaction. These results provide further support for the cognitive theory of social anxiety and extend the application of VR in future cognitive and affective studies.

A review of artificial intelligence methods enabled music-evoked EEG emotion recognition and their applications

Music is an archaic form of emotional expression and arousal that can induce strong emotional experiences in listeners, which has important research and practical value in related fields such as emotion regulation. Among the various emotion recognition methods, the music-evoked emotion recognition method utilizing EEG signals provides real-time and direct brain response data, playing a crucial role in elucidating the neural mechanisms underlying music-induced emotions. Artificial intelligence technology has greatly facilitated the research on the recognition of music-evoked EEG emotions. AI algorithms have ushered in a new era for the extraction of characteristic frequency signals and the identification of novel feature signals. The robust computational capabilities of AI have provided fresh perspectives for the development of innovative quantitative models of emotions, tailored to various emotion recognition paradigms. The discourse surrounding AI algorithms in the context of emotional classification models is gaining momentum, with their applications in music therapy, neuroscience, and social activities increasingly coming under the spotlight. Through an in-depth analysis of the complete process of emotion recognition induced by music through electroencephalography (EEG) signals, we have systematically elucidated the influence of AI on pertinent research issues. This analysis offers a trove of innovative approaches that could pave the way for future research endeavors.

Effective Emotion Recognition by Learning Discriminative Graph Topologies in EEG Brain Networks

Multichannel electroencephalogram (EEG) is an array signal that represents brain neural networks and can be applied to characterize information propagation patterns for different emotional states. To reveal these inherent spatial graph features and increase the stability of emotion recognition, we propose an effective emotion recognition model that performs multicategory emotion recognition with multiple emotion-related spatial network topology patterns (MESNPs) by learning discriminative graph topologies in EEG brain networks. To evaluate the performance of our proposed MESNP model, we conducted single-subject and multisubject four-class classification experiments on two public datasets, MAHNOB-HCI and DEAP. Compared with existing feature extraction methods, the MESNP model significantly enhances the multiclass emotional classification performance in the single-subject and multisubject conditions. To evaluate the online version of the proposed MESNP model, we designed an online emotion monitoring system. We recruited 14 participants to conduct the online emotion decoding experiments. The average online experimental accuracy of the 14 participants was 84.56%, indicating that our model can be applied in affective brain-computer interface (aBCI) systems. The offline and online experimental results demonstrate that the proposed MESNP model effectively captures discriminative graph topology patterns and significantly improves emotion classification performance. Moreover, the proposed MESNP model provides a new scheme for extracting features from strongly coupled array signals.

Small Urban Green Spaces: Insights into Perception, Preference, and Psychological Well-being in a Densely Populated Areas of Tehran, Iran

In metropolitan areas worldwide, abandoned properties are prevalent, prompting a need for small urban green spaces (SUGS) to meet the growing demand. Understanding residents' preferences and perceptions of transformed spaces is vital for effective urban design. This study delves into residents' preferences and perceptions regarding the transformation of such spaces into SUGS and their impact on psychological well-being. By examining how these preferences and perceived health benefits shape the value of transformed spaces, the research aims to inform effective urban design strategies. The participants underwent visual stimulation, with psychological reactions recorded through Electroencephalogram (EEG) readings and assessed via Questionnaire. Machine learning techniques analyzed EEG sub-band data, achieving an average accuracy of 92.8% when comparing leftover and designed spaces. Results revealed that different types of transformed spaces provoke distinct physiological and preference responses. Specifically, viewing SUGS was associated with significant changes in gamma wave power, suggesting a correlation between enhanced gamma activity and increased feelings of empathy. Moreover, participants also reported enhanced comfort, relaxation, and overall mood, and a strong preference for SUGS over untransformed spaces, emphasizing the value placed on these areas for their health benefits. This research highlights the positive impact of even SUGS on mental health, using EEG data to assess emotional states triggered by urban spaces. The study concludes with a call for further research to investigate the long-term benefits of SUGS on well-being, alongside an exploration of the gamma band as a neural marker for emotional restoration in urban green spaces. This research highlights the crucial role of urban design in fostering psychological well-being through the strategic development of green spaces, suggesting a paradigm shift toward more inclusive, health-promoting urban environments.

Constructing Bodily Emotion Maps Based on High-Density Body Surface Potentials for Psychophysiological Computing

Emotion is a complex physiological and psychological activity, accompanied by subjective physiological sensations and objective physiological changes. The body sensation map describes the changes in body sensation associated with emotion in a topographic manner, but it relies on subjective evaluations from participants. Physiological signals are a more reliable measure of emotion, but most research focuses on the central nervous system, neglecting the importance of the peripheral nervous system. In this study, a body surface potential mapping (BSPM) system was constructed, and an experiment was designed to induce emotions and obtain high-density body surface potential information under negative and non-negative emotions. Then, by constructing and analyzing the functional connectivity network of BSPs, the high-density electrophysiological characteristics are obtained and visualized as bodily emotion maps. The results showed that the functional connectivity network of BSPs under negative emotions had denser connections, and emotion maps based on local clustering coefficient (LCC) are consistent with BSMs under negative emotions. in addition, our features can classify negative and non-negative emotions with the highest classification accuracy of 80.77%. In conclusion, this study constructs an emotion map based on high-density BSPs, which offers a novel approach to psychophysiological computing.

The role of attention in immersion: The two-competitor model

Currently, we face an exponentially increasing interest in immersion, especially sensory-driven immersion, mainly due to the rapid development of ideas and business models centered around a digital virtual universe as well as the increasing availability of affordable immersive technologies for education, communication, and entertainment. However, a clear definition of 'immersion', in terms of established neurocognitive concepts and measurable properties, remains elusive, slowing research on the human side of immersive interfaces. To address this problem, we propose a conceptual, taxonomic model of attention in immersion. We argue (a) modeling immersion theoretically as well as studying immersion experimentally requires a detailed characterization of the role of attention in immersion, even though (b) attention, while necessary, cannot be a sufficient condition for defining immersion. Our broader goal is to characterize immersion in terms that will be compatible with established psychophysiolgical measures that could then in principle be used for the assessment and eventually the optimization of an immersive experience. We start from the perspective that immersion requires the projection of attention to an induced reality, and build on accepted taxonomies of different modes of attention for the development of our two-competitor model. The two-competitor model allows for a quantitative implementation and has an easy graphical interpretation. It helps to highlight the important link between different modes of attention and affect in studying immersion.

Decoding emotion with phase-amplitude fusion features of EEG functional connectivity network

Decoding emotional neural representations from the electroencephalographic (EEG)-based functional connectivity network (FCN) is of great scientific importance for uncovering emotional cognition mechanisms and developing harmonious human-computer interactions. However, existing methods mainly rely on phase-based FCN measures (e.g., phase locking value [PLV]) to capture dynamic interactions between brain oscillations in emotional states, which fail to reflect the energy fluctuation of cortical oscillations over time. In this study, we initially examined the efficacy of amplitude-based functional networks (e.g., amplitude envelope correlation [AEC]) in representing emotional states. Subsequently, we proposed an efficient phase-amplitude fusion framework (PAF) to fuse PLV and AEC and used common spatial pattern (CSP) to extract fused spatial topological features from PAF for multi-class emotion recognition. We conducted extensive experiments on the DEAP and MAHNOB-HCI datasets. The results showed that: (1) AEC-derived discriminative spatial network topological features possess the ability to characterize emotional states, and the differential network patterns of AEC reflect dynamic interactions in brain regions associated with emotional cognition. (2) The proposed fusion features outperformed other state-of-the-art methods in terms of classification accuracy for both datasets. Moreover, the spatial filter learned from PAF is separable and interpretable, enabling a description of affective activation patterns from both phase and amplitude perspectives.

Dynamic Neural Patterns of Human Emotions in Virtual Reality: Insights from EEG Microstate Analysis

Emotions play a crucial role in human life and affect mental health. Understanding the neural patterns associated with emotions is essential. Previous studies carried out some exploration of the neural features of emotions, but most have designed experiments in two-dimensional (2D) environments, which differs from real-life scenarios. To create a more real environment, this study investigated emotion-related brain activity using electroencephalography (EEG) microstate analysis in a virtual reality (VR) environment. We recruited 42 healthy volunteers to participate in our study. We explored the dynamic features of different emotions, and four characteristic microstates were analyzed. In the alpha band, microstate A exhibited a higher occurrence in both negative and positive emotions than in neutral emotions. Microstate C exhibited a prolonged duration of negative emotions compared to positive emotions, and a higher occurrence was observed in both microstates C and D during positive emotions. Notably, a unique transition pair was observed between microstates B and C during positive emotions, whereas a unique transition pair was observed between microstates A and D during negative emotions. This study emphasizes the potential of integrating virtual reality (VR) and EEG to facilitate experimental design. Furthermore, this study enhances our comprehension of neural activities during various emotional states.

Classification of EEG Signals Based on Sparrow Search Algorithm-Deep Belief Network for Brain-Computer Interface

In brain-computer interface (BCI) systems, challenges are presented by the recognition of motor imagery (MI) brain signals. Established recognition approaches have achieved favorable performance from patterns like SSVEP, AEP, and P300, whereas the classification methods for MI need to be improved. Hence, seeking a classification method that exhibits high accuracy and robustness for application in MI-BCI systems is essential. In this study, the Sparrow search algorithm (SSA)-optimized Deep Belief Network (DBN), called SSA-DBN, is designed to recognize the EEG features extracted by the Empirical Mode Decomposition (EMD). The performance of the DBN is enhanced by the optimized hyper-parameters obtained through the SSA. Our method's efficacy was tested on three datasets: two public and one private. Results indicate a relatively high accuracy rate, outperforming three baseline methods. Specifically, on the private dataset, our approach achieved an accuracy of 87.83%, marking a significant 10.38% improvement over the standard DBN algorithm. For the BCI IV 2a dataset, we recorded an accuracy of 86.14%, surpassing the DBN algorithm by 9.33%. In the SMR-BCI dataset, our method attained a classification accuracy of 87.21%, which is 5.57% higher than that of the conventional DBN algorithm. This study demonstrates enhanced classification capabilities in MI-BCI, potentially contributing to advancements in the field of BCI.

Altered local spontaneous brain activity pattern in children with right-eye amblyopia of varying degrees: evidence from fMRI

PURPOSE

To investigate the abnormal changes of local brain activity in children with right-eye amblyopia of varying degrees.

METHODS

Data of resting-state functional magnetic resonance imaging were collected from 16 children with severe amblyopia, 17 children with mild to moderate amblyopia, and 15 children with normal binocular vision. Local brain activity was analyzed using the amplitude of low-frequency fluctuations (ALFF) and regional homogeneity (ReHo).

RESULTS

There were extensive ALFF differences among the three groups in 10 brain regions. There were extensive differences in ReHo among the three groups in 11 brain regions. The ALFF and ReHo of the right orbital part of the middle frontal gyrus displayed a significantly positive correlation with the best-corrected visual acuity of the right eye, respectively. The ALFF value and ReHo value of the right orbital part of the middle frontal gyrus followed the pattern of normal control < mild to moderate amblyopia < severe amblyopia.

CONCLUSION

This study demonstrated that there were changes in specific patterns of ALFF and ReHo in children with right-eye amblyopia of different degrees in brain regions performing visual sensorimotor and attentional control functions.

Emo-regulator: An emotion-regulation training system fusing virtual reality and EEG-based neurofeedback

Good emotion-regulation ability (ERA) is a vital sign of psychological health; conversely, emotion dysregulation may lead to mental or neurological disorders, such as anxiety disorders and depression. This study developed an emotion-regulation training system, Emo-regulator, fusing virtual reality (VR) and EEG-based neurofeedback to enhance subjects' ability to down-regulate negative emotions. Emo-regulator first elicited negative emotions in subjects through VR scenarios and then asked them to regulate emotions using cognitive reappraisal to change the emotional responses elicited by the VRs. Meanwhile, EEG signals from the subjects were collected and analyzed in real time by machine learning to predict the emotional states of the subjects (negative or positive). Emo-regulator changed the VR scenarios according to the prediction results and completed the feedback. Eight subjects used Emo-regulator for two weeks, and the results showed it could help the subjects improve their emotion regulation, and its usability is above average.Clinical Relevance-Emo-regulator can help subjects improve their ability to down-regulate negative emotions and increase the frequency of cognitive reappraisal use during emotion regulation.

An Adaptive Joint CCA-ICA Method for Ocular Artifact Removal and its Application to Emotion Classification

BACKGROUND

The quality of Electroencephalogram (EEG) signals is critical for revealing the neural mechanism of emotions. However, ocular artifacts decreased the signal to noise ratio (SNR) and covered the inherent cognitive component of EEGs, which pose a great challenge in neuroscience research.

NEW METHOD

We proposed a novel unsupervised learning algorithm to adaptively remove the ocular artifacts by combining canonical correlation analysis (CCA), independent component analysis (ICA), higher-order statistics, empirical mode decomposition (EMD), and wavelet denoising techniques. Specifically, the combination of CCA and ICA aimed to improve the quality of source separation, while the higher-order statistics further located the source of ocular artifacts. Subsequently, these noised sources were further corrected by EMD and wavelet denoising to improve SNR of EEG signals.

RESULTS

We evaluated the performance of our proposed method with simulation studies and real EEG applications. The results of simulation study showed our proposed method could significantly improve the quality of signals under almost all noise conditions compared to four state-of-art methods. Consistently, the experiments of real EEG applications showed that the proposed methods could efficiently restrict the components of ocular artifacts and preserve the inherent information of cognition processing to improve the reliability of related analysis such as power spectral density (PSD) and emotion recognition.

COMPARISON WITH EXISTING METHODS

Our proposed model outperforms the comparative methods in EEG recovery, which further improve the application performance such as PSD analysis and emotion recognition.

CONCLUSIONS

The superior performance of our proposed method suggests that it is promising for removing ocular artifacts from EEG signals, which offers an efficient EEG preprocessing technology for the development of brain computer interface such as emotion recognition.

Musical Emotions Recognition Using Entropy Features and Channel Optimization Based on EEG

The dynamic of music is an important factor to arouse emotional experience, but current research mainly uses short-term artificial stimulus materials, which cannot effectively awaken complex emotions and reflect their dynamic brain response. In this paper, we used three long-term stimulus materials with many dynamic emotions inside: the "Waltz No. 2" containing pleasure and excitement, the "No. 14 Couplets" containing excitement, briskness, and nervousness, and the first movement of "Symphony No. 5 in C minor" containing passion, relaxation, cheerfulness, and nervousness. Approximate entropy (ApEn) and sample entropy (SampEn) were applied to extract the non-linear features of electroencephalogram (EEG) signals under long-term dynamic stimulation, and the K-Nearest Neighbor (KNN) method was used to recognize emotions. Further, a supervised feature vector dimensionality reduction method was proposed. Firstly, the optimal channel set for each subject was obtained by using a particle swarm optimization (PSO) algorithm, and then the number of times to select each channel in the optimal channel set of all subjects was counted. If the number was greater than or equal to the threshold, it was a common channel suitable for all subjects. The recognition results based on the optimal channel set demonstrated that each accuracy of two categories of emotions based on "Waltz No. 2" and three categories of emotions based on "No. 14 Couplets" was generally above 80%, respectively, and the recognition accuracy of four categories based on the first movement of "Symphony No. 5 in C minor" was about 70%. The recognition accuracy based on the common channel set was about 10% lower than that based on the optimal channel set, but not much different from that based on the whole channel set. This result suggested that the common channel could basically reflect the universal features of the whole subjects while realizing feature dimension reduction. The common channels were mainly distributed in the frontal lobe, central region, parietal lobe, occipital lobe, and temporal lobe. The channel number distributed in the frontal lobe was greater than the ones in other regions, indicating that the frontal lobe was the main emotional response region. Brain region topographic map based on the common channel set showed that there were differences in entropy intensity between different brain regions of the same emotion and the same brain region of different emotions. The number of times to select each channel in the optimal channel set of all 30 subjects showed that the principal component channels representing five brain regions were Fp1/F3 in the frontal lobe, CP5 in the central region, Pz in the parietal lobe, O2 in the occipital lobe, and T8 in the temporal lobe, respectively.

Use of Differential Entropy for Automated Emotion Recognition in a Virtual Reality Environment with EEG Signals

Emotion recognition is one of the most important issues in human–computer interaction (HCI), neuroscience, and psychology fields. It is generally accepted that emotion recognition with neural data such as electroencephalography (EEG) signals, functional magnetic resonance imaging (fMRI), and near-infrared spectroscopy (NIRS) is better than other emotion detection methods such as speech, mimics, body language, facial expressions, etc., in terms of reliability and accuracy. In particular, EEG signals are bioelectrical signals that are frequently used because of the many advantages they offer in the field of emotion recognition. This study proposes an improved approach for EEG-based emotion recognition on a publicly available newly published dataset, VREED. Differential entropy (DE) features were extracted from four wavebands (theta 4–8 Hz, alpha 8–13 Hz, beta 13–30 Hz, and gamma 30–49 Hz) to classify two emotional states (positive/negative). Five classifiers, namely Support Vector Machine (SVM), k-Nearest Neighbor (kNN), Naïve Bayesian (NB), Decision Tree (DT), and Logistic Regression (LR) were employed with DE features for the automated classification of two emotional states. In this work, we obtained the best average accuracy of 76.22% ± 2.06 with the SVM classifier in the classification of two states. Moreover, we observed from the results that the highest average accuracy score was produced with the gamma band, as previously reported in studies in EEG-based emotion recognition.

EEG-Based Identification of Emotional Neural State Evoked by Virtual Environment Interaction

Classifying emotional states is critical for brain–computer interfaces and psychology-related domains. In previous studies, researchers have tried to identify emotions using neural data such as electroencephalography (EEG) signals or brain functional magnetic resonance imaging (fMRI). In this study, we propose a machine learning framework for emotion state classification using EEG signals in virtual reality (VR) environments. To arouse emotional neural states in brain signals, we provided three VR stimuli scenarios to 15 participants. Fifty-four features were extracted from the collected EEG signals under each scenario. To find the optimal classification in our research design, three machine learning algorithms (XGBoost classifier, support vector classifier, and logistic regression) were applied. Additionally, various class conditions were used in machine learning classifiers to validate the performance of our framework. To evaluate the classification performance, we utilized five evaluation metrics (precision, recall, f1-score, accuracy, and AUROC). Among the three classifiers, the XGBoost classifiers showed the best performance under all experimental conditions. Furthermore, the usability of features, including differential asymmetry and frequency band pass categories, were checked from the feature importance of XGBoost classifiers. We expect that our framework can be applied widely not only to psychological research but also to mental health-related issues.