1
|
Li C, Li P, Zhang Y, Li N, Si Y, Li F, Cao Z, Chen H, Chen B, Yao D, Xu P. Effective Emotion Recognition by Learning Discriminative Graph Topologies in EEG Brain Networks. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:10258-10272. [PMID: 37022389 DOI: 10.1109/tnnls.2023.3238519] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Multichannel electroencephalogram (EEG) is an array signal that represents brain neural networks and can be applied to characterize information propagation patterns for different emotional states. To reveal these inherent spatial graph features and increase the stability of emotion recognition, we propose an effective emotion recognition model that performs multicategory emotion recognition with multiple emotion-related spatial network topology patterns (MESNPs) by learning discriminative graph topologies in EEG brain networks. To evaluate the performance of our proposed MESNP model, we conducted single-subject and multisubject four-class classification experiments on two public datasets, MAHNOB-HCI and DEAP. Compared with existing feature extraction methods, the MESNP model significantly enhances the multiclass emotional classification performance in the single-subject and multisubject conditions. To evaluate the online version of the proposed MESNP model, we designed an online emotion monitoring system. We recruited 14 participants to conduct the online emotion decoding experiments. The average online experimental accuracy of the 14 participants was 84.56%, indicating that our model can be applied in affective brain-computer interface (aBCI) systems. The offline and online experimental results demonstrate that the proposed MESNP model effectively captures discriminative graph topology patterns and significantly improves emotion classification performance. Moreover, the proposed MESNP model provides a new scheme for extracting features from strongly coupled array signals.
Collapse
|
2
|
Gunda NK, Khalaf MI, Bhatnagar S, Quraishi A, Gudala L, Venkata AKP, Alghayadh FY, Alsubai S, Bhatnagar V. Lightweight attention mechanisms for EEG emotion recognition for brain computer interface. J Neurosci Methods 2024; 410:110223. [PMID: 39032522 DOI: 10.1016/j.jneumeth.2024.110223] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2024] [Revised: 06/18/2024] [Accepted: 07/17/2024] [Indexed: 07/23/2024]
Abstract
BACKGROUND In the realm of brain-computer interfaces (BCI), identifying emotions from electroencephalogram (EEG) data is a difficult endeavor because of the volume of data, the intricacy of the signals, and the several channels that make up the signals. NEW METHODS Using dual-stream structure scaling and multiple attention mechanisms (LDMGEEG), a lightweight network is provided to maximize the accuracy and performance of EEG-based emotion identification. Reducing the number of computational parameters while maintaining the current level of classification accuracy is the aim. This network employs a symmetric dual-stream architecture to assess separately time-domain and frequency-domain spatio-temporal maps constructed using differential entropy features of EEG signals as inputs. RESULT The experimental results show that after significantly lowering the number of parameters, the model achieved the best possible performance in the field, with a 95.18 % accuracy on the SEED dataset. COMPARISON WITH EXISTING METHODS Moreover, it reduced the number of parameters by 98 % when compared to existing models. CONCLUSION The proposed method distinct channel-time/frequency-space multiple attention and post-attention methods enhance the model's ability to aggregate features and result in lightweight performance.
Collapse
Affiliation(s)
- Naresh Kumar Gunda
- Information Technology Management, Campbellsville Univeristy, Campbellsville, KY, United States.
| | - Mohammed I Khalaf
- Department of computer science, Al Maarif University College, Al Anbar 31001, Iraq.
| | - Shaleen Bhatnagar
- Department of Computer Science and Engineering, Manipal Institute of Technology Bengaluru, Manipal Academy of Higher Education, Manipal, India.
| | - Aadam Quraishi
- M. D. Research, Interventional Treatment Institute, Al Anbar, TX, United States.
| | | | | | - Faisal Yousef Alghayadh
- Computer Science and Information Systems Department, College of Applied Sciences, AlMaarefa University, Riyadh, Saudi Arabia.
| | - Shtwai Alsubai
- Department of Computer Science, College of Computer Engineering and Sciences in Al-Kharj, Prince Sattam bin Abdulaziz University, P.O. Box 151, Al-Kharj 11942, Saudi Arabia.
| | - Vaibhav Bhatnagar
- Department of Computer Applications, Manipal University Jaipur, India.
| |
Collapse
|
3
|
Hamzah HA, Abdalla KK. EEG-based emotion recognition systems; comprehensive study. Heliyon 2024; 10:e31485. [PMID: 38818173 PMCID: PMC11137547 DOI: 10.1016/j.heliyon.2024.e31485] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2024] [Accepted: 05/16/2024] [Indexed: 06/01/2024] Open
Abstract
Emotion recognition technology through EEG signal analysis is currently a fundamental concept in artificial intelligence. This recognition has major practical implications in emotional health care, human-computer interaction, and so on. This paper provides a comprehensive study of different methods for extracting electroencephalography (EEG) features for emotion recognition from four different perspectives, including time domain features, frequency domain features, time-frequency features, and nonlinear features. We summarize the current pattern recognition methods adopted in most related works, and with the rapid development of deep learning (DL) attracting the attention of researchers in this field, we pay more attention to deep learning-based studies and analyse the characteristics, advantages, disadvantages, and applicable scenarios. Finally, the current challenges and future development directions in this field were summarized. This paper can help novice researchers in this field gain a systematic understanding of the current status of emotion recognition research based on EEG signals and provide ideas for subsequent related research.
Collapse
Affiliation(s)
- Hussein Ali Hamzah
- Electrical Engineering Department, College of Engineering, University of Babylon, Iraq
| | - Kasim K. Abdalla
- Electrical Engineering Department, College of Engineering, University of Babylon, Iraq
| |
Collapse
|
4
|
Hu L, Tan C, Xu J, Qiao R, Hu Y, Tian Y. Decoding emotion with phase-amplitude fusion features of EEG functional connectivity network. Neural Netw 2024; 172:106148. [PMID: 38309138 DOI: 10.1016/j.neunet.2024.106148] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2023] [Revised: 12/20/2023] [Accepted: 01/23/2024] [Indexed: 02/05/2024]
Abstract
Decoding emotional neural representations from the electroencephalographic (EEG)-based functional connectivity network (FCN) is of great scientific importance for uncovering emotional cognition mechanisms and developing harmonious human-computer interactions. However, existing methods mainly rely on phase-based FCN measures (e.g., phase locking value [PLV]) to capture dynamic interactions between brain oscillations in emotional states, which fail to reflect the energy fluctuation of cortical oscillations over time. In this study, we initially examined the efficacy of amplitude-based functional networks (e.g., amplitude envelope correlation [AEC]) in representing emotional states. Subsequently, we proposed an efficient phase-amplitude fusion framework (PAF) to fuse PLV and AEC and used common spatial pattern (CSP) to extract fused spatial topological features from PAF for multi-class emotion recognition. We conducted extensive experiments on the DEAP and MAHNOB-HCI datasets. The results showed that: (1) AEC-derived discriminative spatial network topological features possess the ability to characterize emotional states, and the differential network patterns of AEC reflect dynamic interactions in brain regions associated with emotional cognition. (2) The proposed fusion features outperformed other state-of-the-art methods in terms of classification accuracy for both datasets. Moreover, the spatial filter learned from PAF is separable and interpretable, enabling a description of affective activation patterns from both phase and amplitude perspectives.
Collapse
Affiliation(s)
- Liangliang Hu
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; West China Institute of Children's Brain and Cognition, Chongqing University of Education, Chongqing 400065, China.
| | - Congming Tan
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China.
| | - Jiayang Xu
- School of Bioinformatics, Chongqing University of Posts and Telecommunications, Chongqing 400065, China.
| | - Rui Qiao
- School of Bioinformatics, Chongqing University of Posts and Telecommunications, Chongqing 400065, China.
| | - Yilin Hu
- School of Bioinformatics, Chongqing University of Posts and Telecommunications, Chongqing 400065, China.
| | - Yin Tian
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; School of Bioinformatics, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; Institute for Advanced Sciences, Chongqing University of Posts and Telecommunications, Chongqing 400065, China; Chongqing Institute for Brain and Intelligence, Guangyang Bay Laboratory, Chongqing 400064, China.
| |
Collapse
|
5
|
Hara T, Hamano M, Ho BQ, Ota J, Yoshimoto Y, Arimitsu N. Method for analyzing sequential services using EEG: Micro-meso analysis of emotional changes in real flight service. Physiol Behav 2023; 272:114359. [PMID: 37769860 DOI: 10.1016/j.physbeh.2023.114359] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Revised: 09/15/2023] [Accepted: 09/25/2023] [Indexed: 10/03/2023]
Abstract
Capturing customers' emotional changes in sequential service should be realized using physiological measurements to assess customer delight. Questionnaire-based customer surveys may miss significant and dissipating emotional responses. This study developed a micro‑meso analysis method of capturing emotional changes for sequential service using electroencephalograph (EEG) measurement, dealing with both service encounters (micro-level) and servicescape (meso‑level) over a couple of hours. Customers' emotion states were defined based on emotional arousal and valence. Emotional responses caused by human interactions were evaluated, and periods of high positive affect throughout the customer journey were visualized. Experiments in actual flight services demonstrated successful emotion estimation across flight phases using a single-channel EEG measurement over two hours. Analysis results on the measurement data revealed emotional peaks outside service encounters that are not captured in customers' individual self-reports. The results also statistically revealed that two individual services (asking about a refill and conversations started by flight attendants) evoked high positive affect. Temporal dynamic analyses around high positive affect suggested patterns of interplay between joy and surprise, which are key components of customer delight. Compared with questionnaire-based evaluation, the proposed method contributes significantly to empirical studies on sequential services in marketing and design by enabling the extraction of "high positive affect," which needs to be identified for customer delight. This study supplements existing research on the interactions among physiology (EEG), behavior (emotional changes), and customer service research.
Collapse
Affiliation(s)
- Tatsunori Hara
- Graduate School of Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656 Japan.
| | - Masafumi Hamano
- Graduate School of Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656 Japan
| | - Bach Q Ho
- Human Augmentation Research Center, National Institute of Advanced Industrial Science and Technology (AIST), Kashiwa II Campus, University of Tokyo, 6-2-3 Kashiwanoha, Kashiwa, Chiba 277-0882 Japan
| | - Jun Ota
- Graduate School of Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656 Japan
| | - Yoko Yoshimoto
- ANA Strategic Research Institute Co., Ltd., 1-5-2 Shimbashi, Higashishimbashi, Minato-ku, Tokyo 105-7140, Japan
| | - Narito Arimitsu
- ANA Strategic Research Institute Co., Ltd., 1-5-2 Shimbashi, Higashishimbashi, Minato-ku, Tokyo 105-7140, Japan
| |
Collapse
|
6
|
Jafari M, Shoeibi A, Khodatars M, Bagherzadeh S, Shalbaf A, García DL, Gorriz JM, Acharya UR. Emotion recognition in EEG signals using deep learning methods: A review. Comput Biol Med 2023; 165:107450. [PMID: 37708717 DOI: 10.1016/j.compbiomed.2023.107450] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2023] [Revised: 08/03/2023] [Accepted: 09/01/2023] [Indexed: 09/16/2023]
Abstract
Emotions are a critical aspect of daily life and serve a crucial role in human decision-making, planning, reasoning, and other mental states. As a result, they are considered a significant factor in human interactions. Human emotions can be identified through various sources, such as facial expressions, speech, behavior (gesture/position), or physiological signals. The use of physiological signals can enhance the objectivity and reliability of emotion detection. Compared with peripheral physiological signals, electroencephalogram (EEG) recordings are directly generated by the central nervous system and are closely related to human emotions. EEG signals have the great spatial resolution that facilitates the evaluation of brain functions, making them a popular modality in emotion recognition studies. Emotion recognition using EEG signals presents several challenges, including signal variability due to electrode positioning, individual differences in signal morphology, and lack of a universal standard for EEG signal processing. Moreover, identifying the appropriate features for emotion recognition from EEG data requires further research. Finally, there is a need to develop more robust artificial intelligence (AI) including conventional machine learning (ML) and deep learning (DL) methods to handle the complex and diverse EEG signals associated with emotional states. This paper examines the application of DL techniques in emotion recognition from EEG signals and provides a detailed discussion of relevant articles. The paper explores the significant challenges in emotion recognition using EEG signals, highlights the potential of DL techniques in addressing these challenges, and suggests the scope for future research in emotion recognition using DL techniques. The paper concludes with a summary of its findings.
Collapse
Affiliation(s)
- Mahboobeh Jafari
- Data Science and Computational Intelligence Institute, University of Granada, Spain
| | - Afshin Shoeibi
- Data Science and Computational Intelligence Institute, University of Granada, Spain.
| | - Marjane Khodatars
- Data Science and Computational Intelligence Institute, University of Granada, Spain
| | - Sara Bagherzadeh
- Department of Biomedical Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran
| | - Ahmad Shalbaf
- Department of Biomedical Engineering and Medical Physics, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - David López García
- Data Science and Computational Intelligence Institute, University of Granada, Spain
| | - Juan M Gorriz
- Data Science and Computational Intelligence Institute, University of Granada, Spain; Department of Psychiatry, University of Cambridge, UK
| | - U Rajendra Acharya
- School of Mathematics, Physics and Computing, University of Southern Queensland, Springfield, Australia
| |
Collapse
|
7
|
Ahmad Z, Khan N. A Survey on Physiological Signal-Based Emotion Recognition. Bioengineering (Basel) 2022; 9:688. [PMID: 36421089 PMCID: PMC9687364 DOI: 10.3390/bioengineering9110688] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2022] [Revised: 10/28/2022] [Accepted: 10/31/2022] [Indexed: 12/26/2023] Open
Abstract
Physiological signals are the most reliable form of signals for emotion recognition, as they cannot be controlled deliberately by the subject. Existing review papers on emotion recognition based on physiological signals surveyed only the regular steps involved in the workflow of emotion recognition such as pre-processing, feature extraction, and classification. While these are important steps, such steps are required for any signal processing application. Emotion recognition poses its own set of challenges that are very important to address for a robust system. Thus, to bridge the gap in the existing literature, in this paper, we review the effect of inter-subject data variance on emotion recognition, important data annotation techniques for emotion recognition and their comparison, data pre-processing techniques for each physiological signal, data splitting techniques for improving the generalization of emotion recognition models and different multimodal fusion techniques and their comparison. Finally, we discuss key challenges and future directions in this field.
Collapse
Affiliation(s)
- Zeeshan Ahmad
- Department of Electrical, Computer and Biomedical Engineering, Toronto Metropolitan University, Toronto, ON M5B 2K3, Canada
| | | |
Collapse
|
8
|
Liang Z, Zhang X, Zhou R, Zhang L, Li L, Huang G, Zhang Z. Cross-individual affective detection using EEG signals with audio-visual embedding. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.09.078] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
|