1
|
Multisource Wasserstein Adaptation Coding Network for EEG emotion recognition. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103687] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
|
2
|
Wu CT, Huang HC, Huang S, Chen IM, Liao SC, Chen CK, Lin C, Lee SH, Chen MH, Tsai CF, Weng CH, Ko LW, Jung TP, Liu YH. Resting-State EEG Signal for Major Depressive Disorder Detection: A Systematic Validation on a Large and Diverse Dataset. BIOSENSORS 2021; 11:499. [PMID: 34940256 PMCID: PMC8699348 DOI: 10.3390/bios11120499] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/15/2021] [Revised: 11/26/2021] [Accepted: 12/04/2021] [Indexed: 05/09/2023]
Abstract
Major depressive disorder (MDD) is a global healthcare issue and one of the leading causes of disability. Machine learning combined with non-invasive electroencephalography (EEG) has recently been shown to have the potential to diagnose MDD. However, most of these studies analyzed small samples of participants recruited from a single source, raising serious concerns about the generalizability of these results in clinical practice. Thus, it has become critical to re-evaluate the efficacy of various common EEG features for MDD detection across large and diverse datasets. To address this issue, we collected resting-state EEG data from 400 participants across four medical centers and tested classification performance of four common EEG features: band power (BP), coherence, Higuchi's fractal dimension, and Katz's fractal dimension. Then, a sequential backward selection (SBS) method was used to determine the optimal subset. To overcome the large data variability due to an increased data size and multi-site EEG recordings, we introduced the conformal kernel (CK) transformation to further improve the MDD as compared with the healthy control (HC) classification performance of support vector machine (SVM). The results show that (1) coherence features account for 98% of the optimal feature subset; (2) the CK-SVM outperforms other classifiers such as K-nearest neighbors (K-NN), linear discriminant analysis (LDA), and SVM; (3) the combination of the optimal feature subset and CK-SVM achieves a high five-fold cross-validation accuracy of 91.07% on the training set (140 MDD and 140 HC) and 84.16% on the independent test set (60 MDD and 60 HC). The current results suggest that the coherence-based connectivity is a more reliable feature for achieving high and generalizable MDD detection performance in real-life clinical practice.
Collapse
Affiliation(s)
- Chien-Te Wu
- International Research Center for Neurointelligence (WPI-IRCN), The University of Tokyo Institutes for Advanced Study (UTIAS), The University of Tokyo, Tokyo 113-0033, Japan;
| | - Hao-Chuan Huang
- Hipposcreen Neurotech Corp. (HNC), Taipei 114, Taiwan; (H.-C.H.); (S.H.); (C.-H.W.)
| | - Shiuan Huang
- Hipposcreen Neurotech Corp. (HNC), Taipei 114, Taiwan; (H.-C.H.); (S.H.); (C.-H.W.)
| | - I-Ming Chen
- Division of Psychosomatic Medicine, Department of Psychiatry, National Taiwan University Hospital, Taipei 100229, Taiwan; (I.-M.C.); (S.-C.L.)
- Institute of Health Policy and Management, National Taiwan University, Taipei 10617, Taiwan
| | - Shih-Cheng Liao
- Division of Psychosomatic Medicine, Department of Psychiatry, National Taiwan University Hospital, Taipei 100229, Taiwan; (I.-M.C.); (S.-C.L.)
| | - Chih-Ken Chen
- Department of Psychiatry & Community Medicine Research Center, Chang Gung Memorial Hospital, Keelung 204, Taiwan; (C.-K.C.); (C.L.)
- College of Medicine, Chang Gung University, Taoyuan 33302, Taiwan;
| | - Chemin Lin
- Department of Psychiatry & Community Medicine Research Center, Chang Gung Memorial Hospital, Keelung 204, Taiwan; (C.-K.C.); (C.L.)
- College of Medicine, Chang Gung University, Taoyuan 33302, Taiwan;
| | - Shwu-Hua Lee
- College of Medicine, Chang Gung University, Taoyuan 33302, Taiwan;
- Department of Psychiatry, Chang Gung Memorial Hospital, Taoyuan 33305, Taiwan
| | - Mu-Hong Chen
- Department of Psychiatry, Taipei Veterans General Hospital, Taipei 11217, Taiwan; (M.-H.C.); (C.-F.T.)
- Faculty of Medicine, National Yang Ming Chiao Tung University, Taipei 11217, Taiwan
| | - Chia-Fen Tsai
- Department of Psychiatry, Taipei Veterans General Hospital, Taipei 11217, Taiwan; (M.-H.C.); (C.-F.T.)
- Faculty of Medicine, National Yang Ming Chiao Tung University, Taipei 11217, Taiwan
| | - Chang-Hsin Weng
- Hipposcreen Neurotech Corp. (HNC), Taipei 114, Taiwan; (H.-C.H.); (S.H.); (C.-H.W.)
| | - Li-Wei Ko
- Department of Bio Science & Tech., National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan;
| | - Tzyy-Ping Jung
- Institute for Neural Computation, University of California, San Diego, CA 92093, USA
| | - Yi-Hung Liu
- Department of Mechanical Engineering, National Taiwan University of Science and Technology, Taipei 106335, Taiwan
| |
Collapse
|
3
|
Gao Z, Wang X, Yang Y, Li Y, Ma K, Chen G. A Channel-Fused Dense Convolutional Network for EEG-Based Emotion Recognition. IEEE Trans Cogn Dev Syst 2021. [DOI: 10.1109/tcds.2020.2976112] [Citation(s) in RCA: 30] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
4
|
Abstract
Classification between individuals with mild cognitive impairment (MCI) and healthy controls (HC) based on electroencephalography (EEG) has been considered a challenging task to be addressed for the purpose of its early detection. In this study, we proposed a novel EEG feature, the kernel eigen-relative-power (KERP) feature, for achieving high classification accuracy of MCI versus HC. First, we introduced the relative powers (RPs) between pairs of electrodes across 21 different subbands of 2-Hz width as the features, which have not yet been used in previous MCI-HC classification studies. Next, the Fisher’s class separability criterion was applied to determine the best electrode pairs (five electrodes) as well as the frequency subbands for extracting the most sensitive RP features. The kernel principal component analysis (kernel PCA) algorithm was further performed to extract a few more discriminating nonlinear principal components from the optimal RPs, and these components form a KERP feature vector. Results carried out on 51 participants (24 MCI and 27 HC) show that the newly introduced subband RP feature showed superior classification performance to commonly used spectral power features, including the band power, single-electrode relative power, and also the RP based on the conventional frequency bands. A high leave-one-participant-out cross-validation (LOPO-CV) classification accuracy 86.27% was achieved by the RP feature, using a simple linear discriminant analysis (LDA) classifier. Moreover, with the same classifier, the proposed KERP further improved the accuracy to 88.24%. Finally, cascading the KERP feature to a nonlinear classifier, the support vector machine (SVM), yields a high MCI-HC classification accuracy of 90.20% (sensitivity = 87.50% and specificity = 92.59%). The proposed method demonstrated a high accuracy and a high usability (only five electrodes are required), and therefore, has great potential to further develop an EEG-based computer-aided diagnosis system that can be applied for the early detection of MCI.
Collapse
|
5
|
Emotion Recognition: An Evaluation of ERP Features Acquired from Frontal EEG Electrodes. APPLIED SCIENCES-BASEL 2021. [DOI: 10.3390/app11094131] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
The challenge to develop an affective Brain Computer Interface requires the understanding of emotions psychologically, physiologically as well as analytically. To make the analysis and classification of emotions possible, emotions have been represented in a two-dimensional or three-dimensional space represented by arousal and valence domains or arousal, valence and dominance domains, respectively. This paper presents the classification of emotions into four classes in an arousal–valence plane using the orthogonal nature of emotions. The average Event Related Potential (ERP) attributes and differential of average ERPs acquired from the frontal region of 24 subjects have been used to classify emotions into four classes. The attributes acquired from the frontal electrodes, viz., Fp1, Fp2, F3, F4, F8 and Fz, have been used for developing a classifier. The four-class subject-independent emotion classification results in the range of 67–83% have been obtained. Using three classifiers, a mid-range accuracy of 85% has been obtained, which is considerably better than existing studies on ERPs.
Collapse
|
6
|
Jiang C, Li Y, Tang Y, Guan C. Enhancing EEG-Based Classification of Depression Patients Using Spatial Information. IEEE Trans Neural Syst Rehabil Eng 2021; 29:566-575. [PMID: 33587703 DOI: 10.1109/tnsre.2021.3059429] [Citation(s) in RCA: 26] [Impact Index Per Article: 8.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
BACKGROUND Depression has become a leading mental disorder worldwide. Evidence has shown that subjects with depression exhibit different spatial responses in neurophysiological signals from the healthy controls when they are exposed to positive and negative stimuli. METHODS We proposed an effective electroencephalogram-based detection method for depression classification using spatial information. A face-in-the-crowd task, including positive and negative emotional facial expressions, was presented to 30 participants, including 16 depression patients and 14 healthy controls. Differential entropy and the genetic algorithm were used for feature extraction and selection, and a support vector machine was used for classification. A task-related common spatial pattern (TCSP) was proposed to enhance the spatial differences before the feature extraction. RESULTS AND DISCUSSION We achieved a leave-one-subject-out cross-validation classification result of 84% and 85.7% for positive and negative stimuli, respectively, using TCSP, which is statistically significantly higher than 81.7% and 83.2%, respectively, acquired without the TCSP (p < 0.05). We also evaluated the classification performance using individual frequency bands and found that the contribution of the gamma band was predominant. In addition, we evaluated different classifiers, including k-nearest neighbor and logistic regression, which showed similar trends in the improvement of classification by employing TCSP. CONCLUSION The results show that our proposed method, employing spatial information, significantly improves the accuracy of classifying depression patients.
Collapse
|
7
|
EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowl Based Syst 2020. [DOI: 10.1016/j.knosys.2020.106243] [Citation(s) in RCA: 53] [Impact Index Per Article: 13.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022]
|
8
|
Raheel A, Majid M, Alnowami M, Anwar SM. Physiological Sensors Based Emotion Recognition While Experiencing Tactile Enhanced Multimedia. SENSORS 2020; 20:s20144037. [PMID: 32708056 PMCID: PMC7411620 DOI: 10.3390/s20144037] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/29/2020] [Revised: 05/12/2020] [Accepted: 05/14/2020] [Indexed: 12/18/2022]
Abstract
Emotion recognition has increased the potential of affective computing by getting an instant feedback from users and thereby, have a better understanding of their behavior. Physiological sensors have been used to recognize human emotions in response to audio and video content that engages single (auditory) and multiple (two: auditory and vision) human senses, respectively. In this study, human emotions were recognized using physiological signals observed in response to tactile enhanced multimedia content that engages three (tactile, vision, and auditory) human senses. The aim was to give users an enhanced real-world sensation while engaging with multimedia content. To this end, four videos were selected and synchronized with an electric fan and a heater, based on timestamps within the scenes, to generate tactile enhanced content with cold and hot air effect respectively. Physiological signals, i.e., electroencephalography (EEG), photoplethysmography (PPG), and galvanic skin response (GSR) were recorded using commercially available sensors, while experiencing these tactile enhanced videos. The precision of the acquired physiological signals (including EEG, PPG, and GSR) is enhanced using pre-processing with a Savitzky-Golay smoothing filter. Frequency domain features (rational asymmetry, differential asymmetry, and correlation) from EEG, time domain features (variance, entropy, kurtosis, and skewness) from GSR, heart rate and heart rate variability from PPG data are extracted. The K nearest neighbor classifier is applied to the extracted features to classify four (happy, relaxed, angry, and sad) emotions. Our experimental results show that among individual modalities, PPG-based features gives the highest accuracy of 78.57% as compared to EEG- and GSR-based features. The fusion of EEG, GSR, and PPG features further improved the classification accuracy to 79.76% (for four emotions) when interacting with tactile enhanced multimedia.
Collapse
Affiliation(s)
- Aasim Raheel
- Department of Computer Engineering, University of Engineering and Technology, Taxila 47050, Pakistan;
| | - Muhammad Majid
- Department of Computer Engineering, University of Engineering and Technology, Taxila 47050, Pakistan;
- Correspondence:
| | - Majdi Alnowami
- Department of Nuclear Engineering, King Abdulaziz University, Jeddah 21589, Saudi Arabia;
| | - Syed Muhammad Anwar
- Department of Software Engineering, University of Engineering and Technology, Taxila 47050, Pakistan;
| |
Collapse
|
9
|
Dzedzickis A, Kaklauskas A, Bucinskas V. Human Emotion Recognition: Review of Sensors and Methods. SENSORS (BASEL, SWITZERLAND) 2020; 20:E592. [PMID: 31973140 PMCID: PMC7037130 DOI: 10.3390/s20030592] [Citation(s) in RCA: 99] [Impact Index Per Article: 24.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/11/2019] [Revised: 01/10/2020] [Accepted: 01/12/2020] [Indexed: 11/16/2022]
Abstract
Automated emotion recognition (AEE) is an important issue in various fields of activities which use human emotional reactions as a signal for marketing, technical equipment, or human-robot interaction. This paper analyzes scientific research and technical papers for sensor use analysis, among various methods implemented or researched. This paper covers a few classes of sensors, using contactless methods as well as contact and skin-penetrating electrodes for human emotion detection and the measurement of their intensity. The results of the analysis performed in this paper present applicable methods for each type of emotion and their intensity and propose their classification. The classification of emotion sensors is presented to reveal area of application and expected outcomes from each method, as well as their limitations. This paper should be relevant for researchers using human emotion evaluation and analysis, when there is a need to choose a proper method for their purposes or to find alternative decisions. Based on the analyzed human emotion recognition sensors and methods, we developed some practical applications for humanizing the Internet of Things (IoT) and affective computing systems.
Collapse
Affiliation(s)
- Andrius Dzedzickis
- Faculty of Mechanics, Vilnius Gediminas Technical University, J. Basanaviciaus g. 28, LT-03224 Vilnius, Lithuania;
| | - Artūras Kaklauskas
- Faculty of Civil engineering, Vilnius Gediminas Technical University, Sauletekio ave. 11, LT-10223 Vilnius, Lithuania;
| | - Vytautas Bucinskas
- Faculty of Mechanics, Vilnius Gediminas Technical University, J. Basanaviciaus g. 28, LT-03224 Vilnius, Lithuania;
| |
Collapse
|
10
|
Mousavinasr SMR, Pourmohammad A, Saffari MSM. Providing a Four-layer Method Based on Deep Belief Network to Improve Emotion Recognition in Electroencephalography in Brain Signals. JOURNAL OF MEDICAL SIGNALS & SENSORS 2019; 9:77-87. [PMID: 31316901 PMCID: PMC6601226 DOI: 10.4103/jmss.jmss_34_17] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Abstract
Background: One of the fields of research in recent years that has been under focused is emotion recognition in electroencephalography (EEG) signals. This study provides a four-layer method to improve people's emotion recognition through these signals and deep belief neural networks. Methods: In this study, using DEAP dataset, a four-layer method is established, which includes (1) preprocessing, (2) extracting features, (3) dimension reduction, and (4) emotion identification and estimation. To find the optimal choice in some of the steps of these layers, three different tests have been conducted. The first is finding the perfect window in feature extraction section that resulted in superiority of Hamming window to the other windows. The second is choosing the most appropriate number of filter bank and the best result was 26. The third test was also emotion recognition that its accuracy was 92.93 for arousal dimension, 92.64 for valence dimension, 93.14 for dominance dimension in two-class experiment and 76.28 for the arousal, 74.83 for the valence, and 75.64 for dominance in three-class experiment. Results: The results of this method show an improvement of 12.34% and 7.74% in two- and three-class levels in the arousal dimension. This improvement in the valence is 12.77 and 8.52, respectively. Conclusion: The results show that the proposed method can be used to improve the accuracy of emotion recognition.
Collapse
Affiliation(s)
| | - Ali Pourmohammad
- Department of Electrical Engineering, Amirkabir University of Technology, Tehran, Iran
| | | |
Collapse
|
11
|
Al-Nafjan A, Al-Wabil A, AlMudhi A, Hosny M. Measuring and monitoring emotional changes in children who stutter. Comput Biol Med 2018; 102:138-150. [PMID: 30278338 DOI: 10.1016/j.compbiomed.2018.09.022] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2018] [Revised: 09/16/2018] [Accepted: 09/24/2018] [Indexed: 10/28/2022]
Abstract
The assessment of clients with speech disorders presents challenges for speech-language pathologists. For example, having a reliable way of measuring the severity of the case, determining which remedial program is aligned with a patient's needs, and measuring of treatment processes. There is potential for brain-computer interface (BCI) applications to enhance speech therapy sessions by providing objective insights and real-time visualization of brain activity during the sessions. This paper presents a study on emotional state detection during speech pathology. The goal of this study is to investigate affective-motivational brain responses to stimuli in children who stutter. To this end, we conducted an experiment that involved recording frontal electroencephalography (EEG) activity from fifteen children with stuttering whilst they looked at visual stimuli. The contribution of our study is to provide a comprehensive background and a framework for emotional state detection experiments as assessment and monitoring tool in speech pathology. It mainly discusses the feasibility and potential benefits of applying EEG-based emotion detection in speech-language therapy contexts of use. The findings of our research indicate that emotional recognition using non-invasive EEG-based BCI system is sufficient to differentiate between affective states of individuals in treatment contexts.
Collapse
Affiliation(s)
- Abeer Al-Nafjan
- College of Computer and Information Sciences, Imam Muhammad Bin Saud University, Riyadh, 11432, Saudi Arabia; Department of Computer Sciences, College of Computer and Information Sciences, King Saud University, Riyadh, 11543, Saudi Arabia.
| | - Areej Al-Wabil
- Center for Complex Engineering Systems at KACST and MIT, King Abdulaziz City for Science and Technology, Riyadh, 11442, Saudi Arabia.
| | - Abdulaziz AlMudhi
- Department of Rehabilitation Sciences, College of Applied Medical Sciences, King Khalid University, Abha, 61421, Saudi Arabia.
| | - Manar Hosny
- Department of Computer Sciences, College of Computer and Information Sciences, King Saud University, Riyadh, 11543, Saudi Arabia.
| |
Collapse
|
12
|
Wang Y, Lv Z, Zheng Y. Automatic Emotion Perception Using Eye Movement Information for E-Healthcare Systems. SENSORS 2018; 18:s18092826. [PMID: 30150554 PMCID: PMC6164228 DOI: 10.3390/s18092826] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/13/2018] [Revised: 08/12/2018] [Accepted: 08/25/2018] [Indexed: 11/22/2022]
Abstract
Facing the adolescents and detecting their emotional state is vital for promoting rehabilitation therapy within an E-Healthcare system. Focusing on a novel approach for a sensor-based E-Healthcare system, we propose an eye movement information-based emotion perception algorithm by collecting and analyzing electrooculography (EOG) signals and eye movement video synchronously. Specifically, we extract the time-frequency eye movement features by firstly applying the short-time Fourier transform (STFT) to raw multi-channel EOG signals. Subsequently, in order to integrate time domain eye movement features (i.e., saccade duration, fixation duration, and pupil diameter), we investigate two feature fusion strategies: feature level fusion (FLF) and decision level fusion (DLF). Recognition experiments have been also performed according to three emotional states: positive, neutral, and negative. The average accuracies are 88.64% (the FLF method) and 88.35% (the DLF with maximal rule method), respectively. Experimental results reveal that eye movement information can effectively reflect the emotional state of the adolescences, which provides a promising tool to improve the performance of the E-Healthcare system.
Collapse
Affiliation(s)
- Yang Wang
- School of Computer Science and Technology, Anhui University, Hefei 230601, China.
| | - Zhao Lv
- School of Computer Science and Technology, Anhui University, Hefei 230601, China.
- Institute of Physical Science and Information Technology, Anhui University, Hefei 230601, China.
| | - Yongjun Zheng
- School of Electronics, Computing and Mathematics, University of Derby, Derby DE22 3AW, UK.
| |
Collapse
|
13
|
Depression Detection Using Relative EEG Power Induced by Emotionally Positive Images and a Conformal Kernel Support Vector Machine. APPLIED SCIENCES-BASEL 2018. [DOI: 10.3390/app8081244] [Citation(s) in RCA: 27] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Electroencephalography (EEG) can assist with the detection of major depressive disorder (MDD). However, the ability to distinguish adults with MDD from healthy individuals using resting-state EEG features has reached a bottleneck. To address this limitation, we collected EEG data as participants engaged with positive pictures from the International Affective Picture System. Because MDD is associated with blunted positive emotions, we reasoned that this approach would yield highly dissimilar EEG features in healthy versus depressed adults. We extracted three types of relative EEG power features from different frequency bands (delta, theta, alpha, beta, and gamma) during the emotion task and resting state. We also applied a novel classifier, called a conformal kernel support vector machine (CK-SVM), to try to improve the generalization performance of conventional SVMs. We then compared CK-SVM performance with three machine learning classifiers: linear discriminant analysis (LDA), conventional SVM, and quadratic discriminant analysis. The results from the initial analyses using the LDA classifier on 55 participants (24 MDD, 31 healthy controls) showed that the participant-independent classification accuracy obtained by leave-one-participant-out cross-validation (LOPO-CV) was higher for the EEG recorded during the positive emotion induction versus the resting state for all types of relative EEG power. Furthermore, the CK-SVM classifier achieved higher LOPO-CV accuracy than the other classifiers. The best accuracy (83.64%; sensitivity = 87.50%, specificity = 80.65%) was achieved by the CK-SVM, using seven relative power features extracted from seven electrodes. Overall, combining positive emotion induction with the CK-SVM classifier proved useful for detecting MDD on the basis of EEG signals. In the future, this approach might be used to develop a brain–computer interface system to assist with the detection of MDD in the clinic. Importantly, such a system could be implemented with a low-density electrode montage (seven electrodes), highlighting its practical utility.
Collapse
|
14
|
Shu L, Xie J, Yang M, Li Z, Li Z, Liao D, Xu X, Yang X. A Review of Emotion Recognition Using Physiological Signals. SENSORS (BASEL, SWITZERLAND) 2018; 18:E2074. [PMID: 29958457 PMCID: PMC6069143 DOI: 10.3390/s18072074] [Citation(s) in RCA: 167] [Impact Index Per Article: 27.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/10/2018] [Revised: 05/26/2018] [Accepted: 06/12/2018] [Indexed: 01/30/2023]
Abstract
Emotion recognition based on physiological signals has been a hot topic and applied in many areas such as safe driving, health care and social security. In this paper, we present a comprehensive review on physiological signal-based emotion recognition, including emotion models, emotion elicitation methods, the published emotional physiological datasets, features, classifiers, and the whole framework for emotion recognition based on the physiological signals. A summary and comparation among the recent studies has been conducted, which reveals the current existing problems and the future work has been discussed.
Collapse
Affiliation(s)
- Lin Shu
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China.
| | - Jinyan Xie
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China.
| | - Mingyue Yang
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China.
| | - Ziyi Li
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China.
| | - Zhenqi Li
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China.
| | - Dan Liao
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China.
| | - Xiangmin Xu
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China.
| | - Xinyi Yang
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510641, China.
| |
Collapse
|
15
|
Liu YH, Lin LF, Chou CW, Chang Y, Hsiao YT, Hsu WC. Analysis of Electroencephalography Event-Related Desynchronisation and Synchronisation Induced by Lower-Limb Stepping Motor Imagery. J Med Biol Eng 2018. [DOI: 10.1007/s40846-018-0379-9] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
16
|
Baltatzis V, Bintsi KM, Apostolidis GK, Hadjileontiadis LJ. Bullying incidences identification within an immersive environment using HD EEG-based analysis: A Swarm Decomposition and Deep Learning approach. Sci Rep 2017; 7:17292. [PMID: 29230046 PMCID: PMC5725430 DOI: 10.1038/s41598-017-17562-0] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2017] [Accepted: 11/28/2017] [Indexed: 11/09/2022] Open
Abstract
Bullying is an everlasting phenomenon and the first, yet difficult, step towards the solution is its detection. Conventional approaches for bullying incidence identification include questionnaires, conversations and psychological tests. Here, unlike the conventional approaches, two experiments are proposed that involve visual stimuli with cases of bullying- and non-bullying- related ones, set within a 2D (simple video preview) and a Virtual Reality (VR) (immersive video preview) context. In both experimental settings, brain activity is recorded using high density (HD) (256 channels) electroencephalogram (EEG), and analyzed to identify the bullying stimuli type (bullying/non-bullying) and context (2D/VR). The proposed classification analysis uses a convolutional neural network (CNN), applying deep learning on the oscillatory modes (OCMs) embedded within the raw HD EEG data. The extraction of OCMs from the HD EEG data is achieved with swarm decomposition (SWD), which efficiently accounts for the non-stationarity and noise contamination of the raw HD EEG data. Experimental results from 17 subjects indicate that the new SWD/CNN approach achieves high discrimination accuracy (AUC = 0.987 between bullying/non-bullying stimuli type; AUC = 0.975, between bullying/non-bullying stimuli type and 2D/VR context), paving the way for better understanding of how brain's responses could act as indicators of bullying experience within immersive environments.
Collapse
Affiliation(s)
- Vasileios Baltatzis
- Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki, GR, 54124, Thessaloniki, Greece
| | - Kyriaki-Margarita Bintsi
- Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki, GR, 54124, Thessaloniki, Greece
| | - Georgios K Apostolidis
- Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki, GR, 54124, Thessaloniki, Greece
| | - Leontios J Hadjileontiadis
- Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki, GR, 54124, Thessaloniki, Greece. .,Department of Electrical and Computer Engineering, Khalifa University of Science and Technology, P. O. BOX 127788, Abu Dhabi, UAE.
| |
Collapse
|
17
|
A P300 potential evaluation wavelet method comparing individuals with high and low risk for alcoholism. Neural Comput Appl 2017. [DOI: 10.1007/s00521-016-2225-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
|
18
|
Review and Classification of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review. APPLIED SCIENCES-BASEL 2017. [DOI: 10.3390/app7121239] [Citation(s) in RCA: 70] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/02/2023]
|
19
|
Liao SC, Wu CT, Huang HC, Cheng WT, Liu YH. Major Depression Detection from EEG Signals Using Kernel Eigen-Filter-Bank Common Spatial Patterns. SENSORS (BASEL, SWITZERLAND) 2017; 17:E1385. [PMID: 28613237 PMCID: PMC5492453 DOI: 10.3390/s17061385] [Citation(s) in RCA: 64] [Impact Index Per Article: 9.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/14/2017] [Revised: 06/07/2017] [Accepted: 06/10/2017] [Indexed: 01/19/2023]
Abstract
Major depressive disorder (MDD) has become a leading contributor to the global burden of disease; however, there are currently no reliable biological markers or physiological measurements for efficiently and effectively dissecting the heterogeneity of MDD. Here we propose a novel method based on scalp electroencephalography (EEG) signals and a robust spectral-spatial EEG feature extractor called kernel eigen-filter-bank common spatial pattern (KEFB-CSP). The KEFB-CSP first filters the multi-channel raw EEG signals into a set of frequency sub-bands covering the range from theta to gamma bands, then spatially transforms the EEG signals of each sub-band from the original sensor space to a new space where the new signals (i.e., CSPs) are optimal for the classification between MDD and healthy controls, and finally applies the kernel principal component analysis (kernel PCA) to transform the vector containing the CSPs from all frequency sub-bands to a lower-dimensional feature vector called KEFB-CSP. Twelve patients with MDD and twelve healthy controls participated in this study, and from each participant we collected 54 resting-state EEGs of 6 s length (5 min and 24 s in total). Our results show that the proposed KEFB-CSP outperforms other EEG features including the powers of EEG frequency bands, and fractal dimension, which had been widely applied in previous EEG-based depression detection studies. The results also reveal that the 8 electrodes from the temporal areas gave higher accuracies than other scalp areas. The KEFB-CSP was able to achieve an average EEG classification accuracy of 81.23% in single-trial analysis when only the 8-electrode EEGs of the temporal area and a support vector machine (SVM) classifier were used. We also designed a voting-based leave-one-participant-out procedure to test the participant-independent individual classification accuracy. The voting-based results show that the mean classification accuracy of about 80% can be achieved by the KEFP-CSP feature and the SVM classifier with only several trials, and this level of accuracy seems to become stable as more trials (i.e., <7 trials) are used. These findings therefore suggest that the proposed method has a great potential for developing an efficient (required only a few 6-s EEG signals from the 8 electrodes over the temporal) and effective (~80% classification accuracy) EEG-based brain-computer interface (BCI) system which may, in the future, help psychiatrists provide individualized and effective treatments for MDD patients.
Collapse
Affiliation(s)
- Shih-Cheng Liao
- Department of Psychiatry, National Taiwan University Hospital, Taipei 10051, Taiwan.
| | - Chien-Te Wu
- Department of Psychiatry, National Taiwan University Hospital, Taipei 10051, Taiwan.
- School of Occupational Therapy, College of Medicine, National Taiwan University, Taipei 10051, Taiwan.
| | - Hao-Chuan Huang
- Graduate Institute of Mechatronics Engineering, National Taipei University of Technology, Taipei 10608, Taiwan.
| | - Wei-Teng Cheng
- Department of Mechanical Engineering, Chung Yuan Christian University, Chungli 32023, Taiwan.
| | - Yi-Hung Liu
- Graduate Institute of Mechatronics Engineering, National Taipei University of Technology, Taipei 10608, Taiwan.
- Department of Mechanical Engineering, National Taipei University of Technology, Taipei 10608, Taiwan.
| |
Collapse
|
20
|
Aliakbaryhosseinabadi S, Kamavuako EN, Jiang N, Farina D, Mrachacz-Kersting N. Classification of EEG signals to identify variations in attention during motor task execution. J Neurosci Methods 2017; 284:27-34. [DOI: 10.1016/j.jneumeth.2017.04.008] [Citation(s) in RCA: 21] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2017] [Revised: 04/12/2017] [Accepted: 04/13/2017] [Indexed: 10/19/2022]
|
21
|
McFarland DJ, Parvaz MA, Sarnacki WA, Goldstein RZ, Wolpaw JR. Prediction of subjective ratings of emotional pictures by EEG features. J Neural Eng 2017; 14:016009. [PMID: 27934776 PMCID: PMC5476954 DOI: 10.1088/1741-2552/14/1/016009] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Emotion dysregulation is an important aspect of many psychiatric disorders. Brain-computer interface (BCI) technology could be a powerful new approach to facilitating therapeutic self-regulation of emotions. One possible BCI method would be to provide stimulus-specific feedback based on subject-specific electroencephalographic (EEG) responses to emotion-eliciting stimuli. APPROACH To assess the feasibility of this approach, we studied the relationships between emotional valence/arousal and three EEG features: amplitude of alpha activity over frontal cortex; amplitude of theta activity over frontal midline cortex; and the late positive potential over central and posterior mid-line areas. For each feature, we evaluated its ability to predict emotional valence/arousal on both an individual and a group basis. Twenty healthy participants (9 men, 11 women; ages 22-68) rated each of 192 pictures from the IAPS collection in terms of valence and arousal twice (96 pictures on each of 4 d over 2 weeks). EEG was collected simultaneously and used to develop models based on canonical correlation to predict subject-specific single-trial ratings. Separate models were evaluated for the three EEG features: frontal alpha activity; frontal midline theta; and the late positive potential. In each case, these features were used to simultaneously predict both the normed ratings and the subject-specific ratings. MAIN RESULTS Models using each of the three EEG features with data from individual subjects were generally successful at predicting subjective ratings on training data, but generalization to test data was less successful. Sparse models performed better than models without regularization. SIGNIFICANCE The results suggest that the frontal midline theta is a better candidate than frontal alpha activity or the late positive potential for use in a BCI-based paradigm designed to modify emotional reactions.
Collapse
Affiliation(s)
- Dennis J. McFarland
- National Center for Adaptive Neurotechnologies, Wadsworth Center, New York State Department of Health, Albany, New York 12201-0509
| | - Muhammad A. Parvaz
- Departments of Psychiatry and Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY 10029-6574
| | - William A. Sarnacki
- National Center for Adaptive Neurotechnologies, Wadsworth Center, New York State Department of Health, Albany, New York 12201-0509
| | - Rita Z. Goldstein
- Departments of Psychiatry and Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY 10029-6574
| | - Jonathan R. Wolpaw
- National Center for Adaptive Neurotechnologies, Wadsworth Center, New York State Department of Health, Albany, New York 12201-0509
| |
Collapse
|
22
|
Quitadamo LR, Cavrini F, Sbernini L, Riillo F, Bianchi L, Seri S, Saggio G. Support vector machines to detect physiological patterns for EEG and EMG-based human-computer interaction: a review. J Neural Eng 2017; 14:011001. [PMID: 28068295 DOI: 10.1088/1741-2552/14/1/011001] [Citation(s) in RCA: 47] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
Support vector machines (SVMs) are widely used classifiers for detecting physiological patterns in human-computer interaction (HCI). Their success is due to their versatility, robustness and large availability of free dedicated toolboxes. Frequently in the literature, insufficient details about the SVM implementation and/or parameters selection are reported, making it impossible to reproduce study analysis and results. In order to perform an optimized classification and report a proper description of the results, it is necessary to have a comprehensive critical overview of the applications of SVM. The aim of this paper is to provide a review of the usage of SVM in the determination of brain and muscle patterns for HCI, by focusing on electroencephalography (EEG) and electromyography (EMG) techniques. In particular, an overview of the basic principles of SVM theory is outlined, together with a description of several relevant literature implementations. Furthermore, details concerning reviewed papers are listed in tables and statistics of SVM use in the literature are presented. Suitability of SVM for HCI is discussed and critical comparisons with other classifiers are reported.
Collapse
Affiliation(s)
- L R Quitadamo
- Department of Electronic Engineering, University of Rome Tor Vergata, Rome, Italy. School of Life and Health Sciences, Aston Brain Center, Aston University, Birmingham, UK
| | | | | | | | | | | | | |
Collapse
|
23
|
|
24
|
López-Gil JM, Virgili-Gomá J, Gil R, García R. Method for Improving EEG Based Emotion Recognition by Combining It with Synchronized Biometric and Eye Tracking Technologies in a Non-invasive and Low Cost Way. Front Comput Neurosci 2016; 10:85. [PMID: 27594831 PMCID: PMC4990561 DOI: 10.3389/fncom.2016.00085] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/29/2016] [Accepted: 08/02/2016] [Indexed: 11/23/2022] Open
Abstract
Technical advances, particularly the integration of wearable and embedded sensors, facilitate tracking of physiological responses in a less intrusive way. Currently, there are many devices that allow gathering biometric measurements from human beings, such as EEG Headsets or Health Bracelets. The massive data sets generated by tracking of EEG and physiology may be used, among other things, to infer knowledge about human moods and emotions. Apart from direct biometric signal measurement, eye tracking systems are nowadays capable of determining the point of gaze of the users when interacting in ICT environments, which provides an added value research on many different areas, such as psychology or marketing. We present a process in which devices for eye tracking, biometric, and EEG signal measurements are synchronously used for studying both basic and complex emotions. We selected the least intrusive devices for different signal data collection given the study requirements and cost constraints, so users would behave in the most natural way possible. On the one hand, we have been able to determine basic emotions participants were experiencing by means of valence and arousal. On the other hand, a complex emotion such as empathy has also been detected. To validate the usefulness of this approach, a study involving forty-four people has been carried out, where they were exposed to a series of affective stimuli while their EEG activity, biometric signals, and eye position were synchronously recorded to detect self-regulation. The hypothesis of the work was that people who self-regulated would show significantly different results when analyzing their EEG data. Participants were divided into two groups depending on whether Electro Dermal Activity (EDA) data indicated they self-regulated or not. The comparison of the results obtained using different machine learning algorithms for emotion recognition shows that using EEG activity alone as a predictor for self-regulation does not allow properly determining whether a person in self-regulation its emotions while watching affective stimuli. However, adequately combining different data sources in a synchronous way to detect emotions makes it possible to overcome the limitations of single detection methods.
Collapse
Affiliation(s)
- Juan-Miguel López-Gil
- Department of Computer Languages and Systems, University of the Basque Country Vitoria-Gasteiz, Spain
| | - Jordi Virgili-Gomá
- Department of Computer Science and Engineering, Universitat de Lleida Lleida, Spain
| | - Rosa Gil
- Department of Computer Science and Engineering, Universitat de Lleida Lleida, Spain
| | - Roberto García
- Department of Computer Science and Engineering, Universitat de Lleida Lleida, Spain
| |
Collapse
|
25
|
A Self-Paced P300 Healthcare Brain-Computer Interface System with SSVEP-Based Switching Control and Kernel FDA + SVM-Based Detector. APPLIED SCIENCES-BASEL 2016. [DOI: 10.3390/app6050142] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
|