1
|
Zhang Y, Liu H, Wang D, Zhang D, Lou T, Zheng Q, Quek C. Cross-modal credibility modelling for EEG-based multimodal emotion recognition. J Neural Eng 2024; 21:026040. [PMID: 38565099 DOI: 10.1088/1741-2552/ad3987] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2023] [Accepted: 04/02/2024] [Indexed: 04/04/2024]
Abstract
Objective.The study of emotion recognition through electroencephalography (EEG) has garnered significant attention recently. Integrating EEG with other peripheral physiological signals may greatly enhance performance in emotion recognition. Nonetheless, existing approaches still suffer from two predominant challenges: modality heterogeneity, stemming from the diverse mechanisms across modalities, and fusion credibility, which arises when one or multiple modalities fail to provide highly credible signals.Approach.In this paper, we introduce a novel multimodal physiological signal fusion model that incorporates both intra-inter modality reconstruction and sequential pattern consistency, thereby ensuring a computable and credible EEG-based multimodal emotion recognition. For the modality heterogeneity issue, we first implement a local self-attention transformer to obtain intra-modal features for each respective modality. Subsequently, we devise a pairwise cross-attention transformer to reveal the inter-modal correlations among different modalities, thereby rendering different modalities compatible and diminishing the heterogeneity concern. For the fusion credibility issue, we introduce the concept of sequential pattern consistency to measure whether different modalities evolve in a consistent way. Specifically, we propose to measure the varying trends of different modalities, and compute the inter-modality consistency scores to ascertain fusion credibility.Main results.We conduct extensive experiments on two benchmarked datasets (DEAP and MAHNOB-HCI) with the subject-dependent paradigm. For the DEAP dataset, our method improves the accuracy by 4.58%, and the F1 score by 0.63%, compared to the state-of-the-art baseline. Similarly, for the MAHNOB-HCI dataset, our method improves the accuracy by 3.97%, and the F1 score by 4.21%. In addition, we gain much insight into the proposed framework through significance test, ablation experiments, confusion matrices and hyperparameter analysis. Consequently, we demonstrate the effectiveness of the proposed credibility modelling through statistical analysis and carefully designed experiments.Significance.All experimental results demonstrate the effectiveness of our proposed architecture and indicate that credibility modelling is essential for multimodal emotion recognition.
Collapse
Affiliation(s)
- Yuzhe Zhang
- School of Computer Science and Technology, MOEKLINNS Lab, Xi'an Jiaotong University, Xi'an 710049, People's Republic of China
| | - Huan Liu
- School of Computer Science and Technology, MOEKLINNS Lab, Xi'an Jiaotong University, Xi'an 710049, People's Republic of China
| | - Di Wang
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798, Singapore
| | - Dalin Zhang
- Department of Computer Science, Aalborg University, Fredrik Bajers Vej 7 K, 9220 Aalborg, Denmark
| | - Tianyu Lou
- School of Computer Science and Technology, MOEKLINNS Lab, Xi'an Jiaotong University, Xi'an 710049, People's Republic of China
| | - Qinghua Zheng
- School of Computer Science and Technology, MOEKLINNS Lab, Xi'an Jiaotong University, Xi'an 710049, People's Republic of China
| | - Chai Quek
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798, Singapore
| |
Collapse
|
2
|
Deny P, Cheon S, Son H, Choi KW. Hierarchical Transformer for Motor Imagery-Based Brain Computer Interface. IEEE J Biomed Health Inform 2023; 27:5459-5470. [PMID: 37578918 DOI: 10.1109/jbhi.2023.3304646] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/16/2023]
Abstract
In this paper, we propose a novel transformer-based classification algorithm for the brain computer interface (BCI) using a motor imagery (MI) electroencephalogram (EEG) signal. To design the MI classification algorithm, we apply an up-to-date deep learning model, the transformer, that has revolutionized the natural language processing (NLP) and successfully widened its application to many other domains such as the computer vision. Within a long MI trial spanning a few seconds, the classification algorithm should give more attention to the time periods during which the intended motor task is imagined by the subject without any artifact. To achieve this goal, we propose a hierarchical transformer architecture that consists of a high-level transformer (HLT) and a low-level transformer (LLT). We break down a long MI trial into a number of short-term intervals. The LLT extracts a feature from each short-term interval, and the HLT pays more attention to the features from more relevant short-term intervals by using the self-attention mechanism of the transformer. We have done extensive tests of the proposed scheme on four open MI datasets, and shown that the proposed hierarchical transformer excels in both the subject-dependent and subject-independent tests.
Collapse
|
3
|
Wang H, Treder MS, Marshall D, Jones DK, Li Y. A Skewed Loss Function for Correcting Predictive Bias in Brain Age Prediction. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:1577-1589. [PMID: 37015392 PMCID: PMC7615262 DOI: 10.1109/tmi.2022.3231730] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/22/2022] [Accepted: 12/18/2022] [Indexed: 06/02/2023]
Abstract
In neuroimaging, the difference between predicted brain age and chronological age, known as brain age delta, has shown its potential as a biomarker related to various pathological phenotypes. There is a frequently observed bias when estimating brain age delta using regression models. This bias manifests as an overestimation of brain age for young participants and an underestimation of brain age for older participants. Therefore, the brain age delta is negatively correlated with chronological age, which can be problematic when evaluating relationships between brain age delta and other age-associated variables. This paper proposes a novel bias correction method for regression models by introducing a skewed loss function to replace the normal symmetric loss function. The regression model then behaves differently depending on whether it makes overestimations or underestimations. Our approach works with any type of MR image and no specific preprocessing is required, as long as the image is sensitive to age-related changes. The proposed approach has been validated using three classic deep learning models, namely ResNet, VGG, and GoogleNet on publicly available neuroimaging aging datasets. It shows flexibility across different model architectures and different choices of hyperparameters. The corrected brain age delta from our approach then has no linear relationship with chronological age and achieves higher predictive accuracy than a commonly-used two-stage approach.
Collapse
Affiliation(s)
- Hanzhi Wang
- School of Computer Science and InformaticsCardiff UniversityCF10 3ATCardiffU.K
| | - Matthias S. Treder
- School of Computer Science and InformaticsCardiff UniversityCF10 3ATCardiffU.K
| | - David Marshall
- School of Computer Science and InformaticsCardiff UniversityCF10 3ATCardiffU.K
| | - Derek K. Jones
- Cardiff University Brain Research Imaging Centre, Cardiff UniversityCF24 4HQCardiffU.K
| | - Yuhua Li
- School of Computer Science and InformaticsCardiff UniversityCF10 3ATCardiffU.K
| |
Collapse
|
4
|
Robust zero-watermarking scheme based on a depthwise overparameterized VGG network in healthcare information security. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104478] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
|
5
|
Mendes Junior JJA, Pontim CE, Dias TS, Campos DP. How do sEMG segmentation parameters influence pattern recognition process? An approach based on wearable sEMG sensor. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104546] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
|
6
|
Pornvoraphat P, Tiankanon K, Pittayanon R, Sunthornwetchapong P, Vateekul P, Rerknimitr R. Real-time gastric intestinal metaplasia diagnosis tailored for bias and noisy-labeled data with multiple endoscopic imaging. Comput Biol Med 2023; 154:106582. [PMID: 36738708 DOI: 10.1016/j.compbiomed.2023.106582] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2022] [Revised: 01/11/2023] [Accepted: 01/22/2023] [Indexed: 01/26/2023]
Abstract
This work presents real-time segmentation viz. gastric intestinal metaplasia (GIM). Recently, GIM segmentation of endoscopic images has been carried out to differentiate GIM from a healthy stomach. However, real-time detection is difficult to achieve. Conditions are challenging, and include multiple color modes (white light endoscopy and narrow-band imaging), other abnormal lesions (erosion and ulcer), noisy labels etc. Herein, our model is based on BiSeNet and can overcome the many issues regarding GIM. Application of auxiliary head and additional loss are seen to improve performance as well as enhance multiple color modes accurately. Further, multiple pre-processing techniques are utilized for leveraging detection performance: namely, location-wise negative sampling, jigsaw augmentation, and label smoothing. Finally, the decision threshold can be adjusted separately for each color mode. Work undertaken at King Chulalongkorn Memorial Hospital examined 940 histologically proven GIM images and 1239 non-GIM images, obtained over 173 frames per second (FPS). In terms of accuracy, our model is seen to outperform all baselines. Our results demonstrate sensitivity, specificity, positive predictive, negative predictive, accuracy, and mean intersection over union (IoU), achieving GIM segmentation values of 91%, 96%, 91%, 91%, 96%, and 55%, respectively.
Collapse
Affiliation(s)
- Passin Pornvoraphat
- Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Pathumwan, Bangkok, Thailand
| | - Kasenee Tiankanon
- Division of Gastroenterology, Department of Medicine, Faculty of Medicine, Chulalongkorn University and King Chulalongkorn Memorial Hospital, Bangkok, Thailand
| | - Rapat Pittayanon
- Division of Gastroenterology, Department of Medicine, Faculty of Medicine, Chulalongkorn University and King Chulalongkorn Memorial Hospital, Bangkok, Thailand
| | - Phanukorn Sunthornwetchapong
- Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Pathumwan, Bangkok, Thailand
| | - Peerapon Vateekul
- Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Pathumwan, Bangkok, Thailand.
| | - Rungsun Rerknimitr
- Division of Gastroenterology, Department of Medicine, Faculty of Medicine, Chulalongkorn University and King Chulalongkorn Memorial Hospital, Bangkok, Thailand.
| |
Collapse
|
7
|
A Deep Model of Visual Attention for Saliency Detection on 3D Objects. Neural Process Lett 2023. [DOI: 10.1007/s11063-023-11180-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/06/2023]
|
8
|
Gupta K, Bajaj V. Deep learning models-based CT-scan image classification for automated screening of COVID-19. Biomed Signal Process Control 2023; 80:104268. [PMID: 36267466 PMCID: PMC9556167 DOI: 10.1016/j.bspc.2022.104268] [Citation(s) in RCA: 15] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2022] [Revised: 09/07/2022] [Accepted: 09/26/2022] [Indexed: 02/01/2023]
Abstract
COVID-19 is the most transmissible disease, caused by the SARS-CoV-2 virus that severely infects the lungs and the upper respiratory tract of the human body. This virus badly affected the lives and wellness of millions of people worldwide and spread widely. Early diagnosis, timely treatment, and proper confinement of the infected patients are some possible ways to control the spreading of coronavirus. Computed tomography (CT) scanning has proven useful in diagnosing several respiratory lung problems, including COVID-19 infections. Automated detection of COVID-19 using chest CT-scan images may reduce the clinician's load and save the lives of thousands of people. This study proposes a robust framework for the automated screening of COVID-19 using chest CT-scan images and deep learning-based techniques. In this work, a publically accessible CT-scan image dataset (contains the 1252 COVID-19 and 1230 non-COVID chest CT images), two pre-trained deep learning models (DLMs) namely, MobileNetV2 and DarkNet19, and a newly-designed lightweight DLM, are utilized for the automated screening of COVID-19. A repeated ten-fold holdout validation method is utilized for the training, validation, and testing of DLMs. The highest classification accuracy of 98.91% is achieved using transfer-learned DarkNet19. The proposed framework is ready to be tested with more CT images. The simulation results with the publicly available COVID-19 CT scan image dataset are included to show the effectiveness of the presented study.
Collapse
|
9
|
Su F, Wei M, Sun M, Jiang L, Dong Z, Wang J, Zhang C. Deep learning-based synapse counting and synaptic ultrastructure analysis of electron microscopy images. J Neurosci Methods 2023; 384:109750. [PMID: 36414102 DOI: 10.1016/j.jneumeth.2022.109750] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2022] [Revised: 11/11/2022] [Accepted: 11/18/2022] [Indexed: 11/21/2022]
Abstract
BACKGROUND Synapses are the connections between neurons in the central nervous system (CNS) or between neurons and other excitable cells in the peripheral nervous system (PNS), where electrical or chemical signals rapidly travel through one cell to another with high spatial precision. Synaptic analysis, based on synapse numbers and fine morphology, is the basis for understanding neurological functions and diseases. Manual analysis of synaptic structures in electron microscopy (EM) images is often limited by low efficiency and subjective bias. NEW METHOD We developed a multifunctional synaptic analysis system based on several advanced deep learning (DL) models. The system achieved synapse counting in low-magnification EM images and synaptic ultrastructure analysis in high-magnification EM images. RESULTS The synapse counting system based on ResNet18 and a Faster R-CNN model had a mean average precision (mAP) of 92.55%. For synaptic ultrastructure analysis, the Faster R-CNN model based on ResNet50 achieved a mAP of 91.60%, the DeepLab v3 + model based on ResNet50 enabled high performance in presynaptic and postsynaptic membrane segmentation with a global accuracy of 0.9811, and the Faster R-CNN model based on ResNet18 achieved a mAP of 91.41% for synaptic vesicle detection. CONCLUSIONS The proposed multifunctional synaptic analysis system may help to overcome the experimental bias inherent in manual analysis, thereby facilitating EM image-based synaptic function studies.
Collapse
Affiliation(s)
- Feng Su
- Department of Neurobiology, School of Basic Medical Sciences, Beijing Key Laboratory of Neural Regeneration and Repair, Capital Medical University, Beijing 100069, China; Chinese Institute for Brain Research, Beijing 102206, China; State Key Laboratory of Translational Medicine and Innovative Drug Development, Nanjing 210000, Jiangsu, China; Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
| | - Mengping Wei
- Department of Neurobiology, School of Basic Medical Sciences, Beijing Key Laboratory of Neural Regeneration and Repair, Capital Medical University, Beijing 100069, China
| | - Meng Sun
- Department of Neurobiology, School of Basic Medical Sciences, Beijing Key Laboratory of Neural Regeneration and Repair, Capital Medical University, Beijing 100069, China
| | - Lixin Jiang
- Peking University Institute of Mental Health (Sixth Hospital), No. 51 Huayuanbei Road, Haidian District, Beijing 100191, China
| | - Zhaoqi Dong
- Department of Neurobiology, School of Basic Medical Sciences, Beijing Key Laboratory of Neural Regeneration and Repair, Capital Medical University, Beijing 100069, China
| | - Jue Wang
- Department of Neurobiology, School of Basic Medical Sciences, Beijing Key Laboratory of Neural Regeneration and Repair, Capital Medical University, Beijing 100069, China
| | - Chen Zhang
- Department of Neurobiology, School of Basic Medical Sciences, Beijing Key Laboratory of Neural Regeneration and Repair, Capital Medical University, Beijing 100069, China; Chinese Institute for Brain Research, Beijing 102206, China; State Key Laboratory of Translational Medicine and Innovative Drug Development, Nanjing 210000, Jiangsu, China.
| |
Collapse
|
10
|
Kumar A, Vishwakarma A, Bajaj V. CRCCN-Net: Automated framework for classification of colorectal tissue using histopathological images. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104172] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/14/2022]
|
11
|
Xie W, Wang L, Yu K, Shi T, Li W. Improved multi-layer binary firefly algorithm for optimizing feature selection and classification of microarray data. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104080] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
12
|
Breast cancer classification by a new approach to assessing deep neural network-based uncertainty quantification methods. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
13
|
RTUNet: Residual transformer UNet specifically for pancreas segmentation. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104173] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
14
|
Tian Z, Li D, Yang Y, Hou F, Yang Z, Song Y, Gao Q. A Novel Domain Adversarial Networks Based on 3D-LSTM and Local Domain Discriminator for Hearing-Impaired Emotion Recognition. IEEE J Biomed Health Inform 2023; 27:363-373. [PMID: 36201412 DOI: 10.1109/jbhi.2022.3212475] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Recent research on emotion recognition suggests that deep network-based adversarial learning has an ability to solve the cross-subject problem of emotion recognition. This study constructed a hearing-impaired electroencephalography (EEG) emotion dataset containing three emotions (positive, neutral, and negative) in 15 subjects. The emotional domain adversarial neural network (EDANN) was carried out to identify hearing-impaired subjects' emotions by learning hidden emotion information between the labeled data and the data with no-label. For the input data, we propose a spatial filter matrix to reduce the overfitting of the training data. A feature extraction network 3DLSTM-ConvNET was used to extract comprehensive emotional information from the time, frequency, and spatial dimensions. Moreover, emotion local domain discriminator and emotion film group local domain discriminator were added to reduce the distribution distance between the same kinds of emotions and different film groups, respectively. According to the experimental results, the average accuracy of subject-dependent is 0.984 (STD: 0.011), and that of subject-independent is 0.679 (STD: 0.140). In addition, by analyzing the discrimination characteristics, we found that the brain regions with emotional recognition in the hearing-impaired are distributed in the wider areas of the parietal and occipital lobes, which may be caused by visual processing.
Collapse
|
15
|
Wu M, Teng W, Fan C, Pei S, Li P, Lv Z. An Investigation of Olfactory-Enhanced Video on EEG-Based Emotion Recognition. IEEE Trans Neural Syst Rehabil Eng 2023; 31:1602-1613. [PMID: 37028354 DOI: 10.1109/tnsre.2023.3253866] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/18/2023]
Abstract
Collecting emotional physiological signals is significant in building affective Human-Computer Interactions (HCI). However, how to evoke subjects' emotions efficiently in EEG-related emotional experiments is still a challenge. In this work, we developed a novel experimental paradigm that allows odors dynamically participate in different stages of video-evoked emotions, to investigate the efficiency of olfactory-enhanced videos in inducing subjects' emotions; According to the period that the odors participated in, the stimuli were divided into four patterns, i.e., the olfactory-enhanced video in early/later stimulus periods (OVEP/OVLP), and the traditional videos in early/later stimulus periods (TVEP/TVLP). The differential entropy (DE) feature and four classifiers were employed to test the efficiency of emotion recognition. The best average accuracies of the OVEP, OVLP, TVEP, and TVLP were 50.54%, 51.49%, 40.22%, and 57.55%, respectively. The experimental results indicated that the OVEP significantly outperformed the TVEP on classification performance, while there was no significant difference between the OVLP and TVLP. Besides, olfactory-enhanced videos achieved higher efficiency in evoking negative emotions than traditional videos. Moreover, we found that the neural patterns in response to emotions under different stimulus methods were stable, and for Fp1, FP2, and F7, there existed significant differences in whether adopt the odors.
Collapse
Affiliation(s)
- Minchao Wu
- Anhui Province Key Laboratory of Multimodal Cognitive Computation and the School of Computer Science and Technology, Anhui University, Hefei, China
| | - Wei Teng
- Anhui Province Key Laboratory of Multimodal Cognitive Computation and the School of Computer Science and Technology, Anhui University, Hefei, China
| | - Cunhang Fan
- Anhui Province Key Laboratory of Multimodal Cognitive Computation and the School of Computer Science and Technology, Anhui University, Hefei, China
| | - Shengbing Pei
- Anhui Province Key Laboratory of Multimodal Cognitive Computation and the School of Computer Science and Technology, Anhui University, Hefei, China
| | - Ping Li
- Anhui Province Key Laboratory of Multimodal Cognitive Computation and the School of Computer Science and Technology, Anhui University, Hefei, China
| | - Zhao Lv
- Anhui Province Key Laboratory of Multimodal Cognitive Computation and the School of Computer Science and Technology, Anhui University, Hefei, China
| |
Collapse
|
16
|
Zhong Z, Du Y, Zhou Y, Cao J, He S. Delving deep into pixelized face recovery and defense. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.09.141] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/31/2022]
|
17
|
An Interpretive Adversarial Attack Method: Attacking Softmax Gradient Layer-Wise Relevance Propagation Based on Cosine Similarity Constraint and TS-Invariant. Neural Process Lett 2022. [DOI: 10.1007/s11063-022-11056-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/01/2022]
|
18
|
Classification of Contrasting Discrete Emotional States Indicated by EEG Based Graph Theoretical Network Measures. Neuroinformatics 2022; 20:863-877. [PMID: 35286574 DOI: 10.1007/s12021-022-09579-2] [Citation(s) in RCA: 31] [Impact Index Per Article: 15.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/22/2022] [Indexed: 12/31/2022]
Abstract
The present study shows new findings that reveal the high association between emotional arousal and neuro-functional brain connectivity measures. For this purpose, contrasting discrete emotional states (happiness vs sadness, amusement vs disgust, calmness vs excitement, calmness vs anger, fear vs anger) are classified by using Support Vector Machines (SVMs) driven by Graph Theoretical segregation (clustering coefficients, transitivity, modularity) and integration (global efficiency, local efficiency) measures of the brain network. Emotional EEG data mediated by short duration video film clips is downloaded from publicly available database called DREAMER. Pearson Correlation (PC) and Spearman Correlation have been examined to estimate statistical dependencies between relatively shorter (6 sec) and longer (12 sec) non-overlapped EEG segments across the cortex. Then the corresponding brain connectivity encoded as a graph is transformed into binary numbers with respect to two different thresholds (60%max and mean). Statistical differences between contrasting emotions are obtained by using both one-way Anova tests and step-wise logistic regression modelling in accordance with variables (dependency estimation, segment length, threshold, network measure). Combined integration measures provided the highest classification accuracies (CAs) (75.00% 80.65%) when PC is applied to longer segments in accordance with particular threshold as the mean. The segregation measures also provided useful CAs (74.13% 80.00%), while the combination of both measures did not. The results reveal that discrete emotional states are characterized by balanced network measures even if both segregation and integration measures vary depending on arousal scores of audio-visual stimuli due to neurotransmitter release during video watching.
Collapse
|
19
|
Huxley muscle model surrogates for high-speed multi-scale simulations of cardiac contraction. Comput Biol Med 2022; 149:105963. [DOI: 10.1016/j.compbiomed.2022.105963] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2022] [Revised: 07/17/2022] [Accepted: 08/13/2022] [Indexed: 11/19/2022]
|
20
|
CheXGAT: A disease correlation-aware network for thorax disease diagnosis from chest X-ray images. Artif Intell Med 2022; 132:102382. [DOI: 10.1016/j.artmed.2022.102382] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2021] [Revised: 08/07/2022] [Accepted: 08/19/2022] [Indexed: 11/23/2022]
|
21
|
Rachkovskij DA. Representation of spatial objects by shift-equivariant similarity-preserving hypervectors. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07619-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/14/2022]
|
22
|
Chen Z, Liu C, Ren J. A commonality-based enhancement for sentence modeling with supervision. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.07.063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
|
23
|
Wu Y, Xia M, Nie L, Zhang Y, Fan A. Simultaneously exploring multi-scale and asymmetric EEG features for emotion recognition. Comput Biol Med 2022; 149:106002. [PMID: 36041272 DOI: 10.1016/j.compbiomed.2022.106002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2022] [Revised: 08/07/2022] [Accepted: 08/14/2022] [Indexed: 11/03/2022]
Abstract
In recent years, emotion recognition based on electroencephalography (EEG) has received growing interests in the brain-computer interaction (BCI) field. The neuroscience researches indicate that the left and right brain hemispheres demonstrate activity differences under different emotional activities, which could be an important principle for designing deep learning (DL) model for emotion recognition. Besides, owing to the nonstationarity of EEG signals, using convolution kernels of a single size may not sufficiently extract the abundant features for EEG classification tasks. Based on these two angles, we proposed a model termed Multi-Scales Bi-hemispheric Asymmetric Model (MSBAM) based on convolutional neural network (CNN) structure. Evaluated on the public DEAP and DREAMER datasets, MSBAM achieved over 99% accuracy for the two-class classification of low-level and high-level states in each of four emotional dimensions, i.e., arousal, valence, dominance and liking, respectively. This study further demonstrated the promising potential to design the DL model from the multi-scale characteristics of the EEG data and the neural mechanisms of the emotion cognition.
Collapse
Affiliation(s)
- Yihan Wu
- School of Computer Science and Technology, Laboratory for Brain Science and Medical Artificial Intelligence, Southwest University of Science and Technology, Mianyang, 621010, China
| | - Min Xia
- School of Computer Science and Technology, Laboratory for Brain Science and Medical Artificial Intelligence, Southwest University of Science and Technology, Mianyang, 621010, China
| | - Li Nie
- School of Computer Science and Technology, Laboratory for Brain Science and Medical Artificial Intelligence, Southwest University of Science and Technology, Mianyang, 621010, China
| | - Yangsong Zhang
- School of Computer Science and Technology, Laboratory for Brain Science and Medical Artificial Intelligence, Southwest University of Science and Technology, Mianyang, 621010, China; MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, China.
| | - Andong Fan
- College of Mathematics and Physics, Chengdu University of Technology, Chengdu, 610059, China.
| |
Collapse
|
24
|
A Lightweight Neural Learning Algorithm for Real-Time Facial Feature Tracking System via Split-Attention and Heterogeneous Convolution. Neural Process Lett 2022. [DOI: 10.1007/s11063-022-10951-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
25
|
HIRE: Distilling high-order relational knowledge from heterogeneous graph neural networks. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.08.022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
26
|
Effects of Rest-Break on mental fatigue recovery based on EEG dynamic functional connectivity. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103806] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
27
|
Shou Y, Meng T, Ai W, Yang S, Li K. Conversational emotion recognition studies based on graph convolutional neural networks and a dependent syntactic analysis. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.06.072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
|
28
|
Wang K, Wang J, Zhu Y, Li H, Liu C, Fietkiewicz C, Loparo KA. Adaptive closed-loop control strategy inhibiting pathological basal ganglia oscillations. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103776] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
29
|
Aydın S. Cross-validated Adaboost Classification of Emotion Regulation Strategies Identified by Spectral Coherence in Resting-State. Neuroinformatics 2022; 20:627-639. [PMID: 34536200 DOI: 10.1007/s12021-021-09542-7] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/18/2021] [Indexed: 12/31/2022]
Abstract
In the present study, quantitative relations between Cognitive Emotion Regulation strategies (CERs) and EEG synchronization levels have been investigated for the first time. For this purpose, spectral coherence (COH), phase locking value and mutual information have been applied to short segments of 62-channel resting state eyes-opened EEG data collected from healthy adults who use contrasting emotion regulation strategies (frequently and rarely use of rumination&distraction, frequently and rarely use of suppression&reappraisal). In tests, the individuals are grouped depending on their self-responses to both emotion regulation questionnaire (ERQ) and cognitive ERQ. Experimental data are downloaded from publicly available data-base, LEMON. Regarding EEG electrode pairs that placed on right and left cortical regions, inter-hemispheric dependency measures are computed for non-overlapped short segments of 2 sec at 2 min duration trials. In addition to full-band EEG analysis, dependency metrics are also obtained for both alpha and beta sub-bands. The contrasting groups are discriminated from each other with respect to the corresponding features using cross-validated adaboost classifiers. High classification accuracies (CA) of 99.44% and 98.33% have been obtained through instant classification driven by full-band COH estimations. Considering regional features that provide the high CA, CERs are found to be highly relevant with associative memory functions and cognition. The new findings may indicate the close relation between neuroplasticity and cognitive skills.
Collapse
Affiliation(s)
- Serap Aydın
- Biophysics Department, Medical Faculty, Hacettepe University, Ankara, Turkey.
| |
Collapse
|
30
|
Yang C, Wang X, Yao L, Long G, Jiang J, Xu G. Attentional Gated Res2Net for Multivariate Time Series Classification. Neural Process Lett 2022. [DOI: 10.1007/s11063-022-10944-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Abstract
AbstractMultivariate time series classification is a critical problem in data mining with broad applications. It requires harnessing the inter-relationship of multiple variables and various ranges of temporal dependencies to assign the correct classification label of the time series. Multivariate time series may come from a wide range of sources and be used in various scenarios, bringing the classifier challenge of temporal representation learning. We propose a novel convolutional neural network architecture called Attentional Gated Res2Net for multivariate time series classification. Our model uses hierarchical residual-like connections to achieve multi-scale receptive fields and capture multi-granular temporal information. The gating mechanism enables the model to consider the relations between the feature maps extracted by receptive fields of multiple sizes for information fusion. Further, we propose two types of attention modules, channel-wise attention and block-wise attention, to better leverage the multi-granular temporal patterns. Our experimental results on 14 benchmark multivariate time-series datasets show that our model outperforms several baselines and state-of-the-art methods by a large margin. Our model outperforms the SOTA by a large margin, the classification accuracy of our model is 10.16% better than the SOTA model. Besides, we demonstrate that our model improves the performance of existing models when used as a plugin. Further, based on our experiments and analysis, we provide practical advice on applying our model to a new problem.
Collapse
|
31
|
Saffaryazdi N, Wasim ST, Dileep K, Nia AF, Nanayakkara S, Broadbent E, Billinghurst M. Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition. Front Psychol 2022; 13:864047. [PMID: 35837650 PMCID: PMC9275379 DOI: 10.3389/fpsyg.2022.864047] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2022] [Accepted: 05/30/2022] [Indexed: 11/13/2022] Open
Abstract
Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that facial micro-expressions are more reliable than facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition.
Collapse
Affiliation(s)
- Nastaran Saffaryazdi
- Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| | - Syed Talal Wasim
- Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| | - Kuldeep Dileep
- Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| | - Alireza Farrokhi Nia
- Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| | - Suranga Nanayakkara
- Augmented Human Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| | - Elizabeth Broadbent
- Department of Psychological Medicine, The University of Auckland, Auckland, New Zealand
| | - Mark Billinghurst
- Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| |
Collapse
|
32
|
A novel deep learning model based on the ICA and Riemannian manifold for EEG-based emotion recognition. J Neurosci Methods 2022; 378:109642. [DOI: 10.1016/j.jneumeth.2022.109642] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Revised: 05/11/2022] [Accepted: 06/05/2022] [Indexed: 11/22/2022]
|
33
|
Zhang R, Zeng Y, Tong L, Shu J, Lu R, Yang K, Li Z, Yan B. ERP-WGAN: A Data Augmentation Method for EEG Single-trial Detection. J Neurosci Methods 2022; 376:109621. [PMID: 35513171 DOI: 10.1016/j.jneumeth.2022.109621] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2021] [Revised: 04/13/2022] [Accepted: 04/28/2022] [Indexed: 10/18/2022]
Abstract
Brain computer interaction based on EEG presents great potential and becomes the research hotspots. However, the insufficient scale of EEG database limits the BCI system performance, especially the positive and negative sample imbalance caused by oddball paradigm. To alleviate the bottleneck problem of scarce EEG sample, we propose a data augmentation method based on generative adversarial network to improve the performance of EEG signal classification. Taking the characteristics of EEG into account in wasserstein generative adversarial networks (WGAN), the problems of model collapse and poor quality of artificial data were solved by using resting noise, smoothing and random amplitude. The quality of artificial data was comprehensively evaluated from verisimilitude, diversity and accuracy. Compared with the three artificial data methods and two data sampling methods, the proposed ERP-WGAN framework significantly improve the performance of both subject and general classifiers, especially the accuracy of general classifiers trained by less than 5 dimensional features is improved by 20-25%. Moreover, we evaluate the training sets performance with different mixing ratios of artificial and real samples. ERP-WGAN can reduced at least 73% of the real subject data and acquisition cost, which greatly saves the test cycle and research cost.
Collapse
Affiliation(s)
- Rongkai Zhang
- Department of Information System Engineering, PLA Strategy Support Force Information Engineering University, Zhengzhou 450001, China
| | - Ying Zeng
- Department of Information System Engineering, PLA Strategy Support Force Information Engineering University, Zhengzhou 450001, China; The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, China
| | - Li Tong
- Department of Information System Engineering, PLA Strategy Support Force Information Engineering University, Zhengzhou 450001, China
| | - Jun Shu
- Department of Information System Engineering, PLA Strategy Support Force Information Engineering University, Zhengzhou 450001, China
| | - Runnan Lu
- Department of Information System Engineering, PLA Strategy Support Force Information Engineering University, Zhengzhou 450001, China
| | - Kai Yang
- Department of Information System Engineering, PLA Strategy Support Force Information Engineering University, Zhengzhou 450001, China
| | - Zhongrui Li
- Department of Information System Engineering, PLA Strategy Support Force Information Engineering University, Zhengzhou 450001, China
| | - Bin Yan
- Department of Information System Engineering, PLA Strategy Support Force Information Engineering University, Zhengzhou 450001, China
| |
Collapse
|
34
|
Li G, Chen N, Jin J. Semi-supervised EEG Emotion Recognition Model Based on Enhanced Graph Fusion and GCN. J Neural Eng 2022; 19. [PMID: 35378516 DOI: 10.1088/1741-2552/ac63ec] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2022] [Accepted: 04/04/2022] [Indexed: 11/12/2022]
Abstract
OBJECTIVE To take full advantage of both labeled data and unlabeled ones, the Graph Convolutional Network (GCN) was introduced in electroencephalography (EEG) based emotion recognition to achieve feature propagation. However, a single feature cannot represent the emotional state entirely and precisely due to the instability of the EEG signal and the complexity of the emotional state. In addition, the noise existing in the graph may affect the performance greatly. To solve these problems, it was necessary to introduce feature/similarity fusion and noise reduction strategies. APPROACH A semi-supervised EEG emotion recognition model combining graph fusion, network enhancement, and feature fusion was proposed. Firstly, different features were extracted from EEG and then compacted by Principal Component Analysis (PCA), respectively. Secondly, a Sample-by-sample Similarity Matrix (SSM) was constructed based on each feature, and Similarity Network Fusion (SNF) was adopted to fuse the graphs corresponding to different SSMs to take advantage of their complementarity. Then, Network Enhancement (NE) was performed on the fused graph to reduce the noise in it. Finally, GCN was performed on the concatenated features and the enhanced fused graph to achieve feature propagation. MAIN RESULTS Experimental results demonstrated that: i) When 5.30% of SEED and 7.20% of SEED-IV samples were chosen as the labeled samples, respectively, the minimum classification accuracy improvement achieved by the proposed scheme over state-of-the-art schemes were 1.52% on SEED and 13.14% on SEED-IV, respectively. ii) When 8.00% of SEED and 9.60% of SEED-IV samples were chosen as the labeled samples, respectively, the minimum training time reduction achieved by the proposed scheme over state-of-the-art schemes were 46.75s and 22.55s, respectively. iii) Graph fusion, network enhancement, and feature fusion all contributed to the performance enhancement. iv) The key hyperparameters that affect the performance were relatively few and easy to set to obtain outstanding performance. SIGNIFICANCE This paper demonstrated that the combination of graph fusion, network enhancement, and feature fusion help to enhance GCN-based EEG emotion recognition.
Collapse
Affiliation(s)
- Guangqiang Li
- School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, CHINA
| | - Ning Chen
- School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, CHINA
| | - Jing Jin
- School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, CHINA
| |
Collapse
|
35
|
Shi Y, Zheng X, Zhang M, Yan X, Li T, Yu X. A Study of Subliminal Emotion Classification Based on Entropy Features. Front Psychol 2022; 13:781448. [PMID: 35401346 PMCID: PMC8989849 DOI: 10.3389/fpsyg.2022.781448] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2021] [Accepted: 02/24/2022] [Indexed: 11/25/2022] Open
Abstract
Electroencephalogram (EEG) has been widely utilized in emotion recognition. Psychologists have found that emotions can be divided into conscious emotion and unconscious emotion. In this article, we explore to classify subliminal emotions (happiness and anger) with EEG signals elicited by subliminal face stimulation, that is to select appropriate features to classify subliminal emotions. First, multi-scale sample entropy (MSpEn), wavelet packet energy (Ei), and wavelet packet entropy (WpEn) of EEG signals are extracted. Then, these features are fed into the decision tree and improved random forest, respectively. The classification accuracy with Ei and WpEn is higher than MSpEn, which shows that Ei and WpEn can be used as effective features to classify subliminal emotions. We compared the classification results of different features combined with the decision tree algorithm and the improved random forest algorithm. The experimental results indicate that the improved random forest algorithm attains the best classification accuracy for subliminal emotions. Finally, subliminal emotions and physiological proof of subliminal affective priming effect are discussed.
Collapse
Affiliation(s)
- Yanjing Shi
- School of Information Science and Engineering, Shandong Normal University, Jinan, China
- Network Information Center, Shandong University of Political Science and Law, Jinan, China
| | - Xiangwei Zheng
- School of Information Science and Engineering, Shandong Normal University, Jinan, China
| | - Min Zhang
- School of Information Science and Engineering, Shandong Normal University, Jinan, China
| | - Xiaoyan Yan
- Geriatrics Center, Affiliated Hospital of Shandong University of Traditional Chinese Medicine, Jinan, China
- *Correspondence: Xiaoyan Yan
| | - Tiantian Li
- Faculty of Education, Shandong Normal University, Jinan, China
| | - Xiaomei Yu
- School of Information Science and Engineering, Shandong Normal University, Jinan, China
- Xiaomei Yu
| |
Collapse
|
36
|
Lehký D, Šomodíková M, Lipowczan M. A utilization of the inverse response surface method for the reliability-based design of structures. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07149-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
37
|
|
38
|
A novel deep ordinal classification approach for aesthetic quality control classification. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07050-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Abstract
AbstractNowadays, decision support systems (DSSs) are widely used in several application domains, from industrial to healthcare and medicine fields. Concerning the industrial scenario, we propose a DSS oriented to the aesthetic quality control (AQC) task, which has quickly established itself as one of the most crucial challenges of Industry 4.0. Taking into account the increasing amount of data in this domain, the application of machine learning (ML) and deep learning (DL) techniques offers great opportunities to automatize the overall AQC process. State-of-the-art is mainly oriented to approach this problem with a nominal DL classification method which does not exploit the ordinal structure of the AQC task, thus not penalizing the error among distant AQC classes (which is a relevant aspect for the real use case). The paper introduces a DL ordinal methodology for the AQC classification. Differently from other deep ordinal methods, we combined the standard categorical cross-entropy with the cumulative link model and we imposed the ordinal constraint via the thresholds and slope parameters. Experimental results were performed for solving an AQC task on a novel image dataset originated from a specific company’s demand (i.e., aesthetic assessment of wooden stocks). We demonstrated how the proposed methodology is able to reduce misclassification errors (up to 0.937 quadratic weight kappa loss) among distant classes while overcoming other state-of-the-art deep ordinal models and reducing the bias factor related to the item geometry. The proposed DL approach was integrated as the main core of a DSS supported by Internet of Things (IoT) architecture that can support the human operator by reducing up to 90% the time needed for the qualitative analysis carried out manually in this specific domain.
Collapse
|
39
|
Ballistocardiogram artifact removal in simultaneous EEG-fMRI using generative adversarial network. J Neurosci Methods 2022; 371:109498. [DOI: 10.1016/j.jneumeth.2022.109498] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2021] [Revised: 01/20/2022] [Accepted: 02/08/2022] [Indexed: 11/18/2022]
|
40
|
Wang Z, Zhang J, Zhang X, Chen P, Wang B. Transformer Model for Functional Near-Infrared Spectroscopy Classification. IEEE J Biomed Health Inform 2022; 26:2559-2569. [PMID: 34986110 DOI: 10.1109/jbhi.2022.3140531] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Functional near-infrared spectroscopy (fNIRS) is a promising neuroimaging technology. The fNIRS classification problem has always been the focus of the brain-computer interface (BCI). Inspired by the success of Transformer based on self-attention mechanism in the fields of natural language processing and computer vision, we propose an fNIRS classification network based on Transformer, named fNIRS-T. We explore the spatial-level and channel-level representation of fNIRS signals to improve data utilization and network representation capacity. Besides, a preprocessing module, which consists of one-dimensional average pooling and layer normalization, is designed to replace filtering and baseline correction of data preprocessing. It makes fNIRS-T an end-to-end network, called fNIRS-PreT. Compared with traditional machine learning classifiers, convolutional neural network (CNN), and long short-term memory (LSTM), the proposed models obtain the best accuracy on three open-access datasets. Specifically, in the most extensive ternary classification task (30 subjects) that includes three types of overt movements, fNIRS-T, CNN, and LSTM obtain 75.49%, 72.89%, and 61.94% on test sets, respectively. Compared to traditional classifiers, fNIRS-T is at least 27.41% higher than statistical features and 6.79% higher than well-designed features. In the individual subject experiment of the ternary classification task, fNIRS-T achieves an average subject accuracy of 78.22% and surpasses CNN and LSTM by a large margin of +4.75% and +11.33%. fNIRS-PreT using raw data also achieves competitive performance to fNIRS-T. Therefore, the proposed models improve the performance of fNIRS-based BCI significantly.
Collapse
|
41
|
Jin X, Zhang Z, Zhang L, Li L, Huang G. Using a new phase-locked visual feedback protocol to affirm simpler models for alpha dynamics. J Neurosci Methods 2022; 368:109473. [PMID: 34990698 DOI: 10.1016/j.jneumeth.2021.109473] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2021] [Revised: 12/04/2021] [Accepted: 12/30/2021] [Indexed: 11/28/2022]
Abstract
Alpha band oscillations are the most prominent rhythmic oscillations in EEG, which are related to various types of mental diseases, such as attention deficit hyperactivity disorder, anxiety, and depression. However, the dynamics of alpha oscillations, especially how the endogenous alpha oscillations be entrained by exogenous stimulus, are still unclear. Recently, a newly-developed phase-locked visual feedback (PLVF) protocol has shown effectiveness in modulating alpha rhythm, which provides empirical evidence for the further investigation of the neural mechanism of alpha dynamics. In this work, extensive numerical simulations based on four well-studied models were used to investigate the questions that (1) What kind of dynamic model exhibits a modulation phenomenon of PLVF? (2) What is the dynamic mechanism of PLVF for alpha modulation? (3) Which factors affect the modulation effects in PLVF? The result indicates that the dynamics of endogenous alpha oscillations are close to a simpler dynamic structure, like fixed-point attractor or limit-cycle attractor, which shows a global consistent dynamic behavior at different phases of the alpha oscillation. The further analysis explains the dynamic mechanism of PLVF for amplitude and frequency modulation of the alpha rhythm, as well as the influence of parameter settings in the modulation. All these findings provide a deeper understanding of the endogenous alpha oscillations entrained by exogenous phased locked visual stimulus and lead in turn to the refinement of a control strategy for alpha modulation, which could potentially be used in developing new neural modulation methods for cognitive enhancement and mental diseases treatment.
Collapse
Affiliation(s)
- Xingyi Jin
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong 518060, China; Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen, Guangdong 518060, China
| | - Zhiguo Zhang
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong 518060, China; Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen, Guangdong 518060, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen 518060, China; Peng Cheng Laboratory, Shenzhen, Guangdong 518055, China
| | - Li Zhang
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong 518060, China; Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen, Guangdong 518060, China
| | - Linling Li
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong 518060, China; Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen, Guangdong 518060, China
| | - Gan Huang
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong 518060, China; Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen, Guangdong 518060, China.
| |
Collapse
|
42
|
Duncan SJ, Kamyla M, Ferguson HJ, Wilkinson DT. Extraction of the GVS electrical artifact from EEG recordings of the motor related cortical potential. J Neurosci Methods 2021; 368:109459. [PMID: 34954254 DOI: 10.1016/j.jneumeth.2021.109459] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2021] [Revised: 12/20/2021] [Accepted: 12/20/2021] [Indexed: 11/29/2022]
Abstract
BACKGROUND Galvanic vestibular stimulation (GVS) involves the administration of low-amplitude trans-mastoidal current which induces a dense electrical field across the scalp that is difficult to remove from the EEG record. In two proof-of-concept experiments, we designed a paradigm to evaluate functional limb movement, and tested a method of blind source separation to remove the scalp artifact induced by low-amplitude, alternating current GVS to allow measurement of the motor-related cortical response (MRCP) during voluntary movement. NEW METHOD Off-line Extended Infomax Independent Component Analysis (ICA) was applied to the concatenated dataset to identify and remove core characteristics of the artifact induced by a trans-mastoidal current (Experiment 1: 0.01Hz, 0.2-3mA; Experiment 2: 0.01Hz, 0.3-0.4mA) during finger (Experiments 1 and 2) and foot tapping (Experiment 2). RESULTS In Experiment 1, a GVS-related independent component was identified and successfully removed without compromising measurement of the MRCP. This success was replicated in Experiment 2 which included both finger and foot tapping, and a higher GVS amplitude, which resulted in the identification of additional GVS-related artifacts. COMPARISON WITH EXISTING METHODS Existing methods of artifact rejection typically use an offline bandpass filter that overlaps with the frequency range of the MRCP. Even when similar ICA-based approaches have been employed, they have been applied during rest rather than active movement, have not been described in sufficient detail to enable replication, and require significant expertise and bespoke software to implement. CONCLUSION The ICA-based approach described here provides a relatively simple and accessible means by which MRCPs can be measured during alternating current GVS. This provides opportunity to identify new biomarkers associated with the therapeutic effects of GVS in people with Parkinson's disease and other disorders of voluntary movement.
Collapse
Affiliation(s)
- Shelley J Duncan
- Faculty of Sport, Health and Social Sciences, Solent University, Southampton, SO14 OYN, UK.
| | - Marques Kamyla
- School of Psychology, University of Kent, Canterbury, UK
| | | | | |
Collapse
|
43
|
Corona-González CE, Alonso-Valerdi LM, Ibarra-Zarate DI. Personalized Theta and Beta Binaural Beats for Brain Entrainment: An Electroencephalographic Analysis. Front Psychol 2021; 12:764068. [PMID: 34867666 PMCID: PMC8636003 DOI: 10.3389/fpsyg.2021.764068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2021] [Accepted: 10/04/2021] [Indexed: 11/22/2022] Open
Abstract
Binaural beats (BB) consist of two slightly distinct auditory frequencies (one in each ear), which are differentiated with clinical electroencephalographic (EEG) bandwidths, namely, delta, theta, alpha, beta, or gamma. This auditory stimulation has been widely used to module brain rhythms and thus inducing the mental condition associated with the EEG bandwidth in use. The aim of this research was to investigate whether personalized BB (specifically those within theta and beta EEG bands) improve brain entrainment. Personalized BB consisted of pure tones with a carrier tone of 500 Hz in the left ear together with an adjustable frequency in the right ear that was defined for theta BB (since fc for theta EEG band was 4.60 Hz ± 0.70 SD) and beta BB (since fc for beta EEG band was 18.42 Hz ± 2.82 SD). The adjustable frequencies were estimated for each participant in accordance with their heart rate by applying the Brain-Body Coupling Theorem postulated by Klimesch. To achieve this aim, 20 healthy volunteers were stimulated with their personalized theta and beta BB for 20 min and their EEG signals were collected with 22 channels. EEG analysis was based on the comparison of power spectral density among three mental conditions: (1) theta BB stimulation, (2) beta BB stimulation, and (3) resting state. Results showed larger absolute power differences for both BB stimulation sessions than resting state on bilateral temporal and parietal regions. This power change seems to be related to auditory perception and sound location. However, no significant differences were found between theta and beta BB sessions when it was expected to achieve different brain entrainments, since theta and beta BB induce relaxation and readiness, respectively. In addition, relative power analysis (theta BB/resting state) revealed alpha band desynchronization in the parieto-occipital region when volunteers listened to theta BB, suggesting that participants felt uncomfortable. In conclusion, neural resynchronization was met with both personalized theta and beta BB, but no different mental conditions seemed to be achieved.
Collapse
|
44
|
Gong S, Xing K, Cichocki A, Li J. Deep Learning in EEG: Advance of the Last Ten-Year Critical Period. IEEE Trans Cogn Dev Syst 2021. [DOI: 10.1109/tcds.2021.3079712] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
45
|
Li W, Huan W, Hou B, Tian Y, Zhang Z, Song A. Can Emotion be Transferred? – A Review on Transfer Learning for EEG-Based Emotion Recognition. IEEE Trans Cogn Dev Syst 2021. [DOI: 10.1109/tcds.2021.3098842] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
46
|
Torkamani-Azar M, Kanik SD, Aydin S, Cetin M. Prediction of Reaction Time and Vigilance Variability From Spatio-Spectral Features of Resting-State EEG in a Long Sustained Attention Task. IEEE J Biomed Health Inform 2020; 24:2550-2558. [DOI: 10.1109/jbhi.2020.2980056] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|