1
|
Wang X, Yang W, Qi W, Wang Y, Ma X, Wang W. STaRNet: A spatio-temporal and Riemannian network for high-performance motor imagery decoding. Neural Netw 2024; 178:106471. [PMID: 38945115 DOI: 10.1016/j.neunet.2024.106471] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2024] [Revised: 06/11/2024] [Accepted: 06/16/2024] [Indexed: 07/02/2024]
Abstract
Brain-computer interfaces (BCIs), representing a transformative form of human-computer interaction, empower users to interact directly with external environments through brain signals. In response to the demands for high accuracy, robustness, and end-to-end capabilities within BCIs based on motor imagery (MI), this paper introduces STaRNet, a novel model that integrates multi-scale spatio-temporal convolutional neural networks (CNNs) with Riemannian geometry. Initially, STaRNet integrates a multi-scale spatio-temporal feature extraction module that captures both global and local features, facilitating the construction of Riemannian manifolds from these comprehensive spatio-temporal features. Subsequently, a matrix logarithm operation transforms the manifold-based features into the tangent space, followed by a dense layer for classification. Without preprocessing, STaRNet surpasses state-of-the-art (SOTA) models by achieving an average decoding accuracy of 83.29% and a kappa value of 0.777 on the BCI Competition IV 2a dataset, and 95.45% accuracy with a kappa value of 0.939 on the High Gamma Dataset. Additionally, a comparative analysis between STaRNet and several SOTA models, focusing on the most challenging subjects from both datasets, highlights exceptional robustness of STaRNet. Finally, the visualizations of learned frequency bands demonstrate that temporal convolutions have learned MI-related frequency bands, and the t-SNE analyses of features across multiple layers of STaRNet exhibit strong feature extraction capabilities. We believe that the accurate, robust, and end-to-end capabilities of the STaRNet will facilitate the advancement of BCIs.
Collapse
Affiliation(s)
- Xingfu Wang
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Wenjie Yang
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Wenxia Qi
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Yu Wang
- National Engineering and Technology Research Center for ASIC Design, Institute of Automation, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Xiaojun Ma
- National Engineering and Technology Research Center for ASIC Design, Institute of Automation, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Wei Wang
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China.
| |
Collapse
|
2
|
Wang Y, Peng Y, Han M, Liu X, Niu H, Cheng J, Chang S, Liu T. GCTNet: a graph convolutional transformer network for major depressive disorder detection based on EEG signals. J Neural Eng 2024; 21:036042. [PMID: 38788706 DOI: 10.1088/1741-2552/ad5048] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2023] [Accepted: 05/24/2024] [Indexed: 05/26/2024]
Abstract
Objective.Identifying major depressive disorder (MDD) using objective physiological signals has become a pressing challenge.Approach.Hence, this paper proposes a graph convolutional transformer network (GCTNet) for accurate and reliable MDD detection using electroencephalogram (EEG) signals. The developed framework integrates a residual graph convolutional network block to capture spatial information and a Transformer block to extract global temporal dynamics. Additionally, we introduce the contrastive cross-entropy (CCE) loss that combines contrastive learning to enhance the stability and discriminability of the extracted features, thereby improving classification performance.Main results. The effectiveness of the GCTNet model and CCE loss was assessed using EEG data from 41 MDD patients and 44 normal controls, in addition to a publicly available dataset. Utilizing a subject-independent data partitioning method and 10-fold cross-validation, the proposed method demonstrated significant performance, achieving an average Area Under the Curve of 0.7693 and 0.9755 across both datasets, respectively. Comparative analyses demonstrated the superiority of the GCTNet framework with CCE loss over state-of-the-art algorithms in MDD detection tasks.Significance. The proposed method offers an objective and effective approach to MDD detection, providing valuable support for clinical-assisted diagnosis.
Collapse
Affiliation(s)
- Yuwen Wang
- Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, People's Republic of China
| | - Yudan Peng
- Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, People's Republic of China
| | - Mingxiu Han
- Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, People's Republic of China
| | - Xinyi Liu
- Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, People's Republic of China
| | - Haijun Niu
- Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, People's Republic of China
| | - Jian Cheng
- School of Computer Science and Engineering, Beihang University, Beijing, People's Republic of China
| | - Suhua Chang
- Peking University Sixth Hospital, Peking University Institute of Mental Health, NHC Key Laboratory of Mental Health (Peking University), National Clinical Research Center for Mental Disorders (Peking University Sixth Hospital), Beijing, People's Republic of China
| | - Tao Liu
- Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, People's Republic of China
| |
Collapse
|
3
|
Yang K, Wang J, Yang L, Bian L, Luo Z, Yang C. A diagonal masking self-attention-based multi-scale network for motor imagery classification. J Neural Eng 2024; 21:036040. [PMID: 38834056 DOI: 10.1088/1741-2552/ad5405] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2024] [Accepted: 06/04/2024] [Indexed: 06/06/2024]
Abstract
Objective. Electroencephalography (EEG)-based motor imagery (MI) is a promising paradigm for brain-computer interface (BCI), but the non-stationarity and low signal-to-noise ratio of EEG signals make it a challenging task.Approach. To achieve high-precision MI classification, we propose a Diagonal Masking Self-Attention-based Multi-Scale Network (DMSA-MSNet) to fully develop, extract, and emphasize features from different scales. First, for local features, a multi-scale temporal-spatial block is proposed to extract features from different receptive fields. Second, an adaptive branch fusion block is specifically designed to bridge the semantic gap between these coded features from different scales. Finally, in order to analyze global information over long ranges, a diagonal masking self-attention block is introduced, which highlights the most valuable features in the data.Main results. The proposed DMSA-MSNet outperforms state-of-the-art models on the BCI Competition IV 2a and the BCI Competition IV 2b datasets.Significance. Our study achieves rich information extraction from EEG signals and provides an effective solution for MI classification.
Collapse
Affiliation(s)
- Kaijun Yang
- Power Systems Engineering Research Center, Ministry of Education, College of Big Data and Information Engineering, Guizhou University, Guiyang 550025, People's Republic of China
| | - Jihong Wang
- Power Systems Engineering Research Center, Ministry of Education, College of Big Data and Information Engineering, Guizhou University, Guiyang 550025, People's Republic of China
| | - Liantao Yang
- Power Systems Engineering Research Center, Ministry of Education, College of Big Data and Information Engineering, Guizhou University, Guiyang 550025, People's Republic of China
| | - Lifeng Bian
- Frontier Institute of Chip and System, Fudan University, Shanghai 200433, People's Republic of China
| | - Zijiang Luo
- Institute of Intelligent Manufacturing, Shunde Polytechnic, Foshan 528300, People's Republic of China
| | - Chen Yang
- Power Systems Engineering Research Center, Ministry of Education, College of Big Data and Information Engineering, Guizhou University, Guiyang 550025, People's Republic of China
| |
Collapse
|
4
|
Pfeffer MA, Ling SSH, Wong JKW. Exploring the frontier: Transformer-based models in EEG signal analysis for brain-computer interfaces. Comput Biol Med 2024; 178:108705. [PMID: 38865781 DOI: 10.1016/j.compbiomed.2024.108705] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2023] [Revised: 05/01/2024] [Accepted: 06/01/2024] [Indexed: 06/14/2024]
Abstract
This review systematically explores the application of transformer-based models in EEG signal processing and brain-computer interface (BCI) development, with a distinct focus on ensuring methodological rigour and adhering to empirical validations within the existing literature. By examining various transformer architectures, such as the Temporal Spatial Transformer Network (TSTN) and EEG Conformer, this review delineates their capabilities in mitigating challenges intrinsic to EEG data, such as noise and artifacts, and their subsequent implications on decoding and classification accuracies across disparate mental tasks. The analytical scope extends to a meticulous examination of attention mechanisms within transformer models, delineating their role in illuminating critical temporal and spatial EEG features and facilitating interpretability in model decision-making processes. The discourse additionally encapsulates emerging works that substantiate the efficacy of transformer models in noise reduction of EEG signals and diversifying applications beyond the conventional motor imagery paradigm. Furthermore, this review elucidates evident gaps and propounds exploratory avenues in the applications of pre-trained transformers in EEG analysis and the potential expansion into real-time and multi-task BCI applications. Collectively, this review distils extant knowledge, navigates through the empirical findings, and puts forward a structured synthesis, thereby serving as a conduit for informed future research endeavours in transformer-enhanced, EEG-based BCI systems.
Collapse
Affiliation(s)
- Maximilian Achim Pfeffer
- Faculty of Engineering and Information Technology, University of Technology Sydney, CB11 81-113, Broadway, Ultimo, 2007, New South Wales, Australia.
| | - Steve Sai Ho Ling
- Faculty of Engineering and Information Technology, University of Technology Sydney, CB11 81-113, Broadway, Ultimo, 2007, New South Wales, Australia.
| | - Johnny Kwok Wai Wong
- Faculty of Design, Architecture and Building, University of Technology Sydney, 15 Broadway, Ultimo, 2007, New South Wales, Australia.
| |
Collapse
|
5
|
Qian D, Zeng H, Cheng W, Liu Y, Bikki T, Pan J. NeuroDM: Decoding and visualizing human brain activity with EEG-guided diffusion model. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 251:108213. [PMID: 38744056 DOI: 10.1016/j.cmpb.2024.108213] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/11/2024] [Revised: 04/30/2024] [Accepted: 05/03/2024] [Indexed: 05/16/2024]
Abstract
BACKGROUND AND OBJECTIVE Brain-Computer Interface (BCI) technology has recently been advancing rapidly, bringing significant hope for improving human health and quality of life. Decoding and visualizing visually evoked electroencephalography (EEG) signals into corresponding images plays a crucial role in the practical application of BCI technology. The recent emergence of diffusion models provides a good modeling basis for this work. However, the existing diffusion models still have great challenges in generating high-quality images from EEG, due to the low signal-to-noise ratio and strong randomness of EEG signals. The purpose of this study is to address the above-mentioned challenges by proposing a framework named NeuroDM that can decode human brain responses to visual stimuli from EEG-recorded brain activity. METHODS In NeuroDM, an EEG-Visual-Transformer (EV-Transformer) is used to extract the visual-related features with high classification accuracy from EEG signals, then an EEG-Guided Diffusion Model (EG-DM) is employed to synthesize high-quality images from the EEG visual-related features. RESULTS We conducted experiments on two EEG datasets (one is a forty-class dataset, and the other is a four-class dataset). In the task of EEG decoding, we achieved average accuracies of 99.80% and 92.07% on two datasets, respectively. In the task of EEG visualization, the Inception Score of the images generated by NeuroDM reached 15.04 and 8.67, respectively. All the above results outperform existing methods. CONCLUSIONS The experimental results on two EEG datasets demonstrate the effectiveness of the NeuroDM framework, achieving state-of-the-art performance in terms of classification accuracy and image quality. Furthermore, our NeuroDM exhibits strong generalization capabilities and the ability to generate diverse images.
Collapse
Affiliation(s)
- Dongguan Qian
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, China
| | - Hong Zeng
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, China
| | - Wenjie Cheng
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, China
| | - Yu Liu
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, China
| | - Taha Bikki
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, China
| | - Jianjiang Pan
- School of Sciences, Hangzhou Dianzi University, Hangzhou, Zhejiang, China.
| |
Collapse
|
6
|
Ma X, Chen W, Pei Z, Zhang Y, Chen J. Attention-based convolutional neural network with multi-modal temporal information fusion for motor imagery EEG decoding. Comput Biol Med 2024; 175:108504. [PMID: 38701593 DOI: 10.1016/j.compbiomed.2024.108504] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2023] [Revised: 04/15/2024] [Accepted: 04/21/2024] [Indexed: 05/05/2024]
Abstract
Convolutional neural network (CNN) has been widely applied in motor imagery (MI)-based brain computer interface (BCI) to decode electroencephalography (EEG) signals. However, due to the limited perceptual field of convolutional kernel, CNN only extracts features from local region without considering long-term dependencies for EEG decoding. Apart from long-term dependencies, multi-modal temporal information is equally important for EEG decoding because it can offer a more comprehensive understanding of the temporal dynamics of neural processes. In this paper, we propose a novel deep learning network that combines CNN with self-attention mechanism to encapsulate multi-modal temporal information and global dependencies. The network first extracts multi-modal temporal information from two distinct perspectives: average and variance. A shared self-attention module is then designed to capture global dependencies along these two feature dimensions. We further design a convolutional encoder to explore the relationship between average-pooled and variance-pooled features and fuse them into more discriminative features. Moreover, a data augmentation method called signal segmentation and recombination is proposed to improve the generalization capability of the proposed network. The experimental results on the BCI Competition IV-2a (BCIC-IV-2a) and BCI Competition IV-2b (BCIC-IV-2b) datasets show that our proposed method outperforms the state-of-the-art methods and achieves 4-class average accuracy of 85.03% on the BCIC-IV-2a dataset. The proposed method implies the effectiveness of multi-modal temporal information fusion in attention-based deep learning networks and provides a new perspective for MI-EEG decoding. The code is available at https://github.com/Ma-Xinzhi/EEG-TransNet.
Collapse
Affiliation(s)
- Xinzhi Ma
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China; Hangzhou Innovation Institute, Beihang University, Hangzhou, China
| | - Weihai Chen
- School of Electrical Engineering and Automation, Anhui University, Hefei, China.
| | - Zhongcai Pei
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China; Hangzhou Innovation Institute, Beihang University, Hangzhou, China
| | - Yue Zhang
- Hangzhou Innovation Institute, Beihang University, Hangzhou, China
| | - Jianer Chen
- Department of Geriatric Rehabilitation, Third Affiliated Hospital, Zhejiang Chinese Medical University, Hangzhou, China
| |
Collapse
|
7
|
Lützow Holm E, Fernández Slezak D, Tagliazucchi E. Contribution of low-level image statistics to EEG decoding of semantic content in multivariate and univariate models with feature optimization. Neuroimage 2024; 293:120626. [PMID: 38677632 DOI: 10.1016/j.neuroimage.2024.120626] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2024] [Revised: 04/23/2024] [Accepted: 04/24/2024] [Indexed: 04/29/2024] Open
Abstract
Spatio-temporal patterns of evoked brain activity contain information that can be used to decode and categorize the semantic content of visual stimuli. However, this procedure can be biased by low-level image features independently of the semantic content present in the stimuli, prompting the need to understand the robustness of different models regarding these confounding factors. In this study, we trained machine learning models to distinguish between concepts included in the publicly available THINGS-EEG dataset using electroencephalography (EEG) data acquired during a rapid serial visual presentation paradigm. We investigated the contribution of low-level image features to decoding accuracy in a multivariate model, utilizing broadband data from all EEG channels. Additionally, we explored a univariate model obtained through data-driven feature selection applied to the spatial and frequency domains. While the univariate models exhibited better decoding accuracy, their predictions were less robust to the confounding effect of low-level image statistics. Notably, some of the models maintained their accuracy even after random replacement of the training dataset with semantically unrelated samples that presented similar low-level content. In conclusion, our findings suggest that model optimization impacts sensitivity to confounding factors, regardless of the resulting classification performance. Therefore, the choice of EEG features for semantic decoding should ideally be informed by criteria beyond classifier performance, such as the neurobiological mechanisms under study.
Collapse
Affiliation(s)
- Eric Lützow Holm
- National Scientific and Technical Research Council (CONICET), Godoy Cruz 2290, CABA 1425, Argentina; Institute of Applied and Interdisciplinary Physics and Department of Physics, University of Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina.
| | - Diego Fernández Slezak
- National Scientific and Technical Research Council (CONICET), Godoy Cruz 2290, CABA 1425, Argentina; Departamento de Computación, Facultad de Ciencias Exactas y Naturales, Universidad de Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina; Instituto de Investigación en Ciencias de la Computación (ICC), CONICET-Universidad de Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina
| | - Enzo Tagliazucchi
- National Scientific and Technical Research Council (CONICET), Godoy Cruz 2290, CABA 1425, Argentina; Institute of Applied and Interdisciplinary Physics and Department of Physics, University of Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina; Latin American Brain Health (BrainLat), Universidad Adolfo Ibáñez, Av. Diag. Las Torres 2640, Peñalolén 7941169, Santiago Región Metropolitana, Chile.
| |
Collapse
|
8
|
Liu L, Li J, Ouyang R, Zhou D, Fan C, Liang W, Li F, Lv Z, Wu X. Multimodal brain-controlled system for rehabilitation training: Combining asynchronous online brain-computer interface and exoskeleton. J Neurosci Methods 2024; 406:110132. [PMID: 38604523 DOI: 10.1016/j.jneumeth.2024.110132] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2023] [Revised: 03/11/2024] [Accepted: 04/03/2024] [Indexed: 04/13/2024]
Abstract
BACKGROUND Traditional therapist-based rehabilitation training for patients with movement impairment is laborious and expensive. In order to reduce the cost and improve the treatment effect of rehabilitation, many methods based on human-computer interaction (HCI) technology have been proposed, such as robot-assisted therapy and functional electrical stimulation (FES). However, due to the lack of active participation of brain, these methods have limited effects on the promotion of damaged nerve remodeling. NEW METHOD Based on the neurofeedback training provided by the combination of brain-computer interface (BCI) and exoskeleton, this paper proposes a multimodal brain-controlled active rehabilitation system to help improve limb function. The joint control mode of steady-state visual evoked potential (SSVEP) and motor imagery (MI) is adopted to achieve self-paced control and thus maximize the degree of brain involvement, and a requirement selection function based on SSVEP design is added to facilitate communication with aphasia patients. COMPARISON WITH EXISTING METHODS In addition, the Transformer is introduced as the MI decoder in the asynchronous online BCI to improve the global perception of electroencephalogram (EEG) signals and maintain the sensitivity and efficiency of the system. RESULTS In two multi-task online experiments for left hand, right hand, foot and idle states, subject achieves 91.25% and 92.50% best accuracy, respectively. CONCLUSION Compared with previous studies, this paper aims to establish a high-performance and low-latency brain-controlled rehabilitation system, and provide an independent and autonomous control mode of the brain, so as to improve the effect of neural remodeling. The performance of the proposed method is evaluated through offline and online experiments.
Collapse
Affiliation(s)
- Lei Liu
- School of Computer Science and Technology, Anhui University, Hefei 230601, China; Anhui Province Key Laboratory of Multimodal Cognitive Computation, Anhui University, Hefei 230601, China
| | - Jian Li
- School of Computer Science and Technology, Anhui University, Hefei 230601, China; Anhui Province Key Laboratory of Multimodal Cognitive Computation, Anhui University, Hefei 230601, China
| | - Rui Ouyang
- School of Computer Science and Technology, Anhui University, Hefei 230601, China; Anhui Province Key Laboratory of Multimodal Cognitive Computation, Anhui University, Hefei 230601, China
| | - Danya Zhou
- National Centre for International Research in Cell and Gene Therapy, School of Basic Medical Sciences, Academy of Medical Sciences, Zhengzhou University, Zhengzhou, China
| | - Cunhang Fan
- School of Computer Science and Technology, Anhui University, Hefei 230601, China; Anhui Province Key Laboratory of Multimodal Cognitive Computation, Anhui University, Hefei 230601, China.
| | - Wen Liang
- Google Inc, United States of America
| | - Fan Li
- Civil Aviation Flight University of China, China
| | - Zhao Lv
- Anhui Province Key Laboratory of Multimodal Cognitive Computation, Anhui University, Hefei 230601, China; Civil Aviation Flight University of China, China
| | - Xiaopei Wu
- School of Computer Science and Technology, Anhui University, Hefei 230601, China; Anhui Province Key Laboratory of Multimodal Cognitive Computation, Anhui University, Hefei 230601, China.
| |
Collapse
|
9
|
Tong C, Ding Y, Zhang Z, Zhang H, JunLiang Lim K, Guan C. TASA: Temporal Attention With Spatial Autoencoder Network for Odor-Induced Emotion Classification Using EEG. IEEE Trans Neural Syst Rehabil Eng 2024; 32:1944-1954. [PMID: 38722724 DOI: 10.1109/tnsre.2024.3399326] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/21/2024]
Abstract
The olfactory system enables humans to smell different odors, which are closely related to emotions. The high temporal resolution and non-invasiveness of Electroencephalogram (EEG) make it suitable to objectively study human preferences for odors. Effectively learning the temporal dynamics and spatial information from EEG is crucial for detecting odor-induced emotional valence. In this paper, we propose a deep learning architecture called Temporal Attention with Spatial Autoencoder Network (TASA) for predicting odor-induced emotions using EEG. TASA consists of a filter-bank layer, a spatial encoder, a time segmentation layer, a Long Short-Term Memory (LSTM) module, a multi-head self-attention (MSA) layer, and a fully connected layer. We improve upon the previous work by utilizing a two-phase learning framework, using the autoencoder module to learn the spatial information among electrodes by reconstructing the given input with a latent representation in the spatial dimension, which aims to minimize information loss compared to spatial filtering with CNN. The second improvement is inspired by the continuous nature of the olfactory process; we propose to use LSTM-MSA in TASA to capture its temporal dynamics by learning the intercorrelation among the time segments of the EEG. TASA is evaluated on an existing olfactory EEG dataset and compared with several existing deep learning architectures to demonstrate its effectiveness in predicting olfactory-triggered emotional responses. Interpretability analyses with DeepLIFT also suggest that TASA learns spatial-spectral features that are relevant to olfactory-induced emotion recognition.
Collapse
|
10
|
Komeiji S, Mitsuhashi T, Iimura Y, Suzuki H, Sugano H, Shinoda K, Tanaka T. Feasibility of decoding covert speech in ECoG with a Transformer trained on overt speech. Sci Rep 2024; 14:11491. [PMID: 38769115 PMCID: PMC11106343 DOI: 10.1038/s41598-024-62230-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2024] [Accepted: 05/15/2024] [Indexed: 05/22/2024] Open
Abstract
Several attempts for speech brain-computer interfacing (BCI) have been made to decode phonemes, sub-words, words, or sentences using invasive measurements, such as the electrocorticogram (ECoG), during auditory speech perception, overt speech, or imagined (covert) speech. Decoding sentences from covert speech is a challenging task. Sixteen epilepsy patients with intracranially implanted electrodes participated in this study, and ECoGs were recorded during overt speech and covert speech of eight Japanese sentences, each consisting of three tokens. In particular, Transformer neural network model was applied to decode text sentences from covert speech, which was trained using ECoGs obtained during overt speech. We first examined the proposed Transformer model using the same task for training and testing, and then evaluated the model's performance when trained with overt task for decoding covert speech. The Transformer model trained on covert speech achieved an average token error rate (TER) of 46.6% for decoding covert speech, whereas the model trained on overt speech achieved a TER of 46.3% ( p > 0.05 ; d = 0.07 ) . Therefore, the challenge of collecting training data for covert speech can be addressed using overt speech. The performance of covert speech can improve by employing several overt speeches.
Collapse
Affiliation(s)
- Shuji Komeiji
- Department of Electronic and Information Engineering, Tokyo University of Agriculture and Technology, 2-24-16 Naka-cho, Koganei-shi, Tokyo, 184-8588, Japan
| | - Takumi Mitsuhashi
- Department of Neurosurgery, Juntendo University School of Medicine, 2-1-1 Hongo, Bunkyo-ku, Tokyo, 113-8421, Japan
| | - Yasushi Iimura
- Department of Neurosurgery, Juntendo University School of Medicine, 2-1-1 Hongo, Bunkyo-ku, Tokyo, 113-8421, Japan
| | - Hiroharu Suzuki
- Department of Neurosurgery, Juntendo University School of Medicine, 2-1-1 Hongo, Bunkyo-ku, Tokyo, 113-8421, Japan
| | - Hidenori Sugano
- Department of Neurosurgery, Juntendo University School of Medicine, 2-1-1 Hongo, Bunkyo-ku, Tokyo, 113-8421, Japan
| | - Koichi Shinoda
- Department of Computer Science, Tokyo Institute of Technology, 2-12-1 Ookayama, Meguro-ku, Tokyo, 152-8550, Japan
| | - Toshihisa Tanaka
- Department of Electronic and Information Engineering, Tokyo University of Agriculture and Technology, 2-24-16 Naka-cho, Koganei-shi, Tokyo, 184-8588, Japan.
| |
Collapse
|
11
|
Eldawlatly S. On the role of generative artificial intelligence in the development of brain-computer interfaces. BMC Biomed Eng 2024; 6:4. [PMID: 38698495 PMCID: PMC11064240 DOI: 10.1186/s42490-024-00080-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2023] [Accepted: 04/24/2024] [Indexed: 05/05/2024] Open
Abstract
Since their inception more than 50 years ago, Brain-Computer Interfaces (BCIs) have held promise to compensate for functions lost by people with disabilities through allowing direct communication between the brain and external devices. While research throughout the past decades has demonstrated the feasibility of BCI to act as a successful assistive technology, the widespread use of BCI outside the lab is still beyond reach. This can be attributed to a number of challenges that need to be addressed for BCI to be of practical use including limited data availability, limited temporal and spatial resolutions of brain signals recorded non-invasively and inter-subject variability. In addition, for a very long time, BCI development has been mainly confined to specific simple brain patterns, while developing other BCI applications relying on complex brain patterns has been proven infeasible. Generative Artificial Intelligence (GAI) has recently emerged as an artificial intelligence domain in which trained models can be used to generate new data with properties resembling that of available data. Given the enhancements observed in other domains that possess similar challenges to BCI development, GAI has been recently employed in a multitude of BCI development applications to generate synthetic brain activity; thereby, augmenting the recorded brain activity. Here, a brief review of the recent adoption of GAI techniques to overcome the aforementioned BCI challenges is provided demonstrating the enhancements achieved using GAI techniques in augmenting limited EEG data, enhancing the spatiotemporal resolution of recorded EEG data, enhancing cross-subject performance of BCI systems and implementing end-to-end BCI applications. GAI could represent the means by which BCI would be transformed into a prevalent assistive technology, thereby improving the quality of life of people with disabilities, and helping in adopting BCI as an emerging human-computer interaction technology for general use.
Collapse
Affiliation(s)
- Seif Eldawlatly
- Computer and Systems Engineering Department, Faculty of Engineering, Ain Shams University, 1 El-Sarayat St., Abbassia, Cairo, Egypt.
- Computer Science and Engineering Department, The American University in Cairo, Cairo, Egypt.
| |
Collapse
|
12
|
Qin Y, Zhang W, Tao X. TBEEG: A Two-Branch Manifold Domain Enhanced Transformer Algorithm for Learning EEG Decoding. IEEE Trans Neural Syst Rehabil Eng 2024; 32:1466-1476. [PMID: 38526885 DOI: 10.1109/tnsre.2024.3380595] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/27/2024]
Abstract
The electroencephalogram-based (EEG) brain-computer interface (BCI) has garnered significant attention in recent research. However, the practicality of EEG remains constrained by the lack of efficient EEG decoding technology. The challenge lies in effectively translating intricate EEG into meaningful, generalizable information. EEG signal decoding primarily relies on either time domain or frequency domain information. There lacks a method capable of simultaneously and effectively extracting both time and frequency domain features, as well as efficiently fuse these features. Addressing these limitations, a two-branch Manifold Domain enhanced transformer algorithm is designed to holistically capture EEG's spatio-temporal information. Our method projects the time-domain information of EEG signals into the Riemannian spaces to fully decode the time dependence of EEG signals. Using wavelet transform, the time domain information is converted into frequency domain information, and the spatial information contained in the frequency domain information of EEG signal is mined through the spectrogram. The effectiveness of the proposed TBEEG algorithm is validated on BCIC-IV-2a dataset and MAMEM-SSVEP-II datasets.
Collapse
|
13
|
Xue Q, Song Y, Wu H, Cheng Y, Pan H. Graph neural network based on brain inspired forward-forward mechanism for motor imagery classification in brain-computer interfaces. Front Neurosci 2024; 18:1309594. [PMID: 38606308 PMCID: PMC11008472 DOI: 10.3389/fnins.2024.1309594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2023] [Accepted: 03/04/2024] [Indexed: 04/13/2024] Open
Abstract
Introduction Within the development of brain-computer interface (BCI) systems, it is crucial to consider the impact of brain network dynamics and neural signal transmission mechanisms on electroencephalogram-based motor imagery (MI-EEG) tasks. However, conventional deep learning (DL) methods cannot reflect the topological relationship among electrodes, thereby hindering the effective decoding of brain activity. Methods Inspired by the concept of brain neuronal forward-forward (F-F) mechanism, a novel DL framework based on Graph Neural Network combined forward-forward mechanism (F-FGCN) is presented. F-FGCN framework aims to enhance EEG signal decoding performance by applying functional topological relationships and signal propagation mechanism. The fusion process involves converting the multi-channel EEG into a sequence of signals and constructing a network grounded on the Pearson correlation coeffcient, effectively representing the associations between channels. Our model initially pre-trains the Graph Convolutional Network (GCN), and fine-tunes the output layer to obtain the feature vector. Moreover, the F-F model is used for advanced feature extraction and classification. Results and discussion Achievement of F-FGCN is assessed on the PhysioNet dataset for a four-class categorization, compared with various classical and state-of-the-art models. The learned features of the F-FGCN substantially amplify the performance of downstream classifiers, achieving the highest accuracy of 96.11% and 82.37% at the subject and group levels, respectively. Experimental results affirm the potency of FFGCN in enhancing EEG decoding performance, thus paving the way for BCI applications.
Collapse
Affiliation(s)
- Qiwei Xue
- Institute of Plasma Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, China
- University of Science and Technology of China, Hefei, China
- Mechanical Department, School of Energy Systems, Lappeenranta University of Technology (LUT), Lappeenranta, Finland
| | - Yuntao Song
- Institute of Plasma Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, China
- University of Science and Technology of China, Hefei, China
| | - Huapeng Wu
- Mechanical Department, School of Energy Systems, Lappeenranta University of Technology (LUT), Lappeenranta, Finland
| | - Yong Cheng
- Institute of Plasma Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, China
| | - Hongtao Pan
- Institute of Plasma Physics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, China
| |
Collapse
|
14
|
Wang Z, Wang Y, Wan X, Tang Y. Cerebral asymmetry representation learning-based deep subdomain adaptation network for electroencephalogram-based emotion recognition. Physiol Meas 2024; 45:035004. [PMID: 38422513 DOI: 10.1088/1361-6579/ad2eb6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2023] [Accepted: 02/29/2024] [Indexed: 03/02/2024]
Abstract
Objective.Extracting discriminative spatial information from multiple electrodes is a crucial and challenging problem for electroencephalogram (EEG)-based emotion recognition. Additionally, the domain shift caused by the individual differences degrades the performance of cross-subject EEG classification.Approach.To deal with the above problems, we propose the cerebral asymmetry representation learning-based deep subdomain adaptation network (CARL-DSAN) to enhance cross-subject EEG-based emotion recognition. Specifically, the CARL module is inspired by the neuroscience findings that asymmetrical activations of the left and right brain hemispheres occur during cognitive and affective processes. In the CARL module, we introduce a novel two-step strategy for extracting discriminative features through intra-hemisphere spatial learning and asymmetry representation learning. Moreover, the transformer encoders within the CARL module can emphasize the contributive electrodes and electrode pairs. Subsequently, the DSAN module, known for its superior performance over global domain adaptation, is adopted to mitigate domain shift and further improve the cross-subject performance by aligning relevant subdomains that share the same class samples.Main Results.To validate the effectiveness of the CARL-DSAN, we conduct subject-independent experiments on the DEAP database, achieving accuracies of 68.67% and 67.11% for arousal and valence classification, respectively, and corresponding accuracies of 67.70% and 67.18% on the MAHNOB-HCI database.Significance.The results demonstrate that CARL-DSAN can achieve an outstanding cross-subject performance in both arousal and valence classification.
Collapse
Affiliation(s)
- Zhe Wang
- The School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093, People's Republic of China
| | - Yongxiong Wang
- The School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093, People's Republic of China
| | - Xin Wan
- The School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093, People's Republic of China
| | - Yiheng Tang
- The School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093, People's Republic of China
| |
Collapse
|
15
|
Zhang J, Liu D, Chen W, Pei Z, Wang J. Boosting lower-limb motor imagery performance through an ensemble method for gait rehabilitation. Comput Biol Med 2024; 169:107910. [PMID: 38183703 DOI: 10.1016/j.compbiomed.2023.107910] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2023] [Revised: 12/05/2023] [Accepted: 12/24/2023] [Indexed: 01/08/2024]
Abstract
Lower-limb exoskeletons have been used extensively in many rehabilitation applications to assist disabled people with their therapies. Brain-machine interfaces (BMIs) further provide effective and natural control schemes. However, the limited performance of brain signal decoding from lower-limb kinematics restricts the broad growth of both BMI and rehabilitation industry. To address these challenges, we propose an ensemble method for lower-limb motor imagery (MI) classification. The proposed model employs multiple techniques to boost performance, including deep and shallow parts. Traditional wavelet transformation followed by filter-bank common spatial pattern (CSP) employs neurophysiologically reasonable patterns, while multi-head self-attention (MSA) followed by temporal convolutional network (TCN) extracts deeper encoded generalized patterns. Experimental results in a customized lower-limb exoskeleton on 8 subjects in 3 consecutive sessions showed that the proposed method achieved 60.27% and 64.20% for three (MI of left leg, MI of right leg, and rest) and two classes (lower-limb MI vs. rest), respectively. Besides, the proposed model achieves improvements of up to 4% and 2% accuracy for the subject-specific and subject-independent modes compared to the current state-of-the-art (SOTA) techniques, respectively. Finally, feature analysis was conducted to show discriminative brain patterns in each MI task and sessions with different feedback modalities. The proposed models integrated in the brain-actuated lower-limb exoskeleton established a potential BMI for gait training and neuroprosthesis.
Collapse
Affiliation(s)
- Jing Zhang
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China; Hangzhou Innovation Institute, Beihang University, Hangzhou, Zhejiang 310052, China.
| | - Dong Liu
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China.
| | - Weihai Chen
- School of Electrical Engineering and Automation, Anhui University, Hefei, Anhui 230601, China.
| | - Zhongcai Pei
- Hangzhou Innovation Institute, Beihang University, Hangzhou, Zhejiang 310052, China.
| | - Jianhua Wang
- Hangzhou Innovation Institute, Beihang University, Hangzhou, Zhejiang 310052, China.
| |
Collapse
|
16
|
Li Z, Zhang R, Zeng Y, Tong L, Lu R, Yan B. MST-net: A multi-scale swin transformer network for EEG-based cognitive load assessment. Brain Res Bull 2024; 206:110834. [PMID: 38049039 DOI: 10.1016/j.brainresbull.2023.110834] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2023] [Revised: 10/22/2023] [Accepted: 11/29/2023] [Indexed: 12/06/2023]
Abstract
Cognitive load assessment plays a crucial role in monitoring safe production, resource allocation, and subjective initiative in human-computer interaction. Due to its high time resolution and convenient acquisition, Electroencephalography (EEG) is widely applied in brain monitoring and cognitive state assessment. In this study, a multi-scale Swin Transformer network (MST-Net) was proposed for cognitive load assessment, which extracts local features with different sensory fields using a multi-scale parallel convolution model and introduces the attention mechanism of the Swin Transformer to obtain the feature correlations among multi-scale local features. The performance of the proposed network was validated using the EEG signals collected during cognitive tasks and N-back tasks with three different load levels. Results show that the MST-Net network achieved the best classification accuracy on both local and public datasets, and was higher than the mainstream Swin Transformer and CNN. Furthermore, results of ablation experiments and feature visualization revealed that the proposed MST-Net could well characterize different cognitive loads, which not only provided novel and powerful tools for cognitive load assessment but also showed potential for broad application in brain-computer interface (BCI) systems.
Collapse
Affiliation(s)
- Zhongrui Li
- Henan Key Laboratory of Imaging and Intelligent Processing, PLA Strategic Support Force Information Engineering University, Zhengzhou, China
| | - Rongkai Zhang
- Henan Key Laboratory of Imaging and Intelligent Processing, PLA Strategic Support Force Information Engineering University, Zhengzhou, China
| | - Ying Zeng
- Henan Key Laboratory of Imaging and Intelligent Processing, PLA Strategic Support Force Information Engineering University, Zhengzhou, China
| | - Li Tong
- Henan Key Laboratory of Imaging and Intelligent Processing, PLA Strategic Support Force Information Engineering University, Zhengzhou, China
| | - Runnan Lu
- Henan Key Laboratory of Imaging and Intelligent Processing, PLA Strategic Support Force Information Engineering University, Zhengzhou, China
| | - Bin Yan
- Henan Key Laboratory of Imaging and Intelligent Processing, PLA Strategic Support Force Information Engineering University, Zhengzhou, China.
| |
Collapse
|
17
|
Sharma N, Upadhyay A, Sharma M, Singhal A. Deep temporal networks for EEG-based motor imagery recognition. Sci Rep 2023; 13:18813. [PMID: 37914729 PMCID: PMC10620382 DOI: 10.1038/s41598-023-41653-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2022] [Accepted: 08/29/2023] [Indexed: 11/03/2023] Open
Abstract
The electroencephalogram (EEG) based motor imagery (MI) signal classification, also known as motion recognition, is a highly popular area of research due to its applications in robotics, gaming, and medical fields. However, the problem is ill-posed as these signals are non-stationary and noisy. Recently, a lot of efforts have been made to improve MI signal classification using a combination of signal decomposition and machine learning techniques but they fail to perform adequately on large multi-class datasets. Previously, researchers have implemented long short-term memory (LSTM), which is capable of learning the time-series information, on the MI-EEG dataset for motion recognition. However, it can not model very long-term dependencies present in the motion recognition data. With the advent of transformer networks in natural language processing (NLP), the long-term dependency issue has been widely addressed. Motivated by the success of transformer algorithms, in this article, we propose a transformer-based deep learning neural network architecture that performs motion recognition on the raw BCI competition III IVa and IV 2a datasets. The validation results show that the proposed method achieves superior performance than the existing state-of-the-art methods. The proposed method produces classification accuracy of 99.7% and 84% on the binary class and the multi-class datasets, respectively. Further, the performance of the proposed transformer-based model is also compared with LSTM.
Collapse
Affiliation(s)
- Neha Sharma
- Department of Electronics and Communication Engineering, Bennett University, Greater Noida, 201310, India
| | - Avinash Upadhyay
- Department of Electronics and Communication Engineering, Bennett University, Greater Noida, 201310, India
| | - Manoj Sharma
- Department of Electronics and Communication Engineering, Bennett University, Greater Noida, 201310, India
| | - Amit Singhal
- Department of Electronics and Communication Engineering, NSUT, New Delhi, 110078, India.
| |
Collapse
|
18
|
Deny P, Cheon S, Son H, Choi KW. Hierarchical Transformer for Motor Imagery-Based Brain Computer Interface. IEEE J Biomed Health Inform 2023; 27:5459-5470. [PMID: 37578918 DOI: 10.1109/jbhi.2023.3304646] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/16/2023]
Abstract
In this paper, we propose a novel transformer-based classification algorithm for the brain computer interface (BCI) using a motor imagery (MI) electroencephalogram (EEG) signal. To design the MI classification algorithm, we apply an up-to-date deep learning model, the transformer, that has revolutionized the natural language processing (NLP) and successfully widened its application to many other domains such as the computer vision. Within a long MI trial spanning a few seconds, the classification algorithm should give more attention to the time periods during which the intended motor task is imagined by the subject without any artifact. To achieve this goal, we propose a hierarchical transformer architecture that consists of a high-level transformer (HLT) and a low-level transformer (LLT). We break down a long MI trial into a number of short-term intervals. The LLT extracts a feature from each short-term interval, and the HLT pays more attention to the features from more relevant short-term intervals by using the self-attention mechanism of the transformer. We have done extensive tests of the proposed scheme on four open MI datasets, and shown that the proposed hierarchical transformer excels in both the subject-dependent and subject-independent tests.
Collapse
|
19
|
He Y, Wang X, Yang Z, Xue L, Chen Y, Ji J, Wan F, Mukhopadhyay SC, Men L, Tong MCF, Li G, Chen S. Classification of attention deficit/hyperactivity disorder based on EEG signals using a EEG-Transformer model ∗. J Neural Eng 2023; 20:056013. [PMID: 37683665 DOI: 10.1088/1741-2552/acf7f5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 09/08/2023] [Indexed: 09/10/2023]
Abstract
Objective. Attention-deficit/hyperactivity disorder (ADHD) is the most common neurodevelopmental disorder in adolescents that can seriously impair a person's attention function, cognitive processes, and learning ability. Currently, clinicians primarily diagnose patients based on the subjective assessments of the Diagnostic and Statistical Manual of Mental Disorders-5, which can lead to delayed diagnosis of ADHD and even misdiagnosis due to low diagnostic efficiency and lack of well-trained diagnostic experts. Deep learning of electroencephalogram (EEG) signals recorded from ADHD patients could provide an objective and accurate method to assist physicians in clinical diagnosis.Approach. This paper proposes the EEG-Transformer deep learning model, which is based on the attention mechanism in the traditional Transformer model, and can perform feature extraction and signal classification processing for the characteristics of EEG signals. A comprehensive comparison was made between the proposed transformer model and three existing convolutional neural network models.Main results. The results showed that the proposed EEG-Transformer model achieved an average accuracy of 95.85% and an average AUC value of 0.9926 with the fastest convergence speed, outperforming the other three models. The function and relationship of each module of the model are studied by ablation experiments. The model with optimal performance was identified by the optimization experiment.Significance. The EEG-Transformer model proposed in this paper can be used as an auxiliary tool for clinical diagnosis of ADHD, and at the same time provides a basic model for transferable learning in the field of EEG signal classification.
Collapse
Affiliation(s)
- Yuchao He
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong 518055, People's Republic of China
- Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen, Guangdong 518055, People's Republic of China
| | - Xin Wang
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong 518055, People's Republic of China
- Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen, Guangdong 518055, People's Republic of China
| | - Zijian Yang
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong 518055, People's Republic of China
- Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen, Guangdong 518055, People's Republic of China
| | - Lingbin Xue
- Department of Otorhinolaryngology, Head and Neck Surgery, The Chinese University of Hong Kong, Hong Kong Special Administrative Region of China 000000, People's Republic of China
| | - Yuming Chen
- School of Psychology, Shenzhen University, Shenzhen 518060, People's Republic of China
| | - Junyu Ji
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong 518055, People's Republic of China
- Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences, Shenzhen, Guangdong 518055, People's Republic of China
- Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen, Guangdong 518055, People's Republic of China
| | - Feng Wan
- Faculty of Science and Technology, University of Macau, Macau 999078, People's Republic of China
| | | | - Lina Men
- Department of Neonatology, Shenzhen Children's Hospital, Shenzhen 518034, People's Republic of China
| | - Michael Chi Fai Tong
- Department of Otorhinolaryngology, Head and Neck Surgery, The Chinese University of Hong Kong, Hong Kong Special Administrative Region of China 000000, People's Republic of China
| | - Guanglin Li
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong 518055, People's Republic of China
- Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen, Guangdong 518055, People's Republic of China
| | - Shixiong Chen
- CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong 518055, People's Republic of China
- Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen, Guangdong 518055, People's Republic of China
| |
Collapse
|
20
|
Gour N, Hassan T, Owais M, Ganapathi II, Khanna P, Seghier ML, Werghi N. Transformers for autonomous recognition of psychiatric dysfunction via raw and imbalanced EEG signals. Brain Inform 2023; 10:25. [PMID: 37689601 PMCID: PMC10492733 DOI: 10.1186/s40708-023-00201-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Accepted: 07/17/2023] [Indexed: 09/11/2023] Open
Abstract
Early identification of mental disorders, based on subjective interviews, is extremely challenging in the clinical setting. There is a growing interest in developing automated screening tools for potential mental health problems based on biological markers. Here, we demonstrate the feasibility of an AI-powered diagnosis of different mental disorders using EEG data. Specifically, this work aims to classify different mental disorders in the following ecological context accurately: (1) using raw EEG data, (2) collected during rest, (3) during both eye open, and eye closed conditions, (4) at short 2-min duration, (5) on participants with different psychiatric conditions, (6) with some overlapping symptoms, and (7) with strongly imbalanced classes. To tackle this challenge, we designed and optimized a transformer-based architecture, where class imbalance is addressed through focal loss and class weight balancing. Using the recently released TDBRAIN dataset (n= 1274 participants), our method classifies each participant as either a neurotypical or suffering from major depressive disorder (MDD), attention deficit hyperactivity disorder (ADHD), subjective memory complaints (SMC), or obsessive-compulsive disorder (OCD). We evaluate the performance of the proposed architecture on both the window-level and the patient-level. The classification of the 2-min raw EEG data into five classes achieved a window-level accuracy of 63.2% and 65.8% for open and closed eye conditions, respectively. When the classification is limited to three main classes (MDD, ADHD, SMC), window level accuracy improved to 75.1% and 69.9% for eye open and eye closed conditions, respectively. Our work paves the way for developing novel AI-based methods for accurately diagnosing mental disorders using raw resting-state EEG data.
Collapse
Affiliation(s)
- Neha Gour
- Khalifa University Center for Autonomous Robotic System and Cyber-Physical Security System Center, Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, United Arab Emirates.
| | - Taimur Hassan
- Khalifa University Center for Autonomous Robotic System and Cyber-Physical Security System Center, Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, United Arab Emirates
- Departement of Electrical and Computer Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates
| | - Muhammad Owais
- Khalifa University Center for Autonomous Robotic System and Cyber-Physical Security System Center, Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, United Arab Emirates
| | - Iyyakutti Iyappan Ganapathi
- Khalifa University Center for Autonomous Robotic System and Cyber-Physical Security System Center, Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, United Arab Emirates
| | - Pritee Khanna
- Department of Computer Science and Engineering, Indian Institute of Information Technology, Design and Manufacturing, Jabalpur, India
| | - Mohamed L Seghier
- Healthcare Engineering Innovation Center, Department of Biomedical Engineering, Khalifa University, Abu Dhabi, United Arab Emirates
| | - Naoufel Werghi
- Khalifa University Center for Autonomous Robotic System and Cyber-Physical Security System Center, Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, United Arab Emirates
| |
Collapse
|
21
|
Shin J, Chung W. Multi-Band CNN With Band-Dependent Kernels and Amalgamated Cross Entropy Loss for Motor Imagery Classification. IEEE J Biomed Health Inform 2023; 27:4466-4477. [PMID: 37410639 DOI: 10.1109/jbhi.2023.3292909] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/08/2023]
Abstract
In this paper, we present a novel MI classification method based on multi-band convolutional neural network (CNN) with band-dependent kernel sizes, named MBK-CNN, to improve classification performance, by resolving the subject dependency issue of the widely used CNN-based approaches due to the kernel size optimization problem. The proposed structure exploits the frequency diversity of the EEG signals and simultaneously resolves the subject dependent kernel size issue. EEG signal is decomposed into overlapping multi-band and passed through multiple CNNs (termed 'branch-CNNs') with different kernel sizes to generate frequency dependent features, which are combined by a simple weighted sum. In contrast to the existing works where single-band multi-branch CNNs with different kernel sizes are used to resolve the subject dependency issue, a unique kernel size per frequency band is used. To prevent possible overfitting induced by a weighted sum, each branch-CNN is additionally trained by tentative cross entropy loss while overall network is optimized by the end-to-end cross entropy loss, which is named amalgamated cross entropy loss. In addition, we further propose multi-band CNN with enhanced spatial diversity, named MBK-LR-CNN, by replacing each branch-CNN with several sub branch-CNNs applied for channel subsets (termed 'local region') to improve the classification performance. We evaluated the performance of the proposed methods, MBK-CNN and MBK-LR-CNN, on publicly available datasets, BCI Competition IV dataset 2a and High Gamma Dataset. The experimental results confirm the performance improvement of the proposed methods compared to the currently existing MI classification methods.
Collapse
|
22
|
Dang R, Yu T, Hu B, Wang Y, Pan Z, Luo R, Wang Q. Temporal transformer-spatial graph convolutional network: an intelligent classification model for anti N-methyl-D-aspartate receptor encephalitis based on electroencephalogram signal. Front Neurosci 2023; 17:1223077. [PMID: 37700752 PMCID: PMC10493270 DOI: 10.3389/fnins.2023.1223077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Accepted: 08/15/2023] [Indexed: 09/14/2023] Open
Abstract
Encephalitis is a disease typically caused by viral infections or autoimmunity. The most common type of autoimmune encephalitis is anti-N-methyl-D-aspartate receptor (NMDAR) antibody-mediated, known as anti-NMDA receptor encephalitis, which is a rare disease. Specific EEG patterns, including "extreme delta brush" (EDB), have been reported in patients with anti-NMDA receptor encephalitis. The aim of this study was to develop an intelligent diagnostic model for encephalitis based on EEG signals. A total of 131 Participants were selected based on reasonable inclusion criteria and divided into three groups: health control (35 participants), viral encephalitis (58 participants), and anti NMDAR receptor encephalitis (55 participants). Due to the low prevalence of anti-NMDAR receptor encephalitis, it took several years to collect participants' EEG signals while they were in an awake state. EEG signals were collected and analyzed following the international 10-20 system layout. We proposed a model called Temporal Transformer-Spatial Graph Convolutional Network (TT-SGCN), which consists of a Preprocess Module, a Temporal Transformer Module (TTM), and a Spatial Graph Convolutional Module (SGCM). The raw EEG signal was preprocessed according to traditional procedures, including filtering, averaging, and Independent Component Analysis (ICA) method. The EEG signal was then segmented and transformed using short-time Fourier transform (STFT) to produce concatenated power density (CPD) maps, which served as inputs for the proposed model. TTM extracted the time-frequency features of each channel, and SGCM fused these features using graph convolutional methods based on the location of electrodes. The model was evaluated in two experiments: classification of the three groups and pairwise classification among the three groups. The model was trained using two stages and achieved the performance, with an accuracy of 82.23%, recall of 80.75%, precision of 82.51%, and F1 score of 81.23% in the classification of the three groups. The proposed model has the potential to become an intelligent auxiliary diagnostic tool for encephalitis.
Collapse
Affiliation(s)
- Ruochen Dang
- Key Laboratory of Spectral Imaging Technology, Xi’an Institute of Optics and Precision Mechanics (XIOPM), Chinese Academy of Sciences, Xi’an, China
- School of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an, China
- University of Chinese Academy of Sciences, Beijing, China
- Key Laboratory of Biomedical Spectroscopy of Xi’an, Xi’an Institute of Optics and Precision Mechanics (XIOPM), Chinese Academy of Sciences, Xi’an, China
| | - Tao Yu
- Department of Pediatrics, West China Second University Hospital, Sichuan University, Chengdu, China
- Key Laboratory of Obstetric and Gynecologic and Pediatric Diseases and Birth Defects of Ministry of Education, Sichuan University, Chengdu, China
| | - Bingliang Hu
- Key Laboratory of Spectral Imaging Technology, Xi’an Institute of Optics and Precision Mechanics (XIOPM), Chinese Academy of Sciences, Xi’an, China
- Key Laboratory of Biomedical Spectroscopy of Xi’an, Xi’an Institute of Optics and Precision Mechanics (XIOPM), Chinese Academy of Sciences, Xi’an, China
| | - Yuqi Wang
- Key Laboratory of Spectral Imaging Technology, Xi’an Institute of Optics and Precision Mechanics (XIOPM), Chinese Academy of Sciences, Xi’an, China
- Key Laboratory of Biomedical Spectroscopy of Xi’an, Xi’an Institute of Optics and Precision Mechanics (XIOPM), Chinese Academy of Sciences, Xi’an, China
| | - Zhibin Pan
- School of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an, China
| | - Rong Luo
- Department of Pediatrics, West China Second University Hospital, Sichuan University, Chengdu, China
- Key Laboratory of Obstetric and Gynecologic and Pediatric Diseases and Birth Defects of Ministry of Education, Sichuan University, Chengdu, China
| | - Quan Wang
- Key Laboratory of Spectral Imaging Technology, Xi’an Institute of Optics and Precision Mechanics (XIOPM), Chinese Academy of Sciences, Xi’an, China
- Key Laboratory of Biomedical Spectroscopy of Xi’an, Xi’an Institute of Optics and Precision Mechanics (XIOPM), Chinese Academy of Sciences, Xi’an, China
| |
Collapse
|
23
|
Zhang J, Li K, Yang B, Han X. Local and global convolutional transformer-based motor imagery EEG classification. Front Neurosci 2023; 17:1219988. [PMID: 37662099 PMCID: PMC10469791 DOI: 10.3389/fnins.2023.1219988] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2023] [Accepted: 08/07/2023] [Indexed: 09/05/2023] Open
Abstract
Transformer, a deep learning model with the self-attention mechanism, combined with the convolution neural network (CNN) has been successfully applied for decoding electroencephalogram (EEG) signals in Motor Imagery (MI) Brain-Computer Interface (BCI). However, the extremely non-linear, nonstationary characteristics of the EEG signals limits the effectiveness and efficiency of the deep learning methods. In addition, the variety of subjects and the experimental sessions impact the model adaptability. In this study, we propose a local and global convolutional transformer-based approach for MI-EEG classification. The local transformer encoder is combined to dynamically extract temporal features and make up for the shortcomings of the CNN model. The spatial features from all channels and the difference in hemispheres are obtained to improve the robustness of the model. To acquire adequate temporal-spatial feature representations, we combine the global transformer encoder and Densely Connected Network to improve the information flow and reuse. To validate the performance of the proposed model, three scenarios including within-session, cross-session and two-session are designed. In the experiments, the proposed method achieves up to 1.46%, 7.49% and 7.46% accuracy improvement respectively in the three scenarios for the public Korean dataset compared with current state-of-the-art models. For the BCI competition IV 2a dataset, the proposed model also achieves a 2.12% and 2.21% improvement for the cross-session and two-session scenarios respectively. The results confirm that the proposed approach can effectively extract much richer set of MI features from the EEG signals and improve the performance in the BCI applications.
Collapse
Affiliation(s)
- Jiayang Zhang
- School of Electrical Engineering, University of Leeds, Leeds, United Kingdom
| | - Kang Li
- School of Electrical Engineering, University of Leeds, Leeds, United Kingdom
| | - Banghua Yang
- School of Mechatronic Engineering and Automation, Shanghai University, Shanghai, China
| | - Xiaofei Han
- School of Electrical Engineering, University of Leeds, Leeds, United Kingdom
| |
Collapse
|
24
|
Huang Y, Zheng J, Xu B, Li X, Liu Y, Wang Z, Feng H, Cao S. An improved model using convolutional sliding window-attention network for motor imagery EEG classification. Front Neurosci 2023; 17:1204385. [PMID: 37662108 PMCID: PMC10469504 DOI: 10.3389/fnins.2023.1204385] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2023] [Accepted: 07/26/2023] [Indexed: 09/05/2023] Open
Abstract
Introduction The classification model of motor imagery-based electroencephalogram (MI-EEG) is a new human-computer interface pattern and a new neural rehabilitation assessment method for diseases such as Parkinson's and stroke. However, existing MI-EEG models often suffer from insufficient richness of spatiotemporal feature extraction, learning ability, and dynamic selection ability. Methods To solve these problems, this work proposed a convolutional sliding window-attention network (CSANet) model composed of novel spatiotemporal convolution, sliding window, and two-stage attention blocks. Results The model outperformed existing state-of-the-art (SOTA) models in within- and between-individual classification tasks on commonly used MI-EEG datasets BCI-2a and Physionet MI-EEG, with classification accuracies improved by 4.22 and 2.02%, respectively. Discussion The experimental results also demonstrated that the proposed type token, sliding window, and local and global multi-head self-attention mechanisms can significantly improve the model's ability to construct, learn, and adaptively select multi-scale spatiotemporal features in MI-EEG signals, and accurately identify electroencephalogram signals in the unilateral motor area. This work provided a novel and accurate classification model for MI-EEG brain-computer interface tasks and proposed a feasible neural rehabilitation assessment scheme based on the model, which could promote the further development and application of MI-EEG methods in neural rehabilitation.
Collapse
Affiliation(s)
- Yuxuan Huang
- School of Computer Science and Technology, Donghua University, Shanghai, China
| | - Jianxu Zheng
- Department of Neurosurgery and State Key Laboratory of Trauma, Burn and Combined Injury, Southwest Hospital, Third Military Medical University (Army Medical University), Chongqing, China
| | - Binxing Xu
- School of Computer Science and Technology, Donghua University, Shanghai, China
| | - Xuhang Li
- School of Computer Science and Technology, Donghua University, Shanghai, China
| | - Yu Liu
- School of Computer Science and Technology, Donghua University, Shanghai, China
| | - Zijian Wang
- School of Computer Science and Technology, Donghua University, Shanghai, China
| | - Hua Feng
- Department of Neurosurgery and State Key Laboratory of Trauma, Burn and Combined Injury, Southwest Hospital, Third Military Medical University (Army Medical University), Chongqing, China
| | - Shiqi Cao
- Department of Orthopaedics of TCM Clinical Unit, The Sixth Medical Center, Chinese PLA General Hospital, Beijing, China
| |
Collapse
|
25
|
Miao Z, Zhao M, Zhang X, Ming D. LMDA-Net:A lightweight multi-dimensional attention network for general EEG-based brain-computer interfaces and interpretability. Neuroimage 2023; 276:120209. [PMID: 37269957 DOI: 10.1016/j.neuroimage.2023.120209] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2023] [Revised: 05/05/2023] [Accepted: 05/30/2023] [Indexed: 06/05/2023] Open
Abstract
Electroencephalography (EEG)-based brain-computer interfaces (BCIs) pose a challenge for decoding due to their low spatial resolution and signal-to-noise ratio. Typically, EEG-based recognition of activities and states involves the use of prior neuroscience knowledge to generate quantitative EEG features, which may limit BCI performance. Although neural network-based methods can effectively extract features, they often encounter issues such as poor generalization across datasets, high predicting volatility, and low model interpretability. To address these limitations, we propose a novel lightweight multi-dimensional attention network, called LMDA-Net. By incorporating two novel attention modules designed specifically for EEG signals, the channel attention module and the depth attention module, LMDA-Net is able to effectively integrate features from multiple dimensions, resulting in improved classification performance across various BCI tasks. LMDA-Net was evaluated on four high-impact public datasets, including motor imagery (MI) and P300-Speller, and was compared with other representative models. The experimental results demonstrate that LMDA-Net outperforms other representative methods in terms of classification accuracy and predicting volatility, achieving the highest accuracy in all datasets within 300 training epochs. Ablation experiments further confirm the effectiveness of the channel attention module and the depth attention module. To facilitate an in-depth understanding of the features extracted by LMDA-Net, we propose class-specific neural network feature interpretability algorithms that are suitable for evoked responses and endogenous activities. By mapping the output of the specific layer of LMDA-Net to the time or spatial domain through class activation maps, the resulting feature visualizations can provide interpretable analysis and establish connections with EEG time-spatial analysis in neuroscience. In summary, LMDA-Net shows great potential as a general decoding model for various EEG tasks.
Collapse
Affiliation(s)
- Zhengqing Miao
- State Key Laboratory of Precision Measuring Technology and Instruments, School of Precision Instrument and Opto-electronics Engineering, Tianjin University, Tianjin 300072, China.
| | - Meirong Zhao
- State Key Laboratory of Precision Measuring Technology and Instruments, School of Precision Instrument and Opto-electronics Engineering, Tianjin University, Tianjin 300072, China.
| | - Xin Zhang
- Laboratory of Neural Engineering and Rehabilitation, Department of Biomedical Engineering, School of Precision Instruments and Optoelectronics Engineering, Tianjin University, China; Tianjin International Joint Research Center for Neural Engineering, Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, China.
| | - Dong Ming
- Laboratory of Neural Engineering and Rehabilitation, Department of Biomedical Engineering, School of Precision Instruments and Optoelectronics Engineering, Tianjin University, China; Tianjin International Joint Research Center for Neural Engineering, Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, China.
| |
Collapse
|
26
|
Zhang D, Li H, Xie J. MI-CAT: A transformer-based domain adaptation network for motor imagery classification. Neural Netw 2023; 165:451-462. [PMID: 37336030 DOI: 10.1016/j.neunet.2023.06.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2022] [Revised: 04/03/2023] [Accepted: 06/02/2023] [Indexed: 06/21/2023]
Abstract
Due to its convenience and safety, electroencephalography (EEG) data is one of the most widely used signals in motor imagery (MI) brain-computer interfaces (BCIs). In recent years, methods based on deep learning have been widely applied to the field of BCIs, and some studies have gradually tried to apply Transformer to EEG signal decoding due to its superior global information focusing ability. However, EEG signals vary from subject to subject. Based on Transformer, how to effectively use data from other subjects (source domain) to improve the classification performance of a single subject (target domain) remains a challenge. To fill this gap, we propose a novel architecture called MI-CAT. The architecture innovatively utilizes Transformer's self-attention and cross-attention mechanisms to interact features to resolve differential distribution between different domains. Specifically, we adopt a patch embedding layer for the extracted source and target features to divide the features into multiple patches. Then, we comprehensively focus on the intra-domain and inter-domain features by stacked multiple Cross-Transformer Blocks (CTBs), which can adaptively conduct bidirectional knowledge transfer and information exchange between domains. Furthermore, we also utilize two non-shared domain-based attention blocks to efficiently capture domain-dependent information, optimizing the features extracted from the source and target domains to assist in feature alignment. To evaluate our method, we conduct extensive experiments on two real public EEG datasets, Dataset IIb and Dataset IIa, achieving competitive performance with an average classification accuracy of 85.26% and 76.81%, respectively. Experimental results demonstrate that our method is a powerful model for decoding EEG signals and facilitates the development of the Transformer for brain-computer interfaces (BCIs).
Collapse
Affiliation(s)
- Dongxue Zhang
- Jilin University, College of Computer Science and Technology, Changchun, Jilin Province, China; Key Laboratory of Symbol Computation and Knowledge Engineering, Jilin University, Changchun 130012, China.
| | - Huiying Li
- Jilin University, College of Computer Science and Technology, Changchun, Jilin Province, China; Key Laboratory of Symbol Computation and Knowledge Engineering, Jilin University, Changchun 130012, China.
| | - Jingmeng Xie
- Xi'an Jiaotong University, College of Electronic information, Xi'an, Shanxi Province, China.
| |
Collapse
|
27
|
Jing Y, Wang W, Wang J, Jiao Y, Xiang K, Lin T, Shi W, Hou ZG. Transformer Based Cross-Subject Mental Workload Classification Using FNIRS for Real-World Application. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-5. [PMID: 38082781 DOI: 10.1109/embc40787.2023.10341167] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Mental state monitoring is a hot topic especially in neurorehabilitation, skill training, etc, for which the functional near-infrared spectroscopy (fNIRS) has been suggested to be used, and fewer detection channels and cross-subject performance are usually required for real-world application. To this goal, we propose a transformer-based method for cross-subject mental workload classification using fewer channels of fNIRS. Firstly, the input fNIRS signals in a window are divided into patches in the temporal order and transformed into embeddings, to which a classification token and learnable position embeddings are added. Then, a transformer encoder is used to learn the long-range dependencies among the embeddings, of which the output classification token is sent to a multilayer perceptron (MLP) head. Mental workload classification results can be represented by the outputs of the MLP head. Finally, comparison experiments were conducted on the open-access fNIRS2MW dataset. The results show that, the proposed method can outperform previous methods in cross-subject classification accuracy, and relatively efficient computation can be obtained.
Collapse
|
28
|
Wang L, Li M, Zhang L. Recognize enhanced temporal-spatial-spectral features with a parallel multi-branch CNN and GRU. Med Biol Eng Comput 2023:10.1007/s11517-023-02857-4. [PMID: 37294411 DOI: 10.1007/s11517-023-02857-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2022] [Accepted: 05/17/2023] [Indexed: 06/10/2023]
Abstract
Deep learning has been applied to the recognition of motor imagery electroencephalograms (MI-EEG) in brain-computer interface, and the performance results depend on data representation as well as neural network structure. Especially, MI-EEG is so complex with the characteristics of non-stationarity, specific rhythms, and uneven distribution; however, its multidimensional feature information is difficult to be fused and enhanced simultaneously in the existing recognition methods. In this paper, a novel channel importance (NCI) based on time-frequency analysis is proposed to develop an image sequence generation method (NCI-ISG) for enhancing the integrity of data representation and highlighting the contribution inequalities of different channels as well. Each electrode of MI-EEG is converted to a time-frequency spectrum by utilizing short-time Fourier transform; the corresponding part to 8-30 Hz is combined with random forest algorithm for computing NCI; and it is further divided into three sub-images covered by α (8-13 Hz), β1 (13-21 Hz), and β2 (21-30 Hz) bands; their spectral powers are further weighted by NCI and interpolated to 2-dimensional electrode coordinates, producing three main sub-band image sequences. Then, a parallel multi-branch convolutional neural network and gate recurrent unit (PMBCG) is designed to successively extract and identify the spatial-spectral and temporal features from the image sequences. Two public four-class MI-EEG datasets are adopted; the proposed classification method respectively achieves the average accuracies of 98.26% and 80.62% by 10-fold cross-validation experiment; and its statistical performance is also evaluated by multi-indexes, such as Kappa value, confusion matrix, and ROC curve. Extensive experiment results show that NCI-ISG + PMBCG can yield great performance on MI-EEG classification compared to state-of-the-art methods. The proposed NCI-ISG can enhance the feature representation of time-frequency-space domains and match well with PMBCG, which improves the recognition accuracies of MI tasks and demonstrates the preferable reliability and distinguishable ability. This paper proposes a novel channel importance (NCI) based on time-frequency analysis to develop an image sequences generation method (NCI-ISG) for enhancing the integrity of data representation and highlighting the contribution inequalities of different channels as well. Then, a parallel multi-branch convolutional neural network and gate recurrent unit (PMBCG) is designed to successively extract and identify the spatial-spectral and temporal features from the image sequences.
Collapse
Affiliation(s)
- Linlin Wang
- Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China
| | - Mingai Li
- Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China.
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing, 100124, China.
- Engineering Research Center of Digital Community, Ministry of Education, Beijing, 100124, China.
| | - Liyuan Zhang
- Faculty of Environment and Life, Beijing University of Technology, Beijing, 100124, China
| |
Collapse
|
29
|
Chen J, Zhang Y, Pan Y, Xu P, Guan C. A transformer-based deep neural network model for SSVEP classification. Neural Netw 2023; 164:521-534. [PMID: 37209444 DOI: 10.1016/j.neunet.2023.04.045] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2022] [Revised: 03/24/2023] [Accepted: 04/26/2023] [Indexed: 05/22/2023]
Abstract
Steady-state visual evoked potential (SSVEP) is one of the most commonly used control signals in the brain-computer interface (BCI) systems. However, the conventional spatial filtering methods for SSVEP classification highly depend on the subject-specific calibration data. The need for the methods that can alleviate the demand for the calibration data becomes urgent. In recent years, developing the methods that can work in inter-subject scenario has become a promising new direction. As a popular deep learning model nowadays, Transformer has been used in EEG signal classification tasks owing to its excellent performance. Therefore, in this study, we proposed a deep learning model for SSVEP classification based on Transformer architecture in inter-subject scenario, termed as SSVEPformer, which was the first application of Transformer on the SSVEP classification. Inspired by previous studies, we adopted the complex spectrum features of SSVEP data as the model input, which could enable the model to simultaneously explore the spectral and spatial information for classification. Furthermore, to fully utilize the harmonic information, an extended SSVEPformer based on the filter bank technology (FB-SSVEPformer) was proposed to improve the classification performance. Experiments were conducted using two open datasets (Dataset 1: 10 subjects, 12 targets; Dataset 2: 35 subjects, 40 targets). The experimental results show that the proposed models could achieve better results in terms of classification accuracy and information transfer rate than other baseline methods. The proposed models validate the feasibility of deep learning models based on Transformer architecture for SSVEP data classification, and could serve as potential models to alleviate the calibration procedure in the practical application of SSVEP-based BCI systems.
Collapse
Affiliation(s)
- Jianbo Chen
- School of Computer Science and Technology, Laboratory for Brain Science and Medical Artificial Intelligence, Southwest University of Science and Technology, Mianyang, China
| | - Yangsong Zhang
- School of Computer Science and Technology, Laboratory for Brain Science and Medical Artificial Intelligence, Southwest University of Science and Technology, Mianyang, China; MOE Key Laboratory for NeuroInformation, Clinical Hospital of Chengdu Brain Science Institute, and Center for Information in BioMedicine, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China.
| | - Yudong Pan
- School of Computer Science and Technology, Laboratory for Brain Science and Medical Artificial Intelligence, Southwest University of Science and Technology, Mianyang, China
| | - Peng Xu
- MOE Key Laboratory for NeuroInformation, Clinical Hospital of Chengdu Brain Science Institute, and Center for Information in BioMedicine, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China.
| | - Cuntai Guan
- School of Computer Science and Engineering, Nanyang Technological University (NTU), Singapore
| |
Collapse
|
30
|
Velasco I, Sipols A, De Blas CS, Pastor L, Bayona S. Motor imagery EEG signal classification with a multivariate time series approach. Biomed Eng Online 2023; 22:29. [PMID: 36959601 PMCID: PMC10035287 DOI: 10.1186/s12938-023-01079-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2022] [Accepted: 02/10/2023] [Indexed: 03/25/2023] Open
Abstract
BACKGROUND Electroencephalogram (EEG) signals record electrical activity on the scalp. Measured signals, especially EEG motor imagery signals, are often inconsistent or distorted, which compromises their classification accuracy. Achieving a reliable classification of motor imagery EEG signals opens the door to possibilities such as the assessment of consciousness, brain computer interfaces or diagnostic tools. We seek a method that works with a reduced number of variables, in order to avoid overfitting and to improve interpretability. This work aims to enhance EEG signal classification accuracy by using methods based on time series analysis. Previous work on this line, usually took a univariate approach, thus losing the possibility to take advantage of the correlation information existing within the time series provided by the different electrodes. To overcome this problem, we propose a multivariate approach that can fully capture the relationships among the different time series included in the EEG data. To perform the multivariate time series analysis, we use a multi-resolution analysis approach based on the discrete wavelet transform, together with a stepwise discriminant that selects the most discriminant variables provided by the discrete wavelet transform analysis RESULTS: Applying this methodology to EEG data to differentiate between the motor imagery tasks of moving either hands or feet has yielded very good classification results, achieving in some cases up to 100% of accuracy for this 2-class pre-processed dataset. Besides, the fact that these results were achieved using a reduced number of variables (55 out of 22,176) can shed light on the relevance and impact of those variables. CONCLUSIONS This work has a potentially large impact, as it enables classification of EEG data based on multivariate time series analysis in an interpretable way with high accuracy. The method allows a model with a reduced number of features, facilitating its interpretability and improving overfitting. Future work will extend the application of this classification method to help in diagnosis procedures for detecting brain pathologies and for its use in brain computer interfaces. In addition, the results presented here suggest that this method could be applied to other fields for the successful analysis of multivariate temporal data.
Collapse
Affiliation(s)
- I Velasco
- Department of Computer Science and Statistics, Rey Juan Carlos University, Madrid, Spain.
| | - A Sipols
- Department of Applied Mathematics, Science and Engineering of Materials and Electronic Technology, Rey Juan Carlos University, Madrid, Spain
| | - C Simon De Blas
- Department of Computer Science and Statistics, Rey Juan Carlos University, Madrid, Spain
| | - L Pastor
- Department of Computer Science and Statistics, Rey Juan Carlos University, Madrid, Spain
- Center for Computational Simulation, Universidad Politecnica de Madrid, Madrid, Spain
| | - S Bayona
- Department of Computer Science and Statistics, Rey Juan Carlos University, Madrid, Spain
- Center for Computational Simulation, Universidad Politecnica de Madrid, Madrid, Spain
| |
Collapse
|
31
|
Sibilano E, Brunetti A, Buongiorno D, Lassi M, Grippo A, Bessi V, Micera S, Mazzoni A, Bevilacqua V. An attention-based deep learning approach for the classification of subjective cognitive decline and mild cognitive impairment using resting-state EEG. J Neural Eng 2023; 20. [PMID: 36745929 DOI: 10.1088/1741-2552/acb96e] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Accepted: 02/06/2023] [Indexed: 02/08/2023]
Abstract
Objective. This study aims to design and implement the first deep learning (DL) model to classify subjects in the prodromic states of Alzheimer's disease (AD) based on resting-state electroencephalographic (EEG) signals.Approach. EEG recordings of 17 healthy controls (HCs), 56 subjective cognitive decline (SCD) and 45 mild cognitive impairment (MCI) subjects were acquired at resting state. After preprocessing, we selected sections corresponding to eyes-closed condition. Five different datasets were created by extracting delta, theta, alpha, beta and delta-to-theta frequency bands using bandpass filters. To classify SCDvsMCI and HCvsSCDvsMCI, we propose a framework based on the transformer architecture, which uses multi-head attention to focus on the most relevant parts of the input signals. We trained and validated the model on each dataset with a leave-one-subject-out cross-validation approach, splitting the signals into 10 s epochs. Subjects were assigned to the same class as the majority of their epochs. Classification performances of the transformer were assessed for both epochs and subjects and compared with other DL models.Main results. Results showed that the delta dataset allowed our model to achieve the best performances for the discrimination of SCD and MCI, reaching an Area Under the ROC Curve (AUC) of 0.807, while the highest results for the HCvsSCDvsMCI classification were obtained on alpha and theta with a micro-AUC higher than 0.74.Significance. We demonstrated that DL approaches can support the adoption of non-invasive and economic techniques as EEG to stratify patients in the clinical population at risk for AD. This result was achieved since the attention mechanism was able to learn temporal dependencies of the signal, focusing on the most discriminative patterns, achieving state-of-the-art results by using a deep model of reduced complexity. Our results were consistent with clinical evidence that changes in brain activity are progressive when considering early stages of AD.
Collapse
Affiliation(s)
- Elena Sibilano
- Department of Electrical and Information Engineering, Polytechnic University of Bari, Via Orabona 4, 70125 Bari, Italy
| | - Antonio Brunetti
- Department of Electrical and Information Engineering, Polytechnic University of Bari, Via Orabona 4, 70125 Bari, Italy
| | - Domenico Buongiorno
- Department of Electrical and Information Engineering, Polytechnic University of Bari, Via Orabona 4, 70125 Bari, Italy
| | - Michael Lassi
- The BioRobotics Institute, Scuola Superiore Sant'Anna, 56025 Pisa, Italy
| | | | - Valentina Bessi
- Department of Neuroscience, Psychology, Drug Research and Child Health, University of Florence, Azienda Ospedaliera Careggi, Florence, Italy
| | - Silvestro Micera
- The BioRobotics Institute, Scuola Superiore Sant'Anna, 56025 Pisa, Italy.,Bertarelli Foundation Chair in Translational Neuroengineering, Center for Neuroprosthetics, Institute of Bioengineering, School of Engineering, Ecole Polytechnique Federale de Lausanne, Lausanne, Switzerland
| | - Alberto Mazzoni
- The BioRobotics Institute, Scuola Superiore Sant'Anna, 56025 Pisa, Italy
| | - Vitoantonio Bevilacqua
- Department of Electrical and Information Engineering, Polytechnic University of Bari, Via Orabona 4, 70125 Bari, Italy
| |
Collapse
|
32
|
Jusseaume K, Valova I. Brain Age Prediction/Classification through Recurrent Deep Learning with Electroencephalogram Recordings of Seizure Subjects. SENSORS (BASEL, SWITZERLAND) 2022; 22:8112. [PMID: 36365809 PMCID: PMC9655329 DOI: 10.3390/s22218112] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Revised: 10/19/2022] [Accepted: 10/20/2022] [Indexed: 06/16/2023]
Abstract
With modern population growth and an increase in the average lifespan, more patients are becoming afflicted with neurodegenerative diseases such as dementia and Alzheimer's. Patients with a history of epilepsy, drug abuse, and mental health disorders such as depression have a larger risk of developing Alzheimer's and other neurodegenerative diseases later in life. Utilizing recordings of patients' brain waves obtained from the Temple University abnormal electroencephalogram (EEG) corpus, deep leaning long short-term memory neural networks are utilized to classify and predict patients' brain ages. The proposed deep learning neural network model structure and brain wave-processing methodology leads to an accuracy of 90% in patients' brain age classification across six age groups, with a mean absolute error value of 7 years for the brain age regression analysis. The achieved results demonstrate that the use of raw patient-sourced brain wave information leads to higher performance metrics than methods utilizing other brain wave-preprocessing methods and outperforms other deep learning models such as convolutional neural networks.
Collapse
|