1
|
Ma X, Chen W, Pei Z, Zhang Y, Chen J. Attention-based convolutional neural network with multi-modal temporal information fusion for motor imagery EEG decoding. Comput Biol Med 2024; 175:108504. [PMID: 38701593 DOI: 10.1016/j.compbiomed.2024.108504] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2023] [Revised: 04/15/2024] [Accepted: 04/21/2024] [Indexed: 05/05/2024]
Abstract
Convolutional neural network (CNN) has been widely applied in motor imagery (MI)-based brain computer interface (BCI) to decode electroencephalography (EEG) signals. However, due to the limited perceptual field of convolutional kernel, CNN only extracts features from local region without considering long-term dependencies for EEG decoding. Apart from long-term dependencies, multi-modal temporal information is equally important for EEG decoding because it can offer a more comprehensive understanding of the temporal dynamics of neural processes. In this paper, we propose a novel deep learning network that combines CNN with self-attention mechanism to encapsulate multi-modal temporal information and global dependencies. The network first extracts multi-modal temporal information from two distinct perspectives: average and variance. A shared self-attention module is then designed to capture global dependencies along these two feature dimensions. We further design a convolutional encoder to explore the relationship between average-pooled and variance-pooled features and fuse them into more discriminative features. Moreover, a data augmentation method called signal segmentation and recombination is proposed to improve the generalization capability of the proposed network. The experimental results on the BCI Competition IV-2a (BCIC-IV-2a) and BCI Competition IV-2b (BCIC-IV-2b) datasets show that our proposed method outperforms the state-of-the-art methods and achieves 4-class average accuracy of 85.03% on the BCIC-IV-2a dataset. The proposed method implies the effectiveness of multi-modal temporal information fusion in attention-based deep learning networks and provides a new perspective for MI-EEG decoding. The code is available at https://github.com/Ma-Xinzhi/EEG-TransNet.
Collapse
Affiliation(s)
- Xinzhi Ma
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China; Hangzhou Innovation Institute, Beihang University, Hangzhou, China
| | - Weihai Chen
- School of Electrical Engineering and Automation, Anhui University, Hefei, China.
| | - Zhongcai Pei
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China; Hangzhou Innovation Institute, Beihang University, Hangzhou, China
| | - Yue Zhang
- Hangzhou Innovation Institute, Beihang University, Hangzhou, China
| | - Jianer Chen
- Department of Geriatric Rehabilitation, Third Affiliated Hospital, Zhejiang Chinese Medical University, Hangzhou, China
| |
Collapse
|
2
|
Pérez-Velasco S, Marcos-Martínez D, Santamaría-Vázquez E, Martínez-Cagigal V, Moreno-Calderón S, Hornero R. Unraveling motor imagery brain patterns using explainable artificial intelligence based on Shapley values. Comput Methods Programs Biomed 2024; 246:108048. [PMID: 38308997 DOI: 10.1016/j.cmpb.2024.108048] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/11/2023] [Revised: 01/22/2024] [Accepted: 01/23/2024] [Indexed: 02/05/2024]
Abstract
BACKGROUND AND OBJECTIVE Motor imagery (MI) based brain-computer interfaces (BCIs) are widely used in rehabilitation due to the close relationship that exists between MI and motor execution (ME). However, the underlying brain mechanisms of MI remain not well understood. Most MI-BCIs use the sensorimotor rhythms elicited in the primary motor cortex (M1) and somatosensory cortex (S1), which consist of an event-related desynchronization followed by an event-related synchronization. Consequently, this has resulted in systems that only record signals around M1 and S1. However, MI could involve a more complex network including sensory, association, and motor areas. In this study, we hypothesize that the superior accuracies achieved by new deep learning (DL) models applied to MI decoding rely on focusing on a broader MI activation of the brain. Parallel to the success of DL, the field of explainable artificial intelligence (XAI) has seen continuous development to provide explanations for DL networks success. The goal of this study is to use XAI in combination with DL to extract information about MI brain activation patterns from non-invasive electroencephalography (EEG) signals. METHODS We applied an adaptation of Shapley additive explanations (SHAP) to EEGSym, a state-of-the-art DL network with exceptional transfer learning capabilities for inter-subject MI classification. We obtained the SHAP values from two public databases comprising 171 users generating left and right hand MI instances with and without real-time feedback. RESULTS We found that EEGSym based most of its prediction on the signal of the frontal electrodes, i.e. F7 and F8, and on the first 1500 ms of the analyzed imagination period. We also found that MI involves a broad network not only based on M1 and S1, but also on the prefrontal cortex (PFC) and the posterior parietal cortex (PPC). We further applied this knowledge to select a 8-electrode configuration that reached inter-subject accuracies of 86.5% ± 10.6% on the Physionet dataset and 88.7% ± 7.0% on the Carnegie Mellon University's dataset. CONCLUSION Our results demonstrate the potential of combining DL and SHAP-based XAI to unravel the brain network involved in producing MI. Furthermore, SHAP values can optimize the requirements for out-of-laboratory BCI applications involving real users.
Collapse
Affiliation(s)
- Sergio Pérez-Velasco
- Biomedical Engineering Group, E.T.S Ingenieros de Telecomunicación, University of Valladolid, Paseo de Belén 15, Valladolid, 47011, Spain; Centro de Investigación Biomédica en Red en Bioingeniería, Biomateriales y Nanomedicina (CIBER-BBN), Spain.
| | - Diego Marcos-Martínez
- Biomedical Engineering Group, E.T.S Ingenieros de Telecomunicación, University of Valladolid, Paseo de Belén 15, Valladolid, 47011, Spain; Centro de Investigación Biomédica en Red en Bioingeniería, Biomateriales y Nanomedicina (CIBER-BBN), Spain
| | - Eduardo Santamaría-Vázquez
- Biomedical Engineering Group, E.T.S Ingenieros de Telecomunicación, University of Valladolid, Paseo de Belén 15, Valladolid, 47011, Spain; Centro de Investigación Biomédica en Red en Bioingeniería, Biomateriales y Nanomedicina (CIBER-BBN), Spain
| | - Víctor Martínez-Cagigal
- Biomedical Engineering Group, E.T.S Ingenieros de Telecomunicación, University of Valladolid, Paseo de Belén 15, Valladolid, 47011, Spain; Centro de Investigación Biomédica en Red en Bioingeniería, Biomateriales y Nanomedicina (CIBER-BBN), Spain
| | - Selene Moreno-Calderón
- Biomedical Engineering Group, E.T.S Ingenieros de Telecomunicación, University of Valladolid, Paseo de Belén 15, Valladolid, 47011, Spain
| | - Roberto Hornero
- Biomedical Engineering Group, E.T.S Ingenieros de Telecomunicación, University of Valladolid, Paseo de Belén 15, Valladolid, 47011, Spain; Centro de Investigación Biomédica en Red en Bioingeniería, Biomateriales y Nanomedicina (CIBER-BBN), Spain
| |
Collapse
|
3
|
Ramkumar E, Paulraj M. Optimized FFNN with multichannel CSP-ICA framework of EEG signal for BCI. Comput Methods Biomech Biomed Engin 2024:1-18. [PMID: 38404196 DOI: 10.1080/10255842.2024.2319701] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Accepted: 02/12/2024] [Indexed: 02/27/2024]
Abstract
The electroencephalogram (EEG) of the patient is used to identify their motor intention, which is then converted into a control signal through a brain-computer interface (BCI) based on motor imagery. Whenever gathering features from EEG signals, making a BCI is difficult in part because of the enormous dimensionality of the data. Three stages make up the suggested methodology: pre-processing, extraction of features, selection, and categorization. To remove unwanted artifacts, the EEG signals are filtered by a fifth-order Butterworth multichannel band-pass filter. This decreases execution time and memory use, both of which improve system performance. Then a novel multichannel optimized CSP-ICA feature extraction technique is used to separate and eliminate non-discriminative information from discriminative information in the EEG channels. Furthermore, CSP uses the concept of an Artificial Bee Colony (ABC) algorithm to automatically identify the simultaneous global ideal frequency band and time interval combination for the extraction and classification of common spatial pattern characteristics. Finally, a Tunable optimized feed-forward neural network (FFNN) classifier is utilized to extract and categorize the temporal and frequency domain features, which employs an FFNN classifier with Tunable-Q wavelet transform. The proposed framework, therefore optimizes signal processing, enabling enhanced EEG signal classification for BCI applications. The result shows that the models that use Tunable optimized FFNN produce higher classification accuracy of more than 20% when compared to the existing models.
Collapse
Affiliation(s)
- E Ramkumar
- Sri Ramakrishna Institute of Technology, Coimbatore, India
| | - M Paulraj
- Sri Ramakrishna Institute of Technology, Coimbatore, India
| |
Collapse
|
4
|
Li J, She Q, Meng M, Du S, Zhang Y. Three-stage transfer learning for motor imagery EEG recognition. Med Biol Eng Comput 2024:10.1007/s11517-024-03036-9. [PMID: 38342784 DOI: 10.1007/s11517-024-03036-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Accepted: 01/17/2024] [Indexed: 02/13/2024]
Abstract
Motor imagery (MI) paradigms have been widely used in neural rehabilitation and drowsiness state assessment. The progress in brain-computer interface (BCI) technology has emphasized the importance of accurately and efficiently detecting motor imagery intentions from electroencephalogram (EEG). Despite the recent breakthroughs made in developing EEG-based algorithms for decoding MI, the accuracy and efficiency of these models remain limited by technical challenges posed by cross-subject heterogeneity in EEG data processing and the scarcity of EEG data for training. Inspired by the optimal transport theory, this study aims to develop a novel three-stage transfer learning (TSTL) method, which uses the existing labeled data from a source domain to improve classification performance on an unlabeled target domain. Notably, the proposed method comprises three components, namely, the Riemannian tangent space mapping (RTSM), source domain transformer (SDT), and optimal subspace mapping (OSM). The RTSM maps a symmetric positive definite matrix from the Riemannian space to the tangent space to minimize the marginal probability distribution drift. The SDT transforms the source domain to a target domain by finding the optimal transport mapping matrix to reduce the joint probability distribution differences. The OSM finally maps the transformed source domain and original target domain to the same subspace to further mitigate the distribution discrepancy. The performance of the proposed method was validated on two public BCI datasets, and the average accuracy of the algorithm on two datasets was 72.24% and 69.29%. Our results demonstrated the improved performance of EEG-based MI detection in comparison with state-of-the-art algorithms.
Collapse
Affiliation(s)
- Junhao Li
- School of Automation, Hangzhou Dianzi University, Hangzhou, 310018, Zhejiang, China
| | - Qingshan She
- School of Automation, Hangzhou Dianzi University, Hangzhou, 310018, Zhejiang, China.
| | - Ming Meng
- School of Automation, Hangzhou Dianzi University, Hangzhou, 310018, Zhejiang, China
| | - Shengzhi Du
- Department of Electrical Engineering, Tshwane University of Technology, Pretoria, 0001, South Africa
| | - Yingchun Zhang
- Department of Biomedical Engineering, University of Houston, Houston, TX, 77204, USA
| |
Collapse
|
5
|
Zhang J, Liu D, Chen W, Pei Z, Wang J. Boosting lower-limb motor imagery performance through an ensemble method for gait rehabilitation. Comput Biol Med 2024; 169:107910. [PMID: 38183703 DOI: 10.1016/j.compbiomed.2023.107910] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2023] [Revised: 12/05/2023] [Accepted: 12/24/2023] [Indexed: 01/08/2024]
Abstract
Lower-limb exoskeletons have been used extensively in many rehabilitation applications to assist disabled people with their therapies. Brain-machine interfaces (BMIs) further provide effective and natural control schemes. However, the limited performance of brain signal decoding from lower-limb kinematics restricts the broad growth of both BMI and rehabilitation industry. To address these challenges, we propose an ensemble method for lower-limb motor imagery (MI) classification. The proposed model employs multiple techniques to boost performance, including deep and shallow parts. Traditional wavelet transformation followed by filter-bank common spatial pattern (CSP) employs neurophysiologically reasonable patterns, while multi-head self-attention (MSA) followed by temporal convolutional network (TCN) extracts deeper encoded generalized patterns. Experimental results in a customized lower-limb exoskeleton on 8 subjects in 3 consecutive sessions showed that the proposed method achieved 60.27% and 64.20% for three (MI of left leg, MI of right leg, and rest) and two classes (lower-limb MI vs. rest), respectively. Besides, the proposed model achieves improvements of up to 4% and 2% accuracy for the subject-specific and subject-independent modes compared to the current state-of-the-art (SOTA) techniques, respectively. Finally, feature analysis was conducted to show discriminative brain patterns in each MI task and sessions with different feedback modalities. The proposed models integrated in the brain-actuated lower-limb exoskeleton established a potential BMI for gait training and neuroprosthesis.
Collapse
Affiliation(s)
- Jing Zhang
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China; Hangzhou Innovation Institute, Beihang University, Hangzhou, Zhejiang 310052, China.
| | - Dong Liu
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China.
| | - Weihai Chen
- School of Electrical Engineering and Automation, Anhui University, Hefei, Anhui 230601, China.
| | - Zhongcai Pei
- Hangzhou Innovation Institute, Beihang University, Hangzhou, Zhejiang 310052, China.
| | - Jianhua Wang
- Hangzhou Innovation Institute, Beihang University, Hangzhou, Zhejiang 310052, China.
| |
Collapse
|
6
|
Wang W, Li B, Wang H, Wang X, Qin Y, Shi X, Liu S. EEG-FMCNN: A fusion multi-branch 1D convolutional neural network for EEG-based motor imagery classification. Med Biol Eng Comput 2024; 62:107-120. [PMID: 37728715 DOI: 10.1007/s11517-023-02931-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Accepted: 09/07/2023] [Indexed: 09/21/2023]
Abstract
Motor imagery (MI) electroencephalogram (EEG) signal is recognized as a promising paradigm for brain-computer interface (BCI) systems and has been extensively employed in various BCI applications, including assisting disabled individuals, controlling devices and environments, and enhancing human capabilities. The high-performance decoding capability of MI-EEG signals is a key issue that impacts the development of the industry. However, decoding MI-EEG signals is challenging due to the low signal-to-noise ratio and inter-subject variability. In response to the aforementioned core problems, this paper proposes a novel end-to-end network, a fusion multi-branch 1D convolutional neural network (EEG-FMCNN), to decode MI-EEG signals without pre-processing. The utilization of multi-branch 1D convolution not only exhibits a certain level of noise tolerance but also addresses the issue of inter-subject variability to some extent. This is attributed to the ability of multi-branch architectures to capture information from different frequency bands, enabling the establishment of optimal convolutional scales and depths. Furthermore, we incorporate 1D squeeze-and-excitation (SE) blocks and shortcut connections at appropriate locations to further enhance the generalization and robustness of the network. In the BCI Competition IV-2a dataset, our proposed model has obtained good experimental results, achieving accuracies of 78.82% and 68.41% for subject-dependent and subject-independent modes, respectively. In addition, extensive ablative experiments and fine-tuning experiments were conducted, resulting in a notable 7% improvement in the average performance of the network, which holds significant implications for the generalization and application of the network.
Collapse
Affiliation(s)
- Wenlong Wang
- The School of Electrical Engineering, Shanghai Dianji University, Shanghai, 201306, China
- Intelligent Decision and Control Technology Institute, Shanghai Dianji University, Shanghai, 201306, China
| | - Baojiang Li
- The School of Electrical Engineering, Shanghai Dianji University, Shanghai, 201306, China.
- Intelligent Decision and Control Technology Institute, Shanghai Dianji University, Shanghai, 201306, China.
| | - Haiyan Wang
- The School of Electrical Engineering, Shanghai Dianji University, Shanghai, 201306, China
- Intelligent Decision and Control Technology Institute, Shanghai Dianji University, Shanghai, 201306, China
| | - Xichao Wang
- The School of Electrical Engineering, Shanghai Dianji University, Shanghai, 201306, China
- Intelligent Decision and Control Technology Institute, Shanghai Dianji University, Shanghai, 201306, China
| | - Yuxin Qin
- The School of Electrical Engineering, Shanghai Dianji University, Shanghai, 201306, China
- Intelligent Decision and Control Technology Institute, Shanghai Dianji University, Shanghai, 201306, China
| | - Xingbin Shi
- The School of Electrical Engineering, Shanghai Dianji University, Shanghai, 201306, China
- Intelligent Decision and Control Technology Institute, Shanghai Dianji University, Shanghai, 201306, China
| | - Shuxin Liu
- The School of Electrical Engineering, Shanghai Dianji University, Shanghai, 201306, China
- The Key Laboratory of Cognitive Computing and Intelligent Information Processing of Fujian Education Institutions (Wuyi University), Fujian, 354300, China
| |
Collapse
|
7
|
Sun H, Jin J, Daly I, Huang Y, Zhao X, Wang X, Cichocki A. Feature learning framework based on EEG graph self-attention networks for motor imagery BCI systems. J Neurosci Methods 2023; 399:109969. [PMID: 37683772 DOI: 10.1016/j.jneumeth.2023.109969] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Revised: 08/18/2023] [Accepted: 09/03/2023] [Indexed: 09/10/2023]
Abstract
Learning distinguishable features from raw EEG signals is crucial for accurate classification of motor imagery (MI) tasks. To incorporate spatial relationships between EEG sources, we developed a feature set based on an EEG graph. In this graph, EEG channels represent the nodes, with power spectral density (PSD) features defining their properties, and the edges preserving the spatial information. We designed an EEG based graph self-attention network (EGSAN) to learn low-dimensional embedding vector for EEG graph, which can be used as distinguishable features for motor imagery task classification. We evaluated our EGSAN model on two publicly available MI EEG datasets, each containing different types of motor imagery tasks. Our experiments demonstrate that our proposed model effectively extracts distinguishable features from EEG graphs, achieving significantly higher classification accuracies than existing state-of-the-art methods.
Collapse
Affiliation(s)
- Hao Sun
- Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China
| | - Jing Jin
- Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China; Shenzhen Research Institute of East China University of Science and Technology, Shen Zhen 518063, China.
| | - Ian Daly
- Brain-Computer Interfacing and Neural Engineering Laboratory, School of Computer Science and Electronic Engineering, University of Essex, Colchester CO4 3SQ, United Kingdom
| | - Yitao Huang
- Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China
| | - Xueqing Zhao
- Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China
| | - Xingyu Wang
- Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China
| | - Andrzej Cichocki
- RIKEN Brain Science Institute, Wako 351-0198, Japan; Nicolaus Copernicus University (UMK), 87-100 Torun, Poland
| |
Collapse
|
8
|
Zhang D, Li H, Xie J, Li D. MI-DAGSC: A domain adaptation approach incorporating comprehensive information from MI-EEG signals. Neural Netw 2023; 167:183-198. [PMID: 37659115 DOI: 10.1016/j.neunet.2023.08.008] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2023] [Revised: 07/24/2023] [Accepted: 08/06/2023] [Indexed: 09/04/2023]
Abstract
Non-stationarity of EEG signals leads to high variability between subjects, making it challenging to directly use data from other subjects (source domain) for the classifier in the current subject (target domain). In this study, we propose MI-DAGSC to address domain adaptation challenges in EEG-based motor imagery (MI) decoding. By combining domain-level information, class-level information, and inter-sample structure information, our model effectively aligns the feature distributions of source and target domains. This work is an extension of our previous domain adaptation work MI-DABAN (Li et al., 2023). Based on MI-DABAN, MI-DAGSC designs Sample-Feature Blocks (SFBs) and Graph Convolution Blocks (GCBs) to focus on intra-sample and inter-sample information. The synergistic integration of SFBs and GCBs enable the model to capture comprehensive information and understand the relationship between samples, thus improving representation learning. Furthermore, we introduce a triplet loss to enhance the alignment and compactness of feature representations. Extensive experiments on real EEG datasets demonstrate the effectiveness of MI-DAGSC, confirming that our method makes a valuable contribution to the MI-EEG decoding. Moreover, it holds great potential for various applications in brain-computer interface systems and neuroscience research. And the code of the proposed architecture in this study is available under https://github.com/zhangdx21/MI-DAGSC.
Collapse
Affiliation(s)
- Dongxue Zhang
- Jilin University, College of Computer Science and Technology, Changchun, Jilin Province, China; Key Laboratory of Symbol Computation and Knowledge Engineering, Jilin University, Changchun 130012, China.
| | - Huiying Li
- Jilin University, College of Computer Science and Technology, Changchun, Jilin Province, China; Key Laboratory of Symbol Computation and Knowledge Engineering, Jilin University, Changchun 130012, China.
| | - Jingmeng Xie
- Xi'an Jiaotong University, College of Electronic information, Xi'an, Shanxi Province, China.
| | - Dajun Li
- Jilin Provincial People's Hospital, Changchun, Jilin Province, China
| |
Collapse
|
9
|
Luo J, Wang Y, Xia S, Lu N, Ren X, Shi Z, Hei X. A shallow mirror transformer for subject-independent motor imagery BCI. Comput Biol Med 2023; 164:107254. [PMID: 37499295 DOI: 10.1016/j.compbiomed.2023.107254] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/02/2023] [Revised: 06/28/2023] [Accepted: 07/07/2023] [Indexed: 07/29/2023]
Abstract
OBJECTIVE Motor imagery BCI plays an increasingly important role in motor disorders rehabilitation. However, the position and duration of the discriminative segment in an EEG trial vary from subject to subject and even trial to trial, and this leads to poor performance of subject-independent motor imagery classification. Thus, determining how to detect and utilize the discriminative signal segments is crucial for improving the performance of subject-independent motor imagery BCI. APPROACH In this paper, a shallow mirror transformer is proposed for subject-independent motor imagery EEG classification. Specifically, a multihead self-attention layer with a global receptive field is employed to detect and utilize the discriminative segment from the entire input EEG trial. Furthermore, the mirror EEG signal and the mirror network structure are constructed to improve the classification precision based on ensemble learning. Finally, the subject-independent setup was used to evaluate the shallow mirror transformer on motor imagery EEG signals from subjects existing in the training set and new subjects. MAIN RESULTS The experiments results on BCI Competition IV datasets 2a and 2b and the OpenBMI dataset demonstrated the promising effectiveness of the proposed shallow mirror transformer. The shallow mirror transformer obtained average accuracies of 74.48% and 76.1% for new subjects and existing subjects, respectively, which were highest among the compared state-of-the-art methods. In addition, visualization of the attention score showed the ability of discriminative EEG segment detection. This paper demonstrated that multihead self-attention is effective in capturing global EEG signal information in motor imagery classification. SIGNIFICANCE This study provides an effective model based on a multihead self-attention layer for subject-independent motor imagery-based BCIs. To the best of our knowledge, this is the shallowest transformer model available, in which a small number of parameters promotes the performance in motor imagery EEG classification for such a small sample problem.
Collapse
Affiliation(s)
- Jing Luo
- Shaanxi Key Laboratory for Network Computing and Security Technology and Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China.
| | - Yaojie Wang
- Shaanxi Key Laboratory for Network Computing and Security Technology and Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China
| | - Shuxiang Xia
- Shaanxi Key Laboratory for Network Computing and Security Technology and Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China
| | - Na Lu
- State Key Laboratory for Manufacturing Systems Engineering, Systems Engineering Institute, Xi'an Jiaotong University, Xi'an, Shaanxi, China
| | - Xiaoyong Ren
- Department of Otolaryngology Head and Neck Surgery & Center of Sleep Medicine, The Second Affiliated Hospital of Xi'an Jiaotong University, Xi'an, Shaanxi, China
| | - Zhenghao Shi
- Shaanxi Key Laboratory for Network Computing and Security Technology and Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China
| | - Xinhong Hei
- Shaanxi Key Laboratory for Network Computing and Security Technology and Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China
| |
Collapse
|
10
|
Zhong XC, Wang Q, Liu D, Liao JX, Yang R, Duan S, Ding G, Sun J. A deep domain adaptation framework with correlation alignment for EEG-based motor imagery classification. Comput Biol Med 2023; 163:107235. [PMID: 37442010 DOI: 10.1016/j.compbiomed.2023.107235] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2023] [Revised: 06/30/2023] [Accepted: 07/01/2023] [Indexed: 07/15/2023]
Abstract
It is impractical to collect sufficient and well-labeled EEG data in Brain-computer interface because of the time-consuming data acquisition and costly annotation. Conventional classification methods reusing EEG data from different subjects and time periods (across domains) significantly decrease the classification accuracy of motor imagery. In this paper, we propose a deep domain adaptation framework with correlation alignment (DDAF-CORAL) to solve the problem of distribution divergence for motor imagery classification across domains. Specifically, a two-stage framework is adopted to extract deep features for raw EEG data. The distribution divergence caused by subjected-related and time-related variations is further minimized by aligning the covariance of the source and target EEG feature distributions. Finally, the classification loss and adaptation loss are optimized simultaneously to achieve sufficient discriminative classification performance and low feature distribution divergence. Extensive experiments on three EEG datasets demonstrate that our proposed method can effectively reduce the distribution divergence between the source and target EEG data. The results show that our proposed method delivers outperformance (an average classification accuracy of 92.9% for within-session, an average kappa value of 0.761 for cross-session, and an average classification accuracy of 83.3% for cross-subject) in two-class classification tasks compared to other state-of-the-art methods.
Collapse
Affiliation(s)
- Xiao-Cong Zhong
- School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China
| | - Qisong Wang
- School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China.
| | - Dan Liu
- School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China
| | - Jing-Xiao Liao
- School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China
| | - Runze Yang
- School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China
| | - Sanhe Duan
- School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China
| | - Guohua Ding
- School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China
| | - Jinwei Sun
- School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, 150001, China
| |
Collapse
|
11
|
Zhang D, Li H, Xie J. MI-CAT: A transformer-based domain adaptation network for motor imagery classification. Neural Netw 2023; 165:451-462. [PMID: 37336030 DOI: 10.1016/j.neunet.2023.06.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2022] [Revised: 04/03/2023] [Accepted: 06/02/2023] [Indexed: 06/21/2023]
Abstract
Due to its convenience and safety, electroencephalography (EEG) data is one of the most widely used signals in motor imagery (MI) brain-computer interfaces (BCIs). In recent years, methods based on deep learning have been widely applied to the field of BCIs, and some studies have gradually tried to apply Transformer to EEG signal decoding due to its superior global information focusing ability. However, EEG signals vary from subject to subject. Based on Transformer, how to effectively use data from other subjects (source domain) to improve the classification performance of a single subject (target domain) remains a challenge. To fill this gap, we propose a novel architecture called MI-CAT. The architecture innovatively utilizes Transformer's self-attention and cross-attention mechanisms to interact features to resolve differential distribution between different domains. Specifically, we adopt a patch embedding layer for the extracted source and target features to divide the features into multiple patches. Then, we comprehensively focus on the intra-domain and inter-domain features by stacked multiple Cross-Transformer Blocks (CTBs), which can adaptively conduct bidirectional knowledge transfer and information exchange between domains. Furthermore, we also utilize two non-shared domain-based attention blocks to efficiently capture domain-dependent information, optimizing the features extracted from the source and target domains to assist in feature alignment. To evaluate our method, we conduct extensive experiments on two real public EEG datasets, Dataset IIb and Dataset IIa, achieving competitive performance with an average classification accuracy of 85.26% and 76.81%, respectively. Experimental results demonstrate that our method is a powerful model for decoding EEG signals and facilitates the development of the Transformer for brain-computer interfaces (BCIs).
Collapse
Affiliation(s)
- Dongxue Zhang
- Jilin University, College of Computer Science and Technology, Changchun, Jilin Province, China; Key Laboratory of Symbol Computation and Knowledge Engineering, Jilin University, Changchun 130012, China.
| | - Huiying Li
- Jilin University, College of Computer Science and Technology, Changchun, Jilin Province, China; Key Laboratory of Symbol Computation and Knowledge Engineering, Jilin University, Changchun 130012, China.
| | - Jingmeng Xie
- Xi'an Jiaotong University, College of Electronic information, Xi'an, Shanxi Province, China.
| |
Collapse
|
12
|
Luo J, Li J, Mao Q, Shi Z, Liu H, Ren X, Hei X. Overlapping filter bank convolutional neural network for multisubject multicategory motor imagery brain-computer interface. BioData Min 2023; 16:19. [PMID: 37434221 DOI: 10.1186/s13040-023-00336-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2022] [Accepted: 07/03/2023] [Indexed: 07/13/2023] Open
Abstract
BACKGROUND Motor imagery brain-computer interfaces (BCIs) is a classic and potential BCI technology achieving brain computer integration. In motor imagery BCI, the operational frequency band of the EEG greatly affects the performance of motor imagery EEG recognition model. However, as most algorithms used a broad frequency band, the discrimination from multiple sub-bands were not fully utilized. Thus, using convolutional neural network (CNNs) to extract discriminative features from EEG signals of different frequency components is a promising method in multisubject EEG recognition. METHODS This paper presents a novel overlapping filter bank CNN to incorporate discriminative information from multiple frequency components in multisubject motor imagery recognition. Specifically, two overlapping filter banks with fixed low-cut frequency or sliding low-cut frequency are employed to obtain multiple frequency component representations of EEG signals. Then, multiple CNN models are trained separately. Finally, the output probabilities of multiple CNN models are integrated to determine the predicted EEG label. RESULTS Experiments were conducted based on four popular CNN backbone models and three public datasets. And the results showed that the overlapping filter bank CNN was efficient and universal in improving multisubject motor imagery BCI performance. Specifically, compared with the original backbone model, the proposed method can improve the average accuracy by 3.69 percentage points, F1 score by 0.04, and AUC by 0.03. In addition, the proposed method performed best among the comparison with the state-of-the-art methods. CONCLUSION The proposed overlapping filter bank CNN framework with fixed low-cut frequency is an efficient and universal method to improve the performance of multisubject motor imagery BCI.
Collapse
Affiliation(s)
- Jing Luo
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, People's Republic of China.
- Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, Xi'an University of Technology, Xi'an, Shaanxi, People's Republic of China.
| | - Jundong Li
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, People's Republic of China
- Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, Xi'an University of Technology, Xi'an, Shaanxi, People's Republic of China
| | - Qi Mao
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, People's Republic of China
- Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, Xi'an University of Technology, Xi'an, Shaanxi, People's Republic of China
| | - Zhenghao Shi
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, People's Republic of China
- Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, Xi'an University of Technology, Xi'an, Shaanxi, People's Republic of China
| | - Haiqin Liu
- Department of Otolaryngology Head and Neck Surgery & Center of Sleep Medicine, The Second Affiliated Hospital of Xi'an Jiaotong University, Xi'an, Shaanxi, People's Republic of China
| | - Xiaoyong Ren
- Department of Otolaryngology Head and Neck Surgery & Center of Sleep Medicine, The Second Affiliated Hospital of Xi'an Jiaotong University, Xi'an, Shaanxi, People's Republic of China
| | - Xinhong Hei
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, People's Republic of China
- Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, Xi'an University of Technology, Xi'an, Shaanxi, People's Republic of China
| |
Collapse
|
13
|
Ma J, Yang B, Qiu W, Zhang J, Yan L, Wang W. Recognizable Rehabilitation Movements of Multiple Unilateral Upper Limb: an fMRI Study of Motor Execution and Motor Imagery. J Neurosci Methods 2023; 392:109861. [PMID: 37075914 DOI: 10.1016/j.jneumeth.2023.109861] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2022] [Revised: 02/18/2023] [Accepted: 04/15/2023] [Indexed: 04/21/2023]
Abstract
BACKGROUND This paper presents a study investigating the recognizability of multiple unilateral upper limb movements in stroke rehabilitation. METHODS A functional magnetic experiment is employed to study motor execution (ME) and motor imagery (MI) of four movements for the unilateral upper limb: hand-grasping, hand-handling, arm-reaching, and wrist-twisting. The functional magnetic resonance imaging (fMRI) images of ME and MI tasks are statistically analyzed to delineate the region of interest (ROI). Then parameter estimation associated with ROIs for each ME and MI task are evaluated, where differences in ROIs for different movements are compared using analysis of covariance (ANCOVA). RESULTS All movements of ME and MI tasks activate motor areas of the brain, and there are significant differences (p<0.05) in ROIs evoked by different movements. The activation area is larger when executing the hand-grasping task instead of the others. CONCLUSION The four movements we propose can be adopted as MI tasks, especially for stroke rehabilitation, since they are highly recognizable and capable of activating more brain areas during MI and ME.
Collapse
Affiliation(s)
- Jun Ma
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain Computer Engineering, Shanghai University, Shanghai, 200441, China
| | - Banghua Yang
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain Computer Engineering, Shanghai University, Shanghai, 200441, China; Engineering Research Center of Traditional Chinese Medicine Intelligent Rehabilitation, Ministry of Education, 201203, Shanghai, China.
| | - Wenzheng Qiu
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain Computer Engineering, Shanghai University, Shanghai, 200441, China
| | - Jian Zhang
- Shanghai Universal Medical Imaging Diagnostic Center, Shanghai University, 200441, Shanghai China
| | - Linfeng Yan
- Department of Radiology & Functional and Molecular Imaging Key Lab of Shaanxi Province, Tangdu Hospital, Fourth Military Medical University, 710038, Shaanxi, China
| | - Wen Wang
- Department of Radiology & Functional and Molecular Imaging Key Lab of Shaanxi Province, Tangdu Hospital, Fourth Military Medical University, 710038, Shaanxi, China.
| |
Collapse
|
14
|
Li H, Zhang D, Xie J. MI-DABAN: A dual-attention-based adversarial network for motor imagery classification. Comput Biol Med 2023; 152:106420. [PMID: 36529022 DOI: 10.1016/j.compbiomed.2022.106420] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Revised: 11/11/2022] [Accepted: 12/11/2022] [Indexed: 12/15/2022]
Abstract
The brain-computer interface (BCI) based on motor imagery electroencephalography (EEG) is widely used because of its convenience and safety. However, due to the distributional disparity between EEG signals, data from other subjects cannot be used directly to train a subject-specific classifier. For efficient use of the labeled data, domain transfer learning and adversarial learning are gradually applied to BCI classification tasks. While these methods improve classification performance, they only align globally and ignore task-specific class boundaries, which may lead to the blurring of features near the classification boundaries. Simultaneously, they employ fully shared generators to extract features, resulting in the loss of domain-specific information and the destruction of performance. To address these issues, we propose a novel dual-attention-based adversarial network for motor imagery classification (MI-DABAN). Our framework leverages multiple subjects' knowledge to improve a single subject's motor imagery classification performance by cleverly using a novel adversarial learning method and two unshared attention blocks. Specifically, without introducing additional domain discriminators, we iteratively maximize and minimize the output difference between the two classifiers to implement adversarial learning to ensure accurate domain alignment. Among them, maximization is used to identify easily confused samples near the decision boundary, and minimization is used to align the source and target domain distributions. Moreover, for the shallow features from source and target domains, we use two non-shared attention blocks to preserve domain-specific information, which can prevent the negative transfer of domain information and further improve the classification performance on test data. We conduct extensive experiments on two publicly available EEG datasets, namely BCI Competition IV Datasets 2a and 2b. The experiment results demonstrate our method's effectiveness and superiority.
Collapse
Affiliation(s)
- Huiying Li
- Jilin University, College of Computer Science and Technology, Changchun, Jilin Province, China; Key Laboratory of Symbol Computation and Knowledge Engineering, Jilin University, Changchun 130012, China.
| | - Dongxue Zhang
- Jilin University, College of Computer Science and Technology, Changchun, Jilin Province, China; Key Laboratory of Symbol Computation and Knowledge Engineering, Jilin University, Changchun 130012, China.
| | - Jingmeng Xie
- Xi'an Jiaotong University, College of Electronic information, Xi'an, Shanxi Province, China.
| |
Collapse
|
15
|
Hosni SMI, Borgheai SB, McLinden J, Zhu S, Huang X, Ostadabbas S, Shahriari Y. A Graph-Based Nonlinear Dynamic Characterization of Motor Imagery Toward an Enhanced Hybrid BCI. Neuroinformatics 2022; 20:1169-1189. [PMID: 35907174 DOI: 10.1007/s12021-022-09595-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/05/2022] [Indexed: 12/31/2022]
Abstract
Decoding neural responses from multimodal information sources, including electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), has the transformative potential to advance hybrid brain-computer interfaces (hBCIs). However, existing modest performance improvement of hBCIs might be attributed to the lack of computational frameworks that exploit complementary synergistic properties in multimodal features. This study proposes a multimodal data fusion framework to represent and decode synergistic multimodal motor imagery (MI) neural responses. We hypothesize that exploiting EEG nonlinear dynamics adds a new informative dimension to the commonly combined EEG-fNIRS features and will ultimately increase the synergy between EEG and fNIRS features toward an enhanced hBCI. The EEG nonlinear dynamics were quantified by extracting graph-based recurrence quantification analysis (RQA) features to complement the commonly used spectral features for an enhanced multimodal configuration when combined with fNIRS. The high-dimensional multimodal features were further given to a feature selection algorithm relying on the least absolute shrinkage and selection operator (LASSO) for fused feature selection. Linear support vector machine (SVM) was then used to evaluate the framework. The mean hybrid classification performance improved by up to 15% and 4% compared to the unimodal EEG and fNIRS, respectively. The proposed graph-based framework substantially increased the contribution of EEG features for hBCI classification from 28.16% up to 52.9% when introduced the nonlinear dynamics and improved the performance by approximately 2%. These findings suggest that graph-based nonlinear dynamics can increase the synergy between EEG and fNIRS features for an enhanced MI response representation that is not dominated by a single modality.
Collapse
Affiliation(s)
- Sarah M I Hosni
- Department of Electrical, Computer & Biomedical Engineering, University of Rhode Island (URI), Kingston, RI, 02881, USA
| | - Seyyed B Borgheai
- Department of Electrical, Computer & Biomedical Engineering, University of Rhode Island (URI), Kingston, RI, 02881, USA
| | - John McLinden
- Department of Electrical, Computer & Biomedical Engineering, University of Rhode Island (URI), Kingston, RI, 02881, USA
| | - Shaotong Zhu
- Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, 02115, USA
| | - Xiaofei Huang
- Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, 02115, USA
| | - Sarah Ostadabbas
- Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, 02115, USA
| | - Yalda Shahriari
- Department of Electrical, Computer & Biomedical Engineering, University of Rhode Island (URI), Kingston, RI, 02881, USA.
| |
Collapse
|
16
|
Xu F, Xu X, Sun Y, Li J, Dong G, Wang Y, Li H, Wang L, Zhang Y, Pang S, Yin S. A framework for motor imagery with LSTM neural network. Comput Methods Programs Biomed 2022; 218:106692. [PMID: 35248817 DOI: 10.1016/j.cmpb.2022.106692] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/14/2020] [Revised: 10/23/2021] [Accepted: 02/07/2022] [Indexed: 06/14/2023]
Abstract
BACKGROUND AND OBJECTIVE How to learn robust representations from brain activities and to improve algorithm performance are the most significant issues for brain-computer interface systems. METHODS This study introduces a long short-term memory recurrent neural network to decode the multichannel electroencephalogram or electrocorticogram for implementing an effective motor imagery-based brain-computer interface system. The unique information processing mechanism of the long short-term memory network characterizes spatio-temporal dynamics in time sequences. This study evaluates the proposed method using publically available electroencephalogram/electrocorticogram datasets. RESULTS The decoded features coupled with a gradient boosting classifier could obtain high recognition accuracies of 99% for electroencephalogram and 100% for electrocorticogram, respectively. CONCLUSIONS The results demonstrated that the proposed model can estimate robust spatial-temporal features and obtain significant performance improvement for motor imagery-based brain-computer interface systems. Further, the proposed method is of low computational complexity.
Collapse
Affiliation(s)
- Fangzhou Xu
- International School for Optoelectronic Engineering, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China.
| | - Xiaoyan Xu
- Patent Examination Cooperation (Beijing) Center of the Patent Office, CNIPA, Beijing 100083, China
| | - Yanan Sun
- International School for Optoelectronic Engineering, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China; School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China
| | - Jincheng Li
- International School for Optoelectronic Engineering, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China; School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China
| | - Gege Dong
- International School for Optoelectronic Engineering, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China; School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China
| | - Yuandong Wang
- International School for Optoelectronic Engineering, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China; School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China
| | - Han Li
- International School for Optoelectronic Engineering, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China; School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China
| | - Lei Wang
- School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China
| | - Yingchun Zhang
- Engineering Training Center, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China
| | - Shaopeng Pang
- School of Electrical Engineering and Automation, Qilu University of Technology (Shandong Academy of Sciences), Jinan 250353, China.
| | - Sen Yin
- Department of Neurology, Qilu Hospital, Cheeloo College of Medicine, Shandong University, Jinan 250012, China.
| |
Collapse
|
17
|
Ouyang R, Jin Z, Tang S, Fan C, Wu X. Low-quality Training Data Detection Method of EEG Signals for Motor Imagery BCI System. J Neurosci Methods 2022; 376:109607. [PMID: 35483505 DOI: 10.1016/j.jneumeth.2022.109607] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2021] [Revised: 03/16/2022] [Accepted: 04/19/2022] [Indexed: 10/18/2022]
Abstract
BACKGROUND The design and implementation of high-performance motor imagery-based brain computer interface (MI-BCI) requires high-quality training samples. However, fluctuation in subjects' physiological and mental states as well as artifacts can produce the low-quality motor imagery electroencephalogram (EEG) signal, which will damage the performance of MI-BCI system. NEW METHOD In order to select high-quality MI-EEG training data, this paper proposes a low-quality training data detection method combining independent component analysis (ICA) and weak classifier cluster. we also design and implement a new online BCI system based on motor imagery to verify the online processing performance of the proposed method. RESULT In order to verify the effectiveness of the proposed method, we conducted offline experiments on the public dataset called BCI Competition IV Data Set 2b. Furthermore, in order to verify the processing performance of the online system, we designed 60 groups of online experiments on 12 subjects. The online experimental results show that the twelve subjects can complete the system task efficiently (the best experiment is 135.6 seconds with 9 trials of subject S1). CONCLUSION This paper demonstrated that the proposed low-quality training data detection method can effectively screen out low-quality training samples, so as to improve the performance of the MI-BCI system.
Collapse
Affiliation(s)
- Rui Ouyang
- Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei, 230601, China; Institute of Physical Science and Information Technology, Anhui University, Hefei, 230601, China; Zhejiang Key Laboratory for Brain-Machine Collaborative Intelligence, Hangzhou Dianzi University, Hangzhou, 310018, China
| | - Zihao Jin
- Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei, 230601, China
| | - Shuhao Tang
- Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei, 230601, China
| | - Cunhang Fan
- Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei, 230601, China.
| | - Xiaopei Wu
- Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei, 230601, China.
| |
Collapse
|
18
|
Shen L, Xia Y, Li Y, Sun M. A multiscale siamese convolutional neural network with cross-channel fusion for motor imagery decoding. J Neurosci Methods 2022; 367:109426. [PMID: 34902364 DOI: 10.1016/j.jneumeth.2021.109426] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2021] [Revised: 10/28/2021] [Accepted: 11/22/2021] [Indexed: 11/22/2022]
Abstract
BACKGROUND Recently, convolutional neural networks (CNN) are widely applied in motor imagery electroencephalography (MI-EEG) signal classification tasks. However, a simple CNN framework is challenging to satisfy the complex MI-EEG signal decoding. NEW METHOD In this study, we propose a multiscale Siamese convolutional neural network with cross-channel fusion (MSCCF-Net) for MI-EEG classification tasks. The proposed network consists of three parts: Siamese cross-channel fusion streams, similarity module and classification module. Each Siamese cross-channel fusion stream contains multiple branches, and each branch is supplemented by cross-channel fusion modules to improve multiscale temporal feature representation capability. The similarity module is adopted to measure the feature similarity between multiple branches. At the same time, the classification module provides a strong constraint to classify the features from all Siamese cross-channel fusion streams. The combination of the similarity module and classification module constitutes a new joint training strategy to further optimize the network performance. RESULTS The experiment is conducted on the public BCI Competition IV 2a and 2b datasets, and the results show that the proposed network achieves an average accuracy of 87.36% and 87.33%, respectively. COMPARISON WITH EXISTING METHODS AND CONCLUSIONS The proposed network adopts cross-channel fusion to learn multiscale temporal characteristics and joint training strategy to optimize the training process. Therefore, the performance outperforms other state-of-the-art MI-EEG signal classification methods.
Collapse
|
19
|
Thenmozhi T, Helen R. Feature Selection Using Extreme Gradient Boosting Bayesian Optimization to upgrade the Classification Performance of Motor Imagery signals for BCI. J Neurosci Methods 2022; 366:109425. [PMID: 34838951 DOI: 10.1016/j.jneumeth.2021.109425] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2021] [Revised: 11/17/2021] [Accepted: 11/20/2021] [Indexed: 11/16/2022]
Abstract
BACKGROUND A motor imagery (MI) based brain computer interface (BCI) is a challenging nonmuscular connection system used to independently perform movement-related tasks. It is gaining increasing prominence in helping paralyzed individuals communicate with the real world. Achieving better classification accuracy is the major concern in the field of motor imagery-based BCI. To upgrade the classification performance, relevant features play a vital role. The relevant features can be selected by the extreme gradient Bayesian optimization (XGBO) method. METHODS In this paper, a combination of time-, frequency-, and spatial-related MI features are extracted to design a reliable MI-BCI system. The proposed method incorporates the XGBO algorithm for feature selection and the random forest for the classification of EEG signals. The potency of the proposed system was investigated using two public EEG datasets (BCI Competition III dataset IIIa and dataset IVa). A novel XGBO algorithm increases the accuracy and reduces the time consumption by reducing the dimensionality of features. The proposed algorithm selects the minimum number of features that increase the computational efficacy for MI-based BCI systems. COMPARISON WITH EXISTING METHODS The proposed method is compared with ANOVA, sequential forward selection, recursive feature elimination, and LASSO methods and the accuracy rate is increased with the lowest computation time. RESULTS The proposed method achieves mean accuracies of 94.44% and 88.72% and classification errors of 5.56% and 11.28% for Datasets IIIa and IVa, respectively. It outperforms four state-of-art methods with 0.87% and 0.59% increases in the accuracy for Datasets IIIa and IVa, respectively.
Collapse
Affiliation(s)
- T Thenmozhi
- Department of Electrical and Electronics Engineering, Thiagarajar College of Engineering, Madurai 625015, India.
| | - R Helen
- Department of Electrical and Electronics Engineering, Thiagarajar College of Engineering, Madurai 625015, India.
| |
Collapse
|
20
|
Zheng M, Yang B. A deep neural network with subdomain adaptation for motor imagery brain-computer interface. Med Eng Phys 2021; 96:29-40. [PMID: 34565550 DOI: 10.1016/j.medengphy.2021.08.006] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2021] [Revised: 08/15/2021] [Accepted: 08/17/2021] [Indexed: 10/20/2022]
Abstract
BACKGROUND The nonstationarity problem of EEG is very serious, especially for spontaneous signals, which leads to the poor effect of machine learning related to spontaneous signals, especially in related tasks across time, which correspondingly limits the practical use of brain-computer interface (BCI). OBJECTIVE In this paper, we proposed a new transfer learning algorithm, which can utilize the labeled motor imagery (MI) EEG data at the previous time to achieve better classification accuracies for a small number of labeled EEG signals at the current time. METHODS We introduced an adaptive layer into the full connection layer of a deep convolution neural network. The objective function of the adaptive layer was designed to minimize the Local Maximum Mean Discrepancy (LMMD) and the prediction error while minimizing the distance within each class (DWC) and maximizing the distance between classes within each domain (DBCWD). We verified the effectiveness of the proposed algorithm on two public datasets. RESULTS The classification accuracy of the proposed algorithm was higher than other comparison algorithms, and the paired t-test results also showed that the performance of the proposed algorithm was significantly different from that of other algorithms. The results of the confusion matrix and feature visualization showed the effectiveness of the proposed algorithm. CONCLUSION Experimental results showed that the proposed algorithm can achieve higher classification accuracy than other algorithms when there was only a small amount of labeled MI EEG data at the current time. It can be promising to be applied to the field of BCI.
Collapse
Affiliation(s)
- Minmin Zheng
- School of Mechatronic Engineering and Automation, Research Center of Brain Computer Engineering, Shanghai University, Shanghai, China; School of Mechanical and Electrical Engineering, Putian University, Fujian, China
| | - Banghua Yang
- School of Mechatronic Engineering and Automation, Research Center of Brain Computer Engineering, Shanghai University, Shanghai, China.
| |
Collapse
|
21
|
Abstract
Background Brain wave signal recognition has gained increased attention in neuro-rehabilitation applications. This has driven the development of brain–computer interface (BCI) systems. Brain wave signals are acquired using electroencephalography (EEG) sensors, processed and decoded to identify the category to which the signal belongs. Once the signal category is determined, it can be used to control external devices. However, the success of such a system essentially relies on significant feature extraction and classification algorithms. One of the commonly used feature extraction technique for BCI systems is common spatial pattern (CSP). Results The performance of the proposed spatial-frequency-temporal feature extraction (SPECTRA) predictor is analysed using three public benchmark datasets. Our proposed predictor outperformed other competing methods achieving lowest average error rates of 8.55%, 17.90% and 20.26%, and highest average kappa coefficient values of 0.829, 0.643 and 0.595 for BCI Competition III dataset IVa, BCI Competition IV dataset I and BCI Competition IV dataset IIb, respectively.
Conclusions Our proposed SPECTRA predictor effectively finds features that are more separable and shows improvement in brain wave signal recognition that can be instrumental in developing improved real-time BCI systems that are computationally efficient.
Collapse
Affiliation(s)
- Shiu Kumar
- School of Electrical and Electronics Engineering, Fiji National University, Suva, Fiji.
| | - Tatsuhiko Tsunoda
- Laboratory for Medical Science Mathematics, RIKEN Center for Integrative Medical Sciences, Yokohama, 230-0045, Japan.,Department of Medical Science Mathematics, Medical Research Institute, Tokyo Medical and Dental University (TMDU), Tokyo, 113-8510, Japan.,Laboratory for Medical Science Mathematics, Department of Biological Sciences, Graduate School of Science, University of Tokyo, Tokyo, 113-0033, Japan
| | - Alok Sharma
- Laboratory for Medical Science Mathematics, RIKEN Center for Integrative Medical Sciences, Yokohama, 230-0045, Japan.,Department of Medical Science Mathematics, Medical Research Institute, Tokyo Medical and Dental University (TMDU), Tokyo, 113-8510, Japan.,School of Engineering and Physics, The University of the South Pacific, Suva, Fiji.,Institute for Integrated and Intelligent Systems, Griffith University, Nathan, Brisbane, QLD, Australia
| |
Collapse
|
22
|
Li H, Fan K, Ma J, Wang B, Qiao X, Yan Y, Du W, Wang L. Massage Therapy's Effectiveness on the Decoding EEG Rhythms of Left/Right Motor Imagery and Motion Execution in Patients With Skeletal Muscle Pain. IEEE J Transl Eng Health Med 2021; 9:2100320. [PMID: 33738147 PMCID: PMC7965939 DOI: 10.1109/jtehm.2021.3056911] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/15/2020] [Revised: 12/18/2020] [Accepted: 01/27/2021] [Indexed: 11/09/2022]
Abstract
Objective: Most of effectiveness assessments of the widely-used Massage therapy were based on subjective routine clinical assessment tools, such as Visual Analogue Scale (VAS) score. However, few studies demonstrated the impact of massage on the Electroencephalograph (EEG) rhythm decoding of Motor imagery (MI) and motion execution (ME) with trunk left/right bending in patients with skeletal muscle pain. Method: We used the sample entropy (SampEn), permutation entropy (PermuEn), common spatial pattern (CSP) features, support vector machine (SVM) and logic regression (LR) classifiers. We also used the convolutional neural network (CNN) and attention-based bi-directional long short-term memory (BiLSTM) for classification. Results: The averaged SampEn and PermuEn values of alpha rhythm decreased in almost fourteen channels for five statuses (quiet, MI with left/right bending, ME with left/right bending). It indicated that massage alleviates the pain for the patients of skeletal pain. Furthermore, compared with the SVM and LR classifiers, the BiLSTM method achieved a better area under curve (AUC) of 0.89 for the classification of MI with trunk left/right bending before massage. The AUC became smaller after massage than that before massage for the classification of MI with trunk left/right bending using CNN and BiLSTM methods. The Permutation direct indicator (PDI) score showed the significant difference for patients in different statuses (before vs after massage, and MI vs ME). Conclusions: Massage not only affects the quiet status, but also affects the MI and ME. Clinical Impact: Massage therapy may affect a bit on the accuracy of MI with trunk left/right bending and it change the topography of MI and ME with trunk left/right bending for the patients with skeletal muscle pain.
Collapse
Affiliation(s)
- Huihui Li
- Shenzhen Institute of Advanced TechnologyChinese Academy of SciencesShenzhen518055China
| | - Kai Fan
- North China Institute of Aerospace EngineeringLangfang065000China
| | - Junsong Ma
- School of Electronic Engineering and AutomationGuilin University of Electronic TechnologyGuilin541004China
| | - Bo Wang
- Electronic and Communication Engineering DepartmentWuhan University of TechnologyWuhan430070China
| | - Xiaohao Qiao
- Electronic and Communication Engineering DepartmentWuhan University of TechnologyWuhan430070China
| | - Yan Yan
- Shenzhen Institute of Advanced TechnologyChinese Academy of SciencesShenzhen518055China
| | - Wenjing Du
- Shenzhen Institute of Advanced TechnologyChinese Academy of SciencesShenzhen518055China
| | - Lei Wang
- Shenzhen Institute of Advanced TechnologyChinese Academy of SciencesShenzhen518055China
| |
Collapse
|
23
|
Luo J, Shi W, Lu N, Wang J, Chen H, Wang Y, Lu X, Wang X, Hei X. Improving the performance of multisubject motor imagery-based BCIs using twin cascaded softmax CNNs. J Neural Eng 2021; 18. [PMID: 33540387 DOI: 10.1088/1741-2552/abe357] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2020] [Accepted: 02/04/2021] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Motor imagery (MI) EEG signals vary greatly among subjects, so scholarly research on motor imagery-based brain-computer interfaces (BCIs) has mainly focused on single-subject systems or subject-dependent systems. However, the single-subject model is applicable only to the target subject, and the small sample number greatly limits the performance of the model. This paper aims to study a convolutional neural network to achieve an adaptable MI-BCI that is applicable to multiple subjects. APPROACH In this paper, a twin cascaded softmax convolutional neural network (TCSCNN) is proposed for multisubject MI-BCIs. The proposed TCSCNN is independent and can be applied to any single-subject MI classification CNN model. First, to reduce the influence of individual differences, subject recognition and MI recognition are accomplished simultaneously. A cascaded softmax structure consisting of two softmax layers, related to subject recognition and MI recognition, is subsequently applied. Second, to improve the MI classification precision, a twin network structure is proposed on the basis of ensemble learning. TCSCNN is built by combining a cascaded softmax structure and twin network structure. MAIN RESULTS Experiments were conducted on three popular CNN models (EEGNet and Shallow ConvNet and Deep ConvNet from EEGDecoding) and three public datasets (BCI Competition IV datasets 2a and 2b and the High-Gamma dataset) to verify the performance of the proposed TCSCNN. The results show that compared with the state-of-the-art CNN model, the proposed TCSCNN obviously improves the precision and convergence of multisubject MI recognition. SIGNIFICANCE This study provides a promising scheme for multisubject MI-BCI, reflecting the progress made in the development and application of MI-BCIs.
Collapse
Affiliation(s)
- Jing Luo
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering , Xi'an University of Technology, No. 5, Jinhua South Road, Xi'an, Shaanxi Province, Xi'an, Shaanxi, 710048, CHINA
| | - Weiwei Shi
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering , Xi'an University of Technology, No. 5, Jinhua South Road, Xi'an, Shaanxi Province, Xi'an, Shaanxi, 710048, CHINA
| | - Na Lu
- Systems Engineering Institute, Xi'an Jiaotong University, No.28, Xianning West Road, Xi'an, Shaanxi, Xi'an, 710049, CHINA
| | - Jie Wang
- State Key Laboratory for Manufacturing System Engineering, System Engineering Institute, Xi'an Jiaotong University, No.28, Xianning West Road, Xi'an, Shaanxi, Xi'an, Shaanxi, 710049, CHINA
| | - Hao Chen
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering, Xi'an University of Technology, No. 5, Jinhua South Road, Xi'an, Shaanxi Province, Xi'an, Shaanxi, 710048, CHINA
| | - Yaojie Wang
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering , Xi'an University of Technology, No. 5, Jinhua South Road, Xi'an, Shaanxi Province, Xi'an, Shaanxi, 710048, CHINA
| | - Xiaofeng Lu
- School of computer science, Xi'an University of Technology, No. 5, Jinhua South Road, Xi'an, Shaanxi Province, Xi'an, Shaanxi, 710048, CHINA
| | - Xiaofan Wang
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering , Xi'an University of Technology, No. 5, Jinhua South Road, Xi'an, Shaanxi Province, Xi'an, Shaanxi, 710048, CHINA
| | - Xinhong Hei
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering , Xi'an University of Technology, No. 5, Jinhua South Road, Xi'an, Shaanxi Province, Xi'an, Shaanxi, 710048, CHINA
| |
Collapse
|
24
|
Kumar S, Sharma R, Sharma A. OPTICAL+: a frequency-based deep learning scheme for recognizing brain wave signals. PeerJ Comput Sci 2021; 7:e375. [PMID: 33817023 PMCID: PMC7959638 DOI: 10.7717/peerj-cs.375] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2020] [Accepted: 01/06/2021] [Indexed: 06/12/2023]
Abstract
A human-computer interaction (HCI) system can be used to detect different categories of the brain wave signals that can be beneficial for neurorehabilitation, seizure detection and sleep stage classification. Research on developing HCI systems using brain wave signals has progressed a lot over the years. However, real-time implementation, computational complexity and accuracy are still a concern. In this work, we address the problem of selecting the appropriate filtering frequency band while also achieving a good system performance by proposing a frequency-based approach using long short-term memory network (LSTM) for recognizing different brain wave signals. Adaptive filtering using genetic algorithm is incorporated for a hybrid system utilizing common spatial pattern and LSTM network. The proposed method (OPTICAL+) achieved an overall average classification error rate of 30.41% and a kappa coefficient value of 0.398, outperforming the state-of-the-art methods. The proposed OPTICAL+ predictor can be used to develop improved HCI systems that will aid in neurorehabilitation and may also be beneficial for sleep stage classification and seizure detection.
Collapse
Affiliation(s)
- Shiu Kumar
- School of Electrical and Electronic Engineering, Fiji National University, Suva, Fiji
| | - Ronesh Sharma
- School of Electrical and Electronic Engineering, Fiji National University, Suva, Fiji
| | - Alok Sharma
- STEMP, University of the South Pacific, Suva, Fiji
- Institute for Integrated and Intelligent Systems, Griffith University, Brisbane, Australia
- Laboratory for Medical Science Mathematics, RIKEN Center for Integrative Medical Sciences, Yokohama, Kanagawa, Japan
| |
Collapse
|
25
|
Varsehi H, Firoozabadi SMP. An EEG channel selection method for motor imagery based brain-computer interface and neurofeedback using Granger causality. Neural Netw 2021; 133:193-206. [PMID: 33220643 DOI: 10.1016/j.neunet.2020.11.002] [Citation(s) in RCA: 35] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2020] [Revised: 10/08/2020] [Accepted: 11/05/2020] [Indexed: 11/21/2022]
Abstract
Motor imagery (MI) brain-computer interface (BCI) and neurofeedback (NF) with electroencephalogram (EEG) signals are commonly used for motor function improvement in healthy subjects and to restore neurological functions in stroke patients. Generally, in order to decrease noisy and redundant information in unrelated EEG channels, channel selection methods are used which provide feasible BCI and NF implementations with better performances. Our assumption is that there are causal interactions between the channels of EEG signal in MI tasks that are repeated in different trials of a BCI and NF experiment. Therefore, a novel method for EEG channel selection is proposed which is based on Granger causality (GC) analysis. Additionally, the machine-learning approach is used to cluster independent component analysis (ICA) components of the EEG signal into artifact and normal EEG clusters. After channel selection, using the common spatial pattern (CSP) and regularized CSP (RCSP), features are extracted and with the k-nearest neighbor (k-NN), support vector machine (SVM) and linear discriminant analysis (LDA) classifiers, MI tasks are classified into left and right hand MI. The goal of this study is to achieve a method resulting in lower EEG channels with higher classification performance in MI-based BCI and NF by causal constraint. The proposed method based on GC, with only eight selected channels, results in 93.03% accuracy, 92.93% sensitivity, and 93.12% specificity, with RCSP feature extractor and best classifier for each subject, after being applied on Physionet MI dataset, which is increased by 3.95%, 3.73%, and 4.13%, in comparison with correlation-based channel selection method.
Collapse
|
26
|
Khan MA, Das R, Iversen HK, Puthusserypady S. Review on motor imagery based BCI systems for upper limb post-stroke neurorehabilitation: From designing to application. Comput Biol Med 2020; 123:103843. [PMID: 32768038 DOI: 10.1016/j.compbiomed.2020.103843] [Citation(s) in RCA: 66] [Impact Index Per Article: 16.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2020] [Revised: 05/18/2020] [Accepted: 06/02/2020] [Indexed: 12/21/2022]
Abstract
Strokes are a growing cause of mortality and many stroke survivors suffer from motor impairment as well as other types of disabilities in their daily life activities. To treat these sequelae, motor imagery (MI) based brain-computer interface (BCI) systems have shown potential to serve as an effective neurorehabilitation tool for post-stroke rehabilitation therapy. In this review, different MI-BCI based strategies, including "Functional Electric Stimulation, Robotics Assistance and Hybrid Virtual Reality based Models," have been comprehensively reported for upper-limb neurorehabilitation. Each of these approaches have been presented to illustrate the in-depth advantages and challenges of the respective BCI systems. Additionally, the current state-of-the-art and main concerns regarding BCI based post-stroke neurorehabilitation devices have also been discussed. Finally, recommendations for future developments have been proposed while discussing the BCI neurorehabilitation systems.
Collapse
Affiliation(s)
- Muhammad Ahmed Khan
- Department of Health Technology, Technical University of Denmark, 2800, Kgs. Lyngby, Denmark.
| | - Rig Das
- Department of Health Technology, Technical University of Denmark, 2800, Kgs. Lyngby, Denmark
| | - Helle K Iversen
- Department of Neurology, University of Copenhagen, Rigshospitalet, 2600, Glostrup, Denmark
| | | |
Collapse
|
27
|
Jin J, Miao Y, Daly I, Zuo C, Hu D, Cichocki A. Correlation-based channel selection and regularized feature optimization for MI-based BCI. Neural Netw 2019; 118:262-270. [PMID: 31326660 DOI: 10.1016/j.neunet.2019.07.008] [Citation(s) in RCA: 126] [Impact Index Per Article: 25.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2019] [Revised: 06/18/2019] [Accepted: 07/07/2019] [Indexed: 10/26/2022]
Abstract
Multi-channel EEG data are usually necessary for spatial pattern identification in motor imagery (MI)-based brain computer interfaces (BCIs). To some extent, signals from some channels containing redundant information and noise may degrade BCI performance. We assume that the channels related to MI should contain common information when participants are executing the MI tasks. Based on this hypothesis, a correlation-based channel selection (CCS) method is proposed to select the channels that contained more correlated information in this study. The aim is to improve the classification performance of MI-based BCIs. Furthermore, a novel regularized common spatial pattern (RCSP) method is used to extract effective features. Finally, a support vector machine (SVM) classifier with the Radial Basis Function (RBF) kernel is trained to accurately identify the MI tasks. An experimental study is implemented on three public EEG datasets (BCI competition IV dataset 1, BCI competition III dataset IVa and BCI competition III dataset IIIa) to validate the effectiveness of the proposed methods. The results show that the CCS algorithm obtained superior classification accuracy (78% versus 56.4% for dataset1, 86.6% versus 76.5% for dataset 2 and 91.3% versus 85.1% for dataset 3) compared to the algorithm using all channels (AC), when CSP is used to extract the features. Furthermore, RCSP could further improve the classification accuracy (81.6% for dataset1, 87.4% for dataset2 and 91.9% for dataset 3), when CCS is used to select the channels.
Collapse
Affiliation(s)
- Jing Jin
- Key Laboratory of Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology, Shanghai, PR China.
| | - Yangyang Miao
- Key Laboratory of Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology, Shanghai, PR China
| | - Ian Daly
- Brain-Computer Interfaces and Neural Engineering Laboratory, School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe Park, Colchester, Essex, CO4 3SQ, UK
| | - Cili Zuo
- Key Laboratory of Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology, Shanghai, PR China
| | - Dewen Hu
- College of Mechatronic Engineering and Automation, National University of Defense Technology Changsha, Hunan 410073, PR China
| | - Andrzej Cichocki
- Skolkowo Institute of Science and Technology (SKOLTECH), 143026 Moscow, Russia; Systems Research Institute PAS, Warsaw, Poland; Nicolaus Copernicus University (UMK), Torun, Poland
| |
Collapse
|
28
|
Costa AP, Møller JS, Iversen HK, Puthusserypady S. An adaptive CSP filter to investigate user independence in a 3-class MI-BCI paradigm. Comput Biol Med 2018; 103:24-33. [PMID: 30336362 DOI: 10.1016/j.compbiomed.2018.09.021] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2018] [Revised: 09/20/2018] [Accepted: 09/24/2018] [Indexed: 01/01/2023]
Abstract
This paper describes the implementation of a Brain Computer Interface (BCI) scheme using a common spatial patterns (CSP) filter in combination with a Recursive Least Squares (RLS) approach to iteratively update the coefficients of the CSP filter. The proposed adaptive CSP (ACSP) algorithm is made more robust by introducing regularization using Diagonal Loading (DL), and thus will be able to significantly reduce the length of training sessions when introducing new patients to the BCI system. The system is tested on a 4-class multi-limb motor imagery (MI) data set from the BCI competition IV (2a), and a more complex single limb 3-class MI dataset recorded in-house. The latter dataset is produced to mimic an upper limb rehabilitation session, e.g., after stroke. The findings indicate that when extensive calibration data is available, the ACSP performs comparably to the CSP (kappa value of 0.523 and 0.502, respectively, for the 4-class problem); for reduced calibration sessions, the ACSP significantly improved the performance of the system (up to 4-fold). The proposed paradigm proved feasible and the ACSP algorithm seems to enable a user or semi user independent scenario, where the need for long system calibration sessions without feedback is eliminated.
Collapse
Affiliation(s)
- Ana P Costa
- Department of Electrical Engineering, Technical University of Denmark, Kongens Lyngby, 2800, Denmark.
| | - Jakob S Møller
- Department of Electrical Engineering, Technical University of Denmark, Kongens Lyngby, 2800, Denmark.
| | - Helle K Iversen
- Department of Neurology, Rigshospitalet, Glostrup, 2600, Denmark.
| | - Sadasivan Puthusserypady
- Department of Electrical Engineering, Technical University of Denmark, Kongens Lyngby, 2800, Denmark.
| |
Collapse
|
29
|
Suwannarat A, Pan-Ngum S, Israsena P. Comparison of EEG measurement of upper limb movement in motor imagery training system. Biomed Eng Online 2018; 17:103. [PMID: 30071853 PMCID: PMC6071373 DOI: 10.1186/s12938-018-0534-0] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2018] [Accepted: 07/21/2018] [Indexed: 11/21/2022] Open
Abstract
BACKGROUND One of the most promising applications for electroencephalogram (EEG)-based brain computer interface is for stroke rehabilitation. Implemented as a standalone motor imagery (MI) training system or as part of a rehabilitation robotic system, many studies have shown benefits of using them to restore motor control in stroke patients. Hand movements have widely been chosen as MI tasks. Although potentially more challenging to analyze, wrist and forearm movement such as wrist flexion/extension and forearm pronation/supination should also be considered for MI tasks, because these movements are part of the main exercises given to patients in conventional stroke rehabilitation. This paper will evaluate the effectiveness of such movements for MI tasks. METHODS Three hand and wrist movement tasks which were hand opening/closing, wrist flexion/extension and forearm pronation/supination were chosen as motor imagery tasks for both hands. Eleven subjects participated in the experiment. All of them completed hand opening/closing task session. Ten subjects completed two MI task sessions which were hand opening/closing and wrist flexion/extension. Five subjects completed all three MI tasks sessions. Each MI task comprised 8 sessions spanning a 4 weeks period. For classification, feature extraction based on common spatial pattern (CSP) algorithm was used. Two types were implemented, one with conventional CSP (termed WB) and one with an increase number of features achieved by filtering EEG data into five bands (termed FB). Classification was done by linear discriminant analysis (LDA) and support vector machine (SVM). RESULTS Eight-fold cross validation was applied on EEG data. LDA and SVM gave comparable classification accuracy. FB achieved significantly higher classification accuracy compared to WB. The accuracy of classifying wrist flexion/extension task were higher than that of classifying hand opening/closing task in all subjects. Classifying forearm pronation/supination task achieved higher accuracy than classifying hand opening/closing task in most subjects but achieved lower accuracy than classifying wrist flexion/extension task in all subjects. Significant improvements of classification accuracy were found in nine subjects when considering individual sessions of experiments of all MI tasks. The results of classifying hand opening/closing task and wrist flexion/extension task were comparable to the results of classifying hand opening/closing task and forearm pronation/supination task. Classification accuracy of wrist flexion/extension task and forearm pronation/supination task was lower than those of hand movement tasks and wrist movement tasks. CONCLUSION High classification accuracy of the three MI tasks support the possibility of using EEG-based stroke rehabilitation system with these movements. Either LDA or SVM can equally be chosen as a classifier since the difference of their accuracies is not statistically significant. Significantly higher classification accuracy made FB more suitable for classifying MI task compared to WB. More training sessions could potentially lead to better accuracy as evident in most subjects in this experiment.
Collapse
Affiliation(s)
- Arpa Suwannarat
- Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Phayathai Road, Wang Mai, Pathumwan, Bangkok, 10330, Thailand
| | - Setha Pan-Ngum
- Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Phayathai Road, Wang Mai, Pathumwan, Bangkok, 10330, Thailand.
| | - Pasin Israsena
- National Electronics and Computer Technology Center, 112 Thailand Science Park, Phahonyothin Road, Khlong Nueng, Khlong Luang, Pathumthani, 12120, Thailand
| |
Collapse
|
30
|
Kumar S, Sharma A. A new parameter tuning approach for enhanced motor imagery EEG signal classification. Med Biol Eng Comput 2018; 56:1861-1874. [PMID: 29616456 DOI: 10.1007/s11517-018-1821-4] [Citation(s) in RCA: 43] [Impact Index Per Article: 7.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2017] [Accepted: 03/19/2018] [Indexed: 12/13/2022]
Abstract
A brain-computer interface (BCI) system allows direct communication between the brain and the external world. Common spatial pattern (CSP) has been used effectively for feature extraction of data used in BCI systems. However, many studies show that the performance of a BCI system using CSP largely depends on the filter parameters. The filter parameters that yield most discriminating information vary from subject to subject and manually tuning of the filter parameters is a difficult and time-consuming exercise. In this paper, we propose a new automated filter tuning approach for motor imagery electroencephalography (EEG) signal classification, which automatically and flexibly finds the filter parameters for optimal performance. We have evaluated the performance of our proposed method on two public benchmark datasets. Compared to the existing conventional CSP approach, our method reduces the average classification error rate by 2.89% and 3.61% for BCI Competition III dataset IVa and BCI Competition IV dataset I, respectively. Moreover, our proposed approach also achieved lowest average classification error rate compared to state-of-the-art methods studied in this paper. Thus, our proposed method can be potentially used for developing improved BCI systems, which can assist people with disabilities to recover their environmental control. It can also be used for enhanced disease recognition such as epileptic seizure detection using EEG signals. Graphical abstract ᅟ.
Collapse
Affiliation(s)
- Shiu Kumar
- Department of Electronics, Instrumentation & Control Engineering, School of Electrical & Electronics Engineering, Fiji National University, Samabula, Fiji
- School of Engineering and Physics, Faculty of Science, Technology & Environment, The University of the South Pacific, Suva, Fiji
| | - Alok Sharma
- School of Engineering and Physics, Faculty of Science, Technology & Environment, The University of the South Pacific, Suva, Fiji.
- Institute for Integrated and Intelligent Systems (IIIS), Griffith University, Brisbane, Australia.
- Laboratory for Medical Science Mathematics, RIKEN Center for Integrative Medical Sciences, Yokohama, 230-0045, Japan.
| |
Collapse
|
31
|
Kumar S, Mamun K, Sharma A. CSP-TSM: Optimizing the performance of Riemannian tangent space mapping using common spatial pattern for MI-BCI. Comput Biol Med 2017; 91:231-242. [PMID: 29100117 DOI: 10.1016/j.compbiomed.2017.10.025] [Citation(s) in RCA: 51] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2017] [Revised: 10/08/2017] [Accepted: 10/23/2017] [Indexed: 11/18/2022]
Abstract
BACKGROUND Classification of electroencephalography (EEG) signals for motor imagery based brain computer interface (MI-BCI) is an exigent task and common spatial pattern (CSP) has been extensively explored for this purpose. In this work, we focused on developing a new framework for classification of EEG signals for MI-BCI. METHOD We propose a single band CSP framework for MI-BCI that utilizes the concept of tangent space mapping (TSM) in the manifold of covariance matrices. The proposed method is named CSP-TSM. Spatial filtering is performed on the bandpass filtered MI EEG signal. Riemannian tangent space is utilized for extracting features from the spatial filtered signal. The TSM features are then fused with the CSP variance based features and feature selection is performed using Lasso. Linear discriminant analysis (LDA) is then applied to the selected features and finally classification is done using support vector machine (SVM) classifier. RESULTS The proposed framework gives improved performance for MI EEG signal classification in comparison with several competing methods. Experiments conducted shows that the proposed framework reduces the overall classification error rate for MI-BCI by 3.16%, 5.10% and 1.70% (for BCI Competition III dataset IVa, BCI Competition IV Dataset I and BCI Competition IV Dataset IIb, respectively) compared to the conventional CSP method under the same experimental settings. CONCLUSION The proposed CSP-TSM method produces promising results when compared with several competing methods in this paper. In addition, the computational complexity is less compared to that of TSM method. Our proposed CSP-TSM framework can be potentially used for developing improved MI-BCI systems.
Collapse
Affiliation(s)
- Shiu Kumar
- Department of Electronics, Instrumentation and Control, School of Electrical & Electronics Engineering, College of Engineering, Science and Technology, Fiji National University, Suva, Fiji; School of Engineering and Physics, Faculty of Science, Technology and Environment, The University of the South Pacific, Suva, Fiji.
| | - Kabir Mamun
- School of Engineering and Physics, Faculty of Science, Technology and Environment, The University of the South Pacific, Suva, Fiji.
| | - Alok Sharma
- School of Engineering and Physics, Faculty of Science, Technology and Environment, The University of the South Pacific, Suva, Fiji; Institute for Integrated and Intelligent Systems (IIIS), Griffith University, Brisbane, Australia; RIKEN Center for Integrative Medical Sciences, Yokohama 230-0045, Japan.
| |
Collapse
|
32
|
Cheng M, Lu Z, Wang H. Regularized common spatial patterns with subject-to-subject transfer of EEG signals. Cogn Neurodyn 2017; 11:173-81. [PMID: 28348648 DOI: 10.1007/s11571-016-9417-x] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2016] [Revised: 10/24/2016] [Accepted: 10/31/2016] [Indexed: 10/20/2022] Open
Abstract
In the context of brain-computer interface (BCI) system, the common spatial patterns (CSP) method has been used to extract discriminative spatial filters for the classification of electroencephalogram (EEG) signals. However, the classification performance of CSP typically deteriorates when a few training samples are collected from a new BCI user. In this paper, we propose an approach that maintains or improves the recognition accuracy of the system with only a small size of training data set. The proposed approach is formulated by regularizing the classical CSP technique with the strategy of transfer learning. Specifically, we incorporate into the CSP analysis inter-subject information involving the same task, by minimizing the difference between the inter-subject features. Experimental results on two data sets from BCI competitions show that the proposed approach greatly improves the classification performance over that of the conventional CSP method; the transformed variant proved to be successful in almost every case, based on a small number of available training samples.
Collapse
|
33
|
Zhang Y, Zhou G, Jin J, Wang X, Cichocki A. Optimizing spatial patterns with sparse filter bands for motor-imagery based brain-computer interface. J Neurosci Methods 2015; 255:85-91. [PMID: 26277421 DOI: 10.1016/j.jneumeth.2015.08.004] [Citation(s) in RCA: 117] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2015] [Revised: 08/03/2015] [Accepted: 08/05/2015] [Indexed: 11/19/2022]
Abstract
BACKGROUND Common spatial pattern (CSP) has been most popularly applied to motor-imagery (MI) feature extraction for classification in brain-computer interface (BCI) application. Successful application of CSP depends on the filter band selection to a large degree. However, the most proper band is typically subject-specific and can hardly be determined manually. NEW METHOD This study proposes a sparse filter band common spatial pattern (SFBCSP) for optimizing the spatial patterns. SFBCSP estimates CSP features on multiple signals that are filtered from raw EEG data at a set of overlapping bands. The filter bands that result in significant CSP features are then selected in a supervised way by exploiting sparse regression. A support vector machine (SVM) is implemented on the selected features for MI classification. RESULTS Two public EEG datasets (BCI Competition III dataset IVa and BCI Competition IV IIb) are used to validate the proposed SFBCSP method. Experimental results demonstrate that SFBCSP help improve the classification performance of MI. COMPARISON WITH EXISTING METHODS The optimized spatial patterns by SFBCSP give overall better MI classification accuracy in comparison with several competing methods. CONCLUSIONS The proposed SFBCSP is a potential method for improving the performance of MI-based BCI.
Collapse
Affiliation(s)
- Yu Zhang
- Key Laboratory for Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology, Shanghai 200237, China.
| | - Guoxu Zhou
- Laboratory for Advanced Brain Signal Processing, RIKEN Brain Science Institute, Wako-shi, Saitama 351-0198, Japan
| | - Jing Jin
- Key Laboratory for Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology, Shanghai 200237, China
| | - Xingyu Wang
- Key Laboratory for Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology, Shanghai 200237, China
| | - Andrzej Cichocki
- Laboratory for Advanced Brain Signal Processing, RIKEN Brain Science Institute, Wako-shi, Saitama 351-0198, Japan; System Research Institute, Polish Academy of Sciences, Warsaw 00-901, Poland
| |
Collapse
|
34
|
Kaiser V, Bauernfeind G, Kreilinger A, Kaufmann T, Kübler A, Neuper C, Müller-Putz GR. Cortical effects of user training in a motor imagery based brain-computer interface measured by fNIRS and EEG. Neuroimage 2013; 85 Pt 1:432-44. [PMID: 23651839 DOI: 10.1016/j.neuroimage.2013.04.097] [Citation(s) in RCA: 102] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2012] [Revised: 04/17/2013] [Accepted: 04/22/2013] [Indexed: 12/14/2022] Open
Abstract
The present study aims to gain insights into the effects of training with a motor imagery (MI)-based brain-computer interface (BCI) on activation patterns of the sensorimotor cortex. We used functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to investigate long-term training effects across 10 sessions using a 2-class (right hand and feet) MI-based BCI in fifteen subjects. In the course of the training a significant enhancement of activation pattern emerges, represented by an [oxy-Hb] increase in fNIRS and a stronger event-related desynchronization in the upper β-frequency band in the EEG. These effects were only visible in participants with relatively low BCI performance (mean accuracy ≤ 70%). We found that training with an MI-based BCI affects cortical activation patterns especially in users with low BCI performance. Our results may serve as a valuable contribution to the field of BCI research and provide information about the effects that training with an MI-based BCI has on cortical activation patterns. This might be useful for clinical applications of BCI which aim at promoting and guiding neuroplasticity.
Collapse
Affiliation(s)
- Vera Kaiser
- Institute for Knowledge Discovery, Graz University of Technology, Graz, Austria.
| | | | | | | | | | | | | |
Collapse
|