1
|
Kueper N, Kim SK, Kirchner EA. Avoidance of specific calibration sessions in motor intention recognition for exoskeleton-supported rehabilitation through transfer learning on EEG data. Sci Rep 2024; 14:16690. [PMID: 39030206 PMCID: PMC11271642 DOI: 10.1038/s41598-024-65910-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2024] [Accepted: 06/25/2024] [Indexed: 07/21/2024] Open
Abstract
Exoskeleton-based support for patients requires the learning of individual machine-learning models to recognize movement intentions of patients based on the electroencephalogram (EEG). A major issue in EEG-based movement intention recognition is the long calibration time required to train a model. In this paper, we propose a transfer learning approach that eliminates the need for a calibration session. This approach is validated on healthy subjects in this study. We will use the proposed approach in our future rehabilitation application, where the movement intention of the affected arm of a patient can be inferred from the EEG data recorded during bilateral arm movements enabled by the exoskeleton mirroring arm movements from the unaffected to the affected arm. For the initial evaluation, we compared two trained models for predicting unilateral and bilateral movement intentions without applying a classifier transfer. For the main evaluation, we predicted unilateral movement intentions without a calibration session by transferring the classifier trained on data from bilateral movement intentions. Our results showed that the classification performance for the transfer case was comparable to that in the non-transfer case, even with only 4 or 8 EEG channels. Our results contribute to robotic rehabilitation by eliminating the need for a calibration session, since EEG data for training is recorded during the rehabilitation session, and only a small number of EEG channels are required for model training.
Collapse
Affiliation(s)
- Niklas Kueper
- Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI), 28359, Bremen, Germany
| | - Su Kyoung Kim
- Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI), 28359, Bremen, Germany
| | - Elsa Andrea Kirchner
- Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI), 28359, Bremen, Germany.
- Institute of Medical Technology Systems, University of Duisburg-Essen, 47057, Duisburg, Germany.
| |
Collapse
|
2
|
Li J, She Q, Meng M, Du S, Zhang Y. Three-stage transfer learning for motor imagery EEG recognition. Med Biol Eng Comput 2024; 62:1689-1701. [PMID: 38342784 DOI: 10.1007/s11517-024-03036-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Accepted: 01/17/2024] [Indexed: 02/13/2024]
Abstract
Motor imagery (MI) paradigms have been widely used in neural rehabilitation and drowsiness state assessment. The progress in brain-computer interface (BCI) technology has emphasized the importance of accurately and efficiently detecting motor imagery intentions from electroencephalogram (EEG). Despite the recent breakthroughs made in developing EEG-based algorithms for decoding MI, the accuracy and efficiency of these models remain limited by technical challenges posed by cross-subject heterogeneity in EEG data processing and the scarcity of EEG data for training. Inspired by the optimal transport theory, this study aims to develop a novel three-stage transfer learning (TSTL) method, which uses the existing labeled data from a source domain to improve classification performance on an unlabeled target domain. Notably, the proposed method comprises three components, namely, the Riemannian tangent space mapping (RTSM), source domain transformer (SDT), and optimal subspace mapping (OSM). The RTSM maps a symmetric positive definite matrix from the Riemannian space to the tangent space to minimize the marginal probability distribution drift. The SDT transforms the source domain to a target domain by finding the optimal transport mapping matrix to reduce the joint probability distribution differences. The OSM finally maps the transformed source domain and original target domain to the same subspace to further mitigate the distribution discrepancy. The performance of the proposed method was validated on two public BCI datasets, and the average accuracy of the algorithm on two datasets was 72.24% and 69.29%. Our results demonstrated the improved performance of EEG-based MI detection in comparison with state-of-the-art algorithms.
Collapse
Affiliation(s)
- Junhao Li
- School of Automation, Hangzhou Dianzi University, Hangzhou, 310018, Zhejiang, China
| | - Qingshan She
- School of Automation, Hangzhou Dianzi University, Hangzhou, 310018, Zhejiang, China.
| | - Ming Meng
- School of Automation, Hangzhou Dianzi University, Hangzhou, 310018, Zhejiang, China
| | - Shengzhi Du
- Department of Electrical Engineering, Tshwane University of Technology, Pretoria, 0001, South Africa
| | - Yingchun Zhang
- Department of Biomedical Engineering, University of Houston, Houston, TX, 77204, USA
| |
Collapse
|
3
|
Huang X, Choi KS, Liang S, Zhang Y, Zhang Y, Poon S, Pedrycz W. Frequency Domain Channel-Wise Attack to CNN Classifiers in Motor Imagery Brain-Computer Interfaces. IEEE Trans Biomed Eng 2024; 71:1587-1598. [PMID: 38113159 DOI: 10.1109/tbme.2023.3344295] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/21/2023]
Abstract
OBJECTIVE Convolutional neural network (CNN), a classical structure in deep learning, has been commonly deployed in the motor imagery brain-computer interface (MIBCI). Many methods have been proposed to evaluate the vulnerability of such CNN models, primarily by attacking them using direct temporal perturbations. In this work, we propose a novel attacking approach based on perturbations in the frequency domain instead. METHODS For a given natural MI trial in the frequency domain, the proposed approach, called frequency domain channel-wise attack (FDCA), generates perturbations at each channel one after another to fool the CNN classifiers. The advances of this strategy are two-fold. First, instead of focusing on the temporal domain, perturbations are generated in the frequency domain where discriminative patterns can be extracted for motor imagery (MI) classification tasks. Second, the perturbing optimization is performed based on differential evolution algorithm in a black-box scenario where detailed model knowledge is not required. RESULTS Experimental results demonstrate the effectiveness of the proposed FDCA which achieves a significantly higher success rate than the baselines and existing methods in attacking three major CNN classifiers on four public MI benchmarks. CONCLUSION Perturbations generated in the frequency domain yield highly competitive results in attacking MIBCI deployed by CNN models even in a black-box setting, where the model information is well-protected. SIGNIFICANCE To our best knowledge, existing MIBCI attack approaches are all gradient-based methods and require details about the victim model, e.g., the parameters and objective function. We provide a more flexible strategy that does not require model details but still produces an effective attack outcome.
Collapse
|
4
|
Chu C, Zhu L, Huang A, Xu P, Ying N, Zhang J. Transfer learning with data alignment and optimal transport for EEG based motor imagery classification. J Neural Eng 2024; 21:016015. [PMID: 38232381 DOI: 10.1088/1741-2552/ad1f7a] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2023] [Accepted: 01/17/2024] [Indexed: 01/19/2024]
Abstract
Objective. The non-stationarity of electroencephalogram (EEG) signals and the variability among different subjects present significant challenges in current Brain-Computer Interfaces (BCI) research, which requires a time-consuming specific calibration procedure to address. Transfer Learning (TL) offers a potential solution by leveraging data or models from one or more source domains to facilitate learning in the target domain, so as to address these challenges.Approach. In this paper, a novel Multi-source domain Transfer Learning Fusion (MTLF) framework is proposed to address the calibration problem. Firstly, the method transforms the source domain data with the resting state segment data, in order to decrease the differences between the source domain and the target domain. Subsequently, feature extraction is performed using common spatial pattern. Finally, an improved TL classifier is employed to classify the target samples. Notably, this method does not require the label information of target domain samples, while concurrently reducing the calibration workload.Main results. The proposed MTLF is assessed on Datasets 2a and 2b from the BCI Competition IV. Compared with other algorithms, our method performed relatively the best and achieved mean classification accuracy of 73.69% and 70.83% on Datasets 2a and 2b respectively.Significance.Experimental results demonstrate that the MTLF framework effectively reduces the discrepancy between the source and target domains and acquires better classification performance on two motor imagery datasets.
Collapse
Affiliation(s)
- Chao Chu
- School of Automation, Hangzhou Dianzi University, Hangzhou 310018, People's Republic of China
| | - Lei Zhu
- School of Automation, Hangzhou Dianzi University, Hangzhou 310018, People's Republic of China
| | - Aiai Huang
- School of Automation, Hangzhou Dianzi University, Hangzhou 310018, People's Republic of China
| | - Ping Xu
- School of Automation, Hangzhou Dianzi University, Hangzhou 310018, People's Republic of China
| | - Nanjiao Ying
- School of Automation, Hangzhou Dianzi University, Hangzhou 310018, People's Republic of China
- Center for Drug Inspection of Zhejiang Province, Hangzhou 310018, People's Republic of China
| | - Jianhai Zhang
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, People's Republic of China
- Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou 310018, People's Republic of China
| |
Collapse
|
5
|
Choi J, Kong D, Cho H. Weighted Domain Adaptation Using the Graph-Structured Dataset Representation for Machinery Fault Diagnosis under Varying Operating Conditions. SENSORS (BASEL, SWITZERLAND) 2023; 24:188. [PMID: 38203050 PMCID: PMC10781203 DOI: 10.3390/s24010188] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/08/2023] [Revised: 12/22/2023] [Accepted: 12/26/2023] [Indexed: 01/12/2024]
Abstract
Data-driven fault diagnosis has received significant attention in the era of big data. Most data-driven methods have been developed under the assumption that both training and test data come from identical data distributions. However, in real-world industrial scenarios, data distribution often changes due to varying operating conditions, leading to a degradation of diagnostic performance. Although several domain adaptation methods have shown their feasibility, existing methods have overlooked metadata from the manufacturing process and treated all domains uniformly. To address these limitations, this article proposes a weighted domain adaptation method using a graph-structured dataset representation. Our framework involves encoding a collection of datasets into the proposed graph structure, which captures relations between datasets based on metadata and raw data simultaneously. Then, transferability scores of candidate source datasets for a target are estimated using the constructed graph and a graph embedding model. Finally, the fault diagnosis model is established with a voting ensemble of the base classifiers trained on candidate source datasets and their estimated transferability scores. For validation, two case studies on rotor machinery, specifically tool wear and bearing fault detection, were conducted. The experimental results demonstrate the effectiveness and superiority of the proposed method over other existing domain adaptation methods.
Collapse
Affiliation(s)
| | | | - Hyunbo Cho
- Department of Industrial and Management Engineering, Pohang University of Science and Technology, Pohang 37673, Republic of Korea; (J.C.); (D.K.)
| |
Collapse
|
6
|
Zhan Q, Wang L, Ren L, Huang X. A novel heterogeneous transfer learning method based on data stitching for the sequential coding brain computer interface. Comput Biol Med 2022; 151:106220. [PMID: 36332422 DOI: 10.1016/j.compbiomed.2022.106220] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2022] [Revised: 09/28/2022] [Accepted: 10/15/2022] [Indexed: 12/27/2022]
Abstract
OBJECTIVE For the brain computer interface (BCI), it is necessary to collect enough electroencephalography (EEG) signals to train the classification model. When the operation dimension of BCI is large, it will bring great burden to data acquisition. Fortunately, this problem can be solved by our proposed transfer learning method. METHOD For the sequential coding experimental paradigm, the multi-band data stitching with label alignment and tangent space mapping (MDSLATSM) algorithm is proposed as a novel heterogeneous transfer learning method. After filtering by multi-band filtering, the artificial signals can be obtained by data stitching from the source domain, which build a bridge between the source domain and target domain. To make the distribution of two domains closer, their covariance matrices are aligned by label alignment. After mapping to the tangent space, the features are extracted from the Riemannian manifold. Finally, the classification results are obtained with feature selection and classification. RESULTS Our data set includes the EEG signals from 16 subjects. For the heterogeneous transfer learning of cross-label, the average classification accuracy is 78.28%. MDSLATSM is also tested for cross-subject, and the average classification accuracy is 64.01%, which is better than existing methods. SIGNIFICANCE Combining multi-band filtering, data stitching, label alignment and tangent space mapping, a novel heterogeneous transfer learning method can be achieved with superior performance, which promotes the practical application of the BCI systems.
Collapse
Affiliation(s)
- Qianqian Zhan
- School of Electronics and Communication, Guangzhou University, Guangzhou, 510006, China
| | - Li Wang
- School of Electronics and Communication, Guangzhou University, Guangzhou, 510006, China.
| | - Lingling Ren
- School of Electronics and Communication, Guangzhou University, Guangzhou, 510006, China
| | - Xuewen Huang
- School of Electronics and Communication, Guangzhou University, Guangzhou, 510006, China
| |
Collapse
|
7
|
Du Y, Huang J, Huang X, Shi K, Zhou N. Dual attentive fusion for EEG-based brain-computer interfaces. Front Neurosci 2022; 16:1044631. [PMID: 36507325 PMCID: PMC9727253 DOI: 10.3389/fnins.2022.1044631] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2022] [Accepted: 10/26/2022] [Indexed: 11/24/2022] Open
Abstract
The classification based on Electroencephalogram (EEG) is a challenging task in the brain-computer interface (BCI) field due to data with a low signal-to-noise ratio. Most current deep learning based studies in this challenge focus on designing a desired convolutional neural network (CNN) to learn and classify the raw EEG signals. However, only CNN itself may not capture the highly discriminative patterns of EEG due to a lack of exploration of attentive spatial and temporal dynamics. To improve information utilization, this study proposes a Dual Attentive Fusion Model (DAFM) for the EEG-based BCI. DAFM is employed to capture the spatial and temporal information by modeling the interdependencies between the features from the EEG signals. To our best knowledge, our method is the first to fuse spatial and temporal dimensions in an interactive attention module. This module improves the expression ability of the extracted features. Extensive experiments implemented on four publicly available datasets demonstrate that our method outperforms state-of-the-art methods. Meanwhile, this work also indicates the effectiveness of Dual Attentive Fusion Module.
Collapse
Affiliation(s)
- Yuanhua Du
- College of Applied Mathematics, Chengdu University of Information Technology, Chengdu, China
| | - Jian Huang
- College of Applied Mathematics, Chengdu University of Information Technology, Chengdu, China,*Correspondence: Jian Huang
| | - Xiuyu Huang
- Centre for Smart Health, The Hong Kong Polytechnic University, Hong Kong, Hong Kong SAR, China
| | - Kaibo Shi
- School of Electronic Information and Electronic Engineering, Chengdu University, Chengdu, China
| | - Nan Zhou
- School of Electronic Information and Electronic Engineering, Chengdu University, Chengdu, China
| |
Collapse
|
8
|
Ma J, Yang B, Qiu W, Li Y, Gao S, Xia X. A large EEG dataset for studying cross-session variability in motor imagery brain-computer interface. Sci Data 2022; 9:531. [PMID: 36050394 PMCID: PMC9436944 DOI: 10.1038/s41597-022-01647-1] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2022] [Accepted: 08/09/2022] [Indexed: 11/24/2022] Open
Abstract
In building a practical and robust brain-computer interface (BCI), the classification of motor imagery (MI) from electroencephalography (EEG) across multiple days is a long-standing challenge due to the large variability of the EEG signals. We collected a large dataset of MI from 5 different days with 25 subjects, the first open-access dataset to address BCI issues across 5 different days with a large number of subjects. The dataset includes 5 session data from 5 different days (2–3 days apart) for each subject. Each session contains 100 trials of left-hand and right-hand MI. In this report, we provide the benchmarking classification accuracy for three conditions, namely, within-session classification (WS), cross-session classification (CS), and cross-session adaptation (CSA), with subject-specific models. WS achieves an average classification accuracy of up to 68.8%, while CS degrades the accuracy to 53.7% due to the cross-session variability. However, by adaptation, CSA improves the accuracy to 78.9%. We anticipate this new dataset will significantly push further progress in MI BCI research in addressing the cross-session and cross-subject challenge. Measurement(s) | Electroencephalography | Technology Type(s) | motor imagery |
Collapse
Affiliation(s)
- Jun Ma
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China
| | - Banghua Yang
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China. .,Engineering Research Center of Traditional Chinese Medicine Intelligent Rehabilitation, Ministry of Education, Shanghai, China.
| | - Wenzheng Qiu
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China
| | - Yunzhe Li
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China
| | - Shouwei Gao
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China
| | - Xinxing Xia
- School of Mechatronic Engineering and Automation, School of Medicine, Research Center of Brain-Computer Engineering, Shanghai University, Shanghai, China
| |
Collapse
|
9
|
Xu DQ, Li MA. A dual alignment-based multi-source domain adaptation framework for motor imagery EEG classification. APPL INTELL 2022; 53:10766-10788. [PMID: 36039116 PMCID: PMC9402410 DOI: 10.1007/s10489-022-04077-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/08/2022] [Indexed: 11/25/2022]
Affiliation(s)
- Dong-qin Xu
- Faculty of Information Technology, Beijing University of Technology, Beijing, 100124 China
| | - Ming-ai Li
- Faculty of Information Technology, Beijing University of Technology, Beijing, 100124 China
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing, 100124 China
- Engineering Research Center of Digital Community, Ministry of Education, Beijing, 100124 China
| |
Collapse
|
10
|
Wyser DG, Kanzler CM, Salzmann L, Lambercy O, Wolf M, Scholkmann F, Gassert R. Characterizing reproducibility of cerebral hemodynamic responses when applying short-channel regression in functional near-infrared spectroscopy. NEUROPHOTONICS 2022; 9:015004. [PMID: 35265732 PMCID: PMC8901194 DOI: 10.1117/1.nph.9.1.015004] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/11/2021] [Accepted: 02/11/2022] [Indexed: 05/06/2023]
Abstract
Significance: Functional near-infrared spectroscopy (fNIRS) enables the measurement of brain activity noninvasively. Optical neuroimaging with fNIRS has been shown to be reproducible on the group level and hence is an excellent research tool, but the reproducibility on the single-subject level is still insufficient, challenging the use for clinical applications. Aim: We investigated the effect of short-channel regression (SCR) as an approach to obtain fNIRS measurements with higher reproducibility on a single-subject level. SCR simultaneously considers contributions from long- and short-separation channels and removes confounding physiological changes through the regression of the short-separation channel information. Approach: We performed a test-retest study with a hand grasping task in 15 healthy subjects using a wearable fNIRS device, optoHIVE. Relevant brain regions were localized with transcranial magnetic stimulation to ensure correct placement of the optodes. Reproducibility was assessed by intraclass correlation, correlation analysis, mixed effects modeling, and classification accuracy of the hand grasping task. Further, we characterized the influence of SCR on reproducibility. Results: We found a high reproducibility of fNIRS measurements on a single-subject level ( ICC single = 0.81 and correlation r = 0.81 ). SCR increased the reproducibility from 0.64 to 0.81 ( ICC single ) but did not affect classification (85% overall accuracy). Significant intersubject variability in the reproducibility was observed and was explained by Mayer wave oscillations and low raw signal strength. The raw signal-to-noise ratio (threshold at 40 dB) allowed for distinguishing between persons with weak and strong activations. Conclusions: We report, for the first time, that fNIRS measurements are reproducible on a single-subject level using our optoHIVE fNIRS system and that SCR improves reproducibility. In addition, we give a benchmark to easily assess the ability of a subject to elicit sufficiently strong hemodynamic responses. With these insights, we pave the way for the reliable use of fNIRS neuroimaging in single subjects for neuroscientific research and clinical applications.
Collapse
Affiliation(s)
- Dominik G. Wyser
- ETH Zurich, Rehabilitation Engineering Laboratory, Department of Health Sciences and Technology, Zurich, Switzerland
- University Hospital Zurich, University of Zurich, Biomedical Optics Research Laboratory, Department of Neonatology, Zurich, Switzerland
- Address all correspondence to Dominik G. Wyser,
| | - Christoph M. Kanzler
- ETH Zurich, Rehabilitation Engineering Laboratory, Department of Health Sciences and Technology, Zurich, Switzerland
| | - Lena Salzmann
- ETH Zurich, Rehabilitation Engineering Laboratory, Department of Health Sciences and Technology, Zurich, Switzerland
| | - Olivier Lambercy
- ETH Zurich, Rehabilitation Engineering Laboratory, Department of Health Sciences and Technology, Zurich, Switzerland
| | - Martin Wolf
- University Hospital Zurich, University of Zurich, Biomedical Optics Research Laboratory, Department of Neonatology, Zurich, Switzerland
| | - Felix Scholkmann
- University Hospital Zurich, University of Zurich, Biomedical Optics Research Laboratory, Department of Neonatology, Zurich, Switzerland
- University of Bern, Institute of Complementary and Integrative Medicine, Bern, Switzerland
| | - Roger Gassert
- ETH Zurich, Rehabilitation Engineering Laboratory, Department of Health Sciences and Technology, Zurich, Switzerland
| |
Collapse
|