1
|
Wang X, Yang W, Qi W, Wang Y, Ma X, Wang W. STaRNet: A spatio-temporal and Riemannian network for high-performance motor imagery decoding. Neural Netw 2024; 178:106471. [PMID: 38945115 DOI: 10.1016/j.neunet.2024.106471] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2024] [Revised: 06/11/2024] [Accepted: 06/16/2024] [Indexed: 07/02/2024]
Abstract
Brain-computer interfaces (BCIs), representing a transformative form of human-computer interaction, empower users to interact directly with external environments through brain signals. In response to the demands for high accuracy, robustness, and end-to-end capabilities within BCIs based on motor imagery (MI), this paper introduces STaRNet, a novel model that integrates multi-scale spatio-temporal convolutional neural networks (CNNs) with Riemannian geometry. Initially, STaRNet integrates a multi-scale spatio-temporal feature extraction module that captures both global and local features, facilitating the construction of Riemannian manifolds from these comprehensive spatio-temporal features. Subsequently, a matrix logarithm operation transforms the manifold-based features into the tangent space, followed by a dense layer for classification. Without preprocessing, STaRNet surpasses state-of-the-art (SOTA) models by achieving an average decoding accuracy of 83.29% and a kappa value of 0.777 on the BCI Competition IV 2a dataset, and 95.45% accuracy with a kappa value of 0.939 on the High Gamma Dataset. Additionally, a comparative analysis between STaRNet and several SOTA models, focusing on the most challenging subjects from both datasets, highlights exceptional robustness of STaRNet. Finally, the visualizations of learned frequency bands demonstrate that temporal convolutions have learned MI-related frequency bands, and the t-SNE analyses of features across multiple layers of STaRNet exhibit strong feature extraction capabilities. We believe that the accurate, robust, and end-to-end capabilities of the STaRNet will facilitate the advancement of BCIs.
Collapse
Affiliation(s)
- Xingfu Wang
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Wenjie Yang
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Wenxia Qi
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Yu Wang
- National Engineering and Technology Research Center for ASIC Design, Institute of Automation, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Xiaojun Ma
- National Engineering and Technology Research Center for ASIC Design, Institute of Automation, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Wei Wang
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China.
| |
Collapse
|
2
|
Park H, Jun SC. Connectivity study on resting-state EEG between motor imagery BCI-literate and BCI-illiterate groups. J Neural Eng 2024; 21:046042. [PMID: 38986469 DOI: 10.1088/1741-2552/ad6187] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2023] [Accepted: 07/10/2024] [Indexed: 07/12/2024]
Abstract
Objective.Although motor imagery-based brain-computer interface (MI-BCI) holds significant potential, its practical application faces challenges such as BCI-illiteracy. To mitigate this issue, researchers have attempted to predict BCI-illiteracy by using the resting state, as this was found to be associated with BCI performance. As connectivity's significance in neuroscience has grown, BCI researchers have applied connectivity to it. However, the issues of connectivity have not been considered fully. First, although various connectivity metrics exist, only some have been used to predict BCI-illiteracy. This is problematic because each metric has a distinct hypothesis and perspective to estimate connectivity, resulting in different outcomes according to the metric. Second, the frequency range affects the connectivity estimation. In addition, it is still unknown whether each metric has its own optimal frequency range. Third, the way that estimating connectivity may vary depending upon the dataset has not been investigated. Meanwhile, we still do not know a great deal about how the resting state electroencephalography (EEG) network differs between BCI-literacy and -illiteracy.Approach.To address the issues above, we analyzed three large public EEG datasets using three functional connectivity and three effective connectivity metrics by employing diverse graph theory measures. Our analysis revealed that the appropriate frequency range to predict BCI-illiteracy varies depending upon the metric. The alpha range was found to be suitable for the metrics of the frequency domain, while alpha + theta were found to be appropriate for multivariate Granger causality. The difference in network efficiency between BCI-literate and -illiterate groups was constant regardless of the metrics and datasets used. Although we observed that BCI-literacy had stronger connectivity, no other significant constructional differences were found.Significance.Based upon our findings, we predicted MI-BCI performance for the entire dataset. We discovered that combining several graph features could improve the prediction's accuracy.
Collapse
Affiliation(s)
- Hanjin Park
- AI Graduate School, Gwangju Institute of Science and Technology, Gwangju, Republic of Korea
| | - Sung Chan Jun
- AI Graduate School, Gwangju Institute of Science and Technology, Gwangju, Republic of Korea
- School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology, Gwangju, Republic of Korea
| |
Collapse
|
3
|
Meng J, Li S, Li G, Luo R, Sheng X, Zhu X. A model-based brain switch via periodic motor imagery modulation for asynchronous brain-computer interfaces. J Neural Eng 2024; 21:046035. [PMID: 39029496 DOI: 10.1088/1741-2552/ad6595] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2024] [Accepted: 07/19/2024] [Indexed: 07/21/2024]
Abstract
Objective.Brain switches provide a tangible solution to asynchronized brain-computer interface, which decodes user intention without a pre-programmed structure. However, most brain switches based on electroencephalography signals have high false positive rates (FPRs), resulting in less practicality. This research aims to improve the operating mode and usability of the brain switch.Approach.Here, we propose a novel virtual physical model-based brain switch that leverages periodic active modulation. An optimization problem of minimizing the triggering time subject to a required FPR is formulated, numerical and analytical approximate solutions are obtained based on the model.Main results.Our motor imagery (MI)-based brain switch can reach 0.8FP/h FPR with a median triggering time of 58 s. We evaluated the proposed brain switch during online device control, and their average FPRs substantially outperformed the conventional brain switches in the literature. We further improved the proposed brain switch with the Common Spatial Pattern (CSP) and optimization method. An average FPR of 0.3 FPs/h was obtained for the MI-CSP-based brain switch, and the average triggering time improved to 21.6 s.Significance.This study provides a new approach that could significantly reduce the brain switch's FPR to less than 1 Fps/h, which was less than 10% of the FPR (decreasing by more than a magnitude of order) by other endogenous methods, and the reaction time was comparable to the state-of-the-art approaches. This represents a significant advancement over the current non-invasive asynchronous BCI and will open widespread avenues for translating BCI towards clinical applications.
Collapse
Affiliation(s)
- Jianjun Meng
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
- State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Songwei Li
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
- State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Guangye Li
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
- State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Ruijie Luo
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
- State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Xinjun Sheng
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
- State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Xiangyang Zhu
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
- State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| |
Collapse
|
4
|
Mishra AR, Kumar R, Gupta V, Prabhu S, Upadhyay R, Chhipa PC, Rakesh S, Mokayed H, Das Chakladar D, De K, Liwicki M, Simistira Liwicki F, Saini R. SignEEG v1.0: Multimodal Dataset with Electroencephalography and Hand-written Signature for Biometric Systems. Sci Data 2024; 11:718. [PMID: 38956046 PMCID: PMC11220021 DOI: 10.1038/s41597-024-03546-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2023] [Accepted: 06/18/2024] [Indexed: 07/04/2024] Open
Abstract
Handwritten signatures in biometric authentication leverage unique individual characteristics for identification, offering high specificity through dynamic and static properties. However, this modality faces significant challenges from sophisticated forgery attempts, underscoring the need for enhanced security measures in common applications. To address forgery in signature-based biometric systems, integrating a forgery-resistant modality, namely, noninvasive electroencephalography (EEG), which captures unique brain activity patterns, can significantly enhance system robustness by leveraging multimodality's strengths. By combining EEG, a physiological modality, with handwritten signatures, a behavioral modality, our approach capitalizes on the strengths of both, significantly fortifying the robustness of biometric systems through this multimodal integration. In addition, EEG's resistance to replication offers a high-security level, making it a robust addition to user identification and verification. This study presents a new multimodal SignEEG v1.0 dataset based on EEG and hand-drawn signatures from 70 subjects. EEG signals and hand-drawn signatures have been collected with Emotiv Insight and Wacom One sensors, respectively. The multimodal data consists of three paradigms based on mental, & motor imagery, and physical execution: i) thinking of the signature's image, (ii) drawing the signature mentally, and (iii) drawing a signature physically. Extensive experiments have been conducted to establish a baseline with machine learning classifiers. The results demonstrate that multimodality in biometric systems significantly enhances robustness, achieving high reliability even with limited sample sizes. We release the raw, pre-processed data and easy-to-follow implementation details.
Collapse
Affiliation(s)
- Ashish Ranjan Mishra
- Department of Computer Science and Engineering, Madan Mohan Malaviya University of Technology, Gorakhpur, UP, India.
| | - Rakesh Kumar
- Department of Computer Science and Engineering, Madan Mohan Malaviya University of Technology, Gorakhpur, UP, India
| | - Vibha Gupta
- Department of Molecular and Clinical Medicine, University of Gothenburg, Gothenburg, Sweden
| | - Sameer Prabhu
- Operation, Maintenance and Acoustics, Luleå University of Technology, Luleå, Sweden
| | - Richa Upadhyay
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Prakash Chandra Chhipa
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Sumit Rakesh
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Hamam Mokayed
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Debashis Das Chakladar
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Kanjar De
- Department of Video Communication and Applications, Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute, Berlin, Germany
| | - Marcus Liwicki
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Foteini Simistira Liwicki
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| | - Rajkumar Saini
- Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Intelligent Systems LAB, Luleå, Sweden
| |
Collapse
|
5
|
Kosnoff J, Yu K, Liu C, He B. Transcranial focused ultrasound to V5 enhances human visual motion brain-computer interface by modulating feature-based attention. Nat Commun 2024; 15:4382. [PMID: 38862476 PMCID: PMC11167030 DOI: 10.1038/s41467-024-48576-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Accepted: 05/02/2024] [Indexed: 06/13/2024] Open
Abstract
A brain-computer interface (BCI) enables users to control devices with their minds. Despite advancements, non-invasive BCIs still exhibit high error rates, prompting investigation into the potential reduction through concurrent targeted neuromodulation. Transcranial focused ultrasound (tFUS) is an emerging non-invasive neuromodulation technology with high spatiotemporal precision. This study examines whether tFUS neuromodulation can improve BCI outcomes, and explores the underlying mechanism of action using high-density electroencephalography (EEG) source imaging (ESI). As a result, V5-targeted tFUS significantly reduced the error in a BCI speller task. Source analyses revealed a significantly increase in theta and alpha activities in the tFUS condition at both V5 and downstream in the dorsal visual processing pathway. Correlation analysis indicated that the connection within the dorsal processing pathway was preserved during tFUS stimulation, while the ventral connection was weakened. These findings suggest that V5-targeted tFUS enhances feature-based attention to visual motion.
Collapse
Affiliation(s)
- Joshua Kosnoff
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15237, USA
| | - Kai Yu
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15237, USA
| | - Chang Liu
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15237, USA
- Department of Biomedical Engineering, Boston University, Boston, MA, 02215, USA
| | - Bin He
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15237, USA.
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, 15237, USA.
| |
Collapse
|
6
|
Pan H, Ding P, Wang F, Li T, Zhao L, Nan W, Fu Y, Gong A. Comprehensive evaluation methods for translating BCI into practical applications: usability, user satisfaction and usage of online BCI systems. Front Hum Neurosci 2024; 18:1429130. [PMID: 38903409 PMCID: PMC11188342 DOI: 10.3389/fnhum.2024.1429130] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2024] [Accepted: 05/20/2024] [Indexed: 06/22/2024] Open
Abstract
Although brain-computer interface (BCI) is considered a revolutionary advancement in human-computer interaction and has achieved significant progress, a considerable gap remains between the current technological capabilities and their practical applications. To promote the translation of BCI into practical applications, the gold standard for online evaluation for classification algorithms of BCI has been proposed in some studies. However, few studies have proposed a more comprehensive evaluation method for the entire online BCI system, and it has not yet received sufficient attention from the BCI research and development community. Therefore, the qualitative leap from analyzing and modeling for offline BCI data to the construction of online BCI systems and optimizing their performance is elaborated, and then user-centred is emphasized, and then the comprehensive evaluation methods for translating BCI into practical applications are detailed and reviewed in the article, including the evaluation of the usability (including effectiveness and efficiency of systems), the evaluation of the user satisfaction (including BCI-related aspects, etc.), and the evaluation of the usage (including the match between the system and user, etc.) of online BCI systems. Finally, the challenges faced in the evaluation of the usability and user satisfaction of online BCI systems, the efficacy of online BCI systems, and the integration of BCI and artificial intelligence (AI) and/or virtual reality (VR) and other technologies to enhance the intelligence and user experience of the system are discussed. It is expected that the evaluation methods for online BCI systems elaborated in this review will promote the translation of BCI into practical applications.
Collapse
Affiliation(s)
- He Pan
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Peng Ding
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Fan Wang
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Tianwen Li
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Zhao
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Wenya Nan
- Department of Psychology, School of Education, Shanghai Normal University, Shanghai, China
| | - Yunfa Fu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Anmin Gong
- School of Information Engineering, Chinese People's Armed Police Force Engineering University, Xi’an, China
| |
Collapse
|
7
|
Ma X, Chen W, Pei Z, Zhang Y, Chen J. Attention-based convolutional neural network with multi-modal temporal information fusion for motor imagery EEG decoding. Comput Biol Med 2024; 175:108504. [PMID: 38701593 DOI: 10.1016/j.compbiomed.2024.108504] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2023] [Revised: 04/15/2024] [Accepted: 04/21/2024] [Indexed: 05/05/2024]
Abstract
Convolutional neural network (CNN) has been widely applied in motor imagery (MI)-based brain computer interface (BCI) to decode electroencephalography (EEG) signals. However, due to the limited perceptual field of convolutional kernel, CNN only extracts features from local region without considering long-term dependencies for EEG decoding. Apart from long-term dependencies, multi-modal temporal information is equally important for EEG decoding because it can offer a more comprehensive understanding of the temporal dynamics of neural processes. In this paper, we propose a novel deep learning network that combines CNN with self-attention mechanism to encapsulate multi-modal temporal information and global dependencies. The network first extracts multi-modal temporal information from two distinct perspectives: average and variance. A shared self-attention module is then designed to capture global dependencies along these two feature dimensions. We further design a convolutional encoder to explore the relationship between average-pooled and variance-pooled features and fuse them into more discriminative features. Moreover, a data augmentation method called signal segmentation and recombination is proposed to improve the generalization capability of the proposed network. The experimental results on the BCI Competition IV-2a (BCIC-IV-2a) and BCI Competition IV-2b (BCIC-IV-2b) datasets show that our proposed method outperforms the state-of-the-art methods and achieves 4-class average accuracy of 85.03% on the BCIC-IV-2a dataset. The proposed method implies the effectiveness of multi-modal temporal information fusion in attention-based deep learning networks and provides a new perspective for MI-EEG decoding. The code is available at https://github.com/Ma-Xinzhi/EEG-TransNet.
Collapse
Affiliation(s)
- Xinzhi Ma
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China; Hangzhou Innovation Institute, Beihang University, Hangzhou, China
| | - Weihai Chen
- School of Electrical Engineering and Automation, Anhui University, Hefei, China.
| | - Zhongcai Pei
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China; Hangzhou Innovation Institute, Beihang University, Hangzhou, China
| | - Yue Zhang
- Hangzhou Innovation Institute, Beihang University, Hangzhou, China
| | - Jianer Chen
- Department of Geriatric Rehabilitation, Third Affiliated Hospital, Zhejiang Chinese Medical University, Hangzhou, China
| |
Collapse
|
8
|
Huang D, Wang Y, Fan L, Yu Y, Zhao Z, Zeng P, Wang K, Li N, Shen H. Decoding Subject-Driven Cognitive States from EEG Signals for Cognitive Brain-Computer Interface. Brain Sci 2024; 14:498. [PMID: 38790476 PMCID: PMC11120245 DOI: 10.3390/brainsci14050498] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2024] [Revised: 05/06/2024] [Accepted: 05/14/2024] [Indexed: 05/26/2024] Open
Abstract
In this study, we investigated the feasibility of using electroencephalogram (EEG) signals to differentiate between four distinct subject-driven cognitive states: resting state, narrative memory, music, and subtraction tasks. EEG data were collected from seven healthy male participants while performing these cognitive tasks, and the raw EEG signals were transformed into time-frequency maps using continuous wavelet transform. Based on these time-frequency maps, we developed a convolutional neural network model (TF-CNN-CFA) with a channel and frequency attention mechanism to automatically distinguish between these cognitive states. The experimental results demonstrated that the model achieved an average classification accuracy of 76.14% in identifying these four cognitive states, significantly outperforming traditional EEG signal processing methods and other classical image classification algorithms. Furthermore, we investigated the impact of varying lengths of EEG signals on classification performance and found that TF-CNN-CFA demonstrates consistent performance across different window lengths, indicating its strong generalization capability. This study validates the ability of EEG to differentiate higher cognitive states, which could potentially offer a novel BCI paradigm.
Collapse
Affiliation(s)
- Dingyong Huang
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China; (D.H.); (L.F.); (Y.Y.); (Z.Z.); (P.Z.); (K.W.)
| | - Yingjie Wang
- College of Physical Education and Health, Hebei Normal University of Science & Technology, Qinhuangdao 066004, China;
| | - Liangwei Fan
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China; (D.H.); (L.F.); (Y.Y.); (Z.Z.); (P.Z.); (K.W.)
| | - Yang Yu
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China; (D.H.); (L.F.); (Y.Y.); (Z.Z.); (P.Z.); (K.W.)
| | - Ziyu Zhao
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China; (D.H.); (L.F.); (Y.Y.); (Z.Z.); (P.Z.); (K.W.)
| | - Pu Zeng
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China; (D.H.); (L.F.); (Y.Y.); (Z.Z.); (P.Z.); (K.W.)
| | - Kunqing Wang
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China; (D.H.); (L.F.); (Y.Y.); (Z.Z.); (P.Z.); (K.W.)
| | - Na Li
- Radiology Department, Xiangya 3rd Hospital, Central South University, Changsha 410013, China;
| | - Hui Shen
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China; (D.H.); (L.F.); (Y.Y.); (Z.Z.); (P.Z.); (K.W.)
| |
Collapse
|
9
|
Kim H, Won K, Ahn M, Jun SC. Comparison of recognition methods for an asynchronous (un-cued) BCI system: an investigation with 40-class SSVEP dataset. Biomed Eng Lett 2024; 14:617-630. [PMID: 38645586 PMCID: PMC11026332 DOI: 10.1007/s13534-024-00357-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Revised: 01/16/2024] [Accepted: 01/24/2024] [Indexed: 04/23/2024] Open
Abstract
Steady-state visual evoked potential (SSVEP)-based brain-computer Interface (BCI) has demonstrated the potential to manage multi-command targets to achieve high-speed communication. Recent studies on multi-class SSVEP-based BCI have focused on synchronous systems, which rely on predefined time and task indicators; thus, these systems that use passive approaches may be less suitable for practical applications. Asynchronous systems recognize the user's intention (whether or not the user is willing to use systems) from brain activity; then, after recognizing the user's willingness, they begin to operate by switching swiftly for real-time control. Consequently, various methodologies have been proposed to capture the user's intention. However, in-depth investigation of recognition methods in asynchronous BCI system is lacking. Thus, in this work, three recognition methods (power spectral density analysis, canonical correlation analysis (CCA), and support vector machine (SVM)) used widely in asynchronous SSVEP BCI systems were explored to compare their performance. Further, we categorized asynchronous systems into two approaches (1-stage and 2-stage) based upon the recognition process's design, and compared their performance. To do so, a 40-class SSVEP dataset collected from 40 subjects was introduced. Finally, we found that the CCA-based method in the 2-stage approach demonstrated statistically significantly higher performance with a sensitivity of 97.62 ± 02.06%, specificity of 76.50 ± 23.50%, and accuracy of 75.59 ± 10.09%. Thus, it is expected that the 2-stage approach together with CCA-based recognition and FB-CCA classification have good potential to be implemented in practical asynchronous SSVEP BCI systems.
Collapse
Affiliation(s)
- Heegyu Kim
- School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology, Bukgu, Gwangju, 61005 Korea
| | - Kyungho Won
- Hybrid Team, Inria, Univ Rennes, IRISA, CNRS, F35000 Rennes, France
| | - Minkyu Ahn
- School of Computer Science and Electrical Engineering, Handong Global University, Bukgu, Pohang, 37554 Korea
| | - Sung Chan Jun
- School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology, Bukgu, Gwangju, 61005 Korea
- School of Artificial Intelligence, Gwangju Institute of Science and Technology, Bukgu, Gwangju, 61005 Korea
| |
Collapse
|
10
|
Forenzo D, Zhu H, Shanahan J, Lim J, He B. Continuous Tracking using Deep Learning-based Decoding for Non-invasive Brain-Computer Interface. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.10.12.562084. [PMID: 37905046 PMCID: PMC10614823 DOI: 10.1101/2023.10.12.562084] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/02/2023]
Abstract
Brain-computer interfaces (BCI) using electroencephalography (EEG) provide a non-invasive method for users to interact with external devices without the need for muscle activation. While noninvasive BCIs have the potential to improve the quality of lives of healthy and motor impaired individuals, they currently have limited applications due to inconsistent performance and low degrees of freedom. In this study, we use deep learning (DL)-based decoders for online Continuous Pursuit (CP), a complex BCI task requiring the user to track an object in two-dimensional space. We developed a labeling system to use CP data for supervised learning, trained DL-based decoders based on two architectures, including a newly proposed adaptation of the PointNet architecture, and evaluated the performance over several online sessions. We rigorously evaluated the DL-based decoders in a total of 28 human participants, and found that the DL-based models improved throughout the sessions as more training data became available and significantly outperformed a traditional BCI decoder by the last session. We also performed additional experiments to test an implementation of transfer learning by pre-training models on data from other subjects, and mid-session training to reduce inter-session variability. The results from these experiments showed that pre-training did not significantly improve performance, but updating the models mid-session may have some benefit. Overall, these findings support the use of DL-based decoders for improving BCI performance in complex tasks like CP, which can expand the potential applications of BCI devices and help improve the quality of lives of healthy and motor-impaired individuals. Significance Statement Brain-computer Interfaces (BCI) have the potential to replace or restore motor functions for patients and can benefit the general population by providing a direct link of the brain with robotics or other devices. In this work, we developed a paradigm using deep learning (DL)-based decoders for continuous control of a BCI system and demonstrated its capabilities through extensive online experiments. We also investigate how DL performance is affected by varying amounts of training data and collected more than 150 hours of BCI data that can be used to train new models. The results of this study provide valuable information for developing future DL-based BCI decoders which can improve performance and help bring BCIs closer to practical applications and wide-spread use.
Collapse
|
11
|
Huang C, Shi N, Miao Y, Chen X, Wang Y, Gao X. Visual tracking brain-computer interface. iScience 2024; 27:109376. [PMID: 38510138 PMCID: PMC10951983 DOI: 10.1016/j.isci.2024.109376] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Revised: 01/25/2024] [Accepted: 02/27/2024] [Indexed: 03/22/2024] Open
Abstract
Brain-computer interfaces (BCIs) offer a way to interact with computers without relying on physical movements. Non-invasive electroencephalography-based visual BCIs, known for efficient speed and calibration ease, face limitations in continuous tasks due to discrete stimulus design and decoding methods. To achieve continuous control, we implemented a novel spatial encoding stimulus paradigm and devised a corresponding projection method to enable continuous modulation of decoded velocity. Subsequently, we conducted experiments involving 17 participants and achieved Fitt's information transfer rate (ITR) of 0.55 bps for the fixed tracking task and 0.37 bps for the random tracking task. The proposed BCI with a high Fitt's ITR was then integrated into two applications, including painting and gaming. In conclusion, this study proposed a visual BCI based-control method to go beyond discrete commands, allowing natural continuous control based on neural activity.
Collapse
Affiliation(s)
- Changxing Huang
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China
| | - Nanlin Shi
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China
| | - Yining Miao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China
| | - Xiaogang Chen
- Institute of Biomedical Engineering, Chinese Academy of Medical Sciences and Peking Union Medical College, Tianjin 300192, China
| | - Yijun Wang
- State Key Laboratory on Integrated Optoelectronics, Institute of Semiconductors, Chinese Academy of Sciences Beijing, Beijing 100083, China
| | - Xiaorong Gao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China
| |
Collapse
|
12
|
Della Vedova G, Proverbio AM. Neural signatures of imaginary motivational states: desire for music, movement and social play. Brain Topogr 2024:10.1007/s10548-024-01047-1. [PMID: 38625520 DOI: 10.1007/s10548-024-01047-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Accepted: 03/12/2024] [Indexed: 04/17/2024]
Abstract
The literature has demonstrated the potential for detecting accurate electrical signals that correspond to the will or intention to move, as well as decoding the thoughts of individuals who imagine houses, faces or objects. This investigation examines the presence of precise neural markers of imagined motivational states through the combining of electrophysiological and neuroimaging methods. 20 participants were instructed to vividly imagine the desire to move, listen to music or engage in social activities. Their EEG was recorded from 128 scalp sites and analysed using individual standardized Low-Resolution Brain Electromagnetic Tomographies (LORETAs) in the N400 time window (400-600 ms). The activation of 1056 voxels was examined in relation to the 3 motivational states. The most active dipoles were grouped in eight regions of interest (ROI), including Occipital, Temporal, Fusiform, Premotor, Frontal, OBF/IF, Parietal, and Limbic areas. The statistical analysis revealed that all motivational imaginary states engaged the right hemisphere more than the left hemisphere. Distinct markers were identified for the three motivational states. Specifically, the right temporal area was more relevant for "Social Play", the orbitofrontal/inferior frontal cortex for listening to music, and the left premotor cortex for the "Movement" desire. This outcome is encouraging in terms of the potential use of neural indicators in the realm of brain-computer interface, for interpreting the thoughts and desires of individuals with locked-in syndrome.
Collapse
Affiliation(s)
- Giada Della Vedova
- Cognitive Electrophysiology lab, Dept. of Psychology, University of Milano, Bicocca, Italy
| | - Alice Mado Proverbio
- Cognitive Electrophysiology lab, Dept. of Psychology, University of Milano, Bicocca, Italy.
- NeuroMI, Milan Center for Neuroscience, Milan, Italy.
- Department of Psychology of University of Milano-Bicocca, Piazza dell'Ateneo nuovo 1, Milan, 20162, Italy.
| |
Collapse
|
13
|
Liang G, Cao D, Wang J, Zhang Z, Wu Y. EISATC-Fusion: Inception Self-Attention Temporal Convolutional Network Fusion for Motor Imagery EEG Decoding. IEEE Trans Neural Syst Rehabil Eng 2024; 32:1535-1545. [PMID: 38536681 DOI: 10.1109/tnsre.2024.3382226] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/13/2024]
Abstract
The motor imagery brain-computer interface (MI-BCI) based on electroencephalography (EEG) is a widely used human-machine interface paradigm. However, due to the non-stationarity and individual differences among subjects in EEG signals, the decoding accuracy is limited, affecting the application of the MI-BCI. In this paper, we propose the EISATC-Fusion model for MI EEG decoding, consisting of inception block, multi-head self-attention (MSA), temporal convolutional network (TCN), and layer fusion. Specifically, we design a DS Inception block to extract multi-scale frequency band information. And design a new cnnCosMSA module based on CNN and cos attention to solve the attention collapse and improve the interpretability of the model. The TCN module is improved by the depthwise separable convolution to reduces the parameters of the model. The layer fusion consists of feature fusion and decision fusion, fully utilizing the features output by the model and enhances the robustness of the model. We improve the two-stage training strategy for model training. Early stopping is used to prevent model overfitting, and the accuracy and loss of the validation set are used as indicators for early stopping. The proposed model achieves within-subject classification accuracies of 84.57% and 87.58% on BCI Competition IV Datasets 2a and 2b, respectively. And the model achieves cross-subject classification accuracies of 67.42% and 71.23% (by transfer learning) when training the model with two sessions and one session of Dataset 2a, respectively. The interpretability of the model is demonstrated through weight visualization method.
Collapse
|
14
|
Yan L, Yu H, Liu Y, Xiang B, Cheng Y, Xu J, Wu Y, Yan F. Brain–Computer Interface Based on Motor Imagery With Visual Guidance and its Application in Control of Simulated Unmanned Aerial Vehicle. IEEE SENSORS JOURNAL 2024; 24:10779-10793. [DOI: 10.1109/jsen.2024.3363754] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/22/2024]
Affiliation(s)
- Lirong Yan
- College of Automotive Engineering, Wuhan University of Technology, Wuhan, China
| | - Hao Yu
- College of Automotive Engineering, Wuhan University of Technology, Wuhan, China
| | - Yan Liu
- College of Automotive Engineering, Wuhan University of Technology, Wuhan, China
| | - Biao Xiang
- College of Automotive Engineering, Wuhan University of Technology, Wuhan, China
| | - Yu Cheng
- College of Automotive Engineering, Wuhan University of Technology, Wuhan, China
| | - Jihong Xu
- College of Automotive Engineering, Wuhan University of Technology, Wuhan, China
| | - Yibo Wu
- Wuhan Leishen Special Equipment Company Ltd., Wuhan, China
| | - Fuwu Yan
- College of Automotive Engineering, Wuhan University of Technology, Wuhan, China
| |
Collapse
|
15
|
Forenzo D, Zhu H, Shanahan J, Lim J, He B. Continuous tracking using deep learning-based decoding for noninvasive brain-computer interface. PNAS NEXUS 2024; 3:pgae145. [PMID: 38689706 PMCID: PMC11060102 DOI: 10.1093/pnasnexus/pgae145] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Accepted: 03/28/2024] [Indexed: 05/02/2024]
Abstract
Brain-computer interfaces (BCI) using electroencephalography provide a noninvasive method for users to interact with external devices without the need for muscle activation. While noninvasive BCIs have the potential to improve the quality of lives of healthy and motor-impaired individuals, they currently have limited applications due to inconsistent performance and low degrees of freedom. In this study, we use deep learning (DL)-based decoders for online continuous pursuit (CP), a complex BCI task requiring the user to track an object in 2D space. We developed a labeling system to use CP data for supervised learning, trained DL-based decoders based on two architectures, including a newly proposed adaptation of the PointNet architecture, and evaluated the performance over several online sessions. We rigorously evaluated the DL-based decoders in a total of 28 human participants, and found that the DL-based models improved throughout the sessions as more training data became available and significantly outperformed a traditional BCI decoder by the last session. We also performed additional experiments to test an implementation of transfer learning by pretraining models on data from other subjects, and midsession training to reduce intersession variability. The results from these experiments showed that pretraining did not significantly improve performance, but updating the models' midsession may have some benefit. Overall, these findings support the use of DL-based decoders for improving BCI performance in complex tasks like CP, which can expand the potential applications of BCI devices and help to improve the quality of lives of healthy and motor-impaired individuals.
Collapse
Affiliation(s)
- Dylan Forenzo
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| | - Hao Zhu
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| | - Jenn Shanahan
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| | - Jaehyun Lim
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| | - Bin He
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| |
Collapse
|
16
|
Welter M, Lotte F. Ecological decoding of visual aesthetic preference with oscillatory electroencephalogram features-A mini-review. FRONTIERS IN NEUROERGONOMICS 2024; 5:1341790. [PMID: 38450005 PMCID: PMC10914990 DOI: 10.3389/fnrgo.2024.1341790] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Accepted: 01/19/2024] [Indexed: 03/08/2024]
Abstract
In today's digital information age, human exposure to visual artifacts has reached an unprecedented quasi-omnipresence. Some of these cultural artifacts are elevated to the status of artworks which indicates a special appreciation of these objects. For many persons, the perception of such artworks coincides with aesthetic experiences (AE) that can positively affect health and wellbeing. AEs are composed of complex cognitive and affective mental and physiological states. More profound scientific understanding of the neural dynamics behind AEs would allow the development of passive Brain-Computer-Interfaces (BCI) that offer personalized art presentation to improve AE without the necessity of explicit user feedback. However, previous empirical research in visual neuroaesthetics predominantly investigated functional Magnetic Resonance Imaging and Event-Related-Potentials correlates of AE in unnaturalistic laboratory conditions which might not be the best features for practical neuroaesthetic BCIs. Furthermore, AE has, until recently, largely been framed as the experience of beauty or pleasantness. Yet, these concepts do not encompass all types of AE. Thus, the scope of these concepts is too narrow to allow personalized and optimal art experience across individuals and cultures. This narrative mini-review summarizes the state-of-the-art in oscillatory Electroencephalography (EEG) based visual neuroaesthetics and paints a road map toward the development of ecologically valid neuroaesthetic passive BCI systems that could optimize AEs, as well as their beneficial consequences. We detail reported oscillatory EEG correlates of AEs, as well as machine learning approaches to classify AE. We also highlight current limitations in neuroaesthetics and suggest future directions to improve EEG decoding of AE.
Collapse
Affiliation(s)
- Marc Welter
- Inria Center at the University of Bordeaux/LaBRI, Talence, France
| | | |
Collapse
|
17
|
Wang X, Wang Y, Qi W, Kong D, Wang W. BrainGridNet: A two-branch depthwise CNN for decoding EEG-based multi-class motor imagery. Neural Netw 2024; 170:312-324. [PMID: 38006734 DOI: 10.1016/j.neunet.2023.11.037] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2023] [Revised: 11/14/2023] [Accepted: 11/16/2023] [Indexed: 11/27/2023]
Abstract
Brain-computer interfaces (BCIs) based on motor imagery (MI) enable the disabled to interact with the world through brain signals. To meet demands of real-time, stable, and diverse interactions, it is crucial to develop lightweight networks that can accurately and reliably decode multi-class MI tasks. In this paper, we introduce BrainGridNet, a convolutional neural network (CNN) framework that integrates two intersecting depthwise CNN branches with 3D electroencephalography (EEG) data to decode a five-class MI task. The BrainGridNet attains competitive results in both the time and frequency domains, with superior performance in the frequency domain. As a result, an accuracy of 80.26 percent and a kappa value of 0.753 are achieved by BrainGridNet, surpassing the state-of-the-art (SOTA) model. Additionally, BrainGridNet shows optimal computational efficiency, excels in decoding the most challenging subject, and maintains robust accuracy despite the random loss of 16 electrode signals. Finally, the visualizations demonstrate that BrainGridNet learns discriminative features and identifies critical brain regions and frequency bands corresponding to each MI class. The convergence of BrainGridNet's strong feature extraction capability, high decoding accuracy, steady decoding efficacy, and low computational costs renders it an appealing choice for facilitating the development of BCIs.
Collapse
Affiliation(s)
- Xingfu Wang
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Yu Wang
- Neural Computation and Brain Computer Interaction (NeuBCI) Research Center for Brain-inspired Intelligence, National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Wenxia Qi
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China
| | - Delin Kong
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China
| | - Wei Wang
- CAS Key Laboratory of Space Manufacturing Technology, Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China.
| |
Collapse
|
18
|
Kumar S, Alawieh H, Racz FS, Fakhreddine R, Millán JDR. Transfer learning promotes acquisition of individual BCI skills. PNAS NEXUS 2024; 3:pgae076. [PMID: 38426121 PMCID: PMC10903645 DOI: 10.1093/pnasnexus/pgae076] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Accepted: 02/05/2024] [Indexed: 03/02/2024]
Abstract
Subject training is crucial for acquiring brain-computer interface (BCI) control. Typically, this requires collecting user-specific calibration data due to high inter-subject neural variability that limits the usability of generic decoders. However, calibration is cumbersome and may produce inadequate data for building decoders, especially with naïve subjects. Here, we show that a decoder trained on the data of a single expert is readily transferrable to inexperienced users via domain adaptation techniques allowing calibration-free BCI training. We introduce two real-time frameworks, (i) Generic Recentering (GR) through unsupervised adaptation and (ii) Personally Assisted Recentering (PAR) that extends GR by employing supervised recalibration of the decoder parameters. We evaluated our frameworks on 18 healthy naïve subjects over five online sessions, who operated a customary synchronous bar task with continuous feedback and a more challenging car racing game with asynchronous control and discrete feedback. We show that along with improved task-oriented BCI performance in both tasks, our frameworks promoted subjects' ability to acquire individual BCI skills, as the initial neurophysiological control features of an expert subject evolved and became subject specific. Furthermore, those features were task-specific and were learned in parallel as participants practiced the two tasks in every session. Contrary to previous findings implying that supervised methods lead to improved online BCI control, we observed that longitudinal training coupled with unsupervised domain matching (GR) achieved similar performance to supervised recalibration (PAR). Therefore, our presented frameworks facilitate calibration-free BCIs and have immediate implications for broader populations-such as patients with neurological pathologies-who might struggle to provide suitable initial calibration data.
Collapse
Affiliation(s)
- Satyam Kumar
- Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
| | - Hussein Alawieh
- Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
| | - Frigyes Samuel Racz
- Department of Neurology, The University of Texas at Austin, Austin, TX 78712, USA
- Mulva Clinic for the Neurosciences, The University of Texas at Austin, Austin, TX 78712, USA
| | - Rawan Fakhreddine
- Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
| | - José del R Millán
- Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
- Department of Neurology, The University of Texas at Austin, Austin, TX 78712, USA
- Mulva Clinic for the Neurosciences, The University of Texas at Austin, Austin, TX 78712, USA
- Departement of Biomedical Engineering, The University of Texas at Austin, Austin, TX 78712, USA
| |
Collapse
|
19
|
Mei J, Luo R, Xu L, Zhao W, Wen S, Wang K, Xiao X, Meng J, Huang Y, Tang J, Cheng L, Xu M, Ming D. MetaBCI: An open-source platform for brain-computer interfaces. Comput Biol Med 2024; 168:107806. [PMID: 38081116 DOI: 10.1016/j.compbiomed.2023.107806] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Revised: 11/29/2023] [Accepted: 11/29/2023] [Indexed: 01/10/2024]
Abstract
BACKGROUND Recently, brain-computer interfaces (BCIs) have attracted worldwide attention for their great potential in clinical and real-life applications. To implement a complete BCI system, one must set up several links to translate the brain intent into computer commands. However, there is not an open-source software platform that can cover all links of the BCI chain. METHOD This study developed a one-stop open-source BCI software, namely MetaBCI, to facilitate the construction of a BCI system. MetaBCI is written in Python, and has the functions of stimulus presentation (Brainstim), data loading and processing (Brainda), and online information flow (Brainflow). This paper introduces the detailed information of MetaBCI and presents four typical application cases. RESULTS The results showed that MetaBCI was an extensible and feature-rich software platform for BCI research and application, which could effectively encode, decode, and feedback brain activities. CONCLUSIONS MetaBCI can greatly lower the BCI's technical threshold for BCI beginners and can save time and cost to build up a practical BCI system. The source code is available at https://github.com/TBC-TJU/MetaBCI, expecting new contributions from the BCI community.
Collapse
Affiliation(s)
- Jie Mei
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China; Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, 300072, People's Republic of China.
| | - Ruixin Luo
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China; Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, 300072, People's Republic of China.
| | - Lichao Xu
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China
| | - Wei Zhao
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China; Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, 300072, People's Republic of China
| | - Shengfu Wen
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China
| | - Kun Wang
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China; Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, 300392, People's Republic of China
| | - Xiaolin Xiao
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China; Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, 300072, People's Republic of China; Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, 300392, People's Republic of China
| | - Jiayuan Meng
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China; Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, 300072, People's Republic of China; Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, 300392, People's Republic of China
| | - Yongzhi Huang
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China; Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, 300392, People's Republic of China
| | - Jiabei Tang
- Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, 300392, People's Republic of China; Tiankai Suishi (Tianjin) Intelligence Ltd., Tianjin, 300192, People's Republic of China
| | - Longlong Cheng
- China Electronics Cloud Brain (Tianjin) Technology Co., Ltd., Tianjin, 300392, People's Republic of China
| | - Minpeng Xu
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China; Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, 300072, People's Republic of China; Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, 300392, People's Republic of China.
| | - Dong Ming
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, 300072, People's Republic of China; Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin, 300072, People's Republic of China; Haihe Laboratory of Brain-computer Interaction and Human-machine Integration, Tianjin, 300392, People's Republic of China
| |
Collapse
|
20
|
Luo R, Mai X, Meng J. Effect of motion state variability on error-related potentials during continuous feedback paradigms and their consequences for classification. J Neurosci Methods 2024; 401:109982. [PMID: 37839711 DOI: 10.1016/j.jneumeth.2023.109982] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Revised: 09/11/2023] [Accepted: 10/11/2023] [Indexed: 10/17/2023]
Abstract
BACKGROUND An erroneous motion would elicit the error-related potential (ErrP) when humans monitor the behavior of the external devices. This EEG modality has been largely applied to brain-computer interface in an active or passive manner with discrete visual feedback. However, the effect of variable motion state on ErrP morphology and classification performance raises concerns when the interaction is conducted with continuous visual feedback. NEW METHOD In the present study, we designed a cursor control experiment. Participants monitored a continuously moving cursor to reach the target on one side of the screen. Motion state varied multiple times with two factors: (1) motion direction and (2) motion speed. The effects of these two factors on the morphological characteristics and classification performance of ErrP were analyzed. Furthermore, an offline simulation was performed to evaluate the effectiveness of the proposed extended ErrP-decoder in resolving the interference by motion direction changes. RESULTS The statistical analyses revealed that motion direction and motion speed significantly influenced the amplitude of feedback-ERN and frontal-Pe components, while only motion direction significantly affected the classification performance. COMPARISON WITH EXISTING METHODS Significant deviation was found in ErrP detection utilizing classical correct-versus-erroneous event training. However, this bias can be alleviated by 16% by the extended ErrP-decoder. CONCLUSION The morphology and classification performance of ErrP signal can be affected by motion state variability during continuous feedback paradigms. The results enhance the comprehension of ErrP morphological components and shed light on the detection of BCI's error behavior in practical continuous control.
Collapse
Affiliation(s)
- Ruijie Luo
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Ximing Mai
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Jianjun Meng
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China; State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai, China.
| |
Collapse
|
21
|
Forenzo D, Liu Y, Kim J, Ding Y, Yoon T, He B. Integrating Simultaneous Motor Imagery and Spatial Attention for EEG-BCI Control. IEEE Trans Biomed Eng 2024; 71:282-294. [PMID: 37494151 PMCID: PMC10803074 DOI: 10.1109/tbme.2023.3298957] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/28/2023]
Abstract
OBJECTIVE EEG-based brain-computer interfaces (BCI) are non-invasive approaches for replacing or restoring motor functions in impaired patients, and direct brain-to-device communication in the general population. Motor imagery (MI) is one of the most used BCI paradigms, but its performance varies across individuals and certain users require substantial training to develop control. In this study, we propose to integrate a MI paradigm simultaneously with a recently proposed Overt Spatial Attention (OSA) paradigm, to accomplish BCI control. METHODS We evaluated a cohort of 25 human subjects' ability to control a virtual cursor in one- and two-dimensions over 5 BCI sessions. The subjects used 5 different BCI paradigms: MI alone, OSA alone, MI, and OSA simultaneously towards the same target (MI+OSA), and MI for one axis while OSA controls the other (MI/OSA and OSA/MI). RESULTS Our results show that MI+OSA reached the highest average online performance in 2D tasks at 49% Percent Valid Correct (PVC), and statistically outperforms both MI alone (42%) and OSA alone (45%). MI+OSA had a similar performance to each subject's best individual method between MI alone and OSA alone (50%) and 9 subjects reached their highest average BCI performance using MI+OSA. CONCLUSION Integrating MI and OSA leads to improved performance over both individual methods at the group level and is the best BCI paradigm option for some subjects. SIGNIFICANCE This work proposes a new BCI control paradigm that integrates two existing paradigms and demonstrates its value by showing that it can improve users' BCI performance.
Collapse
Affiliation(s)
- Dylan Forenzo
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| | - Yixuan Liu
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| | - Jeehyun Kim
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| | - Yidan Ding
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| | - Taehyung Yoon
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| | - Bin He
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| |
Collapse
|
22
|
Lin S, Jiang J, Huang K, Li L, He X, Du P, Wu Y, Liu J, Li X, Huang Z, Zhou Z, Yu Y, Gao J, Lei M, Wu H. Advanced Electrode Technologies for Noninvasive Brain-Computer Interfaces. ACS NANO 2023; 17:24487-24513. [PMID: 38064282 DOI: 10.1021/acsnano.3c06781] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/27/2023]
Abstract
Brain-computer interfaces (BCIs) have garnered significant attention in recent years due to their potential applications in medical, assistive, and communication technologies. Building on this, noninvasive BCIs stand out as they provide a safe and user-friendly method for interacting with the human brain. In this work, we provide a comprehensive overview of the latest developments and advancements in material, design, and application of noninvasive BCIs electrode technology. We also explore the challenges and limitations currently faced by noninvasive BCI electrode technology and sketch out the technological roadmap from three dimensions: Materials and Design; Performances; Mode and Function. We aim to unite research efforts within the field of noninvasive BCI electrode technology, focusing on the consolidation of shared goals and fostering integrated development strategies among a diverse array of multidisciplinary researchers.
Collapse
Affiliation(s)
- Sen Lin
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Jingjing Jiang
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Kai Huang
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
- State Key Laboratory of Information Photonics and Optical Communications and School of Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
| | - Lei Li
- National Engineering Research Center of Electric Vehicles, Beijing Institute of Technology, Beijing 100081, China
| | - Xian He
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
| | - Peng Du
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
| | - Yufeng Wu
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
| | - Junchen Liu
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
- State Key Laboratory of Information Photonics and Optical Communications and School of Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
| | - Xilin Li
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
- Advanced Institute for Brain and Intelligence, Guangxi University, Nanning 530004, China
| | - Zhibao Huang
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Zenan Zhou
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Yuanhang Yu
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Jiaxin Gao
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Ming Lei
- State Key Laboratory of Information Photonics and Optical Communications and School of Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
| | - Hui Wu
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
| |
Collapse
|
23
|
Jeong JH, Cho JH, Lee BH, Lee SW. Real-Time Deep Neurolinguistic Learning Enhances Noninvasive Neural Language Decoding for Brain-Machine Interaction. IEEE TRANSACTIONS ON CYBERNETICS 2023; 53:7469-7482. [PMID: 36251899 DOI: 10.1109/tcyb.2022.3211694] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Electroencephalogram (EEG)-based brain-machine interface (BMI) has been utilized to help patients regain motor function and has recently been validated for its use in healthy people because of its ability to directly decipher human intentions. In particular, neurolinguistic research using EEGs has been investigated as an intuitive and naturalistic communication tool between humans and machines. In this study, the human mind directly decoded the neural languages based on speech imagery using the proposed deep neurolinguistic learning. Through real-time experiments, we evaluated whether BMI-based cooperative tasks between multiple users could be accomplished using a variety of neural languages. We successfully demonstrated a BMI system that allows a variety of scenarios, such as essential activity, collaborative play, and emotional interaction. This outcome presents a novel BMI frontier that can interact at the level of human-like intelligence in real time and extends the boundaries of the communication paradigm.
Collapse
|
24
|
Wang W, Qi F, Wipf DP, Cai C, Yu T, Li Y, Zhang Y, Yu Z, Wu W. Sparse Bayesian Learning for End-to-End EEG Decoding. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:15632-15649. [PMID: 37506000 DOI: 10.1109/tpami.2023.3299568] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/30/2023]
Abstract
Decoding brain activity from non-invasive electroencephalography (EEG) is crucial for brain-computer interfaces (BCIs) and the study of brain disorders. Notably, end-to-end EEG decoding has gained widespread popularity in recent years owing to the remarkable advances in deep learning research. However, many EEG studies suffer from limited sample sizes, making it difficult for existing deep learning models to effectively generalize to highly noisy EEG data. To address this fundamental limitation, this paper proposes a novel end-to-end EEG decoding algorithm that utilizes a low-rank weight matrix to encode both spatio-temporal filters and the classifier, all optimized under a principled sparse Bayesian learning (SBL) framework. Importantly, this SBL framework also enables us to learn hyperparameters that optimally penalize the model in a Bayesian fashion. The proposed decoding algorithm is systematically benchmarked on five motor imagery BCI EEG datasets ( N=192) and an emotion recognition EEG dataset ( N=45), in comparison with several contemporary algorithms, including end-to-end deep-learning-based EEG decoding algorithms. The classification results demonstrate that our algorithm significantly outperforms the competing algorithms while yielding neurophysiologically meaningful spatio-temporal patterns. Our algorithm therefore advances the state-of-the-art by providing a novel EEG-tailored machine learning tool for decoding brain activity.
Collapse
|
25
|
Vukelić M, Bui M, Vorreuther A, Lingelbach K. Combining brain-computer interfaces with deep reinforcement learning for robot training: a feasibility study in a simulation environment. FRONTIERS IN NEUROERGONOMICS 2023; 4:1274730. [PMID: 38234482 PMCID: PMC10790930 DOI: 10.3389/fnrgo.2023.1274730] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Accepted: 10/31/2023] [Indexed: 01/19/2024]
Abstract
Deep reinforcement learning (RL) is used as a strategy to teach robot agents how to autonomously learn complex tasks. While sparsity is a natural way to define a reward in realistic robot scenarios, it provides poor learning signals for the agent, thus making the design of good reward functions challenging. To overcome this challenge learning from human feedback through an implicit brain-computer interface (BCI) is used. We combined a BCI with deep RL for robot training in a 3-D physical realistic simulation environment. In a first study, we compared the feasibility of different electroencephalography (EEG) systems (wet- vs. dry-based electrodes) and its application for automatic classification of perceived errors during a robot task with different machine learning models. In a second study, we compared the performance of the BCI-based deep RL training to feedback explicitly given by participants. Our findings from the first study indicate the use of a high-quality dry-based EEG-system can provide a robust and fast method for automatically assessing robot behavior using a sophisticated convolutional neural network machine learning model. The results of our second study prove that the implicit BCI-based deep RL version in combination with the dry EEG-system can significantly accelerate the learning process in a realistic 3-D robot simulation environment. Performance of the BCI-based trained deep RL model was even comparable to that achieved by the approach with explicit human feedback. Our findings emphasize the usage of BCI-based deep RL methods as a valid alternative in those human-robot applications where no access to cognitive demanding explicit human feedback is available.
Collapse
Affiliation(s)
- Mathias Vukelić
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart, Germany
| | - Michael Bui
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart, Germany
| | - Anna Vorreuther
- Applied Neurocognitive Systems, Institute of Human Factors and Technology Management (IAT), University of Stuttgart, Stuttgart, Germany
| | - Katharina Lingelbach
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart, Germany
| |
Collapse
|
26
|
Lin C, Zhang C, Xu J, Liu R, Leng Y, Fu C. Neural Correlation of EEG and Eye Movement in Natural Grasping Intention Estimation. IEEE Trans Neural Syst Rehabil Eng 2023; 31:4329-4337. [PMID: 37883284 DOI: 10.1109/tnsre.2023.3327907] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/28/2023]
Abstract
Decoding the user's natural grasp intent enhances the application of wearable robots, improving the daily lives of individuals with disabilities. Electroencephalogram (EEG) and eye movements are two natural representations when users generate grasp intent in their minds, with current studies decoding human intent by fusing EEG and eye movement signals. However, the neural correlation between these two signals remains unclear. Thus, this paper aims to explore the consistency between EEG and eye movement in natural grasping intention estimation. Specifically, six grasp intent pairs are decoded by combining feature vectors and utilizing the optimal classifier. Extensive experimental results indicate that the coupling between the EEG and eye movements intent patterns remains intact when the user generates a natural grasp intent, and concurrently, the EEG pattern is consistent with the eye movements pattern across the task pairs. Moreover, the findings reveal a solid connection between EEG and eye movements even when taking into account cortical EEG (originating from the visual cortex or motor cortex) and the presence of a suboptimal classifier. Overall, this work uncovers the coupling correlation between EEG and eye movements and provides a reference for intention estimation.
Collapse
|
27
|
Meng L, Jiang X, Huang J, Li W, Luo H, Wu D. User Identity Protection in EEG-Based Brain-Computer Interfaces. IEEE Trans Neural Syst Rehabil Eng 2023; 31:3576-3586. [PMID: 37651476 DOI: 10.1109/tnsre.2023.3310883] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/02/2023]
Abstract
A brain-computer interface (BCI) establishes a direct communication pathway between the brain and an external device. Electroencephalogram (EEG) is the most popular input signal in BCIs, due to its convenience and low cost. Most research on EEG-based BCIs focuses on the accurate decoding of EEG signals; however, EEG signals also contain rich private information, e.g., user identity, emotion, and so on, which should be protected. This paper first exposes a serious privacy problem in EEG-based BCIs, i.e., the user identity in EEG data can be easily learned so that different sessions of EEG data from the same user can be associated together to more reliably mine private information. To address this issue, we further propose two approaches to convert the original EEG data into identity-unlearnable EEG data, i.e., removing the user identity information while maintaining the good performance on the primary BCI task. Experiments on seven EEG datasets from five different BCI paradigms showed that on average the generated identity-unlearnable EEG data can reduce the user identification accuracy from 70.01% to at most 21.36%, greatly facilitating user privacy protection in EEG-based BCIs.
Collapse
|
28
|
Kosnoff J, Yu K, Liu C, He B. Transcranial Focused Ultrasound to V5 Enhances Human Visual Motion Brain-Computer Interface by Modulating Feature-Based Attention. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.09.04.556252. [PMID: 37732253 PMCID: PMC10508752 DOI: 10.1101/2023.09.04.556252] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/22/2023]
Abstract
Paralysis affects roughly 1 in 50 Americans. While there is no cure for the condition, brain-computer interfaces (BCI) can allow users to control a device with their mind, bypassing the paralyzed region. Non-invasive BCIs still have high error rates, which is hypothesized to be reduced with concurrent targeted neuromodulation. This study examines whether transcranial focused ultrasound (tFUS) modulation can improve BCI outcomes, and what the underlying mechanism of action might be through high-density electroencephalography (EEG)-based source imaging (ESI) analyses. V5-targeted tFUS significantly reduced the error for the BCI speller task. ESI analyses showed significantly increased theta activity in the tFUS condition at both V5 and downstream the dorsal visual processing pathway. Correlation analysis indicates that the dorsal processing pathway connection was preserved during tFUS stimulation, whereas extraneous connections were severed. These results suggest that V5-targeted tFUS' mechanism of action is to raise the brain's feature-based attention to visual motion.
Collapse
|
29
|
Luo J, Wang Y, Xia S, Lu N, Ren X, Shi Z, Hei X. A shallow mirror transformer for subject-independent motor imagery BCI. Comput Biol Med 2023; 164:107254. [PMID: 37499295 DOI: 10.1016/j.compbiomed.2023.107254] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/02/2023] [Revised: 06/28/2023] [Accepted: 07/07/2023] [Indexed: 07/29/2023]
Abstract
OBJECTIVE Motor imagery BCI plays an increasingly important role in motor disorders rehabilitation. However, the position and duration of the discriminative segment in an EEG trial vary from subject to subject and even trial to trial, and this leads to poor performance of subject-independent motor imagery classification. Thus, determining how to detect and utilize the discriminative signal segments is crucial for improving the performance of subject-independent motor imagery BCI. APPROACH In this paper, a shallow mirror transformer is proposed for subject-independent motor imagery EEG classification. Specifically, a multihead self-attention layer with a global receptive field is employed to detect and utilize the discriminative segment from the entire input EEG trial. Furthermore, the mirror EEG signal and the mirror network structure are constructed to improve the classification precision based on ensemble learning. Finally, the subject-independent setup was used to evaluate the shallow mirror transformer on motor imagery EEG signals from subjects existing in the training set and new subjects. MAIN RESULTS The experiments results on BCI Competition IV datasets 2a and 2b and the OpenBMI dataset demonstrated the promising effectiveness of the proposed shallow mirror transformer. The shallow mirror transformer obtained average accuracies of 74.48% and 76.1% for new subjects and existing subjects, respectively, which were highest among the compared state-of-the-art methods. In addition, visualization of the attention score showed the ability of discriminative EEG segment detection. This paper demonstrated that multihead self-attention is effective in capturing global EEG signal information in motor imagery classification. SIGNIFICANCE This study provides an effective model based on a multihead self-attention layer for subject-independent motor imagery-based BCIs. To the best of our knowledge, this is the shallowest transformer model available, in which a small number of parameters promotes the performance in motor imagery EEG classification for such a small sample problem.
Collapse
Affiliation(s)
- Jing Luo
- Shaanxi Key Laboratory for Network Computing and Security Technology and Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China.
| | - Yaojie Wang
- Shaanxi Key Laboratory for Network Computing and Security Technology and Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China
| | - Shuxiang Xia
- Shaanxi Key Laboratory for Network Computing and Security Technology and Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China
| | - Na Lu
- State Key Laboratory for Manufacturing Systems Engineering, Systems Engineering Institute, Xi'an Jiaotong University, Xi'an, Shaanxi, China
| | - Xiaoyong Ren
- Department of Otolaryngology Head and Neck Surgery & Center of Sleep Medicine, The Second Affiliated Hospital of Xi'an Jiaotong University, Xi'an, Shaanxi, China
| | - Zhenghao Shi
- Shaanxi Key Laboratory for Network Computing and Security Technology and Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China
| | - Xinhong Hei
- Shaanxi Key Laboratory for Network Computing and Security Technology and Human-Machine Integration Intelligent Robot Shaanxi University Engineering Research Center, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, Shaanxi, China
| |
Collapse
|
30
|
Zheng H, Niu L, Qiu W, Liang D, Long X, Li G, Liu Z, Meng L. The Emergence of Functional Ultrasound for Noninvasive Brain-Computer Interface. RESEARCH (WASHINGTON, D.C.) 2023; 6:0200. [PMID: 37588619 PMCID: PMC10427153 DOI: 10.34133/research.0200] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/13/2023] [Accepted: 07/04/2023] [Indexed: 08/18/2023]
Abstract
A noninvasive brain-computer interface is a central task in the comprehensive analysis and understanding of the brain and is an important challenge in international brain-science research. Current implanted brain-computer interfaces are cranial and invasive, which considerably limits their applications. The development of new noninvasive reading and writing technologies will advance substantial innovations and breakthroughs in the field of brain-computer interfaces. Here, we review the theory and development of the ultrasound brain functional imaging and its applications. Furthermore, we introduce latest advancements in ultrasound brain modulation and its applications in rodents, primates, and human; its mechanism and closed-loop ultrasound neuromodulation based on electroencephalograph are also presented. Finally, high-frequency acoustic noninvasive brain-computer interface is prospected based on ultrasound super-resolution imaging and acoustic tweezers.
Collapse
Affiliation(s)
- Hairong Zheng
- Institute of Biomedical and Health Engineering, Shenzhen Institutes of Advanced Technology,
Chinese Academy of Sciences, Shenzhen, 518055, China
| | - Lili Niu
- Institute of Biomedical and Health Engineering, Shenzhen Institutes of Advanced Technology,
Chinese Academy of Sciences, Shenzhen, 518055, China
| | - Weibao Qiu
- Institute of Biomedical and Health Engineering, Shenzhen Institutes of Advanced Technology,
Chinese Academy of Sciences, Shenzhen, 518055, China
| | - Dong Liang
- Institute of Biomedical and Health Engineering, Shenzhen Institutes of Advanced Technology,
Chinese Academy of Sciences, Shenzhen, 518055, China
| | - Xiaojing Long
- Institute of Biomedical and Health Engineering, Shenzhen Institutes of Advanced Technology,
Chinese Academy of Sciences, Shenzhen, 518055, China
| | - Guanglin Li
- Shenzhen Institute of Advanced Integration Technology, Chinese Academy of Sciences and The Chinese University of Hong Kong, Shenzhen, 518055, China
| | - Zhiyuan Liu
- Shenzhen Institute of Advanced Integration Technology, Chinese Academy of Sciences and The Chinese University of Hong Kong, Shenzhen, 518055, China
| | - Long Meng
- Institute of Biomedical and Health Engineering, Shenzhen Institutes of Advanced Technology,
Chinese Academy of Sciences, Shenzhen, 518055, China
| |
Collapse
|
31
|
Zhou Y, Yu T, Gao W, Huang W, Lu Z, Huang Q, Li Y. Shared Three-Dimensional Robotic Arm Control Based on Asynchronous BCI and Computer Vision. IEEE Trans Neural Syst Rehabil Eng 2023; 31:3163-3175. [PMID: 37498753 DOI: 10.1109/tnsre.2023.3299350] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/29/2023]
Abstract
OBJECTIVE A brain-computer interface (BCI) can be used to translate neuronal activity into commands to control external devices. However, using noninvasive BCI to control a robotic arm for movements in three-dimensional (3D) environments and accomplish complicated daily tasks, such as grasping and drinking, remains a challenge. APPROACH In this study, a shared robotic arm control system based on hybrid asynchronous BCI and computer vision was presented. The BCI model, which combines steady-state visual evoked potentials (SSVEPs) and blink-related electrooculography (EOG) signals, allows users to freely choose from fifteen commands in an asynchronous mode corresponding to robot actions in a 3D workspace and reach targets with a wide movement range, while computer vision can identify objects and assist a robotic arm in completing more precise tasks, such as grasping a target automatically. RESULTS Ten subjects participated in the experiments and achieved an average accuracy of more than 92% and a high trajectory efficiency for robot movement. All subjects were able to perform the reach-grasp-drink tasks successfully using the proposed shared control method, with fewer error commands and shorter completion time than with direct BCI control. SIGNIFICANCE Our results demonstrated the feasibility and efficiency of generating practical multidimensional control of an intuitive robotic arm by merging hybrid asynchronous BCI and computer vision-based recognition.
Collapse
|
32
|
Ai J, Meng J, Mai X, Zhu X. BCI Control of a Robotic Arm Based on SSVEP With Moving Stimuli for Reach and Grasp Tasks. IEEE J Biomed Health Inform 2023; 27:3818-3829. [PMID: 37200132 DOI: 10.1109/jbhi.2023.3277612] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/20/2023]
Abstract
Brain-computer interface (BCI) provides a novel technology for patients and healthy human subjects to control a robotic arm. Currently, BCI control of a robotic arm to complete the reaching and grasping tasks in an unstructured environment is still challenging because the current BCI technology does not meet the requirement of manipulating a multi-degree robotic arm accurately and robustly. BCI based on steady-state visual evoked potential (SSVEP) could output a high information transfer rate; however, the conventional SSVEP paradigm failed to control a robotic arm to move continuously and accurately because the users have to switch their gaze between the flickering stimuli and the target frequently. This study proposed a novel SSVEP paradigm in which the flickering stimuli were attached to the robotic arm's gripper and moved with it. First, an offline experiment was designed to investigate the effects of moving flickering stimuli on the SSVEP's responses and decoding accuracy. After that, contrast experiments were conducted, and twelve subjects were recruited to participate in a robotic arm control experiment using both the paradigm one (P1, with moving flickering stimuli) and the paradigm two (P2, conventional fixed flickering stimuli) using a block randomization design to balance their sequences. Double blinks were used to trigger the grasping action asynchronously whenever the subjects were confident that the position of the robotic arm's gripper was accurate enough. Experimental results showed that the paradigm P1 with moving flickering stimuli provided a much better control performance than the conventional paradigm P2 in completing a reaching and grasping task in an unstructured environment. Subjects' subjective feedback scored by a NASA-TLX mental workload scale also corroborated the BCI control performance. The results of this study suggest that the proposed control interface based on SSVEP BCI provides a better solution for robotic arm control to complete the accurate reaching and grasping tasks.
Collapse
|
33
|
Xu B, Liu D, Xue M, Miao M, Hu C, Song A. Continuous shared control of a mobile robot with brain-computer interface and autonomous navigation for daily assistance. Comput Struct Biotechnol J 2023; 22:3-16. [PMID: 37600142 PMCID: PMC10433001 DOI: 10.1016/j.csbj.2023.07.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2023] [Revised: 07/04/2023] [Accepted: 07/22/2023] [Indexed: 08/22/2023] Open
Abstract
Although the electroencephalography (EEG) based brain-computer interface (BCI) has been successfully developed for rehabilitation and assistance, it is still challenging to achieve continuous control of a brain-actuated mobile robot system. In this study, we propose a continuous shared control strategy combining continuous BCI and autonomous navigation for a mobile robot system. The weight of shared control is designed to dynamically adjust the fusion of continuous BCI control and autonomous navigation. During this process, the system uses the visual-based simultaneous localization and mapping (SLAM) method to construct environmental maps. After obtaining the global optimal path, the system utilizes the brain-based shared control dynamic window approach (BSC-DWA) to evaluate safe and reachable trajectories while considering shared control velocity. Eight subjects participated in two-stage training, and six of these eight subjects participated in online shared control experiments. The training results demonstrated that naïve subjects could achieve continuous control performance with an average percent valid correct rate of approximately 97 % and an average total correct rate of over 80 %. The results of online shared control experiments showed that all of the subjects could complete navigation tasks in an unknown corridor with continuous shared control. Therefore, our experiments verified the feasibility and effectiveness of the proposed system combining continuous BCI, shared control, autonomous navigation, and visual SLAM. The proposed continuous shared control framework shows great promise in BCI-driven tasks, especially navigation tasks for brain-driven assistive mobile robots and wheelchairs in daily applications.
Collapse
Affiliation(s)
- Baoguo Xu
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| | - Deping Liu
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| | - Muhui Xue
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| | - Minmin Miao
- School of Information Engineering, Huzhou University, Huzhou 313000, China
| | - Cong Hu
- Guangxi Key Laboratory of Automatic Detecting Technology and Instruments, Guilin University of Electronic Technology, Guilin 541004, China
| | - Aiguo Song
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| |
Collapse
|
34
|
Kokorin K, Mu J, John SE, Grayden DB. Predictive Shared Control of Robotic Arms Using Simulated Brain-Computer Interface Inputs. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-5. [PMID: 38082602 DOI: 10.1109/embc40787.2023.10340222] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Low decoding accuracy makes brain-computer interface (BCI) control of a robotic arm difficult. Shared control (SC) can overcome limitations of a BCI by leveraging external sensor data and generating commands to assist the user. Our study explored whether reaching targets with a robot end-effector was easier using SC rather than direct control (DC). We simulated a motor imagery BCI using a joystick with noise introduced to explicitly control interface accuracy to be 65% or 79%. Compared to DC, our prediction-based implementation of SC led to a significant reduction in the trajectory length of successful reaches for 4 (3) out of 5 targets using the 65% (79%) accurate interface, with failure rates being equivalent to DC for 2 (1) out of 5 targets. Therefore, this implementation of SC is likely to improve reaching efficiency but at the cost of more failures. Additionally, the NASA Task Load Index results suggest SC reduced user workload.Clinical relevance-Shared control can minimise the impact of BCI decoder errors on robot motion, making robotic arm control using noninvasive BCIs more viable.
Collapse
|
35
|
Peksa J, Mamchur D. State-of-the-Art on Brain-Computer Interface Technology. SENSORS (BASEL, SWITZERLAND) 2023; 23:6001. [PMID: 37447849 DOI: 10.3390/s23136001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/22/2023] [Revised: 06/23/2023] [Accepted: 06/26/2023] [Indexed: 07/15/2023]
Abstract
This paper provides a comprehensive overview of the state-of-the-art in brain-computer interfaces (BCI). It begins by providing an introduction to BCIs, describing their main operation principles and most widely used platforms. The paper then examines the various components of a BCI system, such as hardware, software, and signal processing algorithms. Finally, it looks at current trends in research related to BCI use for medical, educational, and other purposes, as well as potential future applications of this technology. The paper concludes by highlighting some key challenges that still need to be addressed before widespread adoption can occur. By presenting an up-to-date assessment of the state-of-the-art in BCI technology, this paper will provide valuable insight into where this field is heading in terms of progress and innovation.
Collapse
Affiliation(s)
- Janis Peksa
- Department of Information Technologies, Turiba University, Graudu Street 68, LV-1058 Riga, Latvia
- Institute of Information Technology, Riga Technical University, Kalku Street 1, LV-1658 Riga, Latvia
| | - Dmytro Mamchur
- Department of Information Technologies, Turiba University, Graudu Street 68, LV-1058 Riga, Latvia
- Computer Engineering and Electronics Department, Kremenchuk Mykhailo Ostrohradskyi National University, Pershotravneva 20, 39600 Kremenchuk, Ukraine
| |
Collapse
|
36
|
Yang S, Li M, Wang J, Shi Z, He B, Xie J, Xu G. A low-cost and portable wrist exoskeleton using EEG-sEMG combined strategy for prolonged active rehabilitation. Front Neurorobot 2023; 17:1161187. [PMID: 37292117 PMCID: PMC10244749 DOI: 10.3389/fnbot.2023.1161187] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2023] [Accepted: 05/03/2023] [Indexed: 06/10/2023] Open
Abstract
Introduction Hemiparesis is a common consequence of stroke that severely impacts the life quality of the patients. Active training is a key factor in achieving optimal neural recovery, but current systems for wrist rehabilitation present challenges in terms of portability, cost, and the potential for muscle fatigue during prolonged use. Methods To address these challenges, this paper proposes a low-cost, portable wrist rehabilitation system with a control strategy that combines surface electromyogram (sEMG) and electroencephalogram (EEG) signals to encourage patients to engage in consecutive, spontaneous rehabilitation sessions. In addition, a detection method for muscle fatigue based on the Boruta algorithm and a post-processing layer are proposed, allowing for the switch between sEMG and EEG modes when muscle fatigue occurs. Results This method significantly improves accuracy of fatigue detection from 4.90 to 10.49% for four distinct wrist motions, while the Boruta algorithm selects the most essential features and stabilizes the effects of post-processing. The paper also presents an alternative control mode that employs EEG signals to maintain active control, achieving an accuracy of approximately 80% in detecting motion intention. Discussion For the occurrence of muscle fatigue during long term rehabilitation training, the proposed system presents a promising approach to addressing the limitations of existing wrist rehabilitation systems.
Collapse
|
37
|
Srisrisawang N, Müller-Putz GR. Transfer Learning in Trajectory Decoding: Sensor or Source Space? SENSORS (BASEL, SWITZERLAND) 2023; 23:3593. [PMID: 37050653 PMCID: PMC10098869 DOI: 10.3390/s23073593] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/19/2023] [Revised: 03/08/2023] [Accepted: 03/28/2023] [Indexed: 06/19/2023]
Abstract
In this study, across-participant and across-session transfer learning was investigated to minimize the calibration time of the brain-computer interface (BCI) system in the context of continuous hand trajectory decoding. We reanalyzed data from a study with 10 able-bodied participants across three sessions. A leave-one-participant-out (LOPO) model was utilized as a starting model. Recursive exponentially weighted partial least squares regression (REW-PLS) was employed to overcome the memory limitation due to the large pool of training data. We considered four scenarios: generalized with no update (Gen), generalized with cumulative update (GenC), and individual models with cumulative (IndC) and non-cumulative (Ind) updates, with each one trained with sensor-space features or source-space features. The decoding performance in generalized models (Gen and GenC) was lower than the chance level. In individual models, the cumulative update (IndC) showed no significant improvement over the non-cumulative model (Ind). The performance showed the decoder's incapability to generalize across participants and sessions in this task. The results suggested that the best correlation could be achieved with the sensor-space individual model, despite additional anatomical information in the source-space features. The decoding pattern showed a more localized pattern around the precuneus over three sessions in Ind models.
Collapse
Affiliation(s)
- Nitikorn Srisrisawang
- Institute of Neural Engineering, Graz University of Technology, Stremayrgasse 16/IV, 8010 Graz, Austria
| | - Gernot R. Müller-Putz
- Institute of Neural Engineering, Graz University of Technology, Stremayrgasse 16/IV, 8010 Graz, Austria
- BioTechMed Graz, 8010 Graz, Austria
| |
Collapse
|
38
|
Guo R, Lin Y, Luo X, Gao X, Zhang S. A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment. Front Neurorobot 2023; 17:1146415. [PMID: 37051328 PMCID: PMC10083338 DOI: 10.3389/fnbot.2023.1146415] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2023] [Accepted: 03/10/2023] [Indexed: 03/29/2023] Open
Abstract
At present, single-modal brain-computer interface (BCI) still has limitations in practical application, such as low flexibility, poor autonomy, and easy fatigue for subjects. This study developed an asynchronous robotic arm control system based on steady-state visual evoked potentials (SSVEP) and eye-tracking in virtual reality (VR) environment, including simultaneous and sequential modes. For simultaneous mode, target classification was realized by decision-level fusion of electroencephalography (EEG) and eye-gaze. The stimulus duration for each subject was non-fixed, which was determined by an adjustable window method. Subjects could autonomously control the start and stop of the system using triple blink and eye closure, respectively. For sequential mode, no calibration was conducted before operation. First, subjects’ gaze area was obtained through eye-gaze, and then only few stimulus blocks began to flicker. Next, target classification was determined using EEG. Additionally, subjects could reject false triggering commands using eye closure. In this study, the system effectiveness was verified through offline experiment and online robotic-arm grasping experiment. Twenty subjects participated in offline experiment. For simultaneous mode, average ACC and ITR at the stimulus duration of 0.9 s were 90.50% and 60.02 bits/min, respectively. For sequential mode, average ACC and ITR at the stimulus duration of 1.4 s were 90.47% and 45.38 bits/min, respectively. Fifteen subjects successfully completed the online tasks of grabbing balls in both modes, and most subjects preferred the sequential mode. The proposed hybrid brain-computer interface (h-BCI) system could increase autonomy, reduce visual fatigue, meet individual needs, and improve the efficiency of the system.
Collapse
Affiliation(s)
- Rongxiao Guo
- School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, China
| | - Yanfei Lin
- School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, China
- *Correspondence: Yanfei Lin,
| | - Xi Luo
- School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, China
| | - Xiaorong Gao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China
| | - Shangen Zhang
- School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China
| |
Collapse
|
39
|
Śliwowski M, Martin M, Souloumiac A, Blanchart P, Aksenova T. Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performance. Front Hum Neurosci 2023; 17:1111645. [PMID: 37007675 PMCID: PMC10061076 DOI: 10.3389/fnhum.2023.1111645] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Accepted: 02/27/2023] [Indexed: 03/18/2023] Open
Abstract
IntroductionIn brain-computer interfaces (BCI) research, recording data is time-consuming and expensive, which limits access to big datasets. This may influence the BCI system performance as machine learning methods depend strongly on the training dataset size. Important questions arise: taking into account neuronal signal characteristics (e.g., non-stationarity), can we achieve higher decoding performance with more data to train decoders? What is the perspective for further improvement with time in the case of long-term BCI studies? In this study, we investigated the impact of long-term recordings on motor imagery decoding from two main perspectives: model requirements regarding dataset size and potential for patient adaptation.MethodsWe evaluated the multilinear model and two deep learning (DL) models on a long-term BCI & Tetraplegia (ClinicalTrials.gov identifier: NCT02550522) clinical trial dataset containing 43 sessions of ECoG recordings performed with a tetraplegic patient. In the experiment, a participant executed 3D virtual hand translation using motor imagery patterns. We designed multiple computational experiments in which training datasets were increased or translated to investigate the relationship between models' performance and different factors influencing recordings.ResultsOur results showed that DL decoders showed similar requirements regarding the dataset size compared to the multilinear model while demonstrating higher decoding performance. Moreover, high decoding performance was obtained with relatively small datasets recorded later in the experiment, suggesting motor imagery patterns improvement and patient adaptation during the long-term experiment. Finally, we proposed UMAP embeddings and local intrinsic dimensionality as a way to visualize the data and potentially evaluate data quality.DiscussionDL-based decoding is a prospective approach in BCI which may be efficiently applied with real-life dataset size. Patient-decoder co-adaptation is an important factor to consider in long-term clinical BCI.
Collapse
Affiliation(s)
- Maciej Śliwowski
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
- Université Paris-Saclay, CEA, List, Palaiseau, France
| | - Matthieu Martin
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
| | | | | | - Tetiana Aksenova
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
- *Correspondence: Tetiana Aksenova
| |
Collapse
|
40
|
Jiang N, Chen C, He J, Meng J, Pan L, Su S, Zhu X. Bio-robotics research for non-invasive myoelectric neural interfaces for upper-limb prosthetic control: a 10-year perspective review. Natl Sci Rev 2023; 10:nwad048. [PMID: 37056442 PMCID: PMC10089583 DOI: 10.1093/nsr/nwad048] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Revised: 01/01/2023] [Accepted: 02/07/2023] [Indexed: 04/05/2023] Open
Abstract
ABSTRACT
A decade ago, a group of researchers from academia and industry identified a dichotomy between the industrial and academic state-of-the-art in upper-limb prosthesis control, a widely used bio-robotics application. They proposed that four key technical challenges, if addressed, could bridge this gap and translate academic research into clinically and commercially viable products. These challenges are unintuitive control schemes, lack of sensory feedback, poor robustness and single sensor modality. Here, we provide a perspective review on the research effort that occurred in the last decade, aiming at addressing these challenges. In addition, we discuss three research areas essential to the recent development in upper-limb prosthetic control research but were not envisioned in the review 10 years ago: deep learning methods, surface electromyogram decomposition and open-source databases. To conclude the review, we provide an outlook into the near future of the research and development in upper-limb prosthetic control and beyond.
Collapse
Affiliation(s)
| | - Chen Chen
- State Key Laboratory of Mechanical System and Vibration, and Institute of Robotics, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Jiayuan He
- National Clinical Research Center for Geriatrics, West China Hospital, and Med-X Center for Manufacturing, Sichuan University, Chengdu 610041, China
| | - Jianjun Meng
- State Key Laboratory of Mechanical System and Vibration, and Institute of Robotics, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Lizhi Pan
- Key Laboratory of Mechanism Theory and Equipment Design of Ministry of Education, School of Mechanical Engineering, Tianjin University, Tianjin 300350, China
| | - Shiyong Su
- Institute of Neuroscience, Université Catholique Louvain, Brussel B-1348, Belgium
| | - Xiangyang Zhu
- State Key Laboratory of Mechanical System and Vibration, and Institute of Robotics, Shanghai Jiao Tong University, Shanghai 200240, China
| |
Collapse
|
41
|
Forenzo D, Liu Y, Kim J, Ding Y, Yoon T, He B. Integrating simultaneous motor imagery and spatial attention for EEG-BCI control. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.02.20.529307. [PMID: 36865207 PMCID: PMC9980047 DOI: 10.1101/2023.02.20.529307] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/25/2023]
Abstract
OBJECTIVE EEG-based brain-computer interfaces (BCI) are non-invasive approaches for replacing or restoring motor functions in impaired patients, and direct brain-to-device communication in the general population. Motor imagery (MI) is one of the most used BCI paradigms, but its performance varies across individuals and certain users require substantial training to develop control. In this study, we propose to integrate a MI paradigm simultaneously with a recently proposed Overt Spatial Attention (OSA) paradigm, to accomplish BCI control. METHODS We evaluated a cohort of 25 human subjects' ability to control a virtual cursor in one- and two-dimensions over 5 BCI sessions. The subjects used 5 different BCI paradigms: MI alone, OSA alone, MI and OSA simultaneously towards the same target (MI+OSA), and MI for one axis while OSA controls the other (MI/OSA and OSA/MI). RESULTS Our results show that MI+OSA reached the highest average online performance in 2D tasks at 49% Percent Valid Correct (PVC), statistically outperforms MI alone (42%), and was higher, but not statistically significant, than OSA alone (45%). MI+OSA had a similar performance to each subject's best individual method between MI alone and OSA alone (50%) and 9 subjects reached their highest average BCI performance using MI+OSA. CONCLUSION Integrating MI and OSA leads to improved performance over MI alone at the group level and is the best BCI paradigm option for some subjects. SIGNIFICANCE This work proposes a new BCI control paradigm that integrates two existing paradigms and demonstrates its value by showing that it can improve users' BCI performance.
Collapse
|
42
|
Cowley SJ, Gahrn-Andersen R. How systemic cognition enables epistemic engineering. Front Artif Intell 2023; 5:960384. [PMID: 36825254 PMCID: PMC9941702 DOI: 10.3389/frai.2022.960384] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Accepted: 12/30/2022] [Indexed: 02/09/2023] Open
Abstract
Epistemic engineering arises as systems and their parts develop functionality that is construed as valid knowledge. By hypothesis, epistemic engineering is a basic evolutionary principle. It ensures that not only living systems identify the differences that make differences but also ensure that distributed control enables them to construct epistemic change. In tracking such outcomes in human life, we stress that humans act within poly-centered, distributed systems. Similar to how people can act as inert parts of a system, they also actively bring forth intents and vicariant effects. Human cognitive agents use the systemic function to construct epistemic novelties. In the illustration, we used a published experimental study of a cyborg cockroach to consider how an evoneered system enables a human subject to perform as an adaptor with some "thought control" over the animal. Within a wide system, brains enable the techniques to arise ex novo as they attune to the dictates of a device. Human parts act as adaptors that simplify the task. In scaling up, we turn to a case of organizational cognition. We track how adaptor functions spread when drone-based data are brought to the maintenance department of a Danish utility company. While pivoting on how system operators combine experience with the use of software, their expertise sets off epistemically engineered results across the company and beyond. Vicariant effects emerge under the poly-centered control of brains, persons, equipment, and institutional wholes. As a part of culture, epistemic engineering works by reducing entropy.
Collapse
Affiliation(s)
- Stephen J. Cowley
- Department of Language and Communication, University of Southern Denmark, Slagelse, Denmark
| | | |
Collapse
|
43
|
Zhang Z, Li D, Zhao Y, Fan Z, Xiang J, Wang X, Cui X. A flexible speller based on time-space frequency conversion SSVEP stimulation paradigm under dry electrode. Front Comput Neurosci 2023; 17:1101726. [PMID: 36817318 PMCID: PMC9929550 DOI: 10.3389/fncom.2023.1101726] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Accepted: 01/10/2023] [Indexed: 02/04/2023] Open
Abstract
Introduction Speller is the best way to express the performance of the brain-computer interface (BCI) paradigm. Due to its advantages of short analysis time and high accuracy, the SSVEP paradigm has been widely used in the BCI speller system based on the wet electrode. It is widely known that the wet electrode operation is cumbersome and that the subjects have a poor experience. In addition, in the asynchronous SSVEP system based on threshold analysis, the system flickers continuously from the beginning to the end of the experiment, which leads to visual fatigue. The dry electrode has a simple operation and provides a comfortable experience for subjects. The EOG signal can avoid the stimulation of SSVEP for a long time, thus reducing fatigue. Methods This study first designed the brain-controlled switch based on continuous blinking EOG signal and SSVEP signal to improve the flexibility of the BCI speller. Second, in order to increase the number of speller instructions, we designed the time-space frequency conversion (TSFC) SSVEP stimulus paradigm by constantly changing the time and space frequency of SSVEP sub-stimulus blocks, and designed a speller in a dry electrode environment. Results Seven subjects participated and completed the experiments. The results showed that the accuracy of the brain-controlled switch designed in this study was up to 94.64%, and all the subjects could use the speller flexibly. The designed 60-character speller based on the TSFC-SSVEP stimulus paradigm has an accuracy rate of 90.18% and an information transmission rate (ITR) of 117.05 bits/min. All subjects can output the specified characters in a short time. Discussion This study designed and implemented a multi-instruction SSVEP speller based on dry electrode. Through the combination of EOG and SSVEP signals, the speller can be flexibly controlled. The frequency of SSVEP stimulation sub-block is recoded in time and space by TSFC-SSVEP stimulation paradigm, which greatly improves the number of output instructions of BCI system in dry electrode environment. This work only uses FBCCA algorithm to test the stimulus paradigm, which requires a long stimulus time. In the future, we will use trained algorithms to study stimulus paradigm to improve its overall performance.
Collapse
|
44
|
Hu H, Yue K, Guo M, Lu K, Liu Y. Subject Separation Network for Reducing Calibration Time of MI-Based BCI. Brain Sci 2023; 13:brainsci13020221. [PMID: 36831764 PMCID: PMC9954620 DOI: 10.3390/brainsci13020221] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2022] [Revised: 01/11/2023] [Accepted: 01/20/2023] [Indexed: 02/03/2023] Open
Abstract
Motor imagery brain-computer interface (MI-based BCIs) have demonstrated great potential in various applications. However, to well generalize classifiers to new subjects, a time-consuming calibration process is necessary due to high inter-subject variabilities of EEG signals. This process is costly and tedious, hindering the further expansion of MI-based BCIs outside of the laboratory. To reduce the calibration time of MI-based BCIs, we propose a novel domain adaptation framework that adapts multiple source subjects' labeled data to the unseen trials of target subjects. Firstly, we train one Subject Separation Network(SSN) for each of the source subjects in the dataset. Based on adversarial domain adaptation, a shared encoder is constructed to learn similar representations for both domains. Secondly, to model the factors that cause subject variabilities and eliminate the correlated noise existing in common feature space, private feature spaces orthogonal to the shared counterpart are learned for each subject. We use a shared decoder to validate that the model is actually learning from task-relevant neurophysiological information. At last, an ensemble classifier is built by the integration of the SSNs using the information extracted from each subject's task-relevant characteristics. To quantify the efficacy of the framework, we analyze the accuracy-calibration cost trade-off in MI-based BCIs, and theoretically guarantee a generalization bound on the target error. Visualizations of the transformed features illustrate the effectiveness of domain adaptation. The experimental results on the BCI Competition IV-IIa dataset prove the effectiveness of the proposed framework compared with multiple classification methods. We infer from our results that users could learn to control MI-based BCIs without a heavy calibration process. Our study further shows how to design and train Neural Networks to decode task-related information from different subjects and highlights the potential of deep learning methods for inter-subject EEG decoding.
Collapse
Affiliation(s)
- Haochen Hu
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
| | - Kang Yue
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
- Institute of Software, Chinese Academy of Sciences, Beijing 100045, China
| | - Mei Guo
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
| | - Kai Lu
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
| | - Yue Liu
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China
- Correspondence:
| |
Collapse
|
45
|
Han J, Xu M, Xiao X, Yi W, Jung TP, Ming D. A high-speed hybrid brain-computer interface with more than 200 targets. J Neural Eng 2023; 20:016025. [PMID: 36608342 DOI: 10.1088/1741-2552/acb105] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Accepted: 01/06/2023] [Indexed: 01/07/2023]
Abstract
Objective. Brain-computer interfaces (BCIs) have recently made significant strides in expanding their instruction set, which has attracted wide attention from researchers. The number of targets and commands is a key indicator of how well BCIs can decode the brain's intentions. No studies have reported a BCI system with over 200 targets.Approach. This study developed the first high-speed BCI system with up to 216 targets that were encoded by a combination of electroencephalography features, including P300, motion visual evoked potential (mVEP), and steady-state visual evoked potential (SSVEP). Specifically, the hybrid BCI paradigm used the time-frequency division multiple access strategy to elaborately tag targets with P300 and mVEP of different time windows, along with SSVEP of different frequencies. The hybrid features were then decoded by task-discriminant component analysis and linear discriminant analysis. Ten subjects participated in the offline and online cued-guided spelling experiments. Other ten subjects took part in online free-spelling experiments.Main results.The offline results showed that the mVEP and P300 components were prominent in the central, parietal, and occipital regions, while the most distinct SSVEP feature was in the occipital region. The online cued-guided spelling and free-spelling results showed that the proposed BCI system achieved an average accuracy of 85.37% ± 7.49% and 86.00% ± 5.98% for the 216-target classification, resulting in an average information transfer rate (ITR) of 302.83 ± 39.20 bits min-1and 204.47 ± 37.56 bits min-1, respectively. Notably, the peak ITR could reach up to 367.83 bits min-1.Significance.This study developed the first high-speed BCI system with more than 200 targets, which holds promise for extending BCI's application scenarios.
Collapse
Affiliation(s)
- Jin Han
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
| | - Minpeng Xu
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, People's Republic of China
| | - Xiaolin Xiao
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, People's Republic of China
| | - Weibo Yi
- Beijing Machine and Equipment Institute, Beijing 100854, People's Republic of China
| | - Tzyy-Ping Jung
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Swartz Centre for Computational Neuroscience, University of California, San Diego, CA, United States of America
| | - Dong Ming
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, People's Republic of China
| |
Collapse
|
46
|
Duan X, Xie S, Lv Y, Xie X, Obermayer K, Yan H. A transfer learning-based feedback training motivates the performance of SMR-BCI. J Neural Eng 2023; 20. [PMID: 36577144 DOI: 10.1088/1741-2552/acaee7] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2022] [Accepted: 12/28/2022] [Indexed: 12/29/2022]
Abstract
Objective. Feedback training is a practical approach to brain-computer interface (BCI) end-users learning to modulate their sensorimotor rhythms (SMRs). BCI self-regulation learning has been shown to be influenced by subjective psychological factors, such as motivation. However, few studies have taken into account the users' self-motivation as additional guidance for the cognitive process involved in BCI learning. In this study we tested a transfer learning (TL) feedback method designed to increase self-motivation by providing information about past performance.Approach. Electroencephalography (EEG) signals from the previous runs were affine transformed and displayed as points on the screen, along with the newly recorded EEG signals in the current run, giving the subjects a context for self-motivation. Subjects were asked to separate the feedback points for the current run under the display of the separability of prior training. We conducted a between-subject feedback training experiment, in which 24 healthy SMR-BCI naive subjects were trained to imagine left- and right-hand movements. The participants were provided with either TL feedback or typical cursor-bar (CB) feedback (control condition), for three sessions on separate days.Main results. The behavioral results showed an increased challenge and stable mastery confidence, suggesting that subjects' motivation grew as the feedback training went on. The EEG results showed favorable overall training effects with TL feedback in terms of the class distinctiveness and EEG discriminancy. Performance was 28.5% higher in the third session than in the first. About 41.7% of the subjects were 'learners' including not only low-performance subjects, but also good-performance subjects who might be affected by the ceiling effect. Subjects were able to control BCI with TL feedback with a higher performance of 60.5% during the last session compared to CB feedback.Significance. The present study demonstrated that the proposed TL feedback method boosted psychological engagement through the self-motivated context, and further allowed subjects to modulate SMR effectively. The proposed TL feedback method also provided an alternative to typical CB feedback.
Collapse
Affiliation(s)
- Xu Duan
- School of Electronics and Information, Northwestern Polytechnical University, Dongxiang Road 1, Xi'an 710129, People's Republic of China.,Key Laboratory for Artificial Intelligence and Cognitive Neuroscience of Language, Xi'an International Studies University, Wenyuan South Road, Xi'an 710128, People's Republic of China
| | - Songyun Xie
- School of Electronics and Information, Northwestern Polytechnical University, Dongxiang Road 1, Xi'an 710129, People's Republic of China
| | - Yanxia Lv
- School of Electronics and Information, Northwestern Polytechnical University, Dongxiang Road 1, Xi'an 710129, People's Republic of China
| | - Xinzhou Xie
- School of Electronics and Information, Northwestern Polytechnical University, Dongxiang Road 1, Xi'an 710129, People's Republic of China
| | - Klaus Obermayer
- Faculty of Electrical Engineering and Computer Science, Technical University Berlin, Marchstraße 23, Berlin 10587, Germany
| | - Hao Yan
- Key Laboratory for Artificial Intelligence and Cognitive Neuroscience of Language, Xi'an International Studies University, Wenyuan South Road, Xi'an 710128, People's Republic of China
| |
Collapse
|
47
|
Blanco-Díaz CF, Guerrero-Méndez CD, Ruiz-Olaya AF. Enhancing P300 Detection Using a Band-Selective Filter Bank for a Visual P300 Speller. Ing Rech Biomed 2023. [DOI: 10.1016/j.irbm.2022.100751] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/07/2023]
|
48
|
Kim J, Jiang X, Forenzo D, Liu Y, Anderson N, Greco CM, He B. Immediate effects of short-term meditation on sensorimotor rhythm-based brain-computer interface performance. Front Hum Neurosci 2022; 16:1019279. [PMID: 36606248 PMCID: PMC9807599 DOI: 10.3389/fnhum.2022.1019279] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2022] [Accepted: 11/25/2022] [Indexed: 12/24/2022] Open
Abstract
Introduction Meditation has been shown to enhance a user's ability to control a sensorimotor rhythm (SMR)-based brain-computer interface (BCI). For example, prior work have demonstrated that long-term meditation practices and an 8-week mindfulness-based stress reduction (MBSR) training have positive behavioral and neurophysiological effects on SMR-based BCI. However, the effects of short-term meditation practice on SMR-based BCI control are still unknown. Methods In this study, we investigated the immediate effects of a short, 20-minute meditation on SMR-based BCI control. Thirty-seven subjects performed several runs of one-dimensional cursor control tasks before and after two types of 20-minute interventions: a guided mindfulness meditation exercise and a recording of a narrator reading a journal article. Results We found that there is no significant change in BCI performance and Electroencephalography (EEG) BCI control signal following either 20-minute intervention. Moreover, the change in BCI performance between the meditation group and the control group was found to be not significant. Discussion The present results suggest that a longer period of meditation is needed to improve SMR-based BCI control.
Collapse
Affiliation(s)
- Jeehyun Kim
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| | - Xiyuan Jiang
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| | - Dylan Forenzo
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| | - Yixuan Liu
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| | - Nancy Anderson
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| | - Carol M. Greco
- Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, United States
| | - Bin He
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| |
Collapse
|
49
|
Fang T, Wang J, Mu W, Song Z, Zhang X, Zhan G, Wang P, Bin J, Niu L, Zhang L, Kang X. Noninvasive neuroimaging and spatial filter transform enable ultra low delay motor imagery EEG decoding. J Neural Eng 2022; 19. [PMID: 36541542 DOI: 10.1088/1741-2552/aca82d] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2022] [Accepted: 12/01/2022] [Indexed: 12/04/2022]
Abstract
Objective.The brain-computer interface (BCI) system based on sensorimotor rhythm can convert the human spirit into instructions for machine control, and it is a new human-computer interaction system with broad applications. However, the spatial resolution of scalp electroencephalogram (EEG) is limited due to the presence of volume conduction effects. Therefore, it is very meaningful to explore intracranial activities in a noninvasive way and improve the spatial resolution of EEG. Meanwhile, low-delay decoding is an essential factor for the development of a real-time BCI system.Approach.In this paper, EEG conduction is modeled by using public head anatomical templates, and cortical EEG is obtained using dynamic parameter statistical mapping. To solve the problem of a large amount of computation caused by the increase in the number of channels, the filter bank common spatial pattern method is used to obtain a spatial filter kernel, which reduces the computational cost of feature extraction to a linear level. And the feature classification and selection of important features are completed using a neural network containing band-spatial-time domain self-attention mechanisms.Main results.The results show that the method proposed in this paper achieves high accuracy for the four types of motor imagery EEG classification tasks, with fairly low latency and high physiological interpretability.Significance.The proposed decoding framework facilitates the realization of low-latency human-computer interaction systems.
Collapse
Affiliation(s)
- Tao Fang
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Ministry of Education, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Fudan University, Shanghai, People's Republic of China
| | - Junkongshuai Wang
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Ministry of Education, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Fudan University, Shanghai, People's Republic of China
| | - Wei Mu
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Ministry of Education, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Fudan University, Shanghai, People's Republic of China
| | - Zuoting Song
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Ministry of Education, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Fudan University, Shanghai, People's Republic of China
| | - Xueze Zhang
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Ministry of Education, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Fudan University, Shanghai, People's Republic of China
| | - Gege Zhan
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Ministry of Education, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Fudan University, Shanghai, People's Republic of China
| | - Pengchao Wang
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Ministry of Education, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Fudan University, Shanghai, People's Republic of China
| | - Jianxiong Bin
- Ji Hua Laboratory, Foshan, People's Republic of China
| | - Lan Niu
- Ji Hua Laboratory, Foshan, People's Republic of China
| | - Lihua Zhang
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Ministry of Education, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Fudan University, Shanghai, People's Republic of China.,Ji Hua Laboratory, Foshan, People's Republic of China
| | - Xiaoyang Kang
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Ministry of Education, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Fudan University, Shanghai, People's Republic of China.,Ji Hua Laboratory, Foshan, People's Republic of China.,Yiwu Research Institute of Fudan University, Yiwu City, People's Republic of China.,Research Center for Intelligent Sensing, Zhejiang Lab, Hangzhou, People's Republic of China.,Greater Bay Area Institute of Precision Medicine, Guangzhou, People's Republic of China
| |
Collapse
|
50
|
Cho JH, Jeong JH, Lee SW. NeuroGrasp: Real-Time EEG Classification of High-Level Motor Imagery Tasks Using a Dual-Stage Deep Learning Framework. IEEE TRANSACTIONS ON CYBERNETICS 2022; 52:13279-13292. [PMID: 34748509 DOI: 10.1109/tcyb.2021.3122969] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Brain-computer interfaces (BCIs) have been widely employed to identify and estimate a user's intention to trigger a robotic device by decoding motor imagery (MI) from an electroencephalogram (EEG). However, developing a BCI system driven by MI related to natural hand-grasp tasks is challenging due to its high complexity. Although numerous BCI studies have successfully decoded large body parts, such as the movement intention of both hands, arms, or legs, research on MI decoding of high-level behaviors such as hand grasping is essential to further expand the versatility of MI-based BCIs. In this study, we propose NeuroGrasp, a dual-stage deep learning framework that decodes multiple hand grasping from EEG signals under the MI paradigm. The proposed method effectively uses an EEG and electromyography (EMG)-based learning, such that EEG-based inference at test phase becomes possible. The EMG guidance during model training allows BCIs to predict hand grasp types from EEG signals accurately. Consequently, NeuroGrasp improved classification performance offline, and demonstrated a stable classification performance online. Across 12 subjects, we obtained an average offline classification accuracy of 0.68 (±0.09) in four-grasp-type classifications and 0.86 (±0.04) in two-grasp category classifications. In addition, we obtained an average online classification accuracy of 0.65 (±0.09) and 0.79 (±0.09) across six high-performance subjects. Because the proposed method has demonstrated a stable classification performance when evaluated either online or offline, in the future, we expect that the proposed method could contribute to different BCI applications, including robotic hands or neuroprosthetics for handling everyday objects.
Collapse
|