1
|
Chen Y, Wang F, Li T, Zhao L, Gong A, Nan W, Ding P, Fu Y. Considerations and discussions on the clear definition and definite scope of brain-computer interfaces. Front Neurosci 2024; 18:1449208. [PMID: 39161655 PMCID: PMC11330831 DOI: 10.3389/fnins.2024.1449208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2024] [Accepted: 07/22/2024] [Indexed: 08/21/2024] Open
Abstract
Brain-computer interface (BCI) is a revolutionizing human-computer interaction with potential applications in both medical and non-medical fields, emerging as a cutting-edge and trending research direction. Increasing numbers of groups are engaging in BCI research and development. However, in recent years, there has been some confusion regarding BCI, including misleading and hyped propaganda about BCI, and even non-BCI technologies being labeled as BCI. Therefore, a clear definition and a definite scope for BCI are thoroughly considered and discussed in the paper, based on the existing definitions of BCI, including the six key or essential components of BCI. In the review, different from previous definitions of BCI, BCI paradigms and neural coding are explicitly included in the clear definition of BCI provided, and the BCI user (the brain) is clearly identified as a key component of the BCI system. Different people may have different viewpoints on the definition and scope of BCI, as well as some related issues, which are discussed in the article. This review argues that a clear definition and definite scope of BCI will benefit future research and commercial applications. It is hoped that this review will reduce some of the confusion surrounding BCI and promote sustainable development in this field.
Collapse
Affiliation(s)
- Yanxiao Chen
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Fan Wang
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Tianwen Li
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Zhao
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Anmin Gong
- School of Information Engineering, Chinese People’s Armed Police Force Engineering University, Xi’an, China
| | - Wenya Nan
- School of Psychology, Shanghai Normal University, Shanghai, China
| | - Peng Ding
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Yunfa Fu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| |
Collapse
|
2
|
Pan H, Ding P, Wang F, Li T, Zhao L, Nan W, Fu Y, Gong A. Comprehensive evaluation methods for translating BCI into practical applications: usability, user satisfaction and usage of online BCI systems. Front Hum Neurosci 2024; 18:1429130. [PMID: 38903409 PMCID: PMC11188342 DOI: 10.3389/fnhum.2024.1429130] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2024] [Accepted: 05/20/2024] [Indexed: 06/22/2024] Open
Abstract
Although brain-computer interface (BCI) is considered a revolutionary advancement in human-computer interaction and has achieved significant progress, a considerable gap remains between the current technological capabilities and their practical applications. To promote the translation of BCI into practical applications, the gold standard for online evaluation for classification algorithms of BCI has been proposed in some studies. However, few studies have proposed a more comprehensive evaluation method for the entire online BCI system, and it has not yet received sufficient attention from the BCI research and development community. Therefore, the qualitative leap from analyzing and modeling for offline BCI data to the construction of online BCI systems and optimizing their performance is elaborated, and then user-centred is emphasized, and then the comprehensive evaluation methods for translating BCI into practical applications are detailed and reviewed in the article, including the evaluation of the usability (including effectiveness and efficiency of systems), the evaluation of the user satisfaction (including BCI-related aspects, etc.), and the evaluation of the usage (including the match between the system and user, etc.) of online BCI systems. Finally, the challenges faced in the evaluation of the usability and user satisfaction of online BCI systems, the efficacy of online BCI systems, and the integration of BCI and artificial intelligence (AI) and/or virtual reality (VR) and other technologies to enhance the intelligence and user experience of the system are discussed. It is expected that the evaluation methods for online BCI systems elaborated in this review will promote the translation of BCI into practical applications.
Collapse
Affiliation(s)
- He Pan
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Peng Ding
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Fan Wang
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Tianwen Li
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Zhao
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Wenya Nan
- Department of Psychology, School of Education, Shanghai Normal University, Shanghai, China
| | - Yunfa Fu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Anmin Gong
- School of Information Engineering, Chinese People's Armed Police Force Engineering University, Xi’an, China
| |
Collapse
|
3
|
Chen Y, Wang F, Li T, Zhao L, Gong A, Nan W, Ding P, Fu Y. Several inaccurate or erroneous conceptions and misleading propaganda about brain-computer interfaces. Front Hum Neurosci 2024; 18:1391550. [PMID: 38601800 PMCID: PMC11004276 DOI: 10.3389/fnhum.2024.1391550] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2024] [Accepted: 03/18/2024] [Indexed: 04/12/2024] Open
Abstract
Brain-computer interface (BCI) is a revolutionizing human-computer interaction, which has potential applications for specific individuals or groups in specific scenarios. Extensive research has been conducted on the principles and implementation methods of BCI, and efforts are currently being made to bridge the gap from research to real-world applications. However, there are inaccurate or erroneous conceptions about BCI among some members of the public, and certain media outlets, as well as some BCI researchers, developers, manufacturers, and regulators, propagate misleading or overhyped claims about BCI technology. Therefore, this article summarizes the several misconceptions and misleading propaganda about BCI, including BCI being capable of "mind-controlled," "controlling brain," "mind reading," and the ability to "download" or "upload" information from or to the brain using BCI, among others. Finally, the limitations (shortcomings) and limits (boundaries) of BCI, as well as the necessity of conducting research aimed at countering BCI systems are discussed, and several suggestions are offered to reduce misconceptions and misleading claims about BCI.
Collapse
Affiliation(s)
- Yanxiao Chen
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Fan Wang
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Tianwen Li
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Zhao
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Anmin Gong
- School of Information Engineering, Chinese People’s Armed Police Force Engineering University, Xi’an, China
| | - Wenya Nan
- Department of Psychology, School of Education, Shanghai Normal University, Shanghai, China
| | - Peng Ding
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Yunfa Fu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| |
Collapse
|
4
|
Kumar S, Alawieh H, Racz FS, Fakhreddine R, Millán JDR. Transfer learning promotes acquisition of individual BCI skills. PNAS NEXUS 2024; 3:pgae076. [PMID: 38426121 PMCID: PMC10903645 DOI: 10.1093/pnasnexus/pgae076] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Accepted: 02/05/2024] [Indexed: 03/02/2024]
Abstract
Subject training is crucial for acquiring brain-computer interface (BCI) control. Typically, this requires collecting user-specific calibration data due to high inter-subject neural variability that limits the usability of generic decoders. However, calibration is cumbersome and may produce inadequate data for building decoders, especially with naïve subjects. Here, we show that a decoder trained on the data of a single expert is readily transferrable to inexperienced users via domain adaptation techniques allowing calibration-free BCI training. We introduce two real-time frameworks, (i) Generic Recentering (GR) through unsupervised adaptation and (ii) Personally Assisted Recentering (PAR) that extends GR by employing supervised recalibration of the decoder parameters. We evaluated our frameworks on 18 healthy naïve subjects over five online sessions, who operated a customary synchronous bar task with continuous feedback and a more challenging car racing game with asynchronous control and discrete feedback. We show that along with improved task-oriented BCI performance in both tasks, our frameworks promoted subjects' ability to acquire individual BCI skills, as the initial neurophysiological control features of an expert subject evolved and became subject specific. Furthermore, those features were task-specific and were learned in parallel as participants practiced the two tasks in every session. Contrary to previous findings implying that supervised methods lead to improved online BCI control, we observed that longitudinal training coupled with unsupervised domain matching (GR) achieved similar performance to supervised recalibration (PAR). Therefore, our presented frameworks facilitate calibration-free BCIs and have immediate implications for broader populations-such as patients with neurological pathologies-who might struggle to provide suitable initial calibration data.
Collapse
Affiliation(s)
- Satyam Kumar
- Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
| | - Hussein Alawieh
- Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
| | - Frigyes Samuel Racz
- Department of Neurology, The University of Texas at Austin, Austin, TX 78712, USA
- Mulva Clinic for the Neurosciences, The University of Texas at Austin, Austin, TX 78712, USA
| | - Rawan Fakhreddine
- Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
| | - José del R Millán
- Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
- Department of Neurology, The University of Texas at Austin, Austin, TX 78712, USA
- Mulva Clinic for the Neurosciences, The University of Texas at Austin, Austin, TX 78712, USA
- Departement of Biomedical Engineering, The University of Texas at Austin, Austin, TX 78712, USA
| |
Collapse
|
5
|
Nagarajan A, Robinson N, Ang KK, Chua KSG, Chew E, Guan C. Transferring a deep learning model from healthy subjects to stroke patients in a motor imagery brain-computer interface. J Neural Eng 2024; 21:016007. [PMID: 38091617 DOI: 10.1088/1741-2552/ad152f] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Accepted: 12/13/2023] [Indexed: 01/18/2024]
Abstract
Objective.Motor imagery (MI) brain-computer interfaces (BCIs) based on electroencephalogram (EEG) have been developed primarily for stroke rehabilitation, however, due to limited stroke data, current deep learning methods for cross-subject classification rely on healthy data. This study aims to assess the feasibility of applying MI-BCI models pre-trained using data from healthy individuals to detect MI in stroke patients.Approach.We introduce a new transfer learning approach where features from two-class MI data of healthy individuals are used to detect MI in stroke patients. We compare the results of the proposed method with those obtained from analyses within stroke data. Experiments were conducted using Deep ConvNet and state-of-the-art subject-specific machine learning MI classifiers, evaluated on OpenBMI two-class MI-EEG data from healthy subjects and two-class MI versus rest data from stroke patients.Main results.Results of our study indicate that through domain adaptation of a model pre-trained using healthy subjects' data, an average MI detection accuracy of 71.15% (±12.46%) can be achieved across 71 stroke patients. We demonstrate that the accuracy of the pre-trained model increased by 18.15% after transfer learning (p<0.001). Additionally, the proposed transfer learning method outperforms the subject-specific results achieved by Deep ConvNet and FBCSP, with significant enhancements of 7.64% (p<0.001) and 5.55% (p<0.001) in performance, respectively. Notably, the healthy-to-stroke transfer learning approach achieved similar performance to stroke-to-stroke transfer learning, with no significant difference (p>0.05). Explainable AI analyses using transfer models determined channel relevance patterns that indicate contributions from the bilateral motor, frontal, and parietal regions of the cortex towards MI detection in stroke patients.Significance.Transfer learning from healthy to stroke can enhance the clinical use of BCI algorithms by overcoming the challenge of insufficient clinical data for optimal training.
Collapse
Affiliation(s)
- Aarthy Nagarajan
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798, Singapore
| | - Neethu Robinson
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798, Singapore
| | - Kai Keng Ang
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798, Singapore
- Institute for Infocomm Research, Agency of Science, Technology and Research (A*STAR), 1 Fusionopolis Way, Singapore 138632, Singapore
| | - Karen Sui Geok Chua
- Department of Rehabilitation Medicine, Tan Tock Seng Hospital, 11 Jln Tan Tock Seng, Singapore 308433, Singapore
| | - Effie Chew
- National University Health System, 1E Kent Ridge Road, Singapore 119228, Singapore
| | - Cuntai Guan
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798, Singapore
| |
Collapse
|
6
|
Forenzo D, Liu Y, Kim J, Ding Y, Yoon T, He B. Integrating Simultaneous Motor Imagery and Spatial Attention for EEG-BCI Control. IEEE Trans Biomed Eng 2024; 71:282-294. [PMID: 37494151 PMCID: PMC10803074 DOI: 10.1109/tbme.2023.3298957] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/28/2023]
Abstract
OBJECTIVE EEG-based brain-computer interfaces (BCI) are non-invasive approaches for replacing or restoring motor functions in impaired patients, and direct brain-to-device communication in the general population. Motor imagery (MI) is one of the most used BCI paradigms, but its performance varies across individuals and certain users require substantial training to develop control. In this study, we propose to integrate a MI paradigm simultaneously with a recently proposed Overt Spatial Attention (OSA) paradigm, to accomplish BCI control. METHODS We evaluated a cohort of 25 human subjects' ability to control a virtual cursor in one- and two-dimensions over 5 BCI sessions. The subjects used 5 different BCI paradigms: MI alone, OSA alone, MI, and OSA simultaneously towards the same target (MI+OSA), and MI for one axis while OSA controls the other (MI/OSA and OSA/MI). RESULTS Our results show that MI+OSA reached the highest average online performance in 2D tasks at 49% Percent Valid Correct (PVC), and statistically outperforms both MI alone (42%) and OSA alone (45%). MI+OSA had a similar performance to each subject's best individual method between MI alone and OSA alone (50%) and 9 subjects reached their highest average BCI performance using MI+OSA. CONCLUSION Integrating MI and OSA leads to improved performance over both individual methods at the group level and is the best BCI paradigm option for some subjects. SIGNIFICANCE This work proposes a new BCI control paradigm that integrates two existing paradigms and demonstrates its value by showing that it can improve users' BCI performance.
Collapse
Affiliation(s)
- Dylan Forenzo
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| | - Yixuan Liu
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| | - Jeehyun Kim
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| | - Yidan Ding
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| | - Taehyung Yoon
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| | - Bin He
- Department of Biomedical Engineering at Carnegie Mellon University, Pittsburgh, PA
| |
Collapse
|
7
|
Borgheai SB, Zisk AH, McLinden J, Mcintyre J, Sadjadi R, Shahriari Y. Multimodal pre-screening can predict BCI performance variability: A novel subject-specific experimental scheme. Comput Biol Med 2024; 168:107658. [PMID: 37984201 DOI: 10.1016/j.compbiomed.2023.107658] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2023] [Revised: 10/20/2023] [Accepted: 10/31/2023] [Indexed: 11/22/2023]
Abstract
BACKGROUND Brain-computer interface (BCI) systems currently lack the required robustness for long-term daily use due to inter- and intra-subject performance variability. In this study, we propose a novel personalized scheme for a multimodal BCI system, primarily using functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), to identify, predict, and compensate for factors affecting competence-related and interfering factors associated with performance. METHOD 11 (out of 13 recruited) participants, including five participants with motor deficits, completed four sessions on average. During the training sessions, the subjects performed a short pre-screening phase, followed by three variations of a novel visou-mental (VM) protocol. Features extracted from the pre-screening phase were used to construct predictive platforms using stepwise multivariate linear regression (MLR) models. In the test sessions, we employed a task-correction phase where our predictive models were used to predict the ideal task variation to maximize performance, followed by an interference-correction phase. We then investigated the associations between predicted and actual performances and evaluated the outcome of correction strategies. RESULT The predictive models resulted in respective adjusted R-squared values of 0.942, 0.724, and 0.939 for the first, second, and third variation of the task, respectively. The statistical analyses showed significant associations between the performances predicted by predictive models and the actual performances for the first two task variations, with rhos of 0.7289 (p-value = 0.011) and 0.6970 (p-value = 0.017), respectively. For 81.82 % of the subjects, the task/workload correction stage correctly determined which task variation provided the highest accuracy, with an average performance gain of 5.18 % when applying the correction strategies. CONCLUSION Our proposed method can lead to an integrated multimodal predictive framework to compensate for BCI performance variability, particularly, for people with severe motor deficits.
Collapse
Affiliation(s)
- Seyyed Bahram Borgheai
- Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island, Kingston, RI, United States; Neurology Department, Emory University, Atlanta, GA, United States
| | - Alyssa Hillary Zisk
- Interdisciplinary Neuroscience Program, University of Rhode Island, Kingston, RI, United States
| | - John McLinden
- Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island, Kingston, RI, United States
| | - James Mcintyre
- Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island, Kingston, RI, United States
| | - Reza Sadjadi
- Neurology Department, Massachusetts General Hospital, Boston, MA, United States
| | - Yalda Shahriari
- Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island, Kingston, RI, United States; Interdisciplinary Neuroscience Program, University of Rhode Island, Kingston, RI, United States.
| |
Collapse
|
8
|
Ivanov N, Lio A, Chau T. Towards user-centric BCI design: Markov chain-based user assessment for mental imagery EEG-BCIs. J Neural Eng 2023; 20:066037. [PMID: 38128128 DOI: 10.1088/1741-2552/ad17f2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2023] [Accepted: 12/21/2023] [Indexed: 12/23/2023]
Abstract
Objective.While electroencephalography (EEG)-based brain-computer interfaces (BCIs) have many potential clinical applications, their use is impeded by poor performance for many users. To improve BCI performance, either via enhanced signal processing or user training, it is critical to understand and describe each user's ability to perform mental control tasks and produce discernible EEG patterns. While classification accuracy has predominantly been used to assess user performance, limitations and criticisms of this approach have emerged, thus prompting the need to develop novel user assessment approaches with greater descriptive capability. Here, we propose a combination of unsupervised clustering and Markov chain models to assess and describe user skill.Approach.Using unsupervisedK-means clustering, we segmented the EEG signal space into regions representing pattern states that users could produce. A user's movement through these pattern states while performing different tasks was modeled using Markov chains. Finally, using the steady-state distributions and entropy rates of the Markov chains, we proposed two metricstaskDistinctandrelativeTaskInconsistencyto assess, respectively, a user's ability to (i) produce distinct task-specific patterns for each mental task and (ii) maintain consistent patterns during individual tasks.Main results.Analysis of data from 14 adolescents using a three-class BCI revealed significant correlations between thetaskDistinctandrelativeTaskInconsistencymetrics and classification F1 score. Moreover, analysis of the pattern states and Markov chain models yielded descriptive information regarding user performance not immediately apparent from classification accuracy.Significance.Our proposed user assessment method can be used in concert with classifier-based analysis to further understand the extent to which users produce task-specific, time-evolving EEG patterns. In turn, this information could be used to enhance user training or classifier design.
Collapse
Affiliation(s)
- Nicolas Ivanov
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
| | - Aaron Lio
- Division of Engineering Science, University of Toronto, Toronto, ON, Canada
| | - Tom Chau
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
9
|
Xu B, Liu D, Xue M, Miao M, Hu C, Song A. Continuous shared control of a mobile robot with brain-computer interface and autonomous navigation for daily assistance. Comput Struct Biotechnol J 2023; 22:3-16. [PMID: 37600142 PMCID: PMC10433001 DOI: 10.1016/j.csbj.2023.07.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2023] [Revised: 07/04/2023] [Accepted: 07/22/2023] [Indexed: 08/22/2023] Open
Abstract
Although the electroencephalography (EEG) based brain-computer interface (BCI) has been successfully developed for rehabilitation and assistance, it is still challenging to achieve continuous control of a brain-actuated mobile robot system. In this study, we propose a continuous shared control strategy combining continuous BCI and autonomous navigation for a mobile robot system. The weight of shared control is designed to dynamically adjust the fusion of continuous BCI control and autonomous navigation. During this process, the system uses the visual-based simultaneous localization and mapping (SLAM) method to construct environmental maps. After obtaining the global optimal path, the system utilizes the brain-based shared control dynamic window approach (BSC-DWA) to evaluate safe and reachable trajectories while considering shared control velocity. Eight subjects participated in two-stage training, and six of these eight subjects participated in online shared control experiments. The training results demonstrated that naïve subjects could achieve continuous control performance with an average percent valid correct rate of approximately 97 % and an average total correct rate of over 80 %. The results of online shared control experiments showed that all of the subjects could complete navigation tasks in an unknown corridor with continuous shared control. Therefore, our experiments verified the feasibility and effectiveness of the proposed system combining continuous BCI, shared control, autonomous navigation, and visual SLAM. The proposed continuous shared control framework shows great promise in BCI-driven tasks, especially navigation tasks for brain-driven assistive mobile robots and wheelchairs in daily applications.
Collapse
Affiliation(s)
- Baoguo Xu
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| | - Deping Liu
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| | - Muhui Xue
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| | - Minmin Miao
- School of Information Engineering, Huzhou University, Huzhou 313000, China
| | - Cong Hu
- Guangxi Key Laboratory of Automatic Detecting Technology and Instruments, Guilin University of Electronic Technology, Guilin 541004, China
| | - Aiguo Song
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| |
Collapse
|
10
|
Gemborn Nilsson M, Tufvesson P, Heskebeck F, Johansson M. An open-source human-in-the-loop BCI research framework: method and design. Front Hum Neurosci 2023; 17:1129362. [PMID: 37441434 PMCID: PMC10335802 DOI: 10.3389/fnhum.2023.1129362] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Accepted: 06/08/2023] [Indexed: 07/15/2023] Open
Abstract
Brain-computer interfaces (BCIs) translate brain activity into digital commands for interaction with the physical world. The technology has great potential in several applied areas, ranging from medical applications to entertainment industry, and creates new conditions for basic research in cognitive neuroscience. The BCIs of today, however, offer only crude online classification of the user's current state of mind, and more sophisticated decoding of mental states depends on time-consuming offline data analysis. The present paper addresses this limitation directly by leveraging a set of improvements to the analytical pipeline to pave the way for the next generation of online BCIs. Specifically, we introduce an open-source research framework that features a modular and customizable hardware-independent design. This framework facilitates human-in-the-loop (HIL) model training and retraining, real-time stimulus control, and enables transfer learning and cloud computing for the online classification of electroencephalography (EEG) data. Stimuli for the subject and diagnostics for the researcher are shown on separate displays using web browser technologies. Messages are sent using the Lab Streaming Layer standard and websockets. Real-time signal processing and classification, as well as training of machine learning models, is facilitated by the open-source Python package Timeflux. The framework runs on Linux, MacOS, and Windows. While online analysis is the main target of the BCI-HIL framework, offline analysis of the EEG data can be performed with Python, MATLAB, and Julia through packages like MNE, EEGLAB, or FieldTrip. The paper describes and discusses desirable properties of a human-in-the-loop BCI research platform. The BCI-HIL framework is released under MIT license with examples at: bci.lu.se/bci-hil (or at: github.com/bci-hil/bci-hil).
Collapse
Affiliation(s)
| | - Pex Tufvesson
- Department of Automatic Control, Lund University, Lund, Sweden
- Ericsson Research, Lund, Sweden
| | - Frida Heskebeck
- Department of Automatic Control, Lund University, Lund, Sweden
| | | |
Collapse
|
11
|
Śliwowski M, Martin M, Souloumiac A, Blanchart P, Aksenova T. Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performance. Front Hum Neurosci 2023; 17:1111645. [PMID: 37007675 PMCID: PMC10061076 DOI: 10.3389/fnhum.2023.1111645] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Accepted: 02/27/2023] [Indexed: 03/18/2023] Open
Abstract
IntroductionIn brain-computer interfaces (BCI) research, recording data is time-consuming and expensive, which limits access to big datasets. This may influence the BCI system performance as machine learning methods depend strongly on the training dataset size. Important questions arise: taking into account neuronal signal characteristics (e.g., non-stationarity), can we achieve higher decoding performance with more data to train decoders? What is the perspective for further improvement with time in the case of long-term BCI studies? In this study, we investigated the impact of long-term recordings on motor imagery decoding from two main perspectives: model requirements regarding dataset size and potential for patient adaptation.MethodsWe evaluated the multilinear model and two deep learning (DL) models on a long-term BCI & Tetraplegia (ClinicalTrials.gov identifier: NCT02550522) clinical trial dataset containing 43 sessions of ECoG recordings performed with a tetraplegic patient. In the experiment, a participant executed 3D virtual hand translation using motor imagery patterns. We designed multiple computational experiments in which training datasets were increased or translated to investigate the relationship between models' performance and different factors influencing recordings.ResultsOur results showed that DL decoders showed similar requirements regarding the dataset size compared to the multilinear model while demonstrating higher decoding performance. Moreover, high decoding performance was obtained with relatively small datasets recorded later in the experiment, suggesting motor imagery patterns improvement and patient adaptation during the long-term experiment. Finally, we proposed UMAP embeddings and local intrinsic dimensionality as a way to visualize the data and potentially evaluate data quality.DiscussionDL-based decoding is a prospective approach in BCI which may be efficiently applied with real-life dataset size. Patient-decoder co-adaptation is an important factor to consider in long-term clinical BCI.
Collapse
Affiliation(s)
- Maciej Śliwowski
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
- Université Paris-Saclay, CEA, List, Palaiseau, France
| | - Matthieu Martin
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
| | | | | | - Tetiana Aksenova
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
- *Correspondence: Tetiana Aksenova
| |
Collapse
|
12
|
Forenzo D, Liu Y, Kim J, Ding Y, Yoon T, He B. Integrating simultaneous motor imagery and spatial attention for EEG-BCI control. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.02.20.529307. [PMID: 36865207 PMCID: PMC9980047 DOI: 10.1101/2023.02.20.529307] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/25/2023]
Abstract
OBJECTIVE EEG-based brain-computer interfaces (BCI) are non-invasive approaches for replacing or restoring motor functions in impaired patients, and direct brain-to-device communication in the general population. Motor imagery (MI) is one of the most used BCI paradigms, but its performance varies across individuals and certain users require substantial training to develop control. In this study, we propose to integrate a MI paradigm simultaneously with a recently proposed Overt Spatial Attention (OSA) paradigm, to accomplish BCI control. METHODS We evaluated a cohort of 25 human subjects' ability to control a virtual cursor in one- and two-dimensions over 5 BCI sessions. The subjects used 5 different BCI paradigms: MI alone, OSA alone, MI and OSA simultaneously towards the same target (MI+OSA), and MI for one axis while OSA controls the other (MI/OSA and OSA/MI). RESULTS Our results show that MI+OSA reached the highest average online performance in 2D tasks at 49% Percent Valid Correct (PVC), statistically outperforms MI alone (42%), and was higher, but not statistically significant, than OSA alone (45%). MI+OSA had a similar performance to each subject's best individual method between MI alone and OSA alone (50%) and 9 subjects reached their highest average BCI performance using MI+OSA. CONCLUSION Integrating MI and OSA leads to improved performance over MI alone at the group level and is the best BCI paradigm option for some subjects. SIGNIFICANCE This work proposes a new BCI control paradigm that integrates two existing paradigms and demonstrates its value by showing that it can improve users' BCI performance.
Collapse
|
13
|
Ivanov N, Chau T. Riemannian geometry-based metrics to measure and reinforce user performance changes during brain-computer interface user training. Front Comput Neurosci 2023; 17:1108889. [PMID: 36860616 PMCID: PMC9968793 DOI: 10.3389/fncom.2023.1108889] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2022] [Accepted: 01/25/2023] [Indexed: 02/15/2023] Open
Abstract
Despite growing interest and research into brain-computer interfaces (BCI), their usage remains limited outside of research laboratories. One reason for this is BCI inefficiency, the phenomenon where a significant number of potential users are unable to produce machine-discernible brain signal patterns to control the devices. To reduce the prevalence of BCI inefficiency, some have advocated for novel user-training protocols that enable users to more effectively modulate their neural activity. Important considerations for the design of these protocols are the assessment measures that are used for evaluating user performance and for providing feedback that guides skill acquisition. Herein, we present three trial-wise adaptations (running, sliding window and weighted average) of Riemannian geometry-based user-performance metrics (classDistinct reflecting the degree of class separability and classStability reflecting the level of within-class consistency) to enable feedback to the user following each individual trial. We evaluated these metrics, along with conventional classifier feedback, using simulated and previously recorded sensorimotor rhythm-BCI data to assess their correlation with and discrimination of broader trends in user performance. Analysis revealed that the sliding window and weighted average variants of our proposed trial-wise Riemannian geometry-based metrics more accurately reflected performance changes during BCI sessions compared to conventional classifier output. The results indicate the metrics are a viable method for evaluating and tracking user performance changes during BCI-user training and, therefore, further investigation into how these metrics may be presented to users during training is warranted.
Collapse
Affiliation(s)
- Nicolas Ivanov
- PRISM Lab, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
| | - Tom Chau
- PRISM Lab, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
14
|
A Review of Online Classification Performance in Motor Imagery-Based Brain–Computer Interfaces for Stroke Neurorehabilitation. SIGNALS 2023. [DOI: 10.3390/signals4010004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/22/2023] Open
Abstract
Motor imagery (MI)-based brain–computer interfaces (BCI) have shown increased potential for the rehabilitation of stroke patients; nonetheless, their implementation in clinical practice has been restricted due to their low accuracy performance. To date, although a lot of research has been carried out in benchmarking and highlighting the most valuable classification algorithms in BCI configurations, most of them use offline data and are not from real BCI performance during the closed-loop (or online) sessions. Since rehabilitation training relies on the availability of an accurate feedback system, we surveyed articles of current and past EEG-based BCI frameworks who report the online classification of the movement of two upper limbs in both healthy volunteers and stroke patients. We found that the recently developed deep-learning methods do not outperform the traditional machine-learning algorithms. In addition, patients and healthy subjects exhibit similar classification accuracy in current BCI configurations. Lastly, in terms of neurofeedback modality, functional electrical stimulation (FES) yielded the best performance compared to non-FES systems.
Collapse
|
15
|
Kodama M, Iwama S, Morishige M, Ushiba J. Thirty-minute motor imagery exercise aided by EEG sensorimotor rhythm neurofeedback enhances morphing of sensorimotor cortices: a double-blind sham-controlled study. Cereb Cortex 2023:6967448. [PMID: 36600612 DOI: 10.1093/cercor/bhac525] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2022] [Revised: 12/15/2022] [Accepted: 12/16/2022] [Indexed: 01/06/2023] Open
Abstract
Neurofeedback training using electroencephalogram (EEG)-based brain-computer interfaces (BCIs) combined with mental rehearsals of motor behavior has demonstrated successful self-regulation of motor cortical excitability. However, it remains unclear whether the acquisition of skills to voluntarily control neural excitability is accompanied by structural plasticity boosted by neurofeedback. Here, we sought short-term changes in cortical structures induced by 30 min of BCI-based neurofeedback training, which aimed at the regulation of sensorimotor rhythm (SMR) in scalp EEG. When participants performed kinesthetic motor imagery of right finger movement with online feedback of either event-related desynchronisation (ERD) of SMR magnitude from the contralateral sensorimotor cortex (SM1) or those from other participants (i.e. placebo), the learning rate of SMR-ERD control was significantly different. Although overlapped structural changes in gray matter volumes were found in both groups, significant differences revealed by group-by-group comparison were spatially different; whereas the veritable neurofeedback group exhibited sensorimotor area-specific changes, the placebo exhibited spatially distributed changes. The white matter change indicated a significant decrease in the corpus callosum in the verum group. Furthermore, the learning rate of SMR regulation was correlated with the volume changes in the ipsilateral SM1, suggesting the involvement of interhemispheric motor control circuitries in BCI control tasks.
Collapse
Affiliation(s)
- Midori Kodama
- Graduate School of Science and Technology, Keio University, Kanagawa 108-0073, Japan
| | - Seitaro Iwama
- Graduate School of Science and Technology, Keio University, Kanagawa 108-0073, Japan.,Japan Society for the Promotion of Science, Tokyo 102-0082, Japan
| | - Masumi Morishige
- Graduate School of Science and Technology, Keio University, Kanagawa 108-0073, Japan
| | - Junichi Ushiba
- Department of Biosciences and Informatics, Faculty of Science and Technology, Keio University, Kanagawa 108-0073, Japan
| |
Collapse
|
16
|
Bleuzé A, Mattout J, Congedo M. Tangent space alignment: Transfer learning for Brain-Computer Interface. Front Hum Neurosci 2022; 16:1049985. [DOI: 10.3389/fnhum.2022.1049985] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2022] [Accepted: 11/11/2022] [Indexed: 12/04/2022] Open
Abstract
Statistical variability of electroencephalography (EEG) between subjects and between sessions is a common problem faced in the field of Brain-Computer Interface (BCI). Such variability prevents the usage of pre-trained machine learning models and requires the use of a calibration for every new session. This paper presents a new transfer learning (TL) method that deals with this variability. This method aims to reduce calibration time and even improve accuracy of BCI systems by aligning EEG data from one subject to the other in the tangent space of the positive definite matrices Riemannian manifold. We tested the method on 18 BCI databases comprising a total of 349 subjects pertaining to three BCI paradigms, namely, event related potentials (ERP), motor imagery (MI), and steady state visually evoked potentials (SSVEP). We employ a support vector classifier for feature classification. The results demonstrate a significant improvement of classification accuracy, as compared to a classical training-test pipeline, in the case of the ERP paradigm, whereas for both the MI and SSVEP paradigm no deterioration of performance is observed. A global 2.7% accuracy improvement is obtained compared to a previously published Riemannian method, Riemannian Procrustes Analysis (RPA). Interestingly, tangent space alignment has an intrinsic ability to deal with transfer learning for sets of data that have different number of channels, naturally applying to inter-dataset transfer learning.
Collapse
|
17
|
De Novo Brain-Computer Interfacing Deforms Manifold of Populational Neural Activity Patterns in Human Cerebral Cortex. eNeuro 2022; 9:ENEURO.0145-22.2022. [PMID: 36376067 PMCID: PMC9721308 DOI: 10.1523/eneuro.0145-22.2022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2022] [Revised: 10/27/2022] [Accepted: 11/02/2022] [Indexed: 11/15/2022] Open
Abstract
Human brains are capable of modulating innate activities to adapt to novel environments and tasks; for sensorimotor neural system this means acquisition of a rich repertoire of activity patterns that improve behavioral performance. To directly map the process of acquiring the neural repertoire during tasks onto performance improvement, we analyzed net neural populational activity during the learning of its voluntary modulation by brain-computer interface (BCI) operation in female and male humans. The recorded whole-head high-density scalp electroencephalograms (EEGs) were subjected to dimensionality reduction algorithm to capture changes in cortical activity patterns represented by the synchronization of neuronal oscillations during adaptation. Although the preserved variance of targeted features in the reduced dimensions was 20%, we found systematic interactions between the activity patterns and BCI classifiers that detected motor attempt; the neural manifold derived in the embedded space was stretched along with motor-related features of EEG by model-based fixed classifiers but not with adaptive classifiers that were constantly recalibrated to user activity. Moreover, the manifold was deformed to be orthogonal to the boundary by de novo classifiers with a fixed decision boundary based on biologically unnatural features. Collectively, the flexibility of human cortical signaling patterns (i.e., neural plasticity) is only induced by operation of a BCI whose classifier required fixed activities, and the adaptation could be induced even the requirement is not consistent with biologically natural responses. These principles of neural adaptation at a macroscopic level may underlie the ability of humans to learn wide-ranging behavioral repertoires and adapt to novel environments.
Collapse
|
18
|
Tonin L, Perdikis S, Kuzu TD, Pardo J, Orset B, Lee K, Aach M, Schildhauer TA, Martínez-Olivera R, Millán JDR. Learning to control a BMI-driven wheelchair for people with severe tetraplegia. iScience 2022; 25:105418. [PMID: 36590466 PMCID: PMC9801246 DOI: 10.1016/j.isci.2022.105418] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2022] [Revised: 08/14/2022] [Accepted: 10/18/2022] [Indexed: 11/19/2022] Open
Abstract
Mind-controlled wheelchairs are an intriguing assistive mobility solution applicable in complete paralysis. Despite progress in brain-machine interface (BMI) technology, its translation remains elusive. The primary objective of this study is to probe the hypothesis that BMI skill acquisition by end-users is fundamental to control a non-invasive brain-actuated intelligent wheelchair in real-world settings. We demonstrate that three tetraplegic spinal-cord injury users could be trained to operate a non-invasive, self-paced thought-controlled wheelchair and execute complex navigation tasks. However, only the two users exhibiting increasing decoding performance and feature discriminancy, significant neuroplasticity changes and improved BMI command latency, achieved high navigation performance. In addition, we show that dexterous, continuous control of robots is possible through low-degree of freedom, discrete and uncertain control channels like a motor imagery BMI, by blending human and artificial intelligence through shared-control methodologies. We posit that subject learning and shared-control are the key components paving the way for translational non-invasive BMI.
Collapse
Affiliation(s)
- Luca Tonin
- Department of Information Engineering, University of Padova, Padova, Italy,Padova Neuroscience Center, University of Padova, Padova, Italy
| | - Serafeim Perdikis
- Brain-Computer Interfaces and Neural Engineering Laboratory, School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK
| | - Taylan Deniz Kuzu
- Klinik für Neurochirurgie und Wirbelsäulenchirurgie, Universitätsklinikum Bergmannsheil Bochum, Ruhr-Universität Bochum, Bochum, Germany
| | - Jorge Pardo
- Klinik für Neurochirurgie und Wirbelsäulenchirurgie, Universitätsklinikum Bergmannsheil Bochum, Ruhr-Universität Bochum, Bochum, Germany
| | - Bastien Orset
- École Polytechnique Fédérale de Lausanne, Geneva, Switzerland
| | - Kyuhwa Lee
- Wyss Center for Bio and Neuroengineering, Geneva, Switzerland
| | - Mirko Aach
- Chirurgische Universitätsklinik und Poliklinik, Universitätsklinikum Bergmannsheil Bochum, Ruhr-Universität Bochum, Bochum, Germany
| | - Thomas Armin Schildhauer
- Chirurgische Universitätsklinik und Poliklinik, Universitätsklinikum Bergmannsheil Bochum, Ruhr-Universität Bochum, Bochum, Germany
| | - Ramón Martínez-Olivera
- Klinik für Neurochirurgie und Wirbelsäulenchirurgie, Universitätsklinikum Bergmannsheil Bochum, Ruhr-Universität Bochum, Bochum, Germany
| | - José del R. Millán
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA,Department of Neurology, The University of Texas at Austin, Austin, TX, USA,Corresponding author
| |
Collapse
|
19
|
Korik A, McCreadie K, McShane N, Du Bois N, Khodadadzadeh M, Stow J, McElligott J, Carroll Á, Coyle D. Competing at the Cybathlon championship for people with disabilities: long-term motor imagery brain-computer interface training of a cybathlete who has tetraplegia. J Neuroeng Rehabil 2022; 19:95. [PMID: 36068570 PMCID: PMC9446658 DOI: 10.1186/s12984-022-01073-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2021] [Accepted: 08/29/2022] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND The brain-computer interface (BCI) race at the Cybathlon championship, for people with disabilities, challenges teams (BCI researchers, developers and pilots with spinal cord injury) to control an avatar on a virtual racetrack without movement. Here we describe the training regime and results of the Ulster University BCI Team pilot who has tetraplegia and was trained to use an electroencephalography (EEG)-based BCI intermittently over 10 years, to compete in three Cybathlon events. METHODS A multi-class, multiple binary classifier framework was used to decode three kinesthetically imagined movements (motor imagery of left arm, right arm, and feet), and relaxed state. Three game paradigms were used for training i.e., NeuroSensi, Triad, and Cybathlon Race: BrainDriver. An evaluation of the pilot's performance is presented for two Cybathlon competition training periods-spanning 20 sessions over 5 weeks prior to the 2019 competition, and 25 sessions over 5 weeks in the run up to the 2020 competition. RESULTS Having participated in BCI training in 2009 and competed in Cybathlon 2016, the experienced pilot achieved high two-class accuracy on all class pairs when training began in 2019 (decoding accuracy > 90%, resulting in efficient NeuroSensi and Triad game control). The BrainDriver performance (i.e., Cybathlon race completion time) improved significantly during the training period, leading up to the competition day, ranging from 274-156 s (255 ± 24 s to 191 ± 14 s mean ± std), over 17 days (10 sessions) in 2019, and from 230-168 s (214 ± 14 s to 181 ± 4 s), over 18 days (13 sessions) in 2020. However, on both competition occasions, towards the race date, the performance deteriorated significantly. CONCLUSIONS The training regime and framework applied were highly effective in achieving competitive race completion times. The BCI framework did not cope with significant deviation in electroencephalography (EEG) observed in the sessions occurring shortly before and during the race day. Changes in cognitive state as a result of stress, arousal level, and fatigue, associated with the competition challenge and performance pressure, were likely contributing factors to the non-stationary effects that resulted in the BCI and pilot achieving suboptimal performance on race day. Trial registration not registered.
Collapse
Affiliation(s)
- Attila Korik
- Intelligent Systems Research Centre, Ulster University, Derry, UK.
| | - Karl McCreadie
- Intelligent Systems Research Centre, Ulster University, Derry, UK
| | - Niall McShane
- Intelligent Systems Research Centre, Ulster University, Derry, UK
| | - Naomi Du Bois
- Intelligent Systems Research Centre, Ulster University, Derry, UK
| | | | - Jacqui Stow
- National Rehabilitation Hospital of Ireland, Dun Laoghaire, Ireland
| | | | - Áine Carroll
- National Rehabilitation Hospital of Ireland, Dun Laoghaire, Ireland
- University College Dublin, Dublin, Ireland
| | - Damien Coyle
- Intelligent Systems Research Centre, Ulster University, Derry, UK
| |
Collapse
|
20
|
Hsieh JC, Alawieh H, Li Y, Iwane F, Zhao L, Anderson R, Abdullah S, Kevin Tang KW, Wang W, Pyatnitskiy I, Jia Y, Millán JDR, Wang H. A highly stable electrode with low electrode-skin impedance for wearable brain-computer interface. Biosens Bioelectron 2022; 218:114756. [DOI: 10.1016/j.bios.2022.114756] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2022] [Revised: 09/19/2022] [Accepted: 09/23/2022] [Indexed: 11/17/2022]
|
21
|
Simonyan K, Ehrlich SK, Andersen R, Brumberg J, Guenther F, Hallett M, Howard MA, Millán JDR, Reilly RB, Schultz T, Valeriani D. Brain-Computer Interfaces for Treatment of Focal Dystonia. Mov Disord 2022; 37:1798-1802. [PMID: 35947366 PMCID: PMC9474652 DOI: 10.1002/mds.29178] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Revised: 06/20/2022] [Accepted: 07/19/2022] [Indexed: 11/12/2022] Open
Abstract
Task-specificity in isolated focal dystonias is a powerful feature that may successfully be targeted with therapeutic brain-computer interfaces. While performing a symptomatic task, the patient actively modulates momentary brain activity (disorder signature) to match activity during an asymptomatic task (target signature), which is expected to translate into symptom reduction.
Collapse
Affiliation(s)
- Kristina Simonyan
- Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear and Harvard Medical School, Boston, Massachusetts, USA
- Department of Neurology, Massachusetts General Hospital, Boston, Massachusetts, USA
| | - Stefan K. Ehrlich
- Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear and Harvard Medical School, Boston, Massachusetts, USA
| | - Richard Andersen
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, California, USA
| | - Jonathan Brumberg
- Department of Speech-Language-Hearing: Sciences & Disorders, University of Kansas, Lawrence, Kansas, USA
| | - Frank Guenther
- Department of Speech, Language, & Hearing Sciences, Boston University, Boston, Massachusetts, USA
- Department of Biomedical Engineering, Boston University, Boston, Massachusetts, USA
- Department of Radiology, Massachusetts General Hospital, Boston, Massachusetts, USA
- Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
| | - Mark Hallett
- Human Motor Control Section, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, Maryland, USA
| | - Matthew A. Howard
- Department of Neurosurgery, University of Iowa Carver College of Medicine, Iowa City, Iowa, USA
| | - José del R. Millán
- Department of Electrical and Computer Engineering, University of Texas at Austin, Austin, Texas, USA
- Department of Neurology, University of Texas at Austin, Austin, Texas, USA
| | - Richard B. Reilly
- Center for Biomedical Engineering, Trinity College Institute of Neuroscience, School of Medicine, School of Engineering, Trinity College Dublin and the University of Dublin, Dublin, Ireland
| | - Tanja Schultz
- Faculty 03 Mathematics and Computer Science, University of Bremen, Bremen, Germany
| | - Davide Valeriani
- Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear and Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
22
|
Putri F, Susnoschi Luca I, Garcia Pedro JA, Ding H, Vuckovic A. Winners and losers in brain computer interface competitive gaming: Directional connectivity analysis. J Neural Eng 2022; 19. [PMID: 35882224 DOI: 10.1088/1741-2552/ac8451] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2022] [Accepted: 07/26/2022] [Indexed: 11/11/2022]
Abstract
OBJECTIVE to characterize the direction within and between brain connectivity in winning and losing players in a competitive brain-computer interface game. APPROACH ten dyads (26.9 ± 4.7 years old, eight females and 12 males) participated in the study. In a competitive game based on neurofeedback, they used their relative alpha (RA) band power from the electrode location Pz, to control a virtual seesaw. The players in each pair were separated into winners (W) and losers (L) based on their scores. Intrabrain connectivity was analyzed using multivariate Granger Causality (GC) and Directed Transfer Function, while interbrain connectivity was analyzed using bivariate GC. RESULTS linear regression analysis revealed a significant relationship (p<0.05) between RA and individual scores. During the game, W players maintained a higher RA than L players, although it was not higher than their baseline RA. The analysis of intrabrain GC indicated that both groups engaged in general social interactions, but only the W group succeeded in controlling their brain activity at Pz. Group L applied an inappropriate metal strategy, characterized by strong activity in the left frontal cortex, indicative of collaborative gaming. Interbrain GC showed a larger flow of information from the L to the W group, suggesting a higher capability of the W group to monitor the activity of their opponent. SIGNIFICANCE both innate neurological indices and gaming mental strategies contribute to game outcomes. Future studies should investigate whether there is a causal relationship between these two factors.
Collapse
Affiliation(s)
- Finda Putri
- Centre for Rehabilitation Engineering, University of Glasgow, James Watt Building (South), G12 8QQ, Glasgow, Glasgow, G12 8QQ, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Ioana Susnoschi Luca
- Centre for Rehabilitation Engineering, University of Glasgow, James Watt Building (South), G12 8QQ, Glasgow, Glasgow, G12 8QQ, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Jorge Abdullah Garcia Pedro
- Centre for Rehabilitation Engineering, University of Glasgow, James Watt Building (South), G12 8QQ, Glasgow, Glasgow, G12 8QQ, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Hao Ding
- Centre for Rehabilitation Engineering, University of Glasgow, James Watt Building (South), G12 8QQ, Glasgow, Glasgow, G12 8QQ, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Aleksandra Vuckovic
- School of Engineering, Biomedical Engineering, University of Glasgow, James Watt building (south), G12 8QQ, Glasgow, Glasgow, G12 8QQ, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| |
Collapse
|
23
|
Tortora S, Beraldo G, Bettella F, Formaggio E, Rubega M, Del Felice A, Masiero S, Carli R, Petrone N, Menegatti E, Tonin L. Neural correlates of user learning during long-term BCI training for the Cybathlon competition. J Neuroeng Rehabil 2022; 19:69. [PMID: 35790978 PMCID: PMC9254548 DOI: 10.1186/s12984-022-01047-x] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2021] [Accepted: 06/22/2022] [Indexed: 11/15/2022] Open
Abstract
Background Brain-computer interfaces (BCIs) are systems capable of translating human brain patterns, measured through electroencephalography (EEG), into commands for an external device. Despite the great advances in machine learning solutions to enhance the performance of BCI decoders, the translational impact of this technology remains elusive. The reliability of BCIs is often unsatisfactory for end-users, limiting their application outside a laboratory environment. Methods We present the analysis on the data acquired from an end-user during the preparation for two Cybathlon competitions, where our pilot won the gold medal twice in a row. These data are of particular interest given the mutual learning approach adopted during the longitudinal training phase (8 months), the long training break in between the two events (1 year) and the demanding evaluation scenario. A multifaceted perspective on long-term user learning is proposed: we enriched the information gathered through conventional metrics (e.g., accuracy, application performances) by investigating novel neural correlates of learning in different neural domains. Results First, we showed that by focusing the training on user learning, the pilot was capable of significantly improving his performance over time even with infrequent decoder re-calibrations. Second, we revealed that the analysis of the within-class modifications of the pilot’s neural patterns in the Riemannian domain is more effective in tracking the acquisition and the stabilization of BCI skills, especially after the 1-year break. These results further confirmed the key role of mutual learning in the acquisition of BCI skills, and particularly highlighted the importance of user learning as a key to enhance BCI reliability. Conclusion We firmly believe that our work may open new perspectives and fuel discussions in the BCI field to shift the focus of future research: not only to the machine learning of the decoder, but also in investigating novel training procedures to boost the user learning and the stability of the BCI skills in the long-term. To this end, the analyses and the metrics proposed could be used to monitor the user learning during training and provide a marker guiding the decoder re-calibration to maximize the mutual adaptation of the user to the BCI system. Supplementary Information The online version contains supplementary material available at 10.1186/s12984-022-01047-x.
Collapse
|
24
|
Lun X, Liu J, Zhang Y, Hao Z, Hou Y. A Motor Imagery Signals Classification Method via the Difference of EEG Signals Between Left and Right Hemispheric Electrodes. Front Neurosci 2022; 16:865594. [PMID: 35615273 PMCID: PMC9124859 DOI: 10.3389/fnins.2022.865594] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2022] [Accepted: 02/25/2022] [Indexed: 11/15/2022] Open
Abstract
Brain-computer interface (BCI) based on motor imagery (MI) can help patients with limb movement disorders in their normal life. In order to develop an efficient BCI system, it is necessary to decode high-accuracy motion intention by electroencephalogram (EEG) with low signal-to-noise ratio. In this article, a MI classification approach is proposed, combining the difference in EEG signals between the left and right hemispheric electrodes with a dual convolutional neural network (dual-CNN), which effectively improved the decoding performance of BCI. The positive and inverse problems of EEG were solved by the boundary element method (BEM) and weighted minimum norm estimation (WMNE), and then the scalp signals were mapped to the cortex layer. We created nine pairs of new electrodes on the cortex as the region of interest. The time series of the nine electrodes on the left and right hemispheric are respectively used as the input of the dual-CNN model to classify four MI tasks. The results show that this method has good results in both group-level subjects and individual subjects. On the Physionet database, the averaged accuracy on group-level can reach 96.36%, while the accuracies of four MI tasks reach 98.54, 95.02, 93.66, and 96.19%, respectively. As for the individual subject, the highest accuracy is 98.88%, and its four MI accuracies are 99.62, 99.68, 98.47, and 97.73%, respectively.
Collapse
Affiliation(s)
- Xiangmin Lun
- School of Automation Engineering, Northeast Electric Power University, Jilin, China
| | - Jianwei Liu
- School of Automation Engineering, Northeast Electric Power University, Jilin, China
| | - Yifei Zhang
- School of Automation Engineering, Northeast Electric Power University, Jilin, China
| | - Ziqian Hao
- School of Finance and Business, Jinan Vocational College, Jinan, China
| | - Yimin Hou
- School of Automation Engineering, Northeast Electric Power University, Jilin, China
- *Correspondence: Yimin Hou
| |
Collapse
|
25
|
Data augmentation for cross-subject EEG features using Siamese neural network. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103614] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
26
|
Giles J, Ang KK, Phua KS, Arvaneh M. A Transfer Learning Algorithm to Reduce Brain-Computer Interface Calibration Time for Long-Term Users. FRONTIERS IN NEUROERGONOMICS 2022; 3:837307. [PMID: 38235467 PMCID: PMC10790953 DOI: 10.3389/fnrgo.2022.837307] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Accepted: 02/22/2022] [Indexed: 01/19/2024]
Abstract
Current motor imagery-based brain-computer interface (BCI) systems require a long calibration time at the beginning of each session before they can be used with adequate levels of classification accuracy. In particular, this issue can be a significant burden for long term BCI users. This article proposes a novel transfer learning algorithm, called r-KLwDSA, to reduce the BCI calibration time for long-term users. The proposed r-KLwDSA algorithm aligns the user's EEG data collected in previous sessions to the few EEG trials collected in the current session, using a novel linear alignment method. Thereafter, the aligned EEG trials from the previous sessions and the few EEG trials from the current sessions are fused through a weighting mechanism before they are used for calibrating the BCI model. To validate the proposed algorithm, a large dataset containing the EEG data from 11 stroke patients, each performing 18 BCI sessions, was used. The proposed framework demonstrated a significant improvement in the classification accuracy, of over 4% compared to the session-specific algorithm, when there were as few as two trials per class available from the current session. The proposed algorithm was particularly successful in improving the BCI accuracy of the sessions that had initial session-specific accuracy below 60%, with an average improvement of around 10% in the accuracy, leading to more stroke patients having meaningful BCI rehabilitation.
Collapse
Affiliation(s)
- Joshua Giles
- Department of Automatic Control and Systems Engineering, The University of Sheffield, Sheffield, United Kingdom
- Institute for Infocomm Research, Agency for Science, Technology and Research, (A*STAR) Singapore, Singapore
| | - Kai Keng Ang
- Institute for Infocomm Research, Agency for Science, Technology and Research, (A*STAR) Singapore, Singapore
- School of Computer Science and Engineering, Nanyang Technological University, Singapore, Singapore
| | - Kok Soon Phua
- Institute for Infocomm Research, Agency for Science, Technology and Research, (A*STAR) Singapore, Singapore
| | - Mahnaz Arvaneh
- Department of Automatic Control and Systems Engineering, The University of Sheffield, Sheffield, United Kingdom
| |
Collapse
|
27
|
Sadatnejad K, Lotte F. Riemannian channel selection for BCI with between-session non-stationarity reduction capabilities. IEEE Trans Neural Syst Rehabil Eng 2022; 30:1158-1171. [PMID: 35420985 DOI: 10.1109/tnsre.2022.3167262] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
OBJECTIVE Between-session non-stationarity is a major challenge of current Brain-Computer Interfaces (BCIs) that affects system performance. In this paper, we investigate the use of channel selection for reducing between-session non-stationarity with Riemannian BCI classifiers. We use the Riemannian geometry framework of covariance matrices due to its robustness and promising performances. Current Riemannian channel selection methods do not consider between-session non-stationarity and are usually tested on a single session. Here, we propose a new channel selection approach that specifically considers non-stationarity effects and is assessed on multi-session BCI data sets. METHODS We remove the least significant channels using a sequential floating backward selection search strategy. Our contributions include: 1) quantifying the non-stationarity effects on brain activity in multi-class problems by different criteria in a Riemannian framework and 2) a method to predict whether BCI performance can improve using channel selection. RESULTS We evaluate the proposed approaches on three multi-session and multi-class mental tasks (MT)-based BCI datasets. They could lead to significant improvements in performance as compared to using all channels for datasets affected by between-session non-stationarity and to significant superiority to the state-of-the-art Riemannian channel selection methods over all datasets, notably when selecting small channel set sizes. CONCLUSION Reducing non-stationarity by channel selection could significantly improve Riemannian BCI classification accuracy. SIGNIFICANCE Our proposed channel selection approach contributes to make Riemannian BCI classifiers more robust to between-session non-stationarities.
Collapse
|
28
|
Huggins JE, Krusienski D, Vansteensel MJ, Valeriani D, Thelen A, Stavisky S, Norton JJS, Nijholt A, Müller-Putz G, Kosmyna N, Korczowski L, Kapeller C, Herff C, Halder S, Guger C, Grosse-Wentrup M, Gaunt R, Dusang AN, Clisson P, Chavarriaga R, Anderson CW, Allison BZ, Aksenova T, Aarnoutse E. Workshops of the Eighth International Brain-Computer Interface Meeting: BCIs: The Next Frontier. BRAIN-COMPUTER INTERFACES 2022; 9:69-101. [PMID: 36908334 PMCID: PMC9997957 DOI: 10.1080/2326263x.2021.2009654] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2021] [Accepted: 11/15/2021] [Indexed: 12/11/2022]
Abstract
The Eighth International Brain-Computer Interface (BCI) Meeting was held June 7-9th, 2021 in a virtual format. The conference continued the BCI Meeting series' interactive nature with 21 workshops covering topics in BCI (also called brain-machine interface) research. As in the past, workshops covered the breadth of topics in BCI. Some workshops provided detailed examinations of specific methods, hardware, or processes. Others focused on specific BCI applications or user groups. Several workshops continued consensus building efforts designed to create BCI standards and increase the ease of comparisons between studies and the potential for meta-analysis and large multi-site clinical trials. Ethical and translational considerations were both the primary topic for some workshops or an important secondary consideration for others. The range of BCI applications continues to expand, with more workshops focusing on approaches that can extend beyond the needs of those with physical impairments. This paper summarizes each workshop, provides background information and references for further study, presents an overview of the discussion topics, and describes the conclusion, challenges, or initiatives that resulted from the interactions and discussion at the workshop.
Collapse
Affiliation(s)
- Jane E Huggins
- Department of Physical Medicine and Rehabilitation, Department of Biomedical Engineering, Neuroscience Graduate Program, University of Michigan, Ann Arbor, Michigan, United States 325 East Eisenhower, Room 3017; Ann Arbor, Michigan 48108-5744, 734-936-7177
| | - Dean Krusienski
- Department of Biomedical Engineering, Virginia Commonwealth University, Richmond, VA 23219
| | - Mariska J Vansteensel
- UMC Utrecht Brain Center, Dept of Neurosurgery, University Medical Center Utrecht, The Netherlands
| | | | - Antonia Thelen
- eemagine Medical Imaging Solutions GmbH, Berlin, Germany
| | | | - James J S Norton
- National Center for Adaptive Neurotechnologies, US Department of Veterans Affairs, 113 Holland Ave, Albany, NY 12208
| | - Anton Nijholt
- Faculty EEMCS, University of Twente, Enschede, The Netherlands
| | - Gernot Müller-Putz
- Institute of Neural Engineering, GrazBCI Lab, Graz University of Technology, Stremayrgasse 16/4, 8010 Graz, Austria
| | - Nataliya Kosmyna
- Massachusetts Institute of Technology (MIT), Media Lab, E14-548, Cambridge, MA 02139, Unites States
| | | | | | - Christian Herff
- School of Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands
| | | | - Christoph Guger
- g.tec medical engineering GmbH/Guger Technologies OG, Austria, Sierningstrasse 14, 4521 Schiedlberg, Austria, +43725122240-0
| | - Moritz Grosse-Wentrup
- Research Group Neuroinformatics, Faculty of Computer Science, Vienna Cognitive Science Hub, Data Science @ Uni Vienna University of Vienna
| | - Robert Gaunt
- Rehab Neural Engineering Labs, Department of Physical Medicine and Rehabilitation, Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA, 3520 5th Ave, Suite 300, Pittsburgh, PA 15213, 412-383-1426
| | - Aliceson Nicole Dusang
- Department of Electrical and Computer Engineering, School of Engineering, Brown University, Carney Institute for Brain Science, Brown University, Providence, RI
- Department of Veterans Affairs Medical Center, Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence, RI
- Center for Neurotechnology and Neurorecovery, Neurology, Massachusetts General Hospital, Boston, MA
| | | | - Ricardo Chavarriaga
- IEEE Standards Association Industry Connections group on neurotechnologies for brain-machine interface, Center for Artificial Intelligence, School of Engineering, ZHAW-Zurich University of Applied Sciences, Switzerland, Switzerland
| | - Charles W Anderson
- Department of Computer Science, Molecular, Cellular and Integrative Neurosience Program, Colorado State University, Fort Collins, CO 80523
| | - Brendan Z Allison
- Dept. of Cognitive Science, Mail Code 0515, University of California at San Diego, La Jolla, United States, 619-534-9754
| | - Tetiana Aksenova
- University Grenoble Alpes, CEA, LETI, Clinatec, Grenoble 38000, France
| | - Erik Aarnoutse
- UMC Utrecht Brain Center, Department of Neurology & Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
| |
Collapse
|
29
|
Dillen A, Steckelmacher D, Efthymiadis K, Langlois K, De Beir A, Marušič U, Vanderborght B, Nowé A, Meeusen R, Ghaffari F, Romain O, De Pauw K. Deep learning for biosignal control: insights from basic to real-time methods with recommendations. J Neural Eng 2022; 19. [PMID: 35086076 DOI: 10.1088/1741-2552/ac4f9a] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2021] [Accepted: 01/27/2022] [Indexed: 11/11/2022]
Abstract
Biosignal control is an interaction modality that allows users to interact with electronic devices by decoding the biological signals emanating from the movements or thoughts of the user. This manner of interaction with devices can enhance the sense of agency for users and enable persons suffering from a paralyzing condition to interact with everyday devices that would otherwise be challenging for them to use. It can also improve control of prosthetic devices and exoskeletons by making the interaction feel more natural and intuitive. However, with the current state of the art, several issues still need to be addressed to reliably decode user intent from biosignals and provide an improved user experience over other interaction modalities. One solution is to leverage advances in Deep Learning (DL) methods to provide more reliable decoding at the expense of added computational complexity. This scoping review introduces the basic concepts of DL and assists readers in deploying DL methods to a real-time control system that should operate under real-world conditions. The scope of this review covers any electronic device, but with an emphasis on robotic devices, as this is the most active area of research in biosignal control. We review the literature pertaining to the implementation and evaluation of control systems that incorporate DL to identify the main gaps and issues in the field, and formulate suggestions on how to mitigate them. Additionally, we formulate guidelines on the best approach to designing, implementing and evaluating research prototypes that use DL in their biosignal control systems.
Collapse
Affiliation(s)
- Arnau Dillen
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| | | | | | - Kevin Langlois
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| | - Albert De Beir
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| | - Uroš Marušič
- Alma Mater Europaea - Evropski Center Maribor, Slovenska ulica 17, Maribor, Maribor, 2000, SLOVENIA
| | - Bram Vanderborght
- Vrije Universiteit Brussel, Faculty of Applied Sciences, Brussel, Brussel, 1050, BELGIUM
| | - Ann Nowé
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| | - Romain Meeusen
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| | - Fakhreddine Ghaffari
- Equipe Traitement de l'Information et Systèmes, CY Cergy Paris University, 6 Rue du Ponceau, Cergy-Pontoise, 95000 , FRANCE
| | - Olivier Romain
- Equipe Traitement de l'Information et Systèmes, CY Cergy Paris University, 6 Rue du Ponceau, Cergy-Pontoise, 95000 , FRANCE
| | - Kevin De Pauw
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| |
Collapse
|
30
|
Sharma R, Kim M, Gupta A. Motor imagery classification in brain-machine interface with machine learning algorithms: Classical approach to multi-layer perceptron model. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103101] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
31
|
Palumbo A, Gramigna V, Calabrese B, Ielpo N. Motor-Imagery EEG-Based BCIs in Wheelchair Movement and Control: A Systematic Literature Review. SENSORS (BASEL, SWITZERLAND) 2021; 21:6285. [PMID: 34577493 PMCID: PMC8473300 DOI: 10.3390/s21186285] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/07/2021] [Revised: 09/09/2021] [Accepted: 09/14/2021] [Indexed: 02/07/2023]
Abstract
The pandemic emergency of the coronavirus disease 2019 (COVID-19) shed light on the need for innovative aids, devices, and assistive technologies to enable people with severe disabilities to live their daily lives. EEG-based Brain-Computer Interfaces (BCIs) can lead individuals with significant health challenges to improve their independence, facilitate participation in activities, thus enhancing overall well-being and preventing impairments. This systematic review provides state-of-the-art applications of EEG-based BCIs, particularly those using motor-imagery (MI) data, to wheelchair control and movement. It presents a thorough examination of the different studies conducted since 2010, focusing on the algorithm analysis, features extraction, features selection, and classification techniques used as well as on wheelchair components and performance evaluation. The results provided in this paper could highlight the limitations of current biomedical instrumentations applied to people with severe disabilities and bring focus to innovative research topics.
Collapse
Affiliation(s)
- Arrigo Palumbo
- Department of Medical and Surgical Sciences, “Magna Græcia” University, 88100 Catanzaro, Italy; (A.P.); (B.C.); (N.I.)
| | - Vera Gramigna
- Neuroscience Research Center, Magna Græcia University, 88100 Catanzaro, Italy
| | - Barbara Calabrese
- Department of Medical and Surgical Sciences, “Magna Græcia” University, 88100 Catanzaro, Italy; (A.P.); (B.C.); (N.I.)
| | - Nicola Ielpo
- Department of Medical and Surgical Sciences, “Magna Græcia” University, 88100 Catanzaro, Italy; (A.P.); (B.C.); (N.I.)
| |
Collapse
|
32
|
Hossaini A, Valeriani D, Nam CS, Ferrante R, Mahmud M. A Functional BCI Model by the P2731 working group: Physiology. BRAIN-COMPUTER INTERFACES 2021. [DOI: 10.1080/2326263x.2021.1968665] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Affiliation(s)
- Ali Hossaini
- Department of Engineering, King’s College London, London, UK
| | | | - Chang S. Nam
- Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, NC, USA
| | | | - Mufti Mahmud
- School of Science and Technology, Nottingham Trent University, Nottingham, UK
| |
Collapse
|
33
|
Robinson N, Chouhan T, Mihelj E, Kratka P, Debraine F, Wenderoth N, Guan C, Lehner R. Design Considerations for Long Term Non-invasive Brain Computer Interface Training With Tetraplegic CYBATHLON Pilot. Front Hum Neurosci 2021; 15:648275. [PMID: 34211380 PMCID: PMC8239283 DOI: 10.3389/fnhum.2021.648275] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2020] [Accepted: 05/17/2021] [Indexed: 11/13/2022] Open
Abstract
Several studies in the recent past have demonstrated how Brain Computer Interface (BCI) technology can uncover the neural mechanisms underlying various tasks and translate them into control commands. While a multitude of studies have demonstrated the theoretic potential of BCI, a point of concern is that the studies are still confined to lab settings and mostly limited to healthy, able-bodied subjects. The CYBATHLON 2020 BCI race represents an opportunity to further develop BCI design strategies for use in real-time applications with a tetraplegic end user. In this study, as part of the preparation to participate in CYBATHLON 2020 BCI race, we investigate the design aspects of BCI in relation to the choice of its components, in particular, the type of calibration paradigm and its relevance for long-term use. The end goal was to develop a user-friendly and engaging interface suited for long-term use, especially for a spinal-cord injured (SCI) patient. We compared the efficacy of conventional open-loop calibration paradigms with real-time closed-loop paradigms, using pre-trained BCI decoders. Various indicators of performance were analyzed for this study, including the resulting classification performance, game completion time, brain activation maps, and also subjective feedback from the pilot. Our results show that the closed-loop calibration paradigms with real-time feedback is more engaging for the pilot. They also show an indication of achieving better online median classification performance as compared to conventional calibration paradigms (p = 0.0008). We also observe that stronger and more localized brain activation patterns are elicited in the closed-loop paradigm in which the experiment interface closely resembled the end application. Thus, based on this longitudinal evaluation of single-subject data, we demonstrate that BCI-based calibration paradigms with active user-engagement, such as with real-time feedback, could help in achieving better user acceptability and performance.
Collapse
Affiliation(s)
- Neethu Robinson
- School of Computer Science and Engineering, Nanyang Technological University, Singapore, Singapore
| | - Tushar Chouhan
- School of Computer Science and Engineering, Nanyang Technological University, Singapore, Singapore.,Future Health Technologies, Singapore-ETH Centre, Singapore, Singapore
| | - Ernest Mihelj
- Neural Control of Movement Lab, Department of Health Science and Technology, Swiss Federal Institute of Technology Zurich, Zurich, Switzerland
| | - Paulina Kratka
- Neural Control of Movement Lab, Department of Health Science and Technology, Swiss Federal Institute of Technology Zurich, Zurich, Switzerland
| | - Frédéric Debraine
- Neural Control of Movement Lab, Department of Health Science and Technology, Swiss Federal Institute of Technology Zurich, Zurich, Switzerland
| | - Nicole Wenderoth
- Future Health Technologies, Singapore-ETH Centre, Singapore, Singapore.,Neural Control of Movement Lab, Department of Health Science and Technology, Swiss Federal Institute of Technology Zurich, Zurich, Switzerland
| | - Cuntai Guan
- School of Computer Science and Engineering, Nanyang Technological University, Singapore, Singapore.,Future Health Technologies, Singapore-ETH Centre, Singapore, Singapore
| | - Rea Lehner
- Future Health Technologies, Singapore-ETH Centre, Singapore, Singapore.,Neural Control of Movement Lab, Department of Health Science and Technology, Swiss Federal Institute of Technology Zurich, Zurich, Switzerland
| |
Collapse
|
34
|
Stieger JR, Engel SA, Suma D, He B. Benefits of deep learning classification of continuous noninvasive brain-computer interface control. J Neural Eng 2021; 18. [PMID: 34038873 DOI: 10.1088/1741-2552/ac0584] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2021] [Accepted: 05/26/2021] [Indexed: 11/12/2022]
Abstract
Objective. Noninvasive brain-computer interfaces (BCIs) assist paralyzed patients by providing access to the world without requiring surgical intervention. Prior work has suggested that EEG motor imagery based BCI can benefit from increased decoding accuracy through the application of deep learning methods, such as convolutional neural networks (CNNs).Approach. Here, we examine whether these improvements can generalize to practical scenarios such as continuous control tasks (as opposed to prior work reporting one classification per trial), whether valuable information remains latent outside of the motor cortex (as no prior work has compared full scalp coverage to motor only electrode montages), and the existing challenges to the practical implementation of deep-learning based continuous BCI control.Main results. We report that: (1) deep learning methods significantly increase offline performance compared to standard methods on an independent, large, and longitudinal online motor imagery BCI dataset with up to 4-classes and continuous 2D feedback; (2) our results suggest that a variety of neural biomarkers for BCI, including those outside the motor cortex, can be detected and used to improve performance through deep learning methods, and (3) tuning neural network output will be an important step in optimizing online BCI control, as we found the CNN models trained with full scalp EEG also significantly reduce the average trial length in a simulated online cursor control environment.Significance. This work demonstrates the benefits of CNNs classification during BCI control while providing evidence that electrode montage selection and the mapping of CNN output to device control will be important design choices in CNN based BCIs.
Collapse
Affiliation(s)
- James R Stieger
- Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, United States of America.,University of Minnesota, Minneapolis, MN, United States of America
| | - Stephen A Engel
- Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, United States of America.,University of Minnesota, Minneapolis, MN, United States of America
| | - Daniel Suma
- Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, United States of America
| | - Bin He
- Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, United States of America
| |
Collapse
|
35
|
Duan X, Xie S, Xie X, Obermayer K, Cui Y, Wang Z. An Online Data Visualization Feedback Protocol for Motor Imagery-Based BCI Training. Front Hum Neurosci 2021; 15:625983. [PMID: 34163337 PMCID: PMC8215169 DOI: 10.3389/fnhum.2021.625983] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2020] [Accepted: 04/29/2021] [Indexed: 11/13/2022] Open
Abstract
Brain-computer interface (BCI) has developed rapidly over the past two decades, mainly due to advancements in machine learning. Subjects must learn to modulate their brain activities to ensure a successful BCI. Feedback training is a practical approach to this learning process; however, the commonly used classifier-dependent approaches have inherent limitations such as the need for calibration and a lack of continuous feedback over long periods of time. This paper proposes an online data visualization feedback protocol that intuitively reflects the EEG distribution in Riemannian geometry in real time. Rather than learning a hyperplane, the Riemannian geometry formulation allows iterative learning of prototypical covariance matrices that are translated into visualized feedback through diffusion map process. Ten subjects were recruited for MI-BCI (motor imagery-BCI) training experiments. The subjects learned to modulate their sensorimotor rhythm to centralize the points within one category and to separate points belonging to different categories. The results show favorable overall training effects in terms of the class distinctiveness and EEG feature discriminancy over a 3-day training with 30% learners. A steadily increased class distinctiveness in the last three sessions suggests that the advanced training protocol is effective. The optimal frequency band was consistent during the 3-day training, and the difference between subjects with good or low MI-BCI performance could be clearly observed. We believe that the proposed feedback protocol has promising application prospect.
Collapse
Affiliation(s)
- Xu Duan
- School of Electronics and Information, Northwestern Polytechnical University, Xi'an, China
| | - Songyun Xie
- School of Electronics and Information, Northwestern Polytechnical University, Xi'an, China
| | - Xinzhou Xie
- School of Electronics and Information, Northwestern Polytechnical University, Xi'an, China
| | - Klaus Obermayer
- Faculty of Electrical Engineering and Computer Science, Technical University Berlin, Berlin, Germany
| | - Yujie Cui
- School of Electronics and Information, Northwestern Polytechnical University, Xi'an, China
| | - Zhenzhen Wang
- School of Electronics and Information, Northwestern Polytechnical University, Xi'an, China
| |
Collapse
|
36
|
Iwane F, Iturrate I, Chavarriaga R, Millán JDR. Invariability of EEG error-related potentials during continuous feedback protocols elicited by erroneous actions at predicted or unpredicted states. J Neural Eng 2021; 18. [PMID: 33882461 DOI: 10.1088/1741-2552/abfa70] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2020] [Accepted: 04/21/2021] [Indexed: 11/11/2022]
Abstract
Objective.When humans perceive an erroneous action, an EEG error-related potential (ErrP) is elicited as a neural response. ErrPs have been largely investigated in discrete feedback protocols, where actions are executed at discrete steps, to enable seamless brain-computer interaction. However, there are only a few studies that investigate ErrPs in continuous feedback protocols. The objective of the present study is to better understand the differences between two types of ErrPs elicited during continuous feedback protocols, where errors may occur either at predicted or unpredicted states. We hypothesize that ErrPs of the unpredicted state is associated with longer latency as it requires higher cognitive workload to evaluate actions compared to the predicted states.Approach.Participants monitored the trajectory of an autonomous cursor that occasionally made erroneous actions on its way to the target in two conditions, namely, predicted or unpredicted states. After characterizing the ErrP waveform elicited by erroneous actions in the two conditions, we performed single-trial decoding of ErrPs in both synchronous (i.e. time-locked to the onset of the erroneous action) and asynchronous manner. Furthermore, we explored the possibility to transfer decoders built with data of one of the conditions to the other condition.Main results.As hypothesized, erroneous actions at unpredicted states gave rise to ErrPs with higher latency than erroneous actions at predicted states, a correlate of higher cognitive effort in the former condition. Moreover, ErrP decoders trained in a given condition successfully transferred to the other condition with a slight loss of classification performance. This was the case for synchronous as well as asynchronous ErrP decoding, showing the invariability of ErrPs across conditions.Significance.These results advance the characterization of ErrPs during continuous feedback protocols, enlarging the potential use of ErrPs during natural operation of brain-controlled devices as it is not necessary to have different decoders for each kind of erroneous conditions.
Collapse
Affiliation(s)
- Fumiaki Iwane
- Learning Algorithms and Systems Laboratory (LASA) , École Polytechnique Féderale de Lausanne (EPFL), 1015 Lausanne, Switzerland.,Department of Electrical and Computer Engineering , The University of Texas at Austin, Austin, TX 78712, United States of America
| | - Iñaki Iturrate
- École Polytechnique Féderale de Lausanne (EPFL), Campus Biotech , 1202 Genève, Switzerland.,Amazon , Barcelona, Spain
| | - Ricardo Chavarriaga
- École Polytechnique Féderale de Lausanne (EPFL), Campus Biotech , 1202 Genève, Switzerland.,ZHAW Datalab , Zurich University of Applied Sciences, Winterthur, Switzerland
| | - José Del R Millán
- Department of Electrical and Computer Engineering , The University of Texas at Austin, Austin, TX 78712, United States of America.,École Polytechnique Féderale de Lausanne (EPFL), Campus Biotech , 1202 Genève, Switzerland.,Department of Neurology , The University of Texas at Austin, Austin, TX 78712, United States of America
| |
Collapse
|
37
|
Kleih-Dahms SC, Botrel L, Kübler A. The influence of motivation and emotion on sensorimotor rhythm-based brain-computer interface performance. Psychophysiology 2021; 58:e13832. [PMID: 33945156 DOI: 10.1111/psyp.13832] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2020] [Revised: 04/01/2021] [Accepted: 04/06/2021] [Indexed: 01/20/2023]
Abstract
While decades of research have investigated and technically improved brain-computer interface (BCI)-controlled applications, relatively little is known about the psychological aspects of brain-computer interfacing. In 35 healthy students, we investigated whether extrinsic motivation manipulated via monetary reward and emotional state manipulated via video and music would influence behavioral and psychophysiological measures of performance with a sensorimotor rhythm (SMR)-based BCI. We found increased task-related brain activity in extrinsically motivated (rewarded) as compared with nonmotivated participants but no clear effect of emotional state manipulation. Our experiment investigated the short-term effect of motivation and emotion manipulation in a group of young healthy subjects, and thus, the significance for patients in the locked-in state, who may be in need of a BCI, remains to be investigated.
Collapse
Affiliation(s)
| | - Loic Botrel
- Institute of Psychology, University of Würzburg, Würzburg, Germany
| | - Andrea Kübler
- Institute of Psychology, University of Würzburg, Würzburg, Germany
| |
Collapse
|
38
|
Liang H, Maedono S, Yu Y, Liu C, Ueda N, Li P, Zhu C. Exploring Neurofeedback Training for BMI Power Augmentation of Upper Limbs: A Pilot Study. ENTROPY (BASEL, SWITZERLAND) 2021; 23:443. [PMID: 33918833 PMCID: PMC8068929 DOI: 10.3390/e23040443] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/25/2021] [Revised: 03/25/2021] [Accepted: 04/06/2021] [Indexed: 11/16/2022]
Abstract
Electroencephalography neurofeedback (EEG-NFB) training can induce changes in the power of targeted EEG bands. The objective of this study is to enhance and evaluate the specific changes of EEG power spectral density that the brain-machine interface (BMI) users can reliably generate for power augmentation through EEG-NFB training. First, we constructed an EEG-NFB training system for power augmentation. Then, three subjects were assigned to three NFB training stages, based on a 6-day consecutive training session as one stage. The subjects received real-time feedback from their EEG signals by a robotic arm while conducting flexion and extension movement with their elbow and shoulder joints, respectively. EEG signals were compared with each NFB training stage. The training results showed that EEG beta (12-40 Hz) power increased after the NFB training for both the elbow and the shoulder joints' movements. EEG beta power showed sustained improvements during the 3-stage training, which revealed that even the short-term training could improve EEG signals significantly. Moreover, the training effect of the shoulder joints was more obvious than that of the elbow joints. These results suggest that NFB training can improve EEG signals and clarify the specific EEG changes during the movement. Our results may even provide insights into how the neural effects of NFB can be better applied to the BMI power augmentation system and improve the performance of healthy individuals.
Collapse
Affiliation(s)
- Hongbo Liang
- Maebashi Institute of Technology, Center for Regional Collaboration, 460-1 Kamisadori, Maebashi, Gunma 371-0816, Japan
| | - Shota Maedono
- Department of Systems Life Engineering, Graduate School of Engineering, Maebashi Institute of Technology, 460-1 Kamisadori, Maebashi, Gunma 371-0816, Japan; (S.M.); (Y.Y.)
| | - Yingxin Yu
- Department of Systems Life Engineering, Graduate School of Engineering, Maebashi Institute of Technology, 460-1 Kamisadori, Maebashi, Gunma 371-0816, Japan; (S.M.); (Y.Y.)
| | - Chang Liu
- Department of Environment and Life Engineering, Graduate School of Engineering, Maebashi Institute of Technology, 460-1 Kamisadori, Maebashi, Gunma 371-0816, Japan; (C.L.); (N.U.); (P.L.)
| | - Naoya Ueda
- Department of Environment and Life Engineering, Graduate School of Engineering, Maebashi Institute of Technology, 460-1 Kamisadori, Maebashi, Gunma 371-0816, Japan; (C.L.); (N.U.); (P.L.)
| | - Peirang Li
- Department of Environment and Life Engineering, Graduate School of Engineering, Maebashi Institute of Technology, 460-1 Kamisadori, Maebashi, Gunma 371-0816, Japan; (C.L.); (N.U.); (P.L.)
| | - Chi Zhu
- Department of Systems Life Engineering, Maebashi Institute of Technology, 460-1 Kamisadori, Maebashi, Gunma 371-0816, Japan
| |
Collapse
|
39
|
Leeuwis N, Paas A, Alimardani M. Vividness of Visual Imagery and Personality Impact Motor-Imagery Brain Computer Interfaces. Front Hum Neurosci 2021; 15:634748. [PMID: 33889080 PMCID: PMC8055841 DOI: 10.3389/fnhum.2021.634748] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2020] [Accepted: 03/08/2021] [Indexed: 12/19/2022] Open
Abstract
Brain-computer interfaces (BCIs) are communication bridges between a human brain and external world, enabling humans to interact with their environment without muscle intervention. Their functionality, therefore, depends on both the BCI system and the cognitive capacities of the user. Motor-imagery BCIs (MI-BCI) rely on the users' mental imagination of body movements. However, not all users have the ability to sufficiently modulate their brain activity for control of a MI-BCI; a problem known as BCI illiteracy or inefficiency. The underlying mechanism of this phenomenon and the cause of such difference among users is yet not fully understood. In this study, we investigated the impact of several cognitive and psychological measures on MI-BCI performance. Fifty-five novice BCI-users participated in a left- versus right-hand motor imagery task. In addition to their BCI classification error rate and demographics, psychological measures including personality factors, affinity for technology, and motivation during the experiment, as well as cognitive measures including visuospatial memory and spatial ability and Vividness of Visual Imagery were collected. Factors that were found to have a significant impact on MI-BCI performance were Vividness of Visual Imagery, and the personality factors of orderliness and autonomy. These findings shed light on individual traits that lead to difficulty in BCI operation and hence can help with early prediction of inefficiency among users to optimize training for them.
Collapse
Affiliation(s)
- Nikki Leeuwis
- Department of Cognitive Science and Artificial Intelligence, Tilburg University, Tilburg, Netherlands
| | | | | |
Collapse
|
40
|
Stieger JR, Engel SA, He B. Continuous sensorimotor rhythm based brain computer interface learning in a large population. Sci Data 2021; 8:98. [PMID: 33795705 PMCID: PMC8016873 DOI: 10.1038/s41597-021-00883-1] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2020] [Accepted: 02/19/2021] [Indexed: 02/01/2023] Open
Abstract
Brain computer interfaces (BCIs) are valuable tools that expand the nature of communication through bypassing traditional neuromuscular pathways. The non-invasive, intuitive, and continuous nature of sensorimotor rhythm (SMR) based BCIs enables individuals to control computers, robotic arms, wheel-chairs, and even drones by decoding motor imagination from electroencephalography (EEG). Large and uniform datasets are needed to design, evaluate, and improve the BCI algorithms. In this work, we release a large and longitudinal dataset collected during a study that examined how individuals learn to control SMR-BCIs. The dataset contains over 600 hours of EEG recordings collected during online and continuous BCI control from 62 healthy adults, (mostly) right hand dominant participants, across (up to) 11 training sessions per participant. The data record consists of 598 recording sessions, and over 250,000 trials of 4 different motor-imagery-based BCI tasks. The current dataset presents one of the largest and most complex SMR-BCI datasets publicly available to date and should be useful for the development of improved algorithms for BCI control.
Collapse
Affiliation(s)
- James R Stieger
- Carnegie Mellon University, Pittsburgh, PA, USA
- University of Minnesota, Minneapolis, MN, USA
| | | | - Bin He
- Carnegie Mellon University, Pittsburgh, PA, USA.
| |
Collapse
|
41
|
Turi F, Clerc M, Papadopoulo T. Long Multi-Stage Training for a Motor-Impaired User in a BCI Competition. Front Hum Neurosci 2021; 15:647908. [PMID: 33841120 PMCID: PMC8027084 DOI: 10.3389/fnhum.2021.647908] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2020] [Accepted: 02/24/2021] [Indexed: 11/15/2022] Open
Abstract
In a Mental Imagery Brain-Computer Interface the user has to perform a specific mental task that generates electroencephalography (EEG) components, which can be translated in commands to control a BCI system. The development of a high-performance MI-BCI requires a long training, lasting several weeks or months, in order to improve the ability of the user to manage his/her mental tasks. This works aims to present the design of a MI-BCI combining mental imaginary and cognitive tasks for a severely motor impaired user, involved in the BCI race of the Cybathlon event, a competition of people with severe motor disability. In the BCI-race, the user becomes a pilot in a virtual race game against up to three other pilots, in which each pilot has to control his/her virtual car by his/her mental tasks. We present all the procedures followed to realize an effective MI-BCI, from the user's first contact with a BCI technology to actually controlling a video-game through her EEG. We defined a multi-stage user-centered training protocol in order to successfully control a BCI, even in a stressful situation, such as that of a competition. We put a specific focus on the human aspects that influenced the long training phase of the system and the participation to the competition.
Collapse
|
42
|
Benaroch C, Sadatnejad K, Roc A, Appriou A, Monseigne T, Pramij S, Mladenovic J, Pillette L, Jeunet C, Lotte F. Long-Term BCI Training of a Tetraplegic User: Adaptive Riemannian Classifiers and User Training. Front Hum Neurosci 2021; 15:635653. [PMID: 33815081 PMCID: PMC8012558 DOI: 10.3389/fnhum.2021.635653] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2020] [Accepted: 02/18/2021] [Indexed: 11/13/2022] Open
Abstract
While often presented as promising assistive technologies for motor-impaired users, electroencephalography (EEG)-based Brain-Computer Interfaces (BCIs) remain barely used outside laboratories due to low reliability in real-life conditions. There is thus a need to design long-term reliable BCIs that can be used outside-of-the-lab by end-users, e.g., severely motor-impaired ones. Therefore, we propose and evaluate the design of a multi-class Mental Task (MT)-based BCI for longitudinal training (20 sessions over 3 months) of a tetraplegic user for the CYBATHLON BCI series 2019. In this BCI championship, tetraplegic pilots are mentally driving a virtual car in a racing video game. We aimed at combining a progressive user MT-BCI training with a newly designed machine learning pipeline based on adaptive Riemannian classifiers shown to be promising for real-life applications. We followed a two step training process: the first 11 sessions served to train the user to control a 2-class MT-BCI by performing either two cognitive tasks (REST and MENTAL SUBTRACTION) or two motor-imagery tasks (LEFT-HAND and RIGHT-HAND). The second training step (9 remaining sessions) applied an adaptive, session-independent Riemannian classifier that combined all 4 MT classes used before. Moreover, as our Riemannian classifier was incrementally updated in an unsupervised way it would capture both within and between-session non-stationarity. Experimental evidences confirm the effectiveness of this approach. Namely, the classification accuracy improved by about 30% at the end of the training compared to initial sessions. We also studied the neural correlates of this performance improvement. Using a newly proposed BCI user learning metric, we could show our user learned to improve his BCI control by producing EEG signals matching increasingly more the BCI classifier training data distribution, rather than by improving his EEG class discrimination. However, the resulting improvement was effective only on synchronous (cue-based) BCI and it did not translate into improved CYBATHLON BCI game performances. For the sake of overcoming this in the future, we unveil possible reasons for these limited gaming performances and identify a number of promising future research directions. Importantly, we also report on the evolution of the user's neurophysiological patterns and user experience throughout the BCI training and competition.
Collapse
Affiliation(s)
- Camille Benaroch
- Inria Bordeaux Sud-Ouest, Talence, France.,LaBRI (CNRS, Univ. Bordeaux, Bordeaux INP), Talence, France
| | | | - Aline Roc
- Inria Bordeaux Sud-Ouest, Talence, France.,LaBRI (CNRS, Univ. Bordeaux, Bordeaux INP), Talence, France
| | - Aurélien Appriou
- Inria Bordeaux Sud-Ouest, Talence, France.,LaBRI (CNRS, Univ. Bordeaux, Bordeaux INP), Talence, France
| | | | | | - Jelena Mladenovic
- Inria Bordeaux Sud-Ouest, Talence, France.,LaBRI (CNRS, Univ. Bordeaux, Bordeaux INP), Talence, France
| | - Léa Pillette
- Inria Bordeaux Sud-Ouest, Talence, France.,LaBRI (CNRS, Univ. Bordeaux, Bordeaux INP), Talence, France
| | - Camille Jeunet
- CLLE Lab, CNRS, Univ. Toulouse Jean Jaurès, Toulouse, France
| | - Fabien Lotte
- Inria Bordeaux Sud-Ouest, Talence, France.,LaBRI (CNRS, Univ. Bordeaux, Bordeaux INP), Talence, France
| |
Collapse
|
43
|
Hehenberger L, Kobler RJ, Lopes-Dias C, Srisrisawang N, Tumfart P, Uroko JB, Torke PR, Müller-Putz GR. Long-Term Mutual Training for the CYBATHLON BCI Race With a Tetraplegic Pilot: A Case Study on Inter-Session Transfer and Intra-Session Adaptation. Front Hum Neurosci 2021; 15:635777. [PMID: 33716698 PMCID: PMC7952767 DOI: 10.3389/fnhum.2021.635777] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2020] [Accepted: 01/27/2021] [Indexed: 12/13/2022] Open
Abstract
CYBATHLON is an international championship where people with severe physical disabilities compete with the aid of state-of-the-art assistive technology. In one of the disciplines, the BCI Race, tetraplegic pilots compete in a computer game race by controlling an avatar with a brain-computer interface (BCI). This competition offers a perfect opportunity for BCI researchers to study long-term training effects in potential end-users, and to evaluate BCI performance in a realistic environment. In this work, we describe the BCI system designed by the team Mirage91 for participation in the CYBATHLON BCI Series 2019, as well as in the CYBATHLON 2020 Global Edition. Furthermore, we present the BCI’s interface with the game and the main methodological strategies, along with a detailed evaluation of its performance over the course of the training period, which lasted 14 months. The developed system was a 4-class BCI relying on task-specific modulations of brain rhythms. We implemented inter-session transfer learning to reduce calibration time, and to reinforce the stability of the brain patterns. Additionally, in order to compensate for potential intra-session shifts in the features’ distribution, normalization parameters were continuously adapted in an unsupervised fashion. Across the aforementioned 14 months, we recorded 26 game-based training sessions. Between the first eight sessions, and the final eight sessions leading up to the CYBATHLON 2020 Global Edition, the runtimes significantly improved from 255 ± 23 s (mean ± std) to 225 ± 22 s, respectively. Moreover, we observed a significant increase in the classifier’s accuracy from 46 to 53%, driven by more distinguishable brain patterns. Compared to conventional single session, non-adaptive BCIs, the inter-session transfer learning and unsupervised intra-session adaptation techniques significantly improved the performance. This long-term study demonstrates that regular training helped the pilot to significantly increase the distance between task-specific patterns, which resulted in an improvement of performance, both with respect to class separability in the calibration data, and with respect to the game. Furthermore, it shows that our methodological approaches were beneficial in transferring the performance across sessions, and most importantly to the CYBATHLON competitions.
Collapse
Affiliation(s)
- Lea Hehenberger
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria.,Graz BCI Racing Team Mirage 91, Graz University of Technology, Graz, Austria
| | - Reinmar J Kobler
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria.,Graz BCI Racing Team Mirage 91, Graz University of Technology, Graz, Austria.,Information Integration and Neuroscience Team, RIKEN Advanced Intelligence Project, Kyoto, Japan
| | - Catarina Lopes-Dias
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria.,Graz BCI Racing Team Mirage 91, Graz University of Technology, Graz, Austria
| | - Nitikorn Srisrisawang
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria.,Graz BCI Racing Team Mirage 91, Graz University of Technology, Graz, Austria
| | - Peter Tumfart
- Graz BCI Racing Team Mirage 91, Graz University of Technology, Graz, Austria
| | - John B Uroko
- Graz BCI Racing Team Mirage 91, Graz University of Technology, Graz, Austria
| | - Paul R Torke
- Graz BCI Racing Team Mirage 91, Graz University of Technology, Graz, Austria
| | - Gernot R Müller-Putz
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria.,Graz BCI Racing Team Mirage 91, Graz University of Technology, Graz, Austria.,BioTechMed Graz, Graz, Austria
| |
Collapse
|
44
|
Aydarkhanov R, Uscumlic M, Chavarriaga R, Gheorghe L, Millan JDR. Closed-loop EEG study on visual recognition during driving. J Neural Eng 2021; 18. [PMID: 33494072 DOI: 10.1088/1741-2552/abdfb2] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2020] [Accepted: 01/25/2021] [Indexed: 11/12/2022]
Abstract
OBJECTIVE In contrast to the classical visual BCI paradigms, which adhere to a rigid trial structure and restricted user behavior, EEG-based visual recognition decoding during our daily activities remains challenging. The objective of this study is to explore the feasibility of decoding the EEG signature of visual recognition in experimental conditions promoting our natural ocular behavior when interacting with our dynamic environment. APPROACH In our experiment, subjects visually search for a target object among suddenly appearing objects in the environment while driving a car-simulator. Given that subjects exhibit an unconstrained overt visual behavior, we based our study on eye fixation-related potentials (EFRP). We report on gaze behavior and single-trial EFRP decoding performance (fixations on visually similar target vs. non-target objects). In addition, we demonstrate the application of our approach in a closed-loop BCI setup. MAIN RESULTS To identify the target out of four symbol types along a road segment, the BCI system integrated decoding probabilities of multiple EFRP and achieved the average online accuracy of 0.37 ± 0.06 (12 subjects), statistically significantly above the chance level. Using the acquired data, we performed a comparative study of classification algorithms (discriminating target vs. non-target) and feature spaces in a simulated online scenario. The EEG approaches yielded similar moderate performances of at most 0.6 AUC, yet statistically significantly above the chance level. In addition, the gaze duration (dwell time) appears to be an additional informative feature in this context. SIGNIFICANCE These results show that visual recognition of sudden events can be decoded during active driving. Therefore, this study lays a foundation for assistive and recommender systems based on the driver's brain signals.
Collapse
Affiliation(s)
- Ruslan Aydarkhanov
- EPFL, EPFL STI IBI-STI MIPLAB, Ch. des Mines 9, Geneva, 1202, SWITZERLAND
| | - Marija Uscumlic
- Nissan International SA, La Pièce 12, Rolle, 1180, SWITZERLAND
| | - Ricardo Chavarriaga
- Forschungsschwerpunkt Information Engineering, ZHAW, Obere Kirchgasse 2 / Steinberggasse 12/14, Winterthur, 8400, SWITZERLAND
| | - Lucian Gheorghe
- Advanced Materials and Processing Laboratory, Nissan Research Center, Nissan Motors Co. LTD, 1, Natsushima, Yokosuka-shi, 237-8523, JAPAN
| | - Jose Del R Millan
- ECE & Neurology, University of Texas at Austin, HDB 5.306, 1601 TRINITY ST BLDG B, Austin, Texas, 78712, UNITED STATES
| |
Collapse
|
45
|
Jiang X, Lopez E, Stieger JR, Greco CM, He B. Effects of Long-Term Meditation Practices on Sensorimotor Rhythm-Based Brain-Computer Interface Learning. Front Neurosci 2021; 14:584971. [PMID: 33551719 PMCID: PMC7858648 DOI: 10.3389/fnins.2020.584971] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2020] [Accepted: 12/21/2020] [Indexed: 12/20/2022] Open
Abstract
Sensorimotor rhythm (SMR)-based brain-computer interfaces (BCIs) provide an alternative pathway for users to perform motor control using motor imagery. Despite the non-invasiveness, ease of use, and low cost, this kind of BCI has limitations due to long training times and BCI inefficiency-that is, the SMR BCI control paradigm may not work well on a subpopulation of users. Meditation is a mental training method to improve mindfulness and awareness and is reported to have positive effects on one's mental state. Here, we investigated the behavioral and electrophysiological differences between experienced meditators and meditation naïve subjects in one-dimensional (1D) and two-dimensional (2D) cursor control tasks. We found numerical evidence that meditators outperformed control subjects in both tasks (1D and 2D), and there were fewer BCI inefficient subjects in the meditator group. Finally, we also explored the neurophysiological difference between the two groups and showed that the meditators had a higher resting SMR predictor, more stable resting mu rhythm, and a larger control signal contrast than controls during the task.
Collapse
Affiliation(s)
- Xiyuan Jiang
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| | - Emily Lopez
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| | - James R. Stieger
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
- Department of Biomedical Engineering, University of Minnesota, Minneapolis, MN, United States
| | - Carol M. Greco
- Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, United States
| | - Bin He
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| |
Collapse
|
46
|
Jeunet C, Hauw D, Millán JDR. Sport Psychology: Technologies Ahead. Front Sports Act Living 2020; 2:10. [PMID: 33345005 PMCID: PMC7739689 DOI: 10.3389/fspor.2020.00010] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2019] [Accepted: 01/24/2020] [Indexed: 11/17/2022] Open
Affiliation(s)
- Camille Jeunet
- CLLE Lab, CNRS, Univ. Toulouse Jean Jaurès, Toulouse, France.,École Polytechnique Fédérale de Lausanne, Geneva, Switzerland
| | - Denis Hauw
- Institut des Sciences du Sport, Université de Lausanne, Lausanne, Switzerland
| | - Jose Del R Millán
- École Polytechnique Fédérale de Lausanne, Geneva, Switzerland.,Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, United States.,Department of Neurology, The University of Texas at Austin, Austin, TX, United States
| |
Collapse
|
47
|
Kinney-Lang E, Kelly D, Floreani ED, Jadavji Z, Rowley D, Zewdie ET, Anaraki JR, Bahari H, Beckers K, Castelane K, Crawford L, House S, Rauh CA, Michaud A, Mussi M, Silver J, Tuck C, Adams K, Andersen J, Chau T, Kirton A. Advancing Brain-Computer Interface Applications for Severely Disabled Children Through a Multidisciplinary National Network: Summary of the Inaugural Pediatric BCI Canada Meeting. Front Hum Neurosci 2020; 14:593883. [PMID: 33343318 PMCID: PMC7744376 DOI: 10.3389/fnhum.2020.593883] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2020] [Accepted: 11/10/2020] [Indexed: 11/15/2022] Open
Abstract
Thousands of youth suffering from acquired brain injury or other early-life neurological disease live, mature, and learn with only limited communication and interaction with their world. Such cognitively capable children are ideal candidates for brain-computer interfaces (BCI). While BCI systems are rapidly evolving, a fundamental gap exists between technological innovators and the patients and families who stand to benefit. Forays into translating BCI systems to children in recent years have revealed that kids can learn to operate simple BCI with proficiency akin to adults. BCI could bring significant boons to the lives of many children with severe physical impairment, supporting their complex physical and social needs. However, children have been neglected in BCI research and a collaborative BCI research community is required to unite and push pediatric BCI development forward. To this end, the pediatric BCI Canada collaborative network (BCI-CAN) was formed, under a unified goal to cooperatively drive forward pediatric BCI innovation and impact. This article reflects on the topics and discussions raised in the foundational BCI-CAN meeting held in Toronto, ON, Canada in November 2019 and suggests the next steps required to see BCI impact the lives of children with severe neurological disease and their families.
Collapse
Affiliation(s)
- Eli Kinney-Lang
- Department of Pediatrics and Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Alberta Children's Hospital Research Institute, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada
| | - Dion Kelly
- Department of Pediatrics and Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Alberta Children's Hospital Research Institute, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada
| | - Erica D Floreani
- Department of Pediatrics and Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Alberta Children's Hospital Research Institute, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada
| | - Zeanna Jadavji
- Department of Pediatrics and Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Alberta Children's Hospital Research Institute, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada
| | - Danette Rowley
- Department of Pediatrics and Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Alberta Children's Hospital Research Institute, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada
| | - Ephrem Takele Zewdie
- Department of Pediatrics and Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Alberta Children's Hospital Research Institute, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada
| | - Javad R Anaraki
- Department of Rehabilitation Science, Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada.,PRISM Laboratory, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
| | - Hosein Bahari
- I CAN Centre, Glenrose Rehabilitation Hospital, Alberta Health Services, Edmonton, AB, Canada
| | - Kim Beckers
- Department of Pediatrics and Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Alberta Children's Hospital Research Institute, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada
| | - Karen Castelane
- PRISM Laboratory, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
| | - Lindsey Crawford
- PRISM Laboratory, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
| | - Sarah House
- PRISM Laboratory, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
| | - Chelsea A Rauh
- PRISM Laboratory, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
| | - Amber Michaud
- I CAN Centre, Glenrose Rehabilitation Hospital, Alberta Health Services, Edmonton, AB, Canada
| | - Matheus Mussi
- Assistive Technology Laboratory, Faculty of Rehabilitation Medicine, University of Alberta, Edmonton, AB, Canada
| | - Jessica Silver
- PRISM Laboratory, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
| | - Corinne Tuck
- I CAN Centre, Glenrose Rehabilitation Hospital, Alberta Health Services, Edmonton, AB, Canada
| | - Kim Adams
- Assistive Technology Laboratory, Faculty of Rehabilitation Medicine, University of Alberta, Edmonton, AB, Canada
| | - John Andersen
- I CAN Centre, Glenrose Rehabilitation Hospital, Alberta Health Services, Edmonton, AB, Canada.,Assistive Technology Laboratory, Faculty of Rehabilitation Medicine, University of Alberta, Edmonton, AB, Canada
| | - Tom Chau
- Department of Rehabilitation Science, Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada.,PRISM Laboratory, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
| | - Adam Kirton
- Department of Pediatrics and Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Alberta Children's Hospital Research Institute, University of Calgary, Calgary, AB, Canada.,Cumming School of Medicine, Hotchkiss Brain Institute, University of Calgary, Calgary, AB, Canada
| |
Collapse
|
48
|
Tortora S, Tonin L, Chisari C, Micera S, Menegatti E, Artoni F. Hybrid Human-Machine Interface for Gait Decoding Through Bayesian Fusion of EEG and EMG Classifiers. Front Neurorobot 2020; 14:582728. [PMID: 33281593 PMCID: PMC7705173 DOI: 10.3389/fnbot.2020.582728] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2020] [Accepted: 09/30/2020] [Indexed: 01/25/2023] Open
Abstract
Despite the advances in the field of brain computer interfaces (BCI), the use of the sole electroencephalography (EEG) signal to control walking rehabilitation devices is currently not viable in clinical settings, due to its unreliability. Hybrid interfaces (hHMIs) represent a very recent solution to enhance the performance of single-signal approaches. These are classification approaches that combine multiple human-machine interfaces, normally including at least one BCI with other biosignals, such as the electromyography (EMG). However, their use for the decoding of gait activity is still limited. In this work, we propose and evaluate a hybrid human-machine interface (hHMI) to decode walking phases of both legs from the Bayesian fusion of EEG and EMG signals. The proposed hHMI significantly outperforms its single-signal counterparts, by providing high and stable performance even when the reliability of the muscular activity is compromised temporarily (e.g., fatigue) or permanently (e.g., weakness). Indeed, the hybrid approach shows a smooth degradation of classification performance after temporary EMG alteration, with more than 75% of accuracy at 30% of EMG amplitude, with respect to the EMG classifier whose performance decreases below 60% of accuracy. Moreover, the fusion of EEG and EMG information helps keeping a stable recognition rate of each gait phase of more than 80% independently on the permanent level of EMG degradation. From our study and findings from the literature, we suggest that the use of hybrid interfaces may be the key to enhance the usability of technologies restoring or assisting the locomotion on a wider population of patients in clinical applications and outside the laboratory environment.
Collapse
Affiliation(s)
- Stefano Tortora
- Department of Information Engineering, University of Padova, Padova, Italy
| | - Luca Tonin
- Department of Information Engineering, University of Padova, Padova, Italy
| | - Carmelo Chisari
- Unit of Neurorehabilitation, Department of Medical Specialties, University Hospital of Pisa, Pisa, Italy
| | - Silvestro Micera
- Department of Excellence in Robotics and AI Scuola Superiore Sant'Anna, The Biorobotics Institute, Pisa, Italy.,Bertarelli Foundation Chair in Translational Neuroengineering, Center for Neuroprosthetics and Institute of Bioengineering, Lausanne, Switzerland
| | - Emanuele Menegatti
- Department of Information Engineering, University of Padova, Padova, Italy
| | - Fiorenzo Artoni
- Bertarelli Foundation Chair in Translational Neuroengineering, Center for Neuroprosthetics and Institute of Bioengineering, Lausanne, Switzerland.,Functional Brain Mapping Laboratory, Department of Basic Neuroscience, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| |
Collapse
|
49
|
Iwama S, Tsuchimoto S, Hayashi M, Mizuguchi N, Ushiba J. Scalp electroencephalograms over ipsilateral sensorimotor cortex reflect contraction patterns of unilateral finger muscles. Neuroimage 2020; 222:117249. [PMID: 32798684 DOI: 10.1016/j.neuroimage.2020.117249] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2020] [Revised: 08/02/2020] [Accepted: 08/06/2020] [Indexed: 12/17/2022] Open
Abstract
A variety of neural substrates are implicated in the initiation, coordination, and stabilization of voluntary movements underpinned by adaptive contraction and relaxation of agonist and antagonist muscles. To achieve such flexible and purposeful control of the human body, brain systems exhibit extensive modulation during the transition from resting state to motor execution and to maintain proper joint impedance. However, the neural structures contributing to such sensorimotor control under unconstrained and naturalistic conditions are not fully characterized. To elucidate which brain regions are implicated in generating and coordinating voluntary movements, we employed a physiologically inspired, two-stage method to decode relaxation and three patterns of contraction in unilateral finger muscles (i.e., extension, flexion, and co-contraction) from high-density scalp electroencephalograms (EEG). The decoder consisted of two parts employed in series. The first discriminated between relaxation and contraction. If the EEG data were discriminated as contraction, the second stage then discriminated among the three contraction patterns. Despite the difficulty in dissociating detailed contraction patterns of muscles within a limb from scalp EEG signals, the decoder performance was higher than chance-level by 2-fold in the four-class classification. Moreover, weighted features in the trained decoders revealed EEG features differentially contributing to decoding performance. During the first stage, consistent with previous reports, weighted features were localized around sensorimotor cortex (SM1) contralateral to the activated fingers, while those during the second stage were localized around ipsilateral SM1. The loci of these weighted features suggested that the coordination of unilateral finger muscles induced different signaling patterns in ipsilateral SM1 contributing to motor control. Weighted EEG features enabled a deeper understanding of human sensorimotor processing as well as of a more naturalistic control of brain-computer interfaces.
Collapse
Affiliation(s)
- Seitaro Iwama
- School of Fundamental Science and Technology, Graduate School of Keio University, Kanagawa, Japan
| | - Shohei Tsuchimoto
- School of Fundamental Science and Technology, Graduate School of Keio University, Kanagawa, Japan; Center of Assistive Robotics and Rehabilitation for Longevity and Good Health, National Center for Geriatrics and Gerontology, Aichi, Japan
| | - Masaaki Hayashi
- School of Fundamental Science and Technology, Graduate School of Keio University, Kanagawa, Japan
| | - Nobuaki Mizuguchi
- Center of Assistive Robotics and Rehabilitation for Longevity and Good Health, National Center for Geriatrics and Gerontology, Aichi, Japan; Department of Biosciences and informatics, Faculty of Science and Technology, Keio University, 3-14-1 Hiyoshi, Kouhoku-ku, Yokohama, Kanagawa 223-8522, Japan
| | - Junichi Ushiba
- Department of Biosciences and informatics, Faculty of Science and Technology, Keio University, 3-14-1 Hiyoshi, Kouhoku-ku, Yokohama, Kanagawa 223-8522, Japan.
| |
Collapse
|
50
|
Roc A, Pillette L, Mladenovic J, Benaroch C, N'Kaoua B, Jeunet C, Lotte F. A review of user training methods in brain computer interfaces based on mental tasks. J Neural Eng 2020; 18. [PMID: 33181488 DOI: 10.1088/1741-2552/abca17] [Citation(s) in RCA: 28] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2020] [Accepted: 11/12/2020] [Indexed: 12/12/2022]
Abstract
Mental-Tasks based Brain-Computer Interfaces (MT-BCIs) allow their users to interact with an external device solely by using brain signals produced through mental tasks. While MT-BCIs are promising for many applications, they are still barely used outside laboratories due to their lack of reliability. MT-BCIs require their users to develop the ability to self-regulate specific brain signals. However, the human learning process to control a BCI is still relatively poorly understood and how to optimally train this ability is currently under investigation. Despite their promises and achievements, traditional training programs have been shown to be sub-optimal and could be further improved. In order to optimize user training and improve BCI performance, human factors should be taken into account. An interdisciplinary approach should be adopted to provide learners with appropriate and/or adaptive training. In this article, we provide an overview of existing methods for MT-BCI user training - notably in terms of environment, instructions, feedback and exercises. We present a categorization and taxonomy of these training approaches, provide guidelines on how to choose the best methods and identify open challenges and perspectives to further improve MT-BCI user training.
Collapse
Affiliation(s)
| | | | | | - Camille Benaroch
- Inria Centre de recherche Bordeaux Sud-Ouest, Talence, 33405, FRANCE
| | - Bernard N'Kaoua
- Handicap, Activity, Cognition, Health, Inserm / University of Bordeaux, Talence, FRANCE
| | | | | |
Collapse
|