1
|
Kim JS, Kim H, Chung CK, Kim JS. Dual model transfer learning to compensate for individual variability in brain-computer interface. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 254:108294. [PMID: 38943984 DOI: 10.1016/j.cmpb.2024.108294] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/18/2024] [Revised: 06/14/2024] [Accepted: 06/16/2024] [Indexed: 07/01/2024]
Abstract
BACKGROUND AND OBJECTIVE Recent advancements in brain-computer interface (BCI) technology have seen a significant shift towards incorporating complex decoding models such as deep neural networks (DNNs) to enhance performance. These models are particularly crucial for sophisticated tasks such as regression for decoding arbitrary movements. However, these BCI models trained and tested on individual data often face challenges with limited performance and generalizability across different subjects. This limitation is primarily due to a tremendous number of parameters of DNN models. Training complex models demands extensive datasets. Nevertheless, group data from many subjects may not produce sufficient decoding performance because of inherent variability in neural signals both across individuals and over time METHODS: To address these challenges, this study proposed a transfer learning approach that could effectively adapt to subject-specific variability in cortical regions. Our method involved training two separate movement decoding models: one on individual data and another on pooled group data. We then created a salience map for each cortical region from the individual model, which helped us identify the input's contribution variance across subjects. Based on the contribution variance, we combined individual and group models using a modified knowledge distillation framework. This approach allowed the group model to be universally applicable by assigning greater weights to input data, while the individual model was fine-tuned to focus on areas with significant individual variance RESULTS: Our combined model effectively encapsulated individual variability. We validated this approach with nine subjects performing arm-reaching tasks, with our method outperforming (mean correlation coefficient, r = 0.75) both individual (r = 0.70) and group models (r = 0.40) in decoding performance. In particular, there were notable improvements in cases where individual models showed low performances (e.g., r = 0.50 in the individual decoder to r = 0.61 in the proposed decoder) CONCLUSIONS: These results not only demonstrate the potential of our method for robust BCI, but also underscore its ability to generalize individual data for broader applicability.
Collapse
Affiliation(s)
- Jun Su Kim
- Dept. of Brain and Cognitive Sciences, Seoul National University, Seoul, Republic of Korea; Clinical Research Institute, Konkuk University Medical Center Seoul, Republic of Korea
| | - HongJune Kim
- Clinical Research Institute, Konkuk University Medical Center Seoul, Republic of Korea
| | - Chun Kee Chung
- Dept. of Brain and Cognitive Sciences, Seoul National University, Seoul, Republic of Korea; Interdisciplinary Program in Neuroscience, Seoul National University, Seoul, Republic of Korea; Dept. of Neurosurgery, Seoul National University Hospital, Seoul, Republic of Korea; Neuroscience Research Institute, Seoul National University Medical Research Center, Seoul, Republic of Korea
| | - June Sic Kim
- Clinical Research Institute, Konkuk University Medical Center Seoul, Republic of Korea; Research Institute of Biomedical Science & Technology, Konkuk University, Seoul, Republic of Korea.
| |
Collapse
|
2
|
Pulferer HS, Guan C, Müller-Putz GR. Investigating multilevel cognitive processing within error-free and error-prone feedback conditions in executed and observed car driving. Front Hum Neurosci 2024; 18:1383956. [PMID: 38993330 PMCID: PMC11236611 DOI: 10.3389/fnhum.2024.1383956] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2024] [Accepted: 06/17/2024] [Indexed: 07/13/2024] Open
Abstract
Accident analyses repeatedly reported the considerable contribution of run-off-road incidents to fatalities in road traffic, and despite considerable advances in assistive technologies to mitigate devastating consequences, little insight into the drivers' brain response during such accident scenarios has been gained. While various literature documents neural correlates to steering motion, the driver's mental state, and the impact of distraction and fatigue on driving performance, the cortical substrate of continuous deviations of a car from the road - i.e., how the brain represents a varying discrepancy between the intended and observed car position and subsequently assigns customized levels of corrective measures - remains unclear. Furthermore, the superposition of multiple subprocesses, such as visual and erroneous feedback processing, performance monitoring, or motor control, complicates a clear interpretation of engaged brain regions within car driving tasks. In the present study, we thus attempted to disentangle these subprocesses, employing passive and active steering conditions within both error-free and error-prone vehicle operation conditions. We recorded EEG signals of 26 participants in 13 sessions, simultaneously measuring pairs of Executors (actively steering) and Observers (strictly observing) during a car driving task. We observed common brain patterns in the Executors regardless of error-free or error-prone vehicle operation, albeit with a shift in spectral activity from motor beta to occipital alpha oscillations within erroneous conditions. Further, significant frontocentral differences between Observers and Executors, tracing back to the caudal anterior cingulate cortex, arose during active steering conditions, indicating increased levels of motor-behavioral cognitive control. Finally, we present regression results of both the steering signal and the car position, indicating that a regression of continuous deviations from the road utilizing the EEG might be feasible.
Collapse
Affiliation(s)
- Hannah S. Pulferer
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Cuntai Guan
- College of Computing and Data Science, Nanyang Technological University, Singapore, Singapore
| | - Gernot R. Müller-Putz
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
- BioTechMed-Graz, Graz, Austria
| |
Collapse
|
3
|
Tao X, Gao D, Zhang W, Liu T, Du B, Zhang S, Qin Y. A multimodal physiological dataset for driving behaviour analysis. Sci Data 2024; 11:378. [PMID: 38609440 PMCID: PMC11014944 DOI: 10.1038/s41597-024-03222-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2023] [Accepted: 04/03/2024] [Indexed: 04/14/2024] Open
Abstract
Physiological signal monitoring and driver behavior analysis have gained increasing attention in both fundamental research and applied research. This study involved the analysis of driving behavior using multimodal physiological data collected from 35 participants. The data included 59-channel EEG, single-channel ECG, 4-channel EMG, single-channel GSR, and eye movement data obtained via a six-degree-of-freedom driving simulator. We categorized driving behavior into five groups: smooth driving, acceleration, deceleration, lane changing, and turning. Through extensive experiments, we confirmed that both physiological and vehicle data met the requirements. Subsequently, we developed classification models, including linear discriminant analysis (LDA), MMPNet, and EEGNet, to demonstrate the correlation between physiological data and driving behaviors. Notably, we propose a multimodal physiological dataset for analyzing driving behavior(MPDB). The MPDB dataset's scale, accuracy, and multimodality provide unprecedented opportunities for researchers in the autonomous driving field and beyond. With this dataset, we will contribute to the field of traffic psychology and behavior.
Collapse
Affiliation(s)
- Xiaoming Tao
- Tsinghua University, Department of Electronic Engineering, Beijing, 100084, China
- Beijing National Research Center for Information Science and Technology (BNRist), 100084, Beijing, China
| | - Dingcheng Gao
- Tsinghua University, Department of Electronic Engineering, Beijing, 100084, China
- Beijing National Research Center for Information Science and Technology (BNRist), 100084, Beijing, China
| | - Wenqi Zhang
- Tsinghua University, Department of Electronic Engineering, Beijing, 100084, China
- Beijing National Research Center for Information Science and Technology (BNRist), 100084, Beijing, China
| | - Tianqi Liu
- Tsinghua University, Department of Electronic Engineering, Beijing, 100084, China
- Beijing National Research Center for Information Science and Technology (BNRist), 100084, Beijing, China
| | - Bing Du
- University of Science and Technology Beijing, School of Computer and Communication Engineering, Beijing, 100083, China
| | - Shanghang Zhang
- National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University, Beijing, 100871, China
| | - Yanjun Qin
- Tsinghua University, Department of Electronic Engineering, Beijing, 100084, China.
- Beijing National Research Center for Information Science and Technology (BNRist), 100084, Beijing, China.
| |
Collapse
|
4
|
Forenzo D, Zhu H, Shanahan J, Lim J, He B. Continuous tracking using deep learning-based decoding for noninvasive brain-computer interface. PNAS NEXUS 2024; 3:pgae145. [PMID: 38689706 PMCID: PMC11060102 DOI: 10.1093/pnasnexus/pgae145] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Accepted: 03/28/2024] [Indexed: 05/02/2024]
Abstract
Brain-computer interfaces (BCI) using electroencephalography provide a noninvasive method for users to interact with external devices without the need for muscle activation. While noninvasive BCIs have the potential to improve the quality of lives of healthy and motor-impaired individuals, they currently have limited applications due to inconsistent performance and low degrees of freedom. In this study, we use deep learning (DL)-based decoders for online continuous pursuit (CP), a complex BCI task requiring the user to track an object in 2D space. We developed a labeling system to use CP data for supervised learning, trained DL-based decoders based on two architectures, including a newly proposed adaptation of the PointNet architecture, and evaluated the performance over several online sessions. We rigorously evaluated the DL-based decoders in a total of 28 human participants, and found that the DL-based models improved throughout the sessions as more training data became available and significantly outperformed a traditional BCI decoder by the last session. We also performed additional experiments to test an implementation of transfer learning by pretraining models on data from other subjects, and midsession training to reduce intersession variability. The results from these experiments showed that pretraining did not significantly improve performance, but updating the models' midsession may have some benefit. Overall, these findings support the use of DL-based decoders for improving BCI performance in complex tasks like CP, which can expand the potential applications of BCI devices and help to improve the quality of lives of healthy and motor-impaired individuals.
Collapse
Affiliation(s)
- Dylan Forenzo
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| | - Hao Zhu
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| | - Jenn Shanahan
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| | - Jaehyun Lim
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| | - Bin He
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| |
Collapse
|
5
|
Mondini V, Sburlea AI, Müller-Putz GR. Towards unlocking motor control in spinal cord injured by applying an online EEG-based framework to decode motor intention, trajectory and error processing. Sci Rep 2024; 14:4714. [PMID: 38413782 PMCID: PMC10899181 DOI: 10.1038/s41598-024-55413-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2023] [Accepted: 02/23/2024] [Indexed: 02/29/2024] Open
Abstract
Brain-computer interfaces (BCIs) can translate brain signals directly into commands for external devices. Electroencephalography (EEG)-based BCIs mostly rely on the classification of discrete mental states, leading to unintuitive control. The ERC-funded project "Feel Your Reach" aimed to establish a novel framework based on continuous decoding of hand/arm movement intention, for a more natural and intuitive control. Over the years, we investigated various aspects of natural control, however, the individual components had not yet been integrated. Here, we present a first implementation of the framework in a comprehensive online study, combining (i) goal-directed movement intention, (ii) trajectory decoding, and (iii) error processing in a unique closed-loop control paradigm. Testing involved twelve able-bodied volunteers, performing attempted movements, and one spinal cord injured (SCI) participant. Similar movement-related cortical potentials and error potentials to previous studies were revealed, and the attempted movement trajectories were overall reconstructed. Source analysis confirmed the involvement of sensorimotor and posterior parietal areas for goal-directed movement intention and trajectory decoding. The increased experiment complexity and duration led to a decreased performance than each single BCI. Nevertheless, the study contributes to understanding natural motor control, providing insights for more intuitive strategies for individuals with motor impairments.
Collapse
Affiliation(s)
- Valeria Mondini
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Andreea-Ioana Sburlea
- Bernoulli Institute for Mathematics, Computer Science and Artificial Intelligence, University of Groningen, Groningen, The Netherlands
| | - Gernot R Müller-Putz
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria.
- BioTechMed, Graz, Austria.
| |
Collapse
|
6
|
Lian J, Qiao X, Zhao Y, Li S, Wang C, Zhou J. EEG-Based Target Detection Using an RSVP Paradigm under Five Levels of Weak Hidden Conditions. Brain Sci 2023; 13:1583. [PMID: 38002543 PMCID: PMC10670035 DOI: 10.3390/brainsci13111583] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2023] [Revised: 11/06/2023] [Accepted: 11/07/2023] [Indexed: 11/26/2023] Open
Abstract
Although target detection based on electroencephalogram (EEG) signals has been extensively investigated recently, EEG-based target detection under weak hidden conditions remains a problem. In this paper, we proposed a rapid serial visual presentation (RSVP) paradigm for target detection corresponding to five levels of weak hidden conditions quantitively based on the RGB color space. Eighteen subjects participated in the experiment, and the neural signatures, including P300 amplitude and latency, were investigated. Detection performance was evaluated under five levels of weak hidden conditions using the linear discrimination analysis and support vector machine classifiers on different channel sets. The experimental results showed that, compared with the benchmark condition, (1) the P300 amplitude significantly decreased (8.92 ± 1.24 μV versus 7.84 ± 1.40 μV, p = 0.021) and latency was significantly prolonged (582.39 ± 25.02 ms versus 643.83 ± 26.16 ms, p = 0.028) only under the weakest hidden condition, and (2) the detection accuracy decreased by less than 2% (75.04 ± 3.24% versus 73.35 ± 3.15%, p = 0.029) with a more than 90% reduction in channel number (62 channels versus 6 channels), determined using the proposed channel selection method under the weakest hidden condition. Our study can provide new insights into target detection under weak hidden conditions based on EEG signals with a rapid serial visual presentation paradigm. In addition, it may expand the application of brain-computer interfaces in EEG-based target detection areas.
Collapse
Affiliation(s)
- Jinling Lian
- Department of Neural Engineering and Biological Interdisciplinary Studies, Beijing Institute of Basic Medical Sciences, 27 Taiping Rd., Beijing 100850, China; (J.L.); (X.Q.); (Y.Z.); (S.L.)
| | - Xin Qiao
- Department of Neural Engineering and Biological Interdisciplinary Studies, Beijing Institute of Basic Medical Sciences, 27 Taiping Rd., Beijing 100850, China; (J.L.); (X.Q.); (Y.Z.); (S.L.)
| | - Yuwei Zhao
- Department of Neural Engineering and Biological Interdisciplinary Studies, Beijing Institute of Basic Medical Sciences, 27 Taiping Rd., Beijing 100850, China; (J.L.); (X.Q.); (Y.Z.); (S.L.)
| | - Siwei Li
- Department of Neural Engineering and Biological Interdisciplinary Studies, Beijing Institute of Basic Medical Sciences, 27 Taiping Rd., Beijing 100850, China; (J.L.); (X.Q.); (Y.Z.); (S.L.)
| | - Changyong Wang
- Department of Neural Engineering and Biological Interdisciplinary Studies, Beijing Institute of Basic Medical Sciences, 27 Taiping Rd., Beijing 100850, China; (J.L.); (X.Q.); (Y.Z.); (S.L.)
| | - Jin Zhou
- Department of Neural Engineering and Biological Interdisciplinary Studies, Beijing Institute of Basic Medical Sciences, 27 Taiping Rd., Beijing 100850, China; (J.L.); (X.Q.); (Y.Z.); (S.L.)
- Chinese Institute for Brain Research, Zhongguancun Life Science Park, Changping District, Beijing 102206, China
| |
Collapse
|
7
|
Borra D, Mondini V, Magosso E, Müller-Putz GR. Decoding movement kinematics from EEG using an interpretable convolutional neural network. Comput Biol Med 2023; 165:107323. [PMID: 37619325 DOI: 10.1016/j.compbiomed.2023.107323] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Revised: 07/28/2023] [Accepted: 08/07/2023] [Indexed: 08/26/2023]
Abstract
Continuous decoding of hand kinematics has been recently explored for the intuitive control of electroencephalography (EEG)-based Brain-Computer Interfaces (BCIs). Deep neural networks (DNNs) are emerging as powerful decoders, for their ability to automatically learn features from lightly pre-processed signals. However, DNNs for kinematics decoding lack in the interpretability of the learned features and are only used to realize within-subject decoders without testing other training approaches potentially beneficial for reducing calibration time, such as transfer learning. Here, we aim to overcome these limitations by using an interpretable convolutional neural network (ICNN) to decode 2-D hand kinematics (position and velocity) from EEG in a pursuit tracking task performed by 13 participants. The ICNN is trained using both within-subject and cross-subject strategies, and also testing the feasibility of transferring the knowledge learned on other subjects on a new one. Moreover, the network eases the interpretation of learned spectral and spatial EEG features. Our ICNN outperformed most of the other state-of-the-art decoders, showing the best trade-off between performance, size, and training time. Furthermore, transfer learning improved kinematics prediction in the low data regime. The network attributed the highest relevance for decoding to the delta-band across all subjects, and to higher frequencies (alpha, beta, low-gamma) for a cluster of them; contralateral central and parieto-occipital sites were the most relevant, reflecting the involvement of sensorimotor, visual and visuo-motor processing. The approach improved the quality of kinematics prediction from the EEG, at the same time allowing interpretation of the most relevant spectral and spatial features.
Collapse
Affiliation(s)
- Davide Borra
- Department of Electrical, Electronic and Information Engineering "Guglielmo Marconi" (DEI), University of Bologna, Cesena Campus, Cesena, Italy.
| | - Valeria Mondini
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Elisa Magosso
- Department of Electrical, Electronic and Information Engineering "Guglielmo Marconi" (DEI), University of Bologna, Cesena Campus, Cesena, Italy; Alma Mater Research Institute for Human-Centered Artificial Intelligence, University of Bologna, Bologna, Italy; Interdepartmental Center for Industrial Research on Health Sciences & Technologies, University of Bologna, Bologna, Italy
| | - Gernot R Müller-Putz
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria; BioTechMed, Graz, Austria
| |
Collapse
|
8
|
Wang P, Cao X, Zhou Y, Gong P, Yousefnezhad M, Shao W, Zhang D. A comprehensive review on motion trajectory reconstruction for EEG-based brain-computer interface. Front Neurosci 2023; 17:1086472. [PMID: 37332859 PMCID: PMC10272365 DOI: 10.3389/fnins.2023.1086472] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2022] [Accepted: 05/03/2023] [Indexed: 06/20/2023] Open
Abstract
The advance in neuroscience and computer technology over the past decades have made brain-computer interface (BCI) a most promising area of neurorehabilitation and neurophysiology research. Limb motion decoding has gradually become a hot topic in the field of BCI. Decoding neural activity related to limb movement trajectory is considered to be of great help to the development of assistive and rehabilitation strategies for motor-impaired users. Although a variety of decoding methods have been proposed for limb trajectory reconstruction, there does not yet exist a review that covers the performance evaluation of these decoding methods. To alleviate this vacancy, in this paper, we evaluate EEG-based limb trajectory decoding methods regarding their advantages and disadvantages from a variety of perspectives. Specifically, we first introduce the differences in motor execution and motor imagery in limb trajectory reconstruction with different spaces (2D and 3D). Then, we discuss the limb motion trajectory reconstruction methods including experiment paradigm, EEG pre-processing, feature extraction and selection, decoding methods, and result evaluation. Finally, we expound on the open problem and future outlooks.
Collapse
Affiliation(s)
| | | | | | | | | | - Wei Shao
- *Correspondence: Wei Shao, ; Daoqiang Zhang,
| | | |
Collapse
|
9
|
Pulferer HS, Kostoglou K, Müller-Putz GR. Getting off track: Cortical feedback processing network modulated by continuous error signal during target-feedback mismatch. Neuroimage 2023; 274:120144. [PMID: 37121373 DOI: 10.1016/j.neuroimage.2023.120144] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Revised: 04/14/2023] [Accepted: 04/27/2023] [Indexed: 05/02/2023] Open
Abstract
Performance monitoring and feedback processing - especially in the wake of erroneous outcomes - represent a crucial aspect of everyday life, allowing us to deal with imminent threats in the short term but also promoting necessary behavioral adjustments in the long term to avoid future conflicts. Over the last thirty years, research extensively analyzed the neural correlates of processing discrete error stimuli, unveiling the error-related negativity (ERN) and error positivity (Pe) as two main components of the cognitive response. However, the connection between the ERN/Pe and distinct stages of error processing, ranging from action monitoring to subsequent corrective behavior, remains ambiguous. Furthermore, mundane actions such as steering a vehicle already transgress the scope of discrete erroneous events and demand fine-tuned feedback control, and thus, the processing of continuous error signals - a topic scarcely researched at present. We analyzed two electroencephalography datasets to investigate the processing of continuous erroneous signals during a target tracking task, employing feedback in various levels and modalities. We observed significant differences between correct (slightly delayed) and erroneous feedback conditions in the larger one of the two datasets that we analyzed, both in sensor and source space. Furthermore, we found strong error-induced modulations that appeared consistent across datasets and error conditions, indicating a clear order of engagement of specific brain regions that correspond to individual components of error processing.
Collapse
Affiliation(s)
- Hannah S Pulferer
- Institute of Neural Engineering, Graz University of Technology, Stremayrgasse 16/IV, Graz, Austria
| | - Kyriaki Kostoglou
- Institute of Neural Engineering, Graz University of Technology, Stremayrgasse 16/IV, Graz, Austria
| | - Gernot R Müller-Putz
- Institute of Neural Engineering, Graz University of Technology, Stremayrgasse 16/IV, Graz, Austria; BioTechMed-Graz, Graz, Austria.
| |
Collapse
|
10
|
Srisrisawang N, Müller-Putz GR. Transfer Learning in Trajectory Decoding: Sensor or Source Space? SENSORS (BASEL, SWITZERLAND) 2023; 23:3593. [PMID: 37050653 PMCID: PMC10098869 DOI: 10.3390/s23073593] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/19/2023] [Revised: 03/08/2023] [Accepted: 03/28/2023] [Indexed: 06/19/2023]
Abstract
In this study, across-participant and across-session transfer learning was investigated to minimize the calibration time of the brain-computer interface (BCI) system in the context of continuous hand trajectory decoding. We reanalyzed data from a study with 10 able-bodied participants across three sessions. A leave-one-participant-out (LOPO) model was utilized as a starting model. Recursive exponentially weighted partial least squares regression (REW-PLS) was employed to overcome the memory limitation due to the large pool of training data. We considered four scenarios: generalized with no update (Gen), generalized with cumulative update (GenC), and individual models with cumulative (IndC) and non-cumulative (Ind) updates, with each one trained with sensor-space features or source-space features. The decoding performance in generalized models (Gen and GenC) was lower than the chance level. In individual models, the cumulative update (IndC) showed no significant improvement over the non-cumulative model (Ind). The performance showed the decoder's incapability to generalize across participants and sessions in this task. The results suggested that the best correlation could be achieved with the sensor-space individual model, despite additional anatomical information in the source-space features. The decoding pattern showed a more localized pattern around the precuneus over three sessions in Ind models.
Collapse
Affiliation(s)
- Nitikorn Srisrisawang
- Institute of Neural Engineering, Graz University of Technology, Stremayrgasse 16/IV, 8010 Graz, Austria
| | - Gernot R. Müller-Putz
- Institute of Neural Engineering, Graz University of Technology, Stremayrgasse 16/IV, 8010 Graz, Austria
- BioTechMed Graz, 8010 Graz, Austria
| |
Collapse
|
11
|
Li G, Liu Y, Chen Y, Li M, Song J, Li K, Zhang Y, Hu L, Qi X, Wan X, Liu J, He Q, Zhou H. Polyvinyl alcohol/polyacrylamide double-network hydrogel-based semi-dry electrodes for robust electroencephalography recording at hairy scalp for noninvasive brain-computer interfaces. J Neural Eng 2023; 20. [PMID: 36863014 DOI: 10.1088/1741-2552/acc098] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2022] [Accepted: 03/02/2023] [Indexed: 03/04/2023]
Abstract
Objective.Reliable and user-friendly electrodes can continuously and real-time capture the electroencephalography (EEG) signals, which is essential for real-life brain-computer interfaces (BCIs). This study develops a flexible, durable, and low-contact-impedance polyvinyl alcohol/polyacrylamide double-network hydrogel (PVA/PAM DNH)-based semi-dry electrode for robust EEG recording at hairy scalp.Approach.The PVA/PAM DNHs are developed using a cyclic freeze-thaw strategy and used as a saline reservoir for semi-dry electrodes. The PVA/PAM DNHs steadily deliver trace amounts of saline onto the scalp, enabling low and stable electrode-scalp impedance. The hydrogel also conforms well to the wet scalp, stabilizing the electrode-scalp interface. The feasibility of the real-life BCIs is validated by conducting four classic BCI paradigms on 16 participants.Main results.The results show that the PVA/PAM DNHs with 7.5 wt% PVA achieve a satisfactory trade-off between the saline load-unloading capacity and the compressive strength. The proposed semi-dry electrode exhibits a low contact impedance (18 ± 8.9 kΩ at 10 Hz), a small offset potential (0.46 mV), and negligible potential drift (1.5 ± 0.4μV min-1). The temporal cross-correlation between the semi-dry and wet electrodes is 0.91, and the spectral coherence is higher than 0.90 at frequencies below 45 Hz. Furthermore, no significant differences are present in BCI classification accuracy between these two typical electrodes.Significance.Based on the durability, rapid setup, wear-comfort, and robust signals of the developed hydrogel, PVA/PAM DNH-based semi-dry electrodes are a promising alternative to wet electrodes in real-life BCIs.
Collapse
Affiliation(s)
- Guangli Li
- Hunan Key Laboratory of Biomedical Nanomaterials and Devices, College of Life Science and Chemistry, Hunan University of Technology, Zhuzhou 412007, People's Republic of China.,Department of Neurology, Zhuzhou People's Hospital, Zhuzhou 412008, People's Republic of China
| | - Ying Liu
- Hunan Key Laboratory of Biomedical Nanomaterials and Devices, College of Life Science and Chemistry, Hunan University of Technology, Zhuzhou 412007, People's Republic of China
| | - Yuwei Chen
- Hunan Key Laboratory of Biomedical Nanomaterials and Devices, College of Life Science and Chemistry, Hunan University of Technology, Zhuzhou 412007, People's Republic of China
| | - Mingzhe Li
- Wuhan Greentek Pty. Ltd, Wuhan 430074, People's Republic of China
| | - Jian Song
- Department of Neurosurgery, General Hospital of Central Command Theater of PLA, Wuhan 430012, People's Republic of China
| | - Kanghua Li
- Department of Neurology, Zhuzhou People's Hospital, Zhuzhou 412008, People's Republic of China
| | - Youmei Zhang
- Department of Child Psychology, The Third Hospital of Zhuzhou, Zhuzhou 412003, People's Republic of China
| | - Le Hu
- Wuhan Greentek Pty. Ltd, Wuhan 430074, People's Republic of China
| | - Xiaoman Qi
- Hunan Key Laboratory of Biomedical Nanomaterials and Devices, College of Life Science and Chemistry, Hunan University of Technology, Zhuzhou 412007, People's Republic of China
| | - Xuan Wan
- Hunan Key Laboratory of Biomedical Nanomaterials and Devices, College of Life Science and Chemistry, Hunan University of Technology, Zhuzhou 412007, People's Republic of China
| | - Jun Liu
- Hunan Key Laboratory of Biomedical Nanomaterials and Devices, College of Life Science and Chemistry, Hunan University of Technology, Zhuzhou 412007, People's Republic of China
| | - Quanguo He
- Hunan Key Laboratory of Biomedical Nanomaterials and Devices, College of Life Science and Chemistry, Hunan University of Technology, Zhuzhou 412007, People's Republic of China
| | - Haihan Zhou
- Key Laboratory of Chemical Biology and Molecular Engineering of Ministry of Education, Institute of Molecular Science, Shanxi University, Taiyuan 030006, People's Republic of China
| |
Collapse
|
12
|
Zhang X, Li H, Dong R, Lu Z, Li C. Electroencephalogram and surface electromyogram fusion-based precise detection of lower limb voluntary movement using convolution neural network-long short-term memory model. Front Neurosci 2022; 16:954387. [PMID: 36213740 PMCID: PMC9538146 DOI: 10.3389/fnins.2022.954387] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2022] [Accepted: 08/26/2022] [Indexed: 11/13/2022] Open
Abstract
The electroencephalogram (EEG) and surface electromyogram (sEMG) fusion has been widely used in the detection of human movement intention for human–robot interaction, but the internal relationship of EEG and sEMG signals is not clear, so their fusion still has some shortcomings. A precise fusion method of EEG and sEMG using the CNN-LSTM model was investigated to detect lower limb voluntary movement in this study. At first, the EEG and sEMG signal processing of each stage was analyzed so that the response time difference between EEG and sEMG can be estimated to detect lower limb voluntary movement, and it can be calculated by the symbolic transfer entropy. Second, the data fusion and feature of EEG and sEMG were both used for obtaining a data matrix of the model, and a hybrid CNN-LSTM model was established for the EEG and sEMG-based decoding model of lower limb voluntary movement so that the estimated value of time difference was about 24 ∼ 26 ms, and the calculated value was between 25 and 45 ms. Finally, the offline experimental results showed that the accuracy of data fusion was significantly higher than feature fusion-based accuracy in 5-fold cross-validation, and the average accuracy of EEG and sEMG data fusion was more than 95%; the improved average accuracy for eliminating the response time difference between EEG and sEMG was about 0.7 ± 0.26% in data fusion. In the meantime, the online average accuracy of data fusion-based CNN-LSTM was more than 87% in all subjects. These results demonstrated that the time difference had an influence on the EEG and sEMG fusion to detect lower limb voluntary movement, and the proposed CNN-LSTM model can achieve high performance. This work provides a stable and reliable basis for human–robot interaction of the lower limb exoskeleton.
Collapse
Affiliation(s)
- Xiaodong Zhang
- School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
- Shaanxi Key Laboratory of Intelligent Robots, Xi’an Jiaotong University, Xi’an, Shaanxi, China
- Wearable Human Enhancement Technology Innovation Center, Xi’an Jiaotong University, Xi’an, Shaanxi, China
| | - Hanzhe Li
- School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
- Wearable Human Enhancement Technology Innovation Center, Xi’an Jiaotong University, Xi’an, Shaanxi, China
- *Correspondence: Hanzhe Li,
| | - Runlin Dong
- School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
| | - Zhufeng Lu
- School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
| | - Cunxin Li
- School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, Shaanxi, China
| |
Collapse
|
13
|
Mirfathollahi A, Ghodrati MT, Shalchyan V, Daliri MR. Decoding locomotion speed and slope from local field potentials of rat motor cortex. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 223:106961. [PMID: 35759821 DOI: 10.1016/j.cmpb.2022.106961] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/01/2021] [Revised: 06/02/2022] [Accepted: 06/14/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND AND OBJECTIVE Local Field Potentials (LFPs) recorded from the primary motor cortex (M1) have been shown to be very informative for decoding movement parameters, and these signals can be used to decode forelimb kinematic and kinetic parameters accurately. Although locomotion is one of the most basic and important motor abilities of humans and animals, the potential of LFPs in decoding abstract hindlimb locomotor parameters has not been investigated. This study investigates the feasibility of decoding speed and slope of locomotion, as two important abstract parameters of walking, using the LFP signals. METHODS Rats were trained to walk smoothly on a treadmill with different speeds and slopes. The brain signals were recorded using the microwire arrays chronically implanted in the hindlimb area of M1 while rats walked on the treadmill. LFP channels were spatially filtered using optimal common spatial patterns to increase the discriminability of speeds and slopes of locomotion. Logarithmic wavelet band powers were extracted as basic features, and the best features were selected using the statistical dependency criterion before classification. RESULTS Using 5 s LFP trials, the average classification accuracies of four different speeds and seven different slopes reached 90.8% and 86.82%, respectively. The high-frequency LFP band (250-500 Hz) was the most informative band about these parameters and contributed more than other frequency bands in the final decoder model. CONCLUSIONS Our results show that the LFP signals in M1 accurately decode locomotion speed and slope, which can be considered as abstract walking parameters needed for designing long-term brain-computer interfaces for hindlimb locomotion control.
Collapse
Affiliation(s)
- Alavie Mirfathollahi
- Neuroscience and Neuroengineering Research Lab, Biomedical Engineering Department, School of Electrical Engineering, Iran University of Science and Technology (IUST), Narmak, Tehran 16846-13114, Iran; Institute for Cognitive Science Studies (ICSS), Tehran, Pardis 16583-44575, Iran
| | - Mohammad Taghi Ghodrati
- Neuroscience and Neuroengineering Research Lab, Biomedical Engineering Department, School of Electrical Engineering, Iran University of Science and Technology (IUST), Narmak, Tehran 16846-13114, Iran
| | - Vahid Shalchyan
- Neuroscience and Neuroengineering Research Lab, Biomedical Engineering Department, School of Electrical Engineering, Iran University of Science and Technology (IUST), Narmak, Tehran 16846-13114, Iran.
| | - Mohammad Reza Daliri
- Neuroscience and Neuroengineering Research Lab, Biomedical Engineering Department, School of Electrical Engineering, Iran University of Science and Technology (IUST), Narmak, Tehran 16846-13114, Iran
| |
Collapse
|
14
|
Pei D, Olikkal P, Adali T, Vinjamuri R. Reconstructing Synergy-Based Hand Grasp Kinematics from Electroencephalographic Signals. SENSORS (BASEL, SWITZERLAND) 2022; 22:5349. [PMID: 35891029 PMCID: PMC9318424 DOI: 10.3390/s22145349] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/13/2022] [Revised: 07/15/2022] [Accepted: 07/16/2022] [Indexed: 06/15/2023]
Abstract
Brain-machine interfaces (BMIs) have become increasingly popular in restoring the lost motor function in individuals with disabilities. Several research studies suggest that the CNS may employ synergies or movement primitives to reduce the complexity of control rather than controlling each DoF independently, and the synergies can be used as an optimal control mechanism by the CNS in simplifying and achieving complex movements. Our group has previously demonstrated neural decoding of synergy-based hand movements and used synergies effectively in driving hand exoskeletons. In this study, ten healthy right-handed participants were asked to perform six types of hand grasps representative of the activities of daily living while their neural activities were recorded using electroencephalography (EEG). From half of the participants, hand kinematic synergies were derived, and a neural decoder was developed, based on the correlation between hand synergies and corresponding cortical activity, using multivariate linear regression. Using the synergies and the neural decoder derived from the first half of the participants and only cortical activities from the remaining half of the participants, their hand kinematics were reconstructed with an average accuracy above 70%. Potential applications of synergy-based BMIs for controlling assistive devices in individuals with upper limb motor deficits, implications of the results in individuals with stroke and the limitations of the study were discussed.
Collapse
|
15
|
de Seta V, Colamarino E, Cincotti F, Mattia D, Mongiardini E, Pichiorri F, Toppi J. Cortico-Muscular Coupling Allows to Discriminate Different Types of Hand Movements. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:2324-2327. [PMID: 36086292 DOI: 10.1109/embc48229.2022.9871383] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Cortico-muscular coupling (CMC) could be used as potential input of a novel hybrid Brain-Computer Interface (hBCI) for motor re-learning after stroke. Here, we aim of addressing the design of a hBCI able to classify different movement tasks taking into account the interplay between the cerebral and residual or recovered muscular activity involved in a given movement. Hence, we compared the performances of four classification methods based on CMC features to evaluate their ability in discriminating finger extension from grasping movements executed by 17 healthy subjects. We also explored how the variation in the dimensionality of the feature domain would influence the different classifier performances. Results showed that, regardless of the model, few CMC features (up to 10) allow for a successful classification of two different movements type. Moreover, support vector machine classifier with linear kernel showed the best trade-off between performances and system usability (few electrodes). Thus, these results suggest that a hBCI based on brain-muscular interplay holds the potential to enable more informed neural plasticity and functional motor recovery after stroke. Furthermore, this CMC-based BCI could also allow for a more "natural control" (l.e., that resembling physiological control) of prosthetic devices.
Collapse
|
16
|
Hosseini SM, Shalchyan V. Continuous Decoding of Hand Movement From EEG Signals Using Phase-Based Connectivity Features. Front Hum Neurosci 2022; 16:901285. [PMID: 35845243 PMCID: PMC9279670 DOI: 10.3389/fnhum.2022.901285] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2022] [Accepted: 06/06/2022] [Indexed: 11/13/2022] Open
Abstract
The principal goal of the brain-computer interface (BCI) is to translate brain signals into meaningful commands to control external devices or neuroprostheses to restore lost functions of patients with severe motor disabilities. The invasive recording of brain signals involves numerous health issues. Therefore, BCIs based on non-invasive recording modalities such as electroencephalography (EEG) are safer and more comfortable for the patients. The BCI requires reconstructing continuous movement parameters such as position or velocity for practical application of neuroprostheses. The BCI studies in continuous decoding have extensively relied on extracting features from the amplitude of brain signals, whereas the brain connectivity features have rarely been explored. This study aims to investigate the feasibility of using phase-based connectivity features in decoding continuous hand movements from EEG signals. To this end, the EEG data were collected from seven healthy subjects performing a 2D center-out hand movement task in four orthogonal directions. The phase-locking value (PLV) and magnitude-squared coherence (MSC) are exploited as connectivity features along with multiple linear regression (MLR) for decoding hand positions. A brute-force search approach is employed to find the best channel pairs for extracting features related to hand movements. The results reveal that the regression models based on PLV and MSC features achieve the average Pearson correlations of 0.43 ± 0.03 and 0.42 ± 0.06, respectively, between predicted and actual trajectories over all subjects. The delta and alpha band features have the most contribution in regression analysis. The results also demonstrate that both PLV and MSC decoding models lead to superior results on our data compared to two recently proposed feature extraction methods solely based on the amplitude or phase of recording signals (p < 0.05). This study verifies the ability of PLV and MSC features in the continuous decoding of hand movements with linear regression. Thus, our findings suggest that extracting features based on brain connectivity can improve the accuracy of trajectory decoder BCIs.
Collapse
|
17
|
Pulferer HS, Ásgeirsdóttir B, Mondini V, Sburlea AI, Müller-Putz GR. Continuous 2D trajectory decoding from attempted movement: across-session performance in able-bodied and feasibility in a spinal cord injured participant. J Neural Eng 2022; 19. [PMID: 35443233 DOI: 10.1088/1741-2552/ac689f] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2021] [Accepted: 04/19/2022] [Indexed: 11/11/2022]
Abstract
OBJECTIVE In people with a cervical spinal cord injury (SCI) or degenerative diseases leading to limited motor function, restoration of upper limb movement has been a goal of the brain-computer interface (BCI) field for decades. Recently, research from our group investigated non-invasive and real-time decoding of continuous movement in able-bodied participants from low-frequency brain signals during a target-tracking task. To advance our setup towards motor-impaired end users, we consequently chose a new paradigm based on attempted movement. APPROACH Here, we present the results of two studies. During the first study, data of ten able-bodied participants completing a target-tracking/shape-tracing task on-screen were investigated in terms of improvements in decoding performance due to user training. In a second study, a spinal cord injured participant underwent the same tasks. To investigate the merit of employing attempted movement in end users with SCI, data of the spinal cord injured participant were recorded twice; once within an observation only condition, and once while simultaneously attempting movement. MAIN RESULTS We observed mean correlation well above chance level for continuous motor decoding based on attempted movement in able-bodied participants. No global improvement over three sessions, both in sensor and source space, could be observed across all participants and movement parameters. In the participant with SCI, decoding performance well above chance was found. SIGNIFICANCE No presence of a learning effect in continuous attempted movement decoding in able-bodied participants could be observed. In contrast, non-significantly varying decoding patterns may promote the use of source space decoding in terms of generalized decoders utilizing transfer learning. Furthermore, above-chance correlations for attempted movement decoding ranging between those of observation only and executed movement were seen in one spinal cord injured participant, suggesting attempted movement decoding as a possible link between feasibility studies in able-bodied and actual applications in motor impaired end users.
Collapse
Affiliation(s)
| | | | - Valeria Mondini
- Institute of Neural Engineering, Graz University of Technology, Stremayrgasse 16/IV, Graz, 8010, AUSTRIA
| | - Andreea Ioana Sburlea
- Institute of Neural Engineering, Technische Universitat Graz, Stremayrgasse 16/IV, 8010 Graz, Austria, Graz, 8010, AUSTRIA
| | | |
Collapse
|
18
|
Śliwowski M, Martin M, Souloumiac A, Blanchart P, Aksenova T. Decoding ECoG signal into 3D hand translation using deep learning. J Neural Eng 2022; 19. [PMID: 35287119 DOI: 10.1088/1741-2552/ac5d69] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2021] [Accepted: 03/14/2022] [Indexed: 12/29/2022]
Abstract
Objective.Motor brain-computer interfaces (BCIs) are a promising technology that may enable motor-impaired people to interact with their environment. BCIs would potentially compensate for arm and hand function loss, which is the top priority for individuals with tetraplegia. Designing real-time and accurate BCI is crucial to make such devices useful, safe, and easy to use by patients in a real-life environment. Electrocorticography (ECoG)-based BCIs emerge as a good compromise between invasiveness of the recording device and good spatial and temporal resolution of the recorded signal. However, most ECoG signal decoders used to predict continuous hand movements are linear models. These models have a limited representational capacity and may fail to capture the relationship between ECoG signal features and continuous hand movements. Deep learning (DL) models, which are state-of-the-art in many problems, could be a solution to better capture this relationship.Approach.In this study, we tested several DL-based architectures to predict imagined 3D continuous hand translation using time-frequency features extracted from ECoG signals. The dataset used in the analysis is a part of a long-term clinical trial (ClinicalTrials.gov identifier: NCT02550522) and was acquired during a closed-loop experiment with a tetraplegic subject. The proposed architectures include multilayer perceptron, convolutional neural networks (CNNs), and long short-term memory networks (LSTM). The accuracy of the DL-based and multilinear models was compared offline using cosine similarity.Main results.Our results show that CNN-based architectures outperform the current state-of-the-art multilinear model. The best architecture exploited the spatial correlation between neighboring electrodes with CNN and benefited from the sequential character of the desired hand trajectory by using LSTMs. Overall, DL increased the average cosine similarity, compared to the multilinear model, by up to 60%, from 0.189 to 0.302 and from 0.157 to 0.249 for the left and right hand, respectively.Significance.This study shows that DL-based models could increase the accuracy of BCI systems in the case of 3D hand translation prediction in a tetraplegic subject.
Collapse
Affiliation(s)
- Maciej Śliwowski
- Université Grenoble Alpes, CEA, LETI, Clinatec, F-38000 Grenoble, France.,Université Paris-Saclay, CEA, List, F-91120 Palaiseau, France
| | - Matthieu Martin
- Université Grenoble Alpes, CEA, LETI, Clinatec, F-38000 Grenoble, France
| | | | | | - Tetiana Aksenova
- Université Grenoble Alpes, CEA, LETI, Clinatec, F-38000 Grenoble, France
| |
Collapse
|
19
|
Srisrisawang N, Müller-Putz GR. Applying Dimensionality Reduction Techniques in Source-Space Electroencephalography via Template and Magnetic Resonance Imaging-Derived Head Models to Continuously Decode Hand Trajectories. Front Hum Neurosci 2022; 16:830221. [PMID: 35399364 PMCID: PMC8988304 DOI: 10.3389/fnhum.2022.830221] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2021] [Accepted: 02/28/2022] [Indexed: 11/13/2022] Open
Abstract
Several studies showed evidence supporting the possibility of hand trajectory decoding from low-frequency electroencephalography (EEG). However, the decoding in the source space via source localization is scarcely investigated. In this study, we tried to tackle the problem of collinearity due to the higher number of signals in the source space by two folds: first, we selected signals in predefined regions of interest (ROIs); second, we applied dimensionality reduction techniques to each ROI. The dimensionality reduction techniques were computing the mean (Mean), principal component analysis (PCA), and locality preserving projections (LPP). We also investigated the effect of decoding between utilizing a template head model and a subject-specific head model during the source localization. The results indicated that applying source-space decoding with PCA yielded slightly higher correlations and signal-to-noise (SNR) ratios than the sensor-space approach. We also observed slightly higher correlations and SNRs when applying the subject-specific head model than the template head model. However, the statistical tests revealed no significant differences between the source-space and sensor-space approaches and no significant differences between subject-specific and template head models. The decoder with Mean and PCA utilizes information mainly from precuneus and cuneus to decode the velocity kinematics similarly in the subject-specific and template head models.
Collapse
Affiliation(s)
| | - Gernot R. Müller-Putz
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
- BioTechMed Graz, Graz, Austria
- *Correspondence: Gernot R. Müller-Putz,
| |
Collapse
|
20
|
Kaeseler RL, Johansson TW, Struijk LNSA, Jochumsen M. Feature- and classification analysis for detection and classification of tongue movements from single-trial pre-movement EEG. IEEE Trans Neural Syst Rehabil Eng 2022; 30:678-687. [PMID: 35290187 DOI: 10.1109/tnsre.2022.3157959] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Individuals with severe tetraplegia can benefit from brain-computer interfaces (BCIs). While most movement-related BCI systems focus on right/left hand and/or foot movements, very few studies have considered tongue movements to construct a multiclass BCI. The aim of this study was to decode four movement directions of the tongue (left, right, up, and down) from single-trial pre-movement EEG and provide a feature and classifier investigation. In offline analyses (from ten healthy participants) detection and classification were performed using temporal, spectral, entropy, and template features classified using either a linear discriminative analysis, support vector machine, random forest or multilayer perceptron classifiers. Besides the 4-class classification scenario, all possible 3-, and 2-class scenarios were tested to find the most discriminable movement type. The linear discriminant analysis achieved on average, higher classification accuracies for both movement detection and classification. The right- and down tongue movements provided the highest and lowest detection accuracy (95.3±4.3% and 91.7±4.8%), respectively. The 4-class classification achieved an accuracy of 62.6±7.2%, while the best 3-class classification (using left, right, and up movements) and 2-class classification (using left and right movements) achieved an accuracy of 75.6±8.4% and 87.7±8.0%, respectively. Using only a combination of the temporal and template feature groups provided further classification accuracy improvements. Presumably, this is because these feature groups utilize the movement-related cortical potentials, which are noticeably different on the left- versus right brain hemisphere for the different movements. This study shows that the cortical representation of the tongue is useful for extracting control signals for multi-class movement detection BCIs.
Collapse
|
21
|
Müller-Putz GR, Kobler RJ, Pereira J, Lopes-Dias C, Hehenberger L, Mondini V, Martínez-Cagigal V, Srisrisawang N, Pulferer H, Batistić L, Sburlea AI. Feel Your Reach: An EEG-Based Framework to Continuously Detect Goal-Directed Movements and Error Processing to Gate Kinesthetic Feedback Informed Artificial Arm Control. Front Hum Neurosci 2022; 16:841312. [PMID: 35360289 PMCID: PMC8961864 DOI: 10.3389/fnhum.2022.841312] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2021] [Accepted: 02/16/2022] [Indexed: 11/13/2022] Open
Abstract
Establishing the basic knowledge, methodology, and technology for a framework for the continuous decoding of hand/arm movement intention was the aim of the ERC-funded project “Feel Your Reach”. In this work, we review the studies and methods we performed and implemented in the last 6 years, which build the basis for enabling severely paralyzed people to non-invasively control a robotic arm in real-time from electroencephalogram (EEG). In detail, we investigated goal-directed movement detection, decoding of executed and attempted movement trajectories, grasping correlates, error processing, and kinesthetic feedback. Although we have tested some of our approaches already with the target populations, we still need to transfer the “Feel Your Reach” framework to people with cervical spinal cord injury and evaluate the decoders’ performance while participants attempt to perform upper-limb movements. While on the one hand, we made major progress towards this ambitious goal, we also critically discuss current limitations.
Collapse
Affiliation(s)
- Gernot R. Müller-Putz
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
- BioTechMed, Graz, Austria
- *Correspondence: Gernot R. Müller-Putz
| | - Reinmar J. Kobler
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
- RIKEN Center for Advanced Intelligence Project, Kyoto, Japan
| | - Joana Pereira
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
- Brain-State Decoding Lab, Albert-Ludwigs-Universität Freiburg, Freiburg, Germany
- Stereotaxy and Functional Neurosurgery Department, Uniklinik Freiburg, Freiburg, Germany
| | - Catarina Lopes-Dias
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Lea Hehenberger
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Valeria Mondini
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Víctor Martínez-Cagigal
- Biomedical Engineering Group, E.T.S. Ingenieros de Telecomunicación, University of Valladolid, Valladolid, Spain
- Biomedical Research Networking Center in Bioengineering, Biomaterials and Nanomedicine (CIBER-BBN), Valladolid, Spain
| | | | - Hannah Pulferer
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Luka Batistić
- Faculty of Engineering, Department of Computer Engineering, University of Rijeka, Rijeka, Croatia
| | - Andreea I. Sburlea
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| |
Collapse
|
22
|
Continuous Hybrid BCI Control for Robotic Arm Using Noninvasive Electroencephalogram, Computer Vision, and Eye Tracking. MATHEMATICS 2022. [DOI: 10.3390/math10040618] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/10/2022]
Abstract
The controlling of robotic arms based on brain–computer interface (BCI) can revolutionize the quality of life and living conditions for individuals with physical disabilities. Invasive electroencephalography (EEG)-based BCI has been able to control multiple degrees of freedom (DOFs) robotic arms in three dimensions. However, it is still hard to control a multi-DOF robotic arm to reach and grasp the desired target accurately in complex three-dimensional (3D) space by a noninvasive system mainly due to the limitation of EEG decoding performance. In this study, we propose a noninvasive EEG-based BCI for a robotic arm control system that enables users to complete multitarget reach and grasp tasks and avoid obstacles by hybrid control. The results obtained from seven subjects demonstrated that motor imagery (MI) training could modulate brain rhythms, and six of them completed the online tasks using the hybrid-control-based robotic arm system. The proposed system shows effective performance due to the combination of MI-based EEG, computer vision, gaze detection, and partially autonomous guidance, which drastically improve the accuracy of online tasks and reduce the brain burden caused by long-term mental activities.
Collapse
|
23
|
Huggins JE, Krusienski D, Vansteensel MJ, Valeriani D, Thelen A, Stavisky S, Norton JJS, Nijholt A, Müller-Putz G, Kosmyna N, Korczowski L, Kapeller C, Herff C, Halder S, Guger C, Grosse-Wentrup M, Gaunt R, Dusang AN, Clisson P, Chavarriaga R, Anderson CW, Allison BZ, Aksenova T, Aarnoutse E. Workshops of the Eighth International Brain-Computer Interface Meeting: BCIs: The Next Frontier. BRAIN-COMPUTER INTERFACES 2022; 9:69-101. [PMID: 36908334 PMCID: PMC9997957 DOI: 10.1080/2326263x.2021.2009654] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2021] [Accepted: 11/15/2021] [Indexed: 12/11/2022]
Abstract
The Eighth International Brain-Computer Interface (BCI) Meeting was held June 7-9th, 2021 in a virtual format. The conference continued the BCI Meeting series' interactive nature with 21 workshops covering topics in BCI (also called brain-machine interface) research. As in the past, workshops covered the breadth of topics in BCI. Some workshops provided detailed examinations of specific methods, hardware, or processes. Others focused on specific BCI applications or user groups. Several workshops continued consensus building efforts designed to create BCI standards and increase the ease of comparisons between studies and the potential for meta-analysis and large multi-site clinical trials. Ethical and translational considerations were both the primary topic for some workshops or an important secondary consideration for others. The range of BCI applications continues to expand, with more workshops focusing on approaches that can extend beyond the needs of those with physical impairments. This paper summarizes each workshop, provides background information and references for further study, presents an overview of the discussion topics, and describes the conclusion, challenges, or initiatives that resulted from the interactions and discussion at the workshop.
Collapse
Affiliation(s)
- Jane E Huggins
- Department of Physical Medicine and Rehabilitation, Department of Biomedical Engineering, Neuroscience Graduate Program, University of Michigan, Ann Arbor, Michigan, United States 325 East Eisenhower, Room 3017; Ann Arbor, Michigan 48108-5744, 734-936-7177
| | - Dean Krusienski
- Department of Biomedical Engineering, Virginia Commonwealth University, Richmond, VA 23219
| | - Mariska J Vansteensel
- UMC Utrecht Brain Center, Dept of Neurosurgery, University Medical Center Utrecht, The Netherlands
| | | | - Antonia Thelen
- eemagine Medical Imaging Solutions GmbH, Berlin, Germany
| | | | - James J S Norton
- National Center for Adaptive Neurotechnologies, US Department of Veterans Affairs, 113 Holland Ave, Albany, NY 12208
| | - Anton Nijholt
- Faculty EEMCS, University of Twente, Enschede, The Netherlands
| | - Gernot Müller-Putz
- Institute of Neural Engineering, GrazBCI Lab, Graz University of Technology, Stremayrgasse 16/4, 8010 Graz, Austria
| | - Nataliya Kosmyna
- Massachusetts Institute of Technology (MIT), Media Lab, E14-548, Cambridge, MA 02139, Unites States
| | | | | | - Christian Herff
- School of Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands
| | | | - Christoph Guger
- g.tec medical engineering GmbH/Guger Technologies OG, Austria, Sierningstrasse 14, 4521 Schiedlberg, Austria, +43725122240-0
| | - Moritz Grosse-Wentrup
- Research Group Neuroinformatics, Faculty of Computer Science, Vienna Cognitive Science Hub, Data Science @ Uni Vienna University of Vienna
| | - Robert Gaunt
- Rehab Neural Engineering Labs, Department of Physical Medicine and Rehabilitation, Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA, 3520 5th Ave, Suite 300, Pittsburgh, PA 15213, 412-383-1426
| | - Aliceson Nicole Dusang
- Department of Electrical and Computer Engineering, School of Engineering, Brown University, Carney Institute for Brain Science, Brown University, Providence, RI
- Department of Veterans Affairs Medical Center, Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence, RI
- Center for Neurotechnology and Neurorecovery, Neurology, Massachusetts General Hospital, Boston, MA
| | | | - Ricardo Chavarriaga
- IEEE Standards Association Industry Connections group on neurotechnologies for brain-machine interface, Center for Artificial Intelligence, School of Engineering, ZHAW-Zurich University of Applied Sciences, Switzerland, Switzerland
| | - Charles W Anderson
- Department of Computer Science, Molecular, Cellular and Integrative Neurosience Program, Colorado State University, Fort Collins, CO 80523
| | - Brendan Z Allison
- Dept. of Cognitive Science, Mail Code 0515, University of California at San Diego, La Jolla, United States, 619-534-9754
| | - Tetiana Aksenova
- University Grenoble Alpes, CEA, LETI, Clinatec, Grenoble 38000, France
| | - Erik Aarnoutse
- UMC Utrecht Brain Center, Department of Neurology & Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
| |
Collapse
|
24
|
Yeom HG, Jeong H. F-Value Time-Frequency Analysis: Between-Within Variance Analysis. Front Neurosci 2021; 15:729449. [PMID: 34955709 PMCID: PMC8697975 DOI: 10.3389/fnins.2021.729449] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Accepted: 11/22/2021] [Indexed: 11/23/2022] Open
Abstract
Studies on brain mechanisms enable us to treat various brain diseases and develop diverse technologies for daily life. Therefore, an analysis method of neural signals is critical, as it provides the basis for many brain studies. In many cases, researchers want to understand how neural signals change according to different conditions. However, it is challenging to find distinguishing characteristics, and doing so requires complex statistical analysis. In this study, we propose a novel analysis method, FTF (F-value time-frequency) analysis, that applies the F-value of ANOVA to time-frequency analysis. The proposed method shows the statistical differences among conditions in time and frequency. To evaluate the proposed method, electroencephalography (EEG) signals were analyzed using the proposed FTF method. The EEG signals were measured during imagined movement of the left hand, right hand, foot, and tongue. The analysis revealed the important characteristics which were different among different conditions and similar within the same condition. The FTF analysis method will be useful in various fields, as it allows researchers to analyze how frequency characteristics vary according to different conditions.
Collapse
Affiliation(s)
- Hong Gi Yeom
- Department of Electronics Engineering, Chosun University, Gwangju, South Korea
| | - Hyundoo Jeong
- Department of Mechatronics Engineering, Incheon National University, Incheon, South Korea
| |
Collapse
|
25
|
Robust anticipation of continuous steering actions from electroencephalographic data during simulated driving. Sci Rep 2021; 11:23383. [PMID: 34862442 PMCID: PMC8642531 DOI: 10.1038/s41598-021-02750-w] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2021] [Accepted: 10/26/2021] [Indexed: 11/08/2022] Open
Abstract
Driving a car requires high cognitive demands, from sustained attention to perception and action planning. Recent research investigated the neural processes reflecting the planning of driving actions, aiming to better understand the factors leading to driving errors and to devise methodologies to anticipate and prevent such errors by monitoring the driver’s cognitive state and intention. While such anticipation was shown for discrete driving actions, such as emergency braking, there is no evidence for robust neural signatures of continuous action planning. This study aims to fill this gap by investigating continuous steering actions during a driving task in a car simulator with multimodal recordings of behavioural and electroencephalography (EEG) signals. System identification is used to assess whether robust neurophysiological signatures emerge before steering actions. Linear decoding models are then used to determine whether such cortical signals can predict continuous steering actions with progressively longer anticipation. Results point to significant EEG signatures of continuous action planning. Such neural signals show consistent dynamics across participants for anticipations up to 1 s, while individual-subject neural activity could reliably decode steering actions and predict future actions for anticipations up to 1.8 s. Finally, we use canonical correlation analysis to attempt disentangling brain and non-brain contributors to the EEG-based decoding. Our results suggest that low-frequency cortical dynamics are involved in the planning of steering actions and that EEG is sensitive to that neural activity. As a result, we propose a framework to investigate anticipatory neural activity in realistic continuous motor tasks.
Collapse
|
26
|
Colamarino E, de Seta V, Masciullo M, Cincotti F, Mattia D, Pichiorri F, Toppi J. Corticomuscular and Intermuscular Coupling in Simple Hand Movements to Enable a Hybrid Brain-Computer Interface. Int J Neural Syst 2021; 31:2150052. [PMID: 34590990 DOI: 10.1142/s0129065721500520] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
Hybrid Brain-Computer Interfaces (BCIs) for upper limb rehabilitation after stroke should enable the reinforcement of "more normal" brain and muscular activity. Here, we propose the combination of corticomuscular coherence (CMC) and intermuscular coherence (IMC) as control features for a novel hybrid BCI for rehabilitation purposes. Multiple electroencephalographic (EEG) signals and surface electromyography (EMG) from 5 muscles per side were collected in 20 healthy participants performing finger extension (Ext) and grasping (Grasp) with both dominant and non-dominant hand. Grand average of CMC and IMC patterns showed a bilateral sensorimotor area as well as multiple muscles involvement. CMC and IMC values were used as features to classify each task versus rest and Ext versus Grasp. We demonstrated that a combination of CMC and IMC features allows for classification of both movements versus rest with better performance (Area Under the receiver operating characteristic Curve, AUC) for the Ext movement (0.97) with respect to Grasp (0.88). Classification of Ext versus Grasp also showed high performances (0.99). All in all, these preliminary findings indicate that the combination of CMC and IMC could provide for a comprehensive framework for simple hand movements to eventually be employed in a hybrid BCI system for post-stroke rehabilitation.
Collapse
Affiliation(s)
- Emma Colamarino
- Department of Computer, Control and Management Engineering, Sapienza University of Rome, Via Ariosto 25, Rome 00185, Italy.,Fondazione Santa Lucia IRCCS, Via Ardeatina 306-354, Rome 00179, Italy
| | - Valeria de Seta
- Department of Computer, Control and Management Engineering, Sapienza University of Rome, Via Ariosto 25, Rome 00185, Italy.,Fondazione Santa Lucia IRCCS, Via Ardeatina 306-354, Rome 00179, Italy
| | | | - Febo Cincotti
- Department of Computer, Control and Management Engineering, Sapienza University of Rome, Via Ariosto 25, Rome 00185, Italy.,Fondazione Santa Lucia IRCCS, Via Ardeatina 306-354, Rome 00179, Italy
| | - Donatella Mattia
- Fondazione Santa Lucia IRCCS, Via Ardeatina 306-354, Rome 00179, Italy
| | | | - Jlenia Toppi
- Department of Computer, Control and Management Engineering, Sapienza University of Rome, Via Ariosto 25, Rome 00185, Italy.,Fondazione Santa Lucia IRCCS, Via Ardeatina 306-354, Rome 00179, Italy
| |
Collapse
|
27
|
Pereira J, Kobler R, Ofner P, Schwarz A, Müller-Putz GR. Online detection of movement during natural and self-initiated reach-and-grasp actions from EEG signals. J Neural Eng 2021; 18. [PMID: 34130267 DOI: 10.1088/1741-2552/ac0b52] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2021] [Accepted: 06/15/2021] [Indexed: 11/11/2022]
Abstract
Movement intention detection using electroencephalography (EEG) is a challenging but essential component of brain-computer interfaces (BCIs) for people with motor disabilities.Objective.The goal of this study is to develop a new experimental paradigm to perform asynchronous online detection of movement based on low-frequency time-domain EEG features, concretely on movement-related cortical potentials. The paradigm must be easily transferable to people without any residual upper-limb movement function and the BCI must be independent of upper-limb movement onset measurements and external cues.Approach. In a study with non-disabled participants, we evaluated a novel BCI paradigm to detect self-initiated reach-and-grasp movements. Two experimental conditions were involved. In one condition, participants performed reach-and-grasp movements to a target and simultaneously shifted their gaze towards it. In a control condition, participants solely shifted their gaze towards the target (oculomotor task). The participants freely decided when to initiate the tasks. After eye artefact correction, the EEG signals were time-locked to the saccade onset and the resulting amplitude features were exploited on a hierarchical classification approach to detect movement asynchronously.Main results. With regards to BCI performance, 54.1% (14.4% SD) of the movements were correctly identified, and all participants achieved a performance above chance-level (around 12%). An average of 21.5% (14.1% SD) of the oculomotor tasks were falsely detected as upper-limb movement. In an additional rest condition, 1.7 (1.6 SD) false positives per minute were measured. Through source imaging, movement information was mapped to sensorimotor, posterior parietal and occipital areas.Significance. We present a novel approach for movement detection using EEG signals which does not rely on upper-limb movement onset measurements or on the presentation of external cues. The participants' behaviour closely matches the natural behaviour during goal-directed reach-and-grasp movements, which also constitutes an advantage with respect to current BCI protocols.
Collapse
Affiliation(s)
- Joana Pereira
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Reinmar Kobler
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Patrick Ofner
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Andreas Schwarz
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | | |
Collapse
|
28
|
Sosnik R, Li Z. Reconstruction of hand, elbow and shoulder actual and imagined trajectories in 3D space using EEG current source dipoles. J Neural Eng 2021; 18. [PMID: 33752186 DOI: 10.1088/1741-2552/abf0d7] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2020] [Accepted: 03/22/2021] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Growing evidence suggests that EEG electrode (sensor) potential time series (PTS) of slow cortical potentials (SCPs) hold motor neural correlates that can be used for motion trajectory prediction (MTP), commonly by multiple linear regression (mLR). It is not yet known whether arm-joint trajectories can be reliably decoded from current sources, computed from sensor data, from which brain areas they can be decoded and using which neural features. APPROACH In this study, the PTS of 44 sensors were fed into sLORETA source localization software to compute current source activity in 30 regions of interest (ROIs) found in a recent meta-analysis to be engaged in action execution, motor imagery and motor preparation. The current sources PTS and band-power time series (BTS) in several frequency bands and time lags were used to predict actual and imagined trajectories in 3D space of the three velocity components of the hand, elbow and shoulder of nine subjects using an mLR model. MAIN RESULTS For all arm joints and movement types, current source SCPs PTS contributed most to trajectory reconstruction with time lags 150ms, 116ms and 84ms providing the highest contribution, and current source BTS in any of the tested frequency bands was not informative. Person's correlation coefficient (r) averaged across movement types, arm joints and velocity components using source data was slightly lower than using sensor data (r=0.25 and r=0.28, respectively). For each ROI, the three current source dipoles had different contribution to the reconstruction of each of the three velocity components. SIGNIFICANCE Overall, our results demonstrate the feasibility of predicting of actual and imagined 3D trajectories of all arm joints from current sources, computed from scalp EEG. These findings may be used by developers of a future BCI as a validated set of contributing ROIs.
Collapse
Affiliation(s)
- Ronen Sosnik
- Electrical, Electronics and Communication Engineering, Holon Institute of Technology, 52 Golomb St., Holon, 5810201, ISRAEL
| | - Zheng Li
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Yingdong Building, Xinjiekouwai Street 19, Beijing Haidian, Beijing, 100875, CHINA
| |
Collapse
|
29
|
Valenti A, Barsotti M, Bacciu D, Ascari L. A Deep Classifier for Upper-Limbs Motor Anticipation Tasks in an Online BCI Setting. Bioengineering (Basel) 2021; 8:bioengineering8020021. [PMID: 33562814 PMCID: PMC7915535 DOI: 10.3390/bioengineering8020021] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2020] [Revised: 01/25/2021] [Accepted: 01/28/2021] [Indexed: 11/16/2022] Open
Abstract
Decoding motor intentions from non-invasive brain activity monitoring is one of the most challenging aspects in the Brain Computer Interface (BCI) field. This is especially true in online settings, where classification must be performed in real-time, contextually with the user’s movements. In this work, we use a topology-preserving input representation, which is fed to a novel combination of 3D-convolutional and recurrent deep neural networks, capable of performing multi-class continual classification of subjects’ movement intentions. Our model is able to achieve a higher accuracy than a related state-of-the-art model from literature, despite being trained in a much more restrictive setting and using only a simple form of input signal preprocessing. The results suggest that deep learning models are well suited for deployment in challenging real-time BCI applications such as movement intention recognition.
Collapse
Affiliation(s)
- Andrea Valenti
- Department of Computer Science, University of Pisa, 56127 Pisa, Italy;
- Correspondence:
| | | | - Davide Bacciu
- Department of Computer Science, University of Pisa, 56127 Pisa, Italy;
| | - Luca Ascari
- CAMLIN Italy s.r.l., 43121 Parma, Italy; (M.B.); (L.A.)
| |
Collapse
|
30
|
Martinez-Cagigal V, Kobler RJ, Mondini V, Hornero R, Muller-Putz GR. Non-linear online low-frequency EEG decoding of arm movements during a pursuit tracking task. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2020:2981-2985. [PMID: 33018632 DOI: 10.1109/embc44109.2020.9175723] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Decoding upper-limb movements in invasive recordings has become a reality, but neural tuning in non-invasive low-frequency recordings is still under discussion. Recent studies managed to decode movement positions and velocities using linear decoders, even developing an online system. The decoded signals, however, exhibited smaller amplitudes than actual movements, affecting feedback and user experience. Recently, we showed that a non-linear offline decoder can combine directional (e.g., velocity) and non-directional (e.g., speed) information. In this study, it is assessed if the non-linear decoder can be used online to provide real-time feedback. Five healthy subjects were asked to track a moving target by controlling a robotic arm. Initially, the robot was controlled by their right hand; then, the control was gradually switched until it was entirely controlled by the electroencephalogram (EEG). Correlations between actual and decoded movements were generally above chance level. Results suggest that information about speed was also encoded in the EEG, demonstrating that the proposed non-linear decoder is suitable for decoding real-time arm movements.
Collapse
|