1
|
Yang SH, Huang CJ, Huang JS. Increasing Robustness of Intracortical Brain-Computer Interfaces for Recording Condition Changes via Data Augmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 251:108208. [PMID: 38754326 DOI: 10.1016/j.cmpb.2024.108208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/08/2024] [Accepted: 04/30/2024] [Indexed: 05/18/2024]
Abstract
BACKGROUND AND OBJECTIVE Intracortical brain-computer interfaces (iBCIs) aim to help paralyzed individuals restore their motor functions by decoding neural activity into intended movement. However, changes in neural recording conditions hinder the decoding performance of iBCIs, mainly because the neural-to-kinematic mappings shift. Conventional approaches involve either training the neural decoders using large datasets before deploying the iBCI or conducting frequent calibrations during its operation. However, collecting data for extended periods can cause user fatigue, negatively impacting the quality and consistency of neural signals. Furthermore, frequent calibration imposes a substantial computational load. METHODS This study proposes a novel approach to increase iBCIs' robustness against changing recording conditions. The approach uses three neural augmentation operators to generate augmented neural activity that mimics common recording conditions. Then, contrastive learning is used to learn latent factors by maximizing the similarity between the augmented neural activities. The learned factors are expected to remain stable despite varying recording conditions and maintain a consistent correlation with the intended movement. RESULTS Experimental results demonstrate that the proposed iBCI outperformed the state-of-the-art iBCIs and was robust to changing recording conditions across days for long-term use on one publicly available nonhuman primate dataset. It achieved satisfactory offline decoding performance, even when a large training dataset was unavailable. CONCLUSIONS This study paves the way for reducing the need for frequent calibration of iBCIs and collecting a large amount of annotated training data. Potential future works aim to improve offline decoding performance with an ultra-small training dataset and improve the iBCIs' robustness to severely disabled electrodes.
Collapse
Affiliation(s)
- Shih-Hung Yang
- Department of Mechanical Engineering, National Cheng Kung University, Tainan, 701, Taiwan.
| | - Chun-Jui Huang
- Department of Mechanical Engineering, National Cheng Kung University, Tainan, 701, Taiwan
| | - Jhih-Siang Huang
- Department of Mechanical Engineering, National Cheng Kung University, Tainan, 701, Taiwan
| |
Collapse
|
2
|
Wu X, Zhang D, Li G, Gao X, Metcalfe B, Chen L. Data augmentation for invasive brain-computer interfaces based on stereo-electroencephalography (SEEG). J Neural Eng 2024; 21:016026. [PMID: 38237174 DOI: 10.1088/1741-2552/ad200e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2023] [Accepted: 01/18/2024] [Indexed: 02/23/2024]
Abstract
Objective.Deep learning is increasingly used for brain-computer interfaces (BCIs). However, the quantity of available data is sparse, especially for invasive BCIs. Data augmentation (DA) methods, such as generative models, can help to address this sparseness. However, all the existing studies on brain signals were based on convolutional neural networks and ignored the temporal dependence. This paper attempted to enhance generative models by capturing the temporal relationship from a time-series perspective.Approach. A conditional generative network (conditional transformer-based generative adversarial network (cTGAN)) based on the transformer model was proposed. The proposed method was tested using a stereo-electroencephalography (SEEG) dataset which was recorded from eight epileptic patients performing five different movements. Three other commonly used DA methods were also implemented: noise injection (NI), variational autoencoder (VAE), and conditional Wasserstein generative adversarial network with gradient penalty (cWGANGP). Using the proposed method, the artificial SEEG data was generated, and several metrics were used to compare the data quality, including visual inspection, cosine similarity (CS), Jensen-Shannon distance (JSD), and the effect on the performance of a deep learning-based classifier.Main results. Both the proposed cTGAN and the cWGANGP methods were able to generate realistic data, while NI and VAE outputted inferior samples when visualized as raw sequences and in a lower dimensional space. The cTGAN generated the best samples in terms of CS and JSD and outperformed cWGANGP significantly in enhancing the performance of a deep learning-based classifier (each of them yielding a significant improvement of 6% and 3.4%, respectively).Significance. This is the first time that DA methods have been applied to invasive BCIs based on SEEG. In addition, this study demonstrated the advantages of the model that preserves the temporal dependence from a time-series perspective.
Collapse
Affiliation(s)
- Xiaolong Wu
- The Centre for Autonomous Robotics (CENTAUR), Department of Electronic & Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Dingguo Zhang
- The Centre for Autonomous Robotics (CENTAUR), Department of Electronic & Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Guangye Li
- School of Mechanical Engineering, Shanghai Jiao Tong University, People's Republic of China
| | - Xin Gao
- The Centre for Autonomous Robotics (CENTAUR), Department of Electronic & Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Benjamin Metcalfe
- The Centre for Autonomous Robotics (CENTAUR), Department of Electronic & Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Liang Chen
- Liang Chen is with Huashan Hospital, Fudan University, People's Republic of China
| |
Collapse
|
3
|
Wu X, Yang J, Shao Y, Chen X. Mental fatigue assessment by an arbitrary channel EEG based on morphological features and LSTM-CNN. Comput Biol Med 2023; 167:107652. [PMID: 37950945 DOI: 10.1016/j.compbiomed.2023.107652] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2022] [Revised: 10/05/2023] [Accepted: 10/31/2023] [Indexed: 11/13/2023]
Abstract
In order to achieve more sensitive mental fatigue assessment (MFA) based on an arbitrary channel EEG, this study proposed a series of feature extraction methods that combine mathematical morphology (MM), as well as an LSTM-CNN architecture. Firstly, 37 subjects had their resting-state EEGs collected at rested wakefulness (RW) and after 24 h of sleep deprivation (SD) using a 30-channel EEG acquisition device, the RW and SD groups were regarded as the negative and positive groups of mental fatigue, respectively, and the EEG collection were further categorized into two conditions: eye-opened state (EO) and eye-closed state (EC). Then, since MM can reflect the morphological characteristics of EEG rhythms and their potentials relatively independently of the time-frequency analysis and phase calculation, the MM methods were found to better reflect the mental fatigue after SD statistically, whether for single features (ANOVA: p<0.000001), multiple features (clustering by K-means, t-test: p<0.01), or time series feature spaces (calculating CD, t-test: p<0.01) of a single channel. Finally, the LSTM-CNN enhanced the generalization ability when dealing with different single-channel EEG by combining GRUs with convolutional layers: comparing the AUCs of different architectures for MFA based on an arbitrary channel, LSTM-CNN (0.992) > LSTM network (0.94) > CNN (0.831) > MLP (0.754). Moreover, the use of MM also improved the accuracy of analyzed architectures, and the true/false positive rate (TPR/FPR) of the LSTM-CNN architecture for MFA based on an arbitrary channel reached 97.024 %/3.497 %, which provided a feasible solution for the arbitrary channel EEG-based MFA.
Collapse
Affiliation(s)
- Xiaolong Wu
- School of Mechanical Engineering, University of Science and Technology Beijing, Beijing, China; Shunde Innovation School, University of Science and Technology Beijing, Guangdong, China
| | - Jianhong Yang
- School of Mechanical Engineering, University of Science and Technology Beijing, Beijing, China; Shunde Innovation School, University of Science and Technology Beijing, Guangdong, China; Technical Support Center for Prevention and Control of Disastrous Accidents in Metal Smelting, University of Science and Technology Beijing, Beijing, China.
| | - Yongcong Shao
- School of Psychology, Beijing Sport University, Beijing, China; Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Science, Suzhou, China
| | - Xuewei Chen
- Institute of Environmental and Operational Medicine, Academy of Military Sciences, Tianjin, China
| |
Collapse
|
4
|
Safaie M, Chang JC, Park J, Miller LE, Dudman JT, Perich MG, Gallego JA. Preserved neural dynamics across animals performing similar behaviour. Nature 2023; 623:765-771. [PMID: 37938772 PMCID: PMC10665198 DOI: 10.1038/s41586-023-06714-0] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 10/04/2023] [Indexed: 11/09/2023]
Abstract
Animals of the same species exhibit similar behaviours that are advantageously adapted to their body and environment. These behaviours are shaped at the species level by selection pressures over evolutionary timescales. Yet, it remains unclear how these common behavioural adaptations emerge from the idiosyncratic neural circuitry of each individual. The overall organization of neural circuits is preserved across individuals1 because of their common evolutionarily specified developmental programme2-4. Such organization at the circuit level may constrain neural activity5-8, leading to low-dimensional latent dynamics across the neural population9-11. Accordingly, here we suggested that the shared circuit-level constraints within a species would lead to suitably preserved latent dynamics across individuals. We analysed recordings of neural populations from monkey and mouse motor cortex to demonstrate that neural dynamics in individuals from the same species are surprisingly preserved when they perform similar behaviour. Neural population dynamics were also preserved when animals consciously planned future movements without overt behaviour12 and enabled the decoding of planned and ongoing movement across different individuals. Furthermore, we found that preserved neural dynamics extend beyond cortical regions to the dorsal striatum, an evolutionarily older structure13,14. Finally, we used neural network models to demonstrate that behavioural similarity is necessary but not sufficient for this preservation. We posit that these emergent dynamics result from evolutionary constraints on brain development and thus reflect fundamental properties of the neural basis of behaviour.
Collapse
Affiliation(s)
- Mostafa Safaie
- Department of Bioengineering, Imperial College London, London, UK
| | - Joanna C Chang
- Department of Bioengineering, Imperial College London, London, UK
| | - Junchol Park
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, TX, USA
| | - Lee E Miller
- Departments of Physiology, Biomedical Engineering and Physical Medicine and Rehabilitation, Northwestern University and Shirley Ryan Ability Lab, Chicago, IL, USA
| | - Joshua T Dudman
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, TX, USA
| | - Matthew G Perich
- Département de Neurosciences, Faculté de Médecine, Université de Montréal, Montreal, Quebec, Canada.
- Mila, Quebec Artificial Intelligence Institute, Montreal, Quebec, Canada.
| | - Juan A Gallego
- Department of Bioengineering, Imperial College London, London, UK.
| |
Collapse
|
5
|
Dong Y, Wang S, Huang Q, Berg RW, Li G, He J. Neural Decoding for Intracortical Brain-Computer Interfaces. CYBORG AND BIONIC SYSTEMS 2023; 4:0044. [PMID: 37519930 PMCID: PMC10380541 DOI: 10.34133/cbsystems.0044] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2022] [Accepted: 07/04/2023] [Indexed: 08/01/2023] Open
Abstract
Brain-computer interfaces have revolutionized the field of neuroscience by providing a solution for paralyzed patients to control external devices and improve the quality of daily life. To accurately and stably control effectors, it is important for decoders to recognize an individual's motor intention from neural activity either by noninvasive or intracortical neural recording. Intracortical recording is an invasive way of measuring neural electrical activity with high temporal and spatial resolution. Herein, we review recent developments in neural signal decoding methods for intracortical brain-computer interfaces. These methods have achieved good performance in analyzing neural activity and controlling robots and prostheses in nonhuman primates and humans. For more complex paradigms in motor rehabilitation or other clinical applications, there remains more space for further improvements of decoders.
Collapse
Affiliation(s)
- Yuanrui Dong
- School of Mechatronical Engineering and Beijing Advanced Innovation Center for Intelligent Robots,
Beijing Institute of Technology, Beijing 100081, China
| | - Shirong Wang
- School of Mechatronical Engineering and Beijing Advanced Innovation Center for Intelligent Robots,
Beijing Institute of Technology, Beijing 100081, China
| | - Qiang Huang
- School of Mechatronical Engineering and Beijing Advanced Innovation Center for Intelligent Robots,
Beijing Institute of Technology, Beijing 100081, China
| | - Rune W. Berg
- Department of Neuroscience,
University of Copenhagen, Copenhagen 2200, Denmark
| | - Guanghui Li
- Department of Neuroscience,
University of Copenhagen, Copenhagen 2200, Denmark
| | - Jiping He
- School of Mechatronical Engineering and Beijing Advanced Innovation Center for Intelligent Robots,
Beijing Institute of Technology, Beijing 100081, China
| |
Collapse
|
6
|
Maksymenko K, Clarke AK, Mendez Guerra I, Deslauriers-Gauthier S, Farina D. A myoelectric digital twin for fast and realistic modelling in deep learning. Nat Commun 2023; 14:1600. [PMID: 36959193 PMCID: PMC10036636 DOI: 10.1038/s41467-023-37238-w] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2022] [Accepted: 03/08/2023] [Indexed: 03/25/2023] Open
Abstract
Muscle electrophysiology has emerged as a powerful tool to drive human machine interfaces, with many new recent applications outside the traditional clinical domains, such as robotics and virtual reality. However, more sophisticated, functional, and robust decoding algorithms are required to meet the fine control requirements of these applications. Deep learning has shown high potential in meeting these demands, but requires a large amount of high-quality annotated data, which is expensive and time-consuming to acquire. Data augmentation using simulations, a strategy applied in other deep learning applications, has never been attempted in electromyography due to the absence of computationally efficient models. We introduce a concept of Myoelectric Digital Twin - highly realistic and fast computational model tailored for the training of deep learning algorithms. It enables simulation of arbitrary large and perfectly annotated datasets of realistic electromyography signals, allowing new approaches to muscular signal decoding, accelerating the development of human-machine interfaces.
Collapse
Affiliation(s)
| | | | | | | | - Dario Farina
- Department of Bioengineering, Imperial College London, London, UK.
| |
Collapse
|
7
|
Wu X, Yang J. The superiority verification of morphological features in the EEG-based assessment of depression. J Neurosci Methods 2022; 381:109690. [PMID: 36007848 DOI: 10.1016/j.jneumeth.2022.109690] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Revised: 08/16/2022] [Accepted: 08/19/2022] [Indexed: 12/14/2022]
Affiliation(s)
- Xiaolong Wu
- School of Mechanical Engineering, University of Science and Technology Beijing, Beijing, 100083, China; Shunde Graduate School, University of Science and Technology Beijing, Guangdong 528399, China.
| | - Jianhong Yang
- School of Mechanical Engineering, University of Science and Technology Beijing, Beijing, 100083, China; Shunde Graduate School, University of Science and Technology Beijing, Guangdong 528399, China; Technical Support Center for Prevention and Control of Disastrous Accidents in Metal Smelting, University of Science and Technology Beijing, Beijing 100083, China.
| |
Collapse
|
8
|
Awasthi P, Lin TH, Bae J, Miller LE, Danziger ZC. Validation of a non-invasive, real-time, human-in-the-loop model of intracortical brain-computer interfaces. J Neural Eng 2022; 19:056038. [PMID: 36198278 PMCID: PMC9855658 DOI: 10.1088/1741-2552/ac97c3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2022] [Accepted: 10/05/2022] [Indexed: 01/26/2023]
Abstract
Objective. Despite the tremendous promise of invasive brain-computer interfaces (iBCIs), the associated study costs, risks, and ethical considerations limit the opportunity to develop and test the algorithms that decode neural activity into a user's intentions. Our goal was to address this challenge by designing an iBCI model capable of testing many human subjects in closed-loop.Approach. We developed an iBCI model that uses artificial neural networks (ANNs) to translate human finger movements into realistic motor cortex firing patterns, which can then be decoded in real time. We call the model the joint angle BCI, or jaBCI. jaBCI allows readily recruited, healthy subjects to perform closed-loop iBCI tasks using any neural decoder, preserving subjects' control-relevant short-latency error correction and learning dynamics.Main results. We validated jaBCI offline through emulated neuron firing statistics, confirming that emulated neural signals have firing rates, low-dimensional PCA geometry, and rotational jPCA dynamics that are quite similar to the actual neurons (recorded in monkey M1) on which we trained the ANN. We also tested jaBCI in closed-loop experiments, our single study examining roughly as many subjects as have been tested world-wide with iBCIs (n= 25). Performance was consistent with that of the paralyzed, human iBCI users with implanted intracortical electrodes. jaBCI allowed us to imitate the experimental protocols (e.g. the same velocity Kalman filter decoder and center-out task) and compute the same seven behavioral measures used in three critical studies.Significance. These encouraging results suggest the jaBCI's real-time firing rate emulation is a useful means to provide statistically robust sample sizes for rapid prototyping and optimization of decoding algorithms, the study of bi-directional learning in iBCIs, and improving iBCI control.
Collapse
Affiliation(s)
- Peeyush Awasthi
- Department of Biomedical Engineering, Florida International University, Miami, FL, United States of Amercia
| | - Tzu-Hsiang Lin
- Department of Biomedical Engineering, Florida International University, Miami, FL, United States of Amercia
| | - Jihye Bae
- Department of Electrical and Computer Engineering, University of Kentucky, Lexington, KY, United States
| | - Lee E Miller
- Department of Neuroscience, Physical Medicine, and Rehabilitation, Northwestern University, Chicago, IL, United States
| | - Zachary C Danziger
- Department of Biomedical Engineering, Florida International University, Miami, FL, United States of Amercia,Author to whom any correspondence should be addressed
| |
Collapse
|