1
|
Kuo CH, Guan-Tze L, Lee CE, Wu J, Casimo K, Weaver KE, Lo YC, Chen YY, Huang WC, Ojemann JG. Decoding Micro-Electrocorticographic Signals by Using Explainable 3D Convolutional Neural Network to Predict Finger Movements. J Neurosci Methods 2024:110251. [PMID: 39151656 DOI: 10.1016/j.jneumeth.2024.110251] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2024] [Revised: 08/05/2024] [Accepted: 08/13/2024] [Indexed: 08/19/2024]
Abstract
BACKGROUND Electroencephalography (EEG) and electrocorticography (ECoG) recordings have been used to decode finger movements by analyzing brain activity. Traditional methods focused on single bandpass power changes for movement decoding, utilizing machine learning models requiring manual feature extraction. NEW METHOD This study introduces a 3D convolutional neural network (3D-CNN) model to decode finger movements using ECoG data. The model employs adaptive, explainable AI (xAI) techniques to interpret the physiological relevance of brain signals. ECoG signals from epilepsy patients during awake craniotomy were processed to extract power spectral density across multiple frequency bands. These data formed a 3D matrix used to train the 3D-CNN to predict finger trajectories. RESULTS The 3D-CNN model showed significant accuracy in predicting finger movements, with root-mean-square error (RMSE) values of 0.26-0.38 for single finger movements and 0.20-0.24 for combined movements. Explainable AI techniques, Grad-CAM and SHAP, identified the high gamma (HG) band as crucial for movement prediction, showing specific cortical regions involved in different finger movements. These findings highlighted the physiological significance of the HG band in motor control. COMPARISON WITH EXISTING METHODS The 3D-CNN model outperformed traditional machine learning approaches by effectively capturing spatial and temporal patterns in ECoG data. The use of xAI techniques provided clearer insights into the model's decision-making process, unlike the "black box" nature of standard deep learning models. CONCLUSIONS The proposed 3D-CNN model, combined with xAI methods, enhances the decoding accuracy of finger movements from ECoG data. This approach offers a more efficient and interpretable solution for brain-computer interface (BCI) applications, emphasizing the HG band's role in motor control.
Collapse
Affiliation(s)
- Chao-Hung Kuo
- Department of Neurosurgery, Neurological Institute, Taipei Veterans General Hospital, Taipei, Taiwan; bSchool of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan; Department of Biomedical Engineering, National Yang Ming Chiao Tung University, Taipei, Taiwan; Department of Neurological Surgery, University of Washington, Seattle, WA, USA; The Ph.D. Program for Neural Regenerative Medicine, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.
| | - Liu Guan-Tze
- Department of Neurosurgery, Neurological Institute, Taipei Veterans General Hospital, Taipei, Taiwan; bSchool of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
| | - Chi-En Lee
- Department of Biomedical Engineering, National Yang Ming Chiao Tung University, Taipei, Taiwan
| | - Jing Wu
- Department of Bioengineering, University of Washington, Seattle, WA, USA; Center for Neurotechnology, University of Washington, Seattle, WA, USA
| | - Kaitlyn Casimo
- Center for Neurotechnology, University of Washington, Seattle, WA, USA
| | - Kurt E Weaver
- Center for Neurotechnology, University of Washington, Seattle, WA, USA; Department of Radiology, and Integrated Brain Imaging Center, University of Washington, Seattle, WA, USA
| | - Yu-Chun Lo
- The Ph.D. Program for Neural Regenerative Medicine, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan
| | - You-Yin Chen
- Department of Biomedical Engineering, National Yang Ming Chiao Tung University, Taipei, Taiwan; The Ph.D. Program for Neural Regenerative Medicine, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.
| | - Wen-Cheng Huang
- Department of Neurosurgery, Neurological Institute, Taipei Veterans General Hospital, Taipei, Taiwan; bSchool of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
| | - Jeffrey G Ojemann
- Department of Neurological Surgery, University of Washington, Seattle, WA, USA; Center for Neurotechnology, University of Washington, Seattle, WA, USA; Departments of Surgery, Seattle Children's Hospital, Seattle, WA, USA
| |
Collapse
|
2
|
Guo K, Lu J, Wu Y, Hu X, Yang H. The Latest Research Progress on Bionic Artificial Hands: A Systematic Review. MICROMACHINES 2024; 15:891. [PMID: 39064402 PMCID: PMC11278702 DOI: 10.3390/mi15070891] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/25/2024] [Revised: 07/01/2024] [Accepted: 07/03/2024] [Indexed: 07/28/2024]
Abstract
Bionic prosthetic hands hold the potential to replicate the functionality of human hands. The use of bionic limbs can assist amputees in performing everyday activities. This article systematically reviews the research progress on bionic prostheses, with a focus on control mechanisms, sensory feedback integration, and mechanical design innovations. It emphasizes the use of bioelectrical signals, such as electromyography (EMG), for prosthetic control and discusses the application of machine learning algorithms to enhance the accuracy of gesture recognition. Additionally, the paper explores advancements in sensory feedback technologies, including tactile, visual, and auditory modalities, which enhance user interaction by providing essential environmental feedback. The mechanical design of prosthetic hands is also examined, with particular attention to achieving a balance between dexterity, weight, and durability. Our contribution consists of compiling current research trends and identifying key areas for future development, including the enhancement of control system integration and improving the aesthetic and functional resemblance of prostheses to natural limbs. This work aims to inform and inspire ongoing research that seeks to refine the utility and accessibility of prosthetic hands for amputees, emphasizing user-centric innovations.
Collapse
Affiliation(s)
- Kai Guo
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei 230026, China
| | - Jingxin Lu
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
- College of Mechanical and Electrical Engineering, Changchun University of Science and Technology, Changchun 130022, China
| | - Yuwen Wu
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
| | - Xuhui Hu
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
| | - Hongbo Yang
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei 230026, China
- College of Mechanical and Electrical Engineering, Changchun University of Science and Technology, Changchun 130022, China
| |
Collapse
|
3
|
Ajioka T, Nakai N, Yamashita O, Takumi T. End-to-end deep learning approach to mouse behavior classification from cortex-wide calcium imaging. PLoS Comput Biol 2024; 20:e1011074. [PMID: 38478563 PMCID: PMC10986998 DOI: 10.1371/journal.pcbi.1011074] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Revised: 04/02/2024] [Accepted: 03/04/2024] [Indexed: 04/04/2024] Open
Abstract
Deep learning is a powerful tool for neural decoding, broadly applied to systems neuroscience and clinical studies. Interpretable and transparent models that can explain neural decoding for intended behaviors are crucial to identifying essential features of deep learning decoders in brain activity. In this study, we examine the performance of deep learning to classify mouse behavioral states from mesoscopic cortex-wide calcium imaging data. Our convolutional neural network (CNN)-based end-to-end decoder combined with recurrent neural network (RNN) classifies the behavioral states with high accuracy and robustness to individual differences on temporal scales of sub-seconds. Using the CNN-RNN decoder, we identify that the forelimb and hindlimb areas in the somatosensory cortex significantly contribute to behavioral classification. Our findings imply that the end-to-end approach has the potential to be an interpretable deep learning method with unbiased visualization of critical brain regions.
Collapse
Affiliation(s)
- Takehiro Ajioka
- Department of Physiology and Cell Biology, Kobe University School of Medicine, Chuo, Kobe, Japan
| | - Nobuhiro Nakai
- Department of Physiology and Cell Biology, Kobe University School of Medicine, Chuo, Kobe, Japan
| | - Okito Yamashita
- Department of Computational Brain Imaging, ATR Neural Information Analysis Laboratories, Seika, Kyoto, Japan
| | - Toru Takumi
- Department of Physiology and Cell Biology, Kobe University School of Medicine, Chuo, Kobe, Japan
- RIKEN Center for Biosystems Dynamics Research, Chuo, Kobe, Japan
| |
Collapse
|
4
|
Canny E, Vansteensel MJ, van der Salm SMA, Müller-Putz GR, Berezutskaya J. Boosting brain-computer interfaces with functional electrical stimulation: potential applications in people with locked-in syndrome. J Neuroeng Rehabil 2023; 20:157. [PMID: 37980536 PMCID: PMC10656959 DOI: 10.1186/s12984-023-01272-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Accepted: 10/23/2023] [Indexed: 11/20/2023] Open
Abstract
Individuals with a locked-in state live with severe whole-body paralysis that limits their ability to communicate with family and loved ones. Recent advances in brain-computer interface (BCI) technology have presented a potential alternative for these people to communicate by detecting neural activity associated with attempted hand or speech movements and translating the decoded intended movements to a control signal for a computer. A technique that could potentially enrich the communication capacity of BCIs is functional electrical stimulation (FES) of paralyzed limbs and face to restore body and facial movements of paralyzed individuals, allowing to add body language and facial expression to communication BCI utterances. Here, we review the current state of the art of existing BCI and FES work in people with paralysis of body and face and propose that a combined BCI-FES approach, which has already proved successful in several applications in stroke and spinal cord injury, can provide a novel promising mode of communication for locked-in individuals.
Collapse
Affiliation(s)
- Evan Canny
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Mariska J Vansteensel
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Sandra M A van der Salm
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Gernot R Müller-Putz
- Institute of Neural Engineering, Laboratory of Brain-Computer Interfaces, Graz University of Technology, Graz, Austria
| | - Julia Berezutskaya
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands.
| |
Collapse
|
5
|
Gruenwald J, Sieghartsleitner S, Kapeller C, Scharinger J, Kamada K, Brunner P, Guger C. Characterization of High-Gamma Activity in Electrocorticographic Signals. Front Neurosci 2023; 17:1206120. [PMID: 37609450 PMCID: PMC10440607 DOI: 10.3389/fnins.2023.1206120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Accepted: 07/10/2023] [Indexed: 08/24/2023] Open
Abstract
Introduction Electrocorticographic (ECoG) high-gamma activity (HGA) is a widely recognized and robust neural correlate of cognition and behavior. However, fundamental signal properties of HGA, such as the high-gamma frequency band or temporal dynamics of HGA, have never been systematically characterized. As a result, HGA estimators are often poorly adjusted, such that they miss valuable physiological information. Methods To address these issues, we conducted a thorough qualitative and quantitative characterization of HGA in ECoG signals. Our study is based on ECoG signals recorded from 18 epilepsy patients while performing motor control, listening, and visual perception tasks. In this study, we first categorize HGA into HGA types based on the cognitive/behavioral task. For each HGA type, we then systematically quantify three fundamental signal properties of HGA: the high-gamma frequency band, the HGA bandwidth, and the temporal dynamics of HGA. Results The high-gamma frequency band strongly varies across subjects and across cognitive/behavioral tasks. In addition, HGA time courses have lowpass character, with transients limited to 10 Hz. The task-related rise time and duration of these HGA time courses depend on the individual subject and cognitive/behavioral task. Task-related HGA amplitudes are comparable across the investigated tasks. Discussion This study is of high practical relevance because it provides a systematic basis for optimizing experiment design, ECoG acquisition and processing, and HGA estimation. Our results reveal previously unknown characteristics of HGA, the physiological principles of which need to be investigated in further studies.
Collapse
Affiliation(s)
- Johannes Gruenwald
- g.tec medical engineering GmbH, Schiedlberg, Austria
- Institute of Computational Perception, Johannes Kepler University, Linz, Austria
| | - Sebastian Sieghartsleitner
- g.tec medical engineering GmbH, Schiedlberg, Austria
- Institute of Computational Perception, Johannes Kepler University, Linz, Austria
| | | | - Josef Scharinger
- Institute of Computational Perception, Johannes Kepler University, Linz, Austria
| | - Kyousuke Kamada
- Department for Neurosurgery, Asahikawa Medical University, Asahikawa, Japan
- Hokashin Group Megumino Hospital, Sapporo, Japan
| | - Peter Brunner
- National Center for Adaptive Neurotechnologies, Albany, NY, United States
- Department of Neurosurgery, Washington University School of Medicine, St. Louis, MO, United States
| | | |
Collapse
|
6
|
Xu M, Zhou W, Shen X, Wang Y, Mo L, Qiu J. Swin-TCNet: Swin-based temporal-channel cascade network for motor imagery iEEG signal recognition. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104885] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/28/2023]
|
7
|
Bu Y, Harrington DL, Lee RR, Shen Q, Angeles-Quinto A, Ji Z, Hansen H, Hernandez-Lucas J, Baumgartner J, Song T, Nichols S, Baker D, Rao R, Lerman I, Lin T, Tu XM, Huang M. Magnetoencephalogram-based brain-computer interface for hand-gesture decoding using deep learning. Cereb Cortex 2023:7161766. [PMID: 37183188 DOI: 10.1093/cercor/bhad173] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2023] [Revised: 04/28/2023] [Accepted: 04/29/2023] [Indexed: 05/16/2023] Open
Abstract
Advancements in deep learning algorithms over the past decade have led to extensive developments in brain-computer interfaces (BCI). A promising imaging modality for BCI is magnetoencephalography (MEG), which is a non-invasive functional imaging technique. The present study developed a MEG sensor-based BCI neural network to decode Rock-Paper-scissors gestures (MEG-RPSnet). Unique preprocessing pipelines in tandem with convolutional neural network deep-learning models accurately classified gestures. On a single-trial basis, we found an average of 85.56% classification accuracy in 12 subjects. Our MEG-RPSnet model outperformed two state-of-the-art neural network architectures for electroencephalogram-based BCI as well as a traditional machine learning method, and demonstrated equivalent and/or better performance than machine learning methods that have employed invasive, electrocorticography-based BCI using the same task. In addition, MEG-RPSnet classification performance using an intra-subject approach outperformed a model that used a cross-subject approach. Remarkably, we also found that when using only central-parietal-occipital regional sensors or occipitotemporal regional sensors, the deep learning model achieved classification performances that were similar to the whole-brain sensor model. The MEG-RSPnet model also distinguished neuronal features of individual hand gestures with very good accuracy. Altogether, these results show that noninvasive MEG-based BCI applications hold promise for future BCI developments in hand-gesture decoding.
Collapse
Affiliation(s)
- Yifeng Bu
- Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093, USA
| | - Deborah L Harrington
- Radiology, Research Services, VA, San Diego Healthcare System, San Diego, CA 92161, USA
- Department of Radiology, University of California San Diego, La Jolla, CA 92093, USA
| | - Roland R Lee
- Radiology, Research Services, VA, San Diego Healthcare System, San Diego, CA 92161, USA
- Department of Radiology, University of California San Diego, La Jolla, CA 92093, USA
| | - Qian Shen
- Radiology, Research Services, VA, San Diego Healthcare System, San Diego, CA 92161, USA
- Department of Radiology, University of California San Diego, La Jolla, CA 92093, USA
| | - Annemarie Angeles-Quinto
- Radiology, Research Services, VA, San Diego Healthcare System, San Diego, CA 92161, USA
- Department of Radiology, University of California San Diego, La Jolla, CA 92093, USA
| | - Zhengwei Ji
- Department of Radiology, University of California San Diego, La Jolla, CA 92093, USA
| | - Hayden Hansen
- Radiology, Research Services, VA, San Diego Healthcare System, San Diego, CA 92161, USA
| | | | - Jared Baumgartner
- Department of Radiology, University of California San Diego, La Jolla, CA 92093, USA
| | - Tao Song
- Department of Radiology, University of California San Diego, La Jolla, CA 92093, USA
| | - Sharon Nichols
- Department of Neurosciences, University of California San Diego, La Jolla, CA 92093, USA
| | - Dewleen Baker
- VA Center of Excellence for Stress and Mental Health, VA San Diego Healthcare System, San Diego, CA 92161, USA
- Department of Psychiatry, University of California San Diego, La Jolla, CA 92093, USA
| | - Ramesh Rao
- Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093, USA
| | - Imanuel Lerman
- Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093, USA
- VA Center of Excellence for Stress and Mental Health, VA San Diego Healthcare System, San Diego, CA 92161, USA
- Department of Anesthesiology, University of California San Diego, La Jolla, CA 92093, USA
| | - Tuo Lin
- Division of Biostatistics and Bioinformatics, University of California, San Diego, CA 92093, USA
| | - Xin Ming Tu
- Division of Biostatistics and Bioinformatics, University of California, San Diego, CA 92093, USA
| | - Mingxiong Huang
- Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093, USA
- Radiology, Research Services, VA, San Diego Healthcare System, San Diego, CA 92161, USA
- Department of Radiology, University of California San Diego, La Jolla, CA 92093, USA
| |
Collapse
|
8
|
Śliwowski M, Martin M, Souloumiac A, Blanchart P, Aksenova T. Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performance. Front Hum Neurosci 2023; 17:1111645. [PMID: 37007675 PMCID: PMC10061076 DOI: 10.3389/fnhum.2023.1111645] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Accepted: 02/27/2023] [Indexed: 03/18/2023] Open
Abstract
IntroductionIn brain-computer interfaces (BCI) research, recording data is time-consuming and expensive, which limits access to big datasets. This may influence the BCI system performance as machine learning methods depend strongly on the training dataset size. Important questions arise: taking into account neuronal signal characteristics (e.g., non-stationarity), can we achieve higher decoding performance with more data to train decoders? What is the perspective for further improvement with time in the case of long-term BCI studies? In this study, we investigated the impact of long-term recordings on motor imagery decoding from two main perspectives: model requirements regarding dataset size and potential for patient adaptation.MethodsWe evaluated the multilinear model and two deep learning (DL) models on a long-term BCI & Tetraplegia (ClinicalTrials.gov identifier: NCT02550522) clinical trial dataset containing 43 sessions of ECoG recordings performed with a tetraplegic patient. In the experiment, a participant executed 3D virtual hand translation using motor imagery patterns. We designed multiple computational experiments in which training datasets were increased or translated to investigate the relationship between models' performance and different factors influencing recordings.ResultsOur results showed that DL decoders showed similar requirements regarding the dataset size compared to the multilinear model while demonstrating higher decoding performance. Moreover, high decoding performance was obtained with relatively small datasets recorded later in the experiment, suggesting motor imagery patterns improvement and patient adaptation during the long-term experiment. Finally, we proposed UMAP embeddings and local intrinsic dimensionality as a way to visualize the data and potentially evaluate data quality.DiscussionDL-based decoding is a prospective approach in BCI which may be efficiently applied with real-life dataset size. Patient-decoder co-adaptation is an important factor to consider in long-term clinical BCI.
Collapse
Affiliation(s)
- Maciej Śliwowski
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
- Université Paris-Saclay, CEA, List, Palaiseau, France
| | - Matthieu Martin
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
| | | | | | - Tetiana Aksenova
- Université Grenoble Alpes, CEA, LETI, Clinatec, Grenoble, France
- *Correspondence: Tetiana Aksenova
| |
Collapse
|
9
|
Qi Y, Sun Y, Liu Q, Zhang Q, Cai H, Zheng Q. Editorial: The intersection of artificial intelligence and brain for high-performance neuroprosthesis and cyborg systems. Front Neurosci 2023; 17:1133002. [PMID: 37065928 PMCID: PMC10104444 DOI: 10.3389/fnins.2023.1133002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2022] [Accepted: 01/31/2023] [Indexed: 04/18/2023] Open
Affiliation(s)
- Yu Qi
- Affiliated Mental Health Center and Hangzhou Seventh People's Hospital, MOE Frontier Science Center for Brain Science and Brain-Machine Integration, Zhejiang University School of Medicine, Hangzhou, China
- *Correspondence: Yu Qi ✉
| | - Yu Sun
- Key Laboratory for Biomedical Engineering of MOE of China, College of Biomedical Engineering and Instrument Science, State Key Laboratory of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China
| | - Quanying Liu
- Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China
| | - Qiaosheng Zhang
- Department of Anesthesiology, New York University School of Medicine, New York City, NY, United States
| | - Hanshu Cai
- Gansu Provincial Key Laboratory of Wearable Computing, School of Information Science and Engineering, Lanzhou University, Lanzhou, China
| | - Qian Zheng
- The State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China
| |
Collapse
|
10
|
Li J, Qi Y, Pan G. Phase-amplitude coupling-based adaptive filters for neural signal decoding. Front Neurosci 2023; 17:1153568. [PMID: 37205052 PMCID: PMC10185763 DOI: 10.3389/fnins.2023.1153568] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2023] [Accepted: 04/06/2023] [Indexed: 05/21/2023] Open
Abstract
Bandpass filters play a core role in ECoG signal processing. Commonly used frequency bands such as alpha, beta, and gamma bands can reflect the normal rhythm of the brain. However, the universally predefined bands might not be optimal for a specific task. Especially the gamma band usually covers a wide frequency span (i.e., 30-200 Hz) which can be too coarse to capture features that appear in narrow bands. An ideal option is to find the optimal frequency bands for specific tasks in real-time and dynamically. To tackle this problem, we propose an adaptive band filter that selects the useful frequency band in a data-driven way. Specifically, we leverage the phase-amplitude coupling (PAC) of the coupled working mechanism of synchronizing neuron and pyramidal neurons in neuronal oscillations, in which the phase of slower oscillations modulates the amplitude of faster ones, to help locate the fine frequency bands from the gamma range, in a task-specific and individual-specific way. Thus, the information can be more precisely extracted from ECoG signals to improve neural decoding performance. Based on this, an end-to-end decoder (PACNet) is proposed to construct a neural decoding application with adaptive filter banks in a uniform framework. Experiments show that PACNet can improve neural decoding performance universally with different tasks.
Collapse
Affiliation(s)
- Jiajun Li
- State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China
- College of Computer Science and Technology, Zhejiang University, Hangzhou, China
| | - Yu Qi
- State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China
- Affiliated Mental Health Center and Hangzhou Seventh Peoples Hospital, MOE Frontier Science Center for Brain Science and Brain-Machine Integration, Zhejiang University School of Medicine, Hangzhou, China
- *Correspondence: Yu Qi
| | - Gang Pan
- State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China
- College of Computer Science and Technology, Zhejiang University, Hangzhou, China
| |
Collapse
|
11
|
Towards clinical application of implantable brain-computer interfaces for people with late-stage ALS: medical and ethical considerations. J Neurol 2023; 270:1323-1336. [PMID: 36450968 PMCID: PMC9971103 DOI: 10.1007/s00415-022-11464-6] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2022] [Revised: 10/26/2022] [Accepted: 10/27/2022] [Indexed: 12/05/2022]
Abstract
Individuals with amyotrophic lateral sclerosis (ALS) frequently develop speech and communication problems in the course of their disease. Currently available augmentative and alternative communication technologies do not present a solution for many people with advanced ALS, because these devices depend on residual and reliable motor activity. Brain-computer interfaces (BCIs) use neural signals for computer control and may allow people with late-stage ALS to communicate even when conventional technology falls short. Recent years have witnessed fast progression in the development and validation of implanted BCIs, which place neural signal recording electrodes in or on the cortex. Eventual widespread clinical application of implanted BCIs as an assistive communication technology for people with ALS will have significant consequences for their daily life, as well as for the clinical management of the disease, among others because of the potential interaction between the BCI and other procedures people with ALS undergo, such as tracheostomy. This article aims to facilitate responsible real-world implementation of implanted BCIs. We review the state of the art of research on implanted BCIs for communication, as well as the medical and ethical implications of the clinical application of this technology. We conclude that the contribution of all BCI stakeholders, including clinicians of the various ALS-related disciplines, will be needed to develop procedures for, and shape the process of, the responsible clinical application of implanted BCIs.
Collapse
|
12
|
Faes A, Hulle MMV. Finger movement and coactivation predicted from intracranial brain activity using extended block-term tensor regression. J Neural Eng 2022; 19. [PMID: 36240727 DOI: 10.1088/1741-2552/ac9a75] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2022] [Accepted: 10/14/2022] [Indexed: 01/11/2023]
Abstract
Objective.We introduce extended Block-Term Tensor Regression (eBTTR), a novel regression method designed to account for the multilinear nature of human intracranial finger movement recordings.Approach.The proposed method relies on recursive Tucker decomposition combined with automatic component extraction.Main results.eBTTR outperforms state-of-the-art regression approaches, including multilinear and deep learning ones, in accurately predicting finger trajectories as well as unintentional finger coactivations.Significance.eBTTR rivals state-of-the-art approaches while being less computationally expensive which is an advantage when intracranial electrodes are implanted acutely, as part of the patient's presurgical workup, limiting time for decoder development and testing.
Collapse
Affiliation(s)
- A Faes
- Department of Neurosciences, Laboratory for Neuro- & Psychophysiology, KU Leuven-University of Leuven, B-3000 Leuven, Belgium
| | - M M Van Hulle
- Department of Neurosciences, Laboratory for Neuro- & Psychophysiology, KU Leuven-University of Leuven, B-3000 Leuven, Belgium
| |
Collapse
|
13
|
Mirchi N, Warsi NM, Zhang F, Wong SM, Suresh H, Mithani K, Erdman L, Ibrahim GM. Decoding Intracranial EEG With Machine Learning: A Systematic Review. Front Hum Neurosci 2022; 16:913777. [PMID: 35832872 PMCID: PMC9271576 DOI: 10.3389/fnhum.2022.913777] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2022] [Accepted: 05/31/2022] [Indexed: 11/13/2022] Open
Abstract
Advances in intracranial electroencephalography (iEEG) and neurophysiology have enabled the study of previously inaccessible brain regions with high fidelity temporal and spatial resolution. Studies of iEEG have revealed a rich neural code subserving healthy brain function and which fails in disease states. Machine learning (ML), a form of artificial intelligence, is a modern tool that may be able to better decode complex neural signals and enhance interpretation of these data. To date, a number of publications have applied ML to iEEG, but clinician awareness of these techniques and their relevance to neurosurgery, has been limited. The present work presents a review of existing applications of ML techniques in iEEG data, discusses the relative merits and limitations of the various approaches, and examines potential avenues for clinical translation in neurosurgery. One-hundred-seven articles examining artificial intelligence applications to iEEG were identified from 3 databases. Clinical applications of ML from these articles were categorized into 4 domains: i) seizure analysis, ii) motor tasks, iii) cognitive assessment, and iv) sleep staging. The review revealed that supervised algorithms were most commonly used across studies and often leveraged publicly available timeseries datasets. We conclude with recommendations for future work and potential clinical applications.
Collapse
Affiliation(s)
- Nykan Mirchi
- Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Nebras M. Warsi
- Division of Neurosurgery, Hospital for Sick Children, Department of Surgery, University of Toronto, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
| | - Frederick Zhang
- Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Simeon M. Wong
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
- Program in Neuroscience and Mental Health, Hospital for Sick Children Research Institute, Toronto, ON, Canada
| | - Hrishikesh Suresh
- Division of Neurosurgery, Hospital for Sick Children, Department of Surgery, University of Toronto, Toronto, ON, Canada
| | - Karim Mithani
- Division of Neurosurgery, Hospital for Sick Children, Department of Surgery, University of Toronto, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
| | - Lauren Erdman
- Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON, Canada
- Department of Computer Science, University of Toronto, Toronto, ON, Canada
- Hospital for Sick Children, Toronto, ON, Canada
| | - George M. Ibrahim
- Division of Neurosurgery, Hospital for Sick Children, Department of Surgery, University of Toronto, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
- Program in Neuroscience and Mental Health, Hospital for Sick Children Research Institute, Toronto, ON, Canada
- Institute of Medical Science, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
14
|
Wang H, Qi Y, Yu H, Wang Y, Liu C, Hu G, Pan G. RCIT: An RSVP-Based Concealed Information Test Framework Using EEG Signals. IEEE Trans Cogn Dev Syst 2022. [DOI: 10.1109/tcds.2021.3053455] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Affiliation(s)
- Hanwen Wang
- College of Computer Science and Technology, Zhejiang University, Hangzhou, China
| | - Yu Qi
- College of Computer Science and Technology, MOE Frontier Center of Brain Science and Brain–Machine Integration, Key Laboratory for Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou, China
| | - Hang Yu
- College of Computer Science and Technology, Zhejiang University, Hangzhou, China
| | | | | | | | - Gang Pan
- College of Computer Science and Technology, State Key Lab of CAD&CG and the First Affiliated Hospital, College of Medicine, Zhejiang University, Hangzhou, China
| |
Collapse
|
15
|
Hammer J, Schirrmeister RT, Hartmann K, Marusic P, Schulze-Bonhage A, Ball T. Interpretable functional specialization emerges in deep convolutional networks trained on brain signals. J Neural Eng 2022; 19. [PMID: 35421857 DOI: 10.1088/1741-2552/ac6770] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Accepted: 04/14/2022] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Functional specialization is fundamental to neural information processing. Here, we study whether and how functional specialization emerges in artificial deep convolutional neural networks (CNNs) during a brain-computer interfacing (BCI) task. APPROACH We trained CNNs to predict hand movement speed from intracranial EEG (iEEG) and delineated how units across the different CNN hidden layers learned to represent the iEEG signal. MAIN RESULTS We show that distinct, functionally interpretable neural populations emerged as a result of the training process. While some units became sensitive to either iEEG amplitude or phase, others showed bimodal behavior with significant sensitivity to both features. Pruning of highly-sensitive units resulted in a steep drop of decoding accuracy not observed for pruning of less sensitive units, highlighting the functional relevance of the amplitude- and phase-specialized populations. SIGNIFICANCE We anticipate that emergent functional specialization as uncovered here will become a key concept in research towards interpretable deep learning for neuroscience and BCI applications.
Collapse
Affiliation(s)
- Jiri Hammer
- Neuromedical AI Lab, Department of Neurosurgery, University of Freiburg, Engelbergerstraße 21, Freiburg, 79106, GERMANY
| | | | - Kay Hartmann
- Neuromedical AI Lab, Department of Neurosurgery, University of Freiburg, Engelbergerstraße 21, Freiburg, 79106, GERMANY
| | - Petr Marusic
- Department of Neurology, Motol University Hospital, V Úvalu 84, Prague, 150 06, CZECH REPUBLIC
| | - Andreas Schulze-Bonhage
- Epilepsy Center, University Clinics, Albert-Ludwigs-Universitaet Freiburg, Albert-Ludwigs-University,, 79095 Freiburg, Germany, Freiburg, 79095, GERMANY
| | - Tonio Ball
- Epilepsy Center, University Clinics, Albert-Ludwigs-Universitaet Freiburg, Albert-Ludwigs-University,, 79095 Freiburg, Germany, Freiburg, 79106, GERMANY
| |
Collapse
|
16
|
Śliwowski M, Martin M, Souloumiac A, Blanchart P, Aksenova T. Decoding ECoG signal into 3D hand translation using deep learning. J Neural Eng 2022; 19. [PMID: 35287119 DOI: 10.1088/1741-2552/ac5d69] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2021] [Accepted: 03/14/2022] [Indexed: 12/29/2022]
Abstract
Objective.Motor brain-computer interfaces (BCIs) are a promising technology that may enable motor-impaired people to interact with their environment. BCIs would potentially compensate for arm and hand function loss, which is the top priority for individuals with tetraplegia. Designing real-time and accurate BCI is crucial to make such devices useful, safe, and easy to use by patients in a real-life environment. Electrocorticography (ECoG)-based BCIs emerge as a good compromise between invasiveness of the recording device and good spatial and temporal resolution of the recorded signal. However, most ECoG signal decoders used to predict continuous hand movements are linear models. These models have a limited representational capacity and may fail to capture the relationship between ECoG signal features and continuous hand movements. Deep learning (DL) models, which are state-of-the-art in many problems, could be a solution to better capture this relationship.Approach.In this study, we tested several DL-based architectures to predict imagined 3D continuous hand translation using time-frequency features extracted from ECoG signals. The dataset used in the analysis is a part of a long-term clinical trial (ClinicalTrials.gov identifier: NCT02550522) and was acquired during a closed-loop experiment with a tetraplegic subject. The proposed architectures include multilayer perceptron, convolutional neural networks (CNNs), and long short-term memory networks (LSTM). The accuracy of the DL-based and multilinear models was compared offline using cosine similarity.Main results.Our results show that CNN-based architectures outperform the current state-of-the-art multilinear model. The best architecture exploited the spatial correlation between neighboring electrodes with CNN and benefited from the sequential character of the desired hand trajectory by using LSTMs. Overall, DL increased the average cosine similarity, compared to the multilinear model, by up to 60%, from 0.189 to 0.302 and from 0.157 to 0.249 for the left and right hand, respectively.Significance.This study shows that DL-based models could increase the accuracy of BCI systems in the case of 3D hand translation prediction in a tetraplegic subject.
Collapse
Affiliation(s)
- Maciej Śliwowski
- Université Grenoble Alpes, CEA, LETI, Clinatec, F-38000 Grenoble, France.,Université Paris-Saclay, CEA, List, F-91120 Palaiseau, France
| | - Matthieu Martin
- Université Grenoble Alpes, CEA, LETI, Clinatec, F-38000 Grenoble, France
| | | | | | - Tetiana Aksenova
- Université Grenoble Alpes, CEA, LETI, Clinatec, F-38000 Grenoble, France
| |
Collapse
|
17
|
Xu R, Spataro R, Allison BZ, Guger C. Brain-Computer Interfaces in Acute and Subacute Disorders of Consciousness. J Clin Neurophysiol 2022; 39:32-39. [PMID: 34474428 DOI: 10.1097/wnp.0000000000000810] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022] Open
Abstract
SUMMARY Disorders of consciousness include coma, unresponsive wakefulness syndrome (also known as vegetative state), and minimally conscious state. Neurobehavioral scales such as coma recovery scale-revised are the gold standard for disorder of consciousness assessment. Brain-computer interfaces have been emerging as an alternative tool for these patients. The application of brain-computer interfaces in disorders of consciousness can be divided into four fields: assessment, communication, prediction, and rehabilitation. The operational theoretical model of consciousness that brain-computer interfaces explore was reviewed in this article, with a focus on studies with acute and subacute patients. We then proposed a clinically friendly guideline, which could contribute to the implementation of brain-computer interfaces in neurorehabilitation settings. Finally, we discussed limitations and future directions, including major challenges and possible solutions.
Collapse
Affiliation(s)
- Ren Xu
- Guger Technologies OG, Schiedlberg, Austria
| | - Rossella Spataro
- g.tec medical engineering GmbH, Schiedlberg, Austria
- IRCCS Centro Neurolesi Bonino Pulejo, Palermo, Italy; and
| | - Brendan Z Allison
- Cognitive Science Department, University of California San Diego, La Jolla, California, U.S.A
| | - Christoph Guger
- Guger Technologies OG, Schiedlberg, Austria
- g.tec medical engineering GmbH, Schiedlberg, Austria
| |
Collapse
|
18
|
Pradeepkumar J, Anandakumar M, Kugathasan V, Lalitharatne TD, De Silva AC, Kappel SL. Decoding of Hand Gestures from Electrocorticography with LSTM Based Deep Neural Network. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:420-423. [PMID: 34891323 DOI: 10.1109/embc46164.2021.9630958] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Hand gesture decoding is a key component of controlling prosthesis in the area of Brain Computer Interface (BCI). This study is concerned with classification of hand gestures, based on Electrocorticography (ECoG) recordings. Recent studies have utilized the temporal information in ECoG signals for robust hand gesture decoding. In our preliminary analysis on ECoG recordings of hand gestures, we observed different power variations in six frequency bands ranging from 4 to 200 Hz. Therefore, the current trend of including temporal information in the classifier was extended to provide equal importance to power variations in each of these frequency bands. Statistical and Principal Component Analysis (PCA) based feature reduction was implemented for each frequency band separately, and classification was performed with a Long Short-Term Memory (LSTM) based neural network to utilize both temporal and spatial information of each frequency band. The proposed architecture along with each feature reduction method was tested on ECoG recordings of five finger flexions performed by seven subjects from the publicly available 'fingerflex' dataset. An average classification accuracy of 82.4% was achieved with the statistical based channel selection method which is an improvement compared to state-of-the-art methods.
Collapse
|
19
|
Livezey JA, Glaser JI. Deep learning approaches for neural decoding across architectures and recording modalities. Brief Bioinform 2020; 22:1577-1591. [PMID: 33372958 DOI: 10.1093/bib/bbaa355] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2020] [Revised: 10/31/2020] [Accepted: 11/04/2020] [Indexed: 12/19/2022] Open
Abstract
Decoding behavior, perception or cognitive state directly from neural signals is critical for brain-computer interface research and an important tool for systems neuroscience. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmentation. The success of deep networks in other domains has led to a new wave of applications in neuroscience. In this article, we review deep learning approaches to neural decoding. We describe the architectures used for extracting useful features from neural recording modalities ranging from spikes to functional magnetic resonance imaging. Furthermore, we explore how deep learning has been leveraged to predict common outputs including movement, speech and vision, with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets like acoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.
Collapse
Affiliation(s)
- Jesse A Livezey
- Neural Systems and Data Science Laboratory at the Lawrence Berkeley National Laboratory. He obtained his PhD in Physics from the University of California, Berkeley
| | - Joshua I Glaser
- Center for Theoretical Neuroscience and Department of Statistics at Columbia University. He obtained his PhD in Neuroscience from Northwestern University
| |
Collapse
|
20
|
A comprehensive assessment of Brain Computer Interfaces: Recent trends and challenges. J Neurosci Methods 2020; 346:108918. [PMID: 32853592 DOI: 10.1016/j.jneumeth.2020.108918] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2020] [Revised: 07/15/2020] [Accepted: 08/19/2020] [Indexed: 12/23/2022]
Abstract
BACKGROUND An uninterrupted channel of communication and control between the human brain and electronic processing units has led to an increased use of Brain Computer Interfaces (BCIs). This article attempts to present an all-encompassing review on BCI and the scientific advancements associated with it. The ultimate goal of this review is to provide a general overview of the BCI technology and to shed light on different aspects of BCIs. This review also underscores the applications, practical challenges and opportunities associated with BCI technology, which can be used to accelerate future developments in this field. METHODS This review is based on a systematic literature search for tracking down the relevant research annals and proceedings. Using a methodical search strategy, the search was carried out across major technical databases. The retrieved records were screened for their relevance and a total of 369 research chronicles were engulfed in this review based on the inclusion criteria. RESULTS This review describes the present scenario and recent advancements in BCI technology. It also identifies several application areas of BCI technology. This comprehensive review provides evidence that, while we are getting ever closer, significant challenges still exist for the development of BCIs that can seamlessly integrate with the user's biological system. CONCLUSION The findings of this review confirm the importance of BCI technology in various applications. It is concluded that BCI technology, still in its sprouting phase, requires significant explorations for further development.
Collapse
|
21
|
Lian Q, Qi Y, Pan G, Wang Y. Learning graph in graph convolutional neural networks for robust seizure prediction. J Neural Eng 2020; 17:035004. [DOI: 10.1088/1741-2552/ab909d] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
|
22
|
Farrokhi B, Erfanian A. A state-based probabilistic method for decoding hand position during movement from ECoG signals in non-human primate. J Neural Eng 2020; 17:026042. [PMID: 32224511 DOI: 10.1088/1741-2552/ab848b] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022]
Abstract
OBJECTIVE In this study, we proposed a state-based probabilistic method for decoding hand positions during unilateral and bilateral movements using the ECoG signals recorded from the brain of Rhesus monkey. APPROACH A customized electrode array was implanted subdurally in the right hemisphere of the brain covering from the primary motor cortex to the frontal cortex. Three different experimental paradigms were considered: ipsilateral, contralateral, and bilateral movements. During unilateral movement, the monkey was trained to get food with one hand, while during bilateral movement, the monkey used its left and right hands alternately to get food. To estimate the hand positions, a state-based probabilistic method was introduced which was based on the conditional probability of the hand movement state (i.e. idle, right hand movement, and left hand movement) and the conditional expectation of the hand position for each state. Moreover, a hybrid feature extraction method based on linear discriminant analysis and partial least squares (PLS) was introduced. MAIN RESULTS The proposed method could successfully decode the hand positions during ipsilateral, contralateral, and bilateral movements and significantly improved the decoding performance compared to the conventional Kalman and PLS regression methods [Formula: see text]. The proposed hybrid feature extraction method was found to outperform both the PLS and PCA methods [Formula: see text]. Investigating the kinematic information of each frequency band shows that more informative frequency bands were [Formula: see text] (15-30 Hz) and [Formula: see text](50-100 Hz) for ipsilateral and [Formula: see text] and [Formula: see text] (100-200 Hz) for contralateral movements. It is observed that ipsilateral movement was decoded better than contralateral movement for [Formula: see text] (5-15 Hz) and [Formula: see text] bands, while contralateral movements was decoded better for [Formula: see text] (30-200 Hz) and hfECoG (200-400 Hz) bands. SIGNIFICANCE Accurate decoding the bilateral movement using the ECoG recorded from one brain hemisphere is an important issue toward real-life applications of the brain-machine interface technologies.
Collapse
Affiliation(s)
- Behraz Farrokhi
- Department of Biomedical Engineering, School of Electrical Engineering, Iran University of Science and Technology (IUST), Iran Neural Technology Research Centre, Tehran, Iran
| | | |
Collapse
|
23
|
Qian C, Sun X, Yang Z, Pan G, Wang Y. A K-Medoids based Point-Process Modeling on Neural Spike Transformation using Binless Kernel. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2019:4387-4390. [PMID: 31946839 DOI: 10.1109/embc.2019.8856479] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
A neural prosthesis is designed to compensate for cognitive functional losses by modeling the information transmission among cortical areas. Existing methods generally build a generalized linear model to approximate the nonlinear transformation among two areas, and use the temporal information of the neural spike with low efficiency. It is essential to efficiently model the nonlinearity embedded in spike generation and transmission for the real-time. This paper proposes a nonlinear point-process model to describe spike-in and spike-out transformation using the theory of reproducing kernel Hilbert space (RKHS) and the binless kernel on spike trains. The binless kernel efficiently maps exact spike timing information to the RKHS to describe nonlinear transformations with global minimum regardless of the weight initialization. A streaming K-medoids algorithm is introduced to select typical and important features in this nonlinear binless kernel for further modeling. We test our model on the nonlinearly generated synthetic neural spike trains, and compare with the existing spike transformation methods, such as Volterra model and staged point-process model. The results show that our model has higher goodness-of-fit evaluated by Kolmogorov-Smirnov test and less variance on the prediction results, which indicates the potential better modeling approach for neural prosthesis application.
Collapse
|
24
|
Volkova K, Lebedev MA, Kaplan A, Ossadtchi A. Decoding Movement From Electrocorticographic Activity: A Review. Front Neuroinform 2019; 13:74. [PMID: 31849632 PMCID: PMC6901702 DOI: 10.3389/fninf.2019.00074] [Citation(s) in RCA: 37] [Impact Index Per Article: 7.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2019] [Accepted: 11/14/2019] [Indexed: 01/08/2023] Open
Abstract
Electrocorticography (ECoG) holds promise to provide efficient neuroprosthetic solutions for people suffering from neurological disabilities. This recording technique combines adequate temporal and spatial resolution with the lower risks of medical complications compared to the other invasive methods. ECoG is routinely used in clinical practice for preoperative cortical mapping in epileptic patients. During the last two decades, research utilizing ECoG has considerably grown, including the paradigms where behaviorally relevant information is extracted from ECoG activity with decoding algorithms of different complexity. Several research groups have advanced toward the development of assistive devices driven by brain-computer interfaces (BCIs) that decode motor commands from multichannel ECoG recordings. Here we review the evolution of this field and its recent tendencies, and discuss the potential areas for future development.
Collapse
Affiliation(s)
- Ksenia Volkova
- Center for Bioelectric Interfaces, Higher School of Economics, National Research University, Moscow, Russia
| | - Mikhail A. Lebedev
- Center for Bioelectric Interfaces, Higher School of Economics, National Research University, Moscow, Russia
| | - Alexander Kaplan
- Center for Bioelectric Interfaces, Higher School of Economics, National Research University, Moscow, Russia
- Center for Biotechnology Development, National Research Lobachevsky State University of Nizhny Novgorod, Nizhny Novgorod, Russia
- Laboratory for Neurophysiology and Neuro-Computer Interfaces, Faculty of Biology, Lomonosov Moscow State University, Moscow, Russia
| | - Alexei Ossadtchi
- Center for Bioelectric Interfaces, Higher School of Economics, National Research University, Moscow, Russia
| |
Collapse
|
25
|
Gruenwald J, Znobishchev A, Kapeller C, Kamada K, Scharinger J, Guger C. Time-Variant Linear Discriminant Analysis Improves Hand Gesture and Finger Movement Decoding for Invasive Brain-Computer Interfaces. Front Neurosci 2019; 13:901. [PMID: 31616237 PMCID: PMC6775278 DOI: 10.3389/fnins.2019.00901] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2019] [Accepted: 08/12/2019] [Indexed: 11/13/2022] Open
Abstract
Invasive brain-computer interfaces yield remarkable performance in a multitude of applications. For classification experiments, high-gamma bandpower features and linear discriminant analysis (LDA) are commonly used due to simplicity and robustness. However, LDA is inherently static and not suited to account for transient information that is typically present in high-gamma features. To resolve this issue, we here present an extension of LDA to the time-variant feature space. We call this method time-variant linear discriminant analysis (TVLDA). It intrinsically provides a feature reduction stage, which makes external approaches thereto obsolete, such as feature selection techniques or common spatial patterns (CSPs). As well, we propose a time-domain whitening stage which equalizes the pronounced 1/f-shape of the typical brain-wave spectrum. We evaluated our proposed architecture based on recordings from 15 epilepsy patients with temporarily implanted subdural grids, who participated in additional research experiments besides clinical treatment. The experiments featured two different motor tasks involving three high-level gestures and individual finger movement. We used log-transformed bandpower features from the high-gamma band (50-300 Hz, excluding power-line harmonics) for classification. On average, whitening improved the classification performance by about 11%. On whitened data, TVLDA outperformed LDA with feature selection by 11.8%, LDA with CSPs by 13.9%, and regularized LDA with vectorized features by 16.4%. At the same time, TVLDA only required one or two internal features to achieve this. TVLDA provides stable results even if very few trials are available. It is easy to implement, fully automatic and deterministic. Due to its low complexity, TVLDA is suited for real-time brain-computer interfaces. Training is done in less than a second. TVLDA performed particularly well in experiments with data from high-density electrode arrays. For example, the three high-level gestures were correctly identified at a rate of 99% over all subjects. Similarly, the decoding accuracy of individual fingers was 96% on average over all subjects. To our knowledge, these mean accuracies are the highest ever reported for three-class and five-class motor-control BCIs.
Collapse
Affiliation(s)
- Johannes Gruenwald
- g.tec Medical Engineering GmbH, Schiedlberg, Austria.,Institute of Computational Perception, Johannes Kepler University, Linz, Austria
| | | | | | | | - Josef Scharinger
- Institute of Computational Perception, Johannes Kepler University, Linz, Austria
| | - Christoph Guger
- g.tec Medical Engineering GmbH, Schiedlberg, Austria.,Guger Technologies OG, Graz, Austria
| |
Collapse
|
26
|
|
27
|
Fischer B, Schander A, Kreiter AK, Lang W, Wegener D. Visual epidural field potentials possess high functional specificity in single trials. J Neurophysiol 2019; 122:1634-1648. [PMID: 31412218 DOI: 10.1152/jn.00510.2019] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Abstract
Recordings of epidural field potentials (EFPs) allow neuronal activity to be acquired over a large region of cortical tissue with minimal invasiveness. Because electrodes are placed on top of the dura and do not enter the neuronal tissue, EFPs offer intriguing options for both clinical and basic science research. On the other hand, EFPs represent the integrated activity of larger neuronal populations and possess a higher trial-by-trial variability and a reduced signal-to-noise ratio due the additional barrier of the dura. It is thus unclear whether and to what extent EFPs have sufficient spatial selectivity to allow for conclusions about the underlying functional cortical architecture, and whether single EFP trials provide enough information on the short timescales relevant for many clinical and basic neuroscience purposes. We used the high spatial resolution of primary visual cortex to address these issues and investigated the extent to which very short EFP traces allow reliable decoding of spatial information. We briefly presented different visual objects at one of nine closely adjacent locations and recorded neuronal activity with a high-density epidural multielectrode array in three macaque monkeys. With the use of receiver operating characteristics (ROC) to identify the most informative data, machine-learning algorithms provided close-to-perfect classification rates for all 27 stimulus conditions. A binary classifier applying a simple max function on ROC-selected data further showed that single trials might be classified with 100% performance even without advanced offline classifiers. Thus, although highly variable, EFPs constitute an extremely valuable source of information and offer new perspectives for minimally invasive recording of large-scale networks.NEW & NOTEWORTHY Epidural field potential (EFP) recordings provide a minimally invasive approach to investigate large-scale neural networks, but little is known about whether they possess the required specificity for basic and clinical neuroscience. By making use of the spatial selectivity of primary visual cortex, we show that single-trial information can be decoded with close-to-perfect performance, even without using advanced classifiers and based on very few data. This labels EFPs as a highly attractive and widely usable signal.
Collapse
Affiliation(s)
- Benjamin Fischer
- Brain Research Institute, Center for Cognitive Sciences, University of Bremen, Bremen, Germany
| | - Andreas Schander
- Institute for Microsensors, -Actuators, and -Systems, University of Bremen, Bremen, Germany
| | - Andreas K Kreiter
- Brain Research Institute, Center for Cognitive Sciences, University of Bremen, Bremen, Germany
| | - Walter Lang
- Institute for Microsensors, -Actuators, and -Systems, University of Bremen, Bremen, Germany
| | - Detlef Wegener
- Brain Research Institute, Center for Cognitive Sciences, University of Bremen, Bremen, Germany
| |
Collapse
|
28
|
Odor Recognition with a Spiking Neural Network for Bioelectronic Nose. SENSORS 2019; 19:s19050993. [PMID: 30813574 PMCID: PMC6427646 DOI: 10.3390/s19050993] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/05/2019] [Revised: 02/18/2019] [Accepted: 02/20/2019] [Indexed: 12/02/2022]
Abstract
Electronic noses recognize odors using sensor arrays, and usually face difficulties for odor complicacy, while animals have their own biological sensory capabilities for various types of odors. By implanting electrodes into the olfactory bulb of mammalian animals, odors may be recognized by decoding the recorded neural signals, in order to construct a bioelectronic nose. This paper proposes a spiking neural network (SNN)-based odor recognition method from spike trains recorded by the implanted electrode array. The proposed SNN-based approach exploits rich timing information well in precise time points of spikes. To alleviate the overfitting problem, we design a new SNN learning method with a voltage-based regulation strategy. Experiments are carried out using spike train signals recorded from the main olfactory bulb in rats. Results show that our SNN-based approach achieves the state-of-the-art performance, compared with other methods. With the proposed voltage regulation strategy, it achieves about 15% improvement compared with a classical SNN model.
Collapse
|