1
|
Silva AB, Littlejohn KT, Liu JR, Moses DA, Chang EF. The speech neuroprosthesis. Nat Rev Neurosci 2024; 25:473-492. [PMID: 38745103 DOI: 10.1038/s41583-024-00819-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/12/2024] [Indexed: 05/16/2024]
Abstract
Loss of speech after paralysis is devastating, but circumventing motor-pathway injury by directly decoding speech from intact cortical activity has the potential to restore natural communication and self-expression. Recent discoveries have defined how key features of speech production are facilitated by the coordinated activity of vocal-tract articulatory and motor-planning cortical representations. In this Review, we highlight such progress and how it has led to successful speech decoding, first in individuals implanted with intracranial electrodes for clinical epilepsy monitoring and subsequently in individuals with paralysis as part of early feasibility clinical trials to restore speech. We discuss high-spatiotemporal-resolution neural interfaces and the adaptation of state-of-the-art speech computational algorithms that have driven rapid and substantial progress in decoding neural activity into text, audible speech, and facial movements. Although restoring natural speech is a long-term goal, speech neuroprostheses already have performance levels that surpass communication rates offered by current assistive-communication technology. Given this accelerated rate of progress in the field, we propose key evaluation metrics for speed and accuracy, among others, to help standardize across studies. We finish by highlighting several directions to more fully explore the multidimensional feature space of speech and language, which will continue to accelerate progress towards a clinically viable speech neuroprosthesis.
Collapse
Affiliation(s)
- Alexander B Silva
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
| | - Kaylo T Littlejohn
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA, USA
| | - Jessie R Liu
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
| | - David A Moses
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
| | - Edward F Chang
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA.
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA.
| |
Collapse
|
2
|
Crone N, Candrea D, Shah S, Luo S, Angrick M, Rabbani Q, Coogan C, Milsap G, Nathan K, Wester B, Anderson W, Rosenblatt K, Clawson L, Maragakis N, Vansteensel M, Tenore F, Ramsey N, Fifer M, Uchil A. A click-based electrocorticographic brain-computer interface enables long-term high-performance switch-scan spelling. RESEARCH SQUARE 2023:rs.3.rs-3158792. [PMID: 37841873 PMCID: PMC10571601 DOI: 10.21203/rs.3.rs-3158792/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/17/2023]
Abstract
Background Brain-computer interfaces (BCIs) can restore communication in movement- and/or speech-impaired individuals by enabling neural control of computer typing applications. Single command "click" decoders provide a basic yet highly functional capability. Methods We sought to test the performance and long-term stability of click-decoding using a chronically implanted high density electrocorticographic (ECoG) BCI with coverage of the sensorimotor cortex in a human clinical trial participant (ClinicalTrials.gov, NCT03567213) with amyotrophic lateral sclerosis (ALS). We trained the participant's click decoder using a small amount of training data (< 44 minutes across four days) collected up to 21 days prior to BCI use, and then tested it over a period of 90 days without any retraining or updating. Results Using this click decoder to navigate a switch-scanning spelling interface, the study participant was able to maintain a median spelling rate of 10.2 characters per min. Though a transient reduction in signal power modulation interrupted testing with this fixed model, a new click decoder achieved comparable performance despite being trained with even less data (< 15 min, within one day). Conclusion These results demonstrate that a click decoder can be trained with a small ECoG dataset while retaining robust performance for extended periods, providing functional text-based communication to BCI users.
Collapse
|
3
|
Natraj N, Seko S, Abiri R, Yan H, Graham Y, Tu-Chan A, Chang EF, Ganguly K. Flexible regulation of representations on a drifting manifold enables long-term stable complex neuroprosthetic control. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.08.11.551770. [PMID: 37645922 PMCID: PMC10462094 DOI: 10.1101/2023.08.11.551770] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/31/2023]
Abstract
The nervous system needs to balance the stability of neural representations with plasticity. It is unclear what is the representational stability of simple actions, particularly those that are well-rehearsed in humans, and how it changes in new contexts. Using an electrocorticography brain-computer interface (BCI), we found that the mesoscale manifold and relative representational distances for a repertoire of simple imagined movements were remarkably stable. Interestingly, however, the manifold's absolute location demonstrated day-to-day drift. Strikingly, representational statistics, especially variance, could be flexibly regulated to increase discernability during BCI control without somatotopic changes. Discernability strengthened with practice and was specific to the BCI, demonstrating remarkable contextual specificity. Accounting for drift, and leveraging the flexibility of representations, allowed neuroprosthetic control of a robotic arm and hand for over 7 months without recalibration. Our study offers insight into how electrocorticography can both track representational statistics across long periods and allow long-term complex neuroprosthetic control.
Collapse
Affiliation(s)
- Nikhilesh Natraj
- Dept. of Neurology, Weill Institute for Neurosciences, University of California San Francisco, San Francisco, California, USA
- UCSF - Veteran Affairs Medical Center, San Francisco, California, USA
| | - Sarah Seko
- Dept. of Neurology, Weill Institute for Neurosciences, University of California San Francisco, San Francisco, California, USA
- UCSF - Veteran Affairs Medical Center, San Francisco, California, USA
| | - Reza Abiri
- Electrical, Computer and Biomedical Engineering, University of Rhode Island, Rhode Island, USA
| | - Hongyi Yan
- Dept. of Neurology, Weill Institute for Neurosciences, University of California San Francisco, San Francisco, California, USA
- UCSF - Veteran Affairs Medical Center, San Francisco, California, USA
| | - Yasmin Graham
- Dept. of Neurology, Weill Institute for Neurosciences, University of California San Francisco, San Francisco, California, USA
- UCSF - Veteran Affairs Medical Center, San Francisco, California, USA
| | - Adelyn Tu-Chan
- Dept. of Neurology, Weill Institute for Neurosciences, University of California San Francisco, San Francisco, California, USA
- UCSF - Veteran Affairs Medical Center, San Francisco, California, USA
| | - Edward F Chang
- Department of Neurological Surgery, Weill Institute for Neuroscience, University of California-San Francisco, San Francisco, California, USA
| | - Karunesh Ganguly
- Dept. of Neurology, Weill Institute for Neurosciences, University of California San Francisco, San Francisco, California, USA
- UCSF - Veteran Affairs Medical Center, San Francisco, California, USA
| |
Collapse
|
4
|
Hu X, Xu W, Ren Y, Wang Z, He X, Huang R, Ma B, Zhao J, Zhu R, Cheng L. Spinal cord injury: molecular mechanisms and therapeutic interventions. Signal Transduct Target Ther 2023; 8:245. [PMID: 37357239 DOI: 10.1038/s41392-023-01477-6] [Citation(s) in RCA: 78] [Impact Index Per Article: 78.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2022] [Revised: 03/22/2023] [Accepted: 05/07/2023] [Indexed: 06/27/2023] Open
Abstract
Spinal cord injury (SCI) remains a severe condition with an extremely high disability rate. The challenges of SCI repair include its complex pathological mechanisms and the difficulties of neural regeneration in the central nervous system. In the past few decades, researchers have attempted to completely elucidate the pathological mechanism of SCI and identify effective strategies to promote axon regeneration and neural circuit remodeling, but the results have not been ideal. Recently, new pathological mechanisms of SCI, especially the interactions between immune and neural cell responses, have been revealed by single-cell sequencing and spatial transcriptome analysis. With the development of bioactive materials and stem cells, more attention has been focused on forming intermediate neural networks to promote neural regeneration and neural circuit reconstruction than on promoting axonal regeneration in the corticospinal tract. Furthermore, technologies to control physical parameters such as electricity, magnetism and ultrasound have been constantly innovated and applied in neural cell fate regulation. Among these advanced novel strategies and technologies, stem cell therapy, biomaterial transplantation, and electromagnetic stimulation have entered into the stage of clinical trials, and some of them have already been applied in clinical treatment. In this review, we outline the overall epidemiology and pathophysiology of SCI, expound on the latest research progress related to neural regeneration and circuit reconstruction in detail, and propose future directions for SCI repair and clinical applications.
Collapse
Affiliation(s)
- Xiao Hu
- Division of Spine, Department of Orthopaedics, Tongji Hospital, Tongji University School of Medicine, 200065, Shanghai, China
- Key Laboratory of Spine and Spinal cord Injury Repair and Regeneration (Tongji University), Ministry of Education, 200065, Shanghai, China
- Clinical Center For Brain And Spinal Cord Research, Tongji University, 200065, Shanghai, China
| | - Wei Xu
- Division of Spine, Department of Orthopaedics, Tongji Hospital, Tongji University School of Medicine, 200065, Shanghai, China
- Key Laboratory of Spine and Spinal cord Injury Repair and Regeneration (Tongji University), Ministry of Education, 200065, Shanghai, China
- Clinical Center For Brain And Spinal Cord Research, Tongji University, 200065, Shanghai, China
| | - Yilong Ren
- Division of Spine, Department of Orthopaedics, Tongji Hospital, Tongji University School of Medicine, 200065, Shanghai, China
- Key Laboratory of Spine and Spinal cord Injury Repair and Regeneration (Tongji University), Ministry of Education, 200065, Shanghai, China
- Clinical Center For Brain And Spinal Cord Research, Tongji University, 200065, Shanghai, China
| | - Zhaojie Wang
- Division of Spine, Department of Orthopaedics, Tongji Hospital, Tongji University School of Medicine, 200065, Shanghai, China
- Key Laboratory of Spine and Spinal cord Injury Repair and Regeneration (Tongji University), Ministry of Education, 200065, Shanghai, China
- Clinical Center For Brain And Spinal Cord Research, Tongji University, 200065, Shanghai, China
| | - Xiaolie He
- Division of Spine, Department of Orthopaedics, Tongji Hospital, Tongji University School of Medicine, 200065, Shanghai, China
- Key Laboratory of Spine and Spinal cord Injury Repair and Regeneration (Tongji University), Ministry of Education, 200065, Shanghai, China
- Clinical Center For Brain And Spinal Cord Research, Tongji University, 200065, Shanghai, China
| | - Runzhi Huang
- Division of Spine, Department of Orthopaedics, Tongji Hospital, Tongji University School of Medicine, 200065, Shanghai, China
- Key Laboratory of Spine and Spinal cord Injury Repair and Regeneration (Tongji University), Ministry of Education, 200065, Shanghai, China
- Clinical Center For Brain And Spinal Cord Research, Tongji University, 200065, Shanghai, China
| | - Bei Ma
- Division of Spine, Department of Orthopaedics, Tongji Hospital, Tongji University School of Medicine, 200065, Shanghai, China
- Key Laboratory of Spine and Spinal cord Injury Repair and Regeneration (Tongji University), Ministry of Education, 200065, Shanghai, China
- Clinical Center For Brain And Spinal Cord Research, Tongji University, 200065, Shanghai, China
| | - Jingwei Zhao
- Division of Spine, Department of Orthopaedics, Tongji Hospital, Tongji University School of Medicine, 200065, Shanghai, China
- Key Laboratory of Spine and Spinal cord Injury Repair and Regeneration (Tongji University), Ministry of Education, 200065, Shanghai, China
- Clinical Center For Brain And Spinal Cord Research, Tongji University, 200065, Shanghai, China
| | - Rongrong Zhu
- Division of Spine, Department of Orthopaedics, Tongji Hospital, Tongji University School of Medicine, 200065, Shanghai, China.
- Key Laboratory of Spine and Spinal cord Injury Repair and Regeneration (Tongji University), Ministry of Education, 200065, Shanghai, China.
- Clinical Center For Brain And Spinal Cord Research, Tongji University, 200065, Shanghai, China.
| | - Liming Cheng
- Division of Spine, Department of Orthopaedics, Tongji Hospital, Tongji University School of Medicine, 200065, Shanghai, China.
- Key Laboratory of Spine and Spinal cord Injury Repair and Regeneration (Tongji University), Ministry of Education, 200065, Shanghai, China.
- Clinical Center For Brain And Spinal Cord Research, Tongji University, 200065, Shanghai, China.
| |
Collapse
|
5
|
Branco MP, Geukes SH, Aarnoutse EJ, Ramsey NF, Vansteensel MJ. Nine decades of electrocorticography: A comparison between epidural and subdural recordings. Eur J Neurosci 2023; 57:1260-1288. [PMID: 36843389 DOI: 10.1111/ejn.15941] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2022] [Revised: 02/10/2023] [Accepted: 02/18/2023] [Indexed: 02/28/2023]
Abstract
In recent years, electrocorticography (ECoG) has arisen as a neural signal recording tool in the development of clinically viable neural interfaces. ECoG electrodes are generally placed below the dura mater (subdural) but can also be placed on top of the dura (epidural). In deciding which of these modalities best suits long-term implants, complications and signal quality are important considerations. Conceptually, epidural placement may present a lower risk of complications as the dura is left intact but also a lower signal quality due to the dura acting as a signal attenuator. The extent to which complications and signal quality are affected by the dura, however, has been a matter of debate. To improve our understanding of the effects of the dura on complications and signal quality, we conducted a literature review. We inventorized the effect of the dura on signal quality, decodability and longevity of acute and chronic ECoG recordings in humans and non-human primates. Also, we compared the incidence and nature of serious complications in studies that employed epidural and subdural ECoG. Overall, we found that, even though epidural recordings exhibit attenuated signal amplitude over subdural recordings, particularly for high-density grids, the decodability of epidural recorded signals does not seem to be markedly affected. Additionally, we found that the nature of serious complications was comparable between epidural and subdural recordings. These results indicate that both epidural and subdural ECoG may be suited for long-term neural signal recordings, at least for current generations of clinical and high-density ECoG grids.
Collapse
Affiliation(s)
- Mariana P Branco
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Simon H Geukes
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Erik J Aarnoutse
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Nick F Ramsey
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Mariska J Vansteensel
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
6
|
Jang SJ, Yang YJ, Ryun S, Kim JS, Chung CK, Jeong J. Decoding trajectories of imagined hand movement using electrocorticograms for brain-machine interface. J Neural Eng 2022; 19. [PMID: 35985293 DOI: 10.1088/1741-2552/ac8b37] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2022] [Accepted: 08/19/2022] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Reaching hand movement is an important motor skill actively examined in brain-computer interface (BCI). Among various components of movement analyzed is the hand's trajectory, which describes the hand's continuous positions in three-dimensional space. While a large body of studies have investigated the decoding of real movements and the reconstruction of real hand movement trajectories from neural signals, fewer studies have attempted to decode the trajectory of imagined hand movement. To develop BCI systems for patients with hand motor dysfunctions, the systems essentially require to achieve movement-free control of external devices, which is only possible through successful decoding of purely imagined hand movement. APPROACH To achieve this goal, this study used a machine learning technique (i.e., the variational Bayesian least square) to analyze the electrocorticogram (ECoG) of eighteen epilepsy patients obtained from when they performed movement execution (ME) and kinesthetic movement imagination (KMI) of the reach-and-grasp hand action. MAIN RESULTS The variational Bayesian decoding model was able to successfully predict the imagined trajectories of hand movement significantly above chance level. The Pearson's correlation coefficient between imagined and predicted trajectories was 0.3393 and 0.4936 for the KMI (KMI trials only) and MEKMI paradigm (alternating trials of ME and KMI) respectively. SIGNIFICANCE This study demonstrated a high accuracy of prediction for trajectories of imagined hand movement, and more importantly, higher decoding accuracy of imagined trajectories in the MEKMI paradigm than in the KMI paradigm solely.
Collapse
Affiliation(s)
- Sang Jin Jang
- Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, 411 E16-1(YBS Building) Korea Advanced Institute of Science and Technology, 291 Daehak-ro, Yuseong-gu, Daejeon, South Korea 34141, Daejeon, Daejeon, 34141, Korea (the Republic of)
| | - Yu Jin Yang
- Seoul National University College of Natural Sciences, 103, Daehak-ro, Jongno-gu, Seoul, Republic of Korea, Seoul, 03080, Korea (the Republic of)
| | - Seokyun Ryun
- Seoul National University College of Natural Sciences, 103, Daehak-ro, Jongno-gu, Seoul, Republic of Korea, Seoul, 03080, Korea (the Republic of)
| | - June Sic Kim
- Seoul National University College of Natural Sciences, 103, Daehak-ro, Jongno-gu, Seoul, Republic of Korea, Seoul, 03080, Korea (the Republic of)
| | - Chun Kee Chung
- Seoul National University College of Natural Sciences, 103, Daehak-ro, Jongno-gu, Seoul, Republic of Korea, Seoul, 03080, Korea (the Republic of)
| | - Jaeseung Jeong
- Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, 514 E16-1(YBS Building) Korea Advanced Institute of Science and Technology, 291 Daehak-ro, Yuseong-gu, Daejeon, South Korea 34141, Daejeon, 34141, Korea (the Republic of)
| |
Collapse
|
7
|
Existing function in primary visual cortex is not perturbed by new skill acquisition of a non-matched sensory task. Nat Commun 2022; 13:3638. [PMID: 35752622 PMCID: PMC9233699 DOI: 10.1038/s41467-022-31440-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2021] [Accepted: 06/16/2022] [Indexed: 02/07/2023] Open
Abstract
Acquisition of new skills has the potential to disturb existing network function. To directly assess whether previously acquired cortical function is altered during learning, mice were trained in an abstract task in which selected activity patterns were rewarded using an optical brain-computer interface device coupled to primary visual cortex (V1) neurons. Excitatory neurons were longitudinally recorded using 2-photon calcium imaging. Despite significant changes in local neural activity during task performance, tuning properties and stimulus encoding assessed outside of the trained context were not perturbed. Similarly, stimulus tuning was stable in neurons that remained responsive following a different, visual discrimination training task. However, visual discrimination training increased the rate of representational drift. Our results indicate that while some forms of perceptual learning may modify the contribution of individual neurons to stimulus encoding, new skill learning is not inherently disruptive to the quality of stimulus representation in adult V1.
Collapse
|
8
|
Flint RD, Li Y, Wang P, Vaidya M, Barry A, Ghassemi M, Tomic G, Brkic N, Ripley D, Liu C, Kamper D, Do A, Slutzky MW. Noninvasively recorded high-gamma signals improve synchrony of force feedback in a novel neurorehabilitation brain-machine interface for brain injury. J Neural Eng 2022; 19. [PMID: 35576911 PMCID: PMC9728942 DOI: 10.1088/1741-2552/ac7004] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2021] [Accepted: 05/16/2022] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Brain injury is the leading cause of long-term disability worldwide, often resulting in impaired hand function. Brain-machine interfaces (BMIs) offer a potential way to improve hand function. BMIs often target replacing lost function, but may also be employed in neurorehabilitation (nrBMI) by facilitating neural plasticity and functional recovery. Here, we report a novel nrBMI capable of acquiring high-γ (70-115 Hz) information through a unique post-TBI hemicraniectomy window model, and delivering sensory feedback that is synchronized with, and proportional to, intended grasp force. APPROACH We developed the nrBMI to use electroencephalogram recorded over a hemicraniectomy (hEEG) in individuals with traumatic brain injury (TBI). The nrBMI empowered users to exert continuous, proportional control of applied force, and provided continuous force feedback. We report the results of an initial testing group of three human participants with TBI, along with a control group of three skull- and motor-intact volunteers. MAIN RESULTS All participants controlled the nrBMI successfully, with high initial success rates (2 of 6 participants) or performance that improved over time (4 of 6 participants). We observed high-γ modulation with force intent in hEEG but not skull-intact EEG. Most significantly, we found that high-γ control significantly improved the timing synchronization between neural modulation onset and nrBMI output/haptic feedback (compared to low-frequency nrBMI control). SIGNIFICANCE These proof-of-concept results show that high-γ nrBMIs can be used by individuals with impaired ability to control force (without immediately resorting to invasive signals like ECoG). Of note, the nrBMI includes a parameter to change the fraction of control shared between decoded intent and volitional force, to adjust for recovery progress. The improved synchrony between neural modulations and force control for high-γ signals is potentially important for maximizing the ability of nrBMIs to induce plasticity in neural circuits. Inducing plasticity is critical to functional recovery after brain injury.
Collapse
Affiliation(s)
- Robert D Flint
- Department of Physiology, Northwestern University, Northwestern University, The Feinberg School of Medicine, 303 E. Chicago Ave. , Chicago, IL 60611, USA, Chicago, Illinois, 60611, UNITED STATES
| | - Yongcheng Li
- University of California Irvine, 402 E Peltason Dr, Irvine, California, 92617, UNITED STATES
| | - Po Wang
- University of California Irvine, 402 E Peltason Dr, Irvine, California, 92617, UNITED STATES
| | - Mukta Vaidya
- Northwestern University Feinberg School of Medicine, 320 E Superior St, Chicago, Illinois, 60611-3008, UNITED STATES
| | - Alex Barry
- Shirley Ryan AbilityLab, 355 E Erie St, Chicago, Illinois, 60611-2654, UNITED STATES
| | - Mohammad Ghassemi
- North Carolina State University, Engineering Building III, 4130, Raleigh, North Carolina, 27695, UNITED STATES
| | - Goran Tomic
- Department of Physiology, Northwestern University, Northwestern University, The Feinberg School of Medicine, 303 E. Chicago Ave. , Chicago, IL 60611, USA, Chicago, Illinois, 60611, UNITED STATES
| | - Nenad Brkic
- Shirley Ryan AbilityLab, 355 E Erie St, Chicago, Illinois, 60611-2654, UNITED STATES
| | - David Ripley
- Shirley Ryan AbilityLab, 355 E Erie St, Chicago, Illinois, 60611-2654, UNITED STATES
| | - Charles Liu
- University of California Irvine, 402 E Peltason Dr, Irvine, California, 92617, UNITED STATES
| | - Derek Kamper
- North Carolina State University, Engineering Building III, 4130, Raleigh, North Carolina, 27695, UNITED STATES
| | - An Do
- University of California Irvine, 402 E Peltason Dr, Irvine, California, 92617, UNITED STATES
| | - Marc W Slutzky
- Department of Physiology, Northwestern University Medical School, 303 East Chicago Avenue, Chicago, Illinois, 60611, UNITED STATES
| |
Collapse
|
9
|
Moly A, Costecalde T, Martel F, Martin M, Larzabal C, Karakas S, Verney A, Charvet G, Chabardès S, Benabid AL, Aksenova T. An adaptive closed-loop ECoG decoder for long-term and stable bimanual control of an exoskeleton by a tetraplegic. J Neural Eng 2022; 19. [PMID: 35234665 DOI: 10.1088/1741-2552/ac59a0] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2021] [Accepted: 03/01/2022] [Indexed: 11/12/2022]
Abstract
OBJECTIVE The article aims at addressing 2 challenges to step motor BCI out of laboratories: asynchronous control of complex bimanual effectors with large numbers of degrees of freedom, using chronic and safe recorders, and the decoding performance stability over time without frequent decoder recalibration. APPROACH Closed-loop adaptive/incremental decoder training is one strategy to create a model stable over time. Adaptive decoders update their parameters with new incoming data, optimizing the model parameters in real time. It allows cross-session training with multiple recording conditions during closed loop BCI experiments. In the article, an adaptive tensor-based Recursive Exponentially Weighted Markov-Switching multi-Linear Model (REW-MSLM) decoder is proposed. REW-MSLM uses a Mixture of Expert (ME) architecture, mixing or switching independent decoders (experts) according to the probability estimated by a "gating" model. A Hidden Markov model approach is employed as gating model to improve the decoding robustness and to provide strong idle state support. The ME architecture fits the multi-limb paradigm associating an expert to a particular limb or action. MAIN RESULTS Asynchronous control of an exoskeleton by a tetraplegic patient using a chronically implanted epidural electrocorticography (EpiCoG) recorder is reported. The stable over a period of 6 months (without decoder recalibration) 8-dimensional alternative bimanual control of the exoskeleton and its virtual avatar is demonstrated. SIGNIFICANCE Based on the long-term (>36 months) chronic bilateral epidural ECoG recordings in a tetraplegic (ClinicalTrials.gov, NCT02550522), we addressed the poorly explored field of asynchronous bimanual BCI. The new decoder was designed to meet to several challenges: the high-dimensional control of a complex effector in experiments closer to real-world behaviour (point-to-point pursuit versus conventional center-out tasks), with the ability of the BCI system to act as a stand-alone device switching between idle and control states, and a stable performance over a long period of time without decoder recalibration.
Collapse
Affiliation(s)
- Alexandre Moly
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Thomas Costecalde
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Félix Martel
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Matthieu Martin
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des Martyrs, Grenoble, 38000, FRANCE
| | - Christelle Larzabal
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Serpil Karakas
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Alexandre Verney
- Université Paris-Saclay, Palaiseau, Palaiseau, Île-de-France, 91120, FRANCE
| | - Guillaume Charvet
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| | - Stephan Chabardès
- CHU Grenoble Alpes, Boulevard de la Chantourne, La Tronche, Auvergne-Rhône-Alpes, 38700, FRANCE
| | - Alim-Louis Benabid
- Univ. Grenoble Alpes, CEA, LETI, Clinatec, 17, avenue des Martyrs, Grenoble, 38000, FRANCE
| | - Tatiana Aksenova
- Univ. Grenoble Alpes, CEA, LETI, Clinatec,, 17 av. des martyrs, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| |
Collapse
|
10
|
Kim HH, Jeong J. An electrocorticographic decoder for arm movement for brain–machine interface using an echo state network and Gaussian readout. Appl Soft Comput 2022. [DOI: 10.1016/j.asoc.2021.108393] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/02/2023]
|
11
|
Verwoert M, Vansteensel MJ, Freudenburg ZV, Aarnoutse EJ, Leijten FS, Ramsey NF, Branco MP. Decoding four hand gestures with a single bipolar pair of electrocorticography electrodes. J Neural Eng 2021; 18:10.1088/1741-2552/ac2c9f. [PMID: 34607318 PMCID: PMC8744490 DOI: 10.1088/1741-2552/ac2c9f] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2021] [Accepted: 10/04/2021] [Indexed: 11/12/2022]
Abstract
Objective.Electrocorticography (ECoG) based brain-computer interfaces (BCIs) can be used to restore communication in individuals with locked-in syndrome. In motor-based BCIs, the number of degrees-of-freedom, and thus the speed of the BCI, directly depends on the number of classes that can be discriminated from the neural activity in the sensorimotor cortex. When considering minimally invasive BCI implants, the size of the subdural ECoG implant must be minimized without compromising the number of degrees-of-freedom.Approach.Here we investigated if four hand gestures could be decoded using a single ECoG strip of four consecutive electrodes spaced 1 cm apart and compared the performance between a unipolar and a bipolar montage. For that we collected data of seven individuals with intractable epilepsy implanted with ECoG grids, covering the hand region of the sensorimotor cortex. Based on the implanted grids, we generated virtual ECoG strips and compared the decoding accuracy between (a) a single unipolar electrode (Unipolar Electrode), (b) a combination of four unipolar electrodes (Unipolar Strip), (c) a single bipolar pair (Bipolar Pair) and (d) a combination of six bipolar pairs (Bipolar Strip).Main results.We show that four hand gestures can be equally well decoded using 'Unipolar Strips' (mean 67.4 ± 11.7%), 'Bipolar Strips' (mean 66.6 ± 12.1%) and 'Bipolar Pairs' (mean 67.6 ± 9.4%), while 'Unipolar Electrodes' (61.6 ± 5.9%) performed significantly worse compared to 'Unipolar Strips' and 'Bipolar Pairs'.Significance.We conclude that a single bipolar pair is a potential candidate for minimally invasive motor-based BCIs and encourage the use of ECoG as a robust and reliable BCI platform for multi-class movement decoding.
Collapse
Affiliation(s)
- Maxime Verwoert
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Mariska J. Vansteensel
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Zachary V. Freudenburg
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Erik J. Aarnoutse
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Frans S.S. Leijten
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Nick F. Ramsey
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Mariana P. Branco
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
12
|
Foldes ST, Chandrasekaran S, Camerone J, Lowe J, Ramdeo R, Ebersole J, Bouton CE. Case Study: Mapping Evoked Fields in Primary Motor and Sensory Areas via Magnetoencephalography in Tetraplegia. Front Neurol 2021; 12:739693. [PMID: 34630308 PMCID: PMC8497881 DOI: 10.3389/fneur.2021.739693] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2021] [Accepted: 08/13/2021] [Indexed: 12/02/2022] Open
Abstract
Devices interfacing with the brain through implantation in cortical or subcortical structures have great potential for restoration and rehabilitation in patients with sensory or motor dysfunction. Typical implantation surgeries are planned based on maps of brain activity generated from intact function. However, mapping brain activity for planning implantation surgeries is challenging in the target population due to abnormal residual function and, increasingly often, existing MRI-incompatible implanted hardware. Here, we present methods and results for mapping impaired somatosensory and motor function in an individual with paralysis and an existing brain–computer interface (BCI) device. Magnetoencephalography (MEG) was used to directly map the neural activity evoked during transcutaneous electrical stimulation and attempted movement of the impaired hand. Evoked fields were found to align with the expected anatomy and somatotopic organization. This approach may be valuable for guiding implants in other applications, such as cortical stimulation for pain and to improve implant targeting to help reduce the craniotomy size.
Collapse
Affiliation(s)
- Stephen T Foldes
- Department of Neurology, Barrow Neurological Institute, Phoenix, AZ, United States
| | - Santosh Chandrasekaran
- Neural Bypass and Brain-Computer Interface Laboratory, Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States
| | - Joseph Camerone
- MEG Center, Overlook Medical Center, Atlantic Health, Summit, NJ, United States
| | - James Lowe
- MEG Center, Overlook Medical Center, Atlantic Health, Summit, NJ, United States
| | - Richard Ramdeo
- Neural Bypass and Brain-Computer Interface Laboratory, Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States
| | - John Ebersole
- MEG Center, Overlook Medical Center, Atlantic Health, Summit, NJ, United States
| | - Chad E Bouton
- Neural Bypass and Brain-Computer Interface Laboratory, Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States.,Department of Molecular Medicine, Hofstra-Northwell Medical School, New York, NY, United States
| |
Collapse
|
13
|
Olsen S, Zhang J, Liang KF, Lam M, Riaz U, Kao JC. An artificial intelligence that increases simulated brain-computer interface performance. J Neural Eng 2021; 18. [PMID: 33978599 DOI: 10.1088/1741-2552/abfaaa] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2020] [Accepted: 04/22/2021] [Indexed: 12/14/2022]
Abstract
Objective.Brain-computer interfaces (BCIs) translate neural activity into control signals for assistive devices in order to help people with motor disabilities communicate effectively. In this work, we introduce a new BCI architecture that improves control of a BCI computer cursor to type on a virtual keyboard.Approach.Our BCI architecture incorporates an external artificial intelligence (AI) that beneficially augments the movement trajectories of the BCI. This AI-BCI leverages past user actions, at both long (100 s of seconds ago) and short (100 s of milliseconds ago) timescales, to modify the BCI's trajectories.Main results.We tested our AI-BCI in a closed-loop BCI simulator with nine human subjects performing a typing task. We demonstrate that our AI-BCI achieves: (1) categorically higher information communication rates, (2) quicker ballistic movements between targets, (3) improved precision control to 'dial in' on targets, and (4) more efficient movement trajectories. We further show that our AI-BCI increases performance across a wide control quality spectrum from poor to proficient control.Significance.This AI-BCI architecture, by increasing BCI performance across all key metrics evaluated, may increase the clinical viability of BCI systems.
Collapse
Affiliation(s)
- Sebastian Olsen
- Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90024, United States of America
| | - Jianwei Zhang
- Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90024, United States of America
| | - Ken-Fu Liang
- Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90024, United States of America
| | - Michelle Lam
- Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90024, United States of America
| | - Usama Riaz
- Department of Computer Science, University of California, Los Angeles, CA 90024, United States of America
| | - Jonathan C Kao
- Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90024, United States of America.,Neurosciences Program, University of California, Los Angeles, CA 90024, United States of America
| |
Collapse
|
14
|
Larzabal C, Auboiroux V, Karakas S, Charvet G, Benabid AL, Chabardès S, Costecalde T, Bonnet S. The Riemannian Spatial Pattern method: mapping and clustering movement imagery using Riemannian geometry. J Neural Eng 2021; 18. [PMID: 33770779 DOI: 10.1088/1741-2552/abf291] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2020] [Accepted: 03/26/2021] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Over the last decade, Riemannian geometry has shown promising results for motor imagery classification. However, extracting the underlying spatial features is not as straightforward as for applying Common Spatial Pattern (CSP) filtering prior to classification. In this article, we propose a simple way to extract the spatial patterns obtained from Riemannian classification: the Riemannian Spatial Pattern (RSP) method, which is based on the backward channel selection procedure. APPROACH The RSP method was compared to the CSP approach on ECoG data obtained from a quadriplegic patient while performing imagined movements of arm articulations and fingers. MAIN RESULTS Similar results were found between the RSP and CSP methods for mapping each motor imagery task with activations following the classical somatotopic organization. Clustering obtained by pairwise comparisons of imagined motor movements however, revealed higher differentiation for the RSP method compared to the CSP approach. Importantly, the RSP approach could provide a precise comparison of the imagined finger flexions which added supplementary information to the mapping results. SIGNIFICANCE Our new RSP method illustrates the interest of the Riemannian framework in the spatial domain and as such offers new avenues for the neuroimaging community. This study is part of an ongoing clinical trial registered with ClinicalTrials.gov, NCT02550522.
Collapse
Affiliation(s)
| | - Vincent Auboiroux
- CEA de Grenoble, Clinatec, Grenoble, Auvergne-Rhône-Alpes, 38054, FRANCE
| | - Serpil Karakas
- CEA de Grenoble, Clinatec, Grenoble, Auvergne-Rhône-Alpes, 38054, FRANCE
| | - Guillaume Charvet
- CEA de Grenoble, Clinatec, Grenoble, Auvergne-Rhône-Alpes, 38054, FRANCE
| | - Alim-Louis Benabid
- CEA de Grenoble, Clinatec, Grenoble, Auvergne-Rhône-Alpes, 38054, FRANCE
| | | | - Thomas Costecalde
- CEA de Grenoble, Clinatec, Grenoble, Auvergne-Rhône-Alpes, 38054, FRANCE
| | - Stephane Bonnet
- CEA de Grenoble, DTBS, Grenoble, Auvergne-Rhône-Alpes, 38000, FRANCE
| |
Collapse
|
15
|
Plug-and-play control of a brain-computer interface through neural map stabilization. Nat Biotechnol 2020; 39:326-335. [PMID: 32895549 DOI: 10.1038/s41587-020-0662-5] [Citation(s) in RCA: 23] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2020] [Accepted: 08/04/2020] [Indexed: 11/08/2022]
Abstract
Brain-computer interfaces (BCIs) enable control of assistive devices in individuals with severe motor impairments. A limitation of BCIs that has hindered real-world adoption is poor long-term reliability and lengthy daily recalibration times. To develop methods that allow stable performance without recalibration, we used a 128-channel chronic electrocorticography (ECoG) implant in a paralyzed individual, which allowed stable monitoring of signals. We show that long-term closed-loop decoder adaptation, in which decoder weights are carried across sessions over multiple days, results in consolidation of a neural map and 'plug-and-play' control. In contrast, daily reinitialization led to degradation of performance with variable relearning. Consolidation also allowed the addition of control features over days, that is, long-term stacking of dimensions. Our results offer an approach for reliable, stable BCI control by leveraging the stability of ECoG interfaces and neural plasticity.
Collapse
|
16
|
Intra-cortical brain-machine interfaces for controlling upper-limb powered muscle and robotic systems in spinal cord injury. Clin Neurol Neurosurg 2020; 196:106069. [DOI: 10.1016/j.clineuro.2020.106069] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2020] [Revised: 07/03/2020] [Accepted: 07/04/2020] [Indexed: 12/20/2022]
|
17
|
Stabilization of a brain-computer interface via the alignment of low-dimensional spaces of neural activity. Nat Biomed Eng 2020; 4:672-685. [PMID: 32313100 DOI: 10.1038/s41551-020-0542-9] [Citation(s) in RCA: 84] [Impact Index Per Article: 21.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2018] [Accepted: 02/21/2020] [Indexed: 12/31/2022]
Abstract
The instability of neural recordings can render clinical brain-computer interfaces (BCIs) uncontrollable. Here, we show that the alignment of low-dimensional neural manifolds (low-dimensional spaces that describe specific correlation patterns between neurons) can be used to stabilize neural activity, thereby maintaining BCI performance in the presence of recording instabilities. We evaluated the stabilizer with non-human primates during online cursor control via intracortical BCIs in the presence of severe and abrupt recording instabilities. The stabilized BCIs recovered proficient control under different instability conditions and across multiple days. The stabilizer does not require knowledge of user intent and can outperform supervised recalibration. It stabilized BCIs even when neural activity contained little information about the direction of cursor movement. The stabilizer may be applicable to other neural interfaces and may improve the clinical viability of BCIs.
Collapse
|
18
|
Vaughan TM. Brain-computer interfaces for people with amyotrophic lateral sclerosis. HANDBOOK OF CLINICAL NEUROLOGY 2020; 168:33-38. [DOI: 10.1016/b978-0-444-63934-9.00004-4] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/21/2022]
|
19
|
Volkova K, Lebedev MA, Kaplan A, Ossadtchi A. Decoding Movement From Electrocorticographic Activity: A Review. Front Neuroinform 2019; 13:74. [PMID: 31849632 PMCID: PMC6901702 DOI: 10.3389/fninf.2019.00074] [Citation(s) in RCA: 37] [Impact Index Per Article: 7.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2019] [Accepted: 11/14/2019] [Indexed: 01/08/2023] Open
Abstract
Electrocorticography (ECoG) holds promise to provide efficient neuroprosthetic solutions for people suffering from neurological disabilities. This recording technique combines adequate temporal and spatial resolution with the lower risks of medical complications compared to the other invasive methods. ECoG is routinely used in clinical practice for preoperative cortical mapping in epileptic patients. During the last two decades, research utilizing ECoG has considerably grown, including the paradigms where behaviorally relevant information is extracted from ECoG activity with decoding algorithms of different complexity. Several research groups have advanced toward the development of assistive devices driven by brain-computer interfaces (BCIs) that decode motor commands from multichannel ECoG recordings. Here we review the evolution of this field and its recent tendencies, and discuss the potential areas for future development.
Collapse
Affiliation(s)
- Ksenia Volkova
- Center for Bioelectric Interfaces, Higher School of Economics, National Research University, Moscow, Russia
| | - Mikhail A. Lebedev
- Center for Bioelectric Interfaces, Higher School of Economics, National Research University, Moscow, Russia
| | - Alexander Kaplan
- Center for Bioelectric Interfaces, Higher School of Economics, National Research University, Moscow, Russia
- Center for Biotechnology Development, National Research Lobachevsky State University of Nizhny Novgorod, Nizhny Novgorod, Russia
- Laboratory for Neurophysiology and Neuro-Computer Interfaces, Faculty of Biology, Lomonosov Moscow State University, Moscow, Russia
| | - Alexei Ossadtchi
- Center for Bioelectric Interfaces, Higher School of Economics, National Research University, Moscow, Russia
| |
Collapse
|
20
|
Loza CA, Reddy CG, Akella S, Príncipe JC. Discrimination of Movement-Related Cortical Potentials Exploiting Unsupervised Learned Representations From ECoGs. Front Neurosci 2019; 13:1248. [PMID: 31824249 PMCID: PMC6882771 DOI: 10.3389/fnins.2019.01248] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2019] [Accepted: 11/05/2019] [Indexed: 11/13/2022] Open
Abstract
Brain–Computer Interfaces (BCI) aim to bypass the peripheral nervous system to link the brain to external devices via successful modeling of decoding mechanisms. BCI based on electrocorticogram or ECoG represent a viable compromise between clinical practicality, spatial resolution, and signal quality when it comes to extracellular electrical potentials from local neuronal assemblies. Classic analysis of ECoG traces usually falls under the umbrella of Time-Frequency decompositions with adaptations from Fourier analysis and wavelets as its most prominent variants. However, analyzing such high-dimensional, multivariate time series demands for specialized signal processing and neurophysiological principles. We propose a generative model for single-channel ECoGs that is able to fully characterize reoccurring rhythm–specific neuromodulations as weighted activations of prototypical templates over time. The set of timings, weights and indexes comprise a temporal marked point process (TMPP) that accesses a set of bases from vector spaces of different dimensions—a dictionary. The shallow nature of the model admits the equivalence between latent variables and representations. In this way, learning the model parameters is a case of unsupervised representation learning. We exploit principles of Minimum Description Length (MDL) encoding to effectively yield a data-driven framework where prototypical neuromodulations (not restricted to a particular duration) can be estimated alongside the timings and features of the TMPP. We validate the proposed methodology on discrimination of movement-related tasks utilizing 32-electrode grids implanted in the frontal cortex of six epileptic subjects. We show that the learned representations from the high-gamma band (85–145 Hz) are not only interpretable, but also discriminant in a lower dimensional space. The results also underscore the practicality of our algorithm, i.e., 2 main hyperparameters that can be readily set via neurophysiology, and emphasize the need of principled and interpretable representation learning in order to model encoding mechanisms in the brain.
Collapse
Affiliation(s)
- Carlos A. Loza
- Department of Mathematics, Universidad San Francisco de Quito, Quito, Ecuador
- Instituto de Neurociencias, Universidad San Francisco de Quito, Quito, Ecuador
- *Correspondence: Carlos A. Loza
| | - Chandan G. Reddy
- Department of Neurosurgery, University of Iowa, Iowa City, IA, United States
- Department of Neurosurgery, University of Florida, Gainesville, FL, United States
- Computational NeuroEngineering Lab, Electrical and Computer Engineering Department, University of Florida, Gainesville, FL, United States
| | - Shailaja Akella
- Computational NeuroEngineering Lab, Electrical and Computer Engineering Department, University of Florida, Gainesville, FL, United States
| | - José C. Príncipe
- Computational NeuroEngineering Lab, Electrical and Computer Engineering Department, University of Florida, Gainesville, FL, United States
| |
Collapse
|
21
|
Kato K, Sawada M, Nishimura Y. Bypassing stroke-damaged neural pathways via a neural interface induces targeted cortical adaptation. Nat Commun 2019; 10:4699. [PMID: 31619680 PMCID: PMC6796004 DOI: 10.1038/s41467-019-12647-y] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2019] [Accepted: 09/20/2019] [Indexed: 12/03/2022] Open
Abstract
Regaining the function of an impaired limb is highly desirable in paralyzed individuals. One possible avenue to achieve this goal is to bridge the interrupted pathway between preserved neural structures and muscles using a brain–computer interface. Here, we demonstrate that monkeys with subcortical stroke were able to learn to use an artificial cortico-muscular connection (ACMC), which transforms cortical activity into electrical stimulation to the hand muscles, to regain volitional control of a paralysed hand. The ACMC induced an adaptive change of cortical activities throughout an extensive cortical area. In a targeted manner, modulating high-gamma activity became localized around an arbitrarily-selected cortical site controlling stimulation to the muscles. This adaptive change could be reset and localized rapidly to a new cortical site. Thus, the ACMC imparts new function for muscle control to connected cortical sites and triggers cortical adaptation to regain impaired motor function after stroke. Monkeys were trained to use an artificial cortico-muscular connection (ACMC) to regain control over a paralyzed hand following subcortical stroke. Control over the paralyzed hand was accompanied by the appearance of localized high-gamma modulation in the cortex, which could be rapidly reset and relocalized to a different cortical site to reactivate motor control.
Collapse
Affiliation(s)
- Kenji Kato
- Department of Developmental Physiology, National Institute for Physiological Sciences, 38 Nishigonaka, Myodaiji, Okazaki, Aichi, 444-8585, Japan.,Department of Physiological Sciences, School of Life Science, The Graduate University for Advanced Studies, SOKENDAI, Shonan Village, Hayama, Kanagawa, 240-0193, Japan.,Japan Society for The Promotion of Science, Kojimachi Business Center Building, 5-3-1 Kojimachi, Chiyoda, Tokyo, 102-0083, Japan.,Center of Assistive Robotics and Rehabilitation for Longevity and Good Health, National Center for Geriatrics and Gerontology, 7-430, Morioka, Obu, Aichi, 474-8511, Japan
| | - Masahiro Sawada
- Department of Developmental Physiology, National Institute for Physiological Sciences, 38 Nishigonaka, Myodaiji, Okazaki, Aichi, 444-8585, Japan.,Department of Neurosurgery, Graduate School of Kyoto University, 54 Shogoin-kawaharacho, Sakyo, Kyoto, 606-8507, Japan
| | - Yukio Nishimura
- Department of Developmental Physiology, National Institute for Physiological Sciences, 38 Nishigonaka, Myodaiji, Okazaki, Aichi, 444-8585, Japan. .,Department of Physiological Sciences, School of Life Science, The Graduate University for Advanced Studies, SOKENDAI, Shonan Village, Hayama, Kanagawa, 240-0193, Japan. .,Neural Prosthesis Project, Department of Dementia and Higher Brain Function, Tokyo Metropolitan Institute of Medical Science, 2-1-6, Kamikitazawa, Setagaya, Tokyo, 158-8506, Japan. .,Department of Neuroscience, Graduate School of Medicine and Faculty of Medicine, Kyoto University, Yoshida-Konoe, Sakyo, Kyoto, 606-8501, Japan. .,Precursory Research for Embryonic Science and Technology, Japan Science and Technology Agency, Sanban-tyo, Chiyoda, Tokyo, 102-0076, Japan.
| |
Collapse
|
22
|
Branco MP, de Boer LM, Ramsey NF, Vansteensel MJ. Encoding of kinetic and kinematic movement parameters in the sensorimotor cortex: A Brain-Computer Interface perspective. Eur J Neurosci 2019; 50:2755-2772. [PMID: 30633413 PMCID: PMC6625947 DOI: 10.1111/ejn.14342] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2018] [Revised: 11/30/2018] [Accepted: 01/07/2019] [Indexed: 01/23/2023]
Abstract
For severely paralyzed people, Brain-Computer Interfaces (BCIs) can potentially replace lost motor output and provide a brain-based control signal for augmentative and alternative communication devices or neuroprosthetics. Many BCIs focus on neuronal signals acquired from the hand area of the sensorimotor cortex, employing changes in the patterns of neuronal firing or spectral power associated with one or more types of hand movement. Hand and finger movement can be described by two groups of movement features, namely kinematics (spatial and motion aspects) and kinetics (muscles and forces). Despite extensive primate and human research, it is not fully understood how these features are represented in the SMC and how they lead to the appropriate movement. Yet, the available information may provide insight into which features are most suitable for BCI control. To that purpose, the current paper provides an in-depth review on the movement features encoded in the SMC. Even though there is no consensus on how exactly the SMC generates movement, we conclude that some parameters are well represented in the SMC and can be accurately used for BCI control with discrete as well as continuous feedback. However, the vast evidence also suggests that movement should be interpreted as a combination of multiple parameters rather than isolated ones, pleading for further exploration of sensorimotor control models for accurate BCI control.
Collapse
Affiliation(s)
- Mariana P. Branco
- Brain Center Rudolf MagnusDepartment of Neurology and NeurosurgeryUniversity Medical Center UtrechtUtrechtThe Netherlands
| | | | - Nick F. Ramsey
- Brain Center Rudolf MagnusDepartment of Neurology and NeurosurgeryUniversity Medical Center UtrechtUtrechtThe Netherlands
| | - Mariska J. Vansteensel
- Brain Center Rudolf MagnusDepartment of Neurology and NeurosurgeryUniversity Medical Center UtrechtUtrechtThe Netherlands
| |
Collapse
|
23
|
Yang Y, Sani OG, Chang EF, Shanechi MM. Dynamic network modeling and dimensionality reduction for human ECoG activity. J Neural Eng 2019; 16:056014. [DOI: 10.1088/1741-2552/ab2214] [Citation(s) in RCA: 28] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/31/2022]
|
24
|
Pailla T, Miller KJ, Gilja V. Autoencoders for learning template spectrograms in electrocorticographic signals. J Neural Eng 2019; 16:016025. [DOI: 10.1088/1741-2552/aaf13f] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|
25
|
Woods V, Trumpis M, Bent B, Palopoli-Trojani K, Chiang CH, Wang C, Yu C, Insanally MN, Froemke RC, Viventi J. Long-term recording reliability of liquid crystal polymer µECoG arrays. J Neural Eng 2018; 15:066024. [PMID: 30246690 PMCID: PMC6342453 DOI: 10.1088/1741-2552/aae39d] [Citation(s) in RCA: 35] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE The clinical use of microsignals recorded over broad cortical regions is largely limited by the chronic reliability of the implanted interfaces. APPROACH We evaluated the chronic reliability of novel 61-channel micro-electrocorticographic (µECoG) arrays in rats chronically implanted for over one year and using accelerated aging. Devices were encapsulated with polyimide (PI) or liquid crystal polymer (LCP), and fabricated using commercial manufacturing processes. In vitro failure modes and predicted lifetimes were determined from accelerated soak testing. Successful designs were implanted epidurally over the rodent auditory cortex. Trends in baseline signal level, evoked responses and decoding performance were reported for over one year of implantation. MAIN RESULTS Devices fabricated with LCP consistently had longer in vitro lifetimes than PI encapsulation. Our accelerated aging results predicted device integrity beyond 3.4 years. Five implanted arrays showed stable performance over the entire implantation period (247-435 d). Our regression analysis showed that impedance predicted signal quality and information content only in the first 31 d of recordings and had little predictive value in the chronic phase (>31 d). In the chronic phase, site impedances slightly decreased yet decoding performance became statistically uncorrelated with impedance. We also employed an improved statistical model of spatial variation to measure sensitivity to locally varying fields, which is typically concealed in standard signal power calculations. SIGNIFICANCE These findings show that µECoG arrays can reliably perform in chronic applications in vivo for over one year, which facilitates the development of a high-density, clinically viable interface.
Collapse
Affiliation(s)
- Virginia Woods
- Department of Biomedical Engineering, Duke University, Durham, NC, United States of America
| | | | | | | | | | | | | | | | | | | |
Collapse
|
26
|
Choi SI, Han CH, Choi GY, Shin J, Song KS, Im CH, Hwang HJ. On the Feasibility of Using an Ear-EEG to Develop an Endogenous Brain-Computer Interface. SENSORS (BASEL, SWITZERLAND) 2018; 18:E2856. [PMID: 30158505 PMCID: PMC6165202 DOI: 10.3390/s18092856] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/20/2018] [Revised: 08/26/2018] [Accepted: 08/28/2018] [Indexed: 11/27/2022]
Abstract
Brain-computer interface (BCI) studies based on electroencephalography (EEG) measured around the ears (ear-EEGs) have mostly used exogenous paradigms involving brain activity evoked by external stimuli. The objective of this study is to investigate the feasibility of ear-EEGs for development of an endogenous BCI system that uses self-modulated brain activity. We performed preliminary and main experiments where EEGs were measured on the scalp and behind the ears to check the reliability of ear-EEGs as compared to scalp-EEGs. In the preliminary and main experiments, subjects performed eyes-open and eyes-closed tasks, and they performed mental arithmetic (MA) and light cognitive (LC) tasks, respectively. For data analysis, the brain area was divided into four regions of interest (ROIs) (i.e., frontal, central, occipital, and ear area). The preliminary experiment showed that the degree of alpha activity increase of the ear area with eyes closed is comparable to those of other ROIs (occipital > ear > central > frontal). In the main experiment, similar event-related (de)synchronization (ERD/ERS) patterns were observed between the four ROIs during MA and LC, and all ROIs showed the mean classification accuracies above 70% required for effective binary communication (MA vs. LC) (occipital = ear = central = frontal). From the results, we demonstrated that ear-EEG can be used to develop an endogenous BCI system based on cognitive tasks without external stimuli, which allows the usability of ear-EEGs to be extended.
Collapse
Affiliation(s)
- Soo-In Choi
- Department of Medical IT Convergence Engineering, Kumoh National Institute of Technology, Gumi 39177, Korea.
| | - Chang-Hee Han
- Berlin Institute of Technology, Machine Learning Group, Marchstrasse 23, 10587 Berlin, Germany.
| | - Ga-Young Choi
- Department of Medical IT Convergence Engineering, Kumoh National Institute of Technology, Gumi 39177, Korea.
| | - Jaeyoung Shin
- Department of Biomedical Engineering, Hanyang University, Seoul 04763, Korea.
| | - Kwang Soup Song
- Department of Medical IT Convergence Engineering, Kumoh National Institute of Technology, Gumi 39177, Korea.
| | - Chang-Hwan Im
- Department of Biomedical Engineering, Hanyang University, Seoul 04763, Korea.
| | - Han-Jeong Hwang
- Department of Medical IT Convergence Engineering, Kumoh National Institute of Technology, Gumi 39177, Korea.
| |
Collapse
|
27
|
Schaeffer MC, Aksenova T. Data-Driven Transducer Design and Identification for Internally-Paced Motor Brain Computer Interfaces: A Review. Front Neurosci 2018; 12:540. [PMID: 30158847 PMCID: PMC6104172 DOI: 10.3389/fnins.2018.00540] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2017] [Accepted: 07/17/2018] [Indexed: 11/13/2022] Open
Abstract
Brain-Computer Interfaces (BCIs) are systems that establish a direct communication pathway between the users' brain activity and external effectors. They offer the potential to improve the quality of life of motor-impaired patients. Motor BCIs aim to permit severely motor-impaired users to regain limb mobility by controlling orthoses or prostheses. In particular, motor BCI systems benefit patients if the decoded actions reflect the users' intentions with an accuracy that enables them to efficiently interact with their environment. One of the main challenges of BCI systems is to adapt the BCI's signal translation blocks to the user to reach a high decoding accuracy. This paper will review the literature of data-driven and user-specific transducer design and identification approaches and it focuses on internally-paced motor BCIs. In particular, continuous kinematic biomimetic and mental-task decoders are reviewed. Furthermore, static and dynamic decoding approaches, linear and non-linear decoding, offline and real-time identification algorithms are considered. The current progress and challenges related to the design of clinical-compatible motor BCI transducers are additionally discussed.
Collapse
Affiliation(s)
| | - Tetiana Aksenova
- CEA, LETI, CLINATEC, MINATEC Campus, Université Grenoble Alpes, Grenoble, France
| |
Collapse
|