1
|
Alcolea P, Ma X, Bodkin K, Miller LE, Danziger ZC. Less is more: selection from a small set of options improves BCI velocity control. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.03.596241. [PMID: 38895473 PMCID: PMC11185569 DOI: 10.1101/2024.06.03.596241] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/21/2024]
Abstract
We designed the discrete direction selection (DDS) decoder for intracortical brain computer interface (iBCI) cursor control and showed that it outperformed currently used decoders in a human-operated real-time iBCI simulator and in monkey iBCI use. Unlike virtually all existing decoders that map between neural activity and continuous velocity commands, DDS uses neural activity to select among a small menu of preset cursor velocities. We compared closed-loop cursor control across four visits by each of 48 naïve, able-bodied human subjects using either DDS or one of three common continuous velocity decoders: direct regression with assist (an affine map from neural activity to cursor velocity), ReFIT, and the velocity Kalman Filter. DDS outperformed all three by a substantial margin. Subsequently, a monkey using an iBCI also had substantially better performance with DDS than with the Wiener filter decoder (direct regression decoder that includes time history). Discretizing the decoded velocity with DDS effectively traded high resolution velocity commands for less tortuous and lower noise trajectories, highlighting the potential benefits of simplifying online iBCI control.
Collapse
Affiliation(s)
- Pedro Alcolea
- Department of Biomedical Engineering, Florida International University, Miami, FL 33199, USA
| | - Xuan Ma
- Department of Neuroscience, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
| | - Kevin Bodkin
- Department of Neuroscience, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
| | - Lee E. Miller
- Department of Neuroscience, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
- Department of Physical Medicine and Rehabilitation, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
- Department of Biomedical Engineering, McCormick School of Engineering, Northwestern University, Evanston, IL 60208, USA
- Shirley Ryan AbilityLab, Chicago, IL 60611, USA
| | - Zachary C. Danziger
- Department of Biomedical Engineering, Florida International University, Miami, FL 33199, USA
- Department of Rehabilitation Medicine - Division of Physical Therapy, Emory University, Atlanta, GA 30322, USA
- W.H. Coulter Department of Biomedical Engineering, Emory University Atlanta, GA 30322, USA
| |
Collapse
|
2
|
Deo DR, Willett FR, Avansino DT, Hochberg LR, Henderson JM, Shenoy KV. Brain control of bimanual movement enabled by recurrent neural networks. Sci Rep 2024; 14:1598. [PMID: 38238386 PMCID: PMC10796685 DOI: 10.1038/s41598-024-51617-3] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Accepted: 01/07/2024] [Indexed: 01/22/2024] Open
Abstract
Brain-computer interfaces have so far focused largely on enabling the control of a single effector, for example a single computer cursor or robotic arm. Restoring multi-effector motion could unlock greater functionality for people with paralysis (e.g., bimanual movement). However, it may prove challenging to decode the simultaneous motion of multiple effectors, as we recently found that a compositional neural code links movements across all limbs and that neural tuning changes nonlinearly during dual-effector motion. Here, we demonstrate the feasibility of high-quality bimanual control of two cursors via neural network (NN) decoders. Through simulations, we show that NNs leverage a neural 'laterality' dimension to distinguish between left and right-hand movements as neural tuning to both hands become increasingly correlated. In training recurrent neural networks (RNNs) for two-cursor control, we developed a method that alters the temporal structure of the training data by dilating/compressing it in time and re-ordering it, which we show helps RNNs successfully generalize to the online setting. With this method, we demonstrate that a person with paralysis can control two computer cursors simultaneously. Our results suggest that neural network decoders may be advantageous for multi-effector decoding, provided they are designed to transfer to the online setting.
Collapse
Affiliation(s)
- Darrel R Deo
- Department of Neurosurgery, Stanford University, Stanford, CA, USA.
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA.
| | - Francis R Willett
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Donald T Avansino
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Leigh R Hochberg
- School of Engineering, Brown University, Providence, RI, USA
- Carney Institute for Brain Science, Brown University, Providence, RI, USA
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, USA
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Bio-X Institute, Stanford University, Stanford, CA, USA
| | - Krishna V Shenoy
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
- Bio-X Institute, Stanford University, Stanford, CA, USA
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Department of Bioengineering, Stanford University, Stanford, CA, USA
- Department of Neurobiology, Stanford University, Stanford, CA, USA
| |
Collapse
|
3
|
Rizzoglio F, Altan E, Ma X, Bodkin KL, Dekleva BM, Solla SA, Kennedy A, Miller LE. From monkeys to humans: observation-basedEMGbrain-computer interface decoders for humans with paralysis. J Neural Eng 2023; 20:056040. [PMID: 37844567 PMCID: PMC10618714 DOI: 10.1088/1741-2552/ad038e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2023] [Revised: 10/02/2023] [Accepted: 10/16/2023] [Indexed: 10/18/2023]
Abstract
Objective. Intracortical brain-computer interfaces (iBCIs) aim to enable individuals with paralysis to control the movement of virtual limbs and robotic arms. Because patients' paralysis prevents training a direct neural activity to limb movement decoder, most iBCIs rely on 'observation-based' decoding in which the patient watches a moving cursor while mentally envisioning making the movement. However, this reliance on observed target motion for decoder development precludes its application to the prediction of unobservable motor output like muscle activity. Here, we ask whether recordings of muscle activity from a surrogate individual performing the same movement as the iBCI patient can be used as target for an iBCI decoder.Approach. We test two possible approaches, each using data from a human iBCI user and a monkey, both performing similar motor actions. In one approach, we trained a decoder to predict the electromyographic (EMG) activity of a monkey from neural signals recorded from a human. We then contrast this to a second approach, based on the hypothesis that the low-dimensional 'latent' neural representations of motor behavior, known to be preserved across time for a given behavior, might also be preserved across individuals. We 'transferred' an EMG decoder trained solely on monkey data to the human iBCI user after using Canonical Correlation Analysis to align the human latent signals to those of the monkey.Main results. We found that both direct and transfer decoding approaches allowed accurate EMG predictions between two monkeys and from a monkey to a human.Significance. Our findings suggest that these latent representations of behavior are consistent across animals and even primate species. These methods are an important initial step in the development of iBCI decoders that generate EMG predictions that could serve as signals for a biomimetic decoder controlling motion and impedance of a prosthetic arm, or even muscle force directly through functional electrical stimulation.
Collapse
Affiliation(s)
- Fabio Rizzoglio
- Department of Neuroscience, Northwestern University, Chicago, IL, United States of America
| | - Ege Altan
- Department of Neuroscience, Northwestern University, Chicago, IL, United States of America
- Department of Biomedical Engineering, Northwestern University, Evanston, IL, United States of America
| | - Xuan Ma
- Department of Neuroscience, Northwestern University, Chicago, IL, United States of America
| | - Kevin L Bodkin
- Department of Neuroscience, Northwestern University, Chicago, IL, United States of America
| | - Brian M Dekleva
- Rehab Neural Engineering Labs, Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, United States of America
| | - Sara A Solla
- Department of Neuroscience, Northwestern University, Chicago, IL, United States of America
- Department of Physics and Astronomy, Northwestern University, Evanston, IL, United States of America
| | - Ann Kennedy
- Department of Neuroscience, Northwestern University, Chicago, IL, United States of America
| | - Lee E Miller
- Department of Neuroscience, Northwestern University, Chicago, IL, United States of America
- Department of Biomedical Engineering, Northwestern University, Evanston, IL, United States of America
- Shirley Ryan AbilityLab, Chicago, IL, United States of America
- Department of Physical Medicine and Rehabilitation, Northwestern University, Chicago, IL, United States of America
| |
Collapse
|
4
|
Mirfathollahi A, Ghodrati MT, Shalchyan V, Zarrindast MR, Daliri MR. Decoding hand kinetics and kinematics using somatosensory cortex activity in active and passive movement. iScience 2023; 26:107808. [PMID: 37736040 PMCID: PMC10509302 DOI: 10.1016/j.isci.2023.107808] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 07/20/2023] [Accepted: 08/30/2023] [Indexed: 09/23/2023] Open
Abstract
Area 2 of the primary somatosensory cortex (S1), encodes proprioceptive information of limbs. Several studies investigated the encoding of movement parameters in this area. However, the single-trial decoding of these parameters, which can provide additional knowledge about the amount of information available in sub-regions of this area about instantaneous limb movement, has not been well investigated. We decoded kinematic and kinetic parameters of active and passive hand movement during center-out task using conventional and state-based decoders. Our results show that this area can be used to accurately decode position, velocity, force, moment, and joint angles of hand. Kinematics had higher accuracies compared to kinetics and active trials were decoded more accurately than passive trials. Although the state-based decoder outperformed the conventional decoder in the active task, it was the opposite in the passive task. These results can be used in intracortical micro-stimulation procedures to provide proprioceptive feedback to BCI subjects.
Collapse
Affiliation(s)
- Alavie Mirfathollahi
- Institute for Cognitive Science Studies (ICSS), Pardis 16583- 44575 Tehran, Iran
- Neuroscience & Neuroengineering Research Lab, Biomedical Engineering Department, School of Electrical Engineering, Iran University of Science and Technology (IUST), Narmak, Tehran 16846-13114, Iran
| | - Mohammad Taghi Ghodrati
- Neuroscience & Neuroengineering Research Lab, Biomedical Engineering Department, School of Electrical Engineering, Iran University of Science and Technology (IUST), Narmak, Tehran 16846-13114, Iran
| | - Vahid Shalchyan
- Neuroscience & Neuroengineering Research Lab, Biomedical Engineering Department, School of Electrical Engineering, Iran University of Science and Technology (IUST), Narmak, Tehran 16846-13114, Iran
| | - Mohammad Reza Zarrindast
- Institute for Cognitive Science Studies (ICSS), Pardis 16583- 44575 Tehran, Iran
- Department of Pharmacology, School of Medicine, Tehran University of Medical Sciences, Tehran 14166-34793, Iran
| | - Mohammad Reza Daliri
- Institute for Cognitive Science Studies (ICSS), Pardis 16583- 44575 Tehran, Iran
- Neuroscience & Neuroengineering Research Lab, Biomedical Engineering Department, School of Electrical Engineering, Iran University of Science and Technology (IUST), Narmak, Tehran 16846-13114, Iran
| |
Collapse
|
5
|
Ye J, Collinger JL, Wehbe L, Gaunt R. Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.09.18.558113. [PMID: 37781630 PMCID: PMC10541112 DOI: 10.1101/2023.09.18.558113] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/03/2023]
Abstract
The neural population spiking activity recorded by intracortical brain-computer interfaces (iBCIs) contain rich structure. Current models of such spiking activity are largely prepared for individual experimental contexts, restricting data volume to that collectable within a single session and limiting the effectiveness of deep neural networks (DNNs). The purported challenge in aggregating neural spiking data is the pervasiveness of context-dependent shifts in the neural data distributions. However, large scale unsupervised pretraining by nature spans heterogeneous data, and has proven to be a fundamental recipe for successful representation learning across deep learning. We thus develop Neural Data Transformer 2 (NDT2), a spatiotemporal Transformer for neural spiking activity, and demonstrate that pretraining can leverage motor BCI datasets that span sessions, subjects, and experimental tasks. NDT2 enables rapid adaptation to novel contexts in downstream decoding tasks and opens the path to deployment of pretrained DNNs for iBCI control. Code: https://github.com/joel99/context_general_bci.
Collapse
Affiliation(s)
- Joel Ye
- Rehab Neural Engineering Labs, University of Pittsburgh
- Neuroscience Institute, Carnegie Mellon University
- Center for the Neural Basis of Cognition, Pittsburgh
| | - Jennifer L. Collinger
- Rehab Neural Engineering Labs, University of Pittsburgh
- Center for the Neural Basis of Cognition, Pittsburgh
- Department of Physical Medicine and Rehabilitation, University of Pittsburgh
- Department of Bioengineering, University of Pittsburgh
- Department of Biomedical Engineering, Carnegie Mellon University
| | - Leila Wehbe
- Neuroscience Institute, Carnegie Mellon University
- Center for the Neural Basis of Cognition, Pittsburgh
- Machine Learning Department, Carnegie Mellon University
| | - Robert Gaunt
- Rehab Neural Engineering Labs, University of Pittsburgh
- Center for the Neural Basis of Cognition, Pittsburgh
- Department of Physical Medicine and Rehabilitation, University of Pittsburgh
- Department of Bioengineering, University of Pittsburgh
- Department of Biomedical Engineering, Carnegie Mellon University
| |
Collapse
|
6
|
Bergeron D, Iorio-Morin C, Bonizzato M, Lajoie G, Orr Gaucher N, Racine É, Weil AG. Use of Invasive Brain-Computer Interfaces in Pediatric Neurosurgery: Technical and Ethical Considerations. J Child Neurol 2023; 38:223-238. [PMID: 37116888 PMCID: PMC10226009 DOI: 10.1177/08830738231167736] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/20/2022] [Revised: 02/11/2023] [Accepted: 03/17/2023] [Indexed: 04/30/2023]
Abstract
Invasive brain-computer interfaces hold promise to alleviate disabilities in individuals with neurologic injury, with fully implantable brain-computer interface systems expected to reach the clinic in the upcoming decade. Children with severe neurologic disabilities, like quadriplegic cerebral palsy or cervical spine trauma, could benefit from this technology. However, they have been excluded from clinical trials of intracortical brain-computer interface to date. In this manuscript, we discuss the ethical considerations related to the use of invasive brain-computer interface in children with severe neurologic disabilities. We first review the technical hardware and software considerations for the application of intracortical brain-computer interface in children. We then discuss ethical issues related to motor brain-computer interface use in pediatric neurosurgery. Finally, based on the input of a multidisciplinary panel of experts in fields related to brain-computer interface (functional and restorative neurosurgery, pediatric neurosurgery, mathematics and artificial intelligence research, neuroengineering, pediatric ethics, and pragmatic ethics), we then formulate initial recommendations regarding the clinical use of invasive brain-computer interfaces in children.
Collapse
Affiliation(s)
- David Bergeron
- Division of Neurosurgery, Université de Montréal, Montreal, Québec, Canada
| | | | - Marco Bonizzato
- Electrical Engineering Department, Polytechnique Montréal, Montreal, Québec, Canada
- Neuroscience Department and Centre
interdisciplinaire de recherche sur le cerveau et l’apprentissage (CIRCA), Université de Montréal, Montréal, Québec, Canada
| | - Guillaume Lajoie
- Mathematics and Statistics Department, Université de Montréal, Montreal, Québec, Canada
- Mila - Québec AI Institute, Montréal,
Québec, Canada
| | - Nathalie Orr Gaucher
- Department of Pediatric Emergency
Medicine, CHU Sainte-Justine, Montréal, Québec, Canada
- Bureau de l’Éthique clinique, Faculté
de médecine de l’Université de Montréal, Montreal, Québec, Canada
| | - Éric Racine
- Pragmatic Research Unit, Institute de
Recherche Clinique de Montréal (IRCM), Montreal, Québec, Canada
- Department of Medicine and Department
of Social and Preventative Medicine, Université de Montréal, Montréal, Québec, Canada
| | - Alexander G. Weil
- Division of Neurosurgery, Department
of Surgery, Centre Hospitalier Universitaire Sainte-Justine (CHUSJ), Département de
Pédiatrie, Université de Montréal, Montreal, Québec, Canada
- Department of Neuroscience, Université de Montréal, Montréal, Québec, Canada
- Brain and Development Research Axis,
CHU Sainte-Justine Research Center, Montréal, Québec, Canada
| |
Collapse
|
7
|
Wilson GH, Willett FR, Stein EA, Kamdar F, Avansino DT, Hochberg LR, Shenoy KV, Druckmann S, Henderson JM. Long-term unsupervised recalibration of cursor BCIs. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.02.03.527022. [PMID: 36778458 PMCID: PMC9915729 DOI: 10.1101/2023.02.03.527022] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Abstract
Intracortical brain-computer interfaces (iBCIs) require frequent recalibration to maintain robust performance due to changes in neural activity that accumulate over time. Compensating for this nonstationarity would enable consistently high performance without the need for supervised recalibration periods, where users cannot engage in free use of their device. Here we introduce a hidden Markov model (HMM) to infer what targets users are moving toward during iBCI use. We then retrain the system using these inferred targets, enabling unsupervised adaptation to changing neural activity. Our approach outperforms the state of the art in large-scale, closed-loop simulations over two months and in closed-loop with a human iBCI user over one month. Leveraging an offline dataset spanning five years of iBCI recordings, we further show how recently proposed data distribution-matching approaches to recalibration fail over long time scales; only target-inference methods appear capable of enabling long-term unsupervised recalibration. Our results demonstrate how task structure can be used to bootstrap a noisy decoder into a highly-performant one, thereby overcoming one of the major barriers to clinically translating BCIs.
Collapse
|
8
|
Willsey MS, Nason-Tomaszewski SR, Ensel SR, Temmar H, Mender MJ, Costello JT, Patil PG, Chestek CA. Real-time brain-machine interface in non-human primates achieves high-velocity prosthetic finger movements using a shallow feedforward neural network decoder. Nat Commun 2022; 13:6899. [PMID: 36371498 PMCID: PMC9653378 DOI: 10.1038/s41467-022-34452-w] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2022] [Accepted: 10/25/2022] [Indexed: 11/13/2022] Open
Abstract
Despite the rapid progress and interest in brain-machine interfaces that restore motor function, the performance of prosthetic fingers and limbs has yet to mimic native function. The algorithm that converts brain signals to a control signal for the prosthetic device is one of the limitations in achieving rapid and realistic finger movements. To achieve more realistic finger movements, we developed a shallow feed-forward neural network to decode real-time two-degree-of-freedom finger movements in two adult male rhesus macaques. Using a two-step training method, a recalibrated feedback intention-trained (ReFIT) neural network is introduced to further improve performance. In 7 days of testing across two animals, neural network decoders, with higher-velocity and more natural appearing finger movements, achieved a 36% increase in throughput over the ReFIT Kalman filter, which represents the current standard. The neural network decoders introduced herein demonstrate real-time decoding of continuous movements at a level superior to the current state-of-the-art and could provide a starting point to using neural networks for the development of more naturalistic brain-controlled prostheses.
Collapse
Affiliation(s)
- Matthew S. Willsey
- grid.214458.e0000000086837370Department of Neurosurgery, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA
| | - Samuel R. Nason-Tomaszewski
- grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA
| | - Scott R. Ensel
- grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA
| | - Hisham Temmar
- grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA
| | - Matthew J. Mender
- grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA
| | - Joseph T. Costello
- grid.214458.e0000000086837370Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI USA
| | - Parag G. Patil
- grid.214458.e0000000086837370Department of Neurosurgery, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Neuroscience Graduate Program, University of Michigan Medical School, Ann Arbor, MI USA ,grid.214458.e0000000086837370Department of Anesthesiology, University of Michigan, Ann Arbor, MI USA
| | - Cynthia A. Chestek
- grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Neuroscience Graduate Program, University of Michigan Medical School, Ann Arbor, MI USA ,grid.214458.e0000000086837370Robotics Graduate Program, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Biointerfaces Institute, University of Michigan, Ann Arbor, MI USA
| |
Collapse
|
9
|
Shin H, Suma D, He B. Closed-loop motor imagery EEG simulation for brain-computer interfaces. Front Hum Neurosci 2022; 16:951591. [PMID: 36061506 PMCID: PMC9428352 DOI: 10.3389/fnhum.2022.951591] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Accepted: 07/20/2022] [Indexed: 11/13/2022] Open
Abstract
In a brain-computer interface (BCI) system, the testing of decoding algorithms, tasks, and their parameters is critical for optimizing performance. However, conducting human experiments can be costly and time-consuming, especially when investigating broad sets of parameters. Attempts to utilize previously collected data in offline analysis lack a co-adaptive feedback loop between the system and the user present online, limiting the applicability of the conclusions obtained to real-world uses of BCI. As such, a number of studies have attempted to address this cost-wise middle ground between offline and live experimentation with real-time neural activity simulators. We present one such system which generates motor imagery electroencephalography (EEG) via forward modeling and novel motor intention encoding models for conducting sensorimotor rhythm (SMR)-based continuous cursor control experiments in a closed-loop setting. We use the proposed simulator with 10 healthy human subjects to test the effect of three decoder and task parameters across 10 different values. Our simulated approach produces similar statistical conclusions to those produced during parallel, paired, online experimentation, but in 55% of the time. Notably, both online and simulated experimentation expressed a positive effect of cursor velocity limit on performance regardless of subject average performance, supporting the idea of relaxing constraints on cursor gain in online continuous cursor control. We demonstrate the merits of our closed-loop motor imagery EEG simulation, and provide an open-source framework to the community for closed-loop SMR-based BCI studies in the future. All code including the simulator have been made available on GitHub.
Collapse
|
10
|
Pandarinath C, Bensmaia SJ. The science and engineering behind sensitized brain-controlled bionic hands. Physiol Rev 2022; 102:551-604. [PMID: 34541898 PMCID: PMC8742729 DOI: 10.1152/physrev.00034.2020] [Citation(s) in RCA: 22] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2020] [Revised: 09/07/2021] [Accepted: 09/13/2021] [Indexed: 12/13/2022] Open
Abstract
Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.
Collapse
Affiliation(s)
- Chethan Pandarinath
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, Georgia
- Department of Neurosurgery, Emory University, Atlanta, Georgia
| | - Sliman J Bensmaia
- Department of Organismal Biology and Anatomy, University of Chicago, Chicago, Illinois
- Committee on Computational Neuroscience, University of Chicago, Chicago, Illinois
- Grossman Institute for Neuroscience, Quantitative Biology, and Human Behavior, University of Chicago, Chicago, Illinois
| |
Collapse
|
11
|
Ravishankar S, Toneva M, Wehbe L. Single-Trial MEG Data Can Be Denoised Through Cross-Subject Predictive Modeling. Front Comput Neurosci 2021; 15:737324. [PMID: 34858157 PMCID: PMC8632362 DOI: 10.3389/fncom.2021.737324] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2021] [Accepted: 08/23/2021] [Indexed: 11/13/2022] Open
Abstract
A pervasive challenge in brain imaging is the presence of noise that hinders investigation of underlying neural processes, with Magnetoencephalography (MEG) in particular having very low Signal-to-Noise Ratio (SNR). The established strategy to increase MEG's SNR involves averaging multiple repetitions of data corresponding to the same stimulus. However, repetition of stimulus can be undesirable, because underlying neural activity has been shown to change across trials, and repeating stimuli limits the breadth of the stimulus space experienced by subjects. In particular, the rising popularity of naturalistic studies with a single viewing of a movie or story necessitates the discovery of new approaches to increase SNR. We introduce a simple framework to reduce noise in single-trial MEG data by leveraging correlations in neural responses across subjects as they experience the same stimulus. We demonstrate its use in a naturalistic reading comprehension task with 8 subjects, with MEG data collected while they read the same story a single time. We find that our procedure results in data with reduced noise and allows for better discovery of neural phenomena. As proof-of-concept, we show that the N400m's correlation with word surprisal, an established finding in literature, is far more clearly observed in the denoised data than the original data. The denoised data also shows higher decoding and encoding accuracy than the original data, indicating that the neural signals associated with reading are either preserved or enhanced after the denoising procedure.
Collapse
Affiliation(s)
| | - Mariya Toneva
- Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, United States
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, United States
| | - Leila Wehbe
- Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, United States
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, United States
| |
Collapse
|
12
|
Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity. Commun Biol 2021; 4:1055. [PMID: 34556793 PMCID: PMC8460739 DOI: 10.1038/s42003-021-02578-0] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2021] [Accepted: 08/11/2021] [Indexed: 11/17/2022] Open
Abstract
Speech neuroprosthetics aim to provide a natural communication channel to individuals who are unable to speak due to physical or neurological impairments. Real-time synthesis of acoustic speech directly from measured neural activity could enable natural conversations and notably improve quality of life, particularly for individuals who have severely limited means of communication. Recent advances in decoding approaches have led to high quality reconstructions of acoustic speech from invasively measured neural activity. However, most prior research utilizes data collected during open-loop experiments of articulated speech, which might not directly translate to imagined speech processes. Here, we present an approach that synthesizes audible speech in real-time for both imagined and whispered speech conditions. Using a participant implanted with stereotactic depth electrodes, we were able to reliably generate audible speech in real-time. The decoding models rely predominately on frontal activity suggesting that speech processes have similar representations when vocalized, whispered, or imagined. While reconstructed audio is not yet intelligible, our real-time synthesis approach represents an essential step towards investigating how patients will learn to operate a closed-loop speech neuroprosthesis based on imagined speech. Miguel Angrick et al. develop an intracranial EEG-based method to decode imagined speech from a human patient and translate it into audible speech in real-time. This report presents an important proof of concept that acoustic output can be reconstructed on the basis of neural signals, and serves as a valuable step in the development of neuroprostheses to help nonverbal patients interact with their environment.
Collapse
|
13
|
Restoring upper extremity function with brain-machine interfaces. INTERNATIONAL REVIEW OF NEUROBIOLOGY 2021; 159:153-186. [PMID: 34446245 DOI: 10.1016/bs.irn.2021.06.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
Abstract
One of the most exciting advances to emerge in neural interface technologies has been the development of real-time brain-machine interface (BMI) neuroprosthetic devices to restore upper extremity function. BMI neuroprostheses, made possible by synergistic advances in neural recording technologies, high-speed computation and signal processing, and neuroscience, have permitted the restoration of volitional movement to patients suffering the loss of upper-extremity function. In this chapter, we review the scientific and technological advances underlying these remarkable devices. After presenting an introduction to the current state of the field, we provide an accessible technical discussion of the two fundamental requirements of a successful neuroprosthesis: signal extraction from the brain and signal decoding that results in robust prosthetic control. We close with a presentation of emerging technologies that are likely to substantially advance the field.
Collapse
|
14
|
Olsen S, Zhang J, Liang KF, Lam M, Riaz U, Kao JC. An artificial intelligence that increases simulated brain-computer interface performance. J Neural Eng 2021; 18. [PMID: 33978599 DOI: 10.1088/1741-2552/abfaaa] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2020] [Accepted: 04/22/2021] [Indexed: 12/14/2022]
Abstract
Objective.Brain-computer interfaces (BCIs) translate neural activity into control signals for assistive devices in order to help people with motor disabilities communicate effectively. In this work, we introduce a new BCI architecture that improves control of a BCI computer cursor to type on a virtual keyboard.Approach.Our BCI architecture incorporates an external artificial intelligence (AI) that beneficially augments the movement trajectories of the BCI. This AI-BCI leverages past user actions, at both long (100 s of seconds ago) and short (100 s of milliseconds ago) timescales, to modify the BCI's trajectories.Main results.We tested our AI-BCI in a closed-loop BCI simulator with nine human subjects performing a typing task. We demonstrate that our AI-BCI achieves: (1) categorically higher information communication rates, (2) quicker ballistic movements between targets, (3) improved precision control to 'dial in' on targets, and (4) more efficient movement trajectories. We further show that our AI-BCI increases performance across a wide control quality spectrum from poor to proficient control.Significance.This AI-BCI architecture, by increasing BCI performance across all key metrics evaluated, may increase the clinical viability of BCI systems.
Collapse
Affiliation(s)
- Sebastian Olsen
- Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90024, United States of America
| | - Jianwei Zhang
- Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90024, United States of America
| | - Ken-Fu Liang
- Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90024, United States of America
| | - Michelle Lam
- Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90024, United States of America
| | - Usama Riaz
- Department of Computer Science, University of California, Los Angeles, CA 90024, United States of America
| | - Jonathan C Kao
- Department of Electrical and Computer Engineering, University of California, Los Angeles, CA 90024, United States of America.,Neurosciences Program, University of California, Los Angeles, CA 90024, United States of America
| |
Collapse
|
15
|
Kawala-Sterniuk A, Browarska N, Al-Bakri A, Pelc M, Zygarlicki J, Sidikova M, Martinek R, Gorzelanczyk EJ. Summary of over Fifty Years with Brain-Computer Interfaces-A Review. Brain Sci 2021; 11:43. [PMID: 33401571 PMCID: PMC7824107 DOI: 10.3390/brainsci11010043] [Citation(s) in RCA: 47] [Impact Index Per Article: 15.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2020] [Revised: 12/25/2020] [Accepted: 12/27/2020] [Indexed: 11/16/2022] Open
Abstract
Over the last few decades, the Brain-Computer Interfaces have been gradually making their way to the epicenter of scientific interest. Many scientists from all around the world have contributed to the state of the art in this scientific domain by developing numerous tools and methods for brain signal acquisition and processing. Such a spectacular progress would not be achievable without accompanying technological development to equip the researchers with the proper devices providing what is absolutely necessary for any kind of discovery as the core of every analysis: the data reflecting the brain activity. The common effort has resulted in pushing the whole domain to the point where the communication between a human being and the external world through BCI interfaces is no longer science fiction but nowadays reality. In this work we present the most relevant aspects of the BCIs and all the milestones that have been made over nearly 50-year history of this research domain. We mention people who were pioneers in this area as well as we highlight all the technological and methodological advances that have transformed something available and understandable by a very few into something that has a potential to be a breathtaking change for so many. Aiming to fully understand how the human brain works is a very ambitious goal and it will surely take time to succeed. However, even that fraction of what has already been determined is sufficient e.g., to allow impaired people to regain control on their lives and significantly improve its quality. The more is discovered in this domain, the more benefit for all of us this can potentially bring.
Collapse
Affiliation(s)
- Aleksandra Kawala-Sterniuk
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, 45-758 Opole, Poland; (N.B.); (M.P.); (J.Z.)
| | - Natalia Browarska
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, 45-758 Opole, Poland; (N.B.); (M.P.); (J.Z.)
| | - Amir Al-Bakri
- Department of Biomedical Engineering, College of Engineering, University of Babylon, 51001 Babylon, Iraq;
| | - Mariusz Pelc
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, 45-758 Opole, Poland; (N.B.); (M.P.); (J.Z.)
- Department of Computing and Information Systems, University of Greenwich, London SE10 9LS, UK
| | - Jaroslaw Zygarlicki
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, 45-758 Opole, Poland; (N.B.); (M.P.); (J.Z.)
| | - Michaela Sidikova
- Department of Cybernetics and Biomedical Engineering, VSB-Technical University Ostrava—FEECS, 708 00 Ostrava-Poruba, Czech Republic; (M.S.); (R.M.)
| | - Radek Martinek
- Department of Cybernetics and Biomedical Engineering, VSB-Technical University Ostrava—FEECS, 708 00 Ostrava-Poruba, Czech Republic; (M.S.); (R.M.)
| | - Edward Jacek Gorzelanczyk
- Department of Theoretical Basis of BioMedical Sciences and Medical Informatics, Nicolaus Copernicus University, Collegium Medicum, 85-067 Bydgoszcz, Poland;
- Institute of Philosophy, Kazimierz Wielki University, 85-092 Bydgoszcz, Poland
- Babinski Specialist Psychiatric Healthcare Center, Outpatient Addiction Treatment, 91-229 Lodz, Poland
- The Society for the Substitution Treatment of Addiction “Medically Assisted Recovery”, 85-791 Bydgoszcz, Poland
| |
Collapse
|
16
|
Zhang P, Li W, Ma X, He J, Huang J, Li Q. Feature-Selection-Based Transfer Learning for Intracortical Brain-Machine Interface Decoding. IEEE Trans Neural Syst Rehabil Eng 2020; 29:60-73. [PMID: 33108289 DOI: 10.1109/tnsre.2020.3034234] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
The time spent in collecting current samples for decoder calibration and the computational burden brought by high-dimensional neural recordings remain two challenging problems in intracortical brain-machine interfaces (iBMIs). Decoder calibration optimization approaches have been proposed, and neuron selection methods have been used to reduce computational burden. However, few methods can solve both problems simultaneously. In this article, we present a symmetrical-uncertainty-based transfer learning (SUTL) method that combines transfer learning with feature selection. The proposed method uses symmetrical uncertainty to quantitatively measure three indices for feature selection: stationarity, importance and redundancy of the feature. By selecting the stationary features, the disparities between the historical data and current data can be diminished, and the historical data can be effectively used for decoder calibration, thereby reducing the demand for current data. After selecting the important and non-redundant features, only the channels corresponding to them need to work; thus, the computational burden is reduced. The proposed method was tested on neural data recorded from two rhesus macaques to decode the reaching position or grasping gesture. The results showed that the SUTL method diminished the disparities between the historical data and current data, while achieving superior decoding performance with the needs of only ten current samples each category, less than 10% the number of features and 30% the number of neural recording channels. Additionally, unlike most studies on iBMIs, feature selection was implemented instead of neuron selection, and the average decoding accuracy achieved by the former was 6.6% higher.
Collapse
|
17
|
|
18
|
Burkhart MC, Brandman DM, Franco B, Hochberg LR, Harrison MT. The Discriminative Kalman Filter for Bayesian Filtering with Nonlinear and Nongaussian Observation Models. Neural Comput 2020; 32:969-1017. [PMID: 32187000 PMCID: PMC8259355 DOI: 10.1162/neco_a_01275] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
The Kalman filter provides a simple and efficient algorithm to compute the posterior distribution for state-space models where both the latent state and measurement models are linear and gaussian. Extensions to the Kalman filter, including the extended and unscented Kalman filters, incorporate linearizations for models where the observation model p ( observation | state ) is nonlinear. We argue that in many cases, a model for p ( state | observation ) proves both easier to learn and more accurate for latent state estimation. Approximating p ( state | observation ) as gaussian leads to a new filtering algorithm, the discriminative Kalman filter (DKF), which can perform well even when p ( observation | state ) is highly nonlinear and/or nongaussian. The approximation, motivated by the Bernstein-von Mises theorem, improves as the dimensionality of the observations increases. The DKF has computational complexity similar to the Kalman filter, allowing it in some cases to perform much faster than particle filters with similar precision, while better accounting for nonlinear and nongaussian observation models than Kalman-based extensions. When the observation model must be learned from training data prior to filtering, off-the-shelf nonlinear and nonparametric regression techniques can provide a gaussian model for p ( observation | state ) that cleanly integrates with the DKF. As part of the BrainGate2 clinical trial, we successfully implemented gaussian process regression with the DKF framework in a brain-computer interface to provide real-time, closed-loop cursor control to a person with a complete spinal cord injury. In this letter, we explore the theory underlying the DKF, exhibit some illustrative examples, and outline potential extensions.
Collapse
Affiliation(s)
- Michael C Burkhart
- Division of Applied Mathematics, Brown University, Providence, RI 02912, U.S.A.
| | - David M Brandman
- Department of Neuroscience, Brown University, Providence, RI 02912, U.S.A., and Department of Surgery (Neurosurgery), Dalhousie University, Halifax, NS, B3H 4R2, Canada
| | - Brian Franco
- Center for Neurotechnology and Neurorecovery, Neurology, Massachusetts General Hospital, Boston, MA 02114, U.S.A.
| | - Leigh R Hochberg
- Center for Neurotechnology and Neurorecovery, Neurology, Massachusetts General Hospital, Boston, MA 02114, U.S.A.; School of Engineering and Carney Institute for Brain Science, Brown University, Providence, RI 02912, U.S.A.; Neurology, Harvard Medical School, Boston, MA 02115, U.S.A.; and VA RR&D Center for Neurorestoration and Neurotechnology, Providence Veterans Affairs Medical Center, Providence, RI 02908, U.S.A.
| | - Matthew T Harrison
- Division of Applied Mathematics, Brown University, Providence, RI 02912, U.S.A.
| |
Collapse
|
19
|
Yang SH, Wang HL, Lo YC, Lai HY, Chen KY, Lan YH, Kao CC, Chou C, Lin SH, Huang JW, Wang CF, Kuo CH, Chen YY. Inhibition of Long-Term Variability in Decoding Forelimb Trajectory Using Evolutionary Neural Networks With Error-Correction Learning. Front Comput Neurosci 2020; 14:22. [PMID: 32296323 PMCID: PMC7136463 DOI: 10.3389/fncom.2020.00022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2019] [Accepted: 03/04/2020] [Indexed: 11/13/2022] Open
Abstract
Objective: In brain machine interfaces (BMIs), the functional mapping between neural activities and kinematic parameters varied over time owing to changes in neural recording conditions. The variability in neural recording conditions might result in unstable long-term decoding performance. Relevant studies trained decoders with several days of training data to make them inherently robust to changes in neural recording conditions. However, these decoders might not be robust to changes in neural recording conditions when only a few days of training data are available. In time-series prediction and feedback control system, an error feedback was commonly adopted to reduce the effects of model uncertainty. This motivated us to introduce an error feedback to a neural decoder for dealing with the variability in neural recording conditions. Approach: We proposed an evolutionary constructive and pruning neural network with error feedback (ECPNN-EF) as a neural decoder. The ECPNN-EF with partially connected topology decoded the instantaneous firing rates of each sorted unit into forelimb movement of a rat. Furthermore, an error feedback was adopted as an additional input to provide kinematic information and thus compensate for changes in functional mapping. The proposed neural decoder was trained on data collected from a water reward-related lever-pressing task for a rat. The first 2 days of data were used to train the decoder, and the subsequent 10 days of data were used to test the decoder. Main Results: The ECPNN-EF under different settings was evaluated to better understand the impact of the error feedback and partially connected topology. The experimental results demonstrated that the ECPNN-EF achieved significantly higher daily decoding performance with smaller daily variability when using the error feedback and partially connected topology. Significance: These results suggested that the ECPNN-EF with partially connected topology could cope with both within- and across-day changes in neural recording conditions. The error feedback in the ECPNN-EF compensated for decreases in decoding performance when neural recording conditions changed. This mechanism made the ECPNN-EF robust against changes in functional mappings and thus improved the long-term decoding stability when only a few days of training data were available.
Collapse
Affiliation(s)
- Shih-Hung Yang
- Department of Mechanical Engineering, National Cheng Kung University, Tainan, Taiwan
| | - Han-Lin Wang
- Department of Biomedical Engineering, National Yang Ming University, Taipei, Taiwan
| | - Yu-Chun Lo
- The Ph.D. Program for Neural Regenerative Medicine, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan
| | - Hsin-Yi Lai
- Key Laboratory of Medical Neurobiology of Zhejiang Province, Department of Neurology of the Second Affiliated Hospital, Interdisciplinary Institute of Neuroscience and Technology, Zhejiang University School of Medicine, Hangzhou, China
- Key Laboratory of Biomedical Engineering of Ministry of Education, College of Biomedical Engineering and Instrument Science, Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou, China
| | - Kuan-Yu Chen
- Department of Biomedical Engineering, National Yang Ming University, Taipei, Taiwan
| | - Yu-Hao Lan
- Department of Biomedical Engineering, National Yang Ming University, Taipei, Taiwan
| | - Ching-Chia Kao
- Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan
| | - Chin Chou
- Department of Regulatory & Quality Sciences, University of Southern California, Los Angeles, CA, United States
| | - Sheng-Huang Lin
- Buddhist Tzu Chi Medical Foundation, Department of Neurology, Hualien Tzu Chi Hospital, Hualien, Taiwan
- Department of Neurology, School of Medicine, Tzu Chi University, Hualien, Taiwan
| | - Jyun-We Huang
- Department of Mechanical Engineering, National Cheng Kung University, Tainan, Taiwan
| | - Ching-Fu Wang
- Department of Biomedical Engineering, National Yang Ming University, Taipei, Taiwan
| | - Chao-Hung Kuo
- Department of Biomedical Engineering, National Yang Ming University, Taipei, Taiwan
- Department of Neurosurgery, Neurological Institute, Taipei Veterans General Hospital, Taipei, Taiwan
- Department of Neurological Surgery, University of Washington, Seattle, WA, United States
| | - You-Yin Chen
- Department of Biomedical Engineering, National Yang Ming University, Taipei, Taiwan
- The Ph.D. Program for Neural Regenerative Medicine, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan
| |
Collapse
|