1
|
Wairagkar M, Card NS, Singer-Clark T, Hou X, Iacobacci C, Hochberg LR, Brandman DM, Stavisky SD. An instantaneous voice synthesis neuroprosthesis. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.08.14.607690. [PMID: 39229047 PMCID: PMC11370360 DOI: 10.1101/2024.08.14.607690] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 09/05/2024]
Abstract
Brain computer interfaces (BCIs) have the potential to restore communication to people who have lost the ability to speak due to neurological disease or injury. BCIs have been used to translate the neural correlates of attempted speech into text1-3. However, text communication fails to capture the nuances of human speech such as prosody, intonation and immediately hearing one's own voice. Here, we demonstrate a "brain-to-voice" neuroprosthesis that instantaneously synthesizes voice with closed-loop audio feedback by decoding neural activity from 256 microelectrodes implanted into the ventral precentral gyrus of a man with amyotrophic lateral sclerosis and severe dysarthria. We overcame the challenge of lacking ground-truth speech for training the neural decoder and were able to accurately synthesize his voice. Along with phonemic content, we were also able to decode paralinguistic features from intracortical activity, enabling the participant to modulate his BCI-synthesized voice in real-time to change intonation, emphasize words, and sing short melodies. These results demonstrate the feasibility of enabling people with paralysis to speak intelligibly and expressively through a BCI.
Collapse
Affiliation(s)
- Maitreyee Wairagkar
- Department of Neurological Surgery, University of California Davis, Davis, CA
| | - Nicholas S Card
- Department of Neurological Surgery, University of California Davis, Davis, CA
| | - Tyler Singer-Clark
- Department of Neurological Surgery, University of California Davis, Davis, CA
- Department of Biomedical Engineering, University of California Davis, Davis, CA
| | - Xianda Hou
- Department of Neurological Surgery, University of California Davis, Davis, CA
- Department of Computer Science, University of California Davis, Davis, CA
| | - Carrina Iacobacci
- Department of Neurological Surgery, University of California Davis, Davis, CA
| | - Leigh R Hochberg
- School of Engineering and Carney Institute for Brain Sciences, Brown University, Providence, RI
- VA RR&D Center for Neurorestoration and Neurotechnology, VA Providence Healthcare, Providence, RI
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA
| | - David M Brandman
- Department of Neurological Surgery, University of California Davis, Davis, CA
| | - Sergey D Stavisky
- Department of Neurological Surgery, University of California Davis, Davis, CA
| |
Collapse
|
2
|
Wang F, Chen X, Roelfsema PR. Comparison of electrical microstimulation artifact removal methods for high-channel-count prostheses. J Neurosci Methods 2024; 408:110169. [PMID: 38782123 DOI: 10.1016/j.jneumeth.2024.110169] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Revised: 04/15/2024] [Accepted: 05/17/2024] [Indexed: 05/25/2024]
Abstract
BACKGROUND Neuroprostheses are used to electrically stimulate the brain, modulate neural activity and restore sensory and motor function following injury or disease, such as blindness, paralysis, and other movement and psychiatric disorders. Recordings are often made simultaneously with stimulation, allowing the monitoring of neural signals and closed-loop control of devices. However, stimulation-evoked artifacts may obscure neural activity, particularly when stimulation and recording sites are nearby. Several methods have been developed to remove stimulation artifacts, but it remains challenging to validate and compare these methods because the 'ground-truth' of the neuronal signals may be contaminated by artifacts. NEW METHOD Here, we delivered stimulation to the visual cortex via a high-channel-count prosthesis while recording neuronal activity and stimulation artifacts. We quantified the waveforms and temporal properties of stimulation artifacts from the cortical visual prosthesis (CVP) and used them to build a dataset, in which we simulated the neuronal activity and the stimulation artifacts. We illustrate how to use the simulated data to evaluate the performance of six software-based artifact removal methods (Template subtraction, Linear interpolation, Polynomial fitting, Exponential fitting, SALPA and ERAASR) in a CVP application scenario. RESULTS We here focused on stimulation artifacts caused by electrical stimulation through a high-channel-count cortical prosthesis device. We find that the Polynomial fitting and Exponential fitting methods outperform the other methods in recovering spikes and multi-unit activity. Linear interpolation and Template subtraction recovered the local-field potentials. CONCLUSION Polynomial fitting and Exponential fitting provided a good trade-off between the quality of the recovery of spikes and multi-unit activity (MUA) and the computational complexity for a cortical prosthesis.
Collapse
Affiliation(s)
- Feng Wang
- Department of Vision & Cognition, Netherlands Institute for Neuroscience (KNAW), Amsterdam 1105 BA, the Netherlands.
| | - Xing Chen
- Department of Ophthalmology, University of Pittsburgh School of Medicine, 203 Lothrop St, Pittsburgh, PA 15213, US.
| | - Pieter R Roelfsema
- Department of Vision & Cognition, Netherlands Institute for Neuroscience (KNAW), Amsterdam 1105 BA, the Netherlands; Department of Ophthalmology, University of Pittsburgh School of Medicine, 203 Lothrop St, Pittsburgh, PA 15213, US; Department of Integrative Neurophysiology, VU University, De Boelelaan 1085, Amsterdam 1081 HV, the Netherlands; Department of Neurosurgery, Academic Medical Centre, Postbus 22660, Amsterdam 1100 DD, the Netherlands; Laboratory of Visual Brain Therapy, Sorbonne Université, Institut National de la Santé et de la Recherche Médicale, Centre National de la Recherche Scientifique, Institut de la Vision, Paris F-75012, France.
| |
Collapse
|
3
|
Shah NP, Willsey MS, Hahn N, Kamdar F, Avansino DT, Fan C, Hochberg LR, Willett FR, Henderson JM. A flexible intracortical brain-computer interface for typing using finger movements. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.04.22.590630. [PMID: 38712189 PMCID: PMC11071346 DOI: 10.1101/2024.04.22.590630] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2024]
Abstract
Keyboard typing with finger movements is a versatile digital interface for users with diverse skills, needs, and preferences. Currently, such an interface does not exist for people with paralysis. We developed an intracortical brain-computer interface (BCI) for typing with attempted flexion/extension movements of three finger groups on the right hand, or both hands, and demonstrated its flexibility in two dominant typing paradigms. The first paradigm is "point-and-click" typing, where a BCI user selects one key at a time using continuous real-time control, allowing selection of arbitrary sequences of symbols. During cued character selection with this paradigm, a human research participant with paralysis achieved 30-40 selections per minute with nearly 90% accuracy. The second paradigm is "keystroke" typing, where the BCI user selects each character by a discrete movement without real-time feedback, often giving a faster speed for natural language sentences. With 90 cued characters per minute, decoding attempted finger movements and correcting errors using a language model resulted in more than 90% accuracy. Notably, both paradigms matched the state-of-the-art for BCI performance and enabled further flexibility by the simultaneous selection of multiple characters as well as efficient decoder estimation across paradigms. Overall, the high-performance interface is a step towards the wider accessibility of BCI technology by addressing unmet user needs for flexibility.
Collapse
Affiliation(s)
| | - Matthew S Willsey
- Department of Neurosurgery, Stanford University
- Department of Neurosurgery, The University of Texas at Austin, Austin, TX, USA + This work was primarily conducted at Stanford University
| | - Nick Hahn
- Department of Neurosurgery, Stanford University
| | | | - Donald T Avansino
- Department of Neurosurgery, Stanford University
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Chaofei Fan
- Department of Neurosurgery, Stanford University
| | - Leigh R Hochberg
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, USA
- School of Engineering, Brown University, Providence, RI, USA
- Robert J. and Nancy D. Carney Institute for Brain Science, Brown University, Providence, RI, USA
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Francis R Willett
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Bio-X Institute, Stanford University, Stanford, CA, USA
| |
Collapse
|
4
|
Card NS, Wairagkar M, Iacobacci C, Hou X, Singer-Clark T, Willett FR, Kunz EM, Fan C, Nia MV, Deo DR, Srinivasan A, Choi EY, Glasser MF, Hochberg LR, Henderson JM, Shahlaie K, Brandman DM, Stavisky SD. An accurate and rapidly calibrating speech neuroprosthesis. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2023.12.26.23300110. [PMID: 38645254 PMCID: PMC11030484 DOI: 10.1101/2023.12.26.23300110] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/23/2024]
Abstract
Brain-computer interfaces can enable rapid, intuitive communication for people with paralysis by transforming the cortical activity associated with attempted speech into text on a computer screen. Despite recent advances, communication with brain-computer interfaces has been restricted by extensive training data requirements and inaccurate word output. A man in his 40's with ALS with tetraparesis and severe dysarthria (ALSFRS-R = 23) was enrolled into the BrainGate2 clinical trial. He underwent surgical implantation of four microelectrode arrays into his left precentral gyrus, which recorded neural activity from 256 intracortical electrodes. We report a speech neuroprosthesis that decoded his neural activity as he attempted to speak in both prompted and unstructured conversational settings. Decoded words were displayed on a screen, then vocalized using text-to-speech software designed to sound like his pre-ALS voice. On the first day of system use, following 30 minutes of attempted speech training data, the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. On the second day, the size of the possible output vocabulary increased to 125,000 words, and, after 1.4 additional hours of training data, the neuroprosthesis achieved 90.2% accuracy. With further training data, the neuroprosthesis sustained 97.5% accuracy beyond eight months after surgical implantation. The participant has used the neuroprosthesis to communicate in self-paced conversations for over 248 hours. In an individual with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore naturalistic communication after a brief training period.
Collapse
Affiliation(s)
- Nicholas S Card
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| | - Maitreyee Wairagkar
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| | - Carrina Iacobacci
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| | - Xianda Hou
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
- Departments of Computer Science, University of California Davis, Davis, CA, USA
| | - Tyler Singer-Clark
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
- Departments of Biomedical Engineering, University of California Davis, Davis, CA, USA
| | - Francis R Willett
- Departments of Neurosurgery, Stanford University, Stanford, CA, USA
- Departments of Electrical Engineering, Stanford University, Stanford, CA, USA
- Howard Hughes Medical Institute, Stanford University, Stanford, CA, USA
| | - Erin M Kunz
- Departments of Electrical Engineering, Stanford University, Stanford, CA, USA
- Departments of Mechanical Engineering, Stanford University, Stanford, CA, USA
| | - Chaofei Fan
- Departments of Computer Science, Stanford University, Stanford, CA, USA
| | - Maryam Vahdati Nia
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
- Departments of Computer Science, University of California Davis, Davis, CA, USA
| | - Darrel R Deo
- Departments of Neurosurgery, Stanford University, Stanford, CA, USA
| | - Aparna Srinivasan
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
- Departments of Biomedical Engineering, University of California Davis, Davis, CA, USA
| | - Eun Young Choi
- Departments of Neurosurgery, Stanford University, Stanford, CA, USA
| | - Matthew F Glasser
- Departments of Radiology and Neuroscience, Washington University School of Medicine, Saint Louis, MO, USA
| | - Leigh R Hochberg
- School of Engineering and Carney Institute for Brain Sciences, Brown University, Providence, RI, USA
- VA RR&D Center for Neurorestoration and Neurotechnology, VA Providence Healthcare, Providence, RI
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA
| | - Jaimie M Henderson
- Departments of Neurosurgery, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
| | - Kiarash Shahlaie
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| | - David M Brandman
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| | - Sergey D Stavisky
- Departments of Neurological Surgery, University of California Davis, Davis, CA, USA
| |
Collapse
|
5
|
Willsey MS, Shah NP, Avansino DT, Hahn NV, Jamiolkowski RM, Kamdar FB, Hochberg LR, Willett FR, Henderson JM. A real-time, high-performance brain-computer interface for finger decoding and quadcopter control. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.06.578107. [PMID: 38370697 PMCID: PMC10871262 DOI: 10.1101/2024.02.06.578107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
People with paralysis express unmet needs for peer support, leisure activities, and sporting activities. Many within the general population rely on social media and massively multiplayer video games to address these needs. We developed a high-performance finger brain-computer-interface system allowing continuous control of 3 independent finger groups with 2D thumb movements. The system was tested in a human research participant over sequential trials requiring fingers to reach and hold on targets, with an average acquisition rate of 76 targets/minute and completion time of 1.58 ± 0.06 seconds. Performance compared favorably to previous animal studies, despite a 2-fold increase in the decoded degrees-of-freedom (DOF). Finger positions were then used for 4-DOF velocity control of a virtual quadcopter, demonstrating functionality over both fixed and random obstacle courses. This approach shows promise for controlling multiple-DOF end-effectors, such as robotic fingers or digital interfaces for work, entertainment, and socialization.
Collapse
|
6
|
Pellot-Cestero JE, Herring EZ, Graczyk EL, Memberg WD, Kirsch RF, Ajiboye AB, Miller JP. Implanted Electrodes for Functional Electrical Stimulation to Restore Upper and Lower Extremity Function: History and Future Directions. Neurosurgery 2023; 93:965-970. [PMID: 37288972 DOI: 10.1227/neu.0000000000002561] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Accepted: 04/03/2023] [Indexed: 06/09/2023] Open
Abstract
Functional electrical stimulation (FES) to activate nerves and muscles in paralyzed extremities has considerable promise to improve outcome after neurological disease or injury, especially in individuals who have upper motor nerve dysfunction due to central nervous system pathology. Because technology has improved, a wide variety of methods for providing electrical stimulation to create functional movements have been developed, including muscle stimulating electrodes, nerve stimulating electrodes, and hybrid constructs. However, in spite of decades of success in experimental settings with clear functional improvements for individuals with paralysis, the technology has not yet reached widespread clinical translation. In this review, we outline the history of FES techniques and approaches and describe future directions in evolution of the technology.
Collapse
Affiliation(s)
- Joel E Pellot-Cestero
- Department of Neurosurgery, School of Medicine, Case Western Reserve University, Cleveland , Ohio , USA
- Department of Neurosurgery, The Neurological Institute, University Hospital Cleveland Medical Center, Cleveland , Ohio , USA
| | - Eric Z Herring
- Department of Neurosurgery, School of Medicine, Case Western Reserve University, Cleveland , Ohio , USA
- Department of Neurosurgery, The Neurological Institute, University Hospital Cleveland Medical Center, Cleveland , Ohio , USA
| | - Emily L Graczyk
- Department of Neurosurgery, School of Medicine, Case Western Reserve University, Cleveland , Ohio , USA
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland , Ohio , USA
- Louis Stokes Cleveland Department of Veterans Affairs Medical Center, FES Center of Excellence, Rehab. R&D Service, Cleveland , Ohio , USA
| | - William D Memberg
- Department of Neurosurgery, School of Medicine, Case Western Reserve University, Cleveland , Ohio , USA
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland , Ohio , USA
- Louis Stokes Cleveland Department of Veterans Affairs Medical Center, FES Center of Excellence, Rehab. R&D Service, Cleveland , Ohio , USA
| | - Robert F Kirsch
- Department of Neurosurgery, School of Medicine, Case Western Reserve University, Cleveland , Ohio , USA
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland , Ohio , USA
- Louis Stokes Cleveland Department of Veterans Affairs Medical Center, FES Center of Excellence, Rehab. R&D Service, Cleveland , Ohio , USA
| | - A Bolu Ajiboye
- Department of Neurosurgery, School of Medicine, Case Western Reserve University, Cleveland , Ohio , USA
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland , Ohio , USA
- Louis Stokes Cleveland Department of Veterans Affairs Medical Center, FES Center of Excellence, Rehab. R&D Service, Cleveland , Ohio , USA
| | - Jonathan P Miller
- Department of Neurosurgery, School of Medicine, Case Western Reserve University, Cleveland , Ohio , USA
- Department of Neurosurgery, The Neurological Institute, University Hospital Cleveland Medical Center, Cleveland , Ohio , USA
- Louis Stokes Cleveland Department of Veterans Affairs Medical Center, FES Center of Excellence, Rehab. R&D Service, Cleveland , Ohio , USA
| |
Collapse
|
7
|
Willett FR, Kunz E, Fan C, Avansino D, Wilson G, Choi EY, Kamdar F, Hochberg LRH, Druckmann S, Shenoy K, Henderson J. A high-performance speech neuroprosthesis. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.01.21.524489. [PMID: 36711591 PMCID: PMC9882398 DOI: 10.1101/2023.01.21.524489] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/23/2023]
Abstract
Speech brain-computer interfaces (BCIs) have the potential to restore rapid communication to people with paralysis by decoding neural activity evoked by attempted speaking movements into text or sound. Early demonstrations, while promising, have not yet achieved accuracies high enough for communication of unconstrainted sentences from a large vocabulary. Here, we demonstrate the first speech-to-text BCI that records spiking activity from intracortical microelectrode arrays. Enabled by these high-resolution recordings, our study participant, who can no longer speak intelligibly due amyotrophic lateral sclerosis (ALS), achieved a 9.1% word error rate on a 50 word vocabulary (2.7 times fewer errors than the prior state of the art speech BCI2) and a 23.8% word error rate on a 125,000 word vocabulary (the first successful demonstration of large-vocabulary decoding). Our BCI decoded speech at 62 words per minute, which is 3.4 times faster than the prior record for any kind of BCI and begins to approach the speed of natural conversation (160 words per minute). Finally, we highlight two aspects of the neural code for speech that are encouraging for speech BCIs: spatially intermixed tuning to speech articulators that makes accurate decoding possible from only a small region of cortex, and a detailed articulatory representation of phonemes that persists years after paralysis. These results show a feasible path forward for using intracortical speech BCIs to restore rapid communication to people with paralysis who can no longer speak.
Collapse
|
8
|
Sohn WJ, Lim J, Wang PT, Pu H, Malekzadeh-Arasteh O, Shaw SJ, Armacost M, Gong H, Kellis S, Andersen RA, Liu CY, Heydari P, Nenadic Z, Do AH. Benchtop and bedside validation of a low-cost programmable cortical stimulator in a testbed for bi-directional brain-computer-interface research. Front Neurosci 2023; 16:1075971. [PMID: 36711153 PMCID: PMC9878125 DOI: 10.3389/fnins.2022.1075971] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2022] [Accepted: 12/16/2022] [Indexed: 01/15/2023] Open
Abstract
Introduction Bi-directional brain-computer interfaces (BD-BCI) to restore movement and sensation must achieve concurrent operation of recording and decoding of motor commands from the brain and stimulating the brain with somatosensory feedback. Methods A custom programmable direct cortical stimulator (DCS) capable of eliciting artificial sensorimotor response was integrated into an embedded BCI system to form a safe, independent, wireless, and battery powered testbed to explore BD-BCI concepts at a low cost. The BD-BCI stimulator output was tested in phantom brain tissue by assessing its ability to deliver electrical stimulation equivalent to an FDA-approved commercial electrical cortical stimulator. Subsequently, the stimulator was tested in an epilepsy patient with subcortical electrocorticographic (ECoG) implants covering the sensorimotor cortex to assess its ability to elicit equivalent responses as the FDA-approved counterpart. Additional safety features (impedance monitoring, artifact mitigation, and passive and active charge balancing mechanisms) were also implemeneted and tested in phantom brain tissue. Finally, concurrent operation with interleaved stimulation and BCI decoding was tested in a phantom brain as a proof-of-concept operation of BD-BCI system. Results The benchtop prototype BD-BCI stimulator's basic output features (current amplitude, pulse frequency, pulse width, train duration) were validated by demonstrating the output-equivalency to an FDA-approved commercial cortical electrical stimulator (R 2 > 0.99). Charge-neutral stimulation was demonstrated with pulse-width modulation-based correction algorithm preventing steady state voltage deviation. Artifact mitigation achieved a 64.5% peak voltage reduction. Highly accurate impedance monitoring was achieved with R 2 > 0.99 between measured and actual impedance, which in-turn enabled accurate charge density monitoring. An online BCI decoding accuracy of 93.2% between instructional cues and decoded states was achieved while delivering interleaved stimulation. The brain stimulation mapping via ECoG grids in an epilepsy patient showed that the two stimulators elicit equivalent responses. Significance This study demonstrates clinical validation of a fully-programmable electrical stimulator, integrated into an embedded BCI system. This low-cost BD-BCI system is safe and readily applicable as a testbed for BD-BCI research. In particular, it provides an all-inclusive hardware platform that approximates the limitations in a near-future implantable BD-BCI. This successful benchtop/human validation of the programmable electrical stimulator in a BD-BCI system is a critical milestone toward fully-implantable BD-BCI systems.
Collapse
Affiliation(s)
- Won Joon Sohn
- Department of Neurology, University of California, Irvine, Irvine, CA, United States
| | - Jeffrey Lim
- Department of Biomedical Engineering, University of California, Irvine, Irvine, CA, United States
| | - Po T. Wang
- Department of Biomedical Engineering, University of California, Irvine, Irvine, CA, United States
| | - Haoran Pu
- Department of Electrical Engineering and Computer Science, University of California, Irvine, Irvine, CA, United States
| | - Omid Malekzadeh-Arasteh
- Department of Electrical Engineering and Computer Science, University of California, Irvine, Irvine, CA, United States
| | - Susan J. Shaw
- Department of Neurology, Rancho Los Amigos National Rehabilitation Center, Downey, CA, United States
- Department of Neurology, University of Southern California, Los Angeles, CA, United States
| | - Michelle Armacost
- Department of Neurology, Rancho Los Amigos National Rehabilitation Center, Downey, CA, United States
- Department of Neurology, University of Southern California, Los Angeles, CA, United States
| | - Hui Gong
- Department of Neurology, Rancho Los Amigos National Rehabilitation Center, Downey, CA, United States
- Department of Neurology, University of Southern California, Los Angeles, CA, United States
| | - Spencer Kellis
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, United States
| | - Richard A. Andersen
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, United States
| | - Charles Y. Liu
- Department of Neurosurgery, Rancho Los Amigos National Rehabilitation Center, Downey, CA, United States
- Department of Neurological Surgery, University of Southern California, Los Angeles, CA, United States
| | - Payam Heydari
- Department of Electrical Engineering and Computer Science, University of California, Irvine, Irvine, CA, United States
| | - Zoran Nenadic
- Department of Biomedical Engineering, University of California, Irvine, Irvine, CA, United States
- Department of Electrical Engineering and Computer Science, University of California, Irvine, Irvine, CA, United States
| | - An H. Do
- Department of Neurology, University of California, Irvine, Irvine, CA, United States
| |
Collapse
|
9
|
Insausti-Delgado A, López-Larraz E, Nishimura Y, Ziemann U, Ramos-Murguialday A. Non-invasive brain-spine interface: Continuous control of trans-spinal magnetic stimulation using EEG. Front Bioeng Biotechnol 2022; 10:975037. [PMID: 36394044 PMCID: PMC9659618 DOI: 10.3389/fbioe.2022.975037] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2022] [Accepted: 09/23/2022] [Indexed: 08/22/2023] Open
Abstract
Brain-controlled neuromodulation has emerged as a promising tool to promote functional recovery in patients with motor disorders. Brain-machine interfaces exploit this neuromodulatory strategy and could be used for restoring voluntary control of lower limbs. In this work, we propose a non-invasive brain-spine interface (BSI) that processes electroencephalographic (EEG) activity to volitionally control trans-spinal magnetic stimulation (ts-MS), as an approach for lower-limb neurorehabilitation. This novel platform allows to contingently connect motor cortical activation during leg motor imagery with the activation of leg muscles via ts-MS. We tested this closed-loop system in 10 healthy participants using different stimulation conditions. This BSI efficiently removed stimulation artifacts from EEG regardless of ts-MS intensity used, allowing continuous monitoring of cortical activity and real-time closed-loop control of ts-MS. Our BSI induced afferent and efferent evoked responses, being this activation ts-MS intensity-dependent. We demonstrated the feasibility, safety and usability of this non-invasive BSI. The presented system represents a novel non-invasive means of brain-controlled neuromodulation and opens the door towards its integration as a therapeutic tool for lower-limb rehabilitation.
Collapse
Affiliation(s)
- Ainhoa Insausti-Delgado
- Institute of Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany
- International Max Planck Research School (IMPRS) for Cognitive and Systems Neuroscience, Tübingen, Germany
- IKERBASQUE, Basque Foundation for Science, Bilbao, Spain
- TECNALIA, Basque Research and Technology Alliance (BRTA), Donostia-San Sebastián, Spain
| | - Eduardo López-Larraz
- Institute of Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany
- Bitbrain, Zaragoza, Spain
| | - Yukio Nishimura
- Neural Prosthetics Project, Department of Brain and Neuroscience, Tokyo Metropolitan Institute of Medical Science, Tokyo, Japan
| | - Ulf Ziemann
- Department of Neurology and Stroke, University of Tübingen, Tübingen, Germany
- Hertie Institute for Clinical Brain Research, University of Tübingen, Tübingen, Germany
| | - Ander Ramos-Murguialday
- Institute of Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany
- TECNALIA, Basque Research and Technology Alliance (BRTA), Donostia-San Sebastián, Spain
| |
Collapse
|
10
|
Sombeck JT, Heye J, Kumaravelu K, Goetz SM, Peterchev AV, Grill WM, Bensmaia S, Miller LE. Characterizing the short-latency evoked response to intracortical microstimulation across a multi-electrode array. J Neural Eng 2022; 19:10.1088/1741-2552/ac63e8. [PMID: 35378515 PMCID: PMC9142773 DOI: 10.1088/1741-2552/ac63e8] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2021] [Accepted: 04/04/2022] [Indexed: 11/12/2022]
Abstract
Objective.Persons with tetraplegia can use brain-machine interfaces to make visually guided reaches with robotic arms. Without somatosensory feedback, these movements will likely be slow and imprecise, like those of persons who retain movement but have lost proprioception. Intracortical microstimulation (ICMS) has promise for providing artificial somatosensory feedback. ICMS that mimics naturally occurring neural activity, may allow afferent interfaces that are more informative and easier to learn than stimulation evoking unnaturalistic activity. To develop such biomimetic stimulation patterns, it is important to characterize the responses of neurons to ICMS.Approach.Using a Utah multi-electrode array, we recorded activity evoked by both single pulses and trains of ICMS at a wide range of amplitudes and frequencies in two rhesus macaques. As the electrical artifact caused by ICMS typically prevents recording for many milliseconds, we deployed a custom rapid-recovery amplifier with nonlinear gain to limit signal saturation on the stimulated electrode. Across all electrodes after stimulation, we removed the remaining slow return to baseline with acausal high-pass filtering of time-reversed recordings.Main results.After single pulses of stimulation, we recorded what was likely transsynaptically-evoked activity even on the stimulated electrode as early as ∼0.7 ms. This was immediately followed by suppressed neural activity lasting 10-150 ms. After trains, this long-lasting inhibition was replaced by increased firing rates for ∼100 ms. During long trains, the evoked response on the stimulated electrode decayed rapidly while the response was maintained on non-stimulated channels.Significance.The detailed description of the spatial and temporal response to ICMS can be used to better interpret results from experiments that probe circuit connectivity or function of cortical areas. These results can also contribute to the design of stimulation patterns to improve afferent interfaces for artificial sensory feedback.
Collapse
Affiliation(s)
- Joseph T Sombeck
- Department of Biomedical Engineering, Northwestern University, Evanston, IL, United States of America
| | - Juliet Heye
- Department of Neuroscience, Northwestern University, Chicago, IL, United States of America
| | - Karthik Kumaravelu
- Department of Biomedical Engineering, Duke University, Durham, NC, United States of America
| | - Stefan M Goetz
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, United States of America
- Department of Neurosurgery, Duke University, Durham, NC, United States of America
- Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, United States of America
- Duke Institute for Brain Sciences, Duke University, Durham, NC, United States of America
- Department of Engineering, University of Cambridge, Cambridge, United Kingdom
| | - Angel V Peterchev
- Department of Biomedical Engineering, Duke University, Durham, NC, United States of America
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, United States of America
- Department of Neurosurgery, Duke University, Durham, NC, United States of America
- Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, United States of America
- Duke Institute for Brain Sciences, Duke University, Durham, NC, United States of America
| | - Warren M Grill
- Department of Biomedical Engineering, Duke University, Durham, NC, United States of America
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, United States of America
- Department of Neurobiology, Duke University, Durham, NC, United States of America
- Department of Neurosurgery, Duke University, Durham, NC, United States of America
- Duke Institute for Brain Sciences, Duke University, Durham, NC, United States of America
| | - Sliman Bensmaia
- Department of Organismal Biology and Anatomy, University of Chicago, Chicago, IL, United States of America
- Committee on Computational Neuroscience, University of Chicago, Chicago, IL, United States of America
- Neuroscience Institute, University of Chicago, Chicago, IL, United States of America
| | - Lee E Miller
- Department of Biomedical Engineering, Northwestern University, Evanston, IL, United States of America
- Department of Neuroscience, Northwestern University, Chicago, IL, United States of America
- Department of Physical Medicine and Rehabilitation, Northwestern University, Chicago, IL, United States of America
- Shirley Ryan AbilityLab, Chicago, IL, United States of America
| |
Collapse
|
11
|
Pandarinath C, Bensmaia SJ. The science and engineering behind sensitized brain-controlled bionic hands. Physiol Rev 2022; 102:551-604. [PMID: 34541898 PMCID: PMC8742729 DOI: 10.1152/physrev.00034.2020] [Citation(s) in RCA: 22] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2020] [Revised: 09/07/2021] [Accepted: 09/13/2021] [Indexed: 12/13/2022] Open
Abstract
Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.
Collapse
Affiliation(s)
- Chethan Pandarinath
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, Georgia
- Department of Neurosurgery, Emory University, Atlanta, Georgia
| | - Sliman J Bensmaia
- Department of Organismal Biology and Anatomy, University of Chicago, Chicago, Illinois
- Committee on Computational Neuroscience, University of Chicago, Chicago, Illinois
- Grossman Institute for Neuroscience, Quantitative Biology, and Human Behavior, University of Chicago, Chicago, Illinois
| |
Collapse
|
12
|
Schelles M, Wouters J, Asamoah B, Mc Laughlin M, Bertrand A. Objective evaluation of stimulation artefact removal techniques in the context of neural spike sorting. J Neural Eng 2022; 19. [DOI: 10.1088/1741-2552/ac4ecf] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2021] [Accepted: 01/25/2022] [Indexed: 11/12/2022]
Abstract
Abstract
Objective - We present a framework to objectively test and compare stimulation artefact removal techniques in the context of neural spike sorting. Approach - To this end, we used realistic hybrid ground-truth spiking data, with superimposed artefacts from in vivo recordings. We used the framework to evaluate and compare several techniques: blanking, template subtraction by averaging, linear regression, and a multi-channel Wiener filter (MWF). Main results - Our study demonstrates that blanking and template subtraction result in a poorer spike sorting performance than linear regression and MWF, while the latter two perform similarly. Finally, to validate the conclusions found from the hybrid evaluation framework, we also performed a qualitative analysis on in vivo recordings without artificial manipulations. Significance - Our framework allows direct quantification of the impact of the residual artefact on the spike sorting accuracy, thereby allowing for a more objective and more relevant comparison compared to indirect signal quality metrics that are estimated from the signal statistics. Furthermore, the availability of a ground truth in the form of single-unit spiking activity also facilitates a better estimation of such signal quality metrics.
Collapse
|
13
|
Restoring upper extremity function with brain-machine interfaces. INTERNATIONAL REVIEW OF NEUROBIOLOGY 2021; 159:153-186. [PMID: 34446245 DOI: 10.1016/bs.irn.2021.06.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
Abstract
One of the most exciting advances to emerge in neural interface technologies has been the development of real-time brain-machine interface (BMI) neuroprosthetic devices to restore upper extremity function. BMI neuroprostheses, made possible by synergistic advances in neural recording technologies, high-speed computation and signal processing, and neuroscience, have permitted the restoration of volitional movement to patients suffering the loss of upper-extremity function. In this chapter, we review the scientific and technological advances underlying these remarkable devices. After presenting an introduction to the current state of the field, we provide an accessible technical discussion of the two fundamental requirements of a successful neuroprosthesis: signal extraction from the brain and signal decoding that results in robust prosthetic control. We close with a presentation of emerging technologies that are likely to substantially advance the field.
Collapse
|
14
|
EEG-FES-Force-MMG closed-loop control systems of a volunteer with paraplegia considering motor imagery with fatigue recognition and automatic shut-off. Biomed Signal Process Control 2021. [DOI: 10.1016/j.bspc.2021.102662] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022]
|
15
|
Insausti-Delgado A, López-Larraz E, Omedes J, Ramos-Murguialday A. Intensity and Dose of Neuromuscular Electrical Stimulation Influence Sensorimotor Cortical Excitability. Front Neurosci 2021; 14:593360. [PMID: 33519355 PMCID: PMC7845652 DOI: 10.3389/fnins.2020.593360] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2020] [Accepted: 11/30/2020] [Indexed: 12/13/2022] Open
Abstract
Neuromuscular electrical stimulation (NMES) of the nervous system has been extensively used in neurorehabilitation due to its capacity to engage the muscle fibers, improving muscle tone, and the neural pathways, sending afferent volleys toward the brain. Although different neuroimaging tools suggested the capability of NMES to regulate the excitability of sensorimotor cortex and corticospinal circuits, how the intensity and dose of NMES can neuromodulate the brain oscillatory activity measured with electroencephalography (EEG) is still unknown to date. We quantified the effect of NMES parameters on brain oscillatory activity of 12 healthy participants who underwent stimulation of wrist extensors during rest. Three different NMES intensities were included, two below and one above the individual motor threshold, fixing the stimulation frequency to 35 Hz and the pulse width to 300 μs. Firstly, we efficiently removed stimulation artifacts from the EEG recordings. Secondly, we analyzed the effect of amplitude and dose on the sensorimotor oscillatory activity. On the one hand, we observed a significant NMES intensity-dependent modulation of brain activity, demonstrating the direct effect of afferent receptor recruitment. On the other hand, we described a significant NMES intensity-dependent dose-effect on sensorimotor activity modulation over time, with below-motor-threshold intensities causing cortical inhibition and above-motor-threshold intensities causing cortical facilitation. Our results highlight the relevance of intensity and dose of NMES, and show that these parameters can influence the recruitment of the sensorimotor pathways from the muscle to the brain, which should be carefully considered for the design of novel neuromodulation interventions based on NMES.
Collapse
Affiliation(s)
- Ainhoa Insausti-Delgado
- Institute of Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany
- International Max Planck Research School (IMPRS) for Cognitive and Systems Neuroscience, Tübingen, Germany
- IKERBASQUE, Basque Foundation for Science, Bilbao, Spain
| | - Eduardo López-Larraz
- Institute of Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany
- Bitbrain, Zaragoza, Spain
| | - Jason Omedes
- Instituto de Investigación en Ingeniería de Aragón (I3A), Zaragoza, Spain
- Departamento de Informática e Ingeniería de Sistemas (DIIS), University of Zaragoza, Zaragoza, Spain
| | - Ander Ramos-Murguialday
- Institute of Medical Psychology and Behavioral Neurobiology, University of Tübingen, Tübingen, Germany
- Neurotechnology Laboratory, TECNALIA, Basque Research and Technology Alliance (BRTA), Donostia-San Sebastián, Spain
| |
Collapse
|
16
|
Wilson GH, Stavisky SD, Willett FR, Avansino DT, Kelemen JN, Hochberg LR, Henderson JM, Druckmann S, Shenoy KV. Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus. J Neural Eng 2020; 17:066007. [PMID: 33236720 PMCID: PMC8293867 DOI: 10.1088/1741-2552/abbfef] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
Abstract
OBJECTIVE To evaluate the potential of intracortical electrode array signals for brain-computer interfaces (BCIs) to restore lost speech, we measured the performance of decoders trained to discriminate a comprehensive basis set of 39 English phonemes and to synthesize speech sounds via a neural pattern matching method. We decoded neural correlates of spoken-out-loud words in the 'hand knob' area of precentral gyrus, a step toward the eventual goal of decoding attempted speech from ventral speech areas in patients who are unable to speak. APPROACH Neural and audio data were recorded while two BrainGate2 pilot clinical trial participants, each with two chronically-implanted 96-electrode arrays, spoke 420 different words that broadly sampled English phonemes. Phoneme onsets were identified from audio recordings, and their identities were then classified from neural features consisting of each electrode's binned action potential counts or high-frequency local field potential power. Speech synthesis was performed using the 'Brain-to-Speech' pattern matching method. We also examined two potential confounds specific to decoding overt speech: acoustic contamination of neural signals and systematic differences in labeling different phonemes' onset times. MAIN RESULTS A linear decoder achieved up to 29.3% classification accuracy (chance = 6%) across 39 phonemes, while an RNN classifier achieved 33.9% accuracy. Parameter sweeps indicated that performance did not saturate when adding more electrodes or more training data, and that accuracy improved when utilizing time-varying structure in the data. Microphonic contamination and phoneme onset differences modestly increased decoding accuracy, but could be mitigated by acoustic artifact subtraction and using a neural speech onset marker, respectively. Speech synthesis achieved r = 0.523 correlation between true and reconstructed audio. SIGNIFICANCE The ability to decode speech using intracortical electrode array signals from a nontraditional speech area suggests that placing electrode arrays in ventral speech areas is a promising direction for speech BCIs.
Collapse
Affiliation(s)
- Guy H Wilson
- Neurosciences Graduate Program, Stanford University, Stanford, CA, United States of America
| | - Sergey D Stavisky
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
| | - Francis R Willett
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, United States of America
| | - Donald T Avansino
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
| | - Jessica N Kelemen
- Department of Neurology, Harvard Medical School, Boston, MA, United States of America
| | - Leigh R Hochberg
- Department of Neurology, Harvard Medical School, Boston, MA, United States of America
- Center for Neurotechnology and Neurorecovery, Dept. of Neurology, Massachusetts General Hospital, Boston, MA, United States of America
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, United States of America
- Carney Institute for Brain Science and School of Engineering, Brown University, Providence, RI, United States of America
| | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
| | - Shaul Druckmann
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
- Department of Neurobiology, Stanford University, Stanford, CA, United States of America
| | - Krishna V Shenoy
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, United States of America
- Department of Neurobiology, Stanford University, Stanford, CA, United States of America
- Department of Bioengineering, Stanford University, Stanford, CA, United States of America
| |
Collapse
|
17
|
Milosevic M, Marquez-Chin C, Masani K, Hirata M, Nomura T, Popovic MR, Nakazawa K. Why brain-controlled neuroprosthetics matter: mechanisms underlying electrical stimulation of muscles and nerves in rehabilitation. Biomed Eng Online 2020; 19:81. [PMID: 33148270 PMCID: PMC7641791 DOI: 10.1186/s12938-020-00824-w] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2020] [Accepted: 10/10/2020] [Indexed: 12/11/2022] Open
Abstract
Delivering short trains of electric pulses to the muscles and nerves can elicit action potentials resulting in muscle contractions. When the stimulations are sequenced to generate functional movements, such as grasping or walking, the application is referred to as functional electrical stimulation (FES). Implications of the motor and sensory recruitment of muscles using FES go beyond simple contraction of muscles. Evidence suggests that FES can induce short- and long-term neurophysiological changes in the central nervous system by varying the stimulation parameters and delivery methods. By taking advantage of this, FES has been used to restore voluntary movement in individuals with neurological injuries with a technique called FES therapy (FEST). However, long-lasting cortical re-organization (neuroplasticity) depends on the ability to synchronize the descending (voluntary) commands and the successful execution of the intended task using a FES. Brain-computer interface (BCI) technologies offer a way to synchronize cortical commands and movements generated by FES, which can be advantageous for inducing neuroplasticity. Therefore, the aim of this review paper is to discuss the neurophysiological mechanisms of electrical stimulation of muscles and nerves and how BCI-controlled FES can be used in rehabilitation to improve motor function.
Collapse
Affiliation(s)
- Matija Milosevic
- Graduate School of Engineering Science, Department of Mechanical Science and Bioengineering, Osaka University, 1-3 Machikaneyama-cho, Toyonaka, Osaka, 560-8531, Japan.
| | - Cesar Marquez-Chin
- Institute of Biomedical Engineering, University of Toronto, 164 College Street, Toronto, ON, M5S 3G9, Canada
- KITE Research Institute, Toronto Rehabilitation Institute - University Health Network, 520 Sutherland Drive, Toronto, ON, M4G 3V9, Canada
- CRANIA, University Health Network & University of Toronto, 550 University Avenue, Toronto, ON, M5G 2A2, Canada
| | - Kei Masani
- Institute of Biomedical Engineering, University of Toronto, 164 College Street, Toronto, ON, M5S 3G9, Canada
- KITE Research Institute, Toronto Rehabilitation Institute - University Health Network, 520 Sutherland Drive, Toronto, ON, M4G 3V9, Canada
- CRANIA, University Health Network & University of Toronto, 550 University Avenue, Toronto, ON, M5G 2A2, Canada
| | - Masayuki Hirata
- Department of Neurological Diagnosis and Restoration, Graduate School of Medicine, Osaka University, 2-2 Yamadaoka, Suita, Osaka, 565-0871, Japan
| | - Taishin Nomura
- Graduate School of Engineering Science, Department of Mechanical Science and Bioengineering, Osaka University, 1-3 Machikaneyama-cho, Toyonaka, Osaka, 560-8531, Japan
| | - Milos R Popovic
- Institute of Biomedical Engineering, University of Toronto, 164 College Street, Toronto, ON, M5S 3G9, Canada
- KITE Research Institute, Toronto Rehabilitation Institute - University Health Network, 520 Sutherland Drive, Toronto, ON, M4G 3V9, Canada
- CRANIA, University Health Network & University of Toronto, 550 University Avenue, Toronto, ON, M5G 2A2, Canada
| | - Kimitaka Nakazawa
- Department of Life Sciences, Graduate School of Arts and Sciences, University of Tokyo, 3-8-1 Komaba, Meguro, Tokyo, 153-8902, Japan
| |
Collapse
|
18
|
Dunlap CF, Colachis SC, Meyers EC, Bockbrader MA, Friedenberg DA. Classifying Intracortical Brain-Machine Interface Signal Disruptions Based on System Performance and Applicable Compensatory Strategies: A Review. Front Neurorobot 2020; 14:558987. [PMID: 33162885 PMCID: PMC7581895 DOI: 10.3389/fnbot.2020.558987] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2020] [Accepted: 09/09/2020] [Indexed: 12/18/2022] Open
Abstract
Brain-machine interfaces (BMIs) record and translate neural activity into a control signal for assistive or other devices. Intracortical microelectrode arrays (MEAs) enable high degree-of-freedom BMI control for complex tasks by providing fine-resolution neural recording. However, chronically implanted MEAs are subject to a dynamic in vivo environment where transient or systematic disruptions can interfere with neural recording and degrade BMI performance. Typically, neural implant failure modes have been categorized as biological, material, or mechanical. While this categorization provides insight into a disruption's causal etiology, it is less helpful for understanding degree of impact on BMI function or possible strategies for compensation. Therefore, we propose a complementary classification framework for intracortical recording disruptions that is based on duration of impact on BMI performance and requirement for and responsiveness to interventions: (1) Transient disruptions interfere with recordings on the time scale of minutes to hours and can resolve spontaneously; (2) Reversible disruptions cause persistent interference in recordings but the root cause can be remedied by an appropriate intervention; (3) Irreversible compensable disruptions cause persistent or progressive decline in signal quality, but their effects on BMI performance can be mitigated algorithmically; and (4) Irreversible non-compensable disruptions cause permanent signal loss that is not amenable to remediation or compensation. This conceptualization of intracortical BMI disruption types is useful for highlighting specific areas for potential hardware improvements and also identifying opportunities for algorithmic interventions. We review recording disruptions that have been reported for MEAs and demonstrate how biological, material, and mechanical mechanisms of disruption can be further categorized according to their impact on signal characteristics. Then we discuss potential compensatory protocols for each of the proposed disruption classes. Specifically, transient disruptions may be minimized by using robust neural decoder features, data augmentation methods, adaptive machine learning models, and specialized signal referencing techniques. Statistical Process Control methods can identify reparable disruptions for rapid intervention. In-vivo diagnostics such as impedance spectroscopy can inform neural feature selection and decoding models to compensate for irreversible disruptions. Additional compensatory strategies for irreversible disruptions include information salvage techniques, data augmentation during decoder training, and adaptive decoding methods to down-weight damaged channels.
Collapse
Affiliation(s)
- Collin F. Dunlap
- Department of Biomedical Engineering, The Ohio State University, Columbus, OH, United States
- Medical Devices and Neuromodulation, Battelle Memorial Institute, Columbus, OH, United States
| | - Samuel C. Colachis
- Medical Devices and Neuromodulation, Battelle Memorial Institute, Columbus, OH, United States
| | - Eric C. Meyers
- Medical Devices and Neuromodulation, Battelle Memorial Institute, Columbus, OH, United States
| | - Marcia A. Bockbrader
- Department of Physical Medicine and Rehabilitation, The Ohio State University, Columbus, OH, United States
| | - David A. Friedenberg
- Advanced Analytics and Health Research, Battelle Memorial Institute, Columbus, OH, United States
| |
Collapse
|
19
|
Intra-cortical brain-machine interfaces for controlling upper-limb powered muscle and robotic systems in spinal cord injury. Clin Neurol Neurosurg 2020; 196:106069. [DOI: 10.1016/j.clineuro.2020.106069] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2020] [Revised: 07/03/2020] [Accepted: 07/04/2020] [Indexed: 12/20/2022]
|
20
|
Drebitz E, Rausch LP, Kreiter AK. A novel approach for removing micro-stimulation artifacts and reconstruction of broad-band neuronal signals. J Neurosci Methods 2019; 332:108549. [PMID: 31837345 DOI: 10.1016/j.jneumeth.2019.108549] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2019] [Revised: 12/03/2019] [Accepted: 12/10/2019] [Indexed: 01/14/2023]
Abstract
BACKGROUND Electrical stimulation is a widely used method in the neurosciences with a variety of application fields. However, stimulation frequently induces large and long-lasting artifacts, which superimpose on the actual neuronal signal. Existing methods were developed for analyzing fast events such as spikes, but are not well suited for the restoration of LFP signals. NEW METHOD We developed a method that extracts artifact components while also leaving the LFP components of the neuronal signal intact. We based it on an exponential fit of the average artifact shape, which is subsequently adapted to the individual artifacts amplitude and then subtracted. Importantly, we used for fitting of the individual artifact only a short initial time window, in which the artifact is dominating the superimposition with the neuronal signal. Using this short period ensures that LFP components are not part of the fit, which leaves them unaffected by the subsequent artifact removal. RESULTS By using the method presented here, we could diminish the substantial distortions of neuronal signals caused by electrical stimulation to levels that were statistically indistinguishable from the original data. Furthermore, the effect of stimulation on the phases of γ- and β- oscillations was reduced by 85 and 75 %, respectively. COMPARISON WITH EXISTING METHODS This approach avoids signal loss as caused by methods cutting out artifacts and minimizes the distortion of the signal's temporal structure as compared to other approaches. CONCLUSION The method presented here allows for a successful reconstruction of broad-band signals.
Collapse
Affiliation(s)
- Eric Drebitz
- Brain Research Institute, Center for Cognitive Sciences, University of Bremen, Germany.
| | - Lukas-Paul Rausch
- Brain Research Institute, Center for Cognitive Sciences, University of Bremen, Germany
| | - Andreas K Kreiter
- Brain Research Institute, Center for Cognitive Sciences, University of Bremen, Germany
| |
Collapse
|
21
|
Bullard AJ, Hutchison BC, Lee J, Chestek CA, Patil PG. Estimating Risk for Future Intracranial, Fully Implanted, Modular Neuroprosthetic Systems: A Systematic Review of Hardware Complications in Clinical Deep Brain Stimulation and Experimental Human Intracortical Arrays. Neuromodulation 2019; 23:411-426. [DOI: 10.1111/ner.13069] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2019] [Revised: 08/05/2019] [Accepted: 09/10/2019] [Indexed: 01/08/2023]
Affiliation(s)
- Autumn J. Bullard
- Department of Biomedical Engineering University of Michigan Ann Arbor MI USA
| | | | - Jiseon Lee
- Department of Biomedical Engineering University of Michigan Ann Arbor MI USA
| | - Cynthia A. Chestek
- Department of Biomedical Engineering University of Michigan Ann Arbor MI USA
- Department of Electrical Engineering and Computer Science University of Michigan Ann Arbor MI USA
| | - Parag G. Patil
- Department of Biomedical Engineering University of Michigan Ann Arbor MI USA
- Department of Neurosurgery University of Michigan Medical School Ann Arbor MI USA
| |
Collapse
|
22
|
Young D, Willett F, Memberg WD, Murphy B, Rezaii P, Walter B, Sweet J, Miller J, Shenoy KV, Hochberg LR, Kirsch RF, Ajiboye AB. Closed-loop cortical control of virtual reach and posture using Cartesian and joint velocity commands. J Neural Eng 2018; 16:026011. [PMID: 30523839 DOI: 10.1088/1741-2552/aaf606] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023]
Abstract
OBJECTIVE Brain-computer interfaces (BCIs) are a promising technology for the restoration of function to people with paralysis, especially for controlling coordinated reaching. Typical BCI studies decode Cartesian endpoint velocities as commands, but human arm movements might be better controlled in a joint-based coordinate frame, which may match underlying movement encoding in the motor cortex. A better understanding of BCI controlled reaching by people with paralysis may lead to performance improvements in brain-controlled assistive devices. APPROACH Two intracortical BCI participants in the BrainGate2 pilot clinical trial performed a visual 3D endpoint virtual reality reaching task using two decoders: Cartesian and joint velocity. Task performance metrics (i.e. success rate and path efficiency) and single feature and population tuning were compared across the two decoder conditions. The participants also demonstrated the first BCI control of a fourth dimension of reaching, the arm's swivel angle, in a 4D posture matching task. MAIN RESULTS Both users achieved significantly higher success rates using Cartesian velocity control, and joint controlled trajectories were more variable and significantly more curved. Neural tuning analyses showed that most single feature activity was best described by a Cartesian kinematic encoding model, and population analyses revealed only slight differences in aggregate activity between the decoder conditions. Simulations of a BCI user reproduced trajectory features seen during closed-loop joint control when assuming only Cartesian-tuned features passed through a joint decoder. With minimal training, both participants controlled the virtual arm's swivel angle to complete a 4D posture matching task, and achieved significantly higher success using a Cartesian + swivel velocity decoder compared to a joint velocity decoder. SIGNIFICANCE These results suggest that Cartesian velocity command interfaces may provide better BCI control of arm movements than other kinematic variables, even in 4D posture tasks with swivel angle targets.
Collapse
Affiliation(s)
- D Young
- Case Western Reserve University, Cleveland, OH, United States of America. Department of VA Medical Center, FES Center of Excellence, Rehabilitation R&D Service, Louis Stokes Cleveland, Cleveland, OH, United States of America
| | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
23
|
Weiss JM, Flesher SN, Franklin R, Collinger JL, Gaunt RA. Artifact-free recordings in human bidirectional brain–computer interfaces. J Neural Eng 2018; 16:016002. [DOI: 10.1088/1741-2552/aae748] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
|