251
|
Hochberg LR, Cash SS. Freedom of Speech. N Engl J Med 2021; 385:278-279. [PMID: 34260841 DOI: 10.1056/nejme2106392] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Affiliation(s)
- Leigh R Hochberg
- From the Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, and Harvard Medical School, Boston (L.R.H., S.S.C.); and the School of Engineering and Carney Institute for Brain Science, Brown University, and the Department of Veterans Affairs Rehabilitation Research and Development Service Center for Neurorestoration and Neurotechnology - both in Providence, RI (L.R.H.)
| | - Sydney S Cash
- From the Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, and Harvard Medical School, Boston (L.R.H., S.S.C.); and the School of Engineering and Carney Institute for Brain Science, Brown University, and the Department of Veterans Affairs Rehabilitation Research and Development Service Center for Neurorestoration and Neurotechnology - both in Providence, RI (L.R.H.)
| |
Collapse
|
252
|
Haslacher D, Nasr K, Soekadar SR. Advancing sensory neuroprosthetics using artificial brain networks. PATTERNS (NEW YORK, N.Y.) 2021; 2:100304. [PMID: 34286308 PMCID: PMC8276008 DOI: 10.1016/j.patter.2021.100304] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 12/01/2022]
Abstract
Implementation of effective brain or neural stimulation protocols for restoration of complex sensory perception, e.g., in the visual domain, is an unresolved challenge. By leveraging the capacity of deep learning to model the brain's visual system, optic nerve stimulation patterns could be derived that are predictive of neural responses of higher-level cortical visual areas in silico. This novel approach could be generalized to optimize different types of neuroprosthetics or bidirectional brain-computer interfaces (BCIs).
Collapse
Affiliation(s)
- David Haslacher
- Clinical Neurotechnology Laboratory, Neurowissenschaftliches Forschungszentrum (NWFZ), Department of Psychiatry and Psychotherapy (CCM), Charité – Universitätsmedizin Berlin, Charitéplatz 1, 10117 Berlin, Germany
| | - Khaled Nasr
- Clinical Neurotechnology Laboratory, Neurowissenschaftliches Forschungszentrum (NWFZ), Department of Psychiatry and Psychotherapy (CCM), Charité – Universitätsmedizin Berlin, Charitéplatz 1, 10117 Berlin, Germany
| | - Surjo R. Soekadar
- Clinical Neurotechnology Laboratory, Neurowissenschaftliches Forschungszentrum (NWFZ), Department of Psychiatry and Psychotherapy (CCM), Charité – Universitätsmedizin Berlin, Charitéplatz 1, 10117 Berlin, Germany
| |
Collapse
|
253
|
尧 德. [Brain-computer interface: from lab to real scene]. SHENG WU YI XUE GONG CHENG XUE ZA ZHI = JOURNAL OF BIOMEDICAL ENGINEERING = SHENGWU YIXUE GONGCHENGXUE ZAZHI 2021; 38:405-408. [PMID: 34180184 PMCID: PMC9927763 DOI: 10.7507/1001-5515.202105091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Received: 05/28/2021] [Revised: 05/31/2021] [Indexed: 11/03/2022]
Abstract
Brain-computer interface (BCI) can be summarized as a system that uses online brain information to realize communication between brain and computer. BCI has experienced nearly half a century of development, although it now has a high degree of awareness in the public, but the application of BCI in the actual scene is still very limited. This collection invited some BCI teams in China to report their efforts to promote BCI from laboratory to real scene. This paper summarizes the main contents of the invited papers, and looks forward to the future of BCI.
Collapse
Affiliation(s)
- 德中 尧
- 电子科技大学 生命科学与技术学院(成都 610054)School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 610054, P.R.China
- 四川省脑科学与类脑智能研究院(成都 611731)Sichuan Institute for Brain Science and Brain-Inspired Intelligence, Chengdu 611731, P.R.China
- 中国医学科学院神经信息创新单元(2019RU035)(成都 611731)Research Unit of NeuroInformation, 2019RU035, Chinese Academy of Medical Sciences, Chengdu 611731, P.R.China
| |
Collapse
|
254
|
Liu X, Chen S, Shen X, Zhang X, Wang Y. A Nonlinear Maximum Correntropy Information Filter for High-Dimensional Neural Decoding. ENTROPY (BASEL, SWITZERLAND) 2021; 23:743. [PMID: 34204814 PMCID: PMC8231488 DOI: 10.3390/e23060743] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/30/2021] [Revised: 06/07/2021] [Accepted: 06/08/2021] [Indexed: 11/21/2022]
Abstract
Neural signal decoding is a critical technology in brain machine interface (BMI) to interpret movement intention from multi-neural activity collected from paralyzed patients. As a commonly-used decoding algorithm, the Kalman filter is often applied to derive the movement states from high-dimensional neural firing observation. However, its performance is limited and less effective for noisy nonlinear neural systems with high-dimensional measurements. In this paper, we propose a nonlinear maximum correntropy information filter, aiming at better state estimation in the filtering process for a noisy high-dimensional measurement system. We reconstruct the measurement model between the high-dimensional measurements and low-dimensional states using the neural network, and derive the state estimation using the correntropy criterion to cope with the non-Gaussian noise and eliminate large initial uncertainty. Moreover, analyses of convergence and robustness are given. The effectiveness of the proposed algorithm is evaluated by applying it on multiple segments of neural spiking data from two rats to interpret the movement states when the subjects perform a two-lever discrimination task. Our results demonstrate better and more robust state estimation performance when compared with other filters.
Collapse
Affiliation(s)
- Xi Liu
- Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, China; (X.L.); (X.S.); (X.Z.)
| | - Shuhang Chen
- Department of Chemical and Biological Engineering, Hong Kong University of Science and Technology, Hong Kong, China;
| | - Xiang Shen
- Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, China; (X.L.); (X.S.); (X.Z.)
| | - Xiang Zhang
- Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, China; (X.L.); (X.S.); (X.Z.)
| | - Yiwen Wang
- Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, China; (X.L.); (X.S.); (X.Z.)
- Department of Chemical and Biological Engineering, Hong Kong University of Science and Technology, Hong Kong, China;
| |
Collapse
|
255
|
Pei L, Ouyang G. Online recognition of handwritten characters from scalp-recorded brain activities during handwriting. J Neural Eng 2021; 18. [PMID: 34036941 DOI: 10.1088/1741-2552/ac01a0] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2021] [Accepted: 05/14/2021] [Indexed: 12/31/2022]
Abstract
Objective.Brain-computer interfaces aim to build an efficient communication with the world using neural signals, which may bring great benefits to human society, especially to people with physical impairments. To date, the ability to translate brain signals to effective communication outcome remains low. This work explores whether the handwriting process could serve as a potential interface with high performance. To this end, we first examined how much the scalp-recorded brain signals encode information related to handwriting and whether it is feasible to precisely retrieve the handwritten content solely from the scalp-recorded electrical data.Approach.Five participants were instructed to write the sentence 'HELLO, WORLD!' repeatedly on a tablet while their brain signals were simultaneously recorded by electroencephalography (EEG). The EEG signals were first decomposed by independent component analysis for extracting features to be used to train a convolutional neural network (CNN) to recognize the written symbols.Main results.The accuracy of the CNN-based classifier trained and applied on the same participant (training and test data separated) ranged from 76.8% to 97.0%. The accuracy of cross-participant application was more diverse, ranging from 14.7% to 58.7%. These results showed the possibility of recognizing the handwritten content directly from the scalp level brain signal. A demonstration of the recognition system in an online mode was presented. The major factor that grounded the recognition was the close association between the rich dynamics of electroencephalogram source activities and the kinematic information during the handwriting movements.Significance.This work revealed an explicit and precise mapping between scalp-level electrophysiological signals and linguistic information conveyed by handwriting, which provided a novel approach to developing brain computer interfaces that focus on semantic communication.
Collapse
Affiliation(s)
- Leisi Pei
- Faculty of Education, The University of Hong Kong, Pokfulam, Hong Kong SAR, People's Republic of China
| | - Guang Ouyang
- Faculty of Education, The University of Hong Kong, Pokfulam, Hong Kong SAR, People's Republic of China
| |
Collapse
|
256
|
Sahasrabuddhe K, Khan AA, Singh AP, Stern TM, Ng Y, Tadić A, Orel P, LaReau C, Pouzzner D, Nishimura K, Boergens KM, Shivakumar S, Hopper MS, Kerr B, Hanna MES, Edgington RJ, McNamara I, Fell D, Gao P, Babaie-Fishani A, Veijalainen S, Klekachev AV, Stuckey AM, Luyssaert B, Kozai TDY, Xie C, Gilja V, Dierickx B, Kong Y, Straka M, Sohal HS, Angle MR. The Argo: a high channel count recording system for neural recording in vivo. J Neural Eng 2021; 18:015002. [PMID: 33624614 PMCID: PMC8607496 DOI: 10.1088/1741-2552/abd0ce] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Decoding neural activity has been limited by the lack of tools available to record from large numbers of neurons across multiple cortical regions simultaneously with high temporal fidelity. To this end, we developed the Argo system to record cortical neural activity at high data rates. APPROACH Here we demonstrate a massively parallel neural recording system based on platinum-iridium microwire electrode arrays bonded to a CMOS voltage amplifier array. The Argo system is the highest channel count in vivo neural recording system, supporting simultaneous recording from 65 536 channels, sampled at 32 kHz and 12-bit resolution. This system was designed for cortical recordings, compatible with both penetrating and surface microelectrodes. MAIN RESULTS We validated this system through initial bench testing to determine specific gain and noise characteristics of bonded microwires, followed by in-vivo experiments in both rat and sheep cortex. We recorded spiking activity from 791 neurons in rats and surface local field potential activity from over 30 000 channels in sheep. SIGNIFICANCE These are the largest channel count microwire-based recordings in both rat and sheep. While currently adapted for head-fixed recording, the microwire-CMOS architecture is well suited for clinical translation. Thus, this demonstration helps pave the way for a future high data rate intracortical implant.
Collapse
Affiliation(s)
| | - Aamir A Khan
- Paradromics, Inc, Austin, TX, United States of America
| | | | - Tyler M Stern
- Paradromics, Inc, Austin, TX, United States of America
| | - Yeena Ng
- Paradromics, Inc, Austin, TX, United States of America
| | | | - Peter Orel
- Paradromics, Inc, Austin, TX, United States of America
| | - Chris LaReau
- Paradromics, Inc, Austin, TX, United States of America
| | | | | | | | | | | | - Bryan Kerr
- Paradromics, Inc, Austin, TX, United States of America
| | | | | | | | - Devin Fell
- Paradromics, Inc, Austin, TX, United States of America
| | - Peng Gao
- Caeleste CVBA, Mechelen, Belgium
| | | | | | | | | | | | - Takashi D Y Kozai
- Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, United States of America
- Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, United States of America
- McGowan Institute of Regenerative Medicine, University of Pittsburgh, Pittsburgh, PA, United States of America
- Center for Neuroscience, University of Pittsburgh, Pittsburgh, PA, United States of America
- NeuroTech Center, University of Pittsburgh Brain Institute, Pittsburgh, PA, United States of America
| | - Chong Xie
- Department of Electrical and Computer Engineering, Rice University, Houston, TX, United States of America
- Department of Bioengineering, Rice University, Houston, TX, United States of America
- NeuroEngineering Initiative, Rice University, Houston, TX, United States of America
| | - Vikash Gilja
- Department of Electrical and Computer Engineering, University of California, San Diego, La Jolla, CA, United States of America
| | | | - Yifan Kong
- Paradromics, Inc, Austin, TX, United States of America
| | | | | | | |
Collapse
|
257
|
Wilson GH, Stavisky SD, Willett FR, Avansino DT, Kelemen JN, Hochberg LR, Henderson JM, Druckmann S, Shenoy KV. Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus. J Neural Eng 2020; 17:066007. [PMID: 33236720 PMCID: PMC8293867 DOI: 10.1088/1741-2552/abbfef] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
Abstract
OBJECTIVE To evaluate the potential of intracortical electrode array signals for brain-computer interfaces (BCIs) to restore lost speech, we measured the performance of decoders trained to discriminate a comprehensive basis set of 39 English phonemes and to synthesize speech sounds via a neural pattern matching method. We decoded neural correlates of spoken-out-loud words in the 'hand knob' area of precentral gyrus, a step toward the eventual goal of decoding attempted speech from ventral speech areas in patients who are unable to speak. APPROACH Neural and audio data were recorded while two BrainGate2 pilot clinical trial participants, each with two chronically-implanted 96-electrode arrays, spoke 420 different words that broadly sampled English phonemes. Phoneme onsets were identified from audio recordings, and their identities were then classified from neural features consisting of each electrode's binned action potential counts or high-frequency local field potential power. Speech synthesis was performed using the 'Brain-to-Speech' pattern matching method. We also examined two potential confounds specific to decoding overt speech: acoustic contamination of neural signals and systematic differences in labeling different phonemes' onset times. MAIN RESULTS A linear decoder achieved up to 29.3% classification accuracy (chance = 6%) across 39 phonemes, while an RNN classifier achieved 33.9% accuracy. Parameter sweeps indicated that performance did not saturate when adding more electrodes or more training data, and that accuracy improved when utilizing time-varying structure in the data. Microphonic contamination and phoneme onset differences modestly increased decoding accuracy, but could be mitigated by acoustic artifact subtraction and using a neural speech onset marker, respectively. Speech synthesis achieved r = 0.523 correlation between true and reconstructed audio. SIGNIFICANCE The ability to decode speech using intracortical electrode array signals from a nontraditional speech area suggests that placing electrode arrays in ventral speech areas is a promising direction for speech BCIs.
Collapse
Affiliation(s)
- Guy H Wilson
- Neurosciences Graduate Program, Stanford University, Stanford, CA, United States of America
| | - Sergey D Stavisky
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
| | - Francis R Willett
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, United States of America
| | - Donald T Avansino
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
| | - Jessica N Kelemen
- Department of Neurology, Harvard Medical School, Boston, MA, United States of America
| | - Leigh R Hochberg
- Department of Neurology, Harvard Medical School, Boston, MA, United States of America
- Center for Neurotechnology and Neurorecovery, Dept. of Neurology, Massachusetts General Hospital, Boston, MA, United States of America
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, United States of America
- Carney Institute for Brain Science and School of Engineering, Brown University, Providence, RI, United States of America
| | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
| | - Shaul Druckmann
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
- Department of Neurobiology, Stanford University, Stanford, CA, United States of America
| | - Krishna V Shenoy
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, United States of America
- Department of Neurobiology, Stanford University, Stanford, CA, United States of America
- Department of Bioengineering, Stanford University, Stanford, CA, United States of America
| |
Collapse
|