1
|
Dash D, Ferrari P, Wang J. Neural Decoding of Spontaneous Overt and Intended Speech. JOURNAL OF SPEECH, LANGUAGE, AND HEARING RESEARCH : JSLHR 2024:1-10. [PMID: 39106199 DOI: 10.1044/2024_jslhr-24-00046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/09/2024]
Abstract
PURPOSE The aim of this study was to decode intended and overt speech from neuromagnetic signals while the participants performed spontaneous overt speech tasks without cues or prompts (stimuli). METHOD Magnetoencephalography (MEG), a noninvasive neuroimaging technique, was used to collect neural signals from seven healthy adult English speakers performing spontaneous, overt speech tasks. The participants randomly spoke the words yes or no at a self-paced rate without cues. Two machine learning models, namely, linear discriminant analysis (LDA) and one-dimensional convolutional neural network (1D CNN), were employed to classify the two words from the recorded MEG signals. RESULTS LDA and 1D CNN achieved average decoding accuracies of 79.02% and 90.40%, respectively, in decoding overt speech, significantly surpassing the chance level (50%). The accuracy for decoding intended speech was 67.19% using 1D CNN. CONCLUSIONS This study showcases the possibility of decoding spontaneous overt and intended speech directly from neural signals in the absence of perceptual interference. We believe that these findings make a steady step toward the future spontaneous speech-based brain-computer interface.
Collapse
Affiliation(s)
- Debadatta Dash
- Department of Neurology, The University of Texas at Austin
| | - Paul Ferrari
- Helen DeVos Children's Hospital, Corewell Health, Grand Rapids, MI
| | - Jun Wang
- Department of Neurology, The University of Texas at Austin
- Department of Speech, Language, and Hearing Sciences, The University of Texas at Austin
| |
Collapse
|
2
|
Chau G, Wang C, Talukder S, Subramaniam V, Soedarmadji S, Yue Y, Katz B, Barbu A. Population Transformer: Learning Population-level Representations of Intracranial Activity. ARXIV 2024:arXiv:2406.03044v1. [PMID: 38883237 PMCID: PMC11177958] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/18/2024]
Abstract
We present a self-supervised framework that learns population-level codes for intracranial neural recordings at scale, unlocking the benefits of representation learning for a key neuroscience recording modality. The Population Transformer (PopT) lowers the amount of data required for decoding experiments, while increasing accuracy, even on never-before-seen subjects and tasks. We address two key challenges in developing PopT: sparse electrode distribution and varying electrode location across patients. PopT stacks on top of pretrained representations and enhances downstream tasks by enabling learned aggregation of multiple spatially-sparse data channels. Beyond decoding, we interpret the pretrained PopT and fine-tuned models to show how it can be used to provide neuroscience insights learned from massive amounts of data. We release a pretrained PopT to enable off-the-shelf improvements in multi-channel intracranial data decoding and interpretability, and code is available at https://github.com/czlwang/PopulationTransformer.
Collapse
|
3
|
Wandelt SK, Bjånes DA, Pejsa K, Lee B, Liu C, Andersen RA. Representation of internal speech by single neurons in human supramarginal gyrus. Nat Hum Behav 2024; 8:1136-1149. [PMID: 38740984 PMCID: PMC11199147 DOI: 10.1038/s41562-024-01867-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Accepted: 03/16/2024] [Indexed: 05/16/2024]
Abstract
Speech brain-machine interfaces (BMIs) translate brain signals into words or audio outputs, enabling communication for people having lost their speech abilities due to diseases or injury. While important advances in vocalized, attempted and mimed speech decoding have been achieved, results for internal speech decoding are sparse and have yet to achieve high functionality. Notably, it is still unclear from which brain areas internal speech can be decoded. Here two participants with tetraplegia with implanted microelectrode arrays located in the supramarginal gyrus (SMG) and primary somatosensory cortex (S1) performed internal and vocalized speech of six words and two pseudowords. In both participants, we found significant neural representation of internal and vocalized speech, at the single neuron and population level in the SMG. From recorded population activity in the SMG, the internally spoken and vocalized words were significantly decodable. In an offline analysis, we achieved average decoding accuracies of 55% and 24% for each participant, respectively (chance level 12.5%), and during an online internal speech BMI task, we averaged 79% and 23% accuracy, respectively. Evidence of shared neural representations between internal speech, word reading and vocalized speech processes was found in participant 1. SMG represented words as well as pseudowords, providing evidence for phonetic encoding. Furthermore, our decoder achieved high classification with multiple internal speech strategies (auditory imagination/visual imagination). Activity in S1 was modulated by vocalized but not internal speech in both participants, suggesting no articulator movements of the vocal tract occurred during internal speech production. This work represents a proof-of-concept for a high-performance internal speech BMI.
Collapse
Affiliation(s)
- Sarah K Wandelt
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA.
- T&C Chen Brain-Machine Interface Center, California Institute of Technology, Pasadena, CA, USA.
| | - David A Bjånes
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- T&C Chen Brain-Machine Interface Center, California Institute of Technology, Pasadena, CA, USA
- Rancho Los Amigos National Rehabilitation Center, Downey, CA, USA
| | - Kelsie Pejsa
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- T&C Chen Brain-Machine Interface Center, California Institute of Technology, Pasadena, CA, USA
| | - Brian Lee
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- Department of Neurological Surgery, Keck School of Medicine of USC, Los Angeles, CA, USA
- USC Neurorestoration Center, Keck School of Medicine of USC, Los Angeles, CA, USA
| | - Charles Liu
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- Rancho Los Amigos National Rehabilitation Center, Downey, CA, USA
- Department of Neurological Surgery, Keck School of Medicine of USC, Los Angeles, CA, USA
- USC Neurorestoration Center, Keck School of Medicine of USC, Los Angeles, CA, USA
| | - Richard A Andersen
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- T&C Chen Brain-Machine Interface Center, California Institute of Technology, Pasadena, CA, USA
| |
Collapse
|
4
|
Chen J, Chen X, Wang R, Le C, Khalilian-Gourtani A, Jensen E, Dugan P, Doyle W, Devinsky O, Friedman D, Flinker A, Wang Y. Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.11.584533. [PMID: 38559163 PMCID: PMC10980022 DOI: 10.1101/2024.03.11.584533] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Objective This study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training. Approach We propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes, by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train both subject-specific models using data from a single participant as well as multi-patient models exploiting data from multiple participants. Main Results The subject-specific models using only low-density 8x8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi-subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation. Significance The proposed SwinTW decoder enables future speech neuroprostheses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests the exciting possibility of developing speech neuroprostheses for people with speech disability without relying on their own neural data for training, which is not always feasible.
Collapse
Affiliation(s)
- Junbo Chen
- Electrical and Computer Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| | - Xupeng Chen
- Electrical and Computer Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| | - Ran Wang
- Electrical and Computer Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| | - Chenqian Le
- Electrical and Computer Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| | | | - Erika Jensen
- Neurology Department, New York University, 223 East 34th Street, Manhattan, 10016, NY, USA
| | - Patricia Dugan
- Neurology Department, New York University, 223 East 34th Street, Manhattan, 10016, NY, USA
| | - Werner Doyle
- Neurosurgery Department, New York University, 550 1st Avenue, Manhattan, 10016, NY, USA
| | - Orrin Devinsky
- Neurology Department, New York University, 223 East 34th Street, Manhattan, 10016, NY, USA
| | - Daniel Friedman
- Neurology Department, New York University, 223 East 34th Street, Manhattan, 10016, NY, USA
| | - Adeen Flinker
- Neurology Department, New York University, 223 East 34th Street, Manhattan, 10016, NY, USA
- Biomedical Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| | - Yao Wang
- Electrical and Computer Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
- Biomedical Engineering Department, New York University, 370 Jay Street, Brooklyn, 11201, NY, USA
| |
Collapse
|
5
|
Deo DR, Willett FR, Avansino DT, Hochberg LR, Henderson JM, Shenoy KV. Brain control of bimanual movement enabled by recurrent neural networks. Sci Rep 2024; 14:1598. [PMID: 38238386 PMCID: PMC10796685 DOI: 10.1038/s41598-024-51617-3] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Accepted: 01/07/2024] [Indexed: 01/22/2024] Open
Abstract
Brain-computer interfaces have so far focused largely on enabling the control of a single effector, for example a single computer cursor or robotic arm. Restoring multi-effector motion could unlock greater functionality for people with paralysis (e.g., bimanual movement). However, it may prove challenging to decode the simultaneous motion of multiple effectors, as we recently found that a compositional neural code links movements across all limbs and that neural tuning changes nonlinearly during dual-effector motion. Here, we demonstrate the feasibility of high-quality bimanual control of two cursors via neural network (NN) decoders. Through simulations, we show that NNs leverage a neural 'laterality' dimension to distinguish between left and right-hand movements as neural tuning to both hands become increasingly correlated. In training recurrent neural networks (RNNs) for two-cursor control, we developed a method that alters the temporal structure of the training data by dilating/compressing it in time and re-ordering it, which we show helps RNNs successfully generalize to the online setting. With this method, we demonstrate that a person with paralysis can control two computer cursors simultaneously. Our results suggest that neural network decoders may be advantageous for multi-effector decoding, provided they are designed to transfer to the online setting.
Collapse
Affiliation(s)
- Darrel R Deo
- Department of Neurosurgery, Stanford University, Stanford, CA, USA.
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA.
| | - Francis R Willett
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Donald T Avansino
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Leigh R Hochberg
- School of Engineering, Brown University, Providence, RI, USA
- Carney Institute for Brain Science, Brown University, Providence, RI, USA
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, USA
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Bio-X Institute, Stanford University, Stanford, CA, USA
| | - Krishna V Shenoy
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
- Bio-X Institute, Stanford University, Stanford, CA, USA
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Department of Bioengineering, Stanford University, Stanford, CA, USA
- Department of Neurobiology, Stanford University, Stanford, CA, USA
| |
Collapse
|
6
|
Tai P, Ding P, Wang F, Gong A, Li T, Zhao L, Su L, Fu Y. Brain-computer interface paradigms and neural coding. Front Neurosci 2024; 17:1345961. [PMID: 38287988 PMCID: PMC10822902 DOI: 10.3389/fnins.2023.1345961] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Accepted: 12/28/2023] [Indexed: 01/31/2024] Open
Abstract
Brain signal patterns generated in the central nervous system of brain-computer interface (BCI) users are closely related to BCI paradigms and neural coding. In BCI systems, BCI paradigms and neural coding are critical elements for BCI research. However, so far there have been few references that clearly and systematically elaborated on the definition and design principles of the BCI paradigm as well as the definition and modeling principles of BCI neural coding. Therefore, these contents are expounded and the existing main BCI paradigms and neural coding are introduced in the review. Finally, the challenges and future research directions of BCI paradigm and neural coding were discussed, including user-centered design and evaluation for BCI paradigms and neural coding, revolutionizing the traditional BCI paradigms, breaking through the existing techniques for collecting brain signals and combining BCI technology with advanced AI technology to improve brain signal decoding performance. It is expected that the review will inspire innovative research and development of the BCI paradigm and neural coding.
Collapse
Affiliation(s)
- Pengrui Tai
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Peng Ding
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Fan Wang
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Anmin Gong
- School of Information Engineering, Chinese People’s Armed Police Force Engineering University, Xi’an, China
| | - Tianwen Li
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Zhao
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
- Faculty of Science, Kunming University of Science and Technology, Kunming, China
| | - Lei Su
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| | - Yunfa Fu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming, China
| |
Collapse
|
7
|
Koizumi K, Kunii N, Ueda K, Takabatake K, Nagata K, Fujitani S, Shimada S, Nakao M. Intracranial Neurofeedback Modulating Neural Activity in the Mesial Temporal Lobe During Memory Encoding: A Pilot Study. Appl Psychophysiol Biofeedback 2023; 48:439-451. [PMID: 37405548 PMCID: PMC10581957 DOI: 10.1007/s10484-023-09595-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/24/2023] [Indexed: 07/06/2023]
Abstract
Removal of the mesial temporal lobe (MTL) is an established surgical procedure that leads to seizure freedom in patients with intractable MTL epilepsy; however, it carries the potential risk of memory damage. Neurofeedback (NF), which regulates brain function by converting brain activity into perceptible information and providing feedback, has attracted considerable attention in recent years for its potential as a novel complementary treatment for many neurological disorders. However, no research has attempted to artificially reorganize memory functions by applying NF before resective surgery to preserve memory functions. Thus, this study aimed (1) to construct a memory NF system that used intracranial electrodes to feedback neural activity on the language-dominant side of the MTL during memory encoding and (2) to verify whether neural activity and memory function in the MTL change with NF training. Two intractable epilepsy patients with implanted intracranial electrodes underwent at least five sessions of memory NF training to increase the theta power in the MTL. There was an increase in theta power and a decrease in fast beta and gamma powers in one of the patients in the late stage of memory NF sessions. NF signals were not correlated with memory function. Despite its limitations as a pilot study, to our best knowledge, this study is the first to report that intracranial NF may modulate neural activity in the MTL, which is involved in memory encoding. The findings provide important insights into the future development of NF systems for the artificial reorganization of memory functions.
Collapse
Affiliation(s)
- Koji Koizumi
- Department of Mechanical Engineering, Graduate School of Engineering, The University of Tokyo, Tokyo, Japan.
| | - Naoto Kunii
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Kazutaka Ueda
- Department of Mechanical Engineering, Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
| | | | - Keisuke Nagata
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Shigeta Fujitani
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Seijiro Shimada
- Department of Neurosurgery, The University of Tokyo, Tokyo, Japan
| | - Masayuki Nakao
- Department of Mechanical Engineering, Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
8
|
Duraivel S, Rahimpour S, Chiang CH, Trumpis M, Wang C, Barth K, Harward SC, Lad SP, Friedman AH, Southwell DG, Sinha SR, Viventi J, Cogan GB. High-resolution neural recordings improve the accuracy of speech decoding. Nat Commun 2023; 14:6938. [PMID: 37932250 PMCID: PMC10628285 DOI: 10.1038/s41467-023-42555-1] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Accepted: 10/13/2023] [Indexed: 11/08/2023] Open
Abstract
Patients suffering from debilitating neurodegenerative diseases often lose the ability to communicate, detrimentally affecting their quality of life. One solution to restore communication is to decode signals directly from the brain to enable neural speech prostheses. However, decoding has been limited by coarse neural recordings which inadequately capture the rich spatio-temporal structure of human brain signals. To resolve this limitation, we performed high-resolution, micro-electrocorticographic (µECoG) neural recordings during intra-operative speech production. We obtained neural signals with 57× higher spatial resolution and 48% higher signal-to-noise ratio compared to macro-ECoG and SEEG. This increased signal quality improved decoding by 35% compared to standard intracranial signals. Accurate decoding was dependent on the high-spatial resolution of the neural interface. Non-linear decoding models designed to utilize enhanced spatio-temporal neural information produced better results than linear techniques. We show that high-density µECoG can enable high-quality speech decoding for future neural speech prostheses.
Collapse
Affiliation(s)
| | - Shervin Rahimpour
- Department of Neurosurgery, Duke School of Medicine, Durham, NC, USA
- Department of Neurosurgery, Clinical Neuroscience Center, University of Utah, Salt Lake City, UT, USA
| | - Chia-Han Chiang
- Department of Biomedical Engineering, Duke University, Durham, NC, USA
| | - Michael Trumpis
- Department of Biomedical Engineering, Duke University, Durham, NC, USA
| | - Charles Wang
- Department of Biomedical Engineering, Duke University, Durham, NC, USA
| | - Katrina Barth
- Department of Biomedical Engineering, Duke University, Durham, NC, USA
| | - Stephen C Harward
- Department of Neurosurgery, Duke School of Medicine, Durham, NC, USA
- Duke Comprehensive Epilepsy Center, Duke School of Medicine, Durham, NC, USA
| | - Shivanand P Lad
- Department of Neurosurgery, Duke School of Medicine, Durham, NC, USA
| | - Allan H Friedman
- Department of Neurosurgery, Duke School of Medicine, Durham, NC, USA
| | - Derek G Southwell
- Department of Biomedical Engineering, Duke University, Durham, NC, USA
- Department of Neurosurgery, Duke School of Medicine, Durham, NC, USA
- Duke Comprehensive Epilepsy Center, Duke School of Medicine, Durham, NC, USA
- Department of Neurobiology, Duke School of Medicine, Durham, NC, USA
| | - Saurabh R Sinha
- Penn Epilepsy Center, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Jonathan Viventi
- Department of Biomedical Engineering, Duke University, Durham, NC, USA.
- Department of Neurosurgery, Duke School of Medicine, Durham, NC, USA.
- Duke Comprehensive Epilepsy Center, Duke School of Medicine, Durham, NC, USA.
- Department of Neurobiology, Duke School of Medicine, Durham, NC, USA.
| | - Gregory B Cogan
- Department of Biomedical Engineering, Duke University, Durham, NC, USA.
- Department of Neurosurgery, Duke School of Medicine, Durham, NC, USA.
- Duke Comprehensive Epilepsy Center, Duke School of Medicine, Durham, NC, USA.
- Department of Neurology, Duke School of Medicine, Durham, NC, USA.
- Department of Psychology and Neuroscience, Duke University, Durham, NC, USA.
- Center for Cognitive Neuroscience, Duke University, Durham, NC, USA.
| |
Collapse
|
9
|
Berezutskaya J, Freudenburg ZV, Vansteensel MJ, Aarnoutse EJ, Ramsey NF, van Gerven MAJ. Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models. J Neural Eng 2023; 20:056010. [PMID: 37467739 PMCID: PMC10510111 DOI: 10.1088/1741-2552/ace8be] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2022] [Revised: 07/12/2023] [Accepted: 07/19/2023] [Indexed: 07/21/2023]
Abstract
Objective.Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial for advancing the field.Approach.In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task.Main results.We show that (1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; (2) individual word decoding in reconstructed speech achieves 92%-100% accuracy (chance level is 8%); (3) direct reconstruction from sensorimotor brain activity produces intelligible speech.Significance.These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.
Collapse
Affiliation(s)
- Julia Berezutskaya
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands
- Donders Center for Brain, Cognition and Behaviour, Nijmegen 6525 GD, The Netherlands
| | - Zachary V Freudenburg
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands
| | - Mariska J Vansteensel
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands
| | - Erik J Aarnoutse
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands
| | - Nick F Ramsey
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht 3584 CX, The Netherlands
| | - Marcel A J van Gerven
- Donders Center for Brain, Cognition and Behaviour, Nijmegen 6525 GD, The Netherlands
| |
Collapse
|
10
|
Koizumi K, Kunii N, Ueda K, Nagata K, Fujitani S, Shimada S, Nakao M. Paving the Way for Memory Enhancement: Development and Examination of a Neurofeedback System Targeting the Medial Temporal Lobe. Biomedicines 2023; 11:2262. [PMID: 37626758 PMCID: PMC10452721 DOI: 10.3390/biomedicines11082262] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Revised: 08/01/2023] [Accepted: 08/11/2023] [Indexed: 08/27/2023] Open
Abstract
Neurofeedback (NF) shows promise in enhancing memory, but its application to the medial temporal lobe (MTL) still needs to be studied. Therefore, we aimed to develop an NF system for the memory function of the MTL and examine neural activity changes and memory task score changes through NF training. We created a memory NF system using intracranial electrodes to acquire and visualise the neural activity of the MTL during memory encoding. Twenty trials of a tug-of-war game per session were employed for NF and designed to control neural activity bidirectionally (Up/Down condition). NF training was conducted with three patients with drug-resistant epilepsy, and we observed an increasing difference in NF signal between conditions (Up-Down) as NF training progressed. Similarities and negative correlation tendencies between the transition of neural activity and the transition of memory function were also observed. Our findings demonstrate NF's potential to modulate MTL activity and memory encoding. Future research needs further improvements to the NF system to validate its effects on memory functions. Nonetheless, this study represents a crucial step in understanding NF's application to memory and provides valuable insights into developing more efficient memory enhancement strategies.
Collapse
Affiliation(s)
- Koji Koizumi
- Department of Mechanical Engineering, The University of Tokyo, Tokyo 113-8656, Japan; (K.U.); (M.N.)
| | - Naoto Kunii
- Department of Neurosurgery, The University of Tokyo, Tokyo 113-8655, Japan; (N.K.); (K.N.); (S.F.); (S.S.)
| | - Kazutaka Ueda
- Department of Mechanical Engineering, The University of Tokyo, Tokyo 113-8656, Japan; (K.U.); (M.N.)
| | - Keisuke Nagata
- Department of Neurosurgery, The University of Tokyo, Tokyo 113-8655, Japan; (N.K.); (K.N.); (S.F.); (S.S.)
| | - Shigeta Fujitani
- Department of Neurosurgery, The University of Tokyo, Tokyo 113-8655, Japan; (N.K.); (K.N.); (S.F.); (S.S.)
| | - Seijiro Shimada
- Department of Neurosurgery, The University of Tokyo, Tokyo 113-8655, Japan; (N.K.); (K.N.); (S.F.); (S.S.)
| | - Masayuki Nakao
- Department of Mechanical Engineering, The University of Tokyo, Tokyo 113-8656, Japan; (K.U.); (M.N.)
| |
Collapse
|
11
|
Soroush PZ, Herff C, Ries SK, Shih JJ, Schultz T, Krusienski DJ. The nested hierarchy of overt, mouthed, and imagined speech activity evident in intracranial recordings. Neuroimage 2023; 269:119913. [PMID: 36731812 DOI: 10.1016/j.neuroimage.2023.119913] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Revised: 01/05/2023] [Accepted: 01/29/2023] [Indexed: 02/01/2023] Open
Abstract
Recent studies have demonstrated that it is possible to decode and synthesize various aspects of acoustic speech directly from intracranial measurements of electrophysiological brain activity. In order to continue progressing toward the development of a practical speech neuroprosthesis for the individuals with speech impairments, better understanding and modeling of imagined speech processes are required. The present study uses intracranial brain recordings from participants that performed a speaking task with trials consisting of overt, mouthed, and imagined speech modes, representing various degrees of decreasing behavioral output. Speech activity detection models are constructed using spatial, spectral, and temporal brain activity features, and the features and model performances are characterized and compared across the three degrees of behavioral output. The results indicate the existence of a hierarchy in which the relevant channels for the lower behavioral output modes form nested subsets of the relevant channels from the higher behavioral output modes. This provides important insights for the elusive goal of developing more effective imagined speech decoding models with respect to the better-established overt speech decoding counterparts.
Collapse
|
12
|
Branco MP, Geukes SH, Aarnoutse EJ, Ramsey NF, Vansteensel MJ. Nine decades of electrocorticography: A comparison between epidural and subdural recordings. Eur J Neurosci 2023; 57:1260-1288. [PMID: 36843389 DOI: 10.1111/ejn.15941] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2022] [Revised: 02/10/2023] [Accepted: 02/18/2023] [Indexed: 02/28/2023]
Abstract
In recent years, electrocorticography (ECoG) has arisen as a neural signal recording tool in the development of clinically viable neural interfaces. ECoG electrodes are generally placed below the dura mater (subdural) but can also be placed on top of the dura (epidural). In deciding which of these modalities best suits long-term implants, complications and signal quality are important considerations. Conceptually, epidural placement may present a lower risk of complications as the dura is left intact but also a lower signal quality due to the dura acting as a signal attenuator. The extent to which complications and signal quality are affected by the dura, however, has been a matter of debate. To improve our understanding of the effects of the dura on complications and signal quality, we conducted a literature review. We inventorized the effect of the dura on signal quality, decodability and longevity of acute and chronic ECoG recordings in humans and non-human primates. Also, we compared the incidence and nature of serious complications in studies that employed epidural and subdural ECoG. Overall, we found that, even though epidural recordings exhibit attenuated signal amplitude over subdural recordings, particularly for high-density grids, the decodability of epidural recorded signals does not seem to be markedly affected. Additionally, we found that the nature of serious complications was comparable between epidural and subdural recordings. These results indicate that both epidural and subdural ECoG may be suited for long-term neural signal recordings, at least for current generations of clinical and high-density ECoG grids.
Collapse
Affiliation(s)
- Mariana P Branco
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Simon H Geukes
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Erik J Aarnoutse
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Nick F Ramsey
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| | - Mariska J Vansteensel
- Department of Neurology and Neurosurgery, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
13
|
Zuccaroli I, Lucke-Wold B, Palla A, Eremiev A, Sorrentino Z, Zakare-Fagbamila R, McNulty J, Christie C, Chandra V, Mampre D. Neural Bypasses: Literature Review and Future Directions in Developing Artificial Neural Connections. OBM NEUROBIOLOGY 2023; 7:158. [PMID: 36908763 PMCID: PMC9997488 DOI: 10.21926/obm.neurobiol.2301158] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/24/2023]
Abstract
Reported neuro-modulation schemes in the literature are typically classified as closed-loop or open-loop. A novel group of recently developed neuro-modulation devices may be better described as a neural bypass, which attempts to transmit neural data from one location of the nervous system to another. The most common form of neural bypasses in the literature utilize EEG recordings of cortical information paired with functional electrical stimulation for effector muscle output, most commonly for assistive applications and rehabilitation in spinal cord injury or stroke. Other neural bypass locations that have also been described, or may soon be in development, include cortical-spinal bypasses, cortical-cortical bypasses, autonomic bypasses, peripheral-central bypasses, and inter-subject bypasses. The most common recording devices include EEG, ECoG, and microelectrode arrays, while stimulation devices include both invasive and noninvasive electrodes. Several devices are in development to improve the temporal and spatial resolution and biocompatibility for neuronal recording and stimulation. A major barrier to entry includes neuroplasticity and current decoding mechanisms that regularly require retraining. Neural bypasses are a unique class of neuro-modulation. Continued advancement of neural recording and stimulating devices with high spatial and temporal resolution, combined with decoding mechanisms uninhibited by neuroplasticity, can expand the therapeutic capability of neural bypassing. Overall, neural bypasses are a promising modality to improve the treatment of common neurologic disorders, including stroke, spinal cord injury, peripheral nerve injury, brain injury and more.
Collapse
Affiliation(s)
| | | | | | - Alexander Eremiev
- Department of Neurosurgery, New York University School of Medicine, New York, USA
| | | | | | - Jack McNulty
- Department of Neurosurgery, University of Iowa, Iowa City, IA, USA
| | - Carlton Christie
- Department of Neurosurgery, University of Florida, Gainesville, USA
| | - Vyshak Chandra
- Department of Neurosurgery, University of Florida, Gainesville, USA
| | - David Mampre
- Johns Hopkins University, Baltimore, USA
- Department of Neurosurgery, University of Florida, Gainesville, USA
| |
Collapse
|
14
|
Abrego AM, Khan W, Wright CE, Islam MR, Ghajar MH, Bai X, Tandon N, Seymour JP. Sensing local field potentials with a directional and scalable depth electrode array. J Neural Eng 2023; 20:016041. [PMID: 36630716 DOI: 10.1088/1741-2552/acb230] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Accepted: 01/11/2023] [Indexed: 01/13/2023]
Abstract
Objective. A variety of electrophysiology tools are available to the neurosurgeon for diagnosis, functional therapy, and neural prosthetics. However, no tool can currently address these three critical needs: (a) access to all cortical regions in a minimally invasive manner; (b) recordings with microscale, mesoscale, and macroscale resolutions simultaneously; and (c) access to spatially distant multiple brain regions that constitute distributed cognitive networks.Approach.We modeled, designed, and demonstrated a novel device for recording local field potentials (LFPs) with the form factor of a stereo-electroencephalographic electrode and combined with radially distributed microelectrodes.Main results. Electro-quasistatic models demonstrate that the lead body amplifies and shields LFP sources based on direction, enablingdirectional sensitivity andscalability, referred to as thedirectional andscalable (DISC) array.In vivo,DISC demonstrated significantly improved signal-to-noise ratio, directional sensitivity, and decoding accuracy from rat barrel cortex recordings during whisker stimulation. Critical for future translation, DISC demonstrated a higher signal to noise ratio (SNR) than virtual ring electrodes and a noise floor approaching that of large ring electrodes in an unshielded environment after common average referencing. DISC also revealed independent, stereoscopic current source density measures whose direction was verified after histology.Significance. Directional sensitivity of LFPs may significantly improve brain-computer interfaces and many diagnostic procedures, including epilepsy foci detection and deep brain targeting.
Collapse
Affiliation(s)
- Amada M Abrego
- Department of Neurosurgery, University of Texas Health Science Center, Houston, TX 77030, United States of America
| | - Wasif Khan
- Department of Neurosurgery, University of Texas Health Science Center, Houston, TX 77030, United States of America
| | - Christopher E Wright
- Department of Neurosurgery, University of Texas Health Science Center, Houston, TX 77030, United States of America
- Department of Bioengineering, Rice University, Houston, TX 77030, United States of America
| | - M Rabiul Islam
- Department of Neurosurgery, University of Texas Health Science Center, Houston, TX 77030, United States of America
| | - Mohammad H Ghajar
- Department of Neurosurgery, University of Texas Health Science Center, Houston, TX 77030, United States of America
| | - Xiaokang Bai
- Department of Neurosurgery, University of Texas Health Science Center, Houston, TX 77030, United States of America
| | - Nitin Tandon
- Department of Neurosurgery, University of Texas Health Science Center, Houston, TX 77030, United States of America
| | - John P Seymour
- Department of Neurosurgery, University of Texas Health Science Center, Houston, TX 77030, United States of America
- Department of Electrical and Computer Engineering, Rice University, Houston, TX 77030, United States of America
| |
Collapse
|
15
|
Towards clinical application of implantable brain-computer interfaces for people with late-stage ALS: medical and ethical considerations. J Neurol 2023; 270:1323-1336. [PMID: 36450968 PMCID: PMC9971103 DOI: 10.1007/s00415-022-11464-6] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2022] [Revised: 10/26/2022] [Accepted: 10/27/2022] [Indexed: 12/05/2022]
Abstract
Individuals with amyotrophic lateral sclerosis (ALS) frequently develop speech and communication problems in the course of their disease. Currently available augmentative and alternative communication technologies do not present a solution for many people with advanced ALS, because these devices depend on residual and reliable motor activity. Brain-computer interfaces (BCIs) use neural signals for computer control and may allow people with late-stage ALS to communicate even when conventional technology falls short. Recent years have witnessed fast progression in the development and validation of implanted BCIs, which place neural signal recording electrodes in or on the cortex. Eventual widespread clinical application of implanted BCIs as an assistive communication technology for people with ALS will have significant consequences for their daily life, as well as for the clinical management of the disease, among others because of the potential interaction between the BCI and other procedures people with ALS undergo, such as tracheostomy. This article aims to facilitate responsible real-world implementation of implanted BCIs. We review the state of the art of research on implanted BCIs for communication, as well as the medical and ethical implications of the clinical application of this technology. We conclude that the contribution of all BCI stakeholders, including clinicians of the various ALS-related disciplines, will be needed to develop procedures for, and shape the process of, the responsible clinical application of implanted BCIs.
Collapse
|
16
|
Verwoert M, Ottenhoff MC, Goulis S, Colon AJ, Wagner L, Tousseyn S, van Dijk JP, Kubben PL, Herff C. Dataset of Speech Production in intracranial.Electroencephalography. Sci Data 2022; 9:434. [PMID: 35869138 PMCID: PMC9307753 DOI: 10.1038/s41597-022-01542-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2022] [Accepted: 07/08/2022] [Indexed: 11/28/2022] Open
Abstract
Speech production is an intricate process involving a large number of muscles and cognitive processes. The neural processes underlying speech production are not completely understood. As speech is a uniquely human ability, it can not be investigated in animal models. High-fidelity human data can only be obtained in clinical settings and is therefore not easily available to all researchers. Here, we provide a dataset of 10 participants reading out individual words while we measured intracranial EEG from a total of 1103 electrodes. The data, with its high temporal resolution and coverage of a large variety of cortical and sub-cortical brain regions, can help in understanding the speech production process better. Simultaneously, the data can be used to test speech decoding and synthesis approaches from neural data to develop speech Brain-Computer Interfaces and speech neuroprostheses. Measurement(s) | Brain activity | Technology Type(s) | Stereotactic electroencephalography | Sample Characteristic - Organism | Homo sapiens | Sample Characteristic - Environment | Epilepsy monitoring center | Sample Characteristic - Location | The Netherlands |
Collapse
|
17
|
Petrosyan A, Voskoboinikov A, Sukhinin D, Makarova A, Skalnaya A, Arkhipova N, Sinkin M, Ossadtchi A. Speech decoding from a small set of spatially segregated minimally invasive intracranial EEG electrodes with a compact and interpretable neural network. J Neural Eng 2022; 19. [PMID: 36356309 DOI: 10.1088/1741-2552/aca1e1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2022] [Accepted: 11/10/2022] [Indexed: 11/12/2022]
Abstract
Objective. Speech decoding, one of the most intriguing brain-computer interface applications, opens up plentiful opportunities from rehabilitation of patients to direct and seamless communication between human species. Typical solutions rely on invasive recordings with a large number of distributed electrodes implanted through craniotomy. Here we explored the possibility of creating speech prosthesis in a minimally invasive setting with a small number of spatially segregated intracranial electrodes.Approach. We collected one hour of data (from two sessions) in two patients implanted with invasive electrodes. We then used only the contacts that pertained to a single stereotactic electroencephalographic (sEEG) shaft or an electrocorticographic (ECoG) stripe to decode neural activity into 26 words and one silence class. We employed a compact convolutional network-based architecture whose spatial and temporal filter weights allow for a physiologically plausible interpretation.Mainresults. We achieved on average 55% accuracy using only six channels of data recorded with a single minimally invasive sEEG electrode in the first patient and 70% accuracy using only eight channels of data recorded for a single ECoG strip in the second patient in classifying 26+1 overtly pronounced words. Our compact architecture did not require the use of pre-engineered features, learned fast and resulted in a stable, interpretable and physiologically meaningful decision rule successfully operating over a contiguous dataset collected during a different time interval than that used for training. Spatial characteristics of the pivotal neuronal populations corroborate with active and passive speech mapping results and exhibit the inverse space-frequency relationship characteristic of neural activity. Compared to other architectures our compact solution performed on par or better than those recently featured in neural speech decoding literature.Significance. We showcase the possibility of building a speech prosthesis with a small number of electrodes and based on a compact feature engineering free decoder derived from a small amount of training data.
Collapse
Affiliation(s)
- Artur Petrosyan
- Center for Bioelectric Interfaces, Higher School of Economics, Moscow, Russia
| | | | - Dmitrii Sukhinin
- Center for Bioelectric Interfaces, Higher School of Economics, Moscow, Russia
| | - Anna Makarova
- Center for Bioelectric Interfaces, Higher School of Economics, Moscow, Russia
| | | | | | - Mikhail Sinkin
- Moscow State University of Medicine and Dentistry, Scientific Research Institute of First Aid to them. N.V. Sklifosovsky, Moscow, Russia
| | - Alexei Ossadtchi
- Center for Bioelectric Interfaces, Higher School of Economics, Moscow, Russia.,Artificial Intelligence Research Institute, AIRI, Moscow, Russia
| |
Collapse
|
18
|
Go GT, Lee Y, Seo DG, Lee TW. Organic Neuroelectronics: From Neural Interfaces to Neuroprosthetics. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2022; 34:e2201864. [PMID: 35925610 DOI: 10.1002/adma.202201864] [Citation(s) in RCA: 21] [Impact Index Per Article: 10.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/26/2022] [Revised: 07/17/2022] [Indexed: 06/15/2023]
Abstract
Requirements and recent advances in research on organic neuroelectronics are outlined herein. Neuroelectronics such as neural interfaces and neuroprosthetics provide a promising approach to diagnose and treat neurological diseases. However, the current neural interfaces are rigid and not biocompatible, so they induce an immune response and deterioration of neural signal transmission. Organic materials are promising candidates for neural interfaces, due to their mechanical softness, excellent electrochemical properties, and biocompatibility. Also, organic nervetronics, which mimics functional properties of the biological nerve system, is being developed to overcome the limitations of the complex and energy-consuming conventional neuroprosthetics that limit long-term implantation and daily-life usage. Examples of organic materials for neural interfaces and neural signal recordings are reviewed, recent advances of organic nervetronics that use organic artificial synapses are highlighted, and then further requirements for neuroprosthetics are discussed. Finally, the future challenges that must be overcome to achieve ideal organic neuroelectronics for next-generation neuroprosthetics are discussed.
Collapse
Affiliation(s)
- Gyeong-Tak Go
- Department of Materials Science and Engineering, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826, Republic of Korea
| | - Yeongjun Lee
- Department of Chemical Engineering, Stanford University, Stanford, CA, 94305, USA
| | - Dae-Gyo Seo
- Department of Materials Science and Engineering, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826, Republic of Korea
| | - Tae-Woo Lee
- Department of Materials Science and Engineering, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826, Republic of Korea
- Institute of Engineering Research, Research Institute of Advanced Materials, Soft Foundry, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826, Republic of Korea
- School of Chemical and Biological Engineering, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826, Republic of Korea
| |
Collapse
|
19
|
Samanta D. Recent developments in stereo electroencephalography monitoring for epilepsy surgery. Epilepsy Behav 2022; 135:108914. [PMID: 36116362 DOI: 10.1016/j.yebeh.2022.108914] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/18/2022] [Revised: 08/31/2022] [Accepted: 09/02/2022] [Indexed: 11/03/2022]
Abstract
Recently the utilization of the stereo electroencephalography (SEEG) method has exploded globally. It is now the preferred method of intracranial monitoring for epilepsy. Since its inception, the basic tenet of the SEEG method remains the same: strategic implantation of intracerebral electrodes based on a hypothesis grounded on anatomo-electroclinical correlation, interpretation of interictal and ictal abnormalities, and formation of a surgical plan based on these data. However, there are recent advancements in all these domains-electrodes implantations, data interpretation, and therapeutic strategy- that can make the SEEG a more accessible and effective approach. In this narrative review, these newer developments are discussed and summarized. Regarding implantation, efficient commercial robotic systems are now increasingly available, which are also more accurate in implanting electrodes. In terms of ictal and interictal abnormalities, newer studies focused on correlating these abnormalities with pathological substrates and surgical outcomes and analyzing high-frequency oscillations and cortical-subcortical connectivity. These abnormalities can now be further quantified using advanced tools (spectrum, spatiotemporal, connectivity analysis, and machine learning algorithms) for objective and efficient interpretation. Another aspect of recent development is renewed interest in SEEG-based electrical stimulation mapping (ESM). The SEEG-ESM has been used in defining epileptogenic networks, mapping eloquent cortex (primarily language), and analyzing cortico-cortical evoked potential. Regarding SEEG-guided direct therapeutic strategy, several clinical studies evaluated the use of radiofrequency thermocoagulation. As the emerging SEEG-based diagnosis and therapeutics are better evolved, treatments aimed at specific epileptogenic networks without compromising the eloquent cortex will become more easily accessible to improve the lives of individuals with drug-resistant epilepsy (DRE).
Collapse
Affiliation(s)
- Debopam Samanta
- Neurology Division, Department of Pediatrics, University of Arkansas for Medical Sciences, Little Rock, AR, United States.
| |
Collapse
|
20
|
Ganesh A, Cervantes AJ, Kennedy PR. Slow Firing Single Units Are Essential for Optimal Decoding of Silent Speech. Front Hum Neurosci 2022; 16:874199. [PMID: 35992944 PMCID: PMC9382878 DOI: 10.3389/fnhum.2022.874199] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2022] [Accepted: 06/14/2022] [Indexed: 11/18/2022] Open
Abstract
The motivation of someone who is locked-in, that is, paralyzed and mute, is to find relief for their loss of function. The data presented in this report is part of an attempt to restore one of those lost functions, namely, speech. An essential feature of the development of a speech prosthesis is optimal decoding of patterns of recorded neural signals during silent or covert speech, that is, speaking “inside the head” with output that is inaudible due to the paralysis of the articulators. The aim of this paper is to illustrate the importance of both fast and slow single unit firings recorded from an individual with locked-in syndrome and from an intact participant speaking silently. Long duration electrodes were implanted in the motor speech cortex for up to 13 years in the locked-in participant. The data herein provide evidence that slow firing single units are essential for optimal decoding accuracy. Additional evidence indicates that slow firing single units can be conditioned in the locked-in participant 5 years after implantation, further supporting their role in decoding.
Collapse
Affiliation(s)
- Ananya Ganesh
- Neural Signals Inc., Neural Prostheses Laboratory, Duluth, GA, United States
| | | | - Philip R. Kennedy
- Neural Signals Inc., Neural Prostheses Laboratory, Duluth, GA, United States
- *Correspondence: Philip R. Kennedy
| |
Collapse
|
21
|
Soroush PZ, Herff C, Ries S, Shih JJ, Schultz T, Krusienski DJ. Contributions of Stereotactic EEG Electrodes in Grey and White Matter to Speech Activity Detection. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:4789-4792. [PMID: 36086071 DOI: 10.1109/embc48229.2022.9871464] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Recent studies have shown it is possible to decode and synthesize speech directly using brain activity recorded from implanted electrodes. While this activity has been extensively examined using electrocorticographic (ECoG) recordings from cortical surface grey matter, stereotactic electroen-cephalography (sEEG) provides comparatively broader coverage and access to deeper brain structures including both grey and white matter. The present study examines the relative and joint contributions of grey and white matter electrodes for speech activity detection in a brain-computer interface.
Collapse
|
22
|
Paulo DL, Wills KE, Johnson GW, Gonzalez HFJ, Rolston JD, Naftel RP, Reddy SB, Morgan VL, Kang H, Williams Roberson S, Narasimhan S, Englot DJ. SEEG Functional Connectivity Measures to Identify Epileptogenic Zones: Stability, Medication Influence, and Recording Condition. Neurology 2022; 98:e2060-e2072. [PMID: 35338075 PMCID: PMC9162047 DOI: 10.1212/wnl.0000000000200386] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2021] [Accepted: 02/01/2022] [Indexed: 11/15/2022] Open
Abstract
BACKGROUND AND OBJECTIVES Functional connectivity (FC) measures can be used to differentiate epileptogenic zones (EZs) from non-EZs in patients with medically refractory epilepsy. Little work has been done to evaluate the stability of stereo-EEG (SEEG) FC measures over time and their relationship with antiseizure medication (ASM) use, a critical confounder in epilepsy FC studies. We aimed to answer the following questions: Are SEEG FC measures stable over time? Are they influenced by ASMs? Are they affected by patient data collection state? METHODS In 32 patients with medically refractory focal epilepsy, we collected a single 2-minute prospective SEEG resting-state (awake, eyes closed) data set and consecutive 2-minute retrospective pseudo-rest (awake, eyes open) data sets for days 1-7 postimplantation. ASM dosages were recorded for days 1-7 postimplantation and drug load score (DLS) per day was calculated to standardize and compare across patients. FC was evaluated using directed and nondirected measures. Standard clinical interpretation of ictal SEEG was used to classify brain regions as EZs and non-EZs. RESULTS Over 7 days, presumed EZs consistently had higher FC than non-EZs when using between imaginary coherence (ImCoh) and partial directed coherence (PDC) inward strength, without accounting for DLS. These measures were demonstrated to be stable over a short-term period of 3 consecutive days with the same DLS. Between ImCoh FC differences between EZs and non-EZs were reduced with DLS decreases, whereas other measures were not affected by DLS. FC differences between EZs and non-EZs were seen during both resting-state and pseudo-rest conditions; ImCoh values were strongly correlated between the 2 conditions, whereas PDC values were not. DISCUSSION Inward and nondirected SEEG FC is higher in presumed EZs vs non-EZs and measures are stable over time. However, certain measures may be affected by ASM dose, as between ImCoh differences between EZs and non-EZs are less pronounced with lower doses, and other measures such as PDC are poorly correlated across recording conditions. These findings allow novel insight into how SEEG FC measures may aid surgical localization and how they are influenced by ASMs and other factors.
Collapse
Affiliation(s)
- Danika L Paulo
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - Kristin E Wills
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - Graham W Johnson
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - Hernan F J Gonzalez
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - John D Rolston
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - Robert P Naftel
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - Shilpa B Reddy
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - Victoria L Morgan
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - Hakmook Kang
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - Shawniqua Williams Roberson
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - Saramati Narasimhan
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| | - Dario J Englot
- From the Departments of Neurological Surgery (D.L.P., K.E.W., R.P.N., V.L.M., S.N., D.J.E.), Radiology and Radiological Sciences (V.L.M., D.J.E.), Biostatistics (V.L.M., S.W.R., D.J.E.), and Neurology (H.K.), Vanderbilt University Medical Center; Vanderbilt University Institute of Imaging Science (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Vanderbilt Institute for Surgery and Engineering (K.E.W., G.W.J., H.F.J.G., V.L.M., S.N., D.J.E.); Department of Biomedical Engineering (G.W.J., H.F.J.G., V.L.M., S.W.R., S.N., D.J.E.), Vanderbilt University, Nashville, TN; Departments of Neurosurgery and Biomedical Engineering (J.D.R.), University of Utah, Salt Lake City; and Department of Pediatrics (S.B.R.), Vanderbilt Children's Hospital, Nashville, TN
| |
Collapse
|
23
|
Wu X, Li G, Jiang S, Wellington S, Liu S, Wu Z, Metcalfe B, Chen L, Zhang D. Decoding Continuous Kinetic Information of Grasp from Stereo-electroencephalographic (SEEG) Recordings. J Neural Eng 2022; 19. [PMID: 35395645 DOI: 10.1088/1741-2552/ac65b1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2021] [Accepted: 04/08/2022] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Brain-computer interfaces (BCIs) have the potential to bypass damaged neural pathways and restore functionality lost due to injury or disease. Approaches to decoding kinematic information are well documented; however, the decoding of kinetic information has received less attention. Additionally, the possibility of using stereo-electroencephalography (SEEG) for kinetic decoding during hand grasping tasks is still largely unknown. Thus, the objective of this paper is to demonstrate kinetic parameter decoding using SEEG in patients performing a grasping task with two different force levels under two different ascending rates. APPROACH Temporal-spectral representations were studied to investigate frequency modulation under different force tasks. Then, force amplitude was decoded from SEEG recordings using multiple decoders, including a linear model, a partial least squares (PLS) model, an unscented Kalman filter, and three deep learning models (shallow convolutional neural network, deep convolutional neural network and the proposed CNN+RNN neural network). MAIN RESULTS The current study showed that: 1) for some channel, both low-frequency modulation (event-related desynchronization (ERD)) and high-frequency modulation (event-related synchronization (ERS)) were sustained during prolonged force holding periods; 2) continuously changing grasp force can be decoded from the SEEG signals; 3) the novel CNN+RNN deep learning model achieved the best decoding performance, with the predicted force magnitude closely aligned to the ground truth under different force amplitudes and changing rates. SIGNIFICANCE This work verified the possibility of decoding continuously changing grasp force using SEEG recordings. The result presented in this study demonstrated the potential of SEEG recordings for future BCI application.
Collapse
Affiliation(s)
- Xiaolong Wu
- Electric, Electronic and Engineering, University of Bath, Pulteney Court PD42.2,Pulteney Road,BA2 4HL, Bath, Bath, Somerset, BA2 4HL, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Guangye Li
- Shanghai Jiao Tong University, Shanghai JiaoTong university, Shanghai, China, Shanghai, 200240, CHINA
| | - Shize Jiang
- Fudan University Huashan Hospital Department of Neurosurgery, Fudan University Huanshan hospital, Shanghai, Shanghai, 201906, CHINA
| | - Scott Wellington
- University of Bath, University of Bath, Bath, UK, Bath, Bath and North East Somer, BA2 7AY, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Shengjie Liu
- Shanghai Jiao Tong University, Shanghai jIaotong University, Shanghai, China, Shanghai, 200240, CHINA
| | - Zehan Wu
- Huashan Hospital Fudan University, Huashan hospital, Shanghai, China, Shanghai, Shanghai, 200040, CHINA
| | - Benjamin Metcalfe
- University of Bath, University of Bath, UK, Bath, Bath and North East Somer, BA2 7AY, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Liang Chen
- Fudan University Huashan Hospital Department of Neurosurgery, Fudan University Huashan Hospital, Shanghai, China, Shanghai, Shanghai, 201906, CHINA
| | - Dingguo Zhang
- University of Bath, University of Bath, UK, Bath, Somerset, BA2 4HL, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| |
Collapse
|
24
|
Pandarinath C, Bensmaia SJ. The science and engineering behind sensitized brain-controlled bionic hands. Physiol Rev 2022; 102:551-604. [PMID: 34541898 PMCID: PMC8742729 DOI: 10.1152/physrev.00034.2020] [Citation(s) in RCA: 22] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2020] [Revised: 09/07/2021] [Accepted: 09/13/2021] [Indexed: 12/13/2022] Open
Abstract
Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.
Collapse
Affiliation(s)
- Chethan Pandarinath
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, Georgia
- Department of Neurosurgery, Emory University, Atlanta, Georgia
| | - Sliman J Bensmaia
- Department of Organismal Biology and Anatomy, University of Chicago, Chicago, Illinois
- Committee on Computational Neuroscience, University of Chicago, Chicago, Illinois
- Grossman Institute for Neuroscience, Quantitative Biology, and Human Behavior, University of Chicago, Chicago, Illinois
| |
Collapse
|
25
|
Berezutskaya J, Vansteensel MJ, Aarnoutse EJ, Freudenburg ZV, Piantoni G, Branco MP, Ramsey NF. Open multimodal iEEG-fMRI dataset from naturalistic stimulation with a short audiovisual film. Sci Data 2022; 9:91. [PMID: 35314718 PMCID: PMC8938409 DOI: 10.1038/s41597-022-01173-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2021] [Accepted: 01/24/2022] [Indexed: 12/19/2022] Open
Abstract
Intracranial human recordings are a valuable and rare resource of information about the brain. Making such data publicly available not only helps tackle reproducibility issues in science, it helps make more use of these valuable data. This is especially true for data collected using naturalistic tasks. Here, we describe a dataset collected from a large group of human subjects while they watched a short audiovisual film. The dataset has several unique features. First, it includes a large amount of intracranial electroencephalography (iEEG) data (51 participants, age range of 5-55 years, who all performed the same task). Second, it includes functional magnetic resonance imaging (fMRI) recordings (30 participants, age range of 7-47) during the same task. Eighteen participants performed both iEEG and fMRI versions of the task, non-simultaneously. Third, the data were acquired using a rich audiovisual stimulus, for which we provide detailed speech and video annotations. This dataset can be used to study neural mechanisms of multimodal perception and language comprehension, and similarity of neural signals across brain recording modalities.
Collapse
Affiliation(s)
- Julia Berezutskaya
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands.
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands.
| | - Mariska J Vansteensel
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Erik J Aarnoutse
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Zachary V Freudenburg
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Giovanni Piantoni
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Mariana P Branco
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Nick F Ramsey
- Brain Center, Department of Neurology and Neurosurgery, University Medical Center Utrecht, Utrecht, the Netherlands
| |
Collapse
|
26
|
Ye H, Fan Z, Li G, Wu Z, Hu J, Sheng X, Chen L, Zhu X. Spontaneous State Detection Using Time-Frequency and Time-Domain Features Extracted From Stereo-Electroencephalography Traces. Front Neurosci 2022; 16:818214. [PMID: 35368269 PMCID: PMC8968069 DOI: 10.3389/fnins.2022.818214] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2021] [Accepted: 02/15/2022] [Indexed: 11/23/2022] Open
Abstract
As a minimally invasive recording technique, stereo-electroencephalography (SEEG) measures intracranial signals directly by inserting depth electrodes shafts into the human brain, and thus can capture neural activities in both cortical layers and subcortical structures. Despite gradually increasing SEEG-based brain-computer interface (BCI) studies, the features utilized were usually confined to the amplitude of the event-related potential (ERP) or band power, and the decoding capabilities of other time-frequency and time-domain features have not been demonstrated for SEEG recordings yet. In this study, we aimed to verify the validity of time-domain and time-frequency features of SEEG, where classification performances served as evaluating indicators. To do this, using SEEG signals under intermittent auditory stimuli, we extracted features including the average amplitude, root mean square, slope of linear regression, and line-length from the ERP trace and three traces of band power activities (high-gamma, beta, and alpha). These features were used to detect the active state (including activations to two types of names) against the idle state. Results suggested that valid time-domain and time-frequency features distributed across multiple regions, including the temporal lobe, parietal lobe, and deeper structures such as the insula. Among all feature types, the average amplitude, root mean square, and line-length extracted from high-gamma (60–140 Hz) power and the line-length extracted from ERP were the most informative. Using a hidden Markov model (HMM), we could precisely detect the onset and the end of the active state with a sensitivity of 95.7 ± 1.3% and a precision of 91.7 ± 1.6%. The valid features derived from high-gamma power and ERP in this work provided new insights into the feature selection procedure for further SEEG-based BCI applications.
Collapse
Affiliation(s)
- Huanpeng Ye
- State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Zhen Fan
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, China
| | - Guangye Li
- State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Zehan Wu
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, China
| | - Jie Hu
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, China
| | - Xinjun Sheng
- State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Liang Chen
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, China
- *Correspondence: Liang Chen
| | - Xiangyang Zhu
- State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China
- Xiangyang Zhu
| |
Collapse
|
27
|
Huggins JE, Krusienski D, Vansteensel MJ, Valeriani D, Thelen A, Stavisky S, Norton JJS, Nijholt A, Müller-Putz G, Kosmyna N, Korczowski L, Kapeller C, Herff C, Halder S, Guger C, Grosse-Wentrup M, Gaunt R, Dusang AN, Clisson P, Chavarriaga R, Anderson CW, Allison BZ, Aksenova T, Aarnoutse E. Workshops of the Eighth International Brain-Computer Interface Meeting: BCIs: The Next Frontier. BRAIN-COMPUTER INTERFACES 2022; 9:69-101. [PMID: 36908334 PMCID: PMC9997957 DOI: 10.1080/2326263x.2021.2009654] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2021] [Accepted: 11/15/2021] [Indexed: 12/11/2022]
Abstract
The Eighth International Brain-Computer Interface (BCI) Meeting was held June 7-9th, 2021 in a virtual format. The conference continued the BCI Meeting series' interactive nature with 21 workshops covering topics in BCI (also called brain-machine interface) research. As in the past, workshops covered the breadth of topics in BCI. Some workshops provided detailed examinations of specific methods, hardware, or processes. Others focused on specific BCI applications or user groups. Several workshops continued consensus building efforts designed to create BCI standards and increase the ease of comparisons between studies and the potential for meta-analysis and large multi-site clinical trials. Ethical and translational considerations were both the primary topic for some workshops or an important secondary consideration for others. The range of BCI applications continues to expand, with more workshops focusing on approaches that can extend beyond the needs of those with physical impairments. This paper summarizes each workshop, provides background information and references for further study, presents an overview of the discussion topics, and describes the conclusion, challenges, or initiatives that resulted from the interactions and discussion at the workshop.
Collapse
Affiliation(s)
- Jane E Huggins
- Department of Physical Medicine and Rehabilitation, Department of Biomedical Engineering, Neuroscience Graduate Program, University of Michigan, Ann Arbor, Michigan, United States 325 East Eisenhower, Room 3017; Ann Arbor, Michigan 48108-5744, 734-936-7177
| | - Dean Krusienski
- Department of Biomedical Engineering, Virginia Commonwealth University, Richmond, VA 23219
| | - Mariska J Vansteensel
- UMC Utrecht Brain Center, Dept of Neurosurgery, University Medical Center Utrecht, The Netherlands
| | | | - Antonia Thelen
- eemagine Medical Imaging Solutions GmbH, Berlin, Germany
| | | | - James J S Norton
- National Center for Adaptive Neurotechnologies, US Department of Veterans Affairs, 113 Holland Ave, Albany, NY 12208
| | - Anton Nijholt
- Faculty EEMCS, University of Twente, Enschede, The Netherlands
| | - Gernot Müller-Putz
- Institute of Neural Engineering, GrazBCI Lab, Graz University of Technology, Stremayrgasse 16/4, 8010 Graz, Austria
| | - Nataliya Kosmyna
- Massachusetts Institute of Technology (MIT), Media Lab, E14-548, Cambridge, MA 02139, Unites States
| | | | | | - Christian Herff
- School of Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands
| | | | - Christoph Guger
- g.tec medical engineering GmbH/Guger Technologies OG, Austria, Sierningstrasse 14, 4521 Schiedlberg, Austria, +43725122240-0
| | - Moritz Grosse-Wentrup
- Research Group Neuroinformatics, Faculty of Computer Science, Vienna Cognitive Science Hub, Data Science @ Uni Vienna University of Vienna
| | - Robert Gaunt
- Rehab Neural Engineering Labs, Department of Physical Medicine and Rehabilitation, Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA, 3520 5th Ave, Suite 300, Pittsburgh, PA 15213, 412-383-1426
| | - Aliceson Nicole Dusang
- Department of Electrical and Computer Engineering, School of Engineering, Brown University, Carney Institute for Brain Science, Brown University, Providence, RI
- Department of Veterans Affairs Medical Center, Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence, RI
- Center for Neurotechnology and Neurorecovery, Neurology, Massachusetts General Hospital, Boston, MA
| | | | - Ricardo Chavarriaga
- IEEE Standards Association Industry Connections group on neurotechnologies for brain-machine interface, Center for Artificial Intelligence, School of Engineering, ZHAW-Zurich University of Applied Sciences, Switzerland, Switzerland
| | - Charles W Anderson
- Department of Computer Science, Molecular, Cellular and Integrative Neurosience Program, Colorado State University, Fort Collins, CO 80523
| | - Brendan Z Allison
- Dept. of Cognitive Science, Mail Code 0515, University of California at San Diego, La Jolla, United States, 619-534-9754
| | - Tetiana Aksenova
- University Grenoble Alpes, CEA, LETI, Clinatec, Grenoble 38000, France
| | - Erik Aarnoutse
- UMC Utrecht Brain Center, Department of Neurology & Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
| |
Collapse
|
28
|
Li G, Jiang S, Meng J, Chai G, Wu Z, Fan Z, Hu J, Sheng X, Zhang D, Chen L, Zhu X. Assessing differential representation of hand movements in multiple domains using stereo-electroencephalographic recordings. Neuroimage 2022; 250:118969. [DOI: 10.1016/j.neuroimage.2022.118969] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2021] [Revised: 01/28/2022] [Accepted: 02/02/2022] [Indexed: 01/03/2023] Open
|
29
|
Cherian R, Kanaga EG. Theoretical and Methodological Analysis of EEG based Seizure Detection and Prediction: An Exhaustive Review. J Neurosci Methods 2022; 369:109483. [DOI: 10.1016/j.jneumeth.2022.109483] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2021] [Revised: 01/13/2022] [Accepted: 01/13/2022] [Indexed: 02/07/2023]
|
30
|
Luo S, Rabbani Q, Crone NE. Brain-Computer Interface: Applications to Speech Decoding and Synthesis to Augment Communication. Neurotherapeutics 2022; 19:263-273. [PMID: 35099768 PMCID: PMC9130409 DOI: 10.1007/s13311-022-01190-2] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/16/2022] [Indexed: 01/03/2023] Open
Abstract
Damage or degeneration of motor pathways necessary for speech and other movements, as in brainstem strokes or amyotrophic lateral sclerosis (ALS), can interfere with efficient communication without affecting brain structures responsible for language or cognition. In the worst-case scenario, this can result in the locked in syndrome (LIS), a condition in which individuals cannot initiate communication and can only express themselves by answering yes/no questions with eye blinks or other rudimentary movements. Existing augmentative and alternative communication (AAC) devices that rely on eye tracking can improve the quality of life for people with this condition, but brain-computer interfaces (BCIs) are also increasingly being investigated as AAC devices, particularly when eye tracking is too slow or unreliable. Moreover, with recent and ongoing advances in machine learning and neural recording technologies, BCIs may offer the only means to go beyond cursor control and text generation on a computer, to allow real-time synthesis of speech, which would arguably offer the most efficient and expressive channel for communication. The potential for BCI speech synthesis has only recently been realized because of seminal studies of the neuroanatomical and neurophysiological underpinnings of speech production using intracranial electrocorticographic (ECoG) recordings in patients undergoing epilepsy surgery. These studies have shown that cortical areas responsible for vocalization and articulation are distributed over a large area of ventral sensorimotor cortex, and that it is possible to decode speech and reconstruct its acoustics from ECoG if these areas are recorded with sufficiently dense and comprehensive electrode arrays. In this article, we review these advances, including the latest neural decoding strategies that range from deep learning models to the direct concatenation of speech units. We also discuss state-of-the-art vocoders that are integral in constructing natural-sounding audio waveforms for speech BCIs. Finally, this review outlines some of the challenges ahead in directly synthesizing speech for patients with LIS.
Collapse
Affiliation(s)
- Shiyu Luo
- Department of Biomedical Engineering, The Johns Hopkins University School of Medicine, Baltimore, MD, USA.
| | - Qinwan Rabbani
- Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD, USA
| | - Nathan E Crone
- Department of Neurology, The Johns Hopkins University School of Medicine, Baltimore, MD, USA
| |
Collapse
|
31
|
Angrick M, Ottenhoff M, Goulis S, Colon AJ, Wagner L, Krusienski DJ, Kubben PL, Schultz T, Herff C. Speech Synthesis from Stereotactic EEG using an Electrode Shaft Dependent Multi-Input Convolutional Neural Network Approach. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:6045-6048. [PMID: 34892495 DOI: 10.1109/embc46164.2021.9629711] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Neurological disorders can lead to significant impairments in speech communication and, in severe cases, cause the complete loss of the ability to speak. Brain-Computer Interfaces have shown promise as an alternative communication modality by directly transforming neural activity of speech processes into a textual or audible representations. Previous studies investigating such speech neuroprostheses relied on electrocorticography (ECoG) or microelectrode arrays that acquire neural signals from superficial areas on the cortex. While both measurement methods have demonstrated successful speech decoding, they do not capture activity from deeper brain structures and this activity has therefore not been harnessed for speech-related BCIs. In this study, we bridge this gap by adapting a previously presented decoding pipeline for speech synthesis based on ECoG signals to implanted depth electrodes (sEEG). For this purpose, we propose a multi-input convolutional neural network that extracts speech-related activity separately for each electrode shaft and estimates spectral coefficients to reconstruct an audible waveform. We evaluate our approach on open-loop data from 5 patients who conducted a recitation task of Dutch utterances. We achieve correlations of up to 0.80 between original and reconstructed speech spectrograms, which are significantly above chance level for all patients (p < 0.001). Our results indicate that sEEG can yield similar speech decoding performance to prior ECoG studies and is a promising modality for speech BCIs.
Collapse
|
32
|
Ottenhoff MC, Goulis S, Wagner L, Tousseyn S, Colon A, Kubben P, Herff C. Continuously Decoding Grasping Movements using Stereotactic Depth Electrodes. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:6098-6101. [PMID: 34892508 DOI: 10.1109/embc46164.2021.9629639] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Brain-Computer Interfaces (BCIs) that decode a patient's movement intention to control a prosthetic device could restore some independence to paralyzed patients. An important step on the road towards naturalistic prosthetic control is to decode movement continuously with low-latency. BCIs based on intracortical micro-arrays provide continuous control of robotic arms, but require a minor craniotomy. Surface recordings of neural activity using EEG have made great advances over the last years, but suffer from high noise levels and large intra-session variance. Here, we investigate the use of minimally invasive recordings using stereotactically implanted EEG (sEEG). These electrodes provide a sparse sampling across many brain regions. So far, promising decoding results have been presented using data measured from the subthalamic nucleus or trial-to-trial based methods using depth electrodes. In this work, we demonstrate that grasping movements can continuously be decoded using sEEG electrodes, as well. Beta and high-gamma activity was extracted from eight participants performing a grasping task. We demonstrate above chance level decoding of movement vs rest and left vs right, from both frequency bands with accuracies up to 0.94 AUC. The vastly different electrode locations between participants lead to large variability. In the future, we hope that sEEG recordings will provide additional information for the decoding process in neuroprostheses.
Collapse
|
33
|
Liu S, Li G, Jiang S, Wu X, Hu J, Zhang D, Chen L. Investigating Data Cleaning Methods to Improve Performance of Brain-Computer Interfaces Based on Stereo-Electroencephalography. Front Neurosci 2021; 15:725384. [PMID: 34690673 PMCID: PMC8528199 DOI: 10.3389/fnins.2021.725384] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2021] [Accepted: 09/01/2021] [Indexed: 11/13/2022] Open
Abstract
Stereo-electroencephalography (SEEG) utilizes localized and penetrating depth electrodes to directly measure electrophysiological brain activity. The implanted electrodes generally provide a sparse sampling of multiple brain regions, including both cortical and subcortical structures, making the SEEG neural recordings a potential source for the brain–computer interface (BCI) purpose in recent years. For SEEG signals, data cleaning is an essential preprocessing step in removing excessive noises for further analysis. However, little is known about what kinds of effect that different data cleaning methods may exert on BCI decoding performance and, moreover, what are the reasons causing the differentiated effects. To address these questions, we adopted five different data cleaning methods, including common average reference, gray–white matter reference, electrode shaft reference, bipolar reference, and Laplacian reference, to process the SEEG data and evaluated the effect of these methods on improving BCI decoding performance. Additionally, we also comparatively investigated the changes of SEEG signals induced by these different methods from multiple-domain (e.g., spatial, spectral, and temporal domain). The results showed that data cleaning methods could improve the accuracy of gesture decoding, where the Laplacian reference produced the best performance. Further analysis revealed that the superiority of the data cleaning method with excellent performance might be attributed to the increased distinguishability in the low-frequency band. The findings of this work highlighted the importance of applying proper data clean methods for SEEG signals and proposed the application of Laplacian reference for SEEG-based BCI.
Collapse
Affiliation(s)
- Shengjie Liu
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, China
| | - Guangye Li
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, China
| | - Shize Jiang
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, China
| | - Xiaolong Wu
- Department of Electronic and Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Jie Hu
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, China
| | - Dingguo Zhang
- Department of Electronic and Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Liang Chen
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, China
| |
Collapse
|
34
|
Cometa A, D'Orio P, Revay M, Micera S, Artoni F. Stimulus evoked causality estimation in stereo-EEG. J Neural Eng 2021; 18. [PMID: 34534968 DOI: 10.1088/1741-2552/ac27fb] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2021] [Accepted: 09/17/2021] [Indexed: 11/11/2022]
Abstract
Objective.Stereo-electroencephalography (SEEG) has recently gained importance in analyzing brain functions. Its high temporal resolution and spatial specificity make it a powerful tool to investigate the strength, direction, and spectral content of brain networks interactions, especially when these connections are stimulus-evoked. However, choosing the best approach to evaluate the flow of information is not trivial, due to the lack of validated methods explicitly designed for SEEG.Approach.We propose a novel non-parametric statistical test for event-related causality (ERC) assessment on SEEG recordings. Here, we refer to the ERC as the causality evoked by a particular part of the stimulus (a response window (RW)). We also present a data surrogation method to evaluate the performance of a causality estimation algorithm. We finally validated our pipeline using surrogate SEEG data derived from an experimentally collected dataset, and compared the most used and successful measures to estimate effective connectivity, belonging to the Geweke-Granger causality framework.Main results.Here we show that our workflow correctly identified all the directed connections in the RW of the surrogate data and prove the robustness of the procedure against synthetic noise with amplitude exceeding physiological-plausible values. Among the causality measures tested, partial directed coherence performed best.Significance.This is the first non-parametric statistical test for ERC estimation explicitly designed for SEEG datasets. The pipeline, in principle, can also be applied to the analysis of any type of time-varying estimator, if there exists a clearly defined RW.
Collapse
Affiliation(s)
- Andrea Cometa
- BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Viale Rinaldo Piaggio, 34, Pontedera, 56025, Italy
| | - Piergiorgio D'Orio
- 'Claudio Munari' Center for Epilepsy Surgery, ASST GOM Niguarda Hospital, Piazza dell'Ospedale Maggiore, 3, 20162 Milano, Italy.,Institute of Neuroscience, CNR, via Volturno 39E, Parma 43125, Italy
| | - Martina Revay
- 'Claudio Munari' Center for Epilepsy Surgery, ASST GOM Niguarda Hospital, Piazza dell'Ospedale Maggiore, 3, 20162 Milano, Italy.,Department of Biomedical and Clinical Sciences "L. Sacco", Università degli Studi di Milano, Via Giovanni Battista Grassi 74, Milan 20157, Italy
| | - Silvestro Micera
- BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Viale Rinaldo Piaggio, 34, Pontedera, 56025, Italy.,Ecole Polytechnique Federale de Lausanne, Bertarelli Foundation Chair in Translational NeuroEngineering, Center for Neuroprosthetics and School of Engineering, Chemin des Mines, 9, Geneva, GE CH 1202, Switzerland
| | - Fiorenzo Artoni
- BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Viale Rinaldo Piaggio, 34, Pontedera, 56025, Italy.,Ecole Polytechnique Federale de Lausanne, Bertarelli Foundation Chair in Translational NeuroEngineering, Center for Neuroprosthetics and School of Engineering, Chemin des Mines, 9, Geneva, GE CH 1202, Switzerland
| |
Collapse
|
35
|
Zhu B, Shin U, Shoaran M. Closed-Loop Neural Prostheses With On-Chip Intelligence: A Review and a Low-Latency Machine Learning Model for Brain State Detection. IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS 2021; 15:877-897. [PMID: 34529573 PMCID: PMC8733782 DOI: 10.1109/tbcas.2021.3112756] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/22/2023]
Abstract
The application of closed-loop approaches in systems neuroscience and therapeutic stimulation holds great promise for revolutionizing our understanding of the brain and for developing novel neuromodulation therapies to restore lost functions. Neural prostheses capable of multi-channel neural recording, on-site signal processing, rapid symptom detection, and closed-loop stimulation are critical to enabling such novel treatments. However, the existing closed-loop neuromodulation devices are too simplistic and lack sufficient on-chip processing and intelligence. In this paper, we first discuss both commercial and investigational closed-loop neuromodulation devices for brain disorders. Next, we review state-of-the-art neural prostheses with on-chip machine learning, focusing on application-specific integrated circuits (ASIC). System requirements, performance and hardware comparisons, design trade-offs, and hardware optimization techniques are discussed. To facilitate a fair comparison and guide design choices among various on-chip classifiers, we propose a new energy-area (E-A) efficiency figure of merit that evaluates hardware efficiency and multi-channel scalability. Finally, we present several techniques to improve the key design metrics of tree-based on-chip classifiers, both in the context of ensemble methods and oblique structures. A novel Depth-Variant Tree Ensemble (DVTE) is proposed to reduce processing latency (e.g., by 2.5× on seizure detection task). We further develop a cost-aware learning approach to jointly optimize the power and latency metrics. We show that algorithm-hardware co-design enables the energy- and memory-optimized design of tree-based models, while preserving a high accuracy and low latency. Furthermore, we show that our proposed tree-based models feature a highly interpretable decision process that is essential for safety-critical applications such as closed-loop stimulation.
Collapse
|
36
|
Martinek R, Ladrova M, Sidikova M, Jaros R, Behbehani K, Kahankova R, Kawala-Sterniuk A. Advanced Bioelectrical Signal Processing Methods: Past, Present and Future Approach-Part II: Brain Signals. SENSORS (BASEL, SWITZERLAND) 2021; 21:6343. [PMID: 34640663 PMCID: PMC8512967 DOI: 10.3390/s21196343] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/19/2021] [Revised: 09/12/2021] [Accepted: 09/14/2021] [Indexed: 12/14/2022]
Abstract
As it was mentioned in the previous part of this work (Part I)-the advanced signal processing methods are one of the quickest and the most dynamically developing scientific areas of biomedical engineering with their increasing usage in current clinical practice. In this paper, which is a Part II work-various innovative methods for the analysis of brain bioelectrical signals were presented and compared. It also describes both classical and advanced approaches for noise contamination removal such as among the others digital adaptive and non-adaptive filtering, signal decomposition methods based on blind source separation, and wavelet transform.
Collapse
Affiliation(s)
- Radek Martinek
- Department of Cybernetics and Biomedical Engineering, VSB-Technical University Ostrava—FEECS, 708 00 Ostrava-Poruba, Czech Republic; (M.L.); (M.S.); (R.J.); (R.K.)
| | - Martina Ladrova
- Department of Cybernetics and Biomedical Engineering, VSB-Technical University Ostrava—FEECS, 708 00 Ostrava-Poruba, Czech Republic; (M.L.); (M.S.); (R.J.); (R.K.)
| | - Michaela Sidikova
- Department of Cybernetics and Biomedical Engineering, VSB-Technical University Ostrava—FEECS, 708 00 Ostrava-Poruba, Czech Republic; (M.L.); (M.S.); (R.J.); (R.K.)
| | - Rene Jaros
- Department of Cybernetics and Biomedical Engineering, VSB-Technical University Ostrava—FEECS, 708 00 Ostrava-Poruba, Czech Republic; (M.L.); (M.S.); (R.J.); (R.K.)
| | - Khosrow Behbehani
- College of Engineering, The University of Texas in Arlington, Arlington, TX 76019, USA;
| | - Radana Kahankova
- Department of Cybernetics and Biomedical Engineering, VSB-Technical University Ostrava—FEECS, 708 00 Ostrava-Poruba, Czech Republic; (M.L.); (M.S.); (R.J.); (R.K.)
| | - Aleksandra Kawala-Sterniuk
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, 45-758 Opole, Poland
| |
Collapse
|
37
|
Real-time synthesis of imagined speech processes from minimally invasive recordings of neural activity. Commun Biol 2021; 4:1055. [PMID: 34556793 PMCID: PMC8460739 DOI: 10.1038/s42003-021-02578-0] [Citation(s) in RCA: 30] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2021] [Accepted: 08/11/2021] [Indexed: 11/17/2022] Open
Abstract
Speech neuroprosthetics aim to provide a natural communication channel to individuals who are unable to speak due to physical or neurological impairments. Real-time synthesis of acoustic speech directly from measured neural activity could enable natural conversations and notably improve quality of life, particularly for individuals who have severely limited means of communication. Recent advances in decoding approaches have led to high quality reconstructions of acoustic speech from invasively measured neural activity. However, most prior research utilizes data collected during open-loop experiments of articulated speech, which might not directly translate to imagined speech processes. Here, we present an approach that synthesizes audible speech in real-time for both imagined and whispered speech conditions. Using a participant implanted with stereotactic depth electrodes, we were able to reliably generate audible speech in real-time. The decoding models rely predominately on frontal activity suggesting that speech processes have similar representations when vocalized, whispered, or imagined. While reconstructed audio is not yet intelligible, our real-time synthesis approach represents an essential step towards investigating how patients will learn to operate a closed-loop speech neuroprosthesis based on imagined speech. Miguel Angrick et al. develop an intracranial EEG-based method to decode imagined speech from a human patient and translate it into audible speech in real-time. This report presents an important proof of concept that acoustic output can be reconstructed on the basis of neural signals, and serves as a valuable step in the development of neuroprostheses to help nonverbal patients interact with their environment.
Collapse
|
38
|
Li G, Jiang S, Paraskevopoulou SE, Chai G, Wei Z, Liu S, Wang M, Xu Y, Fan Z, Wu Z, Chen L, Zhang D, Zhu X. Detection of human white matter activation and evaluation of its function in movement decoding using stereo-electroencephalography (SEEG). J Neural Eng 2021; 18. [PMID: 34284361 DOI: 10.1088/1741-2552/ac160e] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2021] [Accepted: 07/20/2021] [Indexed: 11/11/2022]
Abstract
Objective. White matter tissue takes up approximately 50% of the human brain volume and it is widely known as a messenger conducting information between areas of the central nervous system. However, the characteristics of white matter neural activity and whether white matter neural recordings can contribute to movement decoding are often ignored and still remain largely unknown. In this work, we make quantitative analyses to investigate these two important questions using invasive neural recordings.Approach. We recorded stereo-electroencephalography (SEEG) data from 32 human subjects during a visually-cued motor task, where SEEG recordings can tap into gray and white matter electrical activity simultaneously. Using the proximal tissue density method, we identified the location (i.e. gray or white matter) of each SEEG contact. Focusing on alpha oscillatory and high gamma activities, we compared the activation patterns between gray matter and white matter. Then, we evaluated the performance of such white matter activation in movement decoding.Main results. The results show that white matter also presents activation under the task, in a similar way with the gray matter but at a significantly lower amplitude. Additionally, this work also demonstrates that combing white matter neural activities together with that of gray matter significantly promotes the movement decoding accuracy than using gray matter signals only.Significance. Taking advantage of SEEG recordings from a large number of subjects, we reveal the response characteristics of white matter neural signals under the task and demonstrate its enhancing function in movement decoding. This study highlights the importance of taking white matter activities into consideration in further scientific research and translational applications.
Collapse
Affiliation(s)
- Guangye Li
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China.,These authors contributed to this paper equally and should be considered as co-first authors
| | - Shize Jiang
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, People's Republic of China.,These authors contributed to this paper equally and should be considered as co-first authors
| | - Sivylla E Paraskevopoulou
- National Center for Adaptive Neurotechnologies, Wadsworth Center, New York State Department of Health, Albany, NY, United States of America.,These authors contributed to this paper equally and should be considered as co-first authors
| | - Guohong Chai
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Zixuan Wei
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, People's Republic of China
| | - Shengjie Liu
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Meng Wang
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Yang Xu
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Zhen Fan
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, People's Republic of China
| | - Zehan Wu
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, People's Republic of China
| | - Liang Chen
- Department of Neurosurgery of Huashan Hospital, Fudan University, Shanghai, People's Republic of China
| | - Dingguo Zhang
- Department of Electronic and Electrical Engineering, University of Bath, Bath, United Kingdom
| | - Xiangyang Zhu
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| |
Collapse
|
39
|
Chandrasekaran S, Bickel S, Herrero JL, Kim JW, Markowitz N, Espinal E, Bhagat NA, Ramdeo R, Xu J, Glasser MF, Bouton CE, Mehta AD. Evoking highly focal percepts in the fingertips through targeted stimulation of sulcal regions of the brain for sensory restoration. Brain Stimul 2021; 14:1184-1196. [PMID: 34358704 PMCID: PMC8884403 DOI: 10.1016/j.brs.2021.07.009] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2021] [Revised: 06/29/2021] [Accepted: 07/19/2021] [Indexed: 01/06/2023] Open
Abstract
Background: Paralysis and neuropathy, affecting millions of people worldwide, can be accompanied by significant loss of somatosensation. With tactile sensation being central to achieving dexterous movement, brain-computer interface (BCI) researchers have used intracortical and cortical surface electrical stimulation to restore somatotopically-relevant sensation to the hand. However, these approaches are restricted to stimulating the gyral areas of the brain. Since representation of distal regions of the hand extends into the sulcal regions of human primary somatosensory cortex (S1), it has been challenging to evoke sensory percepts localized to the fingertips. Objective/hypothesis: Targeted stimulation of sulcal regions of S1, using stereoelectroencephalography (SEEG) depth electrodes, can evoke focal sensory percepts in the fingertips. Methods: Two participants with intractable epilepsy received cortical stimulation both at the gyri via high-density electrocorticography (HD-ECoG) grids and in the sulci via SEEG depth electrode leads. We characterized the evoked sensory percepts localized to the hand. Results: We show that highly focal percepts can be evoked in the fingertips of the hand through sulcal stimulation. fMRI, myelin content, and cortical thickness maps from the Human Connectome Project elucidated specific cortical areas and sub-regions within S1 that evoked these focal percepts. Within-participant comparisons showed that percepts evoked by sulcal stimulation via SEEG electrodes were significantly more focal (80% less area; p = 0.02) and localized to the fingertips more often, than by gyral stimulation via HD-ECoG electrodes. Finally, sulcal locations with consistent modulation of high-frequency neural activity during mechanical tactile stimulation of the fingertips showed the same somatotopic correspondence as cortical stimulation. Conclusions: Our findings indicate minimally invasive sulcal stimulation via SEEG electrodes could be a clinically viable approach to restoring sensation.
Collapse
Affiliation(s)
- Santosh Chandrasekaran
- Neural Bypass and Brain Computer Interface Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA.
| | - Stephan Bickel
- The Human Brain Mapping Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA; Department of Neurosurgery, Northwell, Manhasset, NY, USA; Department of Neurology, Donald and Barbara Zucker School of Medicine at Hofstra, Northwell, Manhasset, NY, USA
| | - Jose L Herrero
- The Human Brain Mapping Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA; Department of Neurosurgery, Northwell, Manhasset, NY, USA
| | - Joo-Won Kim
- Departments of Radiology and Psychiatry, Baylor College of Medicine, Houston, TX, USA
| | - Noah Markowitz
- The Human Brain Mapping Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
| | - Elizabeth Espinal
- The Human Brain Mapping Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
| | - Nikunj A Bhagat
- Neural Bypass and Brain Computer Interface Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
| | - Richard Ramdeo
- Neural Bypass and Brain Computer Interface Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
| | - Junqian Xu
- Departments of Radiology and Psychiatry, Baylor College of Medicine, Houston, TX, USA
| | - Matthew F Glasser
- Departments of Radiology and Neuroscience, Washington University in St Louis, Saint Louis, MO, USA
| | - Chad E Bouton
- Neural Bypass and Brain Computer Interface Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA; Department of Molecular Medicine, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Manhasset, NY, USA.
| | - Ashesh D Mehta
- The Human Brain Mapping Laboratory, Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA; Department of Neurosurgery, Northwell, Manhasset, NY, USA
| |
Collapse
|
40
|
Orlandi S, House SC, Karlsson P, Saab R, Chau T. Brain-Computer Interfaces for Children With Complex Communication Needs and Limited Mobility: A Systematic Review. Front Hum Neurosci 2021; 15:643294. [PMID: 34335203 PMCID: PMC8319030 DOI: 10.3389/fnhum.2021.643294] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2020] [Accepted: 05/18/2021] [Indexed: 11/13/2022] Open
Abstract
Brain-computer interfaces (BCIs) represent a new frontier in the effort to maximize the ability of individuals with profound motor impairments to interact and communicate. While much literature points to BCIs' promise as an alternative access pathway, there have historically been few applications involving children and young adults with severe physical disabilities. As research is emerging in this sphere, this article aims to evaluate the current state of translating BCIs to the pediatric population. A systematic review was conducted using the Scopus, PubMed, and Ovid Medline databases. Studies of children and adolescents that reported BCI performance published in English in peer-reviewed journals between 2008 and May 2020 were included. Twelve publications were identified, providing strong evidence for continued research in pediatric BCIs. Research evidence was generally at multiple case study or exploratory study level, with modest sample sizes. Seven studies focused on BCIs for communication and five on mobility. Articles were categorized and grouped based on type of measurement (i.e., non-invasive and invasive), and the type of brain signal (i.e., sensory evoked potentials or movement-related potentials). Strengths and limitations of studies were identified and used to provide requirements for clinical translation of pediatric BCIs. This systematic review presents the state-of-the-art of pediatric BCIs focused on developing advanced technology to support children and youth with communication disabilities or limited manual ability. Despite a few research studies addressing the application of BCIs for communication and mobility in children, results are encouraging and future works should focus on customizable pediatric access technologies based on brain activity.
Collapse
Affiliation(s)
- Silvia Orlandi
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
| | - Sarah C. House
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
| | - Petra Karlsson
- Cerebral Palsy Alliance, The University of Sydney, Sydney, NSW, Australia
| | - Rami Saab
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
| | - Tom Chau
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
- Institute of Biomedical Engineering (BME), University of Toronto, Toronto, ON, Canada
| |
Collapse
|
41
|
Sheth J, Tankus A, Tran M, Pouratian N, Fried I, Speier W. Generalizing neural signal-to-text brain-computer interfaces. Biomed Phys Eng Express 2021; 7. [PMID: 33836507 DOI: 10.1088/2057-1976/abf6ab] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2020] [Accepted: 04/09/2021] [Indexed: 11/12/2022]
Abstract
Objective:Brain-Computer Interfaces (BCI) may help patients with faltering communication abilities due to neurodegenerative diseases produce text or speech by direct neural processing. However, their practical realization has proven difficult due to limitations in speed, accuracy, and generalizability of existing interfaces. The goal of this study is to evaluate the BCI performance of a robust speech decoding system that translates neural signals evoked by speech to a textual output. While previous studies have approached this problem by using neural signals to choose from a limited set of possible words, we employ a more general model that can type any word from a large corpus of English text.Approach:In this study, we create an end-to-end BCI that translates neural signals associated with overt speech into text output. Our decoding system first isolates frequency bands in the input depth-electrode signal encapsulating differential information regarding production of various phonemic classes. These bands form a feature set that then feeds into a Long Short-Term Memory (LSTM) model which discerns at each time point probability distributions across all phonemes uttered by a subject. Finally, a particle filtering algorithm temporally smooths these probabilities by incorporating prior knowledge of the English language to output text corresponding to the decoded word. The generalizability of our decoder is driven by the lack of a vocabulary constraint on this output word.Main result:This method was evaluated using a dataset of 6 neurosurgical patients implanted with intra-cranial depth electrodes to identify seizure foci for potential surgical treatment of epilepsy. We averaged 32% word accuracy and on the phoneme-level obtained 46% precision, 51% recall and 73.32% average phoneme error rate while also achieving significant increases in speed when compared to several other BCI approaches.Significance:Our study employs a more general neural signal-to-text model which could facilitate communication by patients in everyday environments.
Collapse
Affiliation(s)
- Janaki Sheth
- Department of Physics and Astronomy, UCLA, Los Angeles, CA, United States of America
| | - Ariel Tankus
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel.,Functional Neurosurgery Unit, Tel Aviv, Sourasky Medical Center, Tel Aviv, Israel.,Department of Neurology and Neurosurgery, Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel
| | - Michelle Tran
- Department of Neurosurgery, UCLA, Los Angeles, CA, United States of America
| | - Nader Pouratian
- Department of Neurosurgery, UCLA, Los Angeles, CA, United States of America
| | - Itzhak Fried
- Department of Neurosurgery, UCLA, Los Angeles, CA, United States of America
| | - William Speier
- Department of Radiology, UCLA, Los Angeles, CA, United States of America
| |
Collapse
|
42
|
Patient-specific prediction of SEEG electrode bending for stereotactic neurosurgical planning. Int J Comput Assist Radiol Surg 2021; 16:789-798. [PMID: 33761063 PMCID: PMC8134306 DOI: 10.1007/s11548-021-02347-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2021] [Accepted: 03/05/2021] [Indexed: 12/01/2022]
Abstract
Purpose Electrode bending observed after stereotactic interventions is typically not accounted for in either computer-assisted planning algorithms, where straight trajectories are assumed, or in quality assessment, where only metrics related to entry and target points are reported. Our aim is to provide a fully automated and validated pipeline for the prediction of stereo-electroencephalography (SEEG) electrode bending. Methods We transform electrodes of 86 cases into a common space and compare features-based and image-based neural networks on their ability to regress local displacement (\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\mathbf{lu} $$\end{document}lu) or electrode bending (\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\hat{\mathbf{eb }}$$\end{document}eb^). Electrodes were stratified into six groups based on brain structures at the entry and target point. Models, both with and without Monte Carlo (MC) dropout, were trained and validated using tenfold cross-validation. Results mage-based models outperformed features-based models for all groups, and models that predicted \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\mathbf{lu} $$\end{document}lu performed better than for \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\hat{\mathbf{eb }}$$\end{document}eb^. Image-based model prediction with MC dropout resulted in lower mean squared error (MSE) with improvements up to 12.9% (\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\mathbf{lu} $$\end{document}lu) and 39.9% (\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\hat{\mathbf{eb }}$$\end{document}eb^), compared to no dropout. Using an image of brain tissue types (cortex, white and deep grey matter) resulted in similar, and sometimes better performance, compared to using a T1-weighted MRI when predicting \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\mathbf{lu} $$\end{document}lu. When inferring trajectories of image-based models (brain tissue types), 86.9% of trajectories had an MSE\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\le 1$$\end{document}≤1 mm. Conclusion An image-based approach regressing local displacement with an image of brain tissue types resulted in more accurate electrode bending predictions compared to other approaches, inputs, and outputs. Future work will investigate the integration of electrode bending into planning and quality assessment algorithms. Supplementary Information The online version supplementary material available at 10.1007/s11548-021-02347-8.
Collapse
|
43
|
Peterson SM, Steine-Hanson Z, Davis N, Rao RPN, Brunton BW. Generalized neural decoders for transfer learning across participants and recording modalities. J Neural Eng 2021; 18. [PMID: 33418552 DOI: 10.1088/1741-2552/abda0b] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2020] [Accepted: 01/08/2021] [Indexed: 02/07/2023]
Abstract
OBJECTIVE Advances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants. APPROACH We introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (1) a Hilbert transform that computes spectral power at data-driven frequencies and (2) a layer that projects electrode-level data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant. MAIN RESULTS HTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By fine-tuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet's trained weights and demonstrate its ability to extract physiologically-relevant features. SIGNIFICANCE By generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participant-specific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.
Collapse
Affiliation(s)
- Steven M Peterson
- Biology, University of Washington, 4000 15th Ave NE, Seattle, Washington, 98195, UNITED STATES
| | - Zoe Steine-Hanson
- Computer Science and Engineering, University of Washington, 4000 15th Ave NE, Seattle, Washington, 98195, UNITED STATES
| | - Nathan Davis
- Computer Science and Engineering, University of Washington, 4000 15th Ave NE, Seattle, Washington, 98195, UNITED STATES
| | - Rajesh P N Rao
- Computer Science and Engineering, University of Washington, 185 E Stevens Way NE, Seattle, Washington, 98195, UNITED STATES
| | - Bingni W Brunton
- Biology, University of Washington, 4000 15th Ave NE, Seattle, Washington, 98195, UNITED STATES
| |
Collapse
|
44
|
Kawala-Sterniuk A, Browarska N, Al-Bakri A, Pelc M, Zygarlicki J, Sidikova M, Martinek R, Gorzelanczyk EJ. Summary of over Fifty Years with Brain-Computer Interfaces-A Review. Brain Sci 2021; 11:43. [PMID: 33401571 PMCID: PMC7824107 DOI: 10.3390/brainsci11010043] [Citation(s) in RCA: 47] [Impact Index Per Article: 15.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2020] [Revised: 12/25/2020] [Accepted: 12/27/2020] [Indexed: 11/16/2022] Open
Abstract
Over the last few decades, the Brain-Computer Interfaces have been gradually making their way to the epicenter of scientific interest. Many scientists from all around the world have contributed to the state of the art in this scientific domain by developing numerous tools and methods for brain signal acquisition and processing. Such a spectacular progress would not be achievable without accompanying technological development to equip the researchers with the proper devices providing what is absolutely necessary for any kind of discovery as the core of every analysis: the data reflecting the brain activity. The common effort has resulted in pushing the whole domain to the point where the communication between a human being and the external world through BCI interfaces is no longer science fiction but nowadays reality. In this work we present the most relevant aspects of the BCIs and all the milestones that have been made over nearly 50-year history of this research domain. We mention people who were pioneers in this area as well as we highlight all the technological and methodological advances that have transformed something available and understandable by a very few into something that has a potential to be a breathtaking change for so many. Aiming to fully understand how the human brain works is a very ambitious goal and it will surely take time to succeed. However, even that fraction of what has already been determined is sufficient e.g., to allow impaired people to regain control on their lives and significantly improve its quality. The more is discovered in this domain, the more benefit for all of us this can potentially bring.
Collapse
Affiliation(s)
- Aleksandra Kawala-Sterniuk
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, 45-758 Opole, Poland; (N.B.); (M.P.); (J.Z.)
| | - Natalia Browarska
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, 45-758 Opole, Poland; (N.B.); (M.P.); (J.Z.)
| | - Amir Al-Bakri
- Department of Biomedical Engineering, College of Engineering, University of Babylon, 51001 Babylon, Iraq;
| | - Mariusz Pelc
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, 45-758 Opole, Poland; (N.B.); (M.P.); (J.Z.)
- Department of Computing and Information Systems, University of Greenwich, London SE10 9LS, UK
| | - Jaroslaw Zygarlicki
- Faculty of Electrical Engineering, Automatic Control and Informatics, Opole University of Technology, 45-758 Opole, Poland; (N.B.); (M.P.); (J.Z.)
| | - Michaela Sidikova
- Department of Cybernetics and Biomedical Engineering, VSB-Technical University Ostrava—FEECS, 708 00 Ostrava-Poruba, Czech Republic; (M.S.); (R.M.)
| | - Radek Martinek
- Department of Cybernetics and Biomedical Engineering, VSB-Technical University Ostrava—FEECS, 708 00 Ostrava-Poruba, Czech Republic; (M.S.); (R.M.)
| | - Edward Jacek Gorzelanczyk
- Department of Theoretical Basis of BioMedical Sciences and Medical Informatics, Nicolaus Copernicus University, Collegium Medicum, 85-067 Bydgoszcz, Poland;
- Institute of Philosophy, Kazimierz Wielki University, 85-092 Bydgoszcz, Poland
- Babinski Specialist Psychiatric Healthcare Center, Outpatient Addiction Treatment, 91-229 Lodz, Poland
- The Society for the Substitution Treatment of Addiction “Medically Assisted Recovery”, 85-791 Bydgoszcz, Poland
| |
Collapse
|
45
|
Wilson GH, Stavisky SD, Willett FR, Avansino DT, Kelemen JN, Hochberg LR, Henderson JM, Druckmann S, Shenoy KV. Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus. J Neural Eng 2020; 17:066007. [PMID: 33236720 PMCID: PMC8293867 DOI: 10.1088/1741-2552/abbfef] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
Abstract
OBJECTIVE To evaluate the potential of intracortical electrode array signals for brain-computer interfaces (BCIs) to restore lost speech, we measured the performance of decoders trained to discriminate a comprehensive basis set of 39 English phonemes and to synthesize speech sounds via a neural pattern matching method. We decoded neural correlates of spoken-out-loud words in the 'hand knob' area of precentral gyrus, a step toward the eventual goal of decoding attempted speech from ventral speech areas in patients who are unable to speak. APPROACH Neural and audio data were recorded while two BrainGate2 pilot clinical trial participants, each with two chronically-implanted 96-electrode arrays, spoke 420 different words that broadly sampled English phonemes. Phoneme onsets were identified from audio recordings, and their identities were then classified from neural features consisting of each electrode's binned action potential counts or high-frequency local field potential power. Speech synthesis was performed using the 'Brain-to-Speech' pattern matching method. We also examined two potential confounds specific to decoding overt speech: acoustic contamination of neural signals and systematic differences in labeling different phonemes' onset times. MAIN RESULTS A linear decoder achieved up to 29.3% classification accuracy (chance = 6%) across 39 phonemes, while an RNN classifier achieved 33.9% accuracy. Parameter sweeps indicated that performance did not saturate when adding more electrodes or more training data, and that accuracy improved when utilizing time-varying structure in the data. Microphonic contamination and phoneme onset differences modestly increased decoding accuracy, but could be mitigated by acoustic artifact subtraction and using a neural speech onset marker, respectively. Speech synthesis achieved r = 0.523 correlation between true and reconstructed audio. SIGNIFICANCE The ability to decode speech using intracortical electrode array signals from a nontraditional speech area suggests that placing electrode arrays in ventral speech areas is a promising direction for speech BCIs.
Collapse
Affiliation(s)
- Guy H Wilson
- Neurosciences Graduate Program, Stanford University, Stanford, CA, United States of America
| | - Sergey D Stavisky
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
| | - Francis R Willett
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, United States of America
| | - Donald T Avansino
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
| | - Jessica N Kelemen
- Department of Neurology, Harvard Medical School, Boston, MA, United States of America
| | - Leigh R Hochberg
- Department of Neurology, Harvard Medical School, Boston, MA, United States of America
- Center for Neurotechnology and Neurorecovery, Dept. of Neurology, Massachusetts General Hospital, Boston, MA, United States of America
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, United States of America
- Carney Institute for Brain Science and School of Engineering, Brown University, Providence, RI, United States of America
| | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University, Stanford, CA, United States of America
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
| | - Shaul Druckmann
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
- Department of Neurobiology, Stanford University, Stanford, CA, United States of America
| | - Krishna V Shenoy
- Wu Tsai Neurosciences Institute and Bio-X Institute, Stanford University, Stanford, CA, United States of America
- Department of Electrical Engineering, Stanford University, Stanford, CA, United States of America
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, United States of America
- Department of Neurobiology, Stanford University, Stanford, CA, United States of America
- Department of Bioengineering, Stanford University, Stanford, CA, United States of America
| |
Collapse
|
46
|
Chari A, Budhdeo S, Sparks R, Barone DG, Marcus HJ, Pereira EAC, Tisdall MM. Brain-Machine Interfaces: The Role of the Neurosurgeon. World Neurosurg 2020; 146:140-147. [PMID: 33197630 DOI: 10.1016/j.wneu.2020.11.028] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2020] [Revised: 11/04/2020] [Accepted: 11/05/2020] [Indexed: 12/26/2022]
Abstract
Neurotechnology is set to expand rapidly in the coming years as technological innovations in hardware and software are translated to the clinical setting. Given our unique access to patients with neurologic disorders, expertise with which to guide appropriate treatments, and technical skills to implant brain-machine interfaces (BMIs), neurosurgeons have a key role to play in the progress of this field. We outline the current state and key challenges in this rapidly advancing field, including implant technology, implant recipients, implantation methodology, implant function, and ethical, regulatory, and economic considerations. Our key message is to encourage the neurosurgical community to proactively engage in collaborating with other health care professionals, engineers, scientists, ethicists, and regulators in tackling these issues. By doing so, we will equip ourselves with the skills and expertise to drive the field forward and avoid being mere technicians in an industry driven by those around us.
Collapse
Affiliation(s)
- Aswin Chari
- Developmental Neurosciences, Great Ormond Street Institute of Child Health, University College London, London, United Kingdom; Department of Neurosurgery, Great Ormond Street Hospital, London, United Kingdom.
| | - Sanjay Budhdeo
- Department for Clinical and Movement Neurosciences, Queen Square Institute of Neurology, University College London, London, United Kingdom; Department of Neurology, National Hospital for Neurology and Neurosurgery, London, United Kingdom; OwkinInc, New York, New York, USA
| | - Rachel Sparks
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
| | - Damiano G Barone
- Division of Neurosurgery, Department of Clinical Neurosciences, University of Cambridge, Cambridge, United Kingdom
| | - Hani J Marcus
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, United Kingdom; Wellcome EPSRC Centre for Interventional and Surgical Sciences, University College London, London, United Kingdom
| | - Erlick A C Pereira
- Neurosciences Research Centre, Molecular and Clinical Sciences Research Institute, St George's, University of London, United Kingdom
| | - Martin M Tisdall
- Developmental Neurosciences, Great Ormond Street Institute of Child Health, University College London, London, United Kingdom; Department of Neurosurgery, Great Ormond Street Hospital, London, United Kingdom
| |
Collapse
|
47
|
Detection of Electrophysiological Activity of Amygdala during Anesthesia Using Stereo-EEG: A Preliminary Research in Anesthetized Epileptic Patients. BIOMED RESEARCH INTERNATIONAL 2020; 2020:6932035. [PMID: 33102588 PMCID: PMC7568817 DOI: 10.1155/2020/6932035] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/10/2020] [Revised: 09/13/2020] [Accepted: 09/19/2020] [Indexed: 11/18/2022]
Abstract
Recent studies of anesthesia mechanisms have focused on neuronal network and functional connectivity. The stereo-electroencephalography (SEEG) recordings provide appropriate temporal and spatial resolution to study whole-brain dynamics; however, the feasibility to detect subcortical signals during anesthesia still needs to be studied with clinical evidence. Here, we focus on the amygdala to investigate if SEEG can be used to detect cortical and subcortical electrophysiological activity in anesthetized epileptic patients. Therefore, we present direct evidence in humans that SEEG indeed can be used to record cortical and subcortical electrophysiological activity during anesthesia. The study was carried out in propofol-anesthetized five epileptic patients. The electrophysiology activity of the amygdala and other cortical areas from anesthesia to the recovery of consciousness was investigated using stereo-EEG (SEEG). Results indicated that with the decrease of propofol concentration, power spectral density (PSD) in the delta band of the amygdala significantly decreased. When it was close to recovery, the correlation between the amygdala and ipsilateral temporal lobe significantly decreased followed by a considerable increase when awake. The findings of the current study suggest SEEG as an effective tool for providing direct evidence of the anesthesia mechanism.
Collapse
|