1
|
Cubillos LH, Kelberman MM, Mender MJ, Hite A, Temmar H, Willsey M, Kumar NG, Kung TA, Patil PG, Chestek C, Krishnan C. Exploring Synergies in Brain-Machine Interfaces: Compression vs. Performance. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2025:2025.02.03.636273. [PMID: 39975237 PMCID: PMC11838491 DOI: 10.1101/2025.02.03.636273] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/21/2025]
Abstract
Individuals with severe neurological injuries often rely on assistive technologies, but current methods have limitations in accurately decoding multi-degree-of-freedom (DoF) movements. Intracortical brain-machine interfaces (iBMIs) use neural signals to provide a more natural control method, but currently struggle with higher-DoF movements-something the brain handles effortlessly. It has been theorized that the brain simplifies high-DoF movement through muscle synergies, which link multiple muscles to function as a single unit. These synergies have been studied using dimensionality reduction techniques like principal component analysis (PCA), non-negative matrix factorization (NMF), and demixed PCA (dPCA) and successfully used to reduce noise and improve offline decoder stability in non-invasive applications. However, their effectiveness in improving decoding and generalizability for implanted recordings across varied tasks is unclear. Here, we evaluated if brain and muscle synergies can enhance iBMI performance in non-human primates performing a two-DoF finger task. Specifically, we tested if PCA, dPCA, and NMF could compress and denoise brain and muscle data and improve decoder generalization across tasks. Our results showed that while all methods effectively compressed data with minimal loss in decoding accuracy, none improved performance through denoising. Additionally, none of the methods enhanced generalization across tasks. These findings suggest that while dimensionality reduction can aid data compression, alone it may not reveal the "true" control space needed to improve decoder performance or generalizability. Further research is required to determine whether synergies are the optimal control framework or if alternative approaches are required to enhance decoder robustness in iBMI applications. Significance Statement Many researchers believe that brain and muscle synergies represent a fundamental control strategy and could enhance brain-machine interface (BMI) decoding performance. These synergies, extracted through dimensionality reduction techniques, are thought to simplify complex neural data, improving the efficiency and accuracy of BMI systems. In our study, we evaluated brain and muscle synergies in a dexterous finger task. We found that while these synergies effectively compressed high-dimensional data, they did not improve performance through denoising or generalize well across different contexts. Instead, the highest performance was achieved when using all available data, suggesting that synergies, although useful for data compression, may not provide the "true" control space needed to enhance decoder robustness or adaptability in implanted BMI systems.
Collapse
|
2
|
Willsey MS, Shah NP, Avansino DT, Hahn NV, Jamiolkowski RM, Kamdar FB, Hochberg LR, Willett FR, Henderson JM. A high-performance brain-computer interface for finger decoding and quadcopter game control in an individual with paralysis. Nat Med 2025; 31:96-104. [PMID: 39833405 PMCID: PMC11750708 DOI: 10.1038/s41591-024-03341-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Accepted: 10/03/2024] [Indexed: 01/22/2025]
Abstract
People with paralysis express unmet needs for peer support, leisure activities and sporting activities. Many within the general population rely on social media and massively multiplayer video games to address these needs. We developed a high-performance, finger-based brain-computer-interface system allowing continuous control of three independent finger groups, of which the thumb can be controlled in two dimensions, yielding a total of four degrees of freedom. The system was tested in a human research participant with tetraplegia due to spinal cord injury over sequential trials requiring fingers to reach and hold on targets, with an average acquisition rate of 76 targets per minute and completion time of 1.58 ± 0.06 seconds-comparing favorably to prior animal studies despite a twofold increase in the decoded degrees of freedom. More importantly, finger positions were then used to control a virtual quadcopter-the number-one restorative priority for the participant-using a brain-to-finger-to-computer interface to allow dexterous navigation around fixed- and random-ringed obstacle courses. The participant expressed or demonstrated a sense of enablement, recreation and social connectedness that addresses many of the unmet needs of people with paralysis.
Collapse
Affiliation(s)
- Matthew S Willsey
- Department of Neurosurgery, Stanford University, Stanford, CA, USA.
- Department of Neurosurgery, University of Michigan, Ann Arbor, MI, USA.
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, USA.
- Biointerfaces Institute, University of Michigan, Ann Arbor, MI, USA.
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, MI, USA.
| | - Nishal P Shah
- Department of Neurosurgery, Stanford University, Stanford, CA, USA
| | - Donald T Avansino
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Nick V Hahn
- Department of Neurosurgery, Stanford University, Stanford, CA, USA
| | | | - Foram B Kamdar
- Department of Neurosurgery, Stanford University, Stanford, CA, USA
| | - Leigh R Hochberg
- Robert J. and Nancy D. Carney Institute for Brain Science, Brown University, Providence, RI, USA
- School of Engineering, Brown University, Providence, RI, USA
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, VA Providence Healthcare System, Providence, RI, USA
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | | | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University, Stanford, CA, USA.
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA.
- Bio-X Institute, Stanford University, Stanford, CA, USA.
| |
Collapse
|
3
|
Agudelo-Toro A, Michaels JA, Sheng WA, Scherberger H. Accurate neural control of a hand prosthesis by posture-related activity in the primate grasping circuit. Neuron 2024; 112:4115-4129.e8. [PMID: 39419024 DOI: 10.1016/j.neuron.2024.09.018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Revised: 03/15/2024] [Accepted: 09/19/2024] [Indexed: 10/19/2024]
Abstract
Brain-computer interfaces (BCIs) have the potential to restore hand movement for people with paralysis, but current devices still lack the fine control required to interact with objects of daily living. Following our understanding of cortical activity during arm reaches, hand BCI studies have focused primarily on velocity control. However, mounting evidence suggests that posture, and not velocity, dominates in hand-related areas. To explore whether this signal can causally control a prosthesis, we developed a BCI training paradigm centered on the reproduction of posture transitions. Monkeys trained with this protocol were able to control a multidimensional hand prosthesis with high accuracy, including execution of the very intricate precision grip. Analysis revealed that the posture signal in the target grasping areas was the main contributor to control. We present, for the first time, neural posture control of a multidimensional hand prosthesis, opening the door for future interfaces to leverage this additional information channel.
Collapse
Affiliation(s)
- Andres Agudelo-Toro
- Neurobiology Laboratory, Deutsches Primatenzentrum GmbH, Göttingen 37077, Germany.
| | - Jonathan A Michaels
- Neurobiology Laboratory, Deutsches Primatenzentrum GmbH, Göttingen 37077, Germany; School of Kinesiology and Health Science, Faculty of Health, York University, Toronto, ON M3J 1P3, Canada
| | - Wei-An Sheng
- Neurobiology Laboratory, Deutsches Primatenzentrum GmbH, Göttingen 37077, Germany; Institute of Biomedical Sciences, Academia Sinica, Taipei 115, Taiwan
| | - Hansjörg Scherberger
- Neurobiology Laboratory, Deutsches Primatenzentrum GmbH, Göttingen 37077, Germany; Faculty of Biology and Psychology, University of Göttingen, Göttingen 37073, Germany.
| |
Collapse
|
4
|
Feng J, Gao S, Hu Y, Sun G, Sheng W. Brain-Computer Interface for Patients with Spinal Cord Injury: A Bibliometric Study. World Neurosurg 2024; 192:170-187.e1. [PMID: 39245135 DOI: 10.1016/j.wneu.2024.08.163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2024] [Revised: 08/29/2024] [Accepted: 08/30/2024] [Indexed: 09/10/2024]
Abstract
BACKGROUND Spinal cord injury (SCI) is a debilitating condition with profound implications on patients' quality of life. Recent advancements in brain-computer interface (BCI) technology have provided novel opportunities for individuals with paralysis due to SCI. Consequently, research on the application of BCI for treating SCI has received increasing attention from scholars worldwide. However, there is a lack of rigorous bibliometric studies on the evolution and trends in this field. Hence, the present study aimed to use bibliometric methods to investigate the current status and emerging trends in the field of applying BCI for treating SCI and thus identify novel therapeutic options for SCI. METHODS We conducted a comprehensive review of the relevant literature on BCI applications for treating SCI published between 2005 and 2024 by using the Web of Science Core Collection database. To facilitate visualization and quantitative analysis of the published literature, we used VOSviewer and CiteSpace software tools. These tools enabled the assessment of co-authorships, co-occurrences, citations, and co-citations in the selected literature, thereby providing an overview of the current trends and predictive insights into the field. RESULTS The literature search yielded 714 publications from the Web of Science Core Collection database. The findings indicated a significant upward trend in the number of publications, yielding a total of 24,804 citations, with an average citation rate of 34.74 per publication and an H-index of 75. Research contributions were identified from 54 countries/regions, and the United States, China, and Germany emerged as the predominant contributors. A total of 1114 research institutions contributed to the retrieved literature, with Harvard Medical School, Brown University, and Northwestern University producing the highest number of publications. The published literature was predominantly distributed across 258 academic journals, and the Journal of Neural Engineering was the most frequently utilized publication source. Hochberg, Leigh, Henderson, Jaimie, and Collinger were the prominent authors in this field. CONCLUSIONS In recent years, there has been a steep increase in research on the use of BCI for treating SCI. Existing research focuses on the application of BCI for improving rehabilitation and quality of life of patients with SCI. Interdisciplinary collaboration is the current trend in this field.
Collapse
Affiliation(s)
- Jingsheng Feng
- Department of Spinal Surgery, The First Affiliated Hospital of Xinjiang Medical University, Urumqi, China
| | - Shutao Gao
- Department of Spinal Surgery, The First Affiliated Hospital of Xinjiang Medical University, Urumqi, China
| | - Yukun Hu
- Department of Spinal Surgery, The First Affiliated Hospital of Xinjiang Medical University, Urumqi, China
| | - Guangxu Sun
- Department of Spinal Surgery, The First Affiliated Hospital of Xinjiang Medical University, Urumqi, China
| | - Weibin Sheng
- Department of Spinal Surgery, The First Affiliated Hospital of Xinjiang Medical University, Urumqi, China.
| |
Collapse
|
5
|
Thomson CJ, Tully TN, Stone ES, Morrell CB, Scheme EJ, Warren DJ, Hutchinson DT, Clark GA, George JA. Enhancing neuroprosthesis calibration: the advantage of integrating prior training over exclusive use of new data. J Neural Eng 2024; 21:066020. [PMID: 39569866 PMCID: PMC11605518 DOI: 10.1088/1741-2552/ad94a7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2024] [Revised: 10/11/2024] [Accepted: 11/19/2024] [Indexed: 11/22/2024]
Abstract
Objective.Neuroprostheses typically operate under supervised learning, in which a machine-learning algorithm is trained to correlate neural or myoelectric activity with an individual's motor intent. Due to the stochastic nature of neuromyoelectric signals, algorithm performance decays over time. This decay is accelerated when attempting to regress proportional control of multiple joints in parallel, compared with the more typical classification-based pattern recognition control. To overcome this degradation, neuroprostheses and commercial myoelectric prostheses are often recalibrated and retrained frequently so that only the most recent, up-to-date data influences the algorithm performance. Here, we introduce and validate an alternative training paradigm in which training data from past calibrations is aggregated and reused in future calibrations for regression control.Approach.Using a cohort of four transradial amputees implanted with intramuscular electromyographic recording leads, we demonstrate that aggregating prior datasets improves prosthetic regression-based control in offline analyses and an online human-in-the-loop task. In offline analyses, we compared the performance of a convolutional neural network (CNN) and a modified Kalman filter (MKF) to simultaneously regress the kinematics of an eight-degree-of-freedom prosthesis. Both algorithms were trained under the traditional paradigm using a single dataset, as well as under the new paradigm using aggregated datasets from the past five or ten trainings.Main results.Dataset aggregation reduced the root-mean-squared error (RMSE) of algorithm estimates for both the CNN and MKF, although the CNN saw a greater reduction in error. Further offline analyses revealed that dataset aggregation improved CNN robustness when reusing the same algorithm on subsequent test days, as indicated by a smaller increase in RMSE per day. Finally, data from an online virtual-target-touching task with one amputee showed significantly better real-time prosthetic control when using aggregated training data from just two prior datasets.Significance.Altogether, these results demonstrate that training data from past calibrations should not be discarded but, rather, should be reused in an aggregated training dataset such that the increased amount and diversity of data improve algorithm performance. More broadly, this work supports a paradigm shift for the field of neuroprostheses away from daily data recalibration for linear classification models and towards daily data aggregation for non-linear regression models.
Collapse
Affiliation(s)
- Caleb J Thomson
- Department of Biomedical Engineering, University of Utah, Salt Lake City, UT 84112, United States of America
| | - Troy N Tully
- Department of Biomedical Engineering, University of Utah, Salt Lake City, UT 84112, United States of America
| | - Eric S Stone
- Department of Biomedical Engineering, University of Utah, Salt Lake City, UT 84112, United States of America
| | - Christian B Morrell
- Institute of Biomedical Engineering, University of New Brunswick, Fredericton, New Brunswick E3B 5A3, Canada
| | - Erik J Scheme
- Institute of Biomedical Engineering, University of New Brunswick, Fredericton, New Brunswick E3B 5A3, Canada
| | - David J Warren
- Department of Biomedical Engineering, University of Utah, Salt Lake City, UT 84112, United States of America
- Department of Electrical and Computer Engineering, University of Utah, Salt Lake City, UT 84112, United States of America
| | - Douglas T Hutchinson
- Department of Orthopaedics, University of Utah, Salt Lake City, UT 84112, United States of America
| | - Gregory A Clark
- Department of Biomedical Engineering, University of Utah, Salt Lake City, UT 84112, United States of America
| | - Jacob A George
- Department of Biomedical Engineering, University of Utah, Salt Lake City, UT 84112, United States of America
- Department of Electrical and Computer Engineering, University of Utah, Salt Lake City, UT 84112, United States of America
- Department of Mechanical Engineering, University of Utah, Salt Lake City, UT 84112, United States of America
- Department of Physical Medicine and Rehabilitation, University of Utah, Salt Lake City, UT 84112, United States of America
| |
Collapse
|
6
|
Karpowicz BM, Ye J, Fan C, Tostado-Marcos P, Rizzoglio F, Washington C, Scodeler T, de Lucena D, Nason-Tomaszewski SR, Mender MJ, Ma X, Arneodo EM, Hochberg LR, Chestek CA, Henderson JM, Gentner TQ, Gilja V, Miller LE, Rouse AG, Gaunt RA, Collinger JL, Pandarinath C. Few-shot Algorithms for Consistent Neural Decoding (FALCON) Benchmark. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.09.15.613126. [PMID: 39345641 PMCID: PMC11429771 DOI: 10.1101/2024.09.15.613126] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/01/2024]
Abstract
Intracortical brain-computer interfaces (iBCIs) can restore movement and communication abilities to individuals with paralysis by decoding their intended behavior from neural activity recorded with an implanted device. While this activity yields high-performance decoding over short timescales, neural data are often nonstationary, which can lead to decoder failure if not accounted for. To maintain performance, users must frequently recalibrate decoders, which requires the arduous collection of new neural and behavioral data. Aiming to reduce this burden, several approaches have been developed that either limit recalibration data requirements (few-shot approaches) or eliminate explicit recalibration entirely (zero-shot approaches). However, progress is limited by a lack of standardized datasets and comparison metrics, causing methods to be compared in an ad hoc manner. Here we introduce the FALCON benchmark suite (Few-shot Algorithms for COnsistent Neural decoding) to standardize evaluation of iBCI robustness. FALCON curates five datasets of neural and behavioral data that span movement and communication tasks to focus on behaviors of interest to modern-day iBCIs. Each dataset includes calibration data, optional few-shot recalibration data, and private evaluation data. We implement a flexible evaluation platform which only requires user-submitted code to return behavioral predictions on unseen data. We also seed the benchmark by applying baseline methods spanning several classes of possible approaches. FALCON aims to provide rigorous selection criteria for robust iBCI decoders, easing their translation to real-world devices.
Collapse
|
7
|
Wu Y, Wang L, Yan M, Wang X, Liao X, Zhong C, Ke D, Lu Y. Poly(3,4-Ethylenedioxythiophene)/Functional Gold Nanoparticle films for Improving the Electrode-Neural Interface. Adv Healthc Mater 2024; 13:e2400836. [PMID: 38757738 DOI: 10.1002/adhm.202400836] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Revised: 05/07/2024] [Indexed: 05/18/2024]
Abstract
Implantable neural electrodes are indispensable tools for recording neuron activity, playing a crucial role in neuroscience research. However, traditional neural electrodes suffer from limited electrochemical performance, compromised biocompatibility, and tentative stability, posing great challenges for reliable long-term studies in free-moving animals. In this study, a novel approach employing a hybrid film composed of poly(3,4-ethylenedioxythiophene)/functional gold nanoparticles (PEDOT/3-MPA-Au) to improve the electrode-neural interface is presented. The deposited PEDOT/3-MPA-Au demonstrates superior cathodal charge storage capacity, reduced electrochemical impedance, and remarkable electrochemical and mechanical stability. Upon implantation into the cortex of mice for a duration of 12 weeks, the modified electrodes exhibit notably decreased levels of glial fibrillary acidic protein and increased neuronal nuclei immunostaining compared to counterparts utilizing poly(3,4-ethylenedioxythiophene)/poly(styrene sulfonate). Additionally, the PEDOT/3-MPA-Au modified electrodes consistently capture high-quality, stable long-term electrophysiological signals in vivo, enabling continuous recording of target neurons for up to 16 weeks. This innovative modification strategy offers a promising solution for fabricating low-impedance, tissue-friendly, and long-term stable neural interfaces, thereby addressing the shortcomings of conventional neural electrodes. These findings mark a significant advancement toward the development of more reliable and efficacious neural interfaces, with broad implications for both research and clinical applications.
Collapse
Affiliation(s)
- Yiyong Wu
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen-Hong Kong Institute of Brain Science, Shenzhen, 518055, China
| | - Lulu Wang
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen-Hong Kong Institute of Brain Science, Shenzhen, 518055, China
| | - Mengying Yan
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen-Hong Kong Institute of Brain Science, Shenzhen, 518055, China
| | - Xufang Wang
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen-Hong Kong Institute of Brain Science, Shenzhen, 518055, China
| | - Xin Liao
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen-Hong Kong Institute of Brain Science, Shenzhen, 518055, China
| | - Cheng Zhong
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen-Hong Kong Institute of Brain Science, Shenzhen, 518055, China
| | - Dingning Ke
- Experiment and Innovation Center, Harbin Institute of Technology (Shenzhen), Shenzhen, 518055, China
| | - Yi Lu
- Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen-Hong Kong Institute of Brain Science, Shenzhen, 518055, China
| |
Collapse
|
8
|
Xu J, Mawase F, Schieber MH. Evolution, biomechanics, and neurobiology converge to explain selective finger motor control. Physiol Rev 2024; 104:983-1020. [PMID: 38385888 PMCID: PMC11380997 DOI: 10.1152/physrev.00030.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Revised: 01/16/2024] [Accepted: 02/15/2024] [Indexed: 02/23/2024] Open
Abstract
Humans use their fingers to perform a variety of tasks, from simple grasping to manipulating objects, to typing and playing musical instruments, a variety wider than any other species. The more sophisticated the task, the more it involves individuated finger movements, those in which one or more selected fingers perform an intended action while the motion of other digits is constrained. Here we review the neurobiology of such individuated finger movements. We consider their evolutionary origins, the extent to which finger movements are in fact individuated, and the evolved features of neuromuscular control that both enable and limit individuation. We go on to discuss other features of motor control that combine with individuation to create dexterity, the impairment of individuation by disease, and the broad extent of capabilities that individuation confers on humans. We comment on the challenges facing the development of a truly dexterous bionic hand. We conclude by identifying topics for future investigation that will advance our understanding of how neural networks interact across multiple regions of the central nervous system to create individuated movements for the skills humans use to express their cognitive activity.
Collapse
Affiliation(s)
- Jing Xu
- Department of Kinesiology, University of Georgia, Athens, Georgia, United States
| | - Firas Mawase
- Department of Biomedical Engineering, Israel Institute of Technology, Haifa, Israel
| | - Marc H Schieber
- Departments of Neurology and Neuroscience, University of Rochester, Rochester, New York, United States
| |
Collapse
|
9
|
Shah NP, Willsey MS, Hahn N, Kamdar F, Avansino DT, Fan C, Hochberg LR, Willett FR, Henderson JM. A flexible intracortical brain-computer interface for typing using finger movements. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.04.22.590630. [PMID: 38712189 PMCID: PMC11071346 DOI: 10.1101/2024.04.22.590630] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2024]
Abstract
Keyboard typing with finger movements is a versatile digital interface for users with diverse skills, needs, and preferences. Currently, such an interface does not exist for people with paralysis. We developed an intracortical brain-computer interface (BCI) for typing with attempted flexion/extension movements of three finger groups on the right hand, or both hands, and demonstrated its flexibility in two dominant typing paradigms. The first paradigm is "point-and-click" typing, where a BCI user selects one key at a time using continuous real-time control, allowing selection of arbitrary sequences of symbols. During cued character selection with this paradigm, a human research participant with paralysis achieved 30-40 selections per minute with nearly 90% accuracy. The second paradigm is "keystroke" typing, where the BCI user selects each character by a discrete movement without real-time feedback, often giving a faster speed for natural language sentences. With 90 cued characters per minute, decoding attempted finger movements and correcting errors using a language model resulted in more than 90% accuracy. Notably, both paradigms matched the state-of-the-art for BCI performance and enabled further flexibility by the simultaneous selection of multiple characters as well as efficient decoder estimation across paradigms. Overall, the high-performance interface is a step towards the wider accessibility of BCI technology by addressing unmet user needs for flexibility.
Collapse
Affiliation(s)
| | - Matthew S Willsey
- Department of Neurosurgery, Stanford University
- Department of Neurosurgery, The University of Texas at Austin, Austin, TX, USA + This work was primarily conducted at Stanford University
| | - Nick Hahn
- Department of Neurosurgery, Stanford University
| | | | - Donald T Avansino
- Department of Neurosurgery, Stanford University
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Chaofei Fan
- Department of Neurosurgery, Stanford University
| | - Leigh R Hochberg
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, USA
- School of Engineering, Brown University, Providence, RI, USA
- Robert J. and Nancy D. Carney Institute for Brain Science, Brown University, Providence, RI, USA
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Francis R Willett
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Bio-X Institute, Stanford University, Stanford, CA, USA
| |
Collapse
|
10
|
Temmar H, Willsey MS, Costello JT, Mender MJ, Cubillos LH, Lam JL, Wallace DM, Kelberman MM, Patil PG, Chestek CA. Artificial neural network for brain-machine interface consistently produces more naturalistic finger movements than linear methods. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.01.583000. [PMID: 38496403 PMCID: PMC10942378 DOI: 10.1101/2024.03.01.583000] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/19/2024]
Abstract
Brain-machine interfaces (BMI) aim to restore function to persons living with spinal cord injuries by 'decoding' neural signals into behavior. Recently, nonlinear BMI decoders have outperformed previous state-of-the-art linear decoders, but few studies have investigated what specific improvements these nonlinear approaches provide. In this study, we compare how temporally convolved feedforward neural networks (tcFNNs) and linear approaches predict individuated finger movements in open and closed-loop settings. We show that nonlinear decoders generate more naturalistic movements, producing distributions of velocities 85.3% closer to true hand control than linear decoders. Addressing concerns that neural networks may come to inconsistent solutions, we find that regularization techniques improve the consistency of tcFNN convergence by 194.6%, along with improving average performance, and training speed. Finally, we show that tcFNN can leverage training data from multiple task variations to improve generalization. The results of this study show that nonlinear methods produce more naturalistic movements and show potential for generalizing over less constrained tasks. Teaser A neural network decoder produces consistent naturalistic movements and shows potential for real-world generalization through task variations.
Collapse
|
11
|
Willsey MS, Shah NP, Avansino DT, Hahn NV, Jamiolkowski RM, Kamdar FB, Hochberg LR, Willett FR, Henderson JM. A real-time, high-performance brain-computer interface for finger decoding and quadcopter control. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.06.578107. [PMID: 38370697 PMCID: PMC10871262 DOI: 10.1101/2024.02.06.578107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
People with paralysis express unmet needs for peer support, leisure activities, and sporting activities. Many within the general population rely on social media and massively multiplayer video games to address these needs. We developed a high-performance finger brain-computer-interface system allowing continuous control of 3 independent finger groups with 2D thumb movements. The system was tested in a human research participant over sequential trials requiring fingers to reach and hold on targets, with an average acquisition rate of 76 targets/minute and completion time of 1.58 ± 0.06 seconds. Performance compared favorably to previous animal studies, despite a 2-fold increase in the decoded degrees-of-freedom (DOF). Finger positions were then used for 4-DOF velocity control of a virtual quadcopter, demonstrating functionality over both fixed and random obstacle courses. This approach shows promise for controlling multiple-DOF end-effectors, such as robotic fingers or digital interfaces for work, entertainment, and socialization.
Collapse
|
12
|
Shah NP, Avansino D, Kamdar F, Nicolas C, Kapitonava A, Vargas-Irwin C, Hochberg L, Pandarinath C, Shenoy K, Willett FR, Henderson J. Pseudo-linear Summation explains Neural Geometry of Multi-finger Movements in Human Premotor Cortex. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.10.11.561982. [PMID: 37873182 PMCID: PMC10592742 DOI: 10.1101/2023.10.11.561982] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/25/2023]
Abstract
How does the motor cortex combine simple movements (such as single finger flexion/extension) into complex movements (such hand gestures or playing piano)? Motor cortical activity was recorded using intracortical multi-electrode arrays in two people with tetraplegia as they attempted single, pairwise and higher order finger movements. Neural activity for simultaneous movements was largely aligned with linear summation of corresponding single finger movement activities, with two violations. First, the neural activity was normalized, preventing a large magnitude with an increasing number of moving fingers. Second, the neural tuning direction of weakly represented fingers (e.g. middle) changed significantly as a result of the movement of other fingers. These deviations from linearity resulted in non-linear methods outperforming linear methods for neural decoding. Overall, simultaneous finger movements are thus represented by the combination of individual finger movements by pseudo-linear summation.
Collapse
Affiliation(s)
| | - Donald Avansino
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | | | - Claire Nicolas
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Anastasia Kapitonava
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Carlos Vargas-Irwin
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, USA
- Robert J. and Nancy D. Carney Institute for Brain Science, Brown University, Providence, RI, USA
- Department of Neuroscience, Brown University, Providence, RI, USA
| | - Leigh Hochberg
- VA RR&D Center for Neurorestoration and Neurotechnology, Rehabilitation R&D Service, Providence VA Medical Center, Providence, RI, USA
- School of Engineering, Brown University, Providence, RI, USA
- Robert J. and Nancy D. Carney Institute for Brain Science, Brown University, Providence, RI, USA
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Chethan Pandarinath
- Wallace H. Coulter Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, GA, USA
- Department of Neurosurgery, Emory University, Atlanta, GA, USA
| | - Krishna Shenoy
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Bio-X Institute, Stanford University, Stanford, CA, USA
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Department of Bioengineering, Stanford University, Stanford, CA, USA
- Department of Neurobiology, Stanford University, Stanford, CA, USA
| | - Francis R Willett
- Howard Hughes Medical Institute at Stanford University, Stanford, CA, USA
| | - Jaimie Henderson
- Department of Neurosurgery, Stanford University
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
- Bio-X Institute, Stanford University, Stanford, CA, USA
| |
Collapse
|
13
|
Garwood IC, Major AJ, Antonini MJ, Correa J, Lee Y, Sahasrabudhe A, Mahnke MK, Miller EK, Brown EN, Anikeeva P. Multifunctional fibers enable modulation of cortical and deep brain activity during cognitive behavior in macaques. SCIENCE ADVANCES 2023; 9:eadh0974. [PMID: 37801492 PMCID: PMC10558126 DOI: 10.1126/sciadv.adh0974] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/09/2023] [Accepted: 09/05/2023] [Indexed: 10/08/2023]
Abstract
Recording and modulating neural activity in vivo enables investigations of the neurophysiology underlying behavior and disease. However, there is a dearth of translational tools for simultaneous recording and localized receptor-specific modulation. We address this limitation by translating multifunctional fiber neurotechnology previously only available for rodent studies to enable cortical and subcortical neural recording and modulation in macaques. We record single-neuron and broader oscillatory activity during intracranial GABA infusions in the premotor cortex and putamen. By applying state-space models to characterize changes in electrophysiology, we uncover that neural activity evoked by a working memory task is reshaped by even a modest local inhibition. The recordings provide detailed insight into the electrophysiological effect of neurotransmitter receptor modulation in both cortical and subcortical structures in an awake macaque. Our results demonstrate a first-time application of multifunctional fibers for causal studies of neuronal activity in behaving nonhuman primates and pave the way for clinical translation of fiber-based neurotechnology.
Collapse
Affiliation(s)
- Indie C. Garwood
- Harvard-MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Alex J. Major
- The Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Marc-Joseph Antonini
- Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Josefina Correa
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Youngbin Lee
- Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Atharva Sahasrabudhe
- Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Chemistry, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Meredith K. Mahnke
- The Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Earl K. Miller
- The Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Emery N. Brown
- Harvard-MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology, Cambridge, MA, USA
- The Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Anesthesia, Critical Care, and Pain Medicine, Massachusetts General Hospital, Boston, MA, USA
- Institute for Medical Engineering and Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Anaesthesia, Harvard Medical School, Boston, MA, USA
| | - Polina Anikeeva
- Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA, USA
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA
| |
Collapse
|
14
|
Wallace DM, Benyamini M, Nason-Tomaszewski SR, Costello JT, Cubillos LH, Mender MJ, Temmar H, Willsey MS, Patil PG, Chestek CA, Zacksenhouse M. Error detection and correction in intracortical brain-machine interfaces controlling two finger groups. J Neural Eng 2023; 20:046037. [PMID: 37567222 PMCID: PMC10594236 DOI: 10.1088/1741-2552/acef95] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2023] [Revised: 08/01/2023] [Accepted: 08/11/2023] [Indexed: 08/13/2023]
Abstract
Objective.While brain-machine interfaces (BMIs) are promising technologies that could provide direct pathways for controlling the external world and thus regaining motor capabilities, their effectiveness is hampered by decoding errors. Previous research has demonstrated the detection and correction of BMI outcome errors, which occur at the end of trials. Here we focus on continuous detection and correction of BMI execution errors, which occur during real-time movements.Approach.Two adult male rhesus macaques were implanted with Utah arrays in the motor cortex. The monkeys performed single or two-finger group BMI tasks where a Kalman filter decoded binned spiking-band power into intended finger kinematics. Neural activity was analyzed to determine how it depends not only on the kinematics of the fingers, but also on the distance of each finger-group to its target. We developed a method to detect erroneous movements, i.e. consistent movements away from the target, from the same neural activity used by the Kalman filter. Detected errors were corrected by a simple stopping strategy, and the effect on performance was evaluated.Mainresults.First we show that including distance to target explains significantly more variance of the recorded neural activity. Then, for the first time, we demonstrate that neural activity in motor cortex can be used to detect execution errors during BMI controlled movements. Keeping false positive rate below5%, it was possible to achieve mean true positive rate of28.1%online. Despite requiring 200 ms to detect and react to suspected errors, we were able to achieve a significant improvement in task performance via reduced orbiting time of one finger group.Significance.Neural activity recorded in motor cortex for BMI control can be used to detect and correct BMI errors and thus to improve performance. Further improvements may be obtained by enhancing classification and correction strategies.
Collapse
Affiliation(s)
- Dylan M Wallace
- Department of Robotics, University of Michigan, Ann Arbor, MI, United States of America
| | - Miri Benyamini
- BCI for Rehabilitation Lab., Faculty of Mechanical Engineering, Technion - Israel Institute of Technology, Haifa, Israel
| | - Samuel R Nason-Tomaszewski
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
| | - Joseph T Costello
- Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, United States of America
| | - Luis H Cubillos
- Department of Robotics, University of Michigan, Ann Arbor, MI, United States of America
| | - Matthew J Mender
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
| | - Hisham Temmar
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
| | - Matthew S Willsey
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
- Department of Neurosurgery, University of Michigan, Ann Arbor, MI, United States of America
| | - Parag G Patil
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
- Department of Neurosurgery, University of Michigan, Ann Arbor, MI, United States of America
| | - Cynthia A Chestek
- Department of Robotics, University of Michigan, Ann Arbor, MI, United States of America
- Cortical Neural Prosthetics Lab., Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, United States of America
- Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, United States of America
| | - Miriam Zacksenhouse
- BCI for Rehabilitation Lab., Faculty of Mechanical Engineering, Technion - Israel Institute of Technology, Haifa, Israel
| |
Collapse
|
15
|
Dong Y, Wang S, Huang Q, Berg RW, Li G, He J. Neural Decoding for Intracortical Brain-Computer Interfaces. CYBORG AND BIONIC SYSTEMS 2023; 4:0044. [PMID: 37519930 PMCID: PMC10380541 DOI: 10.34133/cbsystems.0044] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2022] [Accepted: 07/04/2023] [Indexed: 08/01/2023] Open
Abstract
Brain-computer interfaces have revolutionized the field of neuroscience by providing a solution for paralyzed patients to control external devices and improve the quality of daily life. To accurately and stably control effectors, it is important for decoders to recognize an individual's motor intention from neural activity either by noninvasive or intracortical neural recording. Intracortical recording is an invasive way of measuring neural electrical activity with high temporal and spatial resolution. Herein, we review recent developments in neural signal decoding methods for intracortical brain-computer interfaces. These methods have achieved good performance in analyzing neural activity and controlling robots and prostheses in nonhuman primates and humans. For more complex paradigms in motor rehabilitation or other clinical applications, there remains more space for further improvements of decoders.
Collapse
Affiliation(s)
- Yuanrui Dong
- School of Mechatronical Engineering and Beijing Advanced Innovation Center for Intelligent Robots,
Beijing Institute of Technology, Beijing 100081, China
| | - Shirong Wang
- School of Mechatronical Engineering and Beijing Advanced Innovation Center for Intelligent Robots,
Beijing Institute of Technology, Beijing 100081, China
| | - Qiang Huang
- School of Mechatronical Engineering and Beijing Advanced Innovation Center for Intelligent Robots,
Beijing Institute of Technology, Beijing 100081, China
| | - Rune W. Berg
- Department of Neuroscience,
University of Copenhagen, Copenhagen 2200, Denmark
| | - Guanghui Li
- Department of Neuroscience,
University of Copenhagen, Copenhagen 2200, Denmark
| | - Jiping He
- School of Mechatronical Engineering and Beijing Advanced Innovation Center for Intelligent Robots,
Beijing Institute of Technology, Beijing 100081, China
| |
Collapse
|
16
|
Costello JT, Temmar H, Cubillos LH, Mender MJ, Wallace DM, Willsey MS, Patil PG, Chestek CA. Balancing Memorization and Generalization in RNNs for High Performance Brain-Machine Interfaces. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.05.28.542435. [PMID: 37292755 PMCID: PMC10245969 DOI: 10.1101/2023.05.28.542435] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Brain-machine interfaces (BMIs) can restore motor function to people with paralysis but are currently limited by the accuracy of real-time decoding algorithms. Recurrent neural networks (RNNs) using modern training techniques have shown promise in accurately predicting movements from neural signals but have yet to be rigorously evaluated against other decoding algorithms in a closed-loop setting. Here we compared RNNs to other neural network architectures in real-time, continuous decoding of finger movements using intracortical signals from nonhuman primates. Across one and two finger online tasks, LSTMs (a type of RNN) outperformed convolutional and transformer-based neural networks, averaging 18% higher throughput than the convolution network. On simplified tasks with a reduced movement set, RNN decoders were allowed to memorize movement patterns and matched able-bodied control. Performance gradually dropped as the number of distinct movements increased but did not go below fully continuous decoder performance. Finally, in a two-finger task where one degree-of-freedom had poor input signals, we recovered functional control using RNNs trained to act both like a movement classifier and continuous decoder. Our results suggest that RNNs can enable functional real-time BMI control by learning and generating accurate movement patterns.
Collapse
|
17
|
Guan C, Aflalo T, Kadlec K, Gámez de Leon J, Rosario ER, Bari A, Pouratian N, Andersen RA. Decoding and geometry of ten finger movements in human posterior parietal cortex and motor cortex. J Neural Eng 2023; 20:036020. [PMID: 37160127 PMCID: PMC10209510 DOI: 10.1088/1741-2552/acd3b1] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2022] [Revised: 03/24/2023] [Accepted: 05/09/2023] [Indexed: 05/11/2023]
Abstract
Objective. Enable neural control of individual prosthetic fingers for participants with upper-limb paralysis.Approach. Two tetraplegic participants were each implanted with a 96-channel array in the left posterior parietal cortex (PPC). One of the participants was additionally implanted with a 96-channel array near the hand knob of the left motor cortex (MC). Across tens of sessions, we recorded neural activity while the participants attempted to move individual fingers of the right hand. Offline, we classified attempted finger movements from neural firing rates using linear discriminant analysis with cross-validation. The participants then used the neural classifier online to control individual fingers of a brain-machine interface (BMI). Finally, we characterized the neural representational geometry during individual finger movements of both hands.Main Results. The two participants achieved 86% and 92% online accuracy during BMI control of the contralateral fingers (chance = 17%). Offline, a linear decoder achieved ten-finger decoding accuracies of 70% and 66% using respective PPC recordings and 75% using MC recordings (chance = 10%). In MC and in one PPC array, a factorized code linked corresponding finger movements of the contralateral and ipsilateral hands.Significance. This is the first study to decode both contralateral and ipsilateral finger movements from PPC. Online BMI control of contralateral fingers exceeded that of previous finger BMIs. PPC and MC signals can be used to control individual prosthetic fingers, which may contribute to a hand restoration strategy for people with tetraplegia.
Collapse
Affiliation(s)
- Charles Guan
- California Institute of Technology, Pasadena, CA, United States of America
| | - Tyson Aflalo
- California Institute of Technology, Pasadena, CA, United States of America
- T&C Chen Brain-Machine Interface Center at Caltech, Pasadena, CA, United States of America
| | - Kelly Kadlec
- California Institute of Technology, Pasadena, CA, United States of America
| | | | - Emily R Rosario
- Casa Colina Hospital and Centers for Healthcare, Pomona, CA, United States of America
| | - Ausaf Bari
- David Geffen School of Medicine at UCLA, Los Angeles, CA, United States of America
| | - Nader Pouratian
- University of Texas Southwestern Medical Center, Dallas, TX, United States of America
| | - Richard A Andersen
- California Institute of Technology, Pasadena, CA, United States of America
- T&C Chen Brain-Machine Interface Center at Caltech, Pasadena, CA, United States of America
| |
Collapse
|
18
|
Deo DR, Willett FR, Avansino DT, Hochberg LR, Henderson JM, Shenoy KV. Translating deep learning to neuroprosthetic control. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.04.21.537581. [PMID: 37131830 PMCID: PMC10153231 DOI: 10.1101/2023.04.21.537581] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/04/2023]
Abstract
Advances in deep learning have given rise to neural network models of the relationship between movement and brain activity that appear to far outperform prior approaches. Brain-computer interfaces (BCIs) that enable people with paralysis to control external devices, such as robotic arms or computer cursors, might stand to benefit greatly from these advances. We tested recurrent neural networks (RNNs) on a challenging nonlinear BCI problem: decoding continuous bimanual movement of two computer cursors. Surprisingly, we found that although RNNs appeared to perform well in offline settings, they did so by overfitting to the temporal structure of the training data and failed to generalize to real-time neuroprosthetic control. In response, we developed a method that alters the temporal structure of the training data by dilating/compressing it in time and re-ordering it, which we show helps RNNs successfully generalize to the online setting. With this method, we demonstrate that a person with paralysis can control two computer cursors simultaneously, far outperforming standard linear methods. Our results provide evidence that preventing models from overfitting to temporal structure in training data may, in principle, aid in translating deep learning advances to the BCI setting, unlocking improved performance for challenging applications.
Collapse
|
19
|
Shah NP, Willsey MS, Hahn N, Kamdar F, Avansino DT, Hochberg LR, Shenoy KV, Henderson JM. A brain-computer typing interface using finger movements. INTERNATIONAL IEEE/EMBS CONFERENCE ON NEURAL ENGINEERING : [PROCEEDINGS]. INTERNATIONAL IEEE EMBS CONFERENCE ON NEURAL ENGINEERING 2023; 2023:10.1109/ner52421.2023.10123912. [PMID: 37465143 PMCID: PMC10353344 DOI: 10.1109/ner52421.2023.10123912] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/20/2023]
Abstract
Intracortical brain computer interfaces (iBCIs) decode neural activity from the cortex and enable motor and communication prostheses, such as cursor control, handwriting and speech, for people with paralysis. This paper introduces a new iBCI communication prosthesis using a 3D keyboard interface for typing using continuous, closed loop movement of multiple fingers. A participant-specific BCI keyboard prototype was developed for a BrainGate2 clinical trial participant (T5) using neural recordings from the hand-knob area of the left premotor cortex. We assessed the relative decoding accuracy of flexion/extension movements of individual single fingers (5 degrees of freedom (DOF)) vs. three groups of fingers (thumb, index-middle, and ring-small fingers, 3 DOF). Neural decoding using 3 independent DOF was more accurate (95%) than that using 5 DOF (76%). A virtual keyboard was then developed where each finger group moved along a flexion-extension arc to acquire targets that corresponded to English letters and symbols. The locations of these letter/symbols were optimized using natural language statistics, resulting in an approximately a 2× reduction in distance traveled by fingers on average compared to a random keyboard layout. This keyboard was tested using a simple real-time closed loop decoder enabling T5 to type with 31 symbols at 90% accuracy and approximately 2.3 sec/symbol (excluding a 2 second hold time) on average.
Collapse
Affiliation(s)
| | | | | | | | | | - Leigh R Hochberg
- Neurol., Mass. Gen. Hosp; Boston, MA; Brown Univ./VA Medical Center, Providence, RI
| | | | | |
Collapse
|
20
|
Aristides RP, Pons AJ, Cerdeira HA, Masoller C, Tirabassi G. Parameter and coupling estimation in small networks of Izhikevich's neurons. CHAOS (WOODBURY, N.Y.) 2023; 33:043123. [PMID: 37097937 DOI: 10.1063/5.0144499] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/30/2023] [Accepted: 03/22/2023] [Indexed: 06/19/2023]
Abstract
Nowadays, experimental techniques allow scientists to have access to large amounts of data. In order to obtain reliable information from the complex systems that produce these data, appropriate analysis tools are needed. The Kalman filter is a frequently used technique to infer, assuming a model of the system, the parameters of the model from uncertain observations. A well-known implementation of the Kalman filter, the unscented Kalman filter (UKF), was recently shown to be able to infer the connectivity of a set of coupled chaotic oscillators. In this work, we test whether the UKF can also reconstruct the connectivity of small groups of coupled neurons when their links are either electrical or chemical synapses. In particular, we consider Izhikevich neurons and aim to infer which neurons influence each other, considering simulated spike trains as the experimental observations used by the UKF. First, we verify that the UKF can recover the parameters of a single neuron, even when the parameters vary in time. Second, we analyze small neural ensembles and demonstrate that the UKF allows inferring the connectivity between the neurons, even for heterogeneous, directed, and temporally evolving networks. Our results show that time-dependent parameter and coupling estimation is possible in this nonlinearly coupled system.
Collapse
Affiliation(s)
- R P Aristides
- Instituto de Física Teórica, Universidade Estadual Paulista, Rua Dr. Bento Teobaldo Ferraz 271, Bloco II, Barra Funda, 01140-070 São Paulo, Brazil
- Departament de Fisica, Universitat Politecnica de Catalunya, St. Nebridi 22, 08222 Terrassa, Spain
| | - A J Pons
- Departament de Fisica, Universitat Politecnica de Catalunya, St. Nebridi 22, 08222 Terrassa, Spain
| | - H A Cerdeira
- Instituto de Física Teórica, Universidade Estadual Paulista, Rua Dr. Bento Teobaldo Ferraz 271, Bloco II, Barra Funda, 01140-070 São Paulo, Brazil
| | - C Masoller
- Departament de Fisica, Universitat Politecnica de Catalunya, St. Nebridi 22, 08222 Terrassa, Spain
| | - G Tirabassi
- Departament de Fisica, Universitat Politecnica de Catalunya, St. Nebridi 22, 08222 Terrassa, Spain
| |
Collapse
|
21
|
Hu Z, Niu Q, Hsiao BS, Yao X, Zhang Y. Bioactive polymer-enabled conformal neural interface and its application strategies. MATERIALS HORIZONS 2023; 10:808-828. [PMID: 36597872 DOI: 10.1039/d2mh01125e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
Neural interface is a powerful tool to control the varying neuron activities in the brain, where the performance can directly affect the quality of recording neural signals and the reliability of in vivo connection between the brain and external equipment. Recent advances in bioelectronic innovation have provided promising pathways to fabricate flexible electrodes by integrating electrodes on bioactive polymer substrates. These bioactive polymer-based electrodes can enable the conformal contact with irregular tissue and result in low inflammation when compared to conventional rigid inorganic electrodes. In this review, we focus on the use of silk fibroin and cellulose biopolymers as well as certain synthetic polymers to offer the desired flexibility for constructing electrode substrates for a conformal neural interface. First, the development of a neural interface is reviewed, and the signal recording methods and tissue response features of the implanted electrodes are discussed in terms of biocompatibility and flexibility of corresponding neural interfaces. Following this, the material selection, structure design and integration of conformal neural interfaces accompanied by their effective applications are described. Finally, we offer our perspectives on the evolution of desired bioactive polymer-enabled neural interfaces, regarding the biocompatibility, electrical properties and mechanical softness.
Collapse
Affiliation(s)
- Zhanao Hu
- State Key Laboratory for Modification of Chemical Fibers and Polymer Materials, Shanghai Engineering Research Center of Nano-Biomaterials and Regenerative Medicine, College of Materials Science and Engineering, Donghua University, Shanghai, 201620, People's Republic of China.
| | - Qianqian Niu
- State Key Laboratory for Modification of Chemical Fibers and Polymer Materials, Shanghai Engineering Research Center of Nano-Biomaterials and Regenerative Medicine, College of Materials Science and Engineering, Donghua University, Shanghai, 201620, People's Republic of China.
| | - Benjamin S Hsiao
- Department of Chemistry, Stony Brook University, Stony Brook, New York, 11794-3400, USA
| | - Xiang Yao
- State Key Laboratory for Modification of Chemical Fibers and Polymer Materials, Shanghai Engineering Research Center of Nano-Biomaterials and Regenerative Medicine, College of Materials Science and Engineering, Donghua University, Shanghai, 201620, People's Republic of China.
| | - Yaopeng Zhang
- State Key Laboratory for Modification of Chemical Fibers and Polymer Materials, Shanghai Engineering Research Center of Nano-Biomaterials and Regenerative Medicine, College of Materials Science and Engineering, Donghua University, Shanghai, 201620, People's Republic of China.
| |
Collapse
|
22
|
Willsey MS, Nason-Tomaszewski SR, Ensel SR, Temmar H, Mender MJ, Costello JT, Patil PG, Chestek CA. Real-time brain-machine interface in non-human primates achieves high-velocity prosthetic finger movements using a shallow feedforward neural network decoder. Nat Commun 2022; 13:6899. [PMID: 36371498 PMCID: PMC9653378 DOI: 10.1038/s41467-022-34452-w] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2022] [Accepted: 10/25/2022] [Indexed: 11/13/2022] Open
Abstract
Despite the rapid progress and interest in brain-machine interfaces that restore motor function, the performance of prosthetic fingers and limbs has yet to mimic native function. The algorithm that converts brain signals to a control signal for the prosthetic device is one of the limitations in achieving rapid and realistic finger movements. To achieve more realistic finger movements, we developed a shallow feed-forward neural network to decode real-time two-degree-of-freedom finger movements in two adult male rhesus macaques. Using a two-step training method, a recalibrated feedback intention-trained (ReFIT) neural network is introduced to further improve performance. In 7 days of testing across two animals, neural network decoders, with higher-velocity and more natural appearing finger movements, achieved a 36% increase in throughput over the ReFIT Kalman filter, which represents the current standard. The neural network decoders introduced herein demonstrate real-time decoding of continuous movements at a level superior to the current state-of-the-art and could provide a starting point to using neural networks for the development of more naturalistic brain-controlled prostheses.
Collapse
Affiliation(s)
- Matthew S. Willsey
- grid.214458.e0000000086837370Department of Neurosurgery, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA
| | - Samuel R. Nason-Tomaszewski
- grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA
| | - Scott R. Ensel
- grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA
| | - Hisham Temmar
- grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA
| | - Matthew J. Mender
- grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA
| | - Joseph T. Costello
- grid.214458.e0000000086837370Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI USA
| | - Parag G. Patil
- grid.214458.e0000000086837370Department of Neurosurgery, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Neuroscience Graduate Program, University of Michigan Medical School, Ann Arbor, MI USA ,grid.214458.e0000000086837370Department of Anesthesiology, University of Michigan, Ann Arbor, MI USA
| | - Cynthia A. Chestek
- grid.214458.e0000000086837370Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Neuroscience Graduate Program, University of Michigan Medical School, Ann Arbor, MI USA ,grid.214458.e0000000086837370Robotics Graduate Program, University of Michigan, Ann Arbor, MI USA ,grid.214458.e0000000086837370Biointerfaces Institute, University of Michigan, Ann Arbor, MI USA
| |
Collapse
|
23
|
An H, Nason-Tomaszewski SR, Lim J, Kwon K, Willsey MS, Patil PG, Kim HS, Sylvester D, Chestek CA, Blaauw D. A Power-Efficient Brain-Machine Interface System With a Sub-mw Feature Extraction and Decoding ASIC Demonstrated in Nonhuman Primates. IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS 2022; 16:395-408. [PMID: 35594208 PMCID: PMC9375520 DOI: 10.1109/tbcas.2022.3175926] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Intracortical brain-machine interfaces have shown promise for restoring function to people with paralysis, but their translation to portable and implantable devices is hindered by their high power consumption. Recent devices have drastically reduced power consumption compared to standard experimental brain-machine interfaces, but still require wired or wireless connections to computing hardware for feature extraction and inference. Here, we introduce a Neural Recording And Decoding (NeuRAD) application specific integrated circuit (ASIC) in 180 nm CMOS that can extract neural spiking features and predict two-dimensional behaviors in real-time. To reduce amplifier and feature extraction power consumption, the NeuRAD has a hardware accelerator for extracting spiking band power (SBP) from intracortical spiking signals and includes an M0 processor with a fixed-point Matrix Acceleration Unit (MAU) for efficient and flexible decoding. We validated device functionality by recording SBP from a nonhuman primate implanted with a Utah microelectrode array and predicting the one- and two-dimensional finger movements the monkey was attempting to execute in closed-loop using a steady-state Kalman filter (SSKF). Using the NeuRAD's real-time predictions, the monkey achieved 100% success rate and 0.82 s mean target acquisition time to control one-dimensional finger movements using just 581 μW. To predict two-dimensional finger movements, the NeuRAD consumed 588 μW to enable the monkey to achieve a 96% success rate and 2.4 s mean acquisition time. By employing SBP, ASIC brain-machine interfaces can close the gap to enable fully implantable therapies for people with paralysis.
Collapse
|
24
|
Costello JT, Nason SR, An H, Lee J, Mender MJ, Temmar H, Wallace DM, Lim J, Willsey MS, Patil PG, Jang T, Phillips JD, Kim HS, Blaauw D, Chestek CA. A low-power communication scheme for wireless, 1000 channel brain-machine interfaces. J Neural Eng 2022; 19. [PMID: 35613546 DOI: 10.1088/1741-2552/ac7352] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2022] [Accepted: 05/24/2022] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Brain-machine interfaces (BMIs) have the potential to restore motor function but are currently limited by electrode count and long-term recording stability. These challenges may be solved through the use of free-floating "motes" which wirelessly transmit recorded neural signals, if power consumption can be kept within safe levels when scaling to thousands of motes. Here, we evaluated a pulse-interval modulation (PIM) communication scheme for infrared (IR)-based motes that aims to reduce the wireless data rate and system power consumption. APPROACH To test PIM's ability to efficiently communicate neural information, we simulated the communication scheme in a real-time closed-loop BMI with non-human primates. Additionally, we performed circuit simulations of an IR-based 1000-mote system to calculate communication accuracy and total power consumption. MAIN RESULTS We found that PIM at 1kb/s per channel maintained strong correlations with true firing rate and matched online BMI performance of a traditional wired system. Closed-loop BMI tests suggest that lags as small as 30 ms can have significant performance effects. Finally, unlike other IR communication schemes, PIM is feasible in terms of power, and neural data can accurately be recovered on a receiver using 3mW for 1000 channels. SIGNIFICANCE These results suggest that PIM-based communication could significantly reduce power usage of wireless motes to enable higher channel-counts for high-performance BMIs.
Collapse
Affiliation(s)
- Joseph T Costello
- Electrical and Computer Engineering, University of Michigan, 2800 Plymouth Rd, B10, Ann Arbor, Michigan, 48109-1382, UNITED STATES
| | - Samuel R Nason
- Biomedical Engineering, University of Michigan, 2800 Plymouth Rd B10, Ann Arbor, Michigan, 48109-1382, UNITED STATES
| | - Hyochan An
- Electrical and Computer Engineering, University of Michigan, 2800 Plymouth Rd B10, Ann Arbor, Michigan, 48109-1382, UNITED STATES
| | - Jungho Lee
- Electrical and Computer Engineering, University of Michigan, 2800 Plymouth Rd B10, Ann Arbor, Michigan, 48109-1382, UNITED STATES
| | - Matthew J Mender
- Biomedical Engineering, University of Michigan, 2800 Plymouth Rd B10, Ann Arbor, Michigan, 48109-1382, UNITED STATES
| | - Hisham Temmar
- Biomedical Engineering, University of Michigan, 2800 Plymouth Rd B10, Ann Arbor, Michigan, 48109-1382, UNITED STATES
| | - Dylan M Wallace
- Robotics Institute, University of Michigan, 2800 Plymouth Rd B10, Ann Arbor, 48109-1382, UNITED STATES
| | - Jongyup Lim
- Electrical and Computer Engineering, University of Michigan, 2800 Plymouth Rd B10, Ann Arbor, Michigan, 48109-1382, UNITED STATES
| | - Matthew S Willsey
- Department of Neurosurgery, University of Michigan Medical School, 1500 E Medical Center Drive, Ann Arbor, 48109-0624, UNITED STATES
| | - Parag G Patil
- Neurosurgery, University of Michigan, 1500 E Medical Center Drive, Ann Arbor, Michigan, 48109, UNITED STATES
| | - Taekwang Jang
- Department of Information Technology and Electrical Engineering, ETH Zurich, ETH Zurich, Rämistrasse 101, Zurich, 8092, SWITZERLAND
| | - Jamie Dean Phillips
- Department of Electrical and Computer Engineering, University of Delaware, Evans Hall, 139 The Green,, Newark, Delaware, 19716-5600, UNITED STATES
| | - Hun-Seok Kim
- Electrical and Computer Engineering, University of Michigan, 2800 Plymouth Rd B10, Ann Arbor, Michigan, 48109-1382, UNITED STATES
| | - David Blaauw
- University of Michigan, 2800 Plymouth Rd B10, Ann Arbor, Michigan, 48109-1382, UNITED STATES
| | - Cynthia A Chestek
- Biomedical Engineering, University of Michigan, 2800 Plymouth Rd B10, Ann Arbor, Michigan, 48109-1382, UNITED STATES
| |
Collapse
|