Rapid calibration of an intracortical brain-computer interface for people with tetraplegia

Unsupervised, piecewise linear decoding enables an accurate prediction of muscle activity in a multi-task brain computer interface

Objective.Creating an intracortical brain computer interface (iBCI) capable of seamless transitions between tasks and contexts would greatly enhance user experience. However, the nonlinearity in neural activity presents challenges to computing a global iBCI decoder. We aimed to develop a method that differs from a globally optimized decoder to address this issue.Approach.We devised an unsupervised approach that relies on the structure of a low-dimensional neural manifold to implement a piecewise linear decoder. We created a distinctive dataset in which monkeys performed a diverse set of tasks, some trained, others innate, while we recorded neural signals from the motor cortex (M1) and electromyographs (EMGs) from upper limb muscles. We used both linear and nonlinear dimensionality reduction techniques to discover neural manifolds and applied unsupervised algorithms to identify clusters within those spaces. Finally, we fit a linear decoder of EMG for each cluster. A specific decoder was activated corresponding to the cluster each new neural data point belonged to.Main results.We found clusters in the neural manifolds corresponding with the different tasks or task sub-phases. The performance of piecewise decoding improved as the number of clusters increased and plateaued gradually. With only two clusters it already outperformed a global linear decoder, and unexpectedly, it outperformed even a global recurrent neural network decoder with 10-12 clusters.Significance.This study introduced a computationally lightweight solution for creating iBCI decoders that can function effectively across a broad range of tasks. EMG decoding is particularly challenging, as muscle activity is used, under varying contexts, to control interaction forces and limb stiffness, as well as motion. The results suggest that a piecewise linear decoder can provide a good approximation to the nonlinearity between neural activity and motor outputs, a result of our increased understanding of the structure of neural manifolds in motor cortex.

Speech motor cortex enables BCI cursor control and click

Decoding neural activity from ventral (speech) motor cortex is known to enable high-performance speech brain-computer interface (BCI) control. It was previously unknown whether this brain area could also enable computer control via neural cursor and click, as is typically associated with dorsal (arm and hand) motor cortex. We recruited a clinical trial participant with ALS and implanted intracortical microelectrode arrays in ventral precentral gyrus (vPCG), which the participant used to operate a speech BCI in a prior study. We developed a cursor BCI driven by the participant's vPCG neural activity, and evaluated performance on a series of target selection tasks. The reported vPCG cursor BCI enabled rapidly-calibrating (40 seconds), accurate (2.90 bits per second) cursor control and click. The participant also used the BCI to control his own personal computer independently. These results suggest that placing electrodes in vPCG to optimize for speech decoding may also be a viable strategy for building a multi-modal BCI which enables both speech-based communication and computer control via cursor and click.

SoftBoMI: a non-invasive wearable body-machine interface for mapping movement of shoulder to commands

Objective.Customized human-machine interfaces for controlling assistive devices are vital in improving the self-help ability of upper limb amputees and tetraplegic patients. Given that most of them possess residual shoulder mobility, using it to generate commands to operate assistive devices can serve as a complementary approach to brain-computer interfaces.Approach.We propose a hybrid body-machine interface prototype that integrates soft sensors and an inertial measurement unit. This study introduces both a rule-based data decoding method and a user intent inference-based decoding method to map human shoulder movements into continuous commands. Additionally, by incorporating prior knowledge of the user's operational performance into a shared autonomy framework, we implement an adaptive switching command mapping approach. This approach enables seamless transitions between the two decoding methods, enhancing their adaptability across different tasks.Main results.The proposed method has been validated on individuals with cervical spinal cord injury, bilateral arm amputation, and healthy subjects through a series of center-out target reaching tasks and a virtual powered wheelchair driving task. The experimental results show that using both the soft sensors and the gyroscope exhibits the most well-rounded performance in intent inference. Additionally, the rule-based method demonstrates better dynamic performance for wheelchair operation, while the intent inference method is more accurate but has higher latency. Adaptive switching decoding methods offer the best adaptability by seamlessly transitioning between decoding methods for different tasks. Furthermore, we discussed the differences and characteristics among the various types of participants in the experiment.Significance.The proposed method has the potential to be integrated into clothing, enabling non-invasive interaction with assistive devices in daily life, and could serve as a tool for rehabilitation assessment in the future.

Reducing power requirements for high-accuracy decoding in iBCIs

Objective.Current intracortical brain-computer interfaces (iBCIs) rely predominantly on threshold crossings ('spikes') for decoding neural activity into a control signal for an external device. Spiking data can yield high accuracy online control during complex behaviors; however, its dependence on high-sampling-rate data collection can pose challenges. An alternative signal for iBCI decoding is the local field potential (LFP), a continuous-valued signal that can be acquired simultaneously with spiking activity. However, LFPs are seldom used alone for online iBCI control as their decoding performance has yet to achieve parity with spikes.Approach.Here, we present a strategy to improve the performance of LFP-based decoders by first training a neural dynamics model to use LFPs to reconstruct the firing rates underlying spiking data, and then decoding from the estimated rates. We test these models on previously-collected macaque data during center-out and random-target reaching tasks as well as data collected from a human iBCI participant during attempted speech.Main results.In all cases, training models from LFPs enables firing rate reconstruction with accuracy comparable to spiking-based dynamics models. In addition, LFP-based dynamics models enable decoding performance exceeding that of LFPs alone and approaching that of spiking-based models. In all applications except speech, LFP-based dynamics models also facilitate decoding accuracy exceeding that of direct decoding from spikes.Significance.Because LFP-based dynamics models operate on lower bandwidth and with lower sampling rate than spiking models, our findings indicate that iBCI devices can be designed to operate with lower power requirements than devices dependent on recorded spiking activity, without sacrificing high-accuracy decoding.

Implanted cortical neuroprosthetics for speech and movement restoration

Implanted cortical neuroprosthetics (ICNs) are medical devices developed to replace dysfunctional neural pathways by creating information exchange between the brain and a digital system which can facilitate interaction with the external world. Over the last decade, researchers have explored the application of ICNs for diverse conditions including blindness, aphasia, and paralysis. Both transcranial and endovascular approaches have been used to record neural activity in humans, and in a laboratory setting, high-performance decoding of the signals associated with speech intention has been demonstrated. Particular progress towards a device which can move into clinical practice has been made with ICNs focussed on the restoration of speech and movement. This article provides an overview of contemporary ICNs for speech and movement restoration, their mechanisms of action and the unique ethical challenges raised by the field.

Measuring instability in chronic human intracortical neural recordings towards stable, long-term brain-computer interfaces

Intracortical brain-computer interfaces (iBCIs) enable people with tetraplegia to gain intuitive cursor control from movement intentions. To translate to practical use, iBCIs should provide reliable performance for extended periods of time. However, performance begins to degrade as the relationship between kinematic intention and recorded neural activity shifts compared to when the decoder was initially trained. In addition to developing decoders to better handle long-term instability, identifying when to recalibrate will also optimize performance. We propose a method, "MINDFUL", to measure instabilities in neural data for useful long-term iBCI, without needing labels of user intentions. Longitudinal data were analyzed from two BrainGate2 participants with tetraplegia as they used fixed decoders to control a computer cursor spanning 142 days and 28 days, respectively. We demonstrate a measure of instability that correlates with changes in closed-loop cursor performance solely based on the recorded neural activity (Pearson r = 0.93 and 0.72, respectively). This result suggests a strategy to infer online iBCI performance from neural data alone and to determine when recalibration should take place for practical long-term use.

Unsupervised, piecewise linear decoding enables an accurate prediction of muscle activity in a multi-task brain computer interface

Objective

Creating an intracortical brain-computer interface (iBCI) capable of seamless transitions between tasks and contexts would greatly enhance user experience. However, the nonlinearity in neural activity presents challenges to computing a global iBCI decoder. We aimed to develop a method that differs from a globally optimized decoder to address this issue.

Approach

We devised an unsupervised approach that relies on the structure of a low-dimensional neural manifold to implement a piecewise linear decoder. We created a distinctive dataset in which monkeys performed a diverse set of tasks, some trained, others innate, while we recorded neural signals from the motor cortex (M1) and electromyographs (EMGs) from upper limb muscles. We used both linear and nonlinear dimensionality reduction techniques to discover neural manifolds and applied unsupervised algorithms to identify clusters within those spaces. Finally, we fit a linear decoder of EMG for each cluster. A specific decoder was activated corresponding to the cluster each new neural data point belonged to.

Main results

We found clusters in the neural manifolds corresponding with the different tasks or task sub-phases. The performance of piecewise decoding improved as the number of clusters increased and plateaued gradually. With only two clusters it already outperformed a global linear decoder, and unexpectedly, it outperformed even a global recurrent neural network (RNN) decoder with 10-12 clusters.

Significance

This study introduced a computationally lightweight solution for creating iBCI decoders that can function effectively across a broad range of tasks. EMG decoding is particularly challenging, as muscle activity is used, under varying contexts, to control interaction forces and limb stiffness, as well as motion. The results suggest that a piecewise linear decoder can provide a good approximation to the nonlinearity between neural activity and motor outputs, a result of our increased understanding of the structure of neural manifolds in motor cortex.

Motor somatotopy impacts imagery strategy success in human intracortical brain-computer interfaces

The notion of a somatotopically organized motor cortex, with movements of different body parts being controlled by spatially distinct areas of cortex, is well known. However, recent studies have challenged this notion and suggested a more distributed representation of movement control. This shift in perspective has significant implications, particularly when considering the implantation location of electrode arrays for intracortical brain-computer interfaces (iBCIs). We sought to evaluate whether the location of neural recordings from the precentral gyrus, and thus the underlying somatotopy, has any impact on the imagery strategies that can enable successful iBCI control. Three individuals with a spinal cord injury were enrolled in an ongoing clinical trial of an iBCI. Participants had two intracortical microelectrode arrays implanted in the arm and/or hand areas of the precentral gyrus based on presurgical functional imaging. Neural data were recorded while participants attempted to perform movements of the hand, wrist, elbow, and shoulder. We found that electrode arrays that were located more medially recorded significantly more activity during attempted proximal arm movements (elbow, shoulder) than did lateral arrays, which captured more activity related to attempted distal arm movements (hand, wrist). We also evaluated the relative contribution from the two arrays implanted in each participant to decoding accuracy during calibration of an iBCI decoder for translation and grasping tasks. For both task types, imagery strategy (e.g., reaching vs. wrist movements) had a significant impact on the relative contributions of each array to decoding. Overall, we found some evidence of broad tuning to arm and hand movements; however, there was a clear bias in the amount of information accessible about each movement type in spatially distinct areas of cortex. These results demonstrate that classical concepts of somatotopy can have real consequences for iBCI use, and highlight the importance of considering somatotopy when planning iBCI implantation.

A brain machine interface framework for exploring proactive control of smart environments

Brain machine interfaces (BMIs) can substantially improve the quality of life of elderly or disabled people. However, performing complex action sequences with a BMI system is onerous because it requires issuing commands sequentially. Fundamentally different from this, we have designed a BMI system that reads out mental planning activity and issues commands in a proactive manner. To demonstrate this, we recorded brain activity from freely-moving monkeys performing an instructed task and decoded it with an energy-efficient, small and mobile field-programmable gate array hardware decoder triggering real-time action execution on smart devices. Core of this is an adaptive decoding algorithm that can compensate for the day-by-day neuronal signal fluctuations with minimal re-calibration effort. We show that open-loop planning-ahead control is possible using signals from primary and pre-motor areas leading to significant time-gain in the execution of action sequences. This novel approach provides, thus, a stepping stone towards improved and more humane control of different smart environments with mobile brain machine interfaces.

A flexible intracortical brain-computer interface for typing using finger movements

Keyboard typing with finger movements is a versatile digital interface for users with diverse skills, needs, and preferences. Currently, such an interface does not exist for people with paralysis. We developed an intracortical brain-computer interface (BCI) for typing with attempted flexion/extension movements of three finger groups on the right hand, or both hands, and demonstrated its flexibility in two dominant typing paradigms. The first paradigm is "point-and-click" typing, where a BCI user selects one key at a time using continuous real-time control, allowing selection of arbitrary sequences of symbols. During cued character selection with this paradigm, a human research participant with paralysis achieved 30-40 selections per minute with nearly 90% accuracy. The second paradigm is "keystroke" typing, where the BCI user selects each character by a discrete movement without real-time feedback, often giving a faster speed for natural language sentences. With 90 cued characters per minute, decoding attempted finger movements and correcting errors using a language model resulted in more than 90% accuracy. Notably, both paradigms matched the state-of-the-art for BCI performance and enabled further flexibility by the simultaneous selection of multiple characters as well as efficient decoder estimation across paradigms. Overall, the high-performance interface is a step towards the wider accessibility of BCI technology by addressing unmet user needs for flexibility.

BRAND: a platform for closed-loop experiments with deep network models

Objective.Artificial neural networks (ANNs) are state-of-the-art tools for modeling and decoding neural activity, but deploying them in closed-loop experiments with tight timing constraints is challenging due to their limited support in existing real-time frameworks. Researchers need a platform that fully supports high-level languages for running ANNs (e.g. Python and Julia) while maintaining support for languages that are critical for low-latency data acquisition and processing (e.g. C and C++).Approach.To address these needs, we introduce the Backend for Realtime Asynchronous Neural Decoding (BRAND). BRAND comprises Linux processes, termednodes, which communicate with each other in agraphvia streams of data. Its asynchronous design allows for acquisition, control, and analysis to be executed in parallel on streams of data that may operate at different timescales. BRAND uses Redis, an in-memory database, to send data between nodes, which enables fast inter-process communication and supports 54 different programming languages. Thus, developers can easily deploy existing ANN models in BRAND with minimal implementation changes.Main results.In our tests, BRAND achieved <600 microsecond latency between processes when sending large quantities of data (1024 channels of 30 kHz neural data in 1 ms chunks). BRAND runs a brain-computer interface with a recurrent neural network (RNN) decoder with less than 8 ms of latency from neural data input to decoder prediction. In a real-world demonstration of the system, participant T11 in the BrainGate2 clinical trial (ClinicalTrials.gov Identifier: NCT00912041) performed a standard cursor control task, in which 30 kHz signal processing, RNN decoding, task control, and graphics were all executed in BRAND. This system also supports real-time inference with complex latent variable models like Latent Factor Analysis via Dynamical Systems.Significance.By providing a framework that is fast, modular, and language-agnostic, BRAND lowers the barriers to integrating the latest tools in neuroscience and machine learning into closed-loop experiments.

An accurate and rapidly calibrating speech neuroprosthesis

Brain-computer interfaces can enable rapid, intuitive communication for people with paralysis by transforming the cortical activity associated with attempted speech into text on a computer screen. Despite recent advances, communication with brain-computer interfaces has been restricted by extensive training data requirements and inaccurate word output. A man in his 40's with ALS with tetraparesis and severe dysarthria (ALSFRS-R = 23) was enrolled into the BrainGate2 clinical trial. He underwent surgical implantation of four microelectrode arrays into his left precentral gyrus, which recorded neural activity from 256 intracortical electrodes. We report a speech neuroprosthesis that decoded his neural activity as he attempted to speak in both prompted and unstructured conversational settings. Decoded words were displayed on a screen, then vocalized using text-to-speech software designed to sound like his pre-ALS voice. On the first day of system use, following 30 minutes of attempted speech training data, the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. On the second day, the size of the possible output vocabulary increased to 125,000 words, and, after 1.4 additional hours of training data, the neuroprosthesis achieved 90.2% accuracy. With further training data, the neuroprosthesis sustained 97.5% accuracy beyond eight months after surgical implantation. The participant has used the neuroprosthesis to communicate in self-paced conversations for over 248 hours. In an individual with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore naturalistic communication after a brief training period.

Measuring instability in chronic human intracortical neural recordings towards stable, long-term brain-computer interfaces

Intracortical brain-computer interfaces (iBCIs) enable people with tetraplegia to gain intuitive cursor control from movement intentions. To translate to practical use, iBCIs should provide reliable performance for extended periods of time. However, performance begins to degrade as the relationship between kinematic intention and recorded neural activity shifts compared to when the decoder was initially trained. In addition to developing decoders to better handle long-term instability, identifying when to recalibrate will also optimize performance. We propose a method to measure instability in neural data without needing to label user intentions. Longitudinal data were analyzed from two BrainGate2 participants with tetraplegia as they used fixed decoders to control a computer cursor spanning 142 days and 28 days, respectively. We demonstrate a measure of instability that correlates with changes in closed-loop cursor performance solely based on the recorded neural activity (Pearson r = 0.93 and 0.72, respectively). This result suggests a strategy to infer online iBCI performance from neural data alone and to determine when recalibration should take place for practical long-term use.

Brain-Computer Interfaces with Intracortical Implants for Motor and Communication Functions Compensation: Review of Recent Developments

Brain-computer interfaces allow the exchange of data between the brain and an external device, bypassing the muscular system. Clinical studies of invasive brain-computer interface technologies have been conducted for over 20 years. During this time, there has been a continuous improvement of approaches to neuronal signal processing in order to improve the quality of control of external devices. Currently, brain-computer interfaces with intracortical implants allow completely paralyzed patients to control robotic limbs for self-service, use a computer or a tablet, type text, and reproduce speech at an optimal speed. Studies of invasive brain-computer interfaces regularly provide new fundamental data on functioning of the central nervous system. In recent years, breakthrough discoveries and achievements have been annually made in this sphere. This review analyzes the results of clinical experiments of brain-computer interfaces with intracortical implants, provides information on the stages of this technology development, its main discoveries and achievements.

Intracortical brain-computer interfaces for improved motor function: a systematic review

In this systematic review, we address the status of intracortical brain-computer interfaces (iBCIs) applied to the motor cortex to improve function in patients with impaired motor ability. This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 Guidelines for Systematic Reviews. Risk Of Bias In Non-randomized Studies - of Interventions (ROBINS-I) and the Effective Public Health Practice Project (EPHPP) were used to assess bias and quality. Advances in iBCIs in the last two decades demonstrated the use of iBCI to activate limbs for functional tasks, achieve neural typing for communication, and other applications. However, the inconsistency of performance metrics employed by these studies suggests the need for standardization. Each study was a pilot clinical trial consisting of 1-4, majority male (64.28 %) participants, with most trials featuring participants treated for more than 12 months (55.55 %). The systems treated patients with various conditions: amyotrophic lateral sclerosis, stroke, spinocerebellar degeneration without cerebellar involvement, and spinal cord injury. All participants presented with tetraplegia at implantation and were implanted with microelectrode arrays via pneumatic insertion, with nearly all electrode locations solely at the precentral gyrus of the motor cortex (88.88 %). The development of iBCI devices using neural signals from the motor cortex to improve motor-impaired patients has enhanced the ability of these systems to return ability to their users. However, many milestones remain before these devices can prove their feasibility for recovery. This review summarizes the achievements and shortfalls of these systems and their respective trials.

Stable Decoding from a Speech BCI Enables Control for an Individual with ALS without Recalibration for 3 Months

Brain-computer interfaces (BCIs) can be used to control assistive devices by patients with neurological disorders like amyotrophic lateral sclerosis (ALS) that limit speech and movement. For assistive control, it is desirable for BCI systems to be accurate and reliable, preferably with minimal setup time. In this study, a participant with severe dysarthria due to ALS operates computer applications with six intuitive speech commands via a chronic electrocorticographic (ECoG) implant over the ventral sensorimotor cortex. Speech commands are accurately detected and decoded (median accuracy: 90.59%) throughout a 3-month study period without model retraining or recalibration. Use of the BCI does not require exogenous timing cues, enabling the participant to issue self-paced commands at will. These results demonstrate that a chronically implanted ECoG-based speech BCI can reliably control assistive devices over long time periods with only initial model training and calibration, supporting the feasibility of unassisted home use.

From monkeys to humans: observation-basedEMGbrain-computer interface decoders for humans with paralysis

Objective. Intracortical brain-computer interfaces (iBCIs) aim to enable individuals with paralysis to control the movement of virtual limbs and robotic arms. Because patients' paralysis prevents training a direct neural activity to limb movement decoder, most iBCIs rely on 'observation-based' decoding in which the patient watches a moving cursor while mentally envisioning making the movement. However, this reliance on observed target motion for decoder development precludes its application to the prediction of unobservable motor output like muscle activity. Here, we ask whether recordings of muscle activity from a surrogate individual performing the same movement as the iBCI patient can be used as target for an iBCI decoder.Approach. We test two possible approaches, each using data from a human iBCI user and a monkey, both performing similar motor actions. In one approach, we trained a decoder to predict the electromyographic (EMG) activity of a monkey from neural signals recorded from a human. We then contrast this to a second approach, based on the hypothesis that the low-dimensional 'latent' neural representations of motor behavior, known to be preserved across time for a given behavior, might also be preserved across individuals. We 'transferred' an EMG decoder trained solely on monkey data to the human iBCI user after using Canonical Correlation Analysis to align the human latent signals to those of the monkey.Main results. We found that both direct and transfer decoding approaches allowed accurate EMG predictions between two monkeys and from a monkey to a human.Significance. Our findings suggest that these latent representations of behavior are consistent across animals and even primate species. These methods are an important initial step in the development of iBCI decoders that generate EMG predictions that could serve as signals for a biomimetic decoder controlling motion and impedance of a prosthetic arm, or even muscle force directly through functional electrical stimulation.

Implementation of artificial intelligence and machine learning-based methods in brain-computer interaction

Brain-computer interfaces are used for direct two-way communication between the human brain and the computer. Brain signals contain valuable information about the mental state and brain activity of the examined subject. However, due to their non-stationarity and susceptibility to various types of interference, their processing, analysis and interpretation are challenging. For these reasons, the research in the field of brain-computer interfaces is focused on the implementation of artificial intelligence, especially in five main areas: calibration, noise suppression, communication, mental condition estimation, and motor imagery. The use of algorithms based on artificial intelligence and machine learning has proven to be very promising in these application domains, especially due to their ability to predict and learn from previous experience. Therefore, their implementation within medical technologies can contribute to more accurate information about the mental state of subjects, alleviate the consequences of serious diseases or improve the quality of life of disabled patients.

BRAND: A platform for closed-loop experiments with deep network models

Artificial neural networks (ANNs) are state-of-the-art tools for modeling and decoding neural activity, but deploying them in closed-loop experiments with tight timing constraints is challenging due to their limited support in existing real-time frameworks. Researchers need a platform that fully supports high-level languages for running ANNs (e.g., Python and Julia) while maintaining support for languages that are critical for low-latency data acquisition and processing (e.g., C and C++). To address these needs, we introduce the Backend for Realtime Asynchronous Neural Decoding (BRAND). BRAND comprises Linux processes, termed nodes , which communicate with each other in a graph via streams of data. Its asynchronous design allows for acquisition, control, and analysis to be executed in parallel on streams of data that may operate at different timescales. BRAND uses Redis to send data between nodes, which enables fast inter-process communication and supports 54 different programming languages. Thus, developers can easily deploy existing ANN models in BRAND with minimal implementation changes. In our tests, BRAND achieved <600 microsecond latency between processes when sending large quantities of data (1024 channels of 30 kHz neural data in 1-millisecond chunks). BRAND runs a brain-computer interface with a recurrent neural network (RNN) decoder with less than 8 milliseconds of latency from neural data input to decoder prediction. In a real-world demonstration of the system, participant T11 in the BrainGate2 clinical trial performed a standard cursor control task, in which 30 kHz signal processing, RNN decoding, task control, and graphics were all executed in BRAND. This system also supports real-time inference with complex latent variable models like Latent Factor Analysis via Dynamical Systems. By providing a framework that is fast, modular, and language-agnostic, BRAND lowers the barriers to integrating the latest tools in neuroscience and machine learning into closed-loop experiments.

Neural Decoding for Intracortical Brain-Computer Interfaces

Brain-computer interfaces have revolutionized the field of neuroscience by providing a solution for paralyzed patients to control external devices and improve the quality of daily life. To accurately and stably control effectors, it is important for decoders to recognize an individual's motor intention from neural activity either by noninvasive or intracortical neural recording. Intracortical recording is an invasive way of measuring neural electrical activity with high temporal and spatial resolution. Herein, we review recent developments in neural signal decoding methods for intracortical brain-computer interfaces. These methods have achieved good performance in analyzing neural activity and controlling robots and prostheses in nonhuman primates and humans. For more complex paradigms in motor rehabilitation or other clinical applications, there remains more space for further improvements of decoders.

Neural Plasticity in Sensorimotor Brain-Machine Interfaces

Brain-machine interfaces (BMIs) aim to treat sensorimotor neurological disorders by creating artificial motor and/or sensory pathways. Introducing artificial pathways creates new relationships between sensory input and motor output, which the brain must learn to gain dexterous control. This review highlights the role of learning in BMIs to restore movement and sensation, and discusses how BMI design may influence neural plasticity and performance. The close integration of plasticity in sensory and motor function influences the design of both artificial pathways and will be an essential consideration for bidirectional devices that restore both sensory and motor function.

Neuromorphic computing facilitates deep brain-machine fusion for high-performance neuroprosthesis

Brain-machine interfaces (BMI) have developed rapidly in recent years, but still face critical issues such as accuracy and stability. Ideally, a BMI system would be an implantable neuroprosthesis that would be tightly connected and integrated into the brain. However, the heterogeneity of brains and machines hinders deep fusion between the two. Neuromorphic computing models, which mimic the structure and mechanism of biological nervous systems, present a promising approach to developing high-performance neuroprosthesis. The biologically plausible property of neuromorphic models enables homogeneous information representation and computation in the form of discrete spikes between the brain and the machine, promoting deep brain-machine fusion and bringing new breakthroughs for high-performance and long-term usable BMI systems. Furthermore, neuromorphic models can be computed at ultra-low energy costs and thus are suitable for brain-implantable neuroprosthesis devices. The intersection of neuromorphic computing and BMI has great potential to lead the development of reliable, low-power implantable BMI devices and advance the development and application of BMI.

Interim Safety Profile From the Feasibility Study of the BrainGate Neural Interface System

BACKGROUND AND OBJECTIVES

Brain-computer interfaces (BCIs) are being developed to restore mobility, communication, and functional independence to people with paralysis. Though supported by decades of preclinical data, the safety of chronically implanted microelectrode array BCIs in humans is unknown. We report safety results from the prospective, open-label, nonrandomized BrainGate feasibility study (NCT00912041), the largest and longest-running clinical trial of an implanted BCI.

METHODS

Adults aged 18-75 years with quadriparesis from spinal cord injury, brainstem stroke, or motor neuron disease were enrolled through 7 clinical sites in the United States. Participants underwent surgical implantation of 1 or 2 microelectrode arrays in the motor cortex of the dominant cerebral hemisphere. The primary safety outcome was device-related serious adverse events (SAEs) requiring device explantation or resulting in death or permanently increased disability during the 1-year postimplant evaluation period. The secondary outcomes included the type and frequency of other adverse events and the feasibility of the BrainGate system for controlling a computer or other assistive technologies.

RESULTS

From 2004 to 2021, 14 adults enrolled in the BrainGate trial had devices surgically implanted. The average duration of device implantation was 872 days, yielding 12,203 days of safety experience. There were 68 device-related adverse events, including 6 device-related SAEs. The most common device-related adverse event was skin irritation around the percutaneous pedestal. There were no safety events that required device explantation, no unanticipated adverse device events, no intracranial infections, and no participant deaths or adverse events resulting in permanently increased disability related to the investigational device.

DISCUSSION

The BrainGate Neural Interface system has a safety record comparable with other chronically implanted medical devices. Given rapid recent advances in this technology and continued performance gains, these data suggest a favorable risk/benefit ratio in appropriately selected individuals to support ongoing research and development.

TRIAL REGISTRATION INFORMATION

ClinicalTrials.gov Identifier: NCT00912041.

CLASSIFICATION OF EVIDENCE

This study provides Class IV evidence that the neurosurgically placed BrainGate Neural Interface system is associated with a low rate of SAEs defined as those requiring device explantation, resulting in death, or resulting in permanently increased disability during the 1-year postimplant period.

Long-term unsupervised recalibration of cursor BCIs

Intracortical brain-computer interfaces (iBCIs) require frequent recalibration to maintain robust performance due to changes in neural activity that accumulate over time. Compensating for this nonstationarity would enable consistently high performance without the need for supervised recalibration periods, where users cannot engage in free use of their device. Here we introduce a hidden Markov model (HMM) to infer what targets users are moving toward during iBCI use. We then retrain the system using these inferred targets, enabling unsupervised adaptation to changing neural activity. Our approach outperforms the state of the art in large-scale, closed-loop simulations over two months and in closed-loop with a human iBCI user over one month. Leveraging an offline dataset spanning five years of iBCI recordings, we further show how recently proposed data distribution-matching approaches to recalibration fail over long time scales; only target-inference methods appear capable of enabling long-term unsupervised recalibration. Our results demonstrate how task structure can be used to bootstrap a noisy decoder into a highly-performant one, thereby overcoming one of the major barriers to clinically translating BCIs.

Modulating Brain Activity with Invasive Brain-Computer Interface: A Narrative Review

Brain-computer interface (BCI) can be used as a real-time bidirectional information gateway between the brain and machines. In particular, rapid progress in invasive BCI, propelled by recent developments in electrode materials, miniature and power-efficient electronics, and neural signal decoding technologies has attracted wide attention. In this review, we first introduce the concepts of neuronal signal decoding and encoding that are fundamental for information exchanges in BCI. Then, we review the history and recent advances in invasive BCI, particularly through studies using neural signals for controlling external devices on one hand, and modulating brain activity on the other hand. Specifically, regarding modulating brain activity, we focus on two types of techniques, applying electrical stimulation to cortical and deep brain tissues, respectively. Finally, we discuss the related ethical issues concerning the clinical application of this emerging technology.

Online adaptive group-wise sparse Penalized Recursive Exponentially Weighted N-way Partial Least Square for epidural intracranial BCI

Introduction

Motor Brain-Computer Interfaces (BCIs) create new communication pathways between the brain and external effectors for patients with severe motor impairments. Control of complex effectors such as robotic arms or exoskeletons is generally based on the real-time decoding of high-resolution neural signals. However, high-dimensional and noisy brain signals pose challenges, such as limitations in the generalization ability of the decoding model and increased computational demands.

Methods

The use of sparse decoders may offer a way to address these challenges. A sparsity-promoting penalization is a common approach to obtaining a sparse solution. BCI features are naturally structured and grouped according to spatial (electrodes), frequency, and temporal dimensions. Applying group-wise sparsity, where the coefficients of a group are set to zero simultaneously, has the potential to decrease computational time and memory usage, as well as simplify data transfer. Additionally, online closed-loop decoder adaptation (CLDA) is known to be an efficient procedure for BCI decoder training, taking into account neuronal feedback. In this study, we propose a new algorithm for online closed-loop training of group-wise sparse multilinear decoders using L _p -Penalized Recursive Exponentially Weighted N-way Partial Least Square (PREW-NPLS). Three types of sparsity-promoting penalization were explored using L _p with p = 0., 0.5, and 1.

Results

The algorithms were tested offline in a pseudo-online manner for features grouped by spatial dimension. A comparison study was conducted using an epidural ECoG dataset recorded from a tetraplegic individual during long-term BCI experiments for controlling a virtual avatar (left/right-hand 3D translation). Novel algorithms showed comparable or better decoding performance than conventional REW-NPLS, which was achieved with sparse models. The proposed algorithms are compatible with real-time CLDA.

Discussion

The proposed algorithm demonstrated good performance while drastically reducing the computational load and the memory consumption. However, the current study is limited to offline computation on data recorded with a single patient, with penalization restricted to the spatial domain only.

Changes in Primary Somatosensory Cortex Following Allogeneic Hand Transplantation or Autogenic Hand Replantation

Former amputees who undergo allogeneic hand transplantation or autogenic hand replantation (jointly, "hand restoration") present a unique opportunity to measure the range of post-deafferentation plastic changes in the nervous system, especially primary somatosensory cortex (S1). However, few such patients exist, and previous studies compared single cases to small groups of typical adults. Here, we studied 5 individuals (n = 8 sessions: a transplant with 2 sessions, a transplant with 3 sessions, and three replants with 1 session each). We used functional magnetic resonance imaging (fMRI) to measure S1 responsiveness to controlled pneumatic tactile stimulation delivered to each patient's left and right fingertips and lower face. These data were compared with responses acquired from typical adults (n = 29) and current unilateral amputees (n = 19). During stimulation of the affected hand, patients' affected S1 (contralateral to affected hand) responded to stimulation in a manner similar both to amputees and to typical adults. The presence of contralateral responses indicated grossly typical S1 function, but responses were universally at the low end of the range of typical variability. Patients' affected S1 showed substantial individual variability in responses to stimulation of the intact hand: while all patients fell within the range of typical adults, some patient sessions (4/8) had substantial ipsilateral responses similar to those exhibited by current amputees. Unlike hand restoration patients, current amputees exhibited substantial S1 reorganization compared to typical adults, including bilateral S1 responses to stimulation of the intact hand. In all three participant groups, we assessed tactile localization by measuring individuals' ability to identify the location of touch on the palm and fingers. Curiously, while transplant patients improved their tactile sensory localization over time, this was uncorrelated with changes in S1 responses to tactile stimuli. Overall, our results provide the first description of cortical responses to well-controlled tactile stimulation after hand restoration. Our case studies indicate that hand restoration patients show S1 function within the range of both typical adults and amputees, but with low-amplitude and individual-specific responses that indicate a wide range of potential cortical neurological changes following de-afferentation and re-afferentation.

Validation of a non-invasive, real-time, human-in-the-loop model of intracortical brain-computer interfaces

Objective. Despite the tremendous promise of invasive brain-computer interfaces (iBCIs), the associated study costs, risks, and ethical considerations limit the opportunity to develop and test the algorithms that decode neural activity into a user's intentions. Our goal was to address this challenge by designing an iBCI model capable of testing many human subjects in closed-loop.Approach. We developed an iBCI model that uses artificial neural networks (ANNs) to translate human finger movements into realistic motor cortex firing patterns, which can then be decoded in real time. We call the model the joint angle BCI, or jaBCI. jaBCI allows readily recruited, healthy subjects to perform closed-loop iBCI tasks using any neural decoder, preserving subjects' control-relevant short-latency error correction and learning dynamics.Main results. We validated jaBCI offline through emulated neuron firing statistics, confirming that emulated neural signals have firing rates, low-dimensional PCA geometry, and rotational jPCA dynamics that are quite similar to the actual neurons (recorded in monkey M1) on which we trained the ANN. We also tested jaBCI in closed-loop experiments, our single study examining roughly as many subjects as have been tested world-wide with iBCIs (n= 25). Performance was consistent with that of the paralyzed, human iBCI users with implanted intracortical electrodes. jaBCI allowed us to imitate the experimental protocols (e.g. the same velocity Kalman filter decoder and center-out task) and compute the same seven behavioral measures used in three critical studies.Significance. These encouraging results suggest the jaBCI's real-time firing rate emulation is a useful means to provide statistically robust sample sizes for rapid prototyping and optimization of decoding algorithms, the study of bi-directional learning in iBCIs, and improving iBCI control.

Neural Decoders Using Reinforcement Learning in Brain Machine Interfaces: A Technical Review

Creating flexible and robust brain machine interfaces (BMIs) is currently a popular topic of research that has been explored for decades in medicine, engineering, commercial, and machine-learning communities. In particular, the use of techniques using reinforcement learning (RL) has demonstrated impressive results but is under-represented in the BMI community. To shine more light on this promising relationship, this article aims to provide an exhaustive review of RL's applications to BMIs. Our primary focus in this review is to provide a technical summary of various algorithms used in RL-based BMIs to decode neural intention, without emphasizing preprocessing techniques on the neural signals and reward modeling for RL. We first organize the literature based on the type of RL methods used for neural decoding, and then each algorithm's learning strategy is explained along with its application in BMIs. A comparative analysis highlighting the similarities and uniqueness among neural decoders is provided. Finally, we end this review with a discussion about the current stage of RLBMIs including their limitations and promising directions for future research.

Kernel Temporal Differences for EEG-based Reinforcement Learning Brain Machine Interfaces

Kernel temporal differences (KTD) (λ) algorithm integrated in Q-learning (Q-KTD) has shown its applicability and feasibility for reinforcement learning brain machine interfaces (RLBMIs). RLBMI with its unique learning strategy based on trial-error allows continuous learning and adaptation in BMIs. Q-KTD has shown good performance in both open and closed-loop experiments for finding a proper mapping from neural intention to control commands of an external device. However, previous studies have been limited to intracortical BMIs where monkey's firing rates from primary motor cortex were used as inputs to the neural decoder. This study provides the first attempt to investigate Q-KTD algorithm's applicability in EEG-based RLBMIs. Two different publicly available EEG data sets are considered, we refer to them as Data set A and Data set B. EEG motor imagery tasks are integrated in a single step center-out reaching task, and we observe the open-loop RLBMI experiments reach 100% average success rates after sufficient learning experience. Data set A converges after approximately 20 epochs for raw features and Data set B shows convergence after approximately 40 epochs for both raw and Fourier transform features. Although there still exist challenges to overcome in EEG-based RLBMI using Q-KTD, including increasing the learning speed, and optimization of a continuously growing number of kernel units, the results encourage further investigation of Q-KTD in closed-loop RLBMIs using EEG. Clinical Relevance- This study supports feasibility of noninvasive EEG-based RLBMI implementations and addresses benefits and challenges of RLBMI using EEG.

Information sparseness in cortical microelectrode channels while decoding movement direction using an artificial neural network

Implanted microelectrode arrays can directly pick up electrode signals from the primary motor cortex (M1) during movement, and brain-machine interfaces (BMIs) can decode these signals to predict the directions of contemporaneous movements. However, it is not well known how much each individual input is responsible for the overall performance of a BMI decoder. In this paper, we seek to quantify how much each channel contributes to an artificial neural network (ANN)-based decoder, by measuring how much the removal of each individual channel degrades the accuracy of the output. If information on movement direction was equally distributed among channels, then the removal of one would have a minimal effect on decoder accuracy. On the other hand, if that information was distributed sparsely, then the removal of specific information-rich channels would significantly lower decoder accuracy. We found that for most channels, their removal did not significantly affect decoder performance. However, for a subset of channels (16 out of 61), removing them significantly reduced the decoder accuracy. This suggests that information is not uniformly distributed among the recording channels. We propose examining these channels further to optimize BMIs more effectively, as well as understand how M1 functions at the neuronal level.

Learned Motor Patterns Are Replayed in Human Motor Cortex during Sleep

Consolidation of memory is believed to involve offline replay of neural activity. While amply demonstrated in rodents, evidence for replay in humans, particularly regarding motor memory, is less compelling. To determine whether replay occurs after motor learning, we sought to record from motor cortex during a novel motor task and subsequent overnight sleep. A 36-year-old man with tetraplegia secondary to cervical spinal cord injury enrolled in the ongoing BrainGate brain-computer interface pilot clinical trial had two 96-channel intracortical microelectrode arrays placed chronically into left precentral gyrus. Single- and multi-unit activity was recorded while he played a color/sound sequence matching memory game. Intended movements were decoded from motor cortical neuronal activity by a real-time steady-state Kalman filter that allowed the participant to control a neurally driven cursor on the screen. Intracortical neural activity from precentral gyrus and 2-lead scalp EEG were recorded overnight as he slept. When decoded using the same steady-state Kalman filter parameters, intracortical neural signals recorded overnight replayed the target sequence from the memory game at intervals throughout at a frequency significantly greater than expected by chance. Replay events occurred at speeds ranging from 1 to 4 times as fast as initial task execution and were most frequently observed during slow-wave sleep. These results demonstrate that recent visuomotor skill acquisition in humans may be accompanied by replay of the corresponding motor cortex neural activity during sleep.SIGNIFICANCE STATEMENT Within cortex, the acquisition of information is often followed by the offline recapitulation of specific sequences of neural firing. Replay of recent activity is enriched during sleep and may support the consolidation of learning and memory. Using an intracortical brain-computer interface, we recorded and decoded activity from motor cortex as a human research participant performed a novel motor task. By decoding neural activity throughout subsequent sleep, we find that neural sequences underlying the recently practiced motor task are repeated throughout the night, providing direct evidence of replay in human motor cortex during sleep. This approach, using an optimized brain-computer interface decoder to characterize neural activity during sleep, provides a framework for future studies exploring replay, learning, and memory.

Preserved cortical somatotopic and motor representations in tetraplegic humans

A rich literature has documented changes in cortical representations of the body in somatosensory and motor cortex. Recent clinical studies of brain-machine interfaces designed to assist paralyzed patients have afforded the opportunity to record from and stimulate human somatosensory, motor, and action-related areas of the posterior parietal cortex. These studies show considerable preserved structure in the cortical somato-motor system. Motor cortex can immediately control assistive devices, stimulation of somatosensory cortex produces sensations in an orderly somatotopic map, and the posterior parietal cortex shows a high-dimensional representation of cognitive action variables. These results are strikingly similar to what would be expected in a healthy subject, demonstrating considerable stability of adult cortex even after severe injury and despite potential plasticity-induced new activations within the same region of cortex. Clinically, these results emphasize the importance of targeting cortical areas for BMI control signals that are consistent with their normal functional role.

Neuromotor prosthetic to treat stroke-related paresis: N-of-1 trial

Background

Functional recovery of arm movement typically plateaus following a stroke, leaving chronic motor deficits. Brain-computer interfaces (BCI) may be a potential treatment for post-stroke deficits

Methods

In this n-of-1 trial (NCT03913286), a person with chronic subcortical stroke with upper-limb motor impairment used a powered elbow-wrist-hand orthosis that opened and closed the affected hand using cortical activity, recorded from a percutaneous BCI comprised of four microelectrode arrays implanted in the ipsilesional precentral gyrus, based on decoding of spiking patterns and high frequency field potentials generated by imagined hand movements. The system was evaluated in a home setting for 12 weeks

Results

Robust single unit activity, modulating with attempted or imagined movement, was present throughout the precentral gyrus. The participant acquired voluntary control over a hand-orthosis, achieving 10 points on the Action Research Arm Test using the BCI, compared to 0 without any device, and 5 using myoelectric control. Strength, spasticity, the Fugl-Meyer scores improved.

Conclusions

We demonstrate in a human being that ensembles of individual neurons in the cortex overlying a chronic supratentorial, subcortical stroke remain active and engaged in motor representation and planning and can be used to electrically bypass the stroke and promote limb function. The participant’s ability to rapidly acquire control over otherwise paralyzed hand opening, more than 18 months after a stroke, may justify development of a fully implanted movement restoration system to expand the utility of fully implantable BCI to a clinical population that numbers in the tens of millions worldwide.

Stroke is a restriction of blood flow to part of the brain and can lead to chronic issues with a person’s ability to control the limbs. The aim of this study was to see if a new type of device could restore movement in a person with arm weakness due to a stroke that occurred a year earlier. In our trial, a sensor was implanted into the surface of the brain, near the site of the stroke, and was connected to a computer that generated a command to open and close the hand with a motorized brace worn on the hand. This person was able to use their own brain activity to trigger the brace and pick up and move objects. This research could support the development of similar medical devices to restore movement in people who have had strokes.

Serruya et al. test in an N-of-1 trial whether a wearable, powered exoskeletal orthosis, driven by a percutaneous, implanted brain–computer interface can restore voluntary upper extremity function following chronic hemiparesis subsequent to a cerebral subcortical stroke. Using this approach, voluntary opening of the paralyzed hand is restored.

The science and engineering behind sensitized brain-controlled bionic hands

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

An adaptive closed-loop ECoG decoder for long-term and stable bimanual control of an exoskeleton by a tetraplegic

OBJECTIVE

The article aims at addressing 2 challenges to step motor BCI out of laboratories: asynchronous control of complex bimanual effectors with large numbers of degrees of freedom, using chronic and safe recorders, and the decoding performance stability over time without frequent decoder recalibration.

APPROACH

Closed-loop adaptive/incremental decoder training is one strategy to create a model stable over time. Adaptive decoders update their parameters with new incoming data, optimizing the model parameters in real time. It allows cross-session training with multiple recording conditions during closed loop BCI experiments. In the article, an adaptive tensor-based Recursive Exponentially Weighted Markov-Switching multi-Linear Model (REW-MSLM) decoder is proposed. REW-MSLM uses a Mixture of Expert (ME) architecture, mixing or switching independent decoders (experts) according to the probability estimated by a "gating" model. A Hidden Markov model approach is employed as gating model to improve the decoding robustness and to provide strong idle state support. The ME architecture fits the multi-limb paradigm associating an expert to a particular limb or action.

MAIN RESULTS

Asynchronous control of an exoskeleton by a tetraplegic patient using a chronically implanted epidural electrocorticography (EpiCoG) recorder is reported. The stable over a period of 6 months (without decoder recalibration) 8-dimensional alternative bimanual control of the exoskeleton and its virtual avatar is demonstrated.

SIGNIFICANCE

Based on the long-term (>36 months) chronic bilateral epidural ECoG recordings in a tetraplegic (ClinicalTrials.gov, NCT02550522), we addressed the poorly explored field of asynchronous bimanual BCI. The new decoder was designed to meet to several challenges: the high-dimensional control of a complex effector in experiments closer to real-world behaviour (point-to-point pursuit versus conventional center-out tasks), with the ability of the BCI system to act as a stand-alone device switching between idle and control states, and a stable performance over a long period of time without decoder recalibration.

Workshops of the Eighth International Brain-Computer Interface Meeting: BCIs: The Next Frontier

The Eighth International Brain-Computer Interface (BCI) Meeting was held June 7-9th, 2021 in a virtual format. The conference continued the BCI Meeting series' interactive nature with 21 workshops covering topics in BCI (also called brain-machine interface) research. As in the past, workshops covered the breadth of topics in BCI. Some workshops provided detailed examinations of specific methods, hardware, or processes. Others focused on specific BCI applications or user groups. Several workshops continued consensus building efforts designed to create BCI standards and increase the ease of comparisons between studies and the potential for meta-analysis and large multi-site clinical trials. Ethical and translational considerations were both the primary topic for some workshops or an important secondary consideration for others. The range of BCI applications continues to expand, with more workshops focusing on approaches that can extend beyond the needs of those with physical impairments. This paper summarizes each workshop, provides background information and references for further study, presents an overview of the discussion topics, and describes the conclusion, challenges, or initiatives that resulted from the interactions and discussion at the workshop.

Design a novel BCI for neurorehabilitation using concurrent LFP and EEG features: a case study

Brain-computer interfaces (BCI) that enables people with severe motor disabilities to use their brain signals for direct control of objects have attracted increased interest in rehabilitation. To date, no study has investigated feasibility of the BCI framework incorporating both intracortical and scalp signals. Methods: Concurrent local field potential (LFP) from the hand-knob area and scalp EEG were recorded in a paraplegic patient undergoing a spike-based close-loop neurorehabilitation training. Based upon multimodal spatio-spectral feature extraction and Naive Bayes classification, we developed, for the first time, a novel LFP-EEG-BCI for motor intention decoding. A transfer learning (TL) approach was employed to further improve the feasibility. The performance of the proposed LFP-EEG-BCI for four-class upper-limb motor intention decoding was assessed. Results: Using a decision fusion strategy, we showed that the LFP-EEG-BCI significantly (p <0.05) outperformed single modal BCI (LFP-BCI and EEG-BCI) in terms of decoding accuracy with the best performance achieved using regularized common spatial pattern features. Interrogation of feature characteristics revealed discriminative spatial and spectral patterns, which may lead to new insights for better understanding of brain dynamics during different motor imagery tasks and promote development of efficient decoding algorithms. Moreover, we showed that similar classification performance could be obtained with few training trials, therefore highlighting the efficacy of TL. Conclusion: The present findings demonstrated the superiority of the novel LFP-EEG-BCI in motor intention decoding. Significance: This work introduced a novel LFP-EEG-BCI that may lead to new directions for developing practical neurorehabilitation systems with high detection accuracy and multi-paradigm feasibility in clinical applications.

Tracking Fast Neural Adaptation by Globally Adaptive Point Process Estimation for Brain-Machine Interface

Brain-machine interfaces (BMIs) help the disabled restore body functions by translating neural activity into digital commands to control external devices. Neural adaptation, where the brain signals change in response to external stimuli or movements, plays an important role in BMIs. When subjects purely use neural activity to brain-control a prosthesis, some neurons will actively explore a new tuning property to accomplish the movement task. The prediction of this neural tuning property can help subjects adapt more efficiently to brain control and maintain a good decoding performance. Existing prediction methods track the slow change of the tuning property in the manual control, which is not suitable for the fast neural adaptation in brain control. In order to identify the active neurons in brain control and track their tuning property changes, we propose a globally adaptive point process method (GaPP) to estimate the neural modulation state from spike trains, decompose the states into the hyper preferred direction and reconstruct the kinematics in a dual-model framework. We implement the method on real data from rats performing a two-lever discrimination task under manual control and brain control. The results show our method successfully predicts the neural modulation state and identifies the neurons that become active in brain control. Compared to the existing method, ours tracks the fast changes of the hyper preferred direction from manual control to brain control more accurately and efficiently and reconstructs the kinematics better and faster.

Home Use of a Percutaneous Wireless Intracortical Brain-Computer Interface by Individuals With Tetraplegia

OBJECTIVE

Individuals with neurological disease or injury such as amyotrophic lateral sclerosis, spinal cord injury or stroke may become tetraplegic, unable to speak or even locked-in. For people with these conditions, current assistive technologies are often ineffective. Brain-computer interfaces are being developed to enhance independence and restore communication in the absence of physical movement. Over the past decade, individuals with tetraplegia have achieved rapid on-screen typing and point-and-click control of tablet apps using intracortical brain-computer interfaces (iBCIs) that decode intended arm and hand movements from neural signals recorded by implanted microelectrode arrays. However, cables used to convey neural signals from the brain tether participants to amplifiers and decoding computers and require expert oversight, severely limiting when and where iBCIs could be available for use. Here, we demonstrate the first human use of a wireless broadband iBCI.

METHODS

Based on a prototype system previously used in pre-clinical research, we replaced the external cables of a 192-electrode iBCI with wireless transmitters and achieved high-resolution recording and decoding of broadband field potentials and spiking activity from people with paralysis. Two participants in an ongoing pilot clinical trial completed on-screen item selection tasks to assess iBCI-enabled cursor control.

RESULTS

Communication bitrates were equivalent between cabled and wireless configurations. Participants also used the wireless iBCI to control a standard commercial tablet computer to browse the web and use several mobile applications. Within-day comparison of cabled and wireless interfaces evaluated bit error rate, packet loss, and the recovery of spike rates and spike waveforms from the recorded neural signals. In a representative use case, the wireless system recorded intracortical signals from two arrays in one participant continuously through a 24-hour period at home.

SIGNIFICANCE

Wireless multi-electrode recording of broadband neural signals over extended periods introduces a valuable tool for human neuroscience research and is an important step toward practical deployment of iBCI technology for independent use by individuals with paralysis. On-demand access to high-performance iBCI technology in the home promises to enhance independence and restore communication and mobility for individuals with severe motor impairment.

A Nonlinear Maximum Correntropy Information Filter for High-Dimensional Neural Decoding

Neural signal decoding is a critical technology in brain machine interface (BMI) to interpret movement intention from multi-neural activity collected from paralyzed patients. As a commonly-used decoding algorithm, the Kalman filter is often applied to derive the movement states from high-dimensional neural firing observation. However, its performance is limited and less effective for noisy nonlinear neural systems with high-dimensional measurements. In this paper, we propose a nonlinear maximum correntropy information filter, aiming at better state estimation in the filtering process for a noisy high-dimensional measurement system. We reconstruct the measurement model between the high-dimensional measurements and low-dimensional states using the neural network, and derive the state estimation using the correntropy criterion to cope with the non-Gaussian noise and eliminate large initial uncertainty. Moreover, analyses of convergence and robustness are given. The effectiveness of the proposed algorithm is evaluated by applying it on multiple segments of neural spiking data from two rats to interpret the movement states when the subjects perform a two-lever discrimination task. Our results demonstrate better and more robust state estimation performance when compared with other filters.

Hierarchical Dynamical Model for Multiple Cortical Neural Decoding

Motor brain machine interfaces (BMIs) interpret neural activities from motor-related cortical areas in the brain into movement commands to control a prosthesis. As the subject adapts to control the neural prosthesis, the medial prefrontal cortex (mPFC), upstream of the primary motor cortex (M1), is heavily involved in reward-guided motor learning. Thus, considering mPFC and M1 functionality within a hierarchical structure could potentially improve the effectiveness of BMI decoding while subjects are learning. The commonly used Kalman decoding method with only one simple state model may not be able to represent the multiple brain states that evolve over time as well as along the neural pathway. In addition, the performance of Kalman decoders degenerates in heavy-tailed nongaussian noise, which is usually generated due to the nonlinear neural system or influences of movement-related noise in online neural recording. In this letter, we propose a hierarchical model to represent the brain states from multiple cortical areas that evolve along the neural pathway. We then introduce correntropy theory into the hierarchical structure to address the heavy-tailed noise existing in neural recordings. We test the proposed algorithm on in vivo recordings collected from the mPFC and M1 of two rats when the subjects were learning to perform a lever-pressing task. Compared with the classic Kalman filter, our results demonstrate better movement decoding performance due to the hierarchical structure that integrates the past failed trial information over multisite recording and the combination with correntropy criterion to deal with noisy heavy-tailed neural recordings.

Auditory cues reveal intended movement information in middle frontal gyrus neuronal ensemble activity of a person with tetraplegia

Intracortical brain-computer interfaces (iBCIs) allow people with paralysis to directly control assistive devices using neural activity associated with the intent to move. Realizing the full potential of iBCIs critically depends on continued progress in understanding how different cortical areas contribute to movement control. Here we present the first comparison between neuronal ensemble recordings from the left middle frontal gyrus (MFG) and precentral gyrus (PCG) of a person with tetraplegia using an iBCI. As expected, PCG was more engaged in selecting and generating intended movements than in earlier perceptual stages of action planning. By contrast, MFG displayed movement-related information during the sensorimotor processing steps preceding the appearance of the action plan in PCG, but only when the actions were instructed using auditory cues. These results describe a previously unreported function for neurons in the human left MFG in auditory processing contributing to motor control.

Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus

OBJECTIVE

To evaluate the potential of intracortical electrode array signals for brain-computer interfaces (BCIs) to restore lost speech, we measured the performance of decoders trained to discriminate a comprehensive basis set of 39 English phonemes and to synthesize speech sounds via a neural pattern matching method. We decoded neural correlates of spoken-out-loud words in the 'hand knob' area of precentral gyrus, a step toward the eventual goal of decoding attempted speech from ventral speech areas in patients who are unable to speak.

APPROACH

Neural and audio data were recorded while two BrainGate2 pilot clinical trial participants, each with two chronically-implanted 96-electrode arrays, spoke 420 different words that broadly sampled English phonemes. Phoneme onsets were identified from audio recordings, and their identities were then classified from neural features consisting of each electrode's binned action potential counts or high-frequency local field potential power. Speech synthesis was performed using the 'Brain-to-Speech' pattern matching method. We also examined two potential confounds specific to decoding overt speech: acoustic contamination of neural signals and systematic differences in labeling different phonemes' onset times.

MAIN RESULTS

A linear decoder achieved up to 29.3% classification accuracy (chance = 6%) across 39 phonemes, while an RNN classifier achieved 33.9% accuracy. Parameter sweeps indicated that performance did not saturate when adding more electrodes or more training data, and that accuracy improved when utilizing time-varying structure in the data. Microphonic contamination and phoneme onset differences modestly increased decoding accuracy, but could be mitigated by acoustic artifact subtraction and using a neural speech onset marker, respectively. Speech synthesis achieved r = 0.523 correlation between true and reconstructed audio.

SIGNIFICANCE

The ability to decode speech using intracortical electrode array signals from a nontraditional speech area suggests that placing electrode arrays in ventral speech areas is a promising direction for speech BCIs.

Spatial-temporal aspects of continuous EEG-based neurorobotic control

Objective.The goal of this work is to identify the spatio-temporal facets of state-of-the-art electroencephalography (EEG)-based continuous neurorobotics that need to be addressed, prior to deployment in practical applications at home and in the clinic.Approach.Nine healthy human subjects participated in five sessions of one-dimensional (1D) horizontal (LR), 1D vertical (UD) and two-dimensional (2D) neural tracking from EEG. Users controlled a robotic arm and virtual cursor to continuously track a Gaussian random motion target using EEG sensorimotor rhythm modulation via motor imagery (MI) commands. Continuous control quality was analyzed in the temporal and spatial domains separately.Main results.Axis-specific errors during 2D tasks were significantly larger than during 1D counterparts. Fatigue rates were larger for control tasks with higher cognitive demand (LR, left- and right-hand MI) compared to those with lower cognitive demand (UD, both hands MI and rest). Additionally robotic arm and virtual cursor control exhibited equal tracking error during all tasks. However, further spatial error analysis of 2D control revealed a significant reduction in tracking quality that was dependent on the visual interference of the physical device. In fact, robotic arm performance was significantly greater than that of virtual cursor control when the users' sightlines were not obstructed.Significance.This work emphasizes the need for practical interfaces to be designed around real-world tasks of increased complexity. Here, the dependence of control quality on cognitive task demand emphasizes the need for decoders that facilitate the translation of 1D task mastery to 2D control. When device footprint was accounted for, the introduction of a physical robotic arm improved control quality, likely due to increased user engagement. In general, this work demonstrates the need to consider both the physical footprint of devices, the complexity of training tasks, and the synergy of control strategies during the development of neurorobotic control.

Classifying Intracortical Brain-Machine Interface Signal Disruptions Based on System Performance and Applicable Compensatory Strategies: A Review

Brain-machine interfaces (BMIs) record and translate neural activity into a control signal for assistive or other devices. Intracortical microelectrode arrays (MEAs) enable high degree-of-freedom BMI control for complex tasks by providing fine-resolution neural recording. However, chronically implanted MEAs are subject to a dynamic in vivo environment where transient or systematic disruptions can interfere with neural recording and degrade BMI performance. Typically, neural implant failure modes have been categorized as biological, material, or mechanical. While this categorization provides insight into a disruption's causal etiology, it is less helpful for understanding degree of impact on BMI function or possible strategies for compensation. Therefore, we propose a complementary classification framework for intracortical recording disruptions that is based on duration of impact on BMI performance and requirement for and responsiveness to interventions: (1) Transient disruptions interfere with recordings on the time scale of minutes to hours and can resolve spontaneously; (2) Reversible disruptions cause persistent interference in recordings but the root cause can be remedied by an appropriate intervention; (3) Irreversible compensable disruptions cause persistent or progressive decline in signal quality, but their effects on BMI performance can be mitigated algorithmically; and (4) Irreversible non-compensable disruptions cause permanent signal loss that is not amenable to remediation or compensation. This conceptualization of intracortical BMI disruption types is useful for highlighting specific areas for potential hardware improvements and also identifying opportunities for algorithmic interventions. We review recording disruptions that have been reported for MEAs and demonstrate how biological, material, and mechanical mechanisms of disruption can be further categorized according to their impact on signal characteristics. Then we discuss potential compensatory protocols for each of the proposed disruption classes. Specifically, transient disruptions may be minimized by using robust neural decoder features, data augmentation methods, adaptive machine learning models, and specialized signal referencing techniques. Statistical Process Control methods can identify reparable disruptions for rapid intervention. In-vivo diagnostics such as impedance spectroscopy can inform neural feature selection and decoding models to compensate for irreversible disruptions. Additional compensatory strategies for irreversible disruptions include information salvage techniques, data augmentation during decoder training, and adaptive decoding methods to down-weight damaged channels.

Decoding movement direction from cortical microelectrode recordings using an LSTM-based neural network

Brain-machine interfaces (BMIs) allow individuals to communicate with computers using neural signals, and Kalman Filter (KF) are prevailingly used to decode movement directions from these neural signals. In this paper, we implemented a multi-layer long short-term memory (LSTM)based artificial neural network (ANN) for decoding BMI neural signals. We collected motor cortical neural signals from a nonhuman primate (NHP), implanted with microelectrode array (MEA) while performing a directional joystick task. Next, we compared the LSTM model in decoding the joystick trajectories from the neural signals against the prevailing KF model. The results showed that the LSTM model yielded significantly improved decoding accuracy measured by mean correlation coefficient (0.84, p < 10^-7) than the KF model (0.72). In addition, using a principal component analysis (PCA)-based dimensionality reduction technique yielded slightly deteriorated accuracies for both the LSTM (0.80) and KF (0.70) models, but greatly reduced the computational complexity. The results showed that the LSTM decoding model holds promise to improve decoding in BMIs for paralyzed humans.