Self-recalibrating classifiers for intracortical brain-computer interfaces

Measuring instability in chronic human intracortical neural recordings towards stable, long-term brain-computer interfaces

Intracortical brain-computer interfaces (iBCIs) enable people with tetraplegia to gain intuitive cursor control from movement intentions. To translate to practical use, iBCIs should provide reliable performance for extended periods of time. However, performance begins to degrade as the relationship between kinematic intention and recorded neural activity shifts compared to when the decoder was initially trained. In addition to developing decoders to better handle long-term instability, identifying when to recalibrate will also optimize performance. We propose a method to measure instability in neural data without needing to label user intentions. Longitudinal data were analyzed from two BrainGate2 participants with tetraplegia as they used fixed decoders to control a computer cursor spanning 142 days and 28 days, respectively. We demonstrate a measure of instability that correlates with changes in closed-loop cursor performance solely based on the recorded neural activity (Pearson r = 0.93 and 0.72, respectively). This result suggests a strategy to infer online iBCI performance from neural data alone and to determine when recalibration should take place for practical long-term use.

Flexible regulation of representations on a drifting manifold enables long-term stable complex neuroprosthetic control

The nervous system needs to balance the stability of neural representations with plasticity. It is unclear what is the representational stability of simple actions, particularly those that are well-rehearsed in humans, and how it changes in new contexts. Using an electrocorticography brain-computer interface (BCI), we found that the mesoscale manifold and relative representational distances for a repertoire of simple imagined movements were remarkably stable. Interestingly, however, the manifold's absolute location demonstrated day-to-day drift. Strikingly, representational statistics, especially variance, could be flexibly regulated to increase discernability during BCI control without somatotopic changes. Discernability strengthened with practice and was specific to the BCI, demonstrating remarkable contextual specificity. Accounting for drift, and leveraging the flexibility of representations, allowed neuroprosthetic control of a robotic arm and hand for over 7 months without recalibration. Our study offers insight into how electrocorticography can both track representational statistics across long periods and allow long-term complex neuroprosthetic control.

Neural Decoding for Intracortical Brain-Computer Interfaces

Brain-computer interfaces have revolutionized the field of neuroscience by providing a solution for paralyzed patients to control external devices and improve the quality of daily life. To accurately and stably control effectors, it is important for decoders to recognize an individual's motor intention from neural activity either by noninvasive or intracortical neural recording. Intracortical recording is an invasive way of measuring neural electrical activity with high temporal and spatial resolution. Herein, we review recent developments in neural signal decoding methods for intracortical brain-computer interfaces. These methods have achieved good performance in analyzing neural activity and controlling robots and prostheses in nonhuman primates and humans. For more complex paradigms in motor rehabilitation or other clinical applications, there remains more space for further improvements of decoders.

Long-term unsupervised recalibration of cursor BCIs

Intracortical brain-computer interfaces (iBCIs) require frequent recalibration to maintain robust performance due to changes in neural activity that accumulate over time. Compensating for this nonstationarity would enable consistently high performance without the need for supervised recalibration periods, where users cannot engage in free use of their device. Here we introduce a hidden Markov model (HMM) to infer what targets users are moving toward during iBCI use. We then retrain the system using these inferred targets, enabling unsupervised adaptation to changing neural activity. Our approach outperforms the state of the art in large-scale, closed-loop simulations over two months and in closed-loop with a human iBCI user over one month. Leveraging an offline dataset spanning five years of iBCI recordings, we further show how recently proposed data distribution-matching approaches to recalibration fail over long time scales; only target-inference methods appear capable of enabling long-term unsupervised recalibration. Our results demonstrate how task structure can be used to bootstrap a noisy decoder into a highly-performant one, thereby overcoming one of the major barriers to clinically translating BCIs.

Detection and discrimination of electrical stimuli from an upper limb cuff electrode inM. Mulatta

Objective.Peripheral nerve interfaces seek to restore nervous system function through electrical stimulation of peripheral nerves. In clinical use, these devices should function reliably for years or decades. In this study, we assessed evoked sensations from multi-channel cuff electrode stimulation in macaque monkeys up to 711 d post-implantation.Approach.Three trained macaque monkeys received multi-channel cuff electrode implants at the median or ulnar nerves in the upper arm. Electrical stimuli from the cuff interfaces evoked sensations, which we measured via standard psychophysical tasks. We adjusted pulse amplitude or pulse width for each block with various electrode channel configurations to examine the effects of stimulus parameterization on sensation. We measured detection thresholds and just-noticeable differences (JNDs) at irregular, near-daily intervals for several months using Bayesian inferencing from trial data. We examined data trends using classical models such as Weber's Law and the strength-duration relationship using linear regression.Main results.Detection thresholds were similar between blocks with pulse width modulation and blocks with pulse amplitude modulation when represented as charge per pulse, the product of the amplitude and the pulse width. Conversely, Weber fractions-calculated as the slope of the regression between JND charge values and reference stimulus charge-were significantly different between pulse width and pulse amplitude modulation blocks for the discrimination task.Significance.Weber fractions were lower in blocks with amplitude modulation than in blocks with pulse width modulation, suggesting that pulse amplitude modulation allows finer resolution of sensory encoding above threshold. Consequently, amplitude modulation may enable a greater dynamic range for sensory perception with neuroprosthetic devices.

The science and engineering behind sensitized brain-controlled bionic hands

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

Affective Neurofeedback Under Naturalistic Conditions: A Mini-Review of Current Achievements and Open Challenges

Affective neurofeedback training allows for the self-regulation of the putative circuits of emotion regulation. This approach has recently been studied as a possible additional treatment for psychiatric disorders, presenting positive effects in symptoms and behaviors. After neurofeedback training, a critical aspect is the transference of the learned self-regulation strategies to outside the laboratory and how to continue reinforcing these strategies in non-controlled environments. In this mini-review, we discuss the current achievements of affective neurofeedback under naturalistic setups. For this, we first provide a brief overview of the state-of-the-art for affective neurofeedback protocols. We then discuss virtual reality as a transitional step toward the final goal of "in-the-wild" protocols and current advances using mobile neurotechnology. Finally, we provide a discussion of open challenges for affective neurofeedback protocols in-the-wild, including topics such as convenience and reliability, environmental effects in attention and workload, among others.

Feature-Selection-Based Transfer Learning for Intracortical Brain-Machine Interface Decoding

The time spent in collecting current samples for decoder calibration and the computational burden brought by high-dimensional neural recordings remain two challenging problems in intracortical brain-machine interfaces (iBMIs). Decoder calibration optimization approaches have been proposed, and neuron selection methods have been used to reduce computational burden. However, few methods can solve both problems simultaneously. In this article, we present a symmetrical-uncertainty-based transfer learning (SUTL) method that combines transfer learning with feature selection. The proposed method uses symmetrical uncertainty to quantitatively measure three indices for feature selection: stationarity, importance and redundancy of the feature. By selecting the stationary features, the disparities between the historical data and current data can be diminished, and the historical data can be effectively used for decoder calibration, thereby reducing the demand for current data. After selecting the important and non-redundant features, only the channels corresponding to them need to work; thus, the computational burden is reduced. The proposed method was tested on neural data recorded from two rhesus macaques to decode the reaching position or grasping gesture. The results showed that the SUTL method diminished the disparities between the historical data and current data, while achieving superior decoding performance with the needs of only ten current samples each category, less than 10% the number of features and 30% the number of neural recording channels. Additionally, unlike most studies on iBMIs, feature selection was implemented instead of neuron selection, and the average decoding accuracy achieved by the former was 6.6% higher.

Multi-source domain adaptation for decoder calibration of intracortical brain-machine interface

OBJECTIVE

For nonstationarity of neural recordings, daily retraining is required in the decoder calibration of intracortical brain-machine interfaces (iBMIs). Domain adaptation has started to be applied in iBMIs to solve the problem of daily retraining by taking advantage of historical data. However, previous domain adaptation studies used only a single source domain, which might lead to performance instability. In this study, we proposed a multi-source domain adaptation algorithm, by fully utilizing the historical data, to achieve a better and more robust decoding performance while reducing the decoder calibration time.

APPROACH

The neural signals were recorded from two rhesus macaques using intracortical electrodes to decode the reaching and grasping movements. A principal component analysis-based multi-source domain adaptation algorithm was proposed to apply the feature transfer to diminish the disparities between the target domain and each source domain. Moreover, the multiple weighted sub-classifiers based on multi-source domain data and small current sample set were constructed to accomplish the decoding.

MAIN RESULTS

Our algorithm was able to make use of the multi-source domain data and achieve better and more robust decoding performance compared with other methods. Only a small current sample set was needed by our algorithm in order for the decoder calibration time to be effectively reduced.

SIGNIFICANCE

(1) The idea of the multi-source domain adaptation was introduced into the iBMIs to solve the problem of time consumption in the daily decoder retraining. (2) Instead of using only single-source domain data in the previous study, our algorithm made use of multi-day historical data, resulting in better and more robust decoding performance. (3) Our algorithm could be accomplished with only a small current sample set, and it can effectively reduce the decoder calibration time, which is important for further clinical applications.

Classifying Intracortical Brain-Machine Interface Signal Disruptions Based on System Performance and Applicable Compensatory Strategies: A Review

Brain-machine interfaces (BMIs) record and translate neural activity into a control signal for assistive or other devices. Intracortical microelectrode arrays (MEAs) enable high degree-of-freedom BMI control for complex tasks by providing fine-resolution neural recording. However, chronically implanted MEAs are subject to a dynamic in vivo environment where transient or systematic disruptions can interfere with neural recording and degrade BMI performance. Typically, neural implant failure modes have been categorized as biological, material, or mechanical. While this categorization provides insight into a disruption's causal etiology, it is less helpful for understanding degree of impact on BMI function or possible strategies for compensation. Therefore, we propose a complementary classification framework for intracortical recording disruptions that is based on duration of impact on BMI performance and requirement for and responsiveness to interventions: (1) Transient disruptions interfere with recordings on the time scale of minutes to hours and can resolve spontaneously; (2) Reversible disruptions cause persistent interference in recordings but the root cause can be remedied by an appropriate intervention; (3) Irreversible compensable disruptions cause persistent or progressive decline in signal quality, but their effects on BMI performance can be mitigated algorithmically; and (4) Irreversible non-compensable disruptions cause permanent signal loss that is not amenable to remediation or compensation. This conceptualization of intracortical BMI disruption types is useful for highlighting specific areas for potential hardware improvements and also identifying opportunities for algorithmic interventions. We review recording disruptions that have been reported for MEAs and demonstrate how biological, material, and mechanical mechanisms of disruption can be further categorized according to their impact on signal characteristics. Then we discuss potential compensatory protocols for each of the proposed disruption classes. Specifically, transient disruptions may be minimized by using robust neural decoder features, data augmentation methods, adaptive machine learning models, and specialized signal referencing techniques. Statistical Process Control methods can identify reparable disruptions for rapid intervention. In-vivo diagnostics such as impedance spectroscopy can inform neural feature selection and decoding models to compensate for irreversible disruptions. Additional compensatory strategies for irreversible disruptions include information salvage techniques, data augmentation during decoder training, and adaptive decoding methods to down-weight damaged channels.

Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain-Machine Interface

BACKGROUND

For the nonstationarity of neural recordings in intracortical brain-machine interfaces, daily retraining in a supervised manner is always required to maintain the performance of the decoder. This problem can be improved by using a reinforcement learning (RL) based self-recalibrating decoder. However, quickly exploring new knowledge while maintaining a good performance remains a challenge in RL-based decoders.

METHODS

To solve this problem, we proposed an attention-gated RL-based algorithm combining transfer learning, mini-batch, and weight updating schemes to accelerate the weight updating and avoid over-fitting. The proposed algorithm was tested on intracortical neural data recorded from two monkeys to decode their reaching positions and grasping gestures.

RESULTS

The decoding results showed that our proposed algorithm achieved an approximate 20% increase in classification accuracy compared to that obtained by the non-retrained classifier and even achieved better classification accuracy than the daily retraining classifier. Moreover, compared with a conventional RL method, our algorithm improved the accuracy by approximately 10% and the online weight updating speed by approximately 70 times.

CONCLUSIONS

This paper proposed a self-recalibrating decoder which achieved a good and robust decoding performance with fast weight updating and might facilitate its application in wearable device and clinical practice.

Stabilization of a brain-computer interface via the alignment of low-dimensional spaces of neural activity

The instability of neural recordings can render clinical brain-computer interfaces (BCIs) uncontrollable. Here, we show that the alignment of low-dimensional neural manifolds (low-dimensional spaces that describe specific correlation patterns between neurons) can be used to stabilize neural activity, thereby maintaining BCI performance in the presence of recording instabilities. We evaluated the stabilizer with non-human primates during online cursor control via intracortical BCIs in the presence of severe and abrupt recording instabilities. The stabilized BCIs recovered proficient control under different instability conditions and across multiple days. The stabilizer does not require knowledge of user intent and can outperform supervised recalibration. It stabilized BCIs even when neural activity contained little information about the direction of cursor movement. The stabilizer may be applicable to other neural interfaces and may improve the clinical viability of BCIs.

A neural network for online spike classification that improves decoding accuracy

Separating neural signals from noise can improve brain-computer interface performance and stability. However, most algorithms for separating neural action potentials from noise are not suitable for use in real time and have shown mixed effects on decoding performance. With the goal of removing noise that impedes online decoding, we sought to automate the intuition of human spike-sorters to operate in real time with an easily tunable parameter governing the stringency with which spike waveforms are classified. We trained an artificial neural network with one hidden layer on neural waveforms that were hand-labeled as either spikes or noise. The network output was a likelihood metric for each waveform it classified, and we tuned the network's stringency by varying the minimum likelihood value for a waveform to be considered a spike. Using the network's labels to exclude noise waveforms, we decoded remembered target location during a memory-guided saccade task from electrode arrays implanted in prefrontal cortex of rhesus macaque monkeys. The network classified waveforms in real time, and its classifications were qualitatively similar to those of a human spike-sorter. Compared with decoding with threshold crossings, in most sessions we improved decoding performance by removing waveforms with low spike likelihood values. Furthermore, decoding with our network's classifications became more beneficial as time since array implantation increased. Our classifier serves as a feasible preprocessing step, with little risk of harm, that could be applied to both off-line neural data analyses and online decoding.NEW & NOTEWORTHY Although there are many spike-sorting methods that isolate well-defined single units, these methods typically involve human intervention and have inconsistent effects on decoding. We used human classified neural waveforms as training data to create an artificial neural network that could be tuned to separate spikes from noise that impaired decoding. We found that this network operated in real time and was suitable for both off-line data processing and online decoding.

Inhibition of Long-Term Variability in Decoding Forelimb Trajectory Using Evolutionary Neural Networks With Error-Correction Learning

Objective: In brain machine interfaces (BMIs), the functional mapping between neural activities and kinematic parameters varied over time owing to changes in neural recording conditions. The variability in neural recording conditions might result in unstable long-term decoding performance. Relevant studies trained decoders with several days of training data to make them inherently robust to changes in neural recording conditions. However, these decoders might not be robust to changes in neural recording conditions when only a few days of training data are available. In time-series prediction and feedback control system, an error feedback was commonly adopted to reduce the effects of model uncertainty. This motivated us to introduce an error feedback to a neural decoder for dealing with the variability in neural recording conditions. Approach: We proposed an evolutionary constructive and pruning neural network with error feedback (ECPNN-EF) as a neural decoder. The ECPNN-EF with partially connected topology decoded the instantaneous firing rates of each sorted unit into forelimb movement of a rat. Furthermore, an error feedback was adopted as an additional input to provide kinematic information and thus compensate for changes in functional mapping. The proposed neural decoder was trained on data collected from a water reward-related lever-pressing task for a rat. The first 2 days of data were used to train the decoder, and the subsequent 10 days of data were used to test the decoder. Main Results: The ECPNN-EF under different settings was evaluated to better understand the impact of the error feedback and partially connected topology. The experimental results demonstrated that the ECPNN-EF achieved significantly higher daily decoding performance with smaller daily variability when using the error feedback and partially connected topology. Significance: These results suggested that the ECPNN-EF with partially connected topology could cope with both within- and across-day changes in neural recording conditions. The error feedback in the ECPNN-EF compensated for decreases in decoding performance when neural recording conditions changed. This mechanism made the ECPNN-EF robust against changes in functional mappings and thus improved the long-term decoding stability when only a few days of training data were available.

Sparse Ensemble Machine Learning to Improve Robustness of Long-Term Decoding in iBMIs

This paper presents a novel sparse ensemble based machine learning approach to enhance robustness of intracortical Brain Machine Interfaces (iBMIs) in the face of non-stationary distribution of input neural data across time. Each classifier in the ensemble is trained on a randomly sampled (with replacement) set of input channels. These sparse connections ensure that with a high chance, few of the base classifiers should be less affected by the variations in some of the recording channels. We have tested the generality of this technique on different base classifiers - linear discriminant analysis (LDA), support vector machine (SVM), extreme learning machine (ELM) and multilayer perceptron (MLP). Results show decoding accuracy improvements of up to ≈21 %, 13%, 19%, 10% in non-human primate (NHP) A and 7%, 9%, 7%, 9% in NHP B across test days while using the sparse ensemble approach over a single classifier model for LDA, SVM, ELM and MLP algorithms respectively. Furthermore, improvements of up to ≈7(14)%, 8(15)%, 9(19)%, 7(15)% in NHP A and 8(15)%, 12(20)%, 15(23)%, 12(19)% in NHP B over Random Forest (Long-short Term Memory) have been obtained by sparse ensemble LDA, SVM, ELM, MLP respectively.

Decoder calibration with ultra small current sample set for intracortical brain-machine interface

OBJECTIVE

Intracortical brain-machine interfaces (iBMIs) aim to restore efficient communication and movement ability for paralyzed patients. However, frequent recalibration is required for consistency and reliability, and every recalibration will require relatively large most current sample set. The aim in this study is to develop an effective decoder calibration method that can achieve good performance while minimizing recalibration time.

APPROACH

Two rhesus macaques implanted with intracortical microelectrode arrays were trained separately on movement and sensory paradigm. Neural signals were recorded to decode reaching positions or grasping postures. A novel principal component analysis-based domain adaptation (PDA) method was proposed to recalibrate the decoder with only ultra small current sample set by taking advantage of large historical data, and the decoding performance was compared with other three calibration methods for evaluation.

MAIN RESULTS

The PDA method closed the gap between historical and current data effectively, and made it possible to take advantage of large historical data for decoder recalibration in current data decoding. Using only ultra small current sample set (five trials of each category), the decoder calibrated using the PDA method could achieve much better and more robust performance in all sessions than using other three calibration methods in both monkeys.

SIGNIFICANCE

(1) By this study, transfer learning theory was brought into iBMIs decoder calibration for the first time. (2) Different from most transfer learning studies, the target data in this study were ultra small sample set and were transferred to the source data. (3) By taking advantage of historical data, the PDA method was demonstrated to be effective in reducing recalibration time for both movement paradigm and sensory paradigm, indicating a viable generalization. By reducing the demand for large current training data, this new method may facilitate the application of intracortical brain-machine interfaces in clinical practice.

Closed-Loop Continuous Hand Control via Chronic Recording of Regenerative Peripheral Nerve Interfaces

Loss of the upper limb imposes a devastating interruption to everyday life. Full restoration of natural arm control requires the ability to simultaneously control multiple degrees of freedom of the prosthetic arm and maintain that control over an extended period of time. Current clinically available myoelectric prostheses do not provide simultaneous control or consistency for transradial amputees. To address this issue, we have implemented a standard Kalman filter for continuous hand control using intramuscular electromyography (EMG) from both regenerative peripheral nerve interfaces (RPNI) and an intact muscle within non-human primates. Seven RPNIs and one intact muscle were implanted with indwelling bipolar intramuscular electrodes in two rhesus macaques. Following recuperations, function-specific EMG signals were recorded and then fed through the Kalman filter during a hand-movement behavioral task to continuously predict the monkey's finger position. We were able to reconstruct continuous finger movement offline with an average correlation of and a root mean squared error (RMSE) of 0.12 between actual and predicted position from two macaques. This finger movement prediction was also performed in real time to enable closed-loop neural control of a virtual hand. Compared with physical hand control, neural control performance was slightly slower but maintained an average target hit success rate of 96.70%. Recalibration longevity measurements maintained consistent average correlation over time but had a significant change in RMSE ( ). Additionally, extracted single units varied in amplitude by a factor of +18.65% and -25.85% compared with its mean. This is the first demonstration of chronic indwelling electrodes being used for continuous position control via the Kalman filter. Combining these analyses with our novel peripheral nerve interface, we believe that this demonstrates an important step in providing patients with more naturalistic control of their prosthetic limbs.

Implicit Grasp Force Representation in Human Motor Cortical Recordings

In order for brain-computer interface (BCI) systems to maximize functionality, users will need to be able to accurately modulate grasp force to avoid dropping heavy objects while also being able to handle fragile items. We present a case-study consisting of two experiments designed to identify whether intracortical recordings from the motor cortex of a person with tetraplegia could predict intended grasp force. In the first task, we were able classify neural responses to attempted grasps of four objects, each of which required similar grasp kinematics but different implicit grasp force targets, with 69% accuracy. In the second task, the subject attempted to move a virtual robotic arm in space to grasp a simple virtual object. For each trial, the subject was asked to grasp the virtual object with the force appropriate for one of the four objects from the first experiment, with the goal of measuring an implicit representation of grasp force. While the subject knew the grasp force during all phases of the trial, accurate classification was only achieved during active grasping, not while the hand moved to, transported, or released the object. In both tasks, misclassifications were most often to the object with an adjacent force requirement. In addition to the implications for understanding the representation of grasp force in motor cortex, these results are a first step toward creating intelligent algorithms to help BCI users grasp and manipulate a variety of objects that will be encountered in daily life.

Clinical Trial Identifier: NCT01894802 https://clinicaltrials.gov/ct2/show/NCT01894802.

Latent Factors and Dynamics in Motor Cortex and Their Application to Brain-Machine Interfaces

In the 1960s, Evarts first recorded the activity of single neurons in motor cortex of behaving monkeys (Evarts, 1968). In the 50 years since, great effort has been devoted to understanding how single neuron activity relates to movement. Yet these single neurons exist within a vast network, the nature of which has been largely inaccessible. With advances in recording technologies, algorithms, and computational power, the ability to study these networks is increasing exponentially. Recent experimental results suggest that the dynamical properties of these networks are critical to movement planning and execution. Here we discuss this dynamical systems perspective and how it is reshaping our understanding of the motor cortices. Following an overview of key studies in motor cortex, we discuss techniques to uncover the "latent factors" underlying observed neural population activity. Finally, we discuss efforts to use these factors to improve the performance of brain-machine interfaces, promising to make these findings broadly relevant to neuroengineering as well as systems neuroscience.

Meeting brain-computer interface user performance expectations using a deep neural network decoding framework

Brain-computer interface (BCI) neurotechnology has the potential to reduce disability associated with paralysis by translating neural activity into control of assistive devices^1-9. Surveys of potential end-users have identified key BCI system features^10-14, including high accuracy, minimal daily setup, rapid response times, and multifunctionality. These performance characteristics are primarily influenced by the BCI's neural decoding algorithm^1,15, which is trained to associate neural activation patterns with intended user actions. Here, we introduce a new deep neural network¹⁶ decoding framework for BCI systems enabling discrete movements that addresses these four key performance characteristics. Using intracortical data from a participant with tetraplegia, we provide offline results demonstrating that our decoder is highly accurate, sustains this performance beyond a year without explicit daily retraining by combining it with an unsupervised updating procedure^3,17-20, responds faster than competing methods⁸, and can increase functionality with minimal retraining by using a technique known as transfer learning²¹. We then show that our participant can use the decoder in real-time to reanimate his paralyzed forearm with functional electrical stimulation (FES), enabling accurate manipulation of three objects from the grasp and release test (GRT)²². These results demonstrate that deep neural network decoders can advance the clinical translation of BCI technology.

Robust Closed-Loop Control of a Cursor in a Person with Tetraplegia using Gaussian Process Regression

Intracortical brain computer interfaces can enable individuals with paralysis to control external devices through voluntarily modulated brain activity. Decoding quality has been previously shown to degrade with signal nonstationarities-specifically, the changes in the statistics of the data between training and testing data sets. This includes changes to the neural tuning profiles and baseline shifts in firing rates of recorded neurons, as well as nonphysiological noise. While progress has been made toward providing long-term user control via decoder recalibration, relatively little work has been dedicated to making the decoding algorithm more resilient to signal nonstationarities. Here, we describe how principled kernel selection with gaussian process regression can be used within a Bayesian filtering framework to mitigate the effects of commonly encountered nonstationarities. Given a supervised training set of (neural features, intention to move in a direction)-pairs, we use gaussian process regression to predict the intention given the neural data. We apply kernel embedding for each neural feature with the standard radial basis function. The multiple kernels are then summed together across each neural dimension, which allows the kernel to effectively ignore large differences that occur only in a single feature. The summed kernel is used for real-time predictions of the posterior mean and variance under a gaussian process framework. The predictions are then filtered using the discriminative Kalman filter to produce an estimate of the neural intention given the history of neural data. We refer to the multiple kernel approach combined with the discriminative Kalman filter as the MK-DKF. We found that the MK-DKF decoder was more resilient to nonstationarities frequently encountered in-real world settings yet provided similar performance to the currently used Kalman decoder. These results demonstrate a method by which neural decoding can be made more resistant to nonstationarities.

High performance communication by people with paralysis using an intracortical brain-computer interface

Brain-computer interfaces (BCIs) have the potential to restore communication for people with tetraplegia and anarthria by translating neural activity into control signals for assistive communication devices. While previous pre-clinical and clinical studies have demonstrated promising proofs-of-concept (Serruya et al., 2002; Simeral et al., 2011; Bacher et al., 2015; Nuyujukian et al., 2015; Aflalo et al., 2015; Gilja et al., 2015; Jarosiewicz et al., 2015; Wolpaw et al., 1998; Hwang et al., 2012; Spüler et al., 2012; Leuthardt et al., 2004; Taylor et al., 2002; Schalk et al., 2008; Moran, 2010; Brunner et al., 2011; Wang et al., 2013; Townsend and Platsko, 2016; Vansteensel et al., 2016; Nuyujukian et al., 2016; Carmena et al., 2003; Musallam et al., 2004; Santhanam et al., 2006; Hochberg et al., 2006; Ganguly et al., 2011; O'Doherty et al., 2011; Gilja et al., 2012), the performance of human clinical BCI systems is not yet high enough to support widespread adoption by people with physical limitations of speech. Here we report a high-performance intracortical BCI (iBCI) for communication, which was tested by three clinical trial participants with paralysis. The system leveraged advances in decoder design developed in prior pre-clinical and clinical studies (Gilja et al., 2015; Kao et al., 2016; Gilja et al., 2012). For all three participants, performance exceeded previous iBCIs (Bacher et al., 2015; Jarosiewicz et al., 2015) as measured by typing rate (by a factor of 1.4-4.2) and information throughput (by a factor of 2.2-4.0). This high level of performance demonstrates the potential utility of iBCIs as powerful assistive communication devices for people with limited motor function.Clinical Trial No: NCT00912041.

A Sliced Inverse Regression (SIR) Decoding the Forelimb Movement from Neuronal Spikes in the Rat Motor Cortex

Several neural decoding algorithms have successfully converted brain signals into commands to control a computer cursor and prosthetic devices. A majority of decoding methods, such as population vector algorithms (PVA), optimal linear estimators (OLE), and neural networks (NN), are effective in predicting movement kinematics, including movement direction, speed and trajectory but usually require a large number of neurons to achieve desirable performance. This study proposed a novel decoding algorithm even with signals obtained from a smaller numbers of neurons. We adopted sliced inverse regression (SIR) to predict forelimb movement from single-unit activities recorded in the rat primary motor (M1) cortex in a water-reward lever-pressing task. SIR performed weighted principal component analysis (PCA) to achieve effective dimension reduction for nonlinear regression. To demonstrate the decoding performance, SIR was compared to PVA, OLE, and NN. Furthermore, PCA and sequential feature selection (SFS) which are popular feature selection techniques were implemented for comparison of feature selection effectiveness. Among SIR, PVA, OLE, PCA, SFS, and NN decoding methods, the trajectories predicted by SIR (with a root mean square error, RMSE, of 8.47 ± 1.32 mm) was closer to the actual trajectories compared with those predicted by PVA (30.41 ± 11.73 mm), OLE (20.17 ± 6.43 mm), PCA (19.13 ± 0.75 mm), SFS (22.75 ± 2.01 mm), and NN (16.75 ± 2.02 mm). The superiority of SIR was most obvious when the sample size of neurons was small. We concluded that SIR sorted the input data to obtain the effective transform matrices for movement prediction, making it a robust decoding method for conditions with sparse neuronal information.

Making brain-machine interfaces robust to future neural variability

A major hurdle to clinical translation of brain–machine interfaces (BMIs) is that current decoders, which are trained from a small quantity of recent data, become ineffective when neural recording conditions subsequently change. We tested whether a decoder could be made more robust to future neural variability by training it to handle a variety of recording conditions sampled from months of previously collected data as well as synthetic training data perturbations. We developed a new multiplicative recurrent neural network BMI decoder that successfully learned a large variety of neural-to-kinematic mappings and became more robust with larger training data sets. Here we demonstrate that when tested with a non-human primate preclinical BMI model, this decoder is robust under conditions that disabled a state-of-the-art Kalman filter-based decoder. These results validate a new BMI strategy in which accumulated data history are effectively harnessed, and may facilitate reliable BMI use by reducing decoder retraining downtime.

Brain-machine interfaces (BMI) depend on algorithms to decode neural signals, but these decoders cope poorly with signal variability. Here, authors report a BMI decoder which circumvents these problems by using a large and perturbed training dataset to improve performance with variable neural signals.

Retrospectively supervised click decoder calibration for self-calibrating point-and-click brain-computer interfaces

Brain-computer interfaces (BCIs) aim to restore independence to people with severe motor disabilities by allowing control of acursor on a computer screen or other effectors with neural activity. However, physiological and/or recording-related nonstationarities in neural signals can limit long-term decoding stability, and it would be tedious for users to pause use of the BCI whenever neural control degrades to perform decoder recalibration routines. We recently demonstrated that a kinematic decoder (i.e. a decoder that controls cursor movement) can be recalibrated using data acquired during practical point-and-click control of the BCI by retrospectively inferring users' intended movement directions based on their subsequent selections. Here, we extend these methods to allow the click decoder to also be recalibrated using data acquired during practical BCI use. We retrospectively labeled neural data patterns as corresponding to "click" during all time bins in which the click log-likelihood (decoded using linear discriminant analysis, or LDA) had been above the click threshold that was used during real-time neural control. We labeled as "non-click" those periods that the kinematic decoder's retrospective target inference (RTI) heuristics determined to be consistent with intended cursor movement. Once these neural activity patterns were labeled, the click decoder was calibrated using standard supervised classifier training methods. Combined with real-time bias correction and baseline firing rate tracking, this set of "retrospectively labeled" decoder calibration methods enabled a BrainGate participant with amyotrophic lateral sclerosis (T9) to type freely across 11 research sessions spanning 29days, maintaining high-performance neural control over cursor movement and click without needing to interrupt virtual keyboard use for explicit calibration tasks. By eliminating the need for tedious calibration tasks with prescribed targets and pre-specified click times, this approach advances the potential clinical utility of intracortical BCIs for individuals with severe motor disability.

Big data challenges in decoding cortical activity in a human with quadriplegia to inform a brain computer interface

Recent advances in Brain Computer Interfaces (BCIs) have created hope that one day paralyzed patients will be able to regain control of their paralyzed limbs. As part of an ongoing clinical study, we have implanted a 96-electrode Utah array in the motor cortex of a paralyzed human. The array generates almost 3 million data points from the brain every second. This presents several big data challenges towards developing algorithms that should not only process the data in real-time (for the BCI to be responsive) but are also robust to temporal variations and non-stationarities in the sensor data. We demonstrate an algorithmic approach to analyze such data and present a novel method to evaluate such algorithms. We present our methodology with examples of decoding human brain data in real-time to inform a BCI.

New Perspectives on Neuroengineering and Neurotechnologies: NSF-DFG Workshop Report

GOAL

To identify and overcome barriers to creating new neurotechnologies capable of restoring both motor and sensory function in individuals with neurological conditions.

METHODS

This report builds upon the outcomes of a joint workshop between the US National Science Foundation and the German Research Foundation on New Perspectives in Neuroengineering and Neurotechnology convened in Arlington, VA, USA, November 13-14, 2014.

RESULTS

The participants identified key technological challenges for recording and manipulating neural activity, decoding, and interpreting brain data in the presence of plasticity, and early considerations of ethical and social issues pertinent to the adoption of neurotechnologies.

CONCLUSIONS

The envisaged progress in neuroengineering requires tightly integrated hardware and signal processing efforts, advances in understanding of physiological adaptations to closed-loop interactions with neural devices, and an open dialog with stakeholders and potential end-users of neurotechnology.

SIGNIFICANCE

The development of new neurotechnologies (e.g., bidirectional brain-computer interfaces) could significantly improve the quality of life of people living with the effects of brain or spinal cord injury, or other neurodegenerative diseases. Focused efforts aimed at overcoming the remaining barriers at the electrode tissue interface, developing implantable hardware with on-board computation, and refining stimulation methods to precisely activate neural tissue will advance both our understanding of brain function and our ability to treat currently intractable disorders of the nervous system.

The Evolution of Neuroprosthetic Interfaces

The ideal neuroprosthetic interface permits high-quality neural recording and stimulation of the nervous system while reliably providing clinical benefits over chronic periods. Although current technologies have made notable strides in this direction, significant improvements must be made to better achieve these design goals and satisfy clinical needs. This article provides an overview of the state of neuroprosthetic interfaces, starting with the design and placement of these interfaces before exploring the stimulation and recording platforms yielded from contemporary research. Finally, we outline emerging research trends in an effort to explore the potential next generation of neuroprosthetic interfaces.

Tracking Neural Modulation Depth by Dual Sequential Monte Carlo Estimation on Point Processes for Brain-Machine Interfaces

Classic brain-machine interface (BMI) approaches decode neural signals from the brain responsible for achieving specific motor movements, which subsequently command prosthetic devices. Brain activities adaptively change during the control of the neuroprosthesis in BMIs, where the alteration of the preferred direction and the modulation of the gain depth are observed. The static neural tuning models have been limited by fixed codes, resulting in a decay of decoding performance over the course of the movement and subsequent instability in motor performance. To achieve stable performance, we propose a dual sequential Monte Carlo adaptive point process method, which models and decodes the gradually changing modulation depth of individual neuron over the course of a movement. We use multichannel neural spike trains from the primary motor cortex of a monkey trained to perform a target pursuit task using a joystick. Our results show that our computational approach successfully tracks the neural modulation depth over time with better goodness-of-fit than classic static neural tuning models, resulting in smaller errors between the true kinematics and the estimations in both simulated and real data. Our novel decoding approach suggests that the brain may employ such strategies to achieve stable motor output, i.e., plastic neural tuning is a feature of neural systems. BMI users may benefit from this adaptive algorithm to achieve more complex and controlled movement outcomes.

Cortical neuroprosthetics from a clinical perspective

Recent pilot clinical studies have demonstrated that subjects with severe disorders of movement and communication can exert direct neural control over assistive devices using invasive Brain-Machine Interface (BMI) technology, also referred to as 'cortical neuroprosthetics'. These important proof-of-principle studies have generated great interest among those with disability and clinicians who provide general medical, neurological and/or rehabilitative care. Taking into account the perspective of providers who may be unfamiliar with the field, we first review the clinical goals and fundamentals of invasive BMI technology, and then briefly summarize the vast body of basic science research demonstrating its feasibility. We emphasize recent translational progress in the target clinical populations and discuss translational challenges and future directions.

Encoder-decoder optimization for brain-computer interfaces

Neuroprosthetic brain-computer interfaces are systems that decode neural activity into useful control signals for effectors, such as a cursor on a computer screen. It has long been recognized that both the user and decoding system can adapt to increase the accuracy of the end effector. Co-adaptation is the process whereby a user learns to control the system in conjunction with the decoder adapting to learn the user's neural patterns. We provide a mathematical framework for co-adaptation and relate co-adaptation to the joint optimization of the user's control scheme ("encoding model") and the decoding algorithm's parameters. When the assumptions of that framework are respected, co-adaptation cannot yield better performance than that obtainable by an optimal initial choice of fixed decoder, coupled with optimal user learning. For a specific case, we provide numerical methods to obtain such an optimized decoder. We demonstrate our approach in a model brain-computer interface system using an online prosthesis simulator, a simple human-in-the-loop pyschophysics setup which provides a non-invasive simulation of the BCI setting. These experiments support two claims: that users can learn encoders matched to fixed, optimal decoders and that, once learned, our approach yields expected performance advantages.

Combining decoder design and neural adaptation in brain-machine interfaces

Brain-machine interfaces (BMIs) aim to help people with paralysis by decoding movement-related neural signals into control signals for guiding computer cursors, prosthetic arms, and other assistive devices. Despite compelling laboratory experiments and ongoing FDA pilot clinical trials, system performance, robustness, and generalization remain challenges. We provide a perspective on how two complementary lines of investigation, that have focused on decoder design and neural adaptation largely separately, could be brought together to advance BMIs. This BMI paradigm should also yield new scientific insights into the function and dysfunction of the nervous system.

Performance sustaining intracortical neural prostheses

OBJECTIVE

Neural prostheses, or brain-machine interfaces, aim to restore efficient communication and movement ability to those suffering from paralysis. A major challenge these systems face is robust performance, particularly with aging signal sources. The aim in this study was to develop a neural prosthesis that could sustain high performance in spite of signal instability while still minimizing retraining time.

APPROACH

We trained two rhesus macaques implanted with intracortical microelectrode arrays 1-4 years prior to this study to acquire targets with a neurally-controlled cursor. We measured their performance via achieved bitrate (bits per second, bps). This task was repeated over contiguous days to evaluate the sustained performance across time.

MAIN RESULTS

We found that in the monkey with a younger (i.e., two year old) implant and better signal quality, a fixed decoder could sustain performance for a month at a rate of 4 bps, the highest achieved communication rate reported to date. This fixed decoder was evaluated across 22 months and experienced a performance decline at a rate of 0.24 bps yr(-1). In the monkey with the older (i.e., 3.5 year old) implant and poorer signal quality, a fixed decoder could not sustain performance for more than a few days. Nevertheless, performance in this monkey was maintained for two weeks without requiring additional online retraining time by utilizing prior days' experimental data. Upon analysis of the changes in channel tuning, we found that this stability appeared partially attributable to the cancelling-out of neural tuning fluctuations when projected to two-dimensional cursor movements.

SIGNIFICANCE

The findings in this study (1) document the highest-performing communication neural prosthesis in monkeys, (2) confirm and extend prior reports of the stability of fixed decoders, and (3) demonstrate a protocol for system stability under conditions where fixed decoders would otherwise fail. These improvements to decoder stability are important for minimizing training time and should make neural prostheses more practical to use.

Decoding methods for neural prostheses: where have we reached?

This article reviews advances in decoding methods for brain-machine interfaces (BMIs). Recent work has focused on practical considerations for future clinical deployment of prosthetics. This review is organized by open questions in the field such as what variables to decode, how to design neural tuning models, which neurons to select, how to design models of desired actions, how to learn decoder parameters during prosthetic operation, and how to adapt to changes in neural signals and neural tuning. The concluding discussion highlights the need to design and test decoders within the context of their expected use and the need to answer the question of how much control accuracy is good enough for a prosthetic.