High-performance brain-machine interface enabled by an adaptive optimal feedback-controlled point process decoder

Event detection and classification from multimodal time series with application to neural data

The detection of events in time-series data is a common signal-processing problem. When the data can be modeled as a known template signal with an unknown delay in Gaussian noise, detection of the template signal can be done with a traditional matched filter. However, in many applications, the event of interest is represented in multimodal data consisting of both Gaussian and point-process time series. Neuroscience experiments, for example, can simultaneously record multimodal neural signals such as local field potentials (LFPs), which can be modeled as Gaussian, and neuronal spikes, which can be modeled as point processes. Currently, no method exists for event detection from such multimodal data, and as such our objective in this work is to develop a method to meet this need. Here we address this challenge by developing the multimodal event detector (MED) algorithm which simultaneously estimates event times and classes. To do this, we write a multimodal likelihood function for Gaussian and point-process observations and derive the associated maximum likelihood estimator of simultaneous event times and classes. We additionally introduce a cross-modal scaling parameter to account for model mismatch in real datasets. We validate this method in extensive simulations as well as in a neural spike-LFP dataset recorded during an eye-movement task, where the events of interest are eye movements with unknown times and directions. We show that the MED can successfully detect eye movement onset and classify eye movement direction. Further, the MED successfully combines information across data modalities, with multimodal performance exceeding unimodal performance. This method can facilitate applications such as the discovery of latent events in multimodal neural population activity and the development of brain-computer interfaces for naturalistic settings without constrained tasks or prior knowledge of event times.

Post-stimulus encoding of decision confidence in EEG: toward a brain-computer interface for decision making

Objective.When making decisions, humans can evaluate how likely they are to be correct. If this subjective confidence could be reliably decoded from brain activity, it would be possible to build a brain-computer interface (BCI) that improves decision performance by automatically providing more information to the user if needed based on their confidence. But this possibility depends on whether confidence can be decoded right after stimulus presentation and before the response so that a corrective action can be taken in time. Although prior work has shown that decision confidence is represented in brain signals, it is unclear if the representation is stimulus-locked or response-locked, and whether stimulus-locked pre-response decoding is sufficiently accurate for enabling such a BCI.Approach.We investigate the neural correlates of confidence by collecting high-density electroencephalography (EEG) during a perceptual decision task with realistic stimuli. Importantly, we design our task to include a post-stimulus gap that prevents the confounding of stimulus-locked activity by response-locked activity and vice versa, and then compare with a task without this gap.Main results.We perform event-related potential and source-localization analyses. Our analyses suggest that the neural correlates of confidence are stimulus-locked, and that an absence of a post-stimulus gap could cause these correlates to incorrectly appear as response-locked. By preventing response-locked activity from confounding stimulus-locked activity, we then show that confidence can be reliably decoded from single-trial stimulus-locked pre-response EEG alone. We also identify a high-performance classification algorithm by comparing a battery of algorithms. Lastly, we design a simulated BCI framework to show that the EEG classification is accurate enough to build a BCI and that the decoded confidence could be used to improve decision making performance particularly when the task difficulty and cost of errors are high.Significance.Our results show feasibility of non-invasive EEG-based BCIs to improve human decision making.

Auxiliary controller design and performance comparative analysis in closed-loop brain-machine interface system

Brain-machine interface (BMI) can realize information interaction between the brain and external devices, and yet the control accuracy is limited by the change of electroencephalogram signals. The introduction of auxiliary controller can overcome the above problems, but the performance of different auxiliary controllers is quite different. Hence, in this paper, we comprehensively compare and analyze the performance of different auxiliary controllers to provide a theoretical basis for designing BMI system. The main work includes: (1) designing four kinds of auxiliary controllers based on simultaneous perturbation stochastic approximation-function approximator (SPSA-FA), iterative feedback tuning-PID (IFT-PID), model predictive control (MPC) and model-free control (MFC); (2) based on the model of improved single-joint information transmission, constructing the closed-loop BMI systems with the decoder-based Wiener filter; and (3) comparing their performance in the constructed closed-loop BMI systems for dynamic motion restoration. The results show that the order of tracking accuracy is MPC, IFT-PID, SPSA-FA, MFC, and the order of time consumed is opposite. A good control effectiveness is achieved at the expense of time, so a suitable auxiliary controller should be selected according to the actual requirements.

Decoding Kinematic Information From Primary Motor Cortex Ensemble Activities Using a Deep Canonical Correlation Analysis

The control of arm movements through intracortical brain–machine interfaces (BMIs) mainly relies on the activities of the primary motor cortex (M1) neurons and mathematical models that decode their activities. Recent research on decoding process attempts to not only improve the performance but also simultaneously understand neural and behavioral relationships. In this study, we propose an efficient decoding algorithm using a deep canonical correlation analysis (DCCA), which maximizes correlations between canonical variables with the non-linear approximation of mappings from neuronal to canonical variables via deep learning. We investigate the effectiveness of using DCCA for finding a relationship between M1 activities and kinematic information when non-human primates performed a reaching task with one arm. Then, we examine whether using neural activity representations from DCCA improves the decoding performance through linear and non-linear decoders: a linear Kalman filter (LKF) and a long short-term memory in recurrent neural networks (LSTM-RNN). We found that neural representations of M1 activities estimated by DCCA resulted in more accurate decoding of velocity than those estimated by linear canonical correlation analysis, principal component analysis, factor analysis, and linear dynamical system. Decoding with DCCA yielded better performance than decoding the original FRs using LSTM-RNN (6.6 and 16.0% improvement on average for each velocity and position, respectively; Wilcoxon rank sum test, p < 0.05). Thus, DCCA can identify the kinematics-related canonical variables of M1 activities, thus improving the decoding performance. Our results may help advance the design of decoding models for intracortical BMIs.

Rapid calibration of an intracortical brain-computer interface for people with tetraplegia

OBJECTIVE

Brain-computer interfaces (BCIs) can enable individuals with tetraplegia to communicate and control external devices. Though much progress has been made in improving the speed and robustness of neural control provided by intracortical BCIs, little research has been devoted to minimizing the amount of time spent on decoder calibration.

APPROACH

We investigated the amount of time users needed to calibrate decoders and achieve performance saturation using two markedly different decoding algorithms: the steady-state Kalman filter, and a novel technique using Gaussian process regression (GP-DKF).

MAIN RESULTS

Three people with tetraplegia gained rapid closed-loop neural cursor control and peak, plateaued decoder performance within 3 min of initializing calibration. We also show that a BCI-naïve user (T5) was able to rapidly attain closed-loop neural cursor control with the GP-DKF using self-selected movement imagery on his first-ever day of closed-loop BCI use, acquiring a target 37 s after initiating calibration.

SIGNIFICANCE

These results demonstrate the potential for an intracortical BCI to be used immediately after deployment by people with paralysis, without the need for user learning or extensive system calibration.

Bioactive Neuroelectronic Interfaces

Within the neural engineering field, next-generation implantable neuroelectronic interfaces are being developed using biologically-inspired and/or biologically-derived materials to improve upon the stability and functional lifetime of current interfaces. These technologies use biomaterials, bioactive molecules, living cells, or some combination of these, to promote host neuronal survival, reduce the foreign body response, and improve chronic device-tissue integration. This article provides a general overview of the different strategies, milestones, and evolution of bioactive neural interfaces including electrode material properties, biological coatings, and "decoration" with living cells. Another such biohybrid approach developed in our lab uses preformed implantable micro-tissue featuring long-projecting axonal tracts encased within carrier biomaterial micro-columns. These so-called "living electrodes" have been engineered with carefully tailored material, mechanical, and biological properties to enable natural, synaptic based modulation of specific host circuitry while ultimately being under computer control. This article provides an overview of these living electrodes, including design and fabrication, performance attributes, as well as findings to date characterizing in vitro and in vivo functionality.

Review: Human Intracortical Recording and Neural Decoding for Brain-Computer Interfaces

Brain-computer interfaces (BCIs) use neural information recorded from the brain for the voluntary control of external devices. The development of BCI systems has largely focused on improving functional independence for individuals with severe motor impairments, including providing tools for communication and mobility. In this review, we describe recent advances in intracortical BCI technology and provide potential directions for further research.

Multiscale brain-machine interface decoders

Brain-machine interfaces (BMI) have vastly used a single scale of neural activity, e.g., spikes or electrocorticography (ECoG), as their control signal. New technology allows for simultaneous recording of multiple scales of neural activity, from spikes to local field potentials (LFP) and ECoG. These advances introduce the new challenge of modeling and decoding multiple scales of neural activity jointly. Such multi-scale decoding is challenging for two reasons. First, spikes are discrete-valued and ECoG/LFP are continuous-valued, resulting in fundamental differences in statistical characteristics. Second, the time-scales of these signals are different, with spikes having a millisecond time-scale and ECoG/LFP having much slower time-scales on the order of tens of milliseconds. Here we develop a new multiscale modeling and decoding framework that addresses these challenges. Our multiscale decoder extracts information from ECoG/LFP in addition to spikes, while operating at the fast time-scale of the spikes. The multiscale decoder specializes to a Kalman filter (KF) or to a point process filter (PPF) when no spikes or ECoG/LFP are available, respectively. Using closed-loop BMI simulations, we show that compared to PPF decoding of spikes alone or KF decoding of LFP/ECoG alone, the multiscale decoder significantly improves the accuracy and error performance of BMI control and runs at the fast millisecond time-scale of the spikes. This new multiscale modeling and decoding framework has the potential to improve BMI control using simultaneous multiscale neural activity.

Feedback control policies employed by people using intracortical brain-computer interfaces

OBJECTIVE

When using an intracortical BCI (iBCI), users modulate their neural population activity to move an effector towards a target, stop accurately, and correct for movement errors. We call the rules that govern this modulation a 'feedback control policy'. A better understanding of these policies may inform the design of higher-performing neural decoders.

APPROACH

We studied how three participants in the BrainGate2 pilot clinical trial used an iBCI to control a cursor in a 2D target acquisition task. Participants used a velocity decoder with exponential smoothing dynamics. Through offline analyses, we characterized the users' feedback control policies by modeling their neural activity as a function of cursor state and target position. We also tested whether users could adapt their policy to different decoder dynamics by varying the gain (speed scaling) and temporal smoothing parameters of the iBCI.

MAIN RESULTS

We demonstrate that control policy assumptions made in previous studies do not fully describe the policies of our participants. To account for these discrepancies, we propose a new model that captures (1) how the user's neural population activity gradually declines as the cursor approaches the target from afar, then decreases more sharply as the cursor comes into contact with the target, (2) how the user makes constant feedback corrections even when the cursor is on top of the target, and (3) how the user actively accounts for the cursor's current velocity to avoid overshooting the target. Further, we show that users can adapt their control policy to decoder dynamics by attenuating neural modulation when the cursor gain is high and by damping the cursor velocity more strongly when the smoothing dynamics are high.

SIGNIFICANCE

Our control policy model may help to build better decoders, understand how neural activity varies during active iBCI control, and produce better simulations of closed-loop iBCI movements.

Optimal calibration of the learning rate in closed-loop adaptive brain-machine interfaces

Closed-loop decoder adaptation (CLDA) can improve brain-machine interface (BMI) performance. CLDA methods use batches of data to refit the decoder parameters in closed-loop operation. Recently, dynamic state-space algorithms have also been designed to fit the parameters of a point process decoder (PPF). A main design parameter that needs to be selected in any CLDA algorithm is the learning rate, i.e., how fast should the decoder parameters be updated on the basis of new neural observations. So far, the learning rate of CLDA algorithms has been selected empirically using ad-hoc methods. Here we develop a principled framework to calibrate the learning rate in adaptive state-space algorithms. The learning rate introduces a trade-off between the convergence rate and the steady-state error covariance of the estimated decoder parameters. Hence our algorithm first finds an analytical upper-bound on the steady-state error covariance as a function of the learning rate. It then finds the inverse mapping to select the optimal learning rate based on the maximum allowable steady-state error. Using numerical BMI experiments, we show that the calibration algorithm selects the optimal learning rate that meets the requirement on steady-state error level while achieving the fastest convergence rate possible corresponding to this steady-state level.

A Review of Control Strategies in Closed-Loop Neuroprosthetic Systems

It has been widely recognized that closed-loop neuroprosthetic systems achieve more favorable outcomes for users then equivalent open-loop devices. Improved performance of tasks, better usability, and greater embodiment have all been reported in systems utilizing some form of feedback. However, the interdisciplinary work on neuroprosthetic systems can lead to miscommunication due to similarities in well-established nomenclature in different fields. Here we present a review of control strategies in existing experimental, investigational and clinical neuroprosthetic systems in order to establish a baseline and promote a common understanding of different feedback modes and closed-loop controllers. The first section provides a brief discussion of feedback control and control theory. The second section reviews the control strategies of recent Brain Machine Interfaces, neuromodulatory implants, neuroprosthetic systems, and assistive neurorobotic devices. The final section examines the different approaches to feedback in current neuroprosthetic and neurorobotic systems.

Robust Brain-Machine Interface Design Using Optimal Feedback Control Modeling and Adaptive Point Process Filtering

Much progress has been made in brain-machine interfaces (BMI) using decoders such as Kalman filters and finding their parameters with closed-loop decoder adaptation (CLDA). However, current decoders do not model the spikes directly, and hence may limit the processing time-scale of BMI control and adaptation. Moreover, while specialized CLDA techniques for intention estimation and assisted training exist, a unified and systematic CLDA framework that generalizes across different setups is lacking. Here we develop a novel closed-loop BMI training architecture that allows for processing, control, and adaptation using spike events, enables robust control and extends to various tasks. Moreover, we develop a unified control-theoretic CLDA framework within which intention estimation, assisted training, and adaptation are performed. The architecture incorporates an infinite-horizon optimal feedback-control (OFC) model of the brain's behavior in closed-loop BMI control, and a point process model of spikes. The OFC model infers the user's motor intention during CLDA-a process termed intention estimation. OFC is also used to design an autonomous and dynamic assisted training technique. The point process model allows for neural processing, control and decoder adaptation with every spike event and at a faster time-scale than current decoders; it also enables dynamic spike-event-based parameter adaptation unlike current CLDA methods that use batch-based adaptation on much slower adaptation time-scales. We conducted closed-loop experiments in a non-human primate over tens of days to dissociate the effects of these novel CLDA components. The OFC intention estimation improved BMI performance compared with current intention estimation techniques. OFC assisted training allowed the subject to consistently achieve proficient control. Spike-event-based adaptation resulted in faster and more consistent performance convergence compared with batch-based methods, and was robust to parameter initialization. Finally, the architecture extended control to tasks beyond those used for CLDA training. These results have significant implications towards the development of clinically-viable neuroprosthetics.

Advancing brain-machine interfaces: moving beyond linear state space models

Advances in recent years have dramatically improved output control by Brain-Machine Interfaces (BMIs). Such devices nevertheless remain robotic and limited in their movements compared to normal human motor performance. Most current BMIs rely on transforming recorded neural activity to a linear state space composed of a set number of fixed degrees of freedom. Here we consider a variety of ways in which BMI design might be advanced further by applying non-linear dynamics observed in normal motor behavior. We consider (i) the dynamic range and precision of natural movements, (ii) differences between cortical activity and actual body movement, (iii) kinematic and muscular synergies, and (iv) the implications of large neuronal populations. We advance the hypothesis that a given population of recorded neurons may transmit more useful information than can be captured by a single, linear model across all movement phases and contexts. We argue that incorporating these various non-linear characteristics will be an important next step in advancing BMIs to more closely match natural motor performance.