Event detection and classification from multimodal time series with application to neural data

The detection of events in time-series data is a common signal-processing problem. When the data can be modeled as a known template signal with an unknown delay in Gaussian noise, detection of the template signal can be done with a traditional matched filter. However, in many applications, the event of interest is represented in multimodal data consisting of both Gaussian and point-process time series. Neuroscience experiments, for example, can simultaneously record multimodal neural signals such as local field potentials (LFPs), which can be modeled as Gaussian, and neuronal spikes, which can be modeled as point processes. Currently, no method exists for event detection from such multimodal data, and as such our objective in this work is to develop a method to meet this need. Here we address this challenge by developing the multimodal event detector (MED) algorithm which simultaneously estimates event times and classes. To do this, we write a multimodal likelihood function for Gaussian and point-process observations and derive the associated maximum likelihood estimator of simultaneous event times and classes. We additionally introduce a cross-modal scaling parameter to account for model mismatch in real datasets. We validate this method in extensive simulations as well as in a neural spike-LFP dataset recorded during an eye-movement task, where the events of interest are eye movements with unknown times and directions. We show that the MED can successfully detect eye movement onset and classify eye movement direction. Further, the MED successfully combines information across data modalities, with multimodal performance exceeding unimodal performance. This method can facilitate applications such as the discovery of latent events in multimodal neural population activity and the development of brain-computer interfaces for naturalistic settings without constrained tasks or prior knowledge of event times.

Multimodal subspace identification for modeling discrete-continuous spiking and field potential population activity

Objective.Learning dynamical latent state models for multimodal spiking and field potential activity can reveal their collective low-dimensional dynamics and enable better decoding of behavior through multimodal fusion. Toward this goal, developing unsupervised learning methods that are computationally efficient is important, especially for real-time learning applications such as brain-machine interfaces (BMIs). However, efficient learning remains elusive for multimodal spike-field data due to their heterogeneous discrete-continuous distributions and different timescales.Approach.Here, we develop a multiscale subspace identification (multiscale SID) algorithm that enables computationally efficient learning for modeling and dimensionality reduction for multimodal discrete-continuous spike-field data. We describe the spike-field activity as combined Poisson and Gaussian observations, for which we derive a new analytical SID method. Importantly, we also introduce a novel constrained optimization approach to learn valid noise statistics, which is critical for multimodal statistical inference of the latent state, neural activity, and behavior. We validate the method using numerical simulations and with spiking and local field potential population activity recorded during a naturalistic reach and grasp behavior.Main results.We find that multiscale SID accurately learned dynamical models of spike-field signals and extracted low-dimensional dynamics from these multimodal signals. Further, it fused multimodal information, thus better identifying the dynamical modes and predicting behavior compared to using a single modality. Finally, compared to existing multiscale expectation-maximization learning for Poisson-Gaussian observations, multiscale SID had a much lower training time while being better in identifying the dynamical modes and having a better or similar accuracy in predicting neural activity and behavior.Significance.Overall, multiscale SID is an accurate learning method that is particularly beneficial when efficient learning is of interest, such as for online adaptive BMIs to track non-stationary dynamics or for reducing offline training time in neuroscience investigations.

Modeling and dissociation of intrinsic and input-driven neural population dynamics underlying behavior

Neural dynamics can reflect intrinsic dynamics or dynamic inputs, such as sensory inputs or inputs from other brain regions. To avoid misinterpreting temporally structured inputs as intrinsic dynamics, dynamical models of neural activity should account for measured inputs. However, incorporating measured inputs remains elusive in joint dynamical modeling of neural-behavioral data, which is important for studying neural computations of behavior. We first show how training dynamical models of neural activity while considering behavior but not input or input but not behavior may lead to misinterpretations. We then develop an analytical learning method for linear dynamical models that simultaneously accounts for neural activity, behavior, and measured inputs. The method provides the capability to prioritize the learning of intrinsic behaviorally relevant neural dynamics and dissociate them from both other intrinsic dynamics and measured input dynamics. In data from a simulated brain with fixed intrinsic dynamics that performs different tasks, the method correctly finds the same intrinsic dynamics regardless of the task while other methods can be influenced by the task. In neural datasets from three subjects performing two different motor tasks with task instruction sensory inputs, the method reveals low-dimensional intrinsic neural dynamics that are missed by other methods and are more predictive of behavior and/or neural activity. The method also uniquely finds that the intrinsic behaviorally relevant neural dynamics are largely similar across the different subjects and tasks, whereas the overall neural dynamics are not. These input-driven dynamical models of neural-behavioral data can uncover intrinsic dynamics that may otherwise be missed.

The decoder design and performance comparative analysis for closed-loop brain-machine interface system

Brain-machine interface (BMI) can convert electroencephalography signals (EEGs) into the control instructions of external devices, and the key of control performance is the accuracy and efficiency of decoder. However, the performance of different decoders obtaining control instructions from complex and variable EEG signals is very different and irregular in the different neural information transfer model. Aiming at this problem, the off-line and on-line performance of eight decoders based on the improved single-joint information transmission (SJIT) model is compared and analyzed in this paper, which can provide a theoretical guidance for decoder design. Firstly, in order to avoid the different types of neural activities in the decoding process on the decoder performance, eight decoders based on the improved SJIT model are designed. And then the off-line decoding performance of these decoders is tested and compared. Secondly, a closed-loop BMI system which combining by the designed decoder and the random forest encoder based on the improved SJIT model is constructed. Finally, based on the constructed closed-loop BMI system, the on-line decoding performance of decoders is compared and analyzed. The results show that the LSTM-based decoder has better on-line decoding performance than others in the improved SJIT model.

Unsupervised learning of stationary and switching dynamical system models from Poisson observations

Objective. Investigating neural population dynamics underlying behavior requires learning accurate models of the recorded spiking activity, which can be modeled with a Poisson observation distribution. Switching dynamical system models can offer both explanatory power and interpretability by piecing together successive regimes of simpler dynamics to capture more complex ones. However, in many cases, reliable regime labels are not available, thus demanding accurate unsupervised learning methods for Poisson observations. Existing learning methods, however, rely on inference of latent states in neural activity using the Laplace approximation, which may not capture the broader properties of densities and may lead to inaccurate learning. Thus, there is a need for new inference methods that can enable accurate model learning.Approach. To achieve accurate model learning, we derive a novel inference method based on deterministic sampling for Poisson observations called the Poisson Cubature Filter (PCF) and embed it in an unsupervised learning framework. This method takes a minimum mean squared error approach to estimation. Terms that are difficult to find analytically for Poisson observations are approximated in a novel way with deterministic sampling based on numerical integration and cubature rules.Main results. PCF enabled accurate unsupervised learning in both stationary and switching dynamical systems and largely outperformed prior Laplace approximation-based learning methods in both simulations and motor cortical spiking data recorded during a reaching task. These improvements were larger for smaller data sizes, showing that PCF-based learning was more data efficient and enabled more reliable regime identification. In experimental data and unsupervised with respect to behavior, PCF-based learning uncovered interpretable behavior-relevant regimes unlike prior learning methods.Significance. The developed unsupervised learning methods for switching dynamical systems can accurately uncover latent regimes and states in population spiking activity, with important applications in both basic neuroscience and neurotechnology.

Post-stimulus encoding of decision confidence in EEG: toward a brain-computer interface for decision making

Objective.When making decisions, humans can evaluate how likely they are to be correct. If this subjective confidence could be reliably decoded from brain activity, it would be possible to build a brain-computer interface (BCI) that improves decision performance by automatically providing more information to the user if needed based on their confidence. But this possibility depends on whether confidence can be decoded right after stimulus presentation and before the response so that a corrective action can be taken in time. Although prior work has shown that decision confidence is represented in brain signals, it is unclear if the representation is stimulus-locked or response-locked, and whether stimulus-locked pre-response decoding is sufficiently accurate for enabling such a BCI.Approach.We investigate the neural correlates of confidence by collecting high-density electroencephalography (EEG) during a perceptual decision task with realistic stimuli. Importantly, we design our task to include a post-stimulus gap that prevents the confounding of stimulus-locked activity by response-locked activity and vice versa, and then compare with a task without this gap.Main results.We perform event-related potential and source-localization analyses. Our analyses suggest that the neural correlates of confidence are stimulus-locked, and that an absence of a post-stimulus gap could cause these correlates to incorrectly appear as response-locked. By preventing response-locked activity from confounding stimulus-locked activity, we then show that confidence can be reliably decoded from single-trial stimulus-locked pre-response EEG alone. We also identify a high-performance classification algorithm by comparing a battery of algorithms. Lastly, we design a simulated BCI framework to show that the EEG classification is accurate enough to build a BCI and that the decoded confidence could be used to improve decision making performance particularly when the task difficulty and cost of errors are high.Significance.Our results show feasibility of non-invasive EEG-based BCIs to improve human decision making.

Multi-Class Classification of Upper Limb Movements With Filter Bank Task-Related Component Analysis

The classification of limb movements can provide with control commands in non-invasive brain-computer interface. Previous studies on the classification of limb movements have focused on the classification of left/right limbs; however, the classification of different types of upper limb movements has often been ignored despite that it provides more active-evoked control commands in the brain-computer interface. Nevertheless, few machine learning method can be used as the state-of-the-art method in the multi-class classification of limb movements. This work focuses on the multi-class classification of upper limb movements and proposes the multi-class filter bank task-related component analysis (mFBTRCA) method, which consists of three steps: spatial filtering, similarity measuring and filter bank selection. The spatial filter, namely the task-related component analysis, is first used to remove noise from EEG signals. The canonical correlation measures the similarity of the spatial-filtered signals and is used for feature extraction. The correlation features are extracted from multiple low-frequency filter banks. The minimum-redundancy maximum-relevance selects the essential features from all the correlation features, and finally, the support vector machine is used to classify the selected features. The proposed method compared against previously used models is evaluated using two datasets. mFBTRCA achieved a classification accuracy of 0.4193 ± 0.0780 (7 classes) and 0.4032 ± 0.0714 (5 classes), respectively, which improves on the best accuracies achieved using the compared methods (0.3590 ± 0.0645 and 0.3159 ± 0.0736, respectively). The proposed method is expected to provide more control commands in the applications of non-invasive brain-computer interfaces.

Multimodal subspace identification for modeling discrete-continuous spiking and field potential population activity

Learning dynamical latent state models for multimodal spiking and field potential activity can reveal their collective low-dimensional dynamics and enable better decoding of behavior through multimodal fusion. Toward this goal, developing unsupervised learning methods that are computationally efficient is important, especially for real-time learning applications such as brain-machine interfaces (BMIs). However, efficient learning remains elusive for multimodal spike-field data due to their heterogeneous discrete-continuous distributions and different timescales. Here, we develop a multiscale subspace identification (multiscale SID) algorithm that enables computationally efficient modeling and dimensionality reduction for multimodal discrete-continuous spike-field data. We describe the spike-field activity as combined Poisson and Gaussian observations, for which we derive a new analytical subspace identification method. Importantly, we also introduce a novel constrained optimization approach to learn valid noise statistics, which is critical for multimodal statistical inference of the latent state, neural activity, and behavior. We validate the method using numerical simulations and spike-LFP population activity recorded during a naturalistic reach and grasp behavior. We find that multiscale SID accurately learned dynamical models of spike-field signals and extracted low-dimensional dynamics from these multimodal signals. Further, it fused multimodal information, thus better identifying the dynamical modes and predicting behavior compared to using a single modality. Finally, compared to existing multiscale expectation-maximization learning for Poisson-Gaussian observations, multiscale SID had a much lower computational cost while being better in identifying the dynamical modes and having a better or similar accuracy in predicting neural activity. Overall, multiscale SID is an accurate learning method that is particularly beneficial when efficient learning is of interest.

Modeling and dissociation of intrinsic and input-driven neural population dynamics underlying behavior

Neural dynamics can reflect intrinsic dynamics or dynamic inputs, such as sensory inputs or inputs from other regions. To avoid misinterpreting temporally-structured inputs as intrinsic dynamics, dynamical models of neural activity should account for measured inputs. However, incorporating measured inputs remains elusive in joint dynamical modeling of neural-behavioral data, which is important for studying neural computations of a specific behavior. We first show how training dynamical models of neural activity while considering behavior but not input, or input but not behavior may lead to misinterpretations. We then develop a novel analytical learning method that simultaneously accounts for neural activity, behavior, and measured inputs. The method provides the new capability to prioritize the learning of intrinsic behaviorally relevant neural dynamics and dissociate them from both other intrinsic dynamics and measured input dynamics. In data from a simulated brain with fixed intrinsic dynamics that performs different tasks, the method correctly finds the same intrinsic dynamics regardless of task while other methods can be influenced by the change in task. In neural datasets from three subjects performing two different motor tasks with task instruction sensory inputs, the method reveals low-dimensional intrinsic neural dynamics that are missed by other methods and are more predictive of behavior and/or neural activity. The method also uniquely finds that the intrinsic behaviorally relevant neural dynamics are largely similar across the three subjects and two tasks whereas the overall neural dynamics are not. These input-driven dynamical models of neural-behavioral data can uncover intrinsic dynamics that may otherwise be missed.

Modeling and inference methods for switching regime-dependent dynamical systems with multiscale neural observations

Objective.Realizing neurotechnologies that enable long-term neural recordings across multiple spatial-temporal scales during naturalistic behaviors requires new modeling and inference methods that can simultaneously address two challenges. First, the methods should aggregate information across all activity scales from multiple recording sources such as spiking and field potentials. Second, the methods should detect changes in the regimes of behavior and/or neural dynamics during naturalistic scenarios and long-term recordings. Prior regime detection methods are developed for a single scale of activity rather than multiscale activity, and prior multiscale methods have not considered regime switching and are for stationary cases.Approach.Here, we address both challenges by developing a switching multiscale dynamical system model and the associated filtering and smoothing methods. This model describes the encoding of an unobserved brain state in multiscale spike-field activity. It also allows for regime-switching dynamics using an unobserved regime state that dictates the dynamical and encoding parameters at every time-step. We also design the associated switching multiscale inference methods that estimate both the unobserved regime and brain states from simultaneous spike-field activity.Main results.We validate the methods in both extensive numerical simulations and prefrontal spike-field data recorded in a monkey performing saccades for fluid rewards. We show that these methods can successfully combine the spiking and field potential observations to simultaneously track the regime and brain states accurately. Doing so, these methods lead to better state estimation compared with single-scale switching methods or stationary multiscale methods. Also, for single-scale linear Gaussian observations, the new switching smoother can better generalize to diverse system settings compared to prior switching smoothers.Significance.These modeling and inference methods effectively incorporate both regime-detection and multiscale observations. As such, they could facilitate investigation of latent switching neural population dynamics and improve future brain-machine interfaces by enabling inference in naturalistic scenarios where regime-dependent multiscale activity and behavior arise.

Auxiliary controller design and performance comparative analysis in closed-loop brain-machine interface system

Brain-machine interface (BMI) can realize information interaction between the brain and external devices, and yet the control accuracy is limited by the change of electroencephalogram signals. The introduction of auxiliary controller can overcome the above problems, but the performance of different auxiliary controllers is quite different. Hence, in this paper, we comprehensively compare and analyze the performance of different auxiliary controllers to provide a theoretical basis for designing BMI system. The main work includes: (1) designing four kinds of auxiliary controllers based on simultaneous perturbation stochastic approximation-function approximator (SPSA-FA), iterative feedback tuning-PID (IFT-PID), model predictive control (MPC) and model-free control (MFC); (2) based on the model of improved single-joint information transmission, constructing the closed-loop BMI systems with the decoder-based Wiener filter; and (3) comparing their performance in the constructed closed-loop BMI systems for dynamic motion restoration. The results show that the order of tracking accuracy is MPC, IFT-PID, SPSA-FA, MFC, and the order of time consumed is opposite. A good control effectiveness is achieved at the expense of time, so a suitable auxiliary controller should be selected according to the actual requirements.

Modeling multiscale causal interactions between spiking and field potential signals during behavior

Objective. Brain recordings exhibit dynamics at multiple spatiotemporal scales, which are measured with spike trains and larger-scale field potential signals. To study neural processes, it is important to identify and model causal interactions not only at a single scale of activity, but also across multiple scales, i.e. between spike trains and field potential signals. Standard causality measures are not directly applicable here because spike trains are binary-valued but field potentials are continuous-valued. It is thus important to develop computational tools to recover multiscale neural causality during behavior, assess their performance on neural datasets, and study whether modeling multiscale causalities can improve the prediction of neural signals beyond what is possible with single-scale causality. Approach. We design a multiscale model-based Granger-like causality method based on directed information and evaluate its success both in realistic biophysical spike-field simulations and in motor cortical datasets from two non-human primates (NHP) performing a motor behavior. To compute multiscale causality, we learn point-process generalized linear models that predict the spike events at a given time based on the history of both spike trains and field potential signals. We also learn linear Gaussian models that predict the field potential signals at a given time based on their own history as well as either the history of binary spike events or that of latent firing rates. Main results. We find that our method reveals the true multiscale causality network structure in biophysical simulations despite the presence of model mismatch. Further, models with the identified multiscale causalities in the NHP neural datasets lead to better prediction of both spike trains and field potential signals compared to just modeling single-scale causalities. Finally, we find that latent firing rates are better predictors of field potential signals compared with the binary spike events in the NHP datasets. Significance. This multiscale causality method can reveal the directed functional interactions across spatiotemporal scales of brain activity to inform basic science investigations and neurotechnologies.

Closed-loop Modulation of the Self-regulating Brain: A Review on Approaches, Emerging Paradigms, and Experimental Designs

Closed-loop approaches, setups, and experimental designs have been applied within the field of neuroscience to enhance the understanding of basic neurophysiology principles (closed-loop neuroscience; CLNS) and to develop improved procedures for modulating brain circuits and networks for clinical purposes (closed-loop neuromodulation; CLNM). The contents of this review are thus arranged into the following sections. First, we describe basic research findings that have been made using CLNS. Next, we provide an overview of the application, rationale, and therapeutic aspects of CLNM for clinical purposes. Finally, we summarize methodological concerns and critics in clinical practice of neurofeedback and novel applications of closed-loop perspective and techniques to improve and optimize its experiments. Moreover, we outline the theoretical explanations and experimental ideas to test animal models of neurofeedback and discuss technical issues and challenges associated with implementing closed-loop systems. We hope this review is helpful for both basic neuroscientists and clinical/ translationally-oriented scientists interested in applying closed-loop methods to improve mental health and well-being.

CluSem: Accurate clustering-based ensemble method to predict motor imagery tasks from multi-channel EEG data

BACKGROUND

The classification of motor imagery electroencephalogram (MI-EEG) is a pivotal task in the biosignal classification process in the brain-computer interface (BCI) applications. Currently, this bio-engineering-based technology is being employed by researchers in various fields to develop cutting-edge applications. The classification of real-time MI-EEG signals is the most challenging task in these applications. The prediction performance of the existing classification methods is still limited due to the high dimensionality and dynamic behaviors of the real-time EEG data.

PROPOSED METHOD

To enhance the classification performance of real-time BCI applications, this paper presents a new clustering-based ensemble technique called CluSem to mitigate this problem. We also develop a new brain game called CluGame using this method to evaluate the classification performance of real-time motor imagery movements. In this game, real-time EEG signal classification and prediction tabulation through animated balls are controlled via threads. By playing this game, users can control the movements of the balls via the brain signals of motor imagery movements without using any traditional input devices.

RESULTS

Our results demonstrate that CluSem is able to improve the classification accuracy between 5% and 15% compared to the existing methods on our collected as well as the publicly available EEG datasets. The source codes used to implement CluSem and CluGame are publicly available at https://github.com/MdOchiuddinMiah/MI-BCI_ML.

Intention Estimation Based Adaptive Unscented Kalman Filter for Online Neural Decoding

The commonly used fixed discrete Kalman filters (DKF) in neural decoders do not generalize well to the actual relationship between neuronal firing rates and movement intention. This is due to the underlying assumption that the neural activity is linearly related to the output state. They also face the issues of requiring large amount of training datasets to achieve a robust model and a degradation of decoding performance over time. In this paper, an adaptive adjustment is made to the conventional unscented Kalman filter (UKF) via intention estimation. This is done by incorporating a history of newly collected state parameters to develop a new set of model parameters. At each time point, a comparative weighted sum of old and new model parameters using matrix squared sums is used to update the neural decoding model parameters. The effectiveness of the resulting adaptive unscented Kalman filter (AUKF) is compared against the discrete Kalman filter and unscented Kalman filter-based algorithms. The results show that the proposed new algorithm provides higher decoding accuracy and stability while requiring less training data.

The Influence of Frequency Bands and Brain Region on ECoG-Based BMI Learning Performance

Numerous brain–machine interface (BMI) studies have shown that various frequency bands (alpha, beta, and gamma bands) can be utilized in BMI experiments and modulated as neural information for machine control after several BMI learning trial sessions. In addition to frequency range as a neural feature, various areas of the brain, such as the motor cortex or parietal cortex, have been selected as BMI target brain regions. However, although the selection of target frequency and brain region appears to be crucial in obtaining optimal BMI performance, the direct comparison of BMI learning performance as it relates to various brain regions and frequency bands has not been examined in detail. In this study, ECoG-based BMI learning performances were compared using alpha, beta, and gamma bands, respectively, in a single rodent model. Brain area dependence of learning performance was also evaluated in the frontal cortex, the motor cortex, and the parietal cortex. The findings indicated that BMI learning performance was best in the case of the gamma frequency band and worst in the alpha band (one-way ANOVA, F = 4.41, p < 0.05). In brain area dependence experiments, better BMI learning performance appears to be shown in the primary motor cortex (one-way ANOVA, F = 4.36, p < 0.05). In the frontal cortex, two out of four animals failed to learn the feeding tube control even after a maximum of 10 sessions. In conclusion, the findings reported in this study suggest that the selection of target frequency and brain region should be carefully considered when planning BMI protocols and for performing optimized BMI.

Adaptive latent state modeling of brain network dynamics with real-time learning rate optimization

Objective. Dynamic latent state models are widely used to characterize the dynamics of brain network activity for various neural signal types. To date, dynamic latent state models have largely been developed for stationary brain network dynamics. However, brain network dynamics can be non-stationary for example due to learning, plasticity or recording instability. To enable modeling these non-stationarities, two problems need to be resolved. First, novel methods should be developed that can adaptively update the parameters of latent state models, which is difficult due to the state being latent. Second, new methods are needed to optimize the adaptation learning rate, which specifies how fast new neural observations update the model parameters and can significantly influence adaptation accuracy.Approach. We develop a Rate Optimized-adaptive Linear State-Space Modeling (RO-adaptive LSSM) algorithm that solves these two problems. First, to enable adaptation, we derive a computation- and memory-efficient adaptive LSSM fitting algorithm that updates the LSSM parameters recursively and in real time in the presence of the latent state. Second, we develop a real-time learning rate optimization algorithm. We use comprehensive simulations of a broad range of non-stationary brain network dynamics to validate both algorithms, which together constitute the RO-adaptive LSSM.Main results. We show that the adaptive LSSM fitting algorithm can accurately track the broad simulated non-stationary brain network dynamics. We also find that the learning rate significantly affects the LSSM fitting accuracy. Finally, we show that the real-time learning rate optimization algorithm can run in parallel with the adaptive LSSM fitting algorithm. Doing so, the combined RO-adaptive LSSM algorithm rapidly converges to the optimal learning rate and accurately tracks non-stationarities.Significance. These algorithms can be used to study time-varying neural dynamics underlying various brain functions and enhance future neurotechnologies such as brain-machine interfaces and closed-loop brain stimulation systems.

Building an adaptive interface via unsupervised tracking of latent manifolds

In human-machine interfaces, decoder calibration is critical to enable an effective and seamless interaction with the machine. However, recalibration is often necessary as the decoder off-line predictive power does not generally imply ease-of-use, due to closed loop dynamics and user adaptation that cannot be accounted for during the calibration procedure. Here, we propose an adaptive interface that makes use of a non-linear autoencoder trained iteratively to perform online manifold identification and tracking, with the dual goal of reducing the need for interface recalibration and enhancing human-machine joint performance. Importantly, the proposed approach avoids interrupting the operation of the device and it neither relies on information about the state of the task, nor on the existence of a stable neural or movement manifold, allowing it to be applied in the earliest stages of interface operation, when the formation of new neural strategies is still on-going. In order to more directly test the performance of our algorithm, we defined the autoencoder latent space as the control space of a body-machine interface. After an initial offline parameter tuning, we evaluated the performance of the adaptive interface versus that of a static decoder in approximating the evolving low-dimensional manifold of users simultaneously learning to perform reaching movements within the latent space. Results show that the adaptive approach increased the representational efficiency of the interface decoder. Concurrently, it significantly improved users' task-related performance, indicating that the development of a more accurate internal model is encouraged by the online co-adaptation process.

Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification

Neural activity exhibits complex dynamics related to various brain functions, internal states and behaviors. Understanding how neural dynamics explain specific measured behaviors requires dissociating behaviorally relevant and irrelevant dynamics, which is not achieved with current neural dynamic models as they are learned without considering behavior. We develop preferential subspace identification (PSID), which is an algorithm that models neural activity while dissociating and prioritizing its behaviorally relevant dynamics. Modeling data in two monkeys performing three-dimensional reach and grasp tasks, PSID revealed that the behaviorally relevant dynamics are significantly lower-dimensional than otherwise implied. Moreover, PSID discovered distinct rotational dynamics that were more predictive of behavior. Furthermore, PSID more accurately learned behaviorally relevant dynamics for each joint and recording channel. Finally, modeling data in two monkeys performing saccades demonstrated the generalization of PSID across behaviors, brain regions and neural signal types. PSID provides a general new tool to reveal behaviorally relevant neural dynamics that can otherwise go unnoticed.

Power-saving design opportunities for wireless intracortical brain-computer interfaces

The efficacy of wireless intracortical brain-computer interfaces (iBCIs) is limited in part by the number of recording channels, which is constrained by the power budget of the implantable system. Designing wireless iBCIs that provide the high-quality recordings of today's wired neural interfaces may lead to inadvertent over-design at the expense of power consumption and scalability. Here, we report analyses of neural signals collected from experimental iBCI measurements in rhesus macaques and from a clinical-trial participant with implanted 96-channel Utah multielectrode arrays to understand the trade-offs between signal quality and decoder performance. Moreover, we propose an efficient hardware design for clinically viable iBCIs, and suggest that the circuit design parameters of current recording iBCIs can be relaxed considerably without loss of performance. The proposed design may allow for an order-of-magnitude power savings and lead to clinically viable iBCIs with a higher channel count.

An adaptive decoder design based on the receding horizon optimization in BMI system

In a motor brain-machine interface system, since the electroencephalogram signal is changing through out the process of the arm movement, the offline trained decoder with fixed weights is often unable to convert the electroencephalogram signal accurately, resulting in poor recovery of joint motor function. In this paper, a receding horizon optimization strategy is chosen to online update the decoder weights and design an adaptive Wiener-filter-based decoder. Firstly, a classical Wiener-filter-based decoder with fixed weights is brief reviewed. Secondly, the weights in Wiener-filter-based decoder are updated by minimizing the cost function, which is composed by the sum of squared position errors in the given horizon at each sampling time. The simulation shows that the recovery effect of joint motor function and neuron activity in the BMI system with the adaptive decoder are both better than that in the BMI system with the fixed decoder.

Bacomics: a comprehensive cross area originating in the studies of various brain-apparatus conversations

The brain is the most important organ of the human body, and the conversations between the brain and an apparatus can not only reveal a normally functioning or a dysfunctional brain but also can modulate the brain. Here, the apparatus may be a nonbiological instrument, such as a computer, and the consequent brain-computer interface is now a very popular research area with various applications. The apparatus may also be a biological organ or system, such as the gut and muscle, and their efficient conversations with the brain are vital for a healthy life. Are there any common bases that bind these different scenarios? Here, we propose a new comprehensive cross area: Bacomics, which comes from brain-apparatus conversations (BAC) + omics. We take Bacomics to cover at least three situations: (1) The brain is normal, but the conversation channel is disabled, as in amyotrophic lateral sclerosis. The task is to reconstruct or open up new channels to reactivate the brain function. (2) The brain is in disorder, such as in Parkinson's disease, and the work is to utilize existing or open up new channels to intervene, repair and modulate the brain by medications or stimulation. (3) Both the brain and channels are in order, and the goal is to enhance coordinated development between the brain and apparatus. In this paper, we elaborate the connotation of BAC into three aspects according to the information flow: the issue of output to the outside (BAC-1), the issue of input to the brain (BAC-2) and the issue of unity of brain and apparatus (BAC-3). More importantly, there are no less than five principles that may be taken as the cornerstones of Bacomics, such as feedforward and feedback control, brain plasticity, harmony, the unity of opposites and systems principles. Clearly, Bacomics integrates these seemingly disparate domains, but more importantly, opens a much wider door for the research and development of the brain, and the principles further provide the general framework in which to realize or optimize these various conversations.

Force decoding using local field potentials in primary motor cortex: PLS or Kalman filter regression?

The development of brain-computer interface (BCI) systems is an important approach in brain studies. Control of communication devices and prostheses in real-world scenarios requires complex movement parameters. Decoding a variety of neural signals captured by micro-wire arrays is a potential applicant for extracting movement-related information. The present work was conducted to compare the functionality of partial least square (PLS) regression and Kalman filter to predict the force parameter from local field potential (LFP) signals of the primary motor cortex (M1). The signals were recorded using a 16-channel micro-wire array from the forelimb-related area of the M1 of three rats performing a behavioral task in which the force signal of the rat's forelimb paw was generated. Our results show that PLS regression and Kalman filters with the mean performance of 0.75 and 0.72 in terms of the correlation coefficient (CC) and 0.37 and 0.48 in terms of normalized mean square error (NMSE), respectively, are effective methods for decoding the force parameter from LFPs. Kalman filter underperforms PLS both in performance and speed. Although adding nonlinearity to the Kalman filter results in equally accurate CC performance as PLS, it has even more computational cost. Therefore, it is inferred that nonlinear methods do not necessarily have better functionality than linear ones and PLS, as a simple fast linear method could be an effectively applicable regression technique for BCIs.

Spike Rate Estimation Using Bayesian Adaptive Kernel Smoother (BAKS) and Its Application to Brain Machine Interfaces

Brain Machine Interfaces (BMIs) mostly utilise spike rate as an input feature for decoding a desired motor output as it conveys a useful measure to the underlying neuronal activity. The spike rate is typically estimated by a using non-overlap binning method that yields a coarse estimate. There exist several methods that can produce a smooth estimate which could potentially improve the decoding performance. However, these methods are relatively computationally heavy for real-time BMIs. To address this issue, we propose a new method for estimating spike rate that is able to yield a smooth estimate and also amenable to real-time BMIs. The proposed method, referred to as Bayesian adaptive kernel smoother (BAKS), employs kernel smoothing technique that considers the bandwidth as a random variable with prior distribution which is adaptively updated through a Bayesian framework. With appropriate selection of prior distribution and kernel function, an analytical expression can be achieved for the kernel bandwidth. We apply BAKS and evaluate its impact on offline BMI decoding performance using Kalman filter. The results reveal that BAKS can improve the decoding performance compared to the binning method. This suggests the feasibility and the potential use of BAKS for real-time BMIs.

A point-process matched filter for event detection and decoding from population spike trains

OBJECTIVE

Information encoding in neurons can be described through their response fields. The spatial response field of a neuron is the region of space in which a sensory stimulus or a behavioral event causes that neuron to fire. Neurons can also exhibit temporal response fields (TRFs), which characterize a transient response to stimulus or behavioral event onsets. These neurons can thus be described by a spatio-temporal response field (STRF). The activity of neurons with STRFs can be well-described with point process models that characterize binary spike trains with an instantaneous firing rate that is a function of both time and space. However, developing decoders for point process models of neurons that exhibit TRFs is challenging because it requires prior knowledge of event onset times, which are unknown. Indeed, point process filters (PPF) to date have largely focused on decoding neuronal activity without considering TRFs. Also, neural classifiers have required data to be behavior- or stimulus-aligned, i.e. event times to be known, which is often not possible in real-world applications. Our objective in this work is to develop a viable decoder for neurons with STRFs when event times are unknown.

APPROACH

To enable decoding of neurons with STRFs, we develop a novel point-process matched filter (PPMF) that can detect events and estimate their onset times from population spike trains. We also devise a PPF for neurons with transient responses as characterized by STRFs. When neurons exhibit STRFs and event times are unknown, the PPMF can be combined with the PPF or with discrete classifiers for continuous and discrete brain state decoding, respectively.

MAIN RESULTS

We validate our algorithm on two datasets: simulated spikes from neurons that encode visual saliency in response to stimuli, and prefrontal spikes recorded in a monkey performing a delayed-saccade task. We show that the PPMF can estimate the stimulus times and saccade times accurately. Further, the PPMF combined with the PPF can decode visual saliency maps without knowing the stimulus times. Similarly, the PPMF combined with a point process classifier can decode the saccade direction without knowing the saccade times.

SIGNIFICANCE

These event detection and decoding algorithms can help develop neurotechnologies to decode cognitive states from neural responses that exhibit STRFs.

Developing a personalized closed-loop controller of medically-induced coma in a rodent model

OBJECTIVE

Personalized automatic control of medically-induced coma, a critical multi-day therapy in the intensive care unit, could greatly benefit clinical care and further provide a novel scientific tool for investigating how the brain response to anesthetic infusion rate changes during therapy. Personalized control would require real-time tracking of inter- and intra-subject variabilities in the brain response to anesthetic infusion rate while simultaneously delivering the therapy, which has not been achieved. Current control systems for medically-induced coma require a separate offline model fitting experiment to deal with inter-subject variabilities, which would lead to therapy interruption. Removing the need for these offline interruptions could help facilitate clinical feasbility. In addition, current systems do not track intra-subject variabilities. Tracking intra-subject variabilities is essential for studying whether or how the brain response to anesthetic infusion rate changes during therapy. Further, such tracking could enhance control precison and thus help facilitate clinical feasibility.

APPROACH

Here we develop a personalized closed-loop anesthetic delivery (CLAD) system in a rodent model that tracks both inter- and intra-subject variabilities in real time while simultaneously controlling the anesthetic in closed loop. We tested the CLAD in rats by administrating propofol to control the electroencephalogram (EEG) burst suppression. We first examined whether the CLAD can remove the need for offline model fitting interruption. We then used the CLAD as a tool to study whether and how the brain response to anesthetic infusion rate changes as a function of changes in the depth of medically-induced coma. Finally, we studied whether the CLAD can enhance control compared with prior systems by tracking intra-subject variabilities.

MAIN RESULTS

The CLAD precisely controlled the EEG burst suppression in each rat without performing offline model fitting experiments. Further, using the CLAD, we discovered that the brain response to anesthetic infusion rate varied during control, and that these variations correlated with the depth of medically-induced coma in a consistent manner across individual rats. Finally, tracking these variations reduced control bias and error by more than 70% compared with prior systems.

SIGNIFICANCE

This personalized CLAD provides a new tool to study the dynamics of brain response to anesthetic infusion rate and has significant implications for enabling clinically-feasible automatic control of medically-induced coma.

Spike detection and spike sorting with a hidden Markov model improves offline decoding of motor cortical recordings

OBJECTIVE

Detection and sorting (classification) of action potentials from extracellular recordings are two important pre-processing steps for brain-computer interfaces (BCIs) and some neuroscientific studies. Traditional approaches perform these two steps serially, but using shapes of action potential waveforms during detection, i.e. combining the two steps, may lead to better performance, especially during high noise. We propose a hidden Markov model (HMM) based method for combined detecting and sorting of spikes, with the aim of improving the final decoding accuracy of BCIs.

APPROACH

The states of the HMM indicate whether there is a spike, what unit a spike belongs to, and the time course within a waveform. The HMM outputs probabilities of spike detection, and from this we can calculate expectations of spike counts in time bins, which can replace integer spike counts as input to BCI decoders. We evaluate the HMM method on simulated spiking data. We then examine the impact of using this method on decoding real neural data recorded from primary motor cortex of two Rhesus monkeys.

MAIN RESULTS

Our comparisons on simulated data to detection-then-sorting approaches and combined detection-and-sorting algorithms indicate that the HMM method performs more accurately at detection and sorting (0.93 versus 0.73 spike count correlation, 0.73 versus 0.49 adjusted mutual information). On real neural data, the HMM method led to higher adjusted mutual information between spike counts and kinematics (monkey K: 0.034 versus 0.027; monkey M: 0.033 versus 0.022) and better neuron encoding model predictions (K: 0.016 dB improvement; M: 0.056 dB improvement). Lastly, the HMM method facilitated higher offline decoding accuracy (Kalman filter, K: 8.5% mean squared error reduction, M: 18.6% reduction).

SIGNIFICANCE

The HMM spike detection and sorting method offers a new approach to spike pre-processing for BCIs and neuroscientific studies.

Identifying multiscale hidden states to decode behavior

A key element needed in a brain-machine interface (BMI) decoder is the encoding model, which relates the neural activity to intended movement. The vast majority of work have used a representational encoding model, which assumes movement parameters are directly encoded in neural activity. Recent work have in turn suggested the existence of neural dynamics that represent behavior. This recent evidence motivates developing dynamical encoding models with hidden states that encode movement. Regardless of their type, encoding models have vastly characterized a single scale of activity, e.g., either spikes or local field potentials (LFP). In our recent work we developed a multiscale representational encoding model to simultaneously characterize and decode discrete spikes and continuous field activity. However, learning a multiscale dynamical model from simultaneous spike-field recordings in the presence of hidden states is challenging. Here we present an unsupervised learning algorithm for estimating a multiscale state-space model with hidden states and validate it using spike-LFP activity during a reaching movement. We use the learned multiscale statespace model and a corresponding decoder to identify hidden states from spike-LFP activity. We then decode the movement trajectories using these hidden states. We find that the identified states can accurately decode the trajectories. Moreover, we demonstrate that adding LFP to spikes improves the decoding accuracy, suggesting that our unsupervised learning algorithm incorporates information across scales. This learning algorithm could serve as a new tool to study encoding across scales and to enhance future BMI systems.

Optimizing the learning rate for adaptive estimation of neural encoding models

Closed-loop neurotechnologies often need to adaptively learn an encoding model that relates the neural activity to the brain state, and is used for brain state decoding. The speed and accuracy of adaptive learning algorithms are critically affected by the learning rate, which dictates how fast model parameters are updated based on new observations. Despite the importance of the learning rate, currently an analytical approach for its selection is largely lacking and existing signal processing methods vastly tune it empirically or heuristically. Here, we develop a novel analytical calibration algorithm for optimal selection of the learning rate in adaptive Bayesian filters. We formulate the problem through a fundamental trade-off that learning rate introduces between the steady-state error and the convergence time of the estimated model parameters. We derive explicit functions that predict the effect of learning rate on error and convergence time. Using these functions, our calibration algorithm can keep the steady-state parameter error covariance smaller than a desired upper-bound while minimizing the convergence time, or keep the convergence time faster than a desired value while minimizing the error. We derive the algorithm both for discrete-valued spikes modeled as point processes nonlinearly dependent on the brain state, and for continuous-valued neural recordings modeled as Gaussian processes linearly dependent on the brain state. Using extensive closed-loop simulations, we show that the analytical solution of the calibration algorithm accurately predicts the effect of learning rate on parameter error and convergence time. Moreover, the calibration algorithm allows for fast and accurate learning of the encoding model and for fast convergence of decoding to accurate performance. Finally, larger learning rates result in inaccurate encoding models and decoders, and smaller learning rates delay their convergence. The calibration algorithm provides a novel analytical approach to predictably achieve a desired level of error and convergence time in adaptive learning, with application to closed-loop neurotechnologies and other signal processing domains.

Feasibility of Automatic Error Detect-and-Undo System in Human Intracortical Brain-Computer Interfaces

OBJECTIVE

Brain-computer interfaces (BCIs) aim to help people with impaired movement ability by directly translating their movement intentions into command signals for assistive technologies. Despite large performance improvements over the last two decades, BCI systems still make errors that need to be corrected manually by the user. This decreases system performance and is also frustrating for the user. The deleterious effects of errors could be mitigated if the system automatically detected when the user perceives that an error was made and automatically intervened with a corrective action; thus, sparing users from having to make the correction themselves. Our previous preclinical work with monkeys demonstrated that task-outcome correlates exist in motor cortical spiking activity and can be utilized to improve BCI performance. Here, we asked if these signals also exist in the human hand area of motor cortex, and whether they can be decoded with high accuracy.

METHODS

We analyzed posthoc the intracortical neural activity of two BrainGate2 clinical trial participants who were neurally controlling a computer cursor to perform a grid target selection task and a keyboard-typing task.

RESULTS

Our key findings are that: 1) there exists a putative outcome error signal reflected in both the action potentials and local field potentials of the human hand area of motor cortex, and 2) target selection outcomes can be classified with high accuracy (70-85%) of errors successfully detected with minimal (0-3%) misclassifications of success trials, based on neural activity alone.

SIGNIFICANCE

These offline results suggest that it will be possible to improve the performance of clinical intracortical BCIs by incorporating a real-time error detect-and-undo system alongside the decoding of movement intention.