Leveraging neural dynamics to extend functional lifetime of brain-machine interfaces

Similar neural states, but dissimilar decoding patterns for motor control in parietal cortex

Discrete neural states are associated with reaching movements across the fronto-parietal network. Here, the Hidden Markov Model (HMM) applied to spiking activity of the somato-motor parietal area PE revealed a sequence of states similar to those of the contiguous visuomotor areas PEc and V6A. Using a coupled clustering and decoding approach, we proved that these neural states carried spatiotemporal information regarding behaviour in all three posterior parietal areas. However, comparing decoding accuracy, PE was less informative than V6A and PEc. In addition, V6A outperformed PEc in target inference, indicating functional differences among the parietal areas. To check the consistency of these differences, we used both a supervised and an unsupervised variant of the HMM, and compared its performance with two more common classifiers, Support Vector Machine and Long-Short Term Memory. The differences in decoding between areas were invariant to the algorithm used, still showing the dissimilarities found with HMM, thus indicating that these dissimilarities are intrinsic in the information encoded by parietal neurons. These results highlight that, when decoding from the parietal cortex, for example, in brain machine interface implementations, attention should be paid in selecting the most suitable source of neural signals, given the great heterogeneity of this cortical sector.

Invariant neural dynamics drive commands to control different movements

It has been proposed that the nervous system has the capacity to generate a wide variety of movements because it reuses some invariant code. Previous work has identified that dynamics of neural population activity are similar during different movements, where dynamics refer to how the instantaneous spatial pattern of population activity changes in time. Here, we test whether invariant dynamics of neural populations are actually used to issue the commands that direct movement. Using a brain-machine interface (BMI) that transforms rhesus macaques' motor-cortex activity into commands for a neuroprosthetic cursor, we discovered that the same command is issued with different neural-activity patterns in different movements. However, these different patterns were predictable, as we found that the transitions between activity patterns are governed by the same dynamics across movements. These invariant dynamics are low dimensional, and critically, they align with the BMI, so that they predict the specific component of neural activity that actually issues the next command. We introduce a model of optimal feedback control (OFC) that shows that invariant dynamics can help transform movement feedback into commands, reducing the input that the neural population needs to control movement. Altogether our results demonstrate that invariant dynamics drive commands to control a variety of movements and show how feedback can be integrated with invariant dynamics to issue generalizable commands.

Neural Plasticity in Sensorimotor Brain-Machine Interfaces

Brain-machine interfaces (BMIs) aim to treat sensorimotor neurological disorders by creating artificial motor and/or sensory pathways. Introducing artificial pathways creates new relationships between sensory input and motor output, which the brain must learn to gain dexterous control. This review highlights the role of learning in BMIs to restore movement and sensation, and discusses how BMI design may influence neural plasticity and performance. The close integration of plasticity in sensory and motor function influences the design of both artificial pathways and will be an essential consideration for bidirectional devices that restore both sensory and motor function.

Information sparseness in cortical microelectrode channels while decoding movement direction using an artificial neural network

Implanted microelectrode arrays can directly pick up electrode signals from the primary motor cortex (M1) during movement, and brain-machine interfaces (BMIs) can decode these signals to predict the directions of contemporaneous movements. However, it is not well known how much each individual input is responsible for the overall performance of a BMI decoder. In this paper, we seek to quantify how much each channel contributes to an artificial neural network (ANN)-based decoder, by measuring how much the removal of each individual channel degrades the accuracy of the output. If information on movement direction was equally distributed among channels, then the removal of one would have a minimal effect on decoder accuracy. On the other hand, if that information was distributed sparsely, then the removal of specific information-rich channels would significantly lower decoder accuracy. We found that for most channels, their removal did not significantly affect decoder performance. However, for a subset of channels (16 out of 61), removing them significantly reduced the decoder accuracy. This suggests that information is not uniformly distributed among the recording channels. We propose examining these channels further to optimize BMIs more effectively, as well as understand how M1 functions at the neuronal level.

Increasing Robustness of Brain-Computer Interfaces Through Automatic Detection and Removal of Corrupted Input Signals

For brain–computer interfaces (BCIs) to be viable for long-term daily usage, they must be able to quickly identify and adapt to signal disruptions. Furthermore, the detection and mitigation steps need to occur automatically and without the need for user intervention while also being computationally tractable for the low-power hardware that will be used in a deployed BCI system. Here, we focus on disruptions that are likely to occur during chronic use that cause some recording channels to fail but leave the remaining channels unaffected. In these cases, the algorithm that translates recorded neural activity into actions, the neural decoder, should seamlessly identify and adjust to the altered neural signals with minimal inconvenience to the user. First, we introduce an adapted statistical process control (SPC) method that automatically identifies disrupted channels so that both decoding algorithms can be adjusted, and technicians can be alerted. Next, after identifying corrupted channels, we demonstrate the automated and rapid removal of channels from a neural network decoder using a masking approach that does not change the decoding architecture, making it amenable for transfer learning. Finally, using transfer and unsupervised learning techniques, we update the model weights to adjust for the corrupted channels without requiring the user to collect additional calibration data. We demonstrate with both real and simulated neural data that our approach can maintain high-performance while simultaneously minimizing computation time and data storage requirements. This framework is invisible to the user but can dramatically increase BCI robustness and usability.

The science and engineering behind sensitized brain-controlled bionic hands

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

Cortical Control of Virtual Self-Motion Using Task-Specific Subspaces

Brain-machine interfaces (BMIs) for reaching have enjoyed continued performance improvements, yet there remains significant need for BMIs that control other movement classes. Recent scientific findings suggest that the intrinsic covariance structure of neural activity depends strongly on movement class, potentially necessitating different decode algorithms across classes. To address this possibility, we developed a self-motion BMI based on cortical activity as monkeys cycled a hand-held pedal to progress along a virtual track. Unlike during reaching, we found no high-variance dimensions that directly correlated with to-be-decoded variables. This was due to no neurons having consistent correlations between their responses and kinematic variables. Yet we could decode a single variable-self-motion-by nonlinearly leveraging structure that spanned multiple high-variance neural dimensions. Resulting online BMI-control success rates approached those during manual control. These findings make two broad points regarding how to build decode algorithms that harmonize with the empirical structure of neural activity in motor cortex. First, even when decoding from the same cortical region (e.g., arm-related motor cortex), different movement classes may need to employ very different strategies. Although correlations between neural activity and hand velocity are prominent during reaching tasks, they are not a fundamental property of motor cortex and cannot be counted on to be present in general. Second, although one generally desires a low-dimensional readout, it can be beneficial to leverage a multidimensional high-variance subspace. Fully embracing this approach requires highly nonlinear approaches tailored to the task at hand, but can produce near-native levels of performance.SIGNIFICANCE STATEMENT Many brain-machine interface decoders have been constructed for controlling movements normally performed with the arm. Yet it is unclear how these will function beyond the reach-like scenarios where they were developed. Existing decoders implicitly assume that neural covariance structure, and correlations with to-be-decoded kinematic variables, will be largely preserved across tasks. We find that the correlation between neural activity and hand kinematics, a feature typically exploited when decoding reach-like movements, is essentially absent during another task performed with the arm: cycling through a virtual environment. Nevertheless, the use of a different strategy, one focused on leveraging the highest-variance neural signals, supported high performance real-time brain-machine interface control.

Unsupervised Channel Compression Methods in Motor Prostheses Design

The development of high performance brain machine interfaces (BMIs) requires scaling recording channel count to enable simultaneous recording from large populations of neurons. Unfortunately, proposed implantable neural interfaces have power requirements that scale linearly with channel count. To facilitate the design of interfaces with reduced power requirements, we propose and evaluate an unsupervised-learning-based compressed sensing strategy. This strategy suggests novel neural interface architectures which compress neural data by methodically combining channels of spiking activity. We develop an entropy-based compression strategy that models the population of neurons as being generated from a lower dimensional set of latent variables and aims to minimize the loss of information in the latent variables due to compression. We evaluate compressed features by inferring the latent variables from these features and measuring the accuracy with which the activity of held out neurons and arm movements can be estimated. We apply these methods to different cortical regions (PMd and M1) and compare the proposed compression methods to a random projections strategy often employed for compressed sensing and to a supervised regression based channel dropping strategy traditionally applied in BMI applications.

A Framework for Optimizing Co-adaptation in Body-Machine Interfaces

The operation of a human-machine interface is increasingly often referred to as a two-learners problem, where both the human and the interface independently adapt their behavior based on shared information to improve joint performance over a specific task. Drawing inspiration from the field of body-machine interfaces, we take a different perspective and propose a framework for studying co-adaptation in scenarios where the evolution of the interface is dependent on the users' behavior and that do not require task goals to be explicitly defined. Our mathematical description of co-adaptation is built upon the assumption that the interface and the user agents co-adapt toward maximizing the interaction efficiency rather than optimizing task performance. This work describes a mathematical framework for body-machine interfaces where a naïve user interacts with an adaptive interface. The interface, modeled as a linear map from a space with high dimension (the user input) to a lower dimensional feedback, acts as an adaptive “tool” whose goal is to minimize transmission loss following an unsupervised learning procedure and has no knowledge of the task being performed by the user. The user is modeled as a non-stationary multivariate Gaussian generative process that produces a sequence of actions that is either statistically independent or correlated. Dependent data is used to model the output of an action selection module concerned with achieving some unknown goal dictated by the task. The framework assumes that in parallel to this explicit objective, the user is implicitly learning a suitable but not necessarily optimal way to interact with the interface. Implicit learning is modeled as use-dependent learning modulated by a reward-based mechanism acting on the generative distribution. Through simulation, the work quantifies how the system evolves as a function of the learning time scales when a user learns to operate a static vs. an adaptive interface. We show that this novel framework can be directly exploited to readily simulate a variety of interaction scenarios, to facilitate the exploration of the parameters that lead to optimal learning dynamics of the joint system, and to provide an empirical proof for the superiority of human-machine co-adaptation over user adaptation.

Low-frequency stimulation enhances ensemble co-firing and dexterity after stroke

Electrical stimulation is a promising tool for modulating brain networks. However, it is unclear how stimulation interacts with neural patterns underlying behavior. Specifically, how might external stimulation that is not sensitive to the state of ongoing neural dynamics reliably augment neural processing and improve function? Here, we tested how low-frequency epidural alternating current stimulation (ACS) in non-human primates recovering from stroke interacted with task-related activity in perilesional cortex and affected grasping. We found that ACS increased co-firing within task-related ensembles and improved dexterity. Using a neural network model, we found that simulated ACS drove ensemble co-firing and enhanced propagation of neural activity through parts of the network with impaired connectivity, suggesting a mechanism to link increased co-firing to enhanced dexterity. Together, our results demonstrate that ACS restores neural processing in impaired networks and improves dexterity following stroke. More broadly, these results demonstrate approaches to optimize stimulation to target neural dynamics.

Multiscale low-dimensional motor cortical state dynamics predict naturalistic reach-and-grasp behavior

Motor function depends on neural dynamics spanning multiple spatiotemporal scales of population activity, from spiking of neurons to larger-scale local field potentials (LFP). How multiple scales of low-dimensional population dynamics are related in control of movements remains unknown. Multiscale neural dynamics are especially important to study in naturalistic reach-and-grasp movements, which are relatively under-explored. We learn novel multiscale dynamical models for spike-LFP network activity in monkeys performing naturalistic reach-and-grasps. We show low-dimensional dynamics of spiking and LFP activity exhibited several principal modes, each with a unique decay-frequency characteristic. One principal mode dominantly predicted movements. Despite distinct principal modes existing at the two scales, this predictive mode was multiscale and shared between scales, and was shared across sessions and monkeys, yet did not simply replicate behavioral modes. Further, this multiscale mode's decay-frequency explained behavior. We propose that multiscale, low-dimensional motor cortical state dynamics reflect the neural control of naturalistic reach-and-grasp behaviors.

Multi-source domain adaptation for decoder calibration of intracortical brain-machine interface

OBJECTIVE

For nonstationarity of neural recordings, daily retraining is required in the decoder calibration of intracortical brain-machine interfaces (iBMIs). Domain adaptation has started to be applied in iBMIs to solve the problem of daily retraining by taking advantage of historical data. However, previous domain adaptation studies used only a single source domain, which might lead to performance instability. In this study, we proposed a multi-source domain adaptation algorithm, by fully utilizing the historical data, to achieve a better and more robust decoding performance while reducing the decoder calibration time.

APPROACH

The neural signals were recorded from two rhesus macaques using intracortical electrodes to decode the reaching and grasping movements. A principal component analysis-based multi-source domain adaptation algorithm was proposed to apply the feature transfer to diminish the disparities between the target domain and each source domain. Moreover, the multiple weighted sub-classifiers based on multi-source domain data and small current sample set were constructed to accomplish the decoding.

MAIN RESULTS

Our algorithm was able to make use of the multi-source domain data and achieve better and more robust decoding performance compared with other methods. Only a small current sample set was needed by our algorithm in order for the decoder calibration time to be effectively reduced.

SIGNIFICANCE

(1) The idea of the multi-source domain adaptation was introduced into the iBMIs to solve the problem of time consumption in the daily decoder retraining. (2) Instead of using only single-source domain data in the previous study, our algorithm made use of multi-day historical data, resulting in better and more robust decoding performance. (3) Our algorithm could be accomplished with only a small current sample set, and it can effectively reduce the decoder calibration time, which is important for further clinical applications.

Classifying Intracortical Brain-Machine Interface Signal Disruptions Based on System Performance and Applicable Compensatory Strategies: A Review

Brain-machine interfaces (BMIs) record and translate neural activity into a control signal for assistive or other devices. Intracortical microelectrode arrays (MEAs) enable high degree-of-freedom BMI control for complex tasks by providing fine-resolution neural recording. However, chronically implanted MEAs are subject to a dynamic in vivo environment where transient or systematic disruptions can interfere with neural recording and degrade BMI performance. Typically, neural implant failure modes have been categorized as biological, material, or mechanical. While this categorization provides insight into a disruption's causal etiology, it is less helpful for understanding degree of impact on BMI function or possible strategies for compensation. Therefore, we propose a complementary classification framework for intracortical recording disruptions that is based on duration of impact on BMI performance and requirement for and responsiveness to interventions: (1) Transient disruptions interfere with recordings on the time scale of minutes to hours and can resolve spontaneously; (2) Reversible disruptions cause persistent interference in recordings but the root cause can be remedied by an appropriate intervention; (3) Irreversible compensable disruptions cause persistent or progressive decline in signal quality, but their effects on BMI performance can be mitigated algorithmically; and (4) Irreversible non-compensable disruptions cause permanent signal loss that is not amenable to remediation or compensation. This conceptualization of intracortical BMI disruption types is useful for highlighting specific areas for potential hardware improvements and also identifying opportunities for algorithmic interventions. We review recording disruptions that have been reported for MEAs and demonstrate how biological, material, and mechanical mechanisms of disruption can be further categorized according to their impact on signal characteristics. Then we discuss potential compensatory protocols for each of the proposed disruption classes. Specifically, transient disruptions may be minimized by using robust neural decoder features, data augmentation methods, adaptive machine learning models, and specialized signal referencing techniques. Statistical Process Control methods can identify reparable disruptions for rapid intervention. In-vivo diagnostics such as impedance spectroscopy can inform neural feature selection and decoding models to compensate for irreversible disruptions. Additional compensatory strategies for irreversible disruptions include information salvage techniques, data augmentation during decoder training, and adaptive decoding methods to down-weight damaged channels.

Stabilization of a brain-computer interface via the alignment of low-dimensional spaces of neural activity

The instability of neural recordings can render clinical brain-computer interfaces (BCIs) uncontrollable. Here, we show that the alignment of low-dimensional neural manifolds (low-dimensional spaces that describe specific correlation patterns between neurons) can be used to stabilize neural activity, thereby maintaining BCI performance in the presence of recording instabilities. We evaluated the stabilizer with non-human primates during online cursor control via intracortical BCIs in the presence of severe and abrupt recording instabilities. The stabilized BCIs recovered proficient control under different instability conditions and across multiple days. The stabilizer does not require knowledge of user intent and can outperform supervised recalibration. It stabilized BCIs even when neural activity contained little information about the direction of cursor movement. The stabilizer may be applicable to other neural interfaces and may improve the clinical viability of BCIs.

Long-term stability of cortical population dynamics underlying consistent behavior

Animals readily execute learned behaviors in a consistent manner over long periods of time, and yet no equally stable neural correlate has been demonstrated. How does the cortex achieve this stable control? Using the sensorimotor system as a model of cortical processing, we investigated the hypothesis that the dynamics of neural latent activity, which captures the dominant co-variation patterns within the neural population, must be preserved across time. We recorded from populations of neurons in premotor, primary motor and somatosensory cortices as monkeys performed a reaching task, for up to 2 years. Intriguingly, despite a steady turnover in the recorded neurons, the low-dimensional latent dynamics remained stable. The stability allowed reliable decoding of behavioral features for the entire timespan, while fixed decoders based directly on the recorded neural activity degraded substantially. We posit that stable latent cortical dynamics within the manifold are the fundamental building blocks underlying consistent behavioral execution.

Structure and variability of delay activity in premotor cortex

Voluntary movements are widely considered to be planned before they are executed. Recent studies have hypothesized that neural activity in motor cortex during preparation acts as an ‘initial condition’ which seeds the proceeding neural dynamics. Here, we studied these initial conditions in detail by investigating 1) the organization of neural states for different reaches and 2) the variance of these neural states from trial to trial. We examined population-level responses in macaque premotor cortex (PMd) during the preparatory stage of an instructed-delay center-out reaching task with dense target configurations. We found that after target onset the neural activity on single trials converges to neural states that have a clear low-dimensional structure which is organized by both the reach endpoint and maximum speed of the following reach. Further, we found that variability of the neural states during preparation resembles the spatial variability of reaches made in the absence of visual feedback: there is less variability in direction than distance in neural state space. We also used offline decoding to understand the implications of this neural population structure for brain-machine interfaces (BMIs). We found that decoding of angle between reaches is dependent on reach distance, while decoding of arc-length is independent. Thus, it might be more appropriate to quantify decoding performance for discrete BMIs by using arc-length between reach end-points rather than the angle between them. Lastly, we show that in contrast to the common notion that direction can better be decoded than distance, their decoding capabilities are comparable. These results provide new insights into the dynamical neural processes that underline motor control and can inform the design of BMIs.

Early studies of premotor cortex explored how individual neurons directly encode aspects of an upcoming movement during preparation. Recent developments have proposed that the dynamics of populations of neurons underlie motor control, and that neural activity during preparation serves to set up these dynamics. While the dynamics of motor control have been studied extensively, several aspects of preparatory activity remain unresolved. Here, we ask how the patterns of neural activity during preparation for different reaches are related to one another. We found that the neural activity during preparation for reaches to different targets has a clear ‘structure’. Additionally, we found that the activity on a given trial was predictive of the initial trajectory of the reach. Lastly, we assessed the implications of our findings for predicting upcoming movements from neural activity, as in brain-machine interfaces.

Innovations in electrical stimulation harness neural plasticity to restore motor function

Novel technology and innovative stimulation paradigms allow for unprecedented spatiotemporal precision and closed-loop implementation of neurostimulation systems. In turn, precise, closed-loop neurostimulation appears to preferentially drive neural plasticity in motor networks, promoting neural repair. Recent clinical studies demonstrate that electrical stimulation can drive neural plasticity in damaged motor circuits, leading to meaningful improvement in users. Future advances in these areas hold promise for the treatment of a wide range of motor systems disorders.

Estimation of Functional Dependence in High-Dimensional Spike-Field Activity

Behavior is encoded across spatiotemoral scales of brain activity, from small-scale spikes to large-scale local field potentials (LFP). Identifying the functional dependence between spikes and LFP networks during behavior can help understand neural encoding and improve future neurotechnologies, but is difficult to achieve. First, spikes and LFP have different statistical characteristics (binary spikes vs. continuous LFPs) and time-scales. Second, given the prohibitively large number of spike channels and LFP features recorded in today's experiments, learning dependencies between all recorded signals is challenging and prone to overfitting. To solve this challenge, we present a model-based approach to estimate the functional dependence between high-dimensional field features and neuronal spikes. We model the binary time-series of spikes for each neuron as a point process dependent on the behavioral states and LFP features across the network. Given the prohibitively large number of possible spike-LFP dependency parameters, we first employ an Ll-regularization technique to learn the point process model during both supervised and unsupervised learning to ease detection of significant dependency parameters. We then use the Akaike information criterion (AIC) to enforce model sparsity by incorporating only a minimum number of non-zero dependency parameters into the point process model based on a trade-off between model complexity and its prediction power. Using extensive numerical simulations, we show that our method (i) can correctly identify the functional dependencies and thus improve the prediction of spiking activity and (ii) can improve the prediction of spiking activity with significantly fewer number of parameters compared to when regularization is not enforced. Our approach may serve as a tool to investigate brain connectivity patterns across spatiotemporal scales.

Latent Factors and Dynamics in Motor Cortex and Their Application to Brain-Machine Interfaces

In the 1960s, Evarts first recorded the activity of single neurons in motor cortex of behaving monkeys (Evarts, 1968). In the 50 years since, great effort has been devoted to understanding how single neuron activity relates to movement. Yet these single neurons exist within a vast network, the nature of which has been largely inaccessible. With advances in recording technologies, algorithms, and computational power, the ability to study these networks is increasing exponentially. Recent experimental results suggest that the dynamical properties of these networks are critical to movement planning and execution. Here we discuss this dynamical systems perspective and how it is reshaping our understanding of the motor cortices. Following an overview of key studies in motor cortex, we discuss techniques to uncover the "latent factors" underlying observed neural population activity. Finally, we discuss efforts to use these factors to improve the performance of brain-machine interfaces, promising to make these findings broadly relevant to neuroengineering as well as systems neuroscience.

Implantable computer-controlled adaptive multielectrode positioning system

Acute neuronal recordings performed with metal microelectrodes in nonhuman primates allow investigating the neural substrate of complex cognitive behaviors. Yet the daily reinsertion and positioning of the electrodes prevents recording from many neurons simultaneously, limiting the suitability of these types of recordings for brain-computer interface applications or for large-scale population statistical methods on a trial-by-trial basis. In contrast, chronically implanted multielectrode arrays offer the opportunity to record from many neurons simultaneously, but immovable electrodes prevent optimization of the signal during and after implantation and cause the tissue response to progressively impair the transduced signal quality, thereby limiting the number of different neurons that can be recorded over the lifetime of the implant. Semichronically implanted matrices of electrodes, instead, allow individually movable electrodes in depth and achieve higher channel count compared with acute methods, hence partially overcoming these limitations. Existing semichronic systems with higher channel count lack computerized control of electrode movements, leading to limited user-friendliness and uncertainty in depth positioning. Here we demonstrate a chronically implantable adaptive multielectrode positioning system with detachable drive for computerized depth adjustment of individual electrodes over several millimeters. This semichronic 16-channel system is designed to optimize the simultaneous yield of units in an extended period following implantation since the electrodes can be independently depth adjusted with minimal effort and their signal quality continuously assessed. Importantly, the electrode array is designed to remain within a chronic recording chamber for a prolonged time or can be used for acute recordings with high signal-to-noise ratio in the cerebral cortex of nonhuman primates.

NEW & NOTEWORTHY We present a 16-channel motorized, semichronic multielectrode array with individually depth-adjustable electrodes to record in the cerebral cortex of nonhuman primates. Compared with fixed-geometry arrays, this system allows repeated reestablishing of single neuron isolation. Compared with manually adjustable arrays it benefits from computer-controlled positioning. Compared with motorized semichronic systems it allows higher channel counts due to a robotic single actuator approach. Overall the system is designed to optimize the simultaneous yield of units over the course of implantation.