Towards autonomous neuroprosthetic control using Hebbian reinforcement learning

A Kernel Reinforcement Learning Decoding Framework Integrating Neural and Feedback Signals for Brain Control

Brain-Machine Interfaces (BMIs) have the potential to allow subjects to brain control (BC) external devices, where their brain signals could be translated to the action of the neuro-prosthesis by reinforcement learning (RL) based decoder. During the BC task, feedback cues are provided to guide subject's learning. Subjects will adapt the neural signals according to the feedback cues. Concurrently, the RL decoding parameters are adjusted when the subject explores the BC task through trial and error, leading to a co-adaptive process between the subject and the decoder. However, when subjects receive the feedback cues and enhance their learning, the decoder does not actively utilize the feedback cues. If the RL decoder could integrate both neural signals and feedback cues, the training efficiency of the BC task would increase. A major challenge is the different temporal scales of neural signals and feedback cues, making it difficult to integrate them into a single decoder. In this paper, we propose a novel kernel RL decoding method as the first attempt to combine two signals with different temporal scales for RL decoding. The neural signals and the feedback cues comprise the decoding input, which is then projected into individual Reproducing Kernel Hilbert Spaces (RKHSs) respectively. These two RKHSs form a joint feature space, where the action of the neuro-prosthesis could be decoded linearly. We evaluate the proposed method on a simulated brain control cursor-reaching task. Our proposed method is compared with the kernel RL that only uses neural signals as the input. The proposed method has a faster learning speed and better decoding accuracy. The results demonstrate that our proposed method has successfully integrated the information of the feedback cue and facilitates the training procedure for the BC task.Clinical Relevance-This paper provides an integrated reinforcement learning decoding framework, which combines the neural signals and the feedback cues to increase the learning speed and the accuracy of the brain control task. Subjects could learn the task more easily with this decoder.

Neural Decoders Using Reinforcement Learning in Brain Machine Interfaces: A Technical Review

Creating flexible and robust brain machine interfaces (BMIs) is currently a popular topic of research that has been explored for decades in medicine, engineering, commercial, and machine-learning communities. In particular, the use of techniques using reinforcement learning (RL) has demonstrated impressive results but is under-represented in the BMI community. To shine more light on this promising relationship, this article aims to provide an exhaustive review of RL's applications to BMIs. Our primary focus in this review is to provide a technical summary of various algorithms used in RL-based BMIs to decode neural intention, without emphasizing preprocessing techniques on the neural signals and reward modeling for RL. We first organize the literature based on the type of RL methods used for neural decoding, and then each algorithm's learning strategy is explained along with its application in BMIs. A comparative analysis highlighting the similarities and uniqueness among neural decoders is provided. Finally, we end this review with a discussion about the current stage of RLBMIs including their limitations and promising directions for future research.

A behavioral paradigm for cortical control of a robotic actuator by freely moving rats in a one-dimensional two-target reaching task

BACKGROUND

Controlling the trajectory of a neuroprosthesis to reach distant targets is a commonly used brain-machine interface (BMI) task in primates and has not been available for rodents yet.

NEW METHOD

Here, we describe a novel, fine-tuned behavioral paradigm and setup which enables this task for rats in one-dimensional space for reaching two distant targets depending on their limited cognitive and visual capabilities compared to those of primates. An online transform was used to convert the activity of a pair of primary motor cortex (M1) units into two robotic actions. The rats were shaped to adapt to the transform and direct the robotic actuator toward the selected target by modulating the activity of the M1 neurons.

RESULTS

All three rats involved in the study were capable of achieving randomly selected targets with at least 78% accuracy. A total of 9 out of 16 pairs of units examined were eligible for exceeding this success criterion. Two out of three rats were capable of reversal learning, where the mapping between the activity of the M1 units and the robotic actions were reversed.

COMPARISON WITH EXISTING METHODS

The present work is the first demonstration of trajectory-based control of a neuroprosthetic device by rodents to reach two distant targets using visual feedback.

CONCLUSION

The behavioral paradigm and setup introduced here can be used as a cost-effective platform for elucidating the information processing principles in the neural circuits related to neuroprosthetic control and for studying the performance of novel BMI technologies using freely moving rats.

A behavioral paradigm for cortical control of a robotic actuator by freely moving rats in a one-dimensional two-target reaching task.. [DOI: 10.1101/2021.01.24.427325] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Controlling the trajectory of a neuroprosthetic device to reach multiple targets is a commonly used brain-machine interface (BMI) task in primates and has not been available for rodents yet. Here, we describe a novel experimental paradigm which enables this task for rats in one-dimensional space for reaching two distant targets depending on their limited cognitive and visual capabilities compared to primates. An online transform was used to convert the activity of a pair of primary motor cortex (M1) units into two robotic actions. The rats were shaped to adapt to the transform and direct the robotic actuator toward the selected target by modulating the activity of the M1 neurons. All three rats involved in the study were capable of achieving randomly selected targets with at least 78% accuracy. A total of 9 out of 16 pairs of units examined were eligible for exceeding this success criterion. Two out of three rats were capable of reversal learning, where the mapping between the activity of the unit pairs and the robotic actions were reversed. The present work is the first demonstration of trajectory-based control of a neuroprosthetic device by rodents to reach two distant targets using visual feedback. The paradigm introduced here may be used as a cost-effective platform for elucidating the information processing principles in the neural circuits related to neuroprosthetic control and for studying the performance of novel BMI technologies using freely moving rats.

Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain-Machine Interface

BACKGROUND

For the nonstationarity of neural recordings in intracortical brain-machine interfaces, daily retraining in a supervised manner is always required to maintain the performance of the decoder. This problem can be improved by using a reinforcement learning (RL) based self-recalibrating decoder. However, quickly exploring new knowledge while maintaining a good performance remains a challenge in RL-based decoders.

METHODS

To solve this problem, we proposed an attention-gated RL-based algorithm combining transfer learning, mini-batch, and weight updating schemes to accelerate the weight updating and avoid over-fitting. The proposed algorithm was tested on intracortical neural data recorded from two monkeys to decode their reaching positions and grasping gestures.

RESULTS

The decoding results showed that our proposed algorithm achieved an approximate 20% increase in classification accuracy compared to that obtained by the non-retrained classifier and even achieved better classification accuracy than the daily retraining classifier. Moreover, compared with a conventional RL method, our algorithm improved the accuracy by approximately 10% and the online weight updating speed by approximately 70 times.

CONCLUSIONS

This paper proposed a self-recalibrating decoder which achieved a good and robust decoding performance with fast weight updating and might facilitate its application in wearable device and clinical practice.

Single-Finger Neural Basis Information-Based Neural Decoder for Multi-Finger Movements

In this paper, we investigate the relationship between single and multi-finger movements. By exploiting the neural correlation between the temporal firing patterns between movements, we show that the Pearson's correlation coefficient for the physically related movement pairs are greater than those of others; the firing rates of the neurons that are tuned to a single-finger movements also increases when the corresponding multi-finger movements are instructed. We also use a hierarchical cluster analysis to verify not only the relationship between the single and multi-finger movements, but also the relationship between the flexion and extension movements. Furthermore, we propose a novel decoding method of modeling neural firing patterns while omitting the training process of the multi-finger movements. For the decoding, the Skellam and Gaussian probability distributions are used as mathematical models. The probabilistic distribution model of the multi-finger movements was estimated using the neural activity that was acquired during single-finger movements. As a result, the proposed neural decoding accuracy comparable with that of the supervised neural decoding accuracy when all of the neurons were used for the multi-finger movements. These results suggest that only the neural activities of single-finger movements can be exploited for the control of dexterous multi-finger neuroprosthetics.

Robust Closed-Loop Control of a Cursor in a Person with Tetraplegia using Gaussian Process Regression

Intracortical brain computer interfaces can enable individuals with paralysis to control external devices through voluntarily modulated brain activity. Decoding quality has been previously shown to degrade with signal nonstationarities-specifically, the changes in the statistics of the data between training and testing data sets. This includes changes to the neural tuning profiles and baseline shifts in firing rates of recorded neurons, as well as nonphysiological noise. While progress has been made toward providing long-term user control via decoder recalibration, relatively little work has been dedicated to making the decoding algorithm more resilient to signal nonstationarities. Here, we describe how principled kernel selection with gaussian process regression can be used within a Bayesian filtering framework to mitigate the effects of commonly encountered nonstationarities. Given a supervised training set of (neural features, intention to move in a direction)-pairs, we use gaussian process regression to predict the intention given the neural data. We apply kernel embedding for each neural feature with the standard radial basis function. The multiple kernels are then summed together across each neural dimension, which allows the kernel to effectively ignore large differences that occur only in a single feature. The summed kernel is used for real-time predictions of the posterior mean and variance under a gaussian process framework. The predictions are then filtered using the discriminative Kalman filter to produce an estimate of the neural intention given the history of neural data. We refer to the multiple kernel approach combined with the discriminative Kalman filter as the MK-DKF. We found that the MK-DKF decoder was more resilient to nonstationarities frequently encountered in-real world settings yet provided similar performance to the currently used Kalman decoder. These results demonstrate a method by which neural decoding can be made more resistant to nonstationarities.

Applications of Deep Learning and Reinforcement Learning to Biological Data

Rapid advances in hardware-based technologies during the past decades have opened up new possibilities for life scientists to gather multimodal data in various application domains, such as omics, bioimaging, medical imaging, and (brain/body)-machine interfaces. These have generated novel opportunities for development of dedicated data-intensive machine learning techniques. In particular, recent research in deep learning (DL), reinforcement learning (RL), and their combination (deep RL) promise to revolutionize the future of artificial intelligence. The growth in computational power accompanied by faster and increased data storage, and declining computing costs have already allowed scientists in various fields to apply these techniques on data sets that were previously intractable owing to their size and complexity. This paper provides a comprehensive survey on the application of DL, RL, and deep RL techniques in mining biological data. In addition, we compare the performances of DL techniques when applied to different data sets across various application domains. Finally, we outline open issues in this challenging research area and discuss future development perspectives.

Model of a bilateral Brown-type central pattern generator for symmetric and asymmetric locomotion

The coordinated activity of muscles is produced in part by spinal rhythmogenic neural circuits, termed central pattern generators (CPGs). A classical CPG model is a system of coupled oscillators that transform locomotor drive into coordinated and gait-specific patterns of muscle recruitment. The network properties of this conceptual model can be simulated by a system of ordinary differential equations with a physiologically inspired coupling locus of interactions capturing the timing relationship for bilateral coordination of limbs in locomotion. Whereas most similar models are solved numerically, it is intriguing to have a full analytical description of this plausible CPG architecture to illuminate the functionality within this structure and to expand it to include steering control. Here, we provided a closed-form analytical solution contrasted against the previous numerical method. The evaluation time of the analytical solution was decreased by an order of magnitude when compared with the numerical approach (relative errors, <0.01%). The analytical solution tested and supported the previous finding that the input to the model can be expressed in units of the desired limb locomotor speed. Furthermore, we performed parametric sensitivity analysis in the context of controlling steering and documented two possible mechanisms associated with either an external drive or intrinsic CPG parameters. The results identify specific propriospinal pathways that may be associated with adaptations within the CPG structure. The model offered several network configurations that may generate the same behavioral outcomes. NEW & NOTEWORTHY Using a simple process of leaky integration, we developed an analytical solution to a robust model of spinal pattern generation. We analyzed the ability of this neural element to exert locomotor control of the signal associated with limb speeds and tested the ability of this simple structure to embed steering control using the velocity signal in the model's inputs or within the internal connectivity of its elements.

Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation

Brain-machine interfaces (BMIs) combine methods, approaches, and concepts derived from neurophysiology, computer science, and engineering in an effort to establish real-time bidirectional links between living brains and artificial actuators. Although theoretical propositions and some proof of concept experiments on directly linking the brains with machines date back to the early 1960s, BMI research only took off in earnest at the end of the 1990s, when this approach became intimately linked to new neurophysiological methods for sampling large-scale brain activity. The classic goals of BMIs are 1) to unveil and utilize principles of operation and plastic properties of the distributed and dynamic circuits of the brain and 2) to create new therapies to restore mobility and sensations to severely disabled patients. Over the past decade, a wide range of BMI applications have emerged, which considerably expanded these original goals. BMI studies have shown neural control over the movements of robotic and virtual actuators that enact both upper and lower limb functions. Furthermore, BMIs have also incorporated ways to deliver sensory feedback, generated from external actuators, back to the brain. BMI research has been at the forefront of many neurophysiological discoveries, including the demonstration that, through continuous use, artificial tools can be assimilated by the primate brain's body schema. Work on BMIs has also led to the introduction of novel neurorehabilitation strategies. As a result of these efforts, long-term continuous BMI use has been recently implicated with the induction of partial neurological recovery in spinal cord injury patients.

Quantized Attention-Gated Kernel Reinforcement Learning for Brain-Machine Interface Decoding

Reinforcement learning (RL)-based decoders in brain-machine interfaces (BMIs) interpret dynamic neural activity without patients' real limb movements. In conventional RL, the goal state is selected by the user or defined by the physics of the problem, and the decoder finds an optimal policy essentially by assigning credit over time, which is normally very time-consuming. However, BMI tasks require finding a good policy in very few trials, which impose a limit on the complexity of the tasks that can be learned before the animal quits. Therefore, this paper explores the possibility of letting the agent infer potential goals through actions over space with multiple objects, using the instantaneous reward to assign credit spatially. A previous method, attention-gated RL employs a multilayer perceptron trained with backpropagation, but it is prone to local minima entrapment. We propose a quantized attention-gated kernel RL (QAGKRL) to avoid the local minima adaptation in spatial credit assignment and sparsify the network topology. The experimental results show that the QAGKRL achieves higher successful rates and more stable performance, indicating its powerful decoding ability for more sophisticated BMI tasks as required in clinical applications.

Feedback for reinforcement learning based brain-machine interfaces using confidence metrics

OBJECTIVE

For brain-machine interfaces (BMI) to be used in activities of daily living by paralyzed individuals, the BMI should be as autonomous as possible. One of the challenges is how the feedback is extracted and utilized in the BMI. Our long-term goal is to create autonomous BMIs that can utilize an evaluative feedback from the brain to update the decoding algorithm and use it intelligently in order to adapt the decoder. In this study, we show how to extract the necessary evaluative feedback from a biologically realistic (synthetic) source, use both the quantity and the quality of the feedback, and how that feedback information can be incorporated into a reinforcement learning (RL) controller architecture to maximize its performance.

APPROACH

Motivated by the perception-action-reward cycle (PARC) in the brain which links reward for cognitive decision making and goal-directed behavior, we used a reward-based RL architecture named Actor-Critic RL as the model. Instead of using an error signal towards building an autonomous BMI, we envision to use a reward signal from the nucleus accumbens (NAcc) which plays a key role in the linking of reward to motor behaviors. To deal with the complexity and non-stationarity of biological reward signals, we used a confidence metric which was used to indicate the degree of feedback accuracy. This confidence was added to the Actor's weight update equation in the RL controller architecture. If the confidence was high (>0.2), the BMI decoder used this feedback to update its parameters. However, when the confidence was low, the BMI decoder ignored the feedback and did not update its parameters. The range between high confidence and low confidence was termed as the 'ambiguous' region. When the feedback was within this region, the BMI decoder updated its weight at a lower rate than when fully confident, which was decided by the confidence. We used two biologically realistic models to generate synthetic data for MI (Izhikevich model) and NAcc (Humphries model) to validate proposed controller architecture.

MAIN RESULTS

In this work, we show how the overall performance of the BMI was improved by using a threshold close to the decision boundary to reject erroneous feedback. Additionally, we show the stability of the system improved when the feedback was used with a threshold.

SIGNIFICANCE

The result of this study is a step towards making BMIs autonomous. While our method is not fully autonomous, the results demonstrate that extensive training times necessary at the beginning of each BMI session can be significantly decreased. In our approach, decoder training time was only limited to 10 trials in the first BMI session. Subsequent sessions used previous session weights to initialize the decoder. We also present a method where the use of a threshold can be applied to any decoder with a feedback signal that is less than perfect so that erroneous feedback can be avoided and the stability of the system can be increased.

A Review of Control Strategies in Closed-Loop Neuroprosthetic Systems

It has been widely recognized that closed-loop neuroprosthetic systems achieve more favorable outcomes for users then equivalent open-loop devices. Improved performance of tasks, better usability, and greater embodiment have all been reported in systems utilizing some form of feedback. However, the interdisciplinary work on neuroprosthetic systems can lead to miscommunication due to similarities in well-established nomenclature in different fields. Here we present a review of control strategies in existing experimental, investigational and clinical neuroprosthetic systems in order to establish a baseline and promote a common understanding of different feedback modes and closed-loop controllers. The first section provides a brief discussion of feedback control and control theory. The second section reviews the control strategies of recent Brain Machine Interfaces, neuromodulatory implants, neuroprosthetic systems, and assistive neurorobotic devices. The final section examines the different approaches to feedback in current neuroprosthetic and neurorobotic systems.

Intracortical Brain-Machine Interfaces Advance Sensorimotor Neuroscience

Brain-machine interfaces (BMIs) decode brain activity to control external devices. Over the past two decades, the BMI community has grown tremendously and reached some impressive milestones, including the first human clinical trials using chronically implanted intracortical electrodes. It has also contributed experimental paradigms and important findings to basic neuroscience. In this review, we discuss neuroscience achievements stemming from BMI research, specifically that based upon upper limb prosthetic control with intracortical microelectrodes. We will focus on three main areas: first, we discuss progress in neural coding of reaches in motor cortex, describing recent results linking high dimensional representations of cortical activity to muscle activation. Next, we describe recent findings on learning and plasticity in motor cortex on various time scales. Finally, we discuss how bidirectional BMIs have led to better understanding of somatosensation in and related to motor cortex.

Neuroprosthetic Decoder Training as Imitation Learning

Neuroprosthetic brain-computer interfaces function via an algorithm which decodes neural activity of the user into movements of an end effector, such as a cursor or robotic arm. In practice, the decoder is often learned by updating its parameters while the user performs a task. When the user's intention is not directly observable, recent methods have demonstrated value in training the decoder against a surrogate for the user's intended movement. Here we show that training a decoder in this way is a novel variant of an imitation learning problem, where an oracle or expert is employed for supervised training in lieu of direct observations, which are not available. Specifically, we describe how a generic imitation learning meta-algorithm, dataset aggregation (DAgger), can be adapted to train a generic brain-computer interface. By deriving existing learning algorithms for brain-computer interfaces in this framework, we provide a novel analysis of regret (an important metric of learning efficacy) for brain-computer interfaces. This analysis allows us to characterize the space of algorithmic variants and bounds on their regret rates. Existing approaches for decoder learning have been performed in the cursor control setting, but the available design principles for these decoders are such that it has been impossible to scale them to naturalistic settings. Leveraging our findings, we then offer an algorithm that combines imitation learning with optimal control, which should allow for training of arbitrary effectors for which optimal control can generate goal-oriented control. We demonstrate this novel and general BCI algorithm with simulated neuroprosthetic control of a 26 degree-of-freedom model of an arm, a sophisticated and realistic end effector.

Cortical Spiking Network Interfaced with Virtual Musculoskeletal Arm and Robotic Arm

Embedding computational models in the physical world is a critical step towards constraining their behavior and building practical applications. Here we aim to drive a realistic musculoskeletal arm model using a biomimetic cortical spiking model, and make a robot arm reproduce the same trajectories in real time. Our cortical model consisted of a 3-layered cortex, composed of several hundred spiking model-neurons, which display physiologically realistic dynamics. We interconnected the cortical model to a two-joint musculoskeletal model of a human arm, with realistic anatomical and biomechanical properties. The virtual arm received muscle excitations from the neuronal model, and fed back proprioceptive information, forming a closed-loop system. The cortical model was trained using spike timing-dependent reinforcement learning to drive the virtual arm in a 2D reaching task. Limb position was used to simultaneously control a robot arm using an improved network interface. Virtual arm muscle activations responded to motoneuron firing rates, with virtual arm muscles lengths encoded via population coding in the proprioceptive population. After training, the virtual arm performed reaching movements which were smoother and more realistic than those obtained using a simplistic arm model. This system provided access to both spiking network properties and to arm biophysical properties, including muscle forces. The use of a musculoskeletal virtual arm and the improved control system allowed the robot arm to perform movements which were smoother than those reported in our previous paper using a simplistic arm. This work provides a novel approach consisting of bidirectionally connecting a cortical model to a realistic virtual arm, and using the system output to drive a robotic arm in real time. Our techniques are applicable to the future development of brain neuroprosthetic control systems, and may enable enhanced brain-machine interfaces with the possibility for finer control of limb prosthetics.

Toward Building Hybrid Biological/in silico Neural Networks for Motor Neuroprosthetic Control

In this article, we introduce the Bioinspired Neuroprosthetic Design Environment (BNDE) as a practical platform for the development of novel brain–machine interface (BMI) controllers, which are based on spiking model neurons. We built the BNDE around a hard real-time system so that it is capable of creating simulated synapses from extracellularly recorded neurons to model neurons. In order to evaluate the practicality of the BNDE for neuroprosthetic control experiments, a novel, adaptive BMI controller was developed and tested using real-time closed-loop simulations. The present controller consists of two in silico medium spiny neurons, which receive simulated synaptic inputs from recorded motor cortical neurons. In the closed-loop simulations, the recordings from the cortical neurons were imitated using an external, hardware-based neural signal synthesizer. By implementing a reward-modulated spike timing-dependent plasticity rule, the controller achieved perfect target reach accuracy for a two-target reaching task in one-dimensional space. The BNDE combines the flexibility of software-based spiking neural network (SNN) simulations with powerful online data visualization tools and is a low-cost, PC-based, and all-in-one solution for developing neurally inspired BMI controllers. We believe that the BNDE is the first implementation, which is capable of creating hybrid biological/in silico neural networks for motor neuroprosthetic control and utilizes multiple CPU cores for computationally intensive real-time SNN simulations.

Toward an autonomous brain machine interface: integrating sensorimotor reward modulation and reinforcement learning

For decades, neurophysiologists have worked on elucidating the function of the cortical sensorimotor control system from the standpoint of kinematics or dynamics. Recently, computational neuroscientists have developed models that can emulate changes seen in the primary motor cortex during learning. However, these simulations rely on the existence of a reward-like signal in the primary sensorimotor cortex. Reward modulation of the primary sensorimotor cortex has yet to be characterized at the level of neural units. Here we demonstrate that single units/multiunits and local field potentials in the primary motor (M1) cortex of nonhuman primates (Macaca radiata) are modulated by reward expectation during reaching movements and that this modulation is present even while subjects passively view cursor motions that are predictive of either reward or nonreward. After establishing this reward modulation, we set out to determine whether we could correctly classify rewarding versus nonrewarding trials, on a moment-to-moment basis. This reward information could then be used in collaboration with reinforcement learning principles toward an autonomous brain-machine interface. The autonomous brain-machine interface would use M1 for both decoding movement intention and extraction of reward expectation information as evaluative feedback, which would then update the decoding algorithm as necessary. In the work presented here, we show that this, in theory, is possible.

Optimal feedback control successfully explains changes in neural modulations during experiments with brain-machine interfaces

Recent experiments with brain-machine-interfaces (BMIs) indicate that the extent of neural modulations increased abruptly upon starting to operate the interface, and especially after the monkey stopped moving its hand. In contrast, neural modulations that are correlated with the kinematics of the movement remained relatively unchanged. Here we demonstrate that similar changes are produced by simulated neurons that encode the relevant signals generated by an optimal feedback controller during simulated BMI experiments. The optimal feedback controller relies on state estimation that integrates both visual and proprioceptive feedback with prior estimations from an internal model. The processing required for optimal state estimation and control were conducted in the state-space, and neural recording was simulated by modeling two populations of neurons that encode either only the estimated state or also the control signal. Spike counts were generated as realizations of doubly stochastic Poisson processes with linear tuning curves. The model successfully reconstructs the main features of the kinematics and neural activity during regular reaching movements. Most importantly, the activity of the simulated neurons successfully reproduces the observed changes in neural modulations upon switching to brain control. Further theoretical analysis and simulations indicate that increasing the process noise during normal reaching movement results in similar changes in neural modulations. Thus, we conclude that the observed changes in neural modulations during BMI experiments can be attributed to increasing process noise associated with the imperfect BMI filter, and, more directly, to the resulting increase in the variance of the encoded signals associated with state estimation and the required control signal.

Decoding methods for neural prostheses: where have we reached?

This article reviews advances in decoding methods for brain-machine interfaces (BMIs). Recent work has focused on practical considerations for future clinical deployment of prosthetics. This review is organized by open questions in the field such as what variables to decode, how to design neural tuning models, which neurons to select, how to design models of desired actions, how to learn decoder parameters during prosthetic operation, and how to adapt to changes in neural signals and neural tuning. The concluding discussion highlights the need to design and test decoders within the context of their expected use and the need to answer the question of how much control accuracy is good enough for a prosthetic.

Restoring sensorimotor function through intracortical interfaces: progress and looming challenges

The loss of a limb or paralysis resulting from spinal cord injury has devastating consequences on quality of life. One approach to restoring lost sensory and motor abilities in amputees and patients with tetraplegia is to supply them with implants that provide a direct interface with the CNS. Such brain-machine interfaces might enable a patient to exert voluntary control over a prosthetic or robotic limb or over the electrically induced contractions of paralysed muscles. A parallel interface could convey sensory information about the consequences of these movements back to the patient. Recent developments in the algorithms that decode motor intention from neuronal activity and in approaches to convey sensory feedback by electrically stimulating neurons, using biomimetic and adaptation-based approaches, have shown the promise of invasive interfaces with sensorimotor cortices, although substantial challenges remain.

A confidence metric for using neurobiological feedback in actor-critic reinforcement learning based brain-machine interfaces

Brain-Machine Interfaces (BMIs) can be used to restore function in people living with paralysis. Current BMIs require extensive calibration that increase the set-up times and external inputs for decoder training that may be difficult to produce in paralyzed individuals. Both these factors have presented challenges in transitioning the technology from research environments to activities of daily living (ADL). For BMIs to be seamlessly used in ADL, these issues should be handled with minimal external input thus reducing the need for a technician/caregiver to calibrate the system. Reinforcement Learning (RL) based BMIs are a good tool to be used when there is no external training signal and can provide an adaptive modality to train BMI decoders. However, RL based BMIs are sensitive to the feedback provided to adapt the BMI. In actor-critic BMIs, this feedback is provided by the critic and the overall system performance is limited by the critic accuracy. In this work, we developed an adaptive BMI that could handle inaccuracies in the critic feedback in an effort to produce more accurate RL based BMIs. We developed a confidence measure, which indicated how appropriate the feedback is for updating the decoding parameters of the actor. The results show that with the new update formulation, the critic accuracy is no longer a limiting factor for the overall performance. We tested and validated the system onthree different data sets: synthetic data generated by an Izhikevich neural spiking model, synthetic data with a Gaussian noise distribution, and data collected from a non-human primate engaged in a reaching task. All results indicated that the system with the critic confidence built in always outperformed the system without the critic confidence. Results of this study suggest the potential application of the technique in developing an autonomous BMI that does not need an external signal for training or extensive calibration.

Using reinforcement learning to provide stable brain-machine interface control despite neural input reorganization

Brain-machine interface (BMI) systems give users direct neural control of robotic, communication, or functional electrical stimulation systems. As BMI systems begin transitioning from laboratory settings into activities of daily living, an important goal is to develop neural decoding algorithms that can be calibrated with a minimal burden on the user, provide stable control for long periods of time, and can be responsive to fluctuations in the decoder’s neural input space (e.g. neurons appearing or being lost amongst electrode recordings). These are significant challenges for static neural decoding algorithms that assume stationary input/output relationships. Here we use an actor-critic reinforcement learning architecture to provide an adaptive BMI controller that can successfully adapt to dramatic neural reorganizations, can maintain its performance over long time periods, and which does not require the user to produce specific kinetic or kinematic activities to calibrate the BMI. Two marmoset monkeys used the Reinforcement Learning BMI (RLBMI) to successfully control a robotic arm during a two-target reaching task. The RLBMI was initialized using random initial conditions, and it quickly learned to control the robot from brain states using only a binary evaluative feedback regarding whether previously chosen robot actions were good or bad. The RLBMI was able to maintain control over the system throughout sessions spanning multiple weeks. Furthermore, the RLBMI was able to quickly adapt and maintain control of the robot despite dramatic perturbations to the neural inputs, including a series of tests in which the neuron input space was deliberately halved or doubled.