Clustering Based Kernel Reinforcement Learning for Neural Adaptation in Brain-Machine Interfaces

Kernel Reinforcement Learning-Assisted Adaptive Decoder Facilitates Stable and Continuous Brain Control Tasks

Brain-Machine Interfaces (BMIs) assist paralyzed people to brain control (BC) the neuro-prosthesis continuously moving in space. During the BC process, the subject imagines the movement of the real limb and adapts the brain activity according to the sensory feedback. The neural adaptation in the closed-loop control results in complex and changing brain signals. Simultaneously, the decoder interprets the time-varying functional mapping between neural activity and continuous trajectory. It is crucial and challenging to accurately and adaptively track the mapping to help the subject accomplish the BC task with a stable performance. Existing Kalman Filter (KF) based decoders achieve continuous trajectory control by linearly interpreting neural firing observations into self-evolving prosthetic states. However, the linear neural-state mapping might not accurately reflect the movement intention of the subject. In this paper, we propose a novel method that allows subjects to achieve continuous brain control efficiently and stably. The proposed method incorporates a kernel reinforcement learning method into a state-observation model to decode the nonlinearly neural observation into a continuous trajectory state. The state transition function ensures the continuity of the prosthetic state. The kernel reinforcement learning allows the quick adaptation of the nonlinear neural-movement mapping during the BC process. The proposed method is tested in an online brain control reaching task for rats. Compared with KF, our method achieved more successful trials, faster response time, shorter inter-trial time, and remained stable over days. These results demonstrate that the proposed method is an efficient tool to assist subjects in brain control tasks.

Neural Decoders Using Reinforcement Learning in Brain Machine Interfaces: A Technical Review

Creating flexible and robust brain machine interfaces (BMIs) is currently a popular topic of research that has been explored for decades in medicine, engineering, commercial, and machine-learning communities. In particular, the use of techniques using reinforcement learning (RL) has demonstrated impressive results but is under-represented in the BMI community. To shine more light on this promising relationship, this article aims to provide an exhaustive review of RL's applications to BMIs. Our primary focus in this review is to provide a technical summary of various algorithms used in RL-based BMIs to decode neural intention, without emphasizing preprocessing techniques on the neural signals and reward modeling for RL. We first organize the literature based on the type of RL methods used for neural decoding, and then each algorithm's learning strategy is explained along with its application in BMIs. A comparative analysis highlighting the similarities and uniqueness among neural decoders is provided. Finally, we end this review with a discussion about the current stage of RLBMIs including their limitations and promising directions for future research.

Covariant Cluster Transfer for Kernel Reinforcement Learning in Brain-Machine Interface

Brain-Machine Interface (BMI) provides a promising way to help disabled people restore their motor functions. The patients are able to control the external devices directly from their neural signals by the decoder. Due to various reasons such as mental fatigue and distraction, the distribution of the neural signals might change, which might lead to poor performance for the decoder. In this case, we need to calibrate the parameters before each session, which needs the professionals to label the data and is not convenient for the patient's usage at home. In this paper, we propose a covariant cluster transfer mechanism for the kernel reinforcement learning (RL) algorithm to speed up the adaptation across sessions. The parameters of the decoder will adaptively change according to a reward signal, which could be easily set by the patient. More importantly, we cluster the neural patterns in previous sessions. The cluster represents the conditional distribution from neural patterns to actions. When a distinct neural pattern appears in the new session, the nearest cluster will be transferred. In this way, the knowledge from the old session could be utilized to accelerate the learning in the new session. Our proposed algorithm is tested on the simulated neural data where the neural signal's distribution differs across sessions. Compared with the training from random initialization and a weight transfer policy, our proposed cluster transfer mechanism maintains a significantly higher success rate and a faster adaptation when the conditional distribution from neural signals to actions remains similar.

A K-Medoids based Point-Process Modeling on Neural Spike Transformation using Binless Kernel

A neural prosthesis is designed to compensate for cognitive functional losses by modeling the information transmission among cortical areas. Existing methods generally build a generalized linear model to approximate the nonlinear transformation among two areas, and use the temporal information of the neural spike with low efficiency. It is essential to efficiently model the nonlinearity embedded in spike generation and transmission for the real-time. This paper proposes a nonlinear point-process model to describe spike-in and spike-out transformation using the theory of reproducing kernel Hilbert space (RKHS) and the binless kernel on spike trains. The binless kernel efficiently maps exact spike timing information to the RKHS to describe nonlinear transformations with global minimum regardless of the weight initialization. A streaming K-medoids algorithm is introduced to select typical and important features in this nonlinear binless kernel for further modeling. We test our model on the nonlinearly generated synthetic neural spike trains, and compare with the existing spike transformation methods, such as Volterra model and staged point-process model. The results show that our model has higher goodness-of-fit evaluated by Kolmogorov-Smirnov test and less variance on the prediction results, which indicates the potential better modeling approach for neural prosthesis application.

A Weight Transfer Mechanism for Kernel Reinforcement Learning Decoding in Brain-Machine Interfaces

Brain-Machine Interfaces (BMIs) aim to help disabled people brain control the external devices to finish a variety of movement tasks. The neural signals are decoded into the execution commands of the apparatus. However, most of the existing decoding algorithms in BMI are only trained for a single task. When facing a new task, even if it is similar to the previous one, the decoder needs to be re-trained from scratch, which is not efficient. Among the different types of decoders, reinforcement learning (RL) based algorithm has the advantage of adaptive training through trial-and-error over the recalibration used in supervised learning. But most of the RL algorithms in BMI do not actively leverage the acquired knowledge in the old task. In this paper, we propose a kernel RL algorithm with a weight transfer mechanism for new task learning. The existing neural patterns are clustered according to their similarities. A new pattern will be assigned with the weights that are transferred from the closest cluster. In this way, the most similar experiences from the previous task could be re-utilized in the new task to fasten the learning speed. The proposed algorithm is tested on synthetic neural data. Compared with the policy of re-training from scratch, the proposed weight transfer mechanism could maintain a significantly higher performance and achieve a faster learning speed on the new task.