Reinforcement Learning based Decoding Using Internal Reward for Time Delayed Task in Brain Machine Interfaces

Kernel Temporal Difference based Reinforcement Learning for Brain Machine Interfaces. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021;2021:6721-6724. [PMID: 34892650 DOI: 10.1109/embc46164.2021.9631086] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Brain-machine interfaces (BMIs) enable people with disabilities to control external devices with their motor intentions through a decoder. Compared with supervised learning, reinforcement learning (RL) is more promising for the disabled because it can assist them to learn without actual limb movement. Current RL decoders deal with tasks with immediate reward delivery. But for tasks where the reward is only given by the end of the trial, existing RL methods may take a long time to train and are prone to becoming trapped in the local minima. In this paper, we propose to embed temporal difference method (TD) into Quantized Attention-Gated Kernel Reinforcement Learning (QAGKRL) to solve this temporal credit assignment problem. This algorithm utilizes a kernel network to ensure the global linear structure and adopts a softmax policy to efficiently explore the state-action mapping through TD error. We simulate a center-out task where the agent needs several steps to first reach a periphery target and then return to the center to get the external reward. Our proposed algorithm is tested on simulated data and compared with two state-of-the-art models. We find that introducing the TD method to QAGKRL achieves a prediction accuracy of 96.2% ± 0.77% (mean ± std), which is significantly better the other two methods.Clinical Relevance-This paper proposes a novel kernel temporal difference RL method for the multi-step task with delayed reward delivery, which potentially enables BMI online continuous decoding.

Multivariate Encoding Analysis of Medial Prefrontal Cortex Cortical Activity during Task Learning

Studies have shown that medial prefrontal cortex (mPFC) is responsible for outcome evaluation. Some recent studies also suggest that mPFC may play an important role in goal planning and action execution when performing a task. If the information encoded in mPFC can be accurately extracted and identified, it can improve the design of brain-machine interfaces by better reconstructing subjects' motion intention guided by reward information. In this paper, we investigate whether mPFC neural signals simultaneously encode information of goal planning, action execution and outcome evaluation. Linear-nonlinear-Poisson (LNP) model is applied for encoding analysis on mPFC neural spike data when a rat is learning a two-lever-press discrimination task. We use the L²-norm of tuning parameter in LNP model to indicate the importance of the encoded information and compare the spike train prediction performance of LNP model using all information, the most significant information and reward information only. The preliminary results indicate that mPFC activity can encode simultaneously the information of goal planning, action execution and outcome evaluation and that all the relevant information could be reconstructed from mPFC spike trains on a single trial basis.