Task-Independent Cognitive State Transition Detection From Cortical Neurons During 3-D Reach-to-Grasp Movements

Decoding spatial locations from primate lateral prefrontal cortex neural activity during virtual navigation

Objective. Decoding the intended trajectories from brain signals using a brain-computer interface system could be used to improve the mobility of patients with disabilities.Approach. Neuronal activity associated with spatial locations was examined while macaques performed a navigation task within a virtual environment.Main results.Here, we provide proof of principle that multi-unit spiking activity recorded from the lateral prefrontal cortex (LPFC) of non-human primates can be used to predict the location of a subject in a virtual maze during a navigation task. The spatial positions within the maze that require a choice or are associated with relevant task events can be better predicted than the locations where no relevant events occur. Importantly, within a task epoch of a single trial, multiple locations along the maze can be independently identified using a support vector machine model.Significance. Considering that the LPFC of macaques and humans share similar properties, our results suggest that this area could be a valuable implant location for an intracortical brain-computer interface system used for spatial navigation in patients with disabilities.

Decoding Saccade Intention From Primate Prefrontal Cortical Local Field Potentials Using Spectral, Spatial, and Temporal Dimensionality Reduction

Most invasive Brain Computer Interfaces (iBCIs) use spike and Local Field Potentials (LFPs) from the motor or parietal cortices to decode movement intentions. It has been debated whether harvesting signals from other brain areas that encode global cognitive variables, such as the allocation of attention and eye movement goals in a variety of spatial reference frames, may improve the outcome of iBCIs. Here, we explore the ability of LFP signals, sampled from the lateral prefrontal cortex (LPFC) of macaque monkeys, to encode eye-movement intention during the pre-movement fixation period of a delayed saccade task. We use spectral dimensionality reduction to examine the spatiotemporal properties of the extracted non-rhythmic broadband activity and explore its usefulness in decoding saccade goals. The dynamics of the broadband signal in low spatial dimensions across the pre-movement fixation period uncovered saccade target separation; its discriminative potential was confirmed using support vector machine classifications. These findings reveal that broadband LFP from the LPFC can be used to decode intended saccade target location during pre-movement periods. We further provide a general workflow that can be implemented in iBCIs and it is relatively robust to the loss of spikes in individual electrodes.

A point-process matched filter for event detection and decoding from population spike trains

OBJECTIVE

Information encoding in neurons can be described through their response fields. The spatial response field of a neuron is the region of space in which a sensory stimulus or a behavioral event causes that neuron to fire. Neurons can also exhibit temporal response fields (TRFs), which characterize a transient response to stimulus or behavioral event onsets. These neurons can thus be described by a spatio-temporal response field (STRF). The activity of neurons with STRFs can be well-described with point process models that characterize binary spike trains with an instantaneous firing rate that is a function of both time and space. However, developing decoders for point process models of neurons that exhibit TRFs is challenging because it requires prior knowledge of event onset times, which are unknown. Indeed, point process filters (PPF) to date have largely focused on decoding neuronal activity without considering TRFs. Also, neural classifiers have required data to be behavior- or stimulus-aligned, i.e. event times to be known, which is often not possible in real-world applications. Our objective in this work is to develop a viable decoder for neurons with STRFs when event times are unknown.

APPROACH

To enable decoding of neurons with STRFs, we develop a novel point-process matched filter (PPMF) that can detect events and estimate their onset times from population spike trains. We also devise a PPF for neurons with transient responses as characterized by STRFs. When neurons exhibit STRFs and event times are unknown, the PPMF can be combined with the PPF or with discrete classifiers for continuous and discrete brain state decoding, respectively.

MAIN RESULTS

We validate our algorithm on two datasets: simulated spikes from neurons that encode visual saliency in response to stimuli, and prefrontal spikes recorded in a monkey performing a delayed-saccade task. We show that the PPMF can estimate the stimulus times and saccade times accurately. Further, the PPMF combined with the PPF can decode visual saliency maps without knowing the stimulus times. Similarly, the PPMF combined with a point process classifier can decode the saccade direction without knowing the saccade times.

SIGNIFICANCE

These event detection and decoding algorithms can help develop neurotechnologies to decode cognitive states from neural responses that exhibit STRFs.

Mirror Neuron Populations Represent Sequences of Behavioral Epochs During Both Execution and Observation

Mirror neurons (MNs) have the distinguishing characteristic of modulating during both execution and observation of an action. Although most studies of MNs have focused on various features of the observed movement, MNs also may monitor the behavioral circumstances in which the movement is embedded, including time periods preceding and following the observed movement. Here, we recorded multiple MNs simultaneously from implanted electrode arrays as two male monkeys executed and observed a reach, grasp, and manipulate task involving different target objects. MNs were recorded from premotor cortex (PM-MNs) and primary motor cortex (M1-MNs). During execution trials, hidden Markov models (HMMs) applied to the activity of either PM-MN or M1-MN populations most often detected sequences of four hidden states, which we named according to the behavioral epoch during which each state began: initial, reaction, movement, and final. The hidden states of MN populations thus reflected not only the movement, but also three behavioral epochs during which no movement occurred. HMMs trained on execution trials could decode similar sequences of hidden states in observation trials, with complete hidden state sequences decoded more frequently from PM-MN populations than from M1-MN populations. Moreover, population trajectories projected in a 2D plane defined by execution trials were preserved in observation trials more for PM-MN than for M1-MN populations. These results suggest that MN populations represent entire behavioral sequences, including both movement and non-movement. PM-MN populations showed greater similarity than M1-MN populations in their representation of behavioral sequences during execution versus observation.SIGNIFICANCE STATEMENT Mirror neurons (MNs) are thought to provide a neural mechanism for understanding the actions of others. However, for an action to be understood, both the movement per se and the non-movement context before and after the movement need to be represented. We found that simultaneously recorded MN populations encoded sequential hidden neural states corresponding approximately to sequential behavioral epochs of a reach, grasp, and manipulate task. During observation trials, hidden state sequences were similar to those identified in execution trials. Hidden state similarity was stronger for MN populations in premotor cortex than for those in primary motor cortex. Execution/observation similarity of hidden state sequences may contribute to understanding the actions of others without actually performing the action oneself.

Modeling task-specific neuronal ensembles improves decoding of grasp

OBJECTIVE

Dexterous movement involves the activation and coordination of networks of neuronal populations across multiple cortical regions. Attempts to model firing of individual neurons commonly treat the firing rate as directly modulating with motor behavior. However, motor behavior may additionally be associated with modulations in the activity and functional connectivity of neurons in a broader ensemble. Accounting for variations in neural ensemble connectivity may provide additional information about the behavior being performed.

APPROACH

In this study, we examined neural ensemble activity in primary motor cortex (M1) and premotor cortex (PM) of two male rhesus monkeys during performance of a center-out reach, grasp and manipulate task. We constructed point process encoding models of neuronal firing that incorporated task-specific variations in the baseline firing rate as well as variations in functional connectivity with the neural ensemble. Models were evaluated both in terms of their encoding capabilities and their ability to properly classify the grasp being performed.

MAIN RESULTS

Task-specific ensemble models correctly predicted the performed grasp with over 95% accuracy and were shown to outperform models of neuronal activity that assume only a variable baseline firing rate. Task-specific ensemble models exhibited superior decoding performance in 82% of units in both monkeys (p  <  0.01). Inclusion of ensemble activity also broadly improved the ability of models to describe observed spiking. Encoding performance of task-specific ensemble models, measured by spike timing predictability, improved upon baseline models in 62% of units.

SIGNIFICANCE

These results suggest that additional discriminative information about motor behavior found in the variations in functional connectivity of neuronal ensembles located in motor-related cortical regions is relevant to decode complex tasks such as grasping objects, and may serve the basis for more reliable and accurate neural prosthesis.

Decoding Lower Limb Muscle Activity and Kinematics from Cortical Neural Spike Trains during Monkey Performing Stand and Squat Movements

Extensive literatures have shown approaches for decoding upper limb kinematics or muscle activity using multichannel cortical spike recordings toward brain machine interface (BMI) applications. However, similar topics regarding lower limb remain relatively scarce. We previously reported a system for training monkeys to perform visually guided stand and squat tasks. The current study, as a follow-up extension, investigates whether lower limb kinematics and muscle activity characterized by electromyography (EMG) signals during monkey performing stand/squat movements can be accurately decoded from neural spike trains in primary motor cortex (M1). Two monkeys were used in this study. Subdermal intramuscular EMG electrodes were implanted to 8 right leg/thigh muscles. With ample data collected from neurons from a large brain area, we performed a spike triggered average (SpTA) analysis and got a series of density contours which revealed the spatial distributions of different muscle-innervating neurons corresponding to each given muscle. Based on the guidance of these results, we identified the locations optimal for chronic electrode implantation and subsequently carried on chronic neural data recordings. A recursive Bayesian estimation framework was proposed for decoding EMG signals together with kinematics from M1 spike trains. Two specific algorithms were implemented: a standard Kalman filter and an unscented Kalman filter. For the latter one, an artificial neural network was incorporated to deal with the nonlinearity in neural tuning. High correlation coefficient and signal to noise ratio between the predicted and the actual data were achieved for both EMG signals and kinematics on both monkeys. Higher decoding accuracy and faster convergence rate could be achieved with the unscented Kalman filter. These results demonstrate that lower limb EMG signals and kinematics during monkey stand/squat can be accurately decoded from a group of M1 neurons with the proposed algorithms. Our findings provide new insights for extending current BMI design concepts and techniques on upper limbs to lower limb circumstances. Brain controlled exoskeleton, prostheses or neuromuscular electrical stimulators for lower limbs are expected to be developed, which enables the subject to manipulate complex biomechatronic devices with mind in more harmonized manner.

Comparing offline decoding performance in physiologically defined neuronal classes

Objective

Recently, several studies have documented the presence of a bimodal distribution of spike waveform widths in primary motor cortex. Although narrow and wide spiking neurons, corresponding to the two modes of the distribution, exhibit different response properties, it remains unknown if these differences give rise to differential decoding performance between these two classes of cells.

Approach

We used a Gaussian mixture model to classify neurons into narrow and wide physiological classes. Using similar-size, random samples of neurons from these two physiological classes, we trained offline decoding models to predict a variety of movement features. We compared offline decoding performance between these two physiologically defined populations of cells.

Main results

We found that narrow spiking neural ensembles decode motor parameters better than wide spiking neural ensembles including kinematics, kinetics, and muscle activity.

Significance

These findings suggest that the utility of neural ensembles in brain machine interfaces may be predicted from their spike waveform widths.

High Precision Neural Decoding of Complex Movement Trajectories using Recursive Bayesian Estimation with Dynamic Movement Primitives

Brain-machine interfaces (BMIs) are a rapidly progressing technology with the potential to restore function to victims of severe paralysis via neural control of robotic systems. Great strides have been made in directly mapping a user's cortical activity to control of the individual degrees of freedom of robotic end-effectors. While BMIs have yet to achieve the level of reliability desired for widespread clinical use, environmental sensors (e.g. RGB-D cameras for object detection) and prior knowledge of common movement trajectories hold great potential for improving system performance. Here we present a novel sensor fusion paradigm for BMIs that capitalizes on information able to be extracted from the environment to greatly improve the performance of control. This was accomplished by using dynamic movement primitives to model the 3D endpoint trajectories of manipulating various objects. We then used a switching unscented Kalman filter to continuously arbitrate between the 3D endpoint kinematics predicted by the dynamic movement primitives and control derived from neural signals. We experimentally validated our system by decoding 3D endpoint trajectories executed by a non-human primate manipulating four different objects at various locations. Performance using our system showed a dramatic improvement over using neural signals alone, with median distance between actual and decoded trajectories decreasing from 31.1 cm to 9.9 cm, and mean correlation increasing from 0.80 to 0.98. Our results indicate that our sensor fusion framework can dramatically increase the fidelity of neural prosthetic trajectory decoding.

PMv Neuronal Firing May Be Driven by a Movement Command Trajectory within Multidimensional Gaussian Fields

The premotor cortex (PM) is known to be a site of visuo-somatosensory integration for the production of movement. We sought to better understand the ventral PM (PMv) by modeling its signal encoding in greater detail. Neuronal firing data was obtained from 110 PMv neurons in two male rhesus macaques executing four reach-grasp-manipulate tasks. We found that in the large majority of neurons (∼90%) the firing patterns across the four tasks could be explained by assuming that a high-dimensional position/configuration trajectory-like signal evolving ∼250 ms before movement was encoded within a multidimensional Gaussian field (MGF). Our findings are consistent with the possibility that PMv neurons process a visually specified reference command for the intended arm/hand position trajectory with respect to a proprioceptively or visually sensed initial configuration. The estimated MGF were (hyper) disc-like, such that each neuron's firing modulated strongly only with commands that evolved along a single direction within position/configuration space. Thus, many neurons appeared to be tuned to slices of this input signal space that as a collection appeared to well cover the space. The MGF encoding models appear to be consistent with the arm-referent, bell-shaped, visual target tuning curves and target selectivity patterns observed in PMV visual-motor neurons. These findings suggest that PMv may implement a lookup table-like mechanism that helps translate intended movement trajectory into time-varying patterns of activation in motor cortex and spinal cord. MGFs provide an improved nonlinear framework for potentially decoding visually specified, intended multijoint arm/hand trajectories well in advance of movement.

Advancing brain-machine interfaces: moving beyond linear state space models

Advances in recent years have dramatically improved output control by Brain-Machine Interfaces (BMIs). Such devices nevertheless remain robotic and limited in their movements compared to normal human motor performance. Most current BMIs rely on transforming recorded neural activity to a linear state space composed of a set number of fixed degrees of freedom. Here we consider a variety of ways in which BMI design might be advanced further by applying non-linear dynamics observed in normal motor behavior. We consider (i) the dynamic range and precision of natural movements, (ii) differences between cortical activity and actual body movement, (iii) kinematic and muscular synergies, and (iv) the implications of large neuronal populations. We advance the hypothesis that a given population of recorded neurons may transmit more useful information than can be captured by a single, linear model across all movement phases and contexts. We argue that incorporating these various non-linear characteristics will be an important next step in advancing BMIs to more closely match natural motor performance.