Decoding individuated finger movements using volume-constrained neuronal ensembles in the M1 hand area

Chord skill: learning optimized hand postures and bimanual coordination

This reaction time study tested the hypothesis that in the case of finger movements skilled motor control involves the execution of learned hand postures. After delineating hypothetical control mechanisms and their predictions an experiment is described involving 32 participants who practiced 6 chord responses. These responses involved the simultaneous depression of one, two or three keys with either four right-hand fingers or two fingers of both hands. After practicing each of these responses for 240 trials, the participants performed the practiced and also novel chords with the familiar and with the unfamiliar hand configuration of the other practice group. The results suggest that participants learned hand postures rather than spatial or explicit chord representations. Participants practicing with both hands also developed a bimanual coordination skill. Chord execution was most likely slowed by interference between adjacent fingers. This interference seemed eliminated with practice for some chords but not for others. Hence, the results support the notion that skilled control of finger movements is based on learned hand postures that even after practice may be slowed by interference between adjacent fingers.

Decoding Native Cortical Representations for Flexion and Extension at Upper Limb Joints Using Electrocorticography

Brain-machine interface (BMI) researchers have traditionally focused on modeling endpoint reaching tasks to provide the control of neurally driven prosthetic arms. Most previous research has focused on achieving an endpoint control through a Cartesian-coordinate-centered approach. However, a joint-centered approach could potentially be used to intuitively control a wide range of limb movements. We systematically investigated the feasibility of discriminating between flexion and extension of different upper limb joints using electrocorticography(ECoG) recordings from sensorimotor cortex. Four subjects implanted with macro-ECoG (10-mm spacing), high-density ECoG (5-mm spacing), and/or micro-ECoG arrays (0.9-mm spacing and 4 mm × 4 mm coverage), performed randomly cued flexions or extensions of the fingers, wrist, or elbow contralateral to the implanted hemisphere. We trained a linear model to classify six movements using averaged high-gamma power (70-110 Hz) modulations at different latencies with respect to movement onset, and within a time interval restricted to flexion or extension at each joint. Offline decoding models for each subject classified these movements with accuracies of 62%-83%. Our results suggest that the widespread ECoG coverage of sensorimotor cortex could allow a whole limb BMI to sample native cortical representations in order to control flexion and extension at multiple joints.

Single-Finger Neural Basis Information-Based Neural Decoder for Multi-Finger Movements

In this paper, we investigate the relationship between single and multi-finger movements. By exploiting the neural correlation between the temporal firing patterns between movements, we show that the Pearson's correlation coefficient for the physically related movement pairs are greater than those of others; the firing rates of the neurons that are tuned to a single-finger movements also increases when the corresponding multi-finger movements are instructed. We also use a hierarchical cluster analysis to verify not only the relationship between the single and multi-finger movements, but also the relationship between the flexion and extension movements. Furthermore, we propose a novel decoding method of modeling neural firing patterns while omitting the training process of the multi-finger movements. For the decoding, the Skellam and Gaussian probability distributions are used as mathematical models. The probabilistic distribution model of the multi-finger movements was estimated using the neural activity that was acquired during single-finger movements. As a result, the proposed neural decoding accuracy comparable with that of the supervised neural decoding accuracy when all of the neurons were used for the multi-finger movements. These results suggest that only the neural activities of single-finger movements can be exploited for the control of dexterous multi-finger neuroprosthetics.

A 128-Channel Extreme Learning Machine-Based Neural Decoder for Brain Machine Interfaces

Currently, state-of-the-art motor intention decoding algorithms in brain-machine interfaces are mostly implemented on a PC and consume significant amount of power. A machine learning coprocessor in 0.35- μm CMOS for the motor intention decoding in the brain-machine interfaces is presented in this paper. Using Extreme Learning Machine algorithm and low-power analog processing, it achieves an energy efficiency of 3.45 pJ/MAC at a classification rate of 50 Hz. The learning in second stage and corresponding digitally stored coefficients are used to increase robustness of the core analog processor. The chip is verified with neural data recorded in monkey finger movements experiment, achieving a decoding accuracy of 99.3% for movement type. The same coprocessor is also used to decode time of movement from asynchronous neural spikes. With time-delayed feature dimension enhancement, the classification accuracy can be increased by 5% with limited number of input channels. Further, a sparsity promoting training scheme enables reduction of number of programmable weights by ≈ 2X.

Individual finger control of a modular prosthetic limb using high-density electrocorticography in a human subject

OBJECTIVE

We used native sensorimotor representations of fingers in a brain-machine interface (BMI) to achieve immediate online control of individual prosthetic fingers.

APPROACH

Using high gamma responses recorded with a high-density electrocorticography (ECoG) array, we rapidly mapped the functional anatomy of cued finger movements. We used these cortical maps to select ECoG electrodes for a hierarchical linear discriminant analysis classification scheme to predict: (1) if any finger was moving, and, if so, (2) which digit was moving. To account for sensory feedback, we also mapped the spatiotemporal activation elicited by vibrotactile stimulation. Finally, we used this prediction framework to provide immediate online control over individual fingers of the Johns Hopkins University Applied Physics Laboratory modular prosthetic limb.

MAIN RESULTS

The balanced classification accuracy for detection of movements during the online control session was 92% (chance: 50%). At the onset of movement, finger classification was 76% (chance: 20%), and 88% (chance: 25%) if the pinky and ring finger movements were coupled. Balanced accuracy of fully flexing the cued finger was 64%, and 77% had we combined pinky and ring commands. Offline decoding yielded a peak finger decoding accuracy of 96.5% (chance: 20%) when using an optimized selection of electrodes. Offline analysis demonstrated significant finger-specific activations throughout sensorimotor cortex. Activations either prior to movement onset or during sensory feedback led to discriminable finger control.

SIGNIFICANCE

Our results demonstrate the ability of ECoG-based BMIs to leverage the native functional anatomy of sensorimotor cortical populations to immediately control individual finger movements in real time.

Adaptation to elastic loads and BMI robot controls during rat locomotion examined with point-process GLMs

Currently little is known about how a mechanically coupled BMI system's actions are integrated into ongoing body dynamics. We tested a locomotor task augmented with a BMI system driving a robot mechanically interacting with a rat under three conditions: control locomotion (BL), “simple elastic load” (E) and “BMI with elastic load” (BMI/E). The effect of the BMI was to allow compensation of the elastic load as a function of the neural drive. Neurons recorded here were close to one another in cortex, all within a 200 micron diameter horizontal distance of one another. The interactions of these close assemblies of neurons may differ from those among neurons at longer distances in BMI tasks and thus are important to explore. A point process generalized linear model (GLM), was used to examine connectivity at two different binning timescales (1 ms vs. 10 ms). We used GLM models to fit non-Poisson neural dynamics solely using other neurons' prior neural activity as covariates. Models at different timescales were compared based on Kolmogorov-Smirnov (KS) goodness-of-fit and parsimony. About 15% of cells with non-Poisson firing were well fitted with the neuron-to-neuron models alone. More such cells were fitted at the 1 ms binning than 10 ms. Positive connection parameters (“excitation” ~70%) exceeded negative parameters (“inhibition” ~30%). Significant connectivity changes in the GLM determined networks of well-fitted neurons occurred between the conditions. However, a common core of connections comprising at least ~15% of connections persisted between any two of the three conditions. Significantly almost twice as many connections were in common between the two load conditions (~27%), compared to between either load condition and the baseline. This local point process GLM identified neural correlation structure and the changes seen across task conditions in the rats in this neural subset may be intrinsic to cortex or due to feedback and input reorganization in adaptation.

Task-Independent Cognitive State Transition Detection From Cortical Neurons During 3-D Reach-to-Grasp Movements

Complex reach, grasp, and object manipulation tasks require sequential, temporal coordination of movements by neurons in the brain. Detecting cognitive state transitions associated with motor tasks from sequential neural data is pivotal in rehabilitation engineering. The cognitive state detectors proposed thus far rely on task-dependent (TD) models, i.e., the detection strategy exploits a priori knowledge of the movement tasks to determine the actual cognitive states, regardless of whether these cognitive states actually depend on the movement tasks or not. This approach, however, is not viable when the tasks are not known a priori (e.g., the subject performs many different tasks) or there is paucity of neural data for each task. Moreover, some cognitive states (e.g., holding) may be invariant to the movement tasks performed. Here we propose a real-time (online) task-independent (TI) framework to detect cognitive state transitions from spike trains and kinematic measurements. We constructed this detection framework using 452 single-unit neural spike recordings collected via multielectrode arrays in the premotor dorsal and ventral (PMd and PMv) cortical regions of two nonhuman primates performing 3-D multiobject reach-to-grasp tasks. We used the detection latency and accuracy of state transitions to measure the performance. We find that, in both online and offline detection modes: 1) TI models have significantly better performance than corresponding TD models when using neuronal data alone and 2) during movements, the addition of the kinematics history to the TI models further improves detection performance. These findings suggest that TI models may accurately detect cognitive state transitions. Our framework could pave the way for a TI control of neural prosthesis from cortical neurons.

Distinct neural patterns enable grasp types decoding in monkey dorsal premotor cortex

OBJECTIVE

Recent studies have shown that dorsal premotor cortex (PMd), a cortical area in the dorsomedial grasp pathway, is involved in grasp movements. However, the neural ensemble firing property of PMd during grasp movements and the extent to which it can be used for grasp decoding are still unclear.

APPROACH

To address these issues, we used multielectrode arrays to record both spike and local field potential (LFP) signals in PMd in macaque monkeys performing reaching and grasping of one of four differently shaped objects.

MAIN RESULTS

Single and population neuronal activity showed distinct patterns during execution of different grip types. Cluster analysis of neural ensemble signals indicated that the grasp related patterns emerged soon (200-300 ms) after the go cue signal, and faded away during the hold period. The timing and duration of the patterns varied depending on the behaviors of individual monkey. Application of support vector machine model to stable activity patterns revealed classification accuracies of 94% and 89% for each of the two monkeys, indicating a robust, decodable grasp pattern encoded in the PMd. Grasp decoding using LFPs, especially the high-frequency bands, also produced high decoding accuracies.

SIGNIFICANCE

This study is the first to specify the neuronal population encoding of grasp during the time course of grasp. We demonstrate high grasp decoding performance in PMd. These findings, combined with previous evidence for reach related modulation studies, suggest that PMd may play an important role in generation and maintenance of grasp action and may be a suitable locus for brain-machine interface applications.

Liquid state machine with dendritically enhanced readout for low-power, neuromorphic VLSI implementations

In this paper, we describe a new neuro-inspired, hardware-friendly readout stage for the liquid state machine (LSM), a popular model for reservoir computing. Compared to the parallel perceptron architecture trained by the p-delta algorithm, which is the state of the art in terms of performance of readout stages, our readout architecture and learning algorithm can attain better performance with significantly less synaptic resources making it attractive for VLSI implementation. Inspired by the nonlinear properties of dendrites in biological neurons, our readout stage incorporates neurons having multiple dendrites with a lumped nonlinearity (two compartment model). The number of synaptic connections on each branch is significantly lower than the total number of connections from the liquid neurons and the learning algorithm tries to find the best 'combination' of input connections on each branch to reduce the error. Hence, the learning involves network rewiring (NRW) of the readout network similar to structural plasticity observed in its biological counterparts. We show that compared to a single perceptron using analog weights, this architecture for the readout can attain, even by using the same number of binary valued synapses, up to 3.3 times less error for a two-class spike train classification problem and 2.4 times less error for an input rate approximation task. Even with 60 times larger synapses, a group of 60 parallel perceptrons cannot attain the performance of the proposed dendritically enhanced readout. An additional advantage of this method for hardware implementations is that the 'choice' of connectivity can be easily implemented exploiting address event representation (AER) protocols commonly used in current neuromorphic systems where the connection matrix is stored in memory. Also, due to the use of binary synapses, our proposed method is more robust against statistical variations.

Motor cortical correlates of arm resting in the context of a reaching task and implications for prosthetic control

Prosthetic devices are being developed to restore movement for motor-impaired individuals. A robotic arm can be controlled based on models that relate motor-cortical ensemble activity to kinematic parameters. The models are typically built and validated on data from structured trial periods during which a subject actively performs specific movements, but real-world prosthetic devices will need to operate correctly during rest periods as well. To develop a model of motor cortical modulation during rest, we trained monkeys (Macaca mulatta) to perform a reaching task with their own arm while recording motor-cortical single-unit activity. When a monkey spontaneously put its arm down to rest between trials, our traditional movement decoder produced a nonzero velocity prediction, which would cause undesired motion when applied to a prosthetic arm. During these rest periods, a marked shift was found in individual units' tuning functions. The activity pattern of the whole population during rest (Idle state) was highly distinct from that during reaching movements (Active state), allowing us to predict arm resting from instantaneous firing rates with 98% accuracy using a simple classifier. By cascading this state classifier and the movement decoder, we were able to predict zero velocity correctly, which would avoid undesired motion in a prosthetic application. Interestingly, firing rates during hold periods followed the Active pattern even though hold kinematics were similar to those during rest with near-zero velocity. These findings expand our concept of motor-cortical function by showing that population activity reflects behavioral context in addition to the direct parameters of the movement itself.

Decoding repetitive finger movements with brain activity acquired via non-invasive electroencephalography

We investigated how well repetitive finger tapping movements can be decoded from scalp electroencephalography (EEG) signals. A linear decoder with memory was used to infer continuous index finger angular velocities from the low-pass filtered fluctuations of the amplitude of a plurality of EEG signals distributed across the scalp. To evaluate the accuracy of the decoder, the Pearson's correlation coefficient (r) between the observed and predicted trajectories was calculated in a 10-fold cross-validation scheme. We also assessed attempts to decode finger kinematics from EEG data that was cleaned with independent component analysis (ICA), EEG data from peripheral sensors, and EEG data from rest periods. A genetic algorithm (GA) was used to select combinations of EEG channels that maximized decoding accuracies. Our results (lower quartile r = 0.18, median r = 0.36, upper quartile r = 0.50) show that delta-band EEG signals contain useful information that can be used to infer finger kinematics. Further, the highest decoding accuracies were characterized by highly correlated delta band EEG activity mostly localized to the contralateral central areas of the scalp. Spectral analysis of EEG also showed bilateral alpha band (8–13 Hz) event related desynchronizations (ERDs) and contralateral beta band (20–30 Hz) event related synchronizations (ERSs) localized over central scalp areas. Overall, this study demonstrates the feasibility of decoding finger kinematics from scalp EEG signals.

Sensors and decoding for intracortical brain computer interfaces

Intracortical brain computer interfaces (iBCIs) are being developed to enable people to drive an output device, such as a computer cursor, directly from their neural activity. One goal of the technology is to help people with severe paralysis or limb loss. Key elements of an iBCI are the implanted sensor that records the neural signals and the software that decodes the user's intended movement from those signals. Here, we focus on recent advances in these two areas, placing special attention on contributions that are or may soon be adopted by the iBCI research community. We discuss how these innovations increase the technology's capability, accuracy, and longevity, all important steps that are expanding the range of possible future clinical applications.

Common muscle synergies for balance and walking

Little is known about the integration of neural mechanisms for balance and locomotion. Muscle synergies have been studied independently in standing balance and walking, but not compared. Here, we hypothesized that reactive balance and walking are mediated by a common set of lower-limb muscle synergies. In humans, we examined muscle activity during multidirectional support-surface perturbations during standing and walking, as well as unperturbed walking at two speeds. We show that most muscle synergies used in perturbations responses during standing were also used in perturbation responses during walking, suggesting common neural mechanisms for reactive balance across different contexts. We also show that most muscle synergies using in reactive balance were also used during unperturbed walking, suggesting that neural circuits mediating locomotion and reactive balance recruit a common set of muscle synergies to achieve task-level goals. Differences in muscle synergies across conditions reflected differences in the biomechanical demands of the tasks. For example, muscle synergies specific to walking perturbations may reflect biomechanical challenges associated with single limb stance, and muscle synergies used during sagittal balance recovery in standing but not walking were consistent with maintaining the different desired center of mass motions in standing vs. walking. Thus, muscle synergies specifying spatial organization of muscle activation patterns may define a repertoire of biomechanical subtasks available to different neural circuits governing walking and reactive balance and may be recruited based on task-level goals. Muscle synergy analysis may aid in dissociating deficits in spatial vs. temporal organization of muscle activity in motor deficits. Muscle synergy analysis may also provide a more generalizable assessment of motor function by identifying whether common modular mechanisms are impaired across the performance of multiple motor tasks.

State-based decoding of hand and finger kinematics using neuronal ensemble and LFP activity during dexterous reach-to-grasp movements

The performance of brain-machine interfaces (BMIs) that continuously control upper limb neuroprostheses may benefit from distinguishing periods of posture and movement so as to prevent inappropriate movement of the prosthesis. Few studies, however, have investigated how decoding behavioral states and detecting the transitions between posture and movement could be used autonomously to trigger a kinematic decoder. We recorded simultaneous neuronal ensemble and local field potential (LFP) activity from microelectrode arrays in primary motor cortex (M1) and dorsal (PMd) and ventral (PMv) premotor areas of two male rhesus monkeys performing a center-out reach-and-grasp task, while upper limb kinematics were tracked with a motion capture system with markers on the dorsal aspect of the forearm, hand, and fingers. A state decoder was trained to distinguish four behavioral states (baseline, reaction, movement, hold), while a kinematic decoder was trained to continuously decode hand end point position and 18 joint angles of the wrist and fingers. LFP amplitude most accurately predicted transition into the reaction (62%) and movement (73%) states, while spikes most accurately decoded arm, hand, and finger kinematics during movement. Using an LFP-based state decoder to trigger a spike-based kinematic decoder [r = 0.72, root mean squared error (RMSE) = 0.15] significantly improved decoding of reach-to-grasp movements from baseline to final hold, compared with either a spike-based state decoder combined with a spike-based kinematic decoder (r = 0.70, RMSE = 0.17) or a spike-based kinematic decoder alone (r = 0.67, RMSE = 0.17). Combining LFP-based state decoding with spike-based kinematic decoding may be a valuable step toward the realization of BMI control of a multifingered neuroprosthesis performing dexterous manipulation.

Corticomuscular coherence modulation with the pattern of finger force coordination

We assess the corticomuscular coherence (CMC) of the contralateral primary motor cortex and the hand muscles during a finger force-tracking task and explore whether the pattern of finger coordination has an impact on the CMC level. Six healthy subjects (three men and three women) were recruited to conduct the force-tracking tasks comprising two finger patterns, i.e., natural combination of index and middle fingers and unnatural combination of index and middle fingers (i.e., simultaneously producing equal force strength in index and middle finger). During the conducting of the tasks with right index and middle finger, MEG and sEMG signals were recorded from left primary motor cortex (M1) and right flexor digitorum superficialis (FDS), respectively; the contralateral CMC was calculated to assess the neuromuscular interaction. Finger force-tracking tasks of Common-IM only induce beta-band CMC, whereas Uncommon-IM tasks produce CMC in both beta and low-gamma band. Compared to the force-tracking tasks of Common-IM, the Uncommon-IM task is associated with the most intensive contralateral CMC. Our study demonstrated that the pattern of finger coordination had significant impact on the CMC between the contralateral M1 and hand muscles, and more corticomuscular interaction was necessary for unnaturally coordinated finger activities to regulate the fixed neural drive of hand muscles.

Neuron Selection by Relative Importance for Neural Decoding of Dexterous Finger Prosthesis Control Application

Future generations of upper limb prosthesis will have dexterous hand with individual fingers and will be controlled directly by neural signals. Neurons from the primary motor (M1) cortex code for finger movements and provide the source for neural control of dexterous prosthesis. Each neuron's activation can be quantified by the change in firing rate before and after finger movement, and the quantified value is then represented by the neural activity over each trial for the intended movement. Since this neural activity varies with the intended movement, we define the relative importance of each neuron independent of specific intended movements. The relative importance of each neuron is determined by the inter-movement variance of the neural activities for respective intended movements. Neurons are ranked by the relative importance and then a subpopulation of rank-ordered neurons is selected for the neural decoding. The use of the proposed neuron selection method in individual finger movements improved decoding accuracy by 21.5% in the case of decoding with only 5 neurons and by 9.2% in the case of decoding with only 10 neurons. With only 15 highly-ranked neurons, a decoding accuracy of 99.5% was achieved. The performance improvement is still maintained when combined movements of two fingers were included though the decoding accuracy fell to 95.7%. Since the proposed neuron selection method can achieve the targeting accuracy of decoding algorithms with less number of input neurons, it can be significant for developing brain-machine interfaces for direct neural control of hand prostheses.

Detection and classification of multiple finger movements using a chronically implanted Utah Electrode Array

The ability to detect and classify individual and combined finger movements from neural data is rapidly advancing. The work that has been done has demonstrated the feasibility of decoding finger movements from acutely recorded neurons. There is a need for a recording model that meets the chronic requirements of a neuroprosthetic application and to address this need we have developed an algorithm that can detect and classify individual and combined finger movements using neuronal data acquired from a chronically implanted Utah Electrode Array (UEA). The algorithm utilized the firing rates of individual neurons and performed with an average sensitivity and an average specificity that were both greater than 92% across all movement types. These results lend further support that a chronically implanted UEA is suitable for acquiring and decoding neuronal data and also demonstrate a decoding method that can detect and classify finger movements without any a priori knowledge of the data, task, or behavior.

Optimal control-based bayesian detection of clinical and behavioral state transitions

Accurately detecting hidden clinical or behavioral states from sequential measurements is an emerging topic in neuroscience and medicine, which may dramatically impact neural prosthetics, brain-computer interface and drug delivery. For example, early detection of an epileptic seizure from sequential electroencephalographic (EEG) measurements would allow timely administration of anticonvulsant drugs or neurostimulation, thus reducing physical impairment and risks of overtreatment. We develop a Bayesian paradigm for state transition detection that combines optimal control and Markov processes. We define a hidden Markov model of the state evolution and develop a detection policy that minimizes a loss function of both probability of false positives and accuracy (i.e., lag between estimated and actual transition time). Our strategy automatically adapts to each newly acquired measurement based on the state evolution model and the relative loss for false positives and accuracy, thus resulting in a time varying threshold policy. The paradigm was used in two applications: 1) detection of movement onset (behavioral state) from subthalamic single unit recordings in Parkinson's disease patients performing a motor task; 2) early detection of an approaching seizure (clinical state) from multichannel intracranial EEG recordings in rodents treated with pentylenetetrazol chemoconvulsant. Our paradigm performs significantly better than chance and improves over widely used detection algorithms.

Decoding dexterous finger movements in a neural prosthesis model approaching real-world conditions

Dexterous finger movements can be decoded from neuronal action potentials acquired from a nonhuman primate using a chronically implanted Utah Electrode Array. We have developed an algorithm that can, after training, detect and classify individual and combined finger movements without any a priori knowledge of the data, task, or behavior. The algorithm is based on changes in the firing rates of individual neurons that are tuned for one or more finger movement types. Nine different movement types, which consisted of individual flexions, individual extensions, and combined flexions of the thumb, index finger, and middle finger, were decoded. The algorithm performed reliably on data recorded continuously during movement tasks, including a no-movement state, with an overall average sensitivity and specificity that were both > 92%. These results demonstrate a viable algorithm for decoding dexterous finger movements under conditions similar to those required for a real-world neural prosthetic application.

Classification of multichannel ECoG related to individual finger movements with redundant spatial projections

We tackle the problem of classifying multichannel electrocorticogram (ECoG) related to individual finger movements for a brain machine interface (BMI). For this particular aim we applied a recently developed hierarchical spatial projection framework of neural activity for feature extraction from ECoG. The algorithm extends the binary common spatial patterns algorithm to multiclass problem by constructing a redundant set of spatial projections that are tuned for paired and group-wise discrimination of finger movements. The groupings were constructed by merging the data of adjacent fingers and contrasting them to the rest, such as the first two fingers (thumb and index) vs. the others (middle, ring and little). We applied this framework to the BCI competition IV ECoG data recorded from three subjects. We observed that the maximum classification accuracy was obtained from the gamma frequency band (65200 Hz). For this particular frequency range the average classification accuracy over three subjects was 86.3%. These results indicate that the redundant spatial projection framework can be used successfully in decoding finger movements from ECoG for BMI.

Two distinct ipsilateral cortical representations for individuated finger movements

Movements of the upper limb are controlled mostly through the contralateral hemisphere. Although overall activity changes in the ipsilateral motor cortex have been reported, their functional significance remains unclear. Using human functional imaging, we analyzed neural finger representations by studying differences in fine-grained activation patterns for single isometric finger presses. We demonstrate that cortical motor areas encode ipsilateral movements in 2 fundamentally different ways. During unimanual ipsilateral finger presses, primary sensory and motor cortices show, underneath global suppression, finger-specific activity patterns that are nearly identical to those elicited by contralateral mirror-symmetric action. This component vanishes when both motor cortices are functionally engaged during bimanual actions. We suggest that the ipsilateral representation present during unimanual presses arises because otherwise functionally idle circuits are driven by input from the opposite hemisphere. A second type of representation becomes evident in caudal premotor and anterior parietal cortices during bimanual actions. In these regions, ipsilateral actions are represented as nonlinear modulation of activity patterns related to contralateral actions, an encoding scheme that may provide the neural substrate for coordinating bimanual movements. We conclude that ipsilateral cortical representations change their informational content and functional role, depending on the behavioral context.

Asynchronous decoding of finger position and of EMG during precision grip using CM cell activity: application to robot control

Recent brain-machine interfaces (BMI) have demonstrated the use of intracortical signals for the kinematic control of robotic arms. However, for potential restoration of manual dexterity, two issues remain to be addressed: (1) Can hand and digit movements for dexterous manipulation be controlled in a similar way to arm movements? (2) Can the potentially large signal space for decoding of the many degrees of freedom (dof) of hand and digit movements be minimized? The first question addresses BMI control of dexterous prosthetic devices, while the second addresses the problem of whether few, but identified, neurons might provide adequate decoding. Asynchronous decoding of precision grip finger movement kinematics from identified corticomotoneuronal (CM) cell activity was performed with an artificial neural network (ANN). After training over a given session, the ANNs successfully decoded trial-by-trial movement kinematics. Average accuracy over sessions was in the order of 80% and 50% for data sets of two monkeys respectively. Decoding accuracy increased as a function of (1) number of simultaneously recorded CM cells used for prediction, and (2) size of the sliding input window. Subsequently, a robot digit actuated by pneumatic artificial muscles, fed with the predicted trajectory, mimicked the recorded movement offline. Furthermore, CM cell signals were used for decoding of time-varying hand muscle EMG activity. The performance of EMG prediction tended to increase if CM cells that facilitated this particular muscle (compared to CM cells that facilitated other muscles) were used. These results provide evidence that an anthropomorphic robot finger can be controlled offline by spike trains recorded from identified corticospinal neurons. This represents a step towards neuroprosthetic devices for dexterous hand movements.

Adaptation to a cortex-controlled robot attached at the pelvis and engaged during locomotion in rats

Brain-machine interfaces (BMIs) should ideally show robust adaptation of the BMI across different tasks and daily activities. Most BMIs have used overpracticed tasks. Little is known about BMIs in dynamic environments. How are mechanically body-coupled BMIs integrated into ongoing rhythmic dynamics, for example, in locomotion? To examine this, we designed a novel BMI using neural discharge in the hindlimb/trunk motor cortex in rats during locomotion to control a robot attached at the pelvis. We tested neural adaptation when rats experienced (1) control locomotion, (2) "simple elastic load" (a robot load on locomotion without any BMI neural control), and (3) "BMI with elastic load" (in which the robot loaded locomotion and a BMI neural control could counter this load). Rats significantly offset applied loads with the BMI while preserving more normal pelvic height compared with load alone. Adaptation occurred over ∼100-200 step cycles in a trial. Firing rates increased in both the loaded conditions compared with baseline. Mean phases of the discharge of cells in the step cycle shifted significantly between BMI and the simple load condition. Over time, more BMI cells became positively correlated with the external force and modulated more deeply, and the network correlations of neurons on a 100 ms timescale increased. Loading alone showed none of these effects. The BMI neural changes of rate and force correlations persisted or increased over repeated trials. Our results show that rats have the capacity to use motor adaptation and motor learning to fairly rapidly engage hindlimb/trunk-coupled BMIs in their locomotion.

Common muscle synergies for control of center of mass and force in nonstepping and stepping postural behaviors

We investigated muscle activity, ground reaction forces, and center of mass (CoM) acceleration in two different postural behaviors for standing balance control in humans to determine whether common neural mechanisms are used in different postural tasks. We compared nonstepping responses, where the base of support is stationary and balance is recovered by returning CoM back to its initial position, with stepping responses, where the base of support is enlarged and balance is recovered by pushing the CoM away from the initial position. In response to perturbations of the same direction, these two postural behaviors resulted in different muscle activity and ground reaction forces. We hypothesized that a common pool of muscle synergies producing consistent task-level biomechanical functions is used to generate different postural behaviors. Two sets of support-surface translations in 12 horizontal-plane directions were presented, first to evoke stepping responses and then to evoke nonstepping responses. Electromyographs in 16 lower back and leg muscles of the stance leg were measured. Initially (∼100-ms latency), electromyographs, CoM acceleration, and forces were similar in nonstepping and stepping responses, but these diverged in later time periods (∼200 ms), when stepping occurred. We identified muscle synergies using non-negative matrix factorization and functional muscle synergies that quantified correlations between muscle synergy recruitment levels and biomechanical outputs. Functional muscle synergies that produce forces to restore CoM position in nonstepping responses were also used to displace the CoM during stepping responses. These results suggest that muscle synergies represent common neural mechanisms for CoM movement control under different dynamic conditions: stepping and nonstepping postural responses.

Neural interfaces for control of upper limb prostheses: the state of the art and future possibilities

Current treatment of upper limb amputation restores some degree of functional ability, but this ability falls far below the standard set by the natural arm. Although acceptance rates can be high when patients are highly motivated and receive proper training and care, current prostheses often fail to meet the daily needs of amputees and frequently are abandoned. Recent advancements in science and technology have led to promising methods of accessing neural information for communication or control. Researchers have explored invasive and noninvasive methods of connecting with muscles, nerves, or the brain to provide increased functionality for patients experiencing disease or injury, including amputation. These techniques offer hope of more natural and intuitive prosthesis control, and therefore increased quality of life for amputees. In this review, we discuss the current state of the art of neural interfaces, particularly those that may find application within the prosthetics field.

Developing implantable neuroprosthetics: a new model in pig

A new model has been established in the domestic pig for neural prosthetic device development and testing. To this end, we report on a complete neural prosthetic developmental system using a wireless sensor as the implant, a pig as the animal model, and a novel data acquisition paradigm for actuator control. A new type of stereotactic frame with clinically-inspired fixations pins that place the pig brain in standard surgical plane was developed and tested with success during the implantation of the microsystem. The microsystem implanted was an ultra-low power (12.5 mW) 16-channel intracortical/epicranial device transmitting broadband (40 kS/s) data over a wireless infrared telemetric link. Pigs were implanted and neural data was collected over a period of 5 weeks, clearly showing single unit spiking activity.

Electrocorticographic amplitude predicts finger positions during slow grasping motions of the hand

Four human subjects undergoing subdural electrocorticography for epilepsy surgery engaged in a range of finger and hand movements. We observed that the amplitudes of the low-pass filtered electrocorticogram (ECoG), also known as the local motor potential (LMP), over specific peri-Rolandic electrodes were correlated (p < 0.001) with the position of individual fingers as the subjects engaged in slow and deliberate grasping motions. A generalized linear model (GLM) of the LMP amplitudes from those electrodes yielded predictions for positions of the fingers that had a strong congruence with the actual finger positions (correlation coefficient, r; median = 0.51, maximum = 0.91), during displacements of up to 10 cm at the fingertips. For all the subjects, decoding filters trained on data from any given session were remarkably robust in their prediction performance across multiple sessions and days, and were invariant with respect to changes in wrist angle, elbow flexion and hand placement across these sessions (median r = 0.52, maximum r = 0.86). Furthermore, a reasonable prediction accuracy for grasp aperture was achievable with as few as three electrodes in all subjects (median r = 0.49; maximum r = 0.90). These results provide further evidence for the feasibility of robust and practical ECoG-based control of finger movements in upper extremity prosthetics.

Multi-scale recordings for neuroprosthetic control of finger movements

We trained a rhesus monkey to perform individuated and combined finger flexions and extensions of the thumb, index, and middle finger. A Utah Electrode Array (UEA) was implanted into the hand region of the motor cortex contralateral to the monkey's trained hand. We also implanted a microwire electrocorticography grid (microECoG) epidurally so that it covered the UEA. The microECoG grid spanned the arm and hand regions of both the primary motor and somatosensory cortices. Previously this monkey had Implantable MyoElectric Sensors (IMES) surgically implanted into the finger muscles of the monkey's forearm. Action potentials (APs), local field potentials (LFPs), and microECoG signals were recorded from wired head-stage connectors for the UEA and microECoG grids, while EMG was recorded wirelessly. The monkey performed a finger flexion/extension task while neural and EMG data were acquired. We wrote an algorithm that uses the spike data from the UEA to perform a real-time decode of the monkey's finger movements. Also, analyses of the LFP and microECoG data indicate that these data show trial-averaged differences between different finger movements, indicating the data are potentially decodeable.

Cortical decoding of individual finger and wrist kinematics for an upper-limb neuroprosthesis

Previous research has shown that neuronal activity can be used to continuously decode the kinematics of gross movements involving arm and hand trajectory. However, decoding the kinematics of fine motor movements, such as the manipulation of individual fingers, has not been demonstrated. In this study, single unit activities were recorded from task-related neurons in M1 of two trained rhesus monkey as they performed individuated movements of the fingers and wrist. The primates' hand was placed in a manipulandum, and strain gauges at the tips of each finger were used to track the digit's position. Both linear and non-linear filters were designed to simultaneously predict kinematics of each digit and the wrist, and their performance compared using mean squared error and correlation coefficients. All models had high decoding accuracy, but the feedforward ANN (R = 0.76-0.86, MSE = 0.04-0.05) and Kalman filter (R = 0.68-0.86, MSE = 0.04-0.07) performed better than a simple linear regression filter (0.58-0.81, 0.05-0.07). These results suggest that individual finger and wrist kinematics can be decoded with high accuracy, and be used to control a multi-fingered prosthetic hand in real-time.

Continuous detection and decoding of dexterous finger flexions with implantable myoelectric sensors

A rhesus monkey was trained to perform individuated and combined finger flexions of the thumb, index, and middle finger. Nine implantable myoelectric sensors (IMES) were then surgically implanted into the finger muscles of the monkey's forearm, without any adverse effects over two years postimplantation. Using an inductive link, EMG was wirelessly recorded from the IMES as the monkey performed a finger flexion task. The EMG from the different IMES implants showed very little cross correlation. An offline parallel linear discriminant analysis (LDA) based algorithm was used to decode finger activity based on features extracted from continuously presented frames of recorded EMG. The offline parallel LDA was run on intraday sessions as well as on sessions where the algorithm was trained on one day and tested on following days. The performance of the algorithm was evaluated continuously by comparing classification output by the algorithm to the current state of the finger switches. The algorithm detected and classified seven different finger movements, including individual and combined finger flexions, and a no-movement state (chance performance = 12.5%) . When the algorithm was trained and tested on data collected the same day, the average performance was 43.8+/-3.6% n=10. When the training-testing separation period was five months, the average performance of the algorithm was 46.5+/-3.4% n=8. These results demonstrated that using EMG recorded and wirelessly transmitted by IMES offers a promising approach for providing intuitive, dexterous control of artificial limbs where human patients have sufficient, functional residual muscle following amputation.

Human motor cortical activity recorded with Micro-ECoG electrodes, during individual finger movements

In this study human motor cortical activity was recorded with a customized micro-ECoG grid during individual finger movements. The quality of the recorded neural signals was characterized in the frequency domain from three different perspectives: (1) coherence between neural signals recorded from different electrodes, (2) modulation of neural signals by finger movement, and (3) accuracy of finger movement decoding. It was found that, for the high frequency band (60-120 Hz), coherence between neighboring micro-ECoG electrodes was 0.3. In addition, the high frequency band showed significant modulation by finger movement both temporally and spatially, and a classification accuracy of 73% (chance level: 20%) was achieved for individual finger movement using neural signals recorded from the micro-ECoG grid. These results suggest that the micro-ECoG grid presented here offers sufficient spatial and temporal resolution for the development of minimally-invasive brain-computer interface applications.

Ensemble fractional sensitivity: a quantitative approach to neuron selection for decoding motor tasks

A robust method to help identify the population of neurons used for decoding motor tasks is developed. We use sensitivity analysis to develop a new metric for quantifying the relative contribution of a neuron towards the decoded output, called "fractional sensitivity." Previous model-based approaches for neuron ranking have been shown to largely depend on the collection of training data. We suggest the use of an ensemble of models that are trained on random subsets of trials to rank neurons. For this work, we tested a decoding algorithm on neuronal data recorded from two male rhesus monkeys while they performed a reach to grasp a bar at three orientations (45 degrees, 90 degrees, or 135 degrees). An ensemble approach led to a statistically significant increase of 5% in decoding accuracy and 25% increase in identification accuracy of simulated noisy neurons, when compared to a single model. Furthermore, ranking neurons based on the ensemble fractional sensitivities resulted in decoding accuracies 10%-20% greater than when randomly selecting neurons or ranking based on firing rates alone. By systematically reducing the size of the input space, we determine the optimal number of neurons needed for decoding the motor output. This selection approach has practical benefits for other BMI applications where limited number of electrodes and training datasets are available, but high decoding accuracies are desirable.

Listening to Brain Microcircuits for Interfacing With External World-Progress in Wireless Implantable Microelectronic Neuroengineering Devices: Experimental systems are described for electrical recording in the brain using multiple microelectrodes and short range implantable or wearable broadcasting units

Acquiring neural signals at high spatial and temporal resolution directly from brain microcircuits and decoding their activity to interpret commands and/or prior planning activity, such as motion of an arm or a leg, is a prime goal of modern neurotechnology. Its practical aims include assistive devices for subjects whose normal neural information pathways are not functioning due to physical damage or disease. On the fundamental side, researchers are striving to decipher the code of multiple neural microcircuits which collectively make up nature's amazing computing machine, the brain. By implanting biocompatible neural sensor probes directly into the brain, in the form of microelectrode arrays, it is now possible to extract information from interacting populations of neural cells with spatial and temporal resolution at the single cell level. With parallel advances in application of statistical and mathematical techniques tools for deciphering the neural code, extracted populations or correlated neurons, significant understanding has been achieved of those brain commands that control, e.g., the motion of an arm in a primate (monkey or a human subject). These developments are accelerating the work on neural prosthetics where brain derived signals may be employed to bypass, e.g., an injured spinal cord. One key element in achieving the goals for practical and versatile neural prostheses is the development of fully implantable wireless microelectronic "brain-interfaces" within the body, a point of special emphasis of this paper.

Decoding flexion of individual fingers using electrocorticographic signals in humans

Brain signals can provide the basis for a non-muscular communication and control system, a brain-computer interface (BCI), for people with motor disabilities. A common approach to creating BCI devices is to decode kinematic parameters of movements using signals recorded by intracortical microelectrodes. Recent studies have shown that kinematic parameters of hand movements can also be accurately decoded from signals recorded by electrodes placed on the surface of the brain (electrocorticography (ECoG)). In the present study, we extend these results by demonstrating that it is also possible to decode the time course of the flexion of individual fingers using ECoG signals in humans, and by showing that these flexion time courses are highly specific to the moving finger. These results provide additional support for the hypothesis that ECoG could be the basis for powerful clinically practical BCI systems, and also indicate that ECoG is useful for studying cortical dynamics related to motor function.

Neural control of motor prostheses

Neural interfaces (NIs) for motor control have recently become increasingly advanced. This has been possible owing to substantial progress in our understanding of the cortical motor system as well as the development of appropriate decoding methods in both non-human primates and paralyzed patients. So far, neural interfaces have controlled mainly computer screens and robotic arms. An important advancement has been the demonstration of neural interfaces that can directly control the subject's muscles. Furthermore, it has been shown that cortical plasticity alone can optimize neural interface performance in the absence of machine learning, which emphasizes the role of the brain for neural interface adaptation. Future motor prostheses may use also sensory feedback to enhance their control capabilities.

Matching brain-machine interface performance to space applications

A brain-machine interface (BMI) is a particular class of human-machine interface (HMI). BMIs have so far been studied mostly as a communication means for people who have little or no voluntary control of muscle activity. For able-bodied users, such as astronauts, a BMI would only be practical if conceived as an augmenting interface. A method is presented for pointing out effective combinations of HMIs and applications of robotics and automation to space. Latency and throughput are selected as performance measures for a hybrid bionic system (HBS), that is, the combination of a user, a device, and a HMI. We classify and briefly describe HMIs and space applications and then compare the performance of classes of interfaces with the requirements of classes of applications, both in terms of latency and throughput. Regions of overlap correspond to effective combinations. Devices requiring simpler control, such as a rover, a robotic camera, or environmental controls are suitable to be driven by means of BMI technology. Free flyers and other devices with six degrees of freedom can be controlled, but only at low-interactivity levels. More demanding applications require conventional interfaces, although they could be controlled by BMIs once the same levels of performance as currently recorded in animal experiments are attained. Robotic arms and manipulators could be the next frontier for noninvasive BMIs. Integrating smart controllers in HBSs could improve interactivity and boost the use of BMI technology in space applications.

Spectral modulation of LFP activity in M1 during dexterous finger movements

Recent studies have shown that cortical local field potentials (LFP) contain information about planning or executing hand movement. While earlier research has looked at gross motor movements, we investigate the spectral modulation of LFP activity and its dependence on recording location during dexterous motor actions. In this study, we recorded LFP activity from the primary motor cortex of a primate as it performed a fine finger manipulation task involving different switches. The event-related spectral perturbations (ERSP) in four different frequency bands were considered for the analysis; 4 Hz, 6-15 Hz, 17-40 Hz and 75-170 Hz. LFPs recorded from electrodes in the hand area showed the largest change in ERSP for the highest frequency band (75-170 Hz) (p 0.05), while LFPs recorded from electrodes placed more medially in the arm area showed the largest change in ERSP for the lowest frequency band (4 Hz) (p 0.05). Furthermore, the spectral information from the <4 Hz and 75-150 Hz frequency bands was used to successfully decode the three dexterous grasp movements with an average accuracy of up to 81%. Although previous research has shown that multi-unit neuronal activity can be used to decode fine motor movements, these results demonstrate that LFP activity can also be used to decode dexterous motor tasks. This has implications for future neuroprosthetic devices due to the robustness of LFP signals for chronic recording.

Neural decoding of finger movements using Skellam-based maximum-likelihood decoding

We present an optimal method for decoding the activity of primary motor cortex (M1) neurons in a nonhuman primate during single finger movements. The method is based on the maximum-likelihood (ML) inference, which assuming the probability of finger movements is uniform, is equivalent to the maximum a posteriori (MAP) inference. Each neuron's activation is first quantified by the change in firing rate before and after finger movement. We then estimate the probability density function of this activation given finger movement, i.e., Pr(neuronal activation (x) | finger movements (m)). Based on the ML criterion, we choose finger movements to maximize Pr(x |m). Experimentally, data were collected from 115 task-related neurons in M1 as the monkey performed flexion and extension of each finger and the wrist (12 movements). With as few as 20--25 randomly selected neurons, the proposed method decoded single-finger movements with 99% accuracy. Since the training and decoding procedures in the proposed method are simple and computationally efficient, the method can be extended for real-time neuroprosthetic control of a dexterous hand.

Decoupling the cortical power spectrum reveals real-time representation of individual finger movements in humans

During active movement the electric potentials measured from the surface of the motor cortex exhibit consistent modulation, revealing two distinguishable processes in the power spectrum. At frequencies <40 Hz, narrow-band power decreases occur with movement over widely distributed cortical areas, while at higher frequencies there are spatially more focal power increases. These high-frequency changes have commonly been assumed to reflect synchronous rhythms, analogous to lower-frequency phenomena, but it has recently been proposed that they reflect a broad-band spectral change across the entire spectrum, which could be obscured by synchronous rhythms at low frequencies. In 10 human subjects performing a finger movement task, we demonstrate that a principal component type of decomposition can naively separate low-frequency narrow-band rhythms from an asynchronous, broad-spectral, change at all frequencies between 5 and 200 Hz. This broad-spectral change exhibited spatially discrete representation for individual fingers and reproduced the temporal movement trajectories of different individual fingers.

Asynchronous decoding of dexterous finger movements using M1 neurons

Previous efforts in brain-machine interfaces (BMI) have looked at decoding movement intent or hand and arm trajectory, but current cortical control strategies have not focused on the decoding of dexterous [corrected] actions such as finger movements. The present work demonstrates the asynchronous decoding (i.e., where cues indicating the onset of movement are not known) of individual and combined finger movements. Single-unit activities were recorded sequentially from a population of neurons in the M1 hand area of trained rhesus monkeys during flexion and extension movements of each finger and the wrist. Nonlinear filters were designed to detect the onset of movement and decode the movement type from randomly selected neuronal ensembles (assembled from individually recorded single-unit activities). Average asynchronous decoding accuracies as high as 99.8%, 96.2%, and 90.5%, were achieved for individuated finger and wrist movements with three monkeys. Average decoding accuracy was still 92.5% when combined movements of two fingers were included. These results demonstrate that it is possible to asynchronously decode dexterous finger movements from a neuronal ensemble with high accuracy. This work takes an important step towards the development of a BMI for direct neural control of a state-of-the-art, multifingered hand prosthesis.

Decoding individuated finger movements using volume-constrained neuronal ensembles in the M1 hand area

Individuated finger and wrist movements can be decoded using random subpopulations of neurons that are widely distributed in the primary motor (M1) hand area. This work investigates 1) whether it is possible to decode dexterous finger movements using spatially-constrained volumes of neurons as typically recorded from a microelectrode array; and 2) whether decoding accuracy differs due to the configuration or location of the array within the M1 hand area. Single-unit activities were sequentially recorded from task-related neurons in two rhesus monkeys as they performed individuated movements of the fingers and the wrist. Simultaneous neuronal ensembles were simulated by constraining these activities to the recording field dimensions of conventional microelectrode array architectures. Artificial neural network (ANN) based filters were able to decode individuated finger movements with greater than 90% accuracy for the majority of movement types, using as few as 20 neurons from these ensemble activities. Furthermore, for the large majority of cases there were no significant differences (p < 0.01) in decoding accuracy as a function of the location of the recording volume. The results suggest that a brain-machine interface (BMI) for dexterous control of individuated fingers and the wrist can be implemented using microelectrode arrays placed broadly in the M1 hand area.