Prediction of three-dimensional arm trajectories based on ECoG signals recorded from human sensorimotor cortex

Platelets and hemostatic proteins are co-localized with chronic neuroinflammation surrounding implanted intracortical microelectrodes

Intracortical microelectrodes induce vascular injury upon insertion into the cortex. As blood vessels rupture, blood proteins and blood-derived cells (including platelets) are introduced into the 'immune privileged' brain tissues at higher-than-normal levels, passing through the damaged blood-brain barrier. Blood proteins adhere to implant surfaces, increasing the likelihood of cellular recognition leading to activation of immune and inflammatory cells. Persistent neuroinflammation is a major contributing factor to declining microelectrode recording performance. We investigated the spatial and temporal relationship of blood proteins fibrinogen and von Willebrand Factor (vWF), platelets, and type IV collagen, in relation to glial scarring markers for microglia and astrocytes following implantation of non-functional multi-shank silicon microelectrode probes into rats. Together with type IV collagen, fibrinogen and vWF augment platelet recruitment, activation, and aggregation. Our main results indicate blood proteins participating in hemostasis (fibrinogen and vWF) persisted at the microelectrode interface for up to 8-weeks after implantation. Further, type IV collagen and platelets surrounded the probe interface with similar spatial and temporal trends as vWF and fibrinogen. In addition to prolonged blood-brain barrier instability, specific blood and extracellular matrix proteins may play a role in promoting the inflammatory activation of platelets and recruitment to the microelectrode interface. STATEMENT OF SIGNIFICANCE: Implanted microelectrodes have substantial potential for restoring function to people with paralysis and amputation by providing signals that feed into natural control algorithms that drive prosthetic devices. Unfortunately, these microelectrodes do not display robust performance over time. Persistent neuroinflammation is widely thought to be a primary contributor to the devices' progressive decline in performance. Our manuscript reports on the highly local and persistent accumulation of platelets and hemostatic blood proteins around the microelectrode interface of brain implants. To our knowledge neuroinflammation driven by cellular and non-cellular responses associated with hemostasis and coagulation has not been rigorously quantified elsewhere. Our findings identify potential targets for therapeutic intervention and a better understanding of the driving mechanisms to neuroinflammation in the brain.

Investigating the Application of Graph Theory Features in Hand Movement Directions Decoding using EEG Signals

In recent years, functional analysis of brain networks based on graph theory properties has attracted considerable attention. This approach has usually been exploited for structural and functional brain analysis, while its potential in motor decoding tasks has remained unexplored. This study aimed to investigate the feasibility of using graph-based features in hand direction decoding in movement execution and preparation intervals. Hence, EEG signals were recorded from nine healthy subjects while performing a four-target center-out reaching task. The functional brain network was calculated based on the magnitude squared coherence (MSC) at six frequency bands. Then, the features based on eight graph theory metrics were extracted from brain networks. The classification was performed with a support vector machine classifier. The results revealed that in four-class direction discrimination, the mean accuracy of the graph-based method surpassed 63% and 53% on movement and pre-movement data, respectively. Additionally, a feature fusion approach that combines the graph theory features with power features was proposed. The fusion method raised the classification accuracy to 70.8% and 61.2% for movement and pre-movement intervals, respectively. This work has verified the feasibility of using graph theory properties and their superiority over band power features in a hand movement decoding task.

Impact of dataset size and long-term ECoG-based BCI usage on deep learning decoders performance

Introduction

In brain-computer interfaces (BCI) research, recording data is time-consuming and expensive, which limits access to big datasets. This may influence the BCI system performance as machine learning methods depend strongly on the training dataset size. Important questions arise: taking into account neuronal signal characteristics (e.g., non-stationarity), can we achieve higher decoding performance with more data to train decoders? What is the perspective for further improvement with time in the case of long-term BCI studies? In this study, we investigated the impact of long-term recordings on motor imagery decoding from two main perspectives: model requirements regarding dataset size and potential for patient adaptation.

Methods

We evaluated the multilinear model and two deep learning (DL) models on a long-term BCI & Tetraplegia (ClinicalTrials.gov identifier: NCT02550522) clinical trial dataset containing 43 sessions of ECoG recordings performed with a tetraplegic patient. In the experiment, a participant executed 3D virtual hand translation using motor imagery patterns. We designed multiple computational experiments in which training datasets were increased or translated to investigate the relationship between models' performance and different factors influencing recordings.

Results

Our results showed that DL decoders showed similar requirements regarding the dataset size compared to the multilinear model while demonstrating higher decoding performance. Moreover, high decoding performance was obtained with relatively small datasets recorded later in the experiment, suggesting motor imagery patterns improvement and patient adaptation during the long-term experiment. Finally, we proposed UMAP embeddings and local intrinsic dimensionality as a way to visualize the data and potentially evaluate data quality.

Discussion

DL-based decoding is a prospective approach in BCI which may be efficiently applied with real-life dataset size. Patient-decoder co-adaptation is an important factor to consider in long-term clinical BCI.

Decoding trajectories of imagined hand movement using electrocorticograms for brain-machine interface

OBJECTIVE

Reaching hand movement is an important motor skill actively examined in brain-computer interface (BCI). Among various components of movement analyzed is the hand's trajectory, which describes the hand's continuous positions in three-dimensional space. While a large body of studies have investigated the decoding of real movements and the reconstruction of real hand movement trajectories from neural signals, fewer studies have attempted to decode the trajectory of imagined hand movement. To develop BCI systems for patients with hand motor dysfunctions, the systems essentially require to achieve movement-free control of external devices, which is only possible through successful decoding of purely imagined hand movement.

APPROACH

To achieve this goal, this study used a machine learning technique (i.e., the variational Bayesian least square) to analyze the electrocorticogram (ECoG) of eighteen epilepsy patients obtained from when they performed movement execution (ME) and kinesthetic movement imagination (KMI) of the reach-and-grasp hand action.

MAIN RESULTS

The variational Bayesian decoding model was able to successfully predict the imagined trajectories of hand movement significantly above chance level. The Pearson's correlation coefficient between imagined and predicted trajectories was 0.3393 and 0.4936 for the KMI (KMI trials only) and MEKMI paradigm (alternating trials of ME and KMI) respectively.

SIGNIFICANCE

This study demonstrated a high accuracy of prediction for trajectories of imagined hand movement, and more importantly, higher decoding accuracy of imagined trajectories in the MEKMI paradigm than in the KMI paradigm solely.

Continuous Decoding of Hand Movement From EEG Signals Using Phase-Based Connectivity Features

The principal goal of the brain-computer interface (BCI) is to translate brain signals into meaningful commands to control external devices or neuroprostheses to restore lost functions of patients with severe motor disabilities. The invasive recording of brain signals involves numerous health issues. Therefore, BCIs based on non-invasive recording modalities such as electroencephalography (EEG) are safer and more comfortable for the patients. The BCI requires reconstructing continuous movement parameters such as position or velocity for practical application of neuroprostheses. The BCI studies in continuous decoding have extensively relied on extracting features from the amplitude of brain signals, whereas the brain connectivity features have rarely been explored. This study aims to investigate the feasibility of using phase-based connectivity features in decoding continuous hand movements from EEG signals. To this end, the EEG data were collected from seven healthy subjects performing a 2D center-out hand movement task in four orthogonal directions. The phase-locking value (PLV) and magnitude-squared coherence (MSC) are exploited as connectivity features along with multiple linear regression (MLR) for decoding hand positions. A brute-force search approach is employed to find the best channel pairs for extracting features related to hand movements. The results reveal that the regression models based on PLV and MSC features achieve the average Pearson correlations of 0.43 ± 0.03 and 0.42 ± 0.06, respectively, between predicted and actual trajectories over all subjects. The delta and alpha band features have the most contribution in regression analysis. The results also demonstrate that both PLV and MSC decoding models lead to superior results on our data compared to two recently proposed feature extraction methods solely based on the amplitude or phase of recording signals (p < 0.05). This study verifies the ability of PLV and MSC features in the continuous decoding of hand movements with linear regression. Thus, our findings suggest that extracting features based on brain connectivity can improve the accuracy of trajectory decoder BCIs.

A Design of Biomimetic Prosthetic Hand

Many patients with upper limb defects desire myoelectric prosthetic hands, but they are still not used for some reasons. One of the most significant reasons is its external appearance, which has the discomfort caused by the structural difference between a human hand and a robotic link. The structure must be based on human anatomy to create a more natural-looking prosthesis. This study designed a biomimetic prosthetic hand with bones, ligaments, tendons, and multiple muscles based on the human musculoskeletal system. We verified the proposed prosthetic hand using the viscoelastic angle sensor to determine whether it works like a human hand. We also compared the finger force of the prosthetic hand with that of a human finger. It could be capable of controlling the angle and the stiffness of the joint by multiple extensor and flexor muscles, like humans.

Machine learning based brain signal decoding for intelligent adaptive deep brain stimulation

Sensing enabled implantable devices and next-generation neurotechnology allow real-time adjustments of invasive neuromodulation. The identification of symptom and disease-specific biomarkers in invasive brain signal recordings has inspired the idea of demand dependent adaptive deep brain stimulation (aDBS). Expanding the clinical utility of aDBS with machine learning may hold the potential for the next breakthrough in the therapeutic success of clinical brain computer interfaces. To this end, sophisticated machine learning algorithms optimized for decoding of brain states from neural time-series must be developed. To support this venture, this review summarizes the current state of machine learning studies for invasive neurophysiology. After a brief introduction to the machine learning terminology, the transformation of brain recordings into meaningful features for decoding of symptoms and behavior is described. Commonly used machine learning models are explained and analyzed from the perspective of utility for aDBS. This is followed by a critical review on good practices for training and testing to ensure conceptual and practical generalizability for real-time adaptation in clinical settings. Finally, first studies combining machine learning with aDBS are highlighted. This review takes a glimpse into the promising future of intelligent adaptive DBS (iDBS) and concludes by identifying four key ingredients on the road for successful clinical adoption: i) multidisciplinary research teams, ii) publicly available datasets, iii) open-source algorithmic solutions and iv) strong world-wide research collaborations.

AJILE12: Long-term naturalistic human intracranial neural recordings and pose

Understanding the neural basis of human movement in naturalistic scenarios is critical for expanding neuroscience research beyond constrained laboratory paradigms. Here, we describe our Annotated Joints in Long-term Electrocorticography for 12 human participants (AJILE12) dataset, the largest human neurobehavioral dataset that is publicly available; the dataset was recorded opportunistically during passive clinical epilepsy monitoring. AJILE12 includes synchronized intracranial neural recordings and upper body pose trajectories across 55 semi-continuous days of naturalistic movements, along with relevant metadata, including thousands of wrist movement events and annotated behavioral states. Neural recordings are available at 500 Hz from at least 64 electrodes per participant, for a total of 1280 hours. Pose trajectories at 9 upper-body keypoints were estimated from 118 million video frames. To facilitate data exploration and reuse, we have shared AJILE12 on The DANDI Archive in the Neurodata Without Borders (NWB) data standard and developed a browser-based dashboard.

Decoding Continuous Kinetic Information of Grasp from Stereo-electroencephalographic (SEEG) Recordings

OBJECTIVE

Brain-computer interfaces (BCIs) have the potential to bypass damaged neural pathways and restore functionality lost due to injury or disease. Approaches to decoding kinematic information are well documented; however, the decoding of kinetic information has received less attention. Additionally, the possibility of using stereo-electroencephalography (SEEG) for kinetic decoding during hand grasping tasks is still largely unknown. Thus, the objective of this paper is to demonstrate kinetic parameter decoding using SEEG in patients performing a grasping task with two different force levels under two different ascending rates.

APPROACH

Temporal-spectral representations were studied to investigate frequency modulation under different force tasks. Then, force amplitude was decoded from SEEG recordings using multiple decoders, including a linear model, a partial least squares (PLS) model, an unscented Kalman filter, and three deep learning models (shallow convolutional neural network, deep convolutional neural network and the proposed CNN+RNN neural network).

MAIN RESULTS

The current study showed that: 1) for some channel, both low-frequency modulation (event-related desynchronization (ERD)) and high-frequency modulation (event-related synchronization (ERS)) were sustained during prolonged force holding periods; 2) continuously changing grasp force can be decoded from the SEEG signals; 3) the novel CNN+RNN deep learning model achieved the best decoding performance, with the predicted force magnitude closely aligned to the ground truth under different force amplitudes and changing rates.

SIGNIFICANCE

This work verified the possibility of decoding continuously changing grasp force using SEEG recordings. The result presented in this study demonstrated the potential of SEEG recordings for future BCI application.

Decoding ECoG signal into 3D hand translation using deep learning

Objective.Motor brain-computer interfaces (BCIs) are a promising technology that may enable motor-impaired people to interact with their environment. BCIs would potentially compensate for arm and hand function loss, which is the top priority for individuals with tetraplegia. Designing real-time and accurate BCI is crucial to make such devices useful, safe, and easy to use by patients in a real-life environment. Electrocorticography (ECoG)-based BCIs emerge as a good compromise between invasiveness of the recording device and good spatial and temporal resolution of the recorded signal. However, most ECoG signal decoders used to predict continuous hand movements are linear models. These models have a limited representational capacity and may fail to capture the relationship between ECoG signal features and continuous hand movements. Deep learning (DL) models, which are state-of-the-art in many problems, could be a solution to better capture this relationship.Approach.In this study, we tested several DL-based architectures to predict imagined 3D continuous hand translation using time-frequency features extracted from ECoG signals. The dataset used in the analysis is a part of a long-term clinical trial (ClinicalTrials.gov identifier: NCT02550522) and was acquired during a closed-loop experiment with a tetraplegic subject. The proposed architectures include multilayer perceptron, convolutional neural networks (CNNs), and long short-term memory networks (LSTM). The accuracy of the DL-based and multilinear models was compared offline using cosine similarity.Main results.Our results show that CNN-based architectures outperform the current state-of-the-art multilinear model. The best architecture exploited the spatial correlation between neighboring electrodes with CNN and benefited from the sequential character of the desired hand trajectory by using LSTMs. Overall, DL increased the average cosine similarity, compared to the multilinear model, by up to 60%, from 0.189 to 0.302 and from 0.157 to 0.249 for the left and right hand, respectively.Significance.This study shows that DL-based models could increase the accuracy of BCI systems in the case of 3D hand translation prediction in a tetraplegic subject.

Applying Dimensionality Reduction Techniques in Source-Space Electroencephalography via Template and Magnetic Resonance Imaging-Derived Head Models to Continuously Decode Hand Trajectories

Several studies showed evidence supporting the possibility of hand trajectory decoding from low-frequency electroencephalography (EEG). However, the decoding in the source space via source localization is scarcely investigated. In this study, we tried to tackle the problem of collinearity due to the higher number of signals in the source space by two folds: first, we selected signals in predefined regions of interest (ROIs); second, we applied dimensionality reduction techniques to each ROI. The dimensionality reduction techniques were computing the mean (Mean), principal component analysis (PCA), and locality preserving projections (LPP). We also investigated the effect of decoding between utilizing a template head model and a subject-specific head model during the source localization. The results indicated that applying source-space decoding with PCA yielded slightly higher correlations and signal-to-noise (SNR) ratios than the sensor-space approach. We also observed slightly higher correlations and SNRs when applying the subject-specific head model than the template head model. However, the statistical tests revealed no significant differences between the source-space and sensor-space approaches and no significant differences between subject-specific and template head models. The decoder with Mean and PCA utilizes information mainly from precuneus and cuneus to decode the velocity kinematics similarly in the subject-specific and template head models.

An adaptive closed-loop ECoG decoder for long-term and stable bimanual control of an exoskeleton by a tetraplegic

OBJECTIVE

The article aims at addressing 2 challenges to step motor BCI out of laboratories: asynchronous control of complex bimanual effectors with large numbers of degrees of freedom, using chronic and safe recorders, and the decoding performance stability over time without frequent decoder recalibration.

APPROACH

Closed-loop adaptive/incremental decoder training is one strategy to create a model stable over time. Adaptive decoders update their parameters with new incoming data, optimizing the model parameters in real time. It allows cross-session training with multiple recording conditions during closed loop BCI experiments. In the article, an adaptive tensor-based Recursive Exponentially Weighted Markov-Switching multi-Linear Model (REW-MSLM) decoder is proposed. REW-MSLM uses a Mixture of Expert (ME) architecture, mixing or switching independent decoders (experts) according to the probability estimated by a "gating" model. A Hidden Markov model approach is employed as gating model to improve the decoding robustness and to provide strong idle state support. The ME architecture fits the multi-limb paradigm associating an expert to a particular limb or action.

MAIN RESULTS

Asynchronous control of an exoskeleton by a tetraplegic patient using a chronically implanted epidural electrocorticography (EpiCoG) recorder is reported. The stable over a period of 6 months (without decoder recalibration) 8-dimensional alternative bimanual control of the exoskeleton and its virtual avatar is demonstrated.

SIGNIFICANCE

Based on the long-term (>36 months) chronic bilateral epidural ECoG recordings in a tetraplegic (ClinicalTrials.gov, NCT02550522), we addressed the poorly explored field of asynchronous bimanual BCI. The new decoder was designed to meet to several challenges: the high-dimensional control of a complex effector in experiments closer to real-world behaviour (point-to-point pursuit versus conventional center-out tasks), with the ability of the BCI system to act as a stand-alone device switching between idle and control states, and a stable performance over a long period of time without decoder recalibration.

Predictive accuracy of CNN for cortical oscillatory activity in an acute rat model of parkinsonism

In neurological and neuropsychiatric disorders neuronal oscillatory activity between basal ganglia and cortical circuits are altered, which may be useful as biomarker for adaptive deep brain stimulation. We investigated whether changes in the spectral power of oscillatory activity in the motor cortex (MCtx) and the sensorimotor cortex (SMCtx) of rats after injection of the dopamine (DA) receptor antagonist haloperidol (HALO) would be similar to those observed in Parkinson disease. Thereafter, we tested whether a convolutional neural network (CNN) model would identify brain signal alterations in this acute model of parkinsonism. A sixteen channel surface micro-electrocorticogram (ECoG) recording array was placed under the dura above the MCtx and SMCtx areas of one hemisphere under general anaesthesia in rats. Seven days after surgery, micro ECoG was recorded in individual free moving rats in three conditions: (1) basal activity, (2) after injection of HALO (0.5 mg/kg), and (3) with additional injection of apomorphine (APO) (1 mg/kg). Furthermore, a CNN-based classification consisting of 23,530 parameters was applied on the raw data. HALO injection decreased oscillatory theta band activity (4-8 Hz) and enhanced beta (12-30 Hz) and gamma (30-100 Hz) in MCtx and SMCtx, which was compensated after APO injection (P ¡ 0.001). Evaluation of classification performance of the CNN model provided accuracy of 92%, sensitivity of 90% and specificity of 93% on one-dimensional signals. The CNN proposed model requires a minimum of sensory hardware and may be integrated into future research on therapeutic devices for Parkinson disease, such as adaptive closed loop stimulation, thus contributing to more efficient way of treatment.

Accurate Offline Asynchronous Detection of Individual Finger Movement From Intracranial Brain Signals Using a Novel Multiway Approach

Asynchronous motor Brain Computer Interfacing (BCI) is characterized by the continuous decoding of intended muscular activity from brain signals. Such applications have gained widespread interest for enabling users to issue commands volitionally. In conventional motor BCIs features extracted from brain signals are concatenated into vector- or matrix-based (or one-/two-way) representations. Nevertheless, when accounting for the original multimodal or multiway signal structure, decoding performance has been shown to improve jointly with result interpretability. However, as multiway decoders are notorious for the extensive computational cost to train them, conventional ones are still preferred. To curb this limitation, we introduce a novel multiway classifier, called Block-Term Tensor Classifier that inherits the improved accuracy of multiway methods while providing fast training. We show that it can outperform state-of-the-art multiway and two-way Linear Discriminant Analysis classifiers in asynchronous detection of individual finger movements from intracranial recordings, an essential feature to achieve a sense of dexterity with hand prosthetics and exoskeletons.

Behavioral and Neural Variability of Naturalistic Arm Movements

Motor behaviors are central to many functions and dysfunctions of the brain, and understanding their neural basis has consequently been a major focus in neuroscience. However, most studies of motor behaviors have been restricted to artificial, repetitive paradigms, far removed from natural movements performed "in the wild." Here, we leveraged recent advances in machine learning and computer vision to analyze intracranial recordings from 12 human subjects during thousands of spontaneous, unstructured arm reach movements, observed over several days for each subject. These naturalistic movements elicited cortical spectral power patterns consistent with findings from controlled paradigms, but with considerable neural variability across subjects and events. We modeled interevent variability using 10 behavioral and environmental features; the most important features explaining this variability were reach angle and day of recording. Our work is among the first studies connecting behavioral and neural variability across cortex in humans during unstructured movements and contributes to our understanding of long-term naturalistic behavior.

Distance- and speed-informed kinematics decoding improves M/EEG based upper-limb movement decoder accuracy

OBJECTIVE

One of the main goals in brain-computer interface (BCI) research is the replacement or restoration of lost function in individuals with paralysis. One line of research investigates the inference of movement kinematics from brain activity during different volitional states. A growing number of electroencephalography (EEG) and magnetoencephalography (MEG) studies suggest that information about directional (e.g. velocity) and nondirectional (e.g. speed) movement kinematics is accessible noninvasively. We sought to assess if the neural information associated with both types of kinematics can be combined to improve the decoding accuracy.

APPROACH

In an offline analysis, we reanalyzed the data of two previous experiments containing the recordings of 34 healthy participants (15 EEG, 19 MEG). We decoded 2D movement trajectories from low-frequency M/EEG signals in executed and observed tracking movements, and compared the accuracy of an unscented Kalman filter (UKF) that explicitly modeled the nonlinear relation between directional and nondirectional kinematics to the accuracies of linear Kalman (KF) and Wiener filters which did not combine both types of kinematics.

MAIN RESULTS

At the group level, posterior-parietal and parieto-occipital (executed and observed movements) and sensorimotor areas (executed movements) encoded kinematic information. Correlations between the recorded position and velocity trajectories and the UKF decoded ones were on average 0.49 during executed and 0.36 during observed movements. Compared to the other filters, the UKF could achieve the best trade-off between maximizing the signal to noise ratio and minimizing the amplitude mismatch between the recorded and decoded trajectories.

SIGNIFICANCE

We present direct evidence that directional and nondirectional kinematic information is simultaneously detectable in low-frequency M/EEG signals. Moreover, combining directional and nondirectional kinematic information significantly improves the decoding accuracy upon a linear KF.

Application of a neural interface for restoration of leg movements: Intra-spinal stimulation using the brain electrical activity in spinally injured rabbits

This study aimed to design a neural interface that extracts movement commands from the brain to generate appropriate intra-spinal stimulation to restore leg movement. This study comprised four steps: (1) Recording electrocorticographic (ECoG) signals and corresponding leg movements in different trials. (2) Partial laminectomy to induce spinal cord injury (SCI) and detect motor modules in the spinal cord. (3) Delivering appropriate intra-spinal stimulation to the motor modules for restoration of the movements to those documented before SCI. (4) Development of a neural interface created by sparse linear regression (SLiR) model to detect movement commands transmitted from the brain to the modules. Correlation coefficient (CC) and normalized root mean square (NRMS) error was calculated to evaluate the neural interface effectiveness. It was found that by stimulating detected spinal cord modules, joint angle evaluated before SCI was not significantly different from that of post-SCI (P > 0.05). Based on results of SLiR model, overall CC and NRMS values were 0.63 ± 0.14 and 0.34 ± 0.16 (mean ± SD), respectively. These results indicated that ECoG data contained information about intra-spinal stimulations and the developed neural interface could produce intra-spinal stimulation based on ECoG data, for restoration of leg movements after SCI.

A state-based probabilistic method for decoding hand position during movement from ECoG signals in non-human primate

OBJECTIVE

In this study, we proposed a state-based probabilistic method for decoding hand positions during unilateral and bilateral movements using the ECoG signals recorded from the brain of Rhesus monkey.

APPROACH

A customized electrode array was implanted subdurally in the right hemisphere of the brain covering from the primary motor cortex to the frontal cortex. Three different experimental paradigms were considered: ipsilateral, contralateral, and bilateral movements. During unilateral movement, the monkey was trained to get food with one hand, while during bilateral movement, the monkey used its left and right hands alternately to get food. To estimate the hand positions, a state-based probabilistic method was introduced which was based on the conditional probability of the hand movement state (i.e. idle, right hand movement, and left hand movement) and the conditional expectation of the hand position for each state. Moreover, a hybrid feature extraction method based on linear discriminant analysis and partial least squares (PLS) was introduced.

MAIN RESULTS

The proposed method could successfully decode the hand positions during ipsilateral, contralateral, and bilateral movements and significantly improved the decoding performance compared to the conventional Kalman and PLS regression methods [Formula: see text]. The proposed hybrid feature extraction method was found to outperform both the PLS and PCA methods [Formula: see text]. Investigating the kinematic information of each frequency band shows that more informative frequency bands were [Formula: see text] (15-30 Hz) and [Formula: see text](50-100 Hz) for ipsilateral and [Formula: see text] and [Formula: see text] (100-200 Hz) for contralateral movements. It is observed that ipsilateral movement was decoded better than contralateral movement for [Formula: see text] (5-15 Hz) and [Formula: see text] bands, while contralateral movements was decoded better for [Formula: see text] (30-200 Hz) and hfECoG (200-400 Hz) bands.

SIGNIFICANCE

Accurate decoding the bilateral movement using the ECoG recorded from one brain hemisphere is an important issue toward real-life applications of the brain-machine interface technologies.

Using machine learning to reveal the population vector from EEG signals

OBJECTIVE

Since the discovery of the population vector that directly relates neural spiking activity with arm movement direction, it has become feasible to control robotic arms and neuroprostheses using invasively recorded brain signals. For non-invasive approaches, a direct relation between human brain signals and arm movement direction is yet to be established.

APPROACH

Here, we investigated electroencephalographic (EEG) signals in temporal and spectral domains in a continuous, circular arm movement task. Using machine learning methods that respect the linear mixture of brain activity within EEG signals, we show that directional information is represented in the temporal domain in amplitude modulations of the same frequency as the arm movement, and in the spectral domain in power modulations of the 20-24 Hz frequency band.

MAIN RESULTS

In the temporal domain, the directional information was mainly expressed in primary sensorimotor cortex (SM1) and posterior parietal cortex (PPC) contralateral to the moving arm, while in the spectral domain SM1 and PPC of both hemispheres predicted arm movement direction. The different cortical representations suggest distinct neural representations in both domains.

SIGNIFICANCE

This direct relation between neural activity and arm movement direction in both domains demonstrates the potential of machine learning to reveal neuroscientific insights about the dynamics of human arm movements.

Electroencephalography of completely locked-in state patients with amyotrophic lateral sclerosis

Patients in completely locked-in state (CLIS) due to amyotrophic lateral sclerosis (ALS) lose the control of each and every muscle of their body rendering them motionless and without any means of communication. Though some studies have attempted to develop brain-computer interface (BCI)-based communication methods with CLIS patients, little information is available of the neuroelectric brain activity of CLIS patients. However, because of the difficulties with and often loss of communication, the neuroelectric signature may provide some indications of the state of consciousness in these patients. We recorded electroencephalography (EEG) signals from 10 CLIS patients during resting state and compared their power spectral densities with those of healthy participants in fronto-central, central, and centro-parietal channels. The results showed significant power reduction in the high alpha, beta, and gamma bands in CLIS patients, indicating the dominance of slower EEG frequencies in their oscillatory activity. This is the first study showing group-level EEG change of CLIS patients, though the reason for the observed EEG change cannot be concluded without any reliable communication methods with this population.

Decoding Movement From Electrocorticographic Activity: A Review

Electrocorticography (ECoG) holds promise to provide efficient neuroprosthetic solutions for people suffering from neurological disabilities. This recording technique combines adequate temporal and spatial resolution with the lower risks of medical complications compared to the other invasive methods. ECoG is routinely used in clinical practice for preoperative cortical mapping in epileptic patients. During the last two decades, research utilizing ECoG has considerably grown, including the paradigms where behaviorally relevant information is extracted from ECoG activity with decoding algorithms of different complexity. Several research groups have advanced toward the development of assistive devices driven by brain-computer interfaces (BCIs) that decode motor commands from multichannel ECoG recordings. Here we review the evolution of this field and its recent tendencies, and discuss the potential areas for future development.

Decoding Native Cortical Representations for Flexion and Extension at Upper Limb Joints Using Electrocorticography

Brain-machine interface (BMI) researchers have traditionally focused on modeling endpoint reaching tasks to provide the control of neurally driven prosthetic arms. Most previous research has focused on achieving an endpoint control through a Cartesian-coordinate-centered approach. However, a joint-centered approach could potentially be used to intuitively control a wide range of limb movements. We systematically investigated the feasibility of discriminating between flexion and extension of different upper limb joints using electrocorticography(ECoG) recordings from sensorimotor cortex. Four subjects implanted with macro-ECoG (10-mm spacing), high-density ECoG (5-mm spacing), and/or micro-ECoG arrays (0.9-mm spacing and 4 mm × 4 mm coverage), performed randomly cued flexions or extensions of the fingers, wrist, or elbow contralateral to the implanted hemisphere. We trained a linear model to classify six movements using averaged high-gamma power (70-110 Hz) modulations at different latencies with respect to movement onset, and within a time interval restricted to flexion or extension at each joint. Offline decoding models for each subject classified these movements with accuracies of 62%-83%. Our results suggest that the widespread ECoG coverage of sensorimotor cortex could allow a whole limb BMI to sample native cortical representations in order to control flexion and extension at multiple joints.

Invasive Brain Machine Interface System

Because of high spatial-temporal resolution of neural signals obtained by invasive recording, the invasive brain-machine interfaces (BMI) have achieved great progress in the past two decades. With success in animal research, BMI technology is transferring to clinical trials for helping paralyzed people to restore their lost motor functions. This chapter gives a brief review of BMI development from animal experiments to human clinical studies in the following aspects: (1) BMIs based on rodent animals; (2) BMI based on non-human primates; and (3) pilot BMIs studies in clinical trials. In the end, the chapter concludes with a summary of potential opportunities and future challenges in BMI technology.

Control of a Robot Arm Using Decoded Joint Angles from Electrocorticograms in Primate

Electrocorticogram (ECoG) is a well-known recording method for the less invasive brain machine interface (BMI). Our previous studies have succeeded in predicting muscle activities and arm trajectories from ECoG signals. Despite such successful studies, there still remain solving works for the purpose of realizing an ECoG-based prosthesis. We suggest a neuromuscular interface to control robot using decoded muscle activities and joint angles. We used sparse linear regression to find the best fit between band-passed ECoGs and electromyograms (EMG) or joint angles. The best coefficient of determination for 100 s continuous prediction was 0.6333 ± 0.0033 (muscle activations) and 0.6359 ± 0.0929 (joint angles), respectively. We also controlled a 4 degree of freedom (DOF) robot arm using only decoded 4 DOF angles from the ECoGs in this study. Consequently, this study shows the possibility of contributing to future advancements in neuroprosthesis and neurorehabilitation technology.

Data-Driven Transducer Design and Identification for Internally-Paced Motor Brain Computer Interfaces: A Review

Brain-Computer Interfaces (BCIs) are systems that establish a direct communication pathway between the users' brain activity and external effectors. They offer the potential to improve the quality of life of motor-impaired patients. Motor BCIs aim to permit severely motor-impaired users to regain limb mobility by controlling orthoses or prostheses. In particular, motor BCI systems benefit patients if the decoded actions reflect the users' intentions with an accuracy that enables them to efficiently interact with their environment. One of the main challenges of BCI systems is to adapt the BCI's signal translation blocks to the user to reach a high decoding accuracy. This paper will review the literature of data-driven and user-specific transducer design and identification approaches and it focuses on internally-paced motor BCIs. In particular, continuous kinematic biomimetic and mental-task decoders are reviewed. Furthermore, static and dynamic decoding approaches, linear and non-linear decoding, offline and real-time identification algorithms are considered. The current progress and challenges related to the design of clinical-compatible motor BCI transducers are additionally discussed.

A Fully Implantable Wireless ECoG 128-Channel Recording Device for Human Brain-Machine Interfaces: W-HERBS

Brain-machine interfaces (BMIs) are promising devices that can be used as neuroprostheses by severely disabled individuals. Brain surface electroencephalograms (electrocorticograms, ECoGs) can provide input signals that can then be decoded to enable communication with others and to control intelligent prostheses and home electronics. However, conventional systems use wired ECoG recordings. Therefore, the development of wireless systems for clinical ECoG BMIs is a major goal in the field. We developed a fully implantable ECoG signal recording device for human ECoG BMI, i.e., a wireless human ECoG-based real-time BMI system (W-HERBS). In this system, three-dimensional (3D) high-density subdural multiple electrodes are fitted to the brain surface and ECoG measurement units record 128-channel (ch) ECoG signals at a sampling rate of 1 kHz. The units transfer data to the data and power management unit implanted subcutaneously in the abdomen through a subcutaneous stretchable spiral cable. The data and power management unit then communicates with a workstation outside the body and wirelessly receives 400 mW of power from an external wireless transmitter. The workstation records and analyzes the received data in the frequency domain and controls external devices based on analyses. We investigated the performance of the proposed system. We were able to use W-HERBS to detect sine waves with a 4.8-μV amplitude and a 60-200-Hz bandwidth from the ECoG BMIs. W-HERBS is the first fully implantable ECoG-based BMI system with more than 100 ch. It is capable of recording 128-ch subdural ECoG signals with sufficient input-referred noise (3 μVrms) and with an acceptable time delay (250 ms). The system contributes to the clinical application of high-performance BMIs and to experimental brain research.

MEG-BMI to Control Phantom Limb Pain

A brachial plexus root avulsion (BPRA) causes intractable pain in the insensible affected hands. Such pain is partly due to phantom limb pain, which is neuropathic pain occurring after the amputation of a limb and partial or complete deafferentation. Previous studies suggested that the pain was attributable to maladaptive plasticity of the sensorimotor cortex. However, there is little evidence to demonstrate the causal links between the pain and the cortical representation, and how much cortical factors affect the pain. Here, we applied lesioning of the dorsal root entry zone (DREZotomy) and training with a brain–machine interface (BMI) based on real-time magnetoencephalography signals to reconstruct affected hand movements with a robotic hand. The DREZotomy successfully reduced the shooting pain after BPRA, but a part of the pain remained. The BMI training successfully induced some plastic changes in the sensorimotor representation of the phantom hand movements and helped control the remaining pain. When the patient tried to control the robotic hand by moving their phantom hand through association with the representation of the intact hand, this especially decreased the pain while decreasing the classification accuracy of the phantom hand movements. These results strongly suggested that pain after the BPRA was partly attributable to cortical representation of phantom hand movements and that the BMI training controlled the pain by inducing appropriate cortical reorganization. For the treatment of chronic pain, we need to know how to modulate the cortical representation by novel methods.

Decoding unconstrained arm movements in primates using high-density electrocorticography signals for brain-machine interface use

Motor deficit is among the most debilitating aspects of injury to the central nervous system. Despite ongoing progress in brain-machine interface (BMI) development and in the functional electrical stimulation of muscles and nerves, little is understood about how neural signals in the brain may be used to potentially control movement in one’s own unconstrained paralyzed limb. We recorded from high-density electrocorticography (ECoG) electrode arrays in the ventral premotor cortex (PMv) of a rhesus macaque and used real-time motion tracking techniques to correlate spatial-temporal changes in neural activity with arm movements made towards objects in three-dimensional space at millisecond precision. We found that neural activity from a small number of electrodes within the PMv can be used to accurately predict reach-return movement onset and directionality. Also, whereas higher gamma frequency field activity was more predictive about movement direction during performance, mid-band (beta and low gamma) activity was more predictive of movement prior to onset. We speculate these dual spatiotemporal signals may be used to optimize both planning and execution of movement during natural reaching, with prospective relevance to the future development of neural prosthetics aimed at restoring motor control over one’s own paralyzed limb.

Training in Use of Brain-Machine Interface-Controlled Robotic Hand Improves Accuracy Decoding Two Types of Hand Movements

Objective: Brain-machine interfaces (BMIs) are useful for inducing plastic changes in cortical representation. A BMI first decodes hand movements using cortical signals and then converts the decoded information into movements of a robotic hand. By using the BMI robotic hand, the cortical representation decoded by the BMI is modulated to improve decoding accuracy. We developed a BMI based on real-time magnetoencephalography (MEG) signals to control a robotic hand using decoded hand movements. Subjects were trained to use the BMI robotic hand freely for 10 min to evaluate plastic changes in the cortical representation due to the training.

Method: We trained nine young healthy subjects with normal motor function. In open-loop conditions, they were instructed to grasp or open their right hands during MEG recording. Time-averaged MEG signals were then used to train a real decoder to control the robotic arm in real time. Then, subjects were instructed to control the BMI-controlled robotic hand by moving their right hands for 10 min while watching the robot's movement. During this closed-loop session, subjects tried to improve their ability to control the robot. Finally, subjects performed the same offline task to compare cortical activities related to the hand movements. As a control, we used a random decoder trained by the MEG signals with shuffled movement labels. We performed the same experiments with the random decoder as a crossover trial. To evaluate the cortical representation, cortical currents were estimated using a source localization technique. Hand movements were also decoded by a support vector machine using the MEG signals during the offline task. The classification accuracy of the movements was compared among offline tasks.

Results: During the BMI training with the real decoder, the subjects succeeded in improving their accuracy in controlling the BMI robotic hand with correct rates of 0.28 ± 0.13 to 0.50 ± 0.11 (p = 0.017, n = 8, paired Student's t-test). Moreover, the classification accuracy of hand movements during the offline task was significantly increased after BMI training with the real decoder from 62.7 ± 6.5 to 70.0 ± 11.1% (p = 0.022, n = 8, t₍₇₎ = 2.93, paired Student's t-test), whereas accuracy did not significantly change after BMI training with the random decoder from 63.0 ± 8.8 to 66.4 ± 9.0% (p = 0.225, n = 8, t₍₇₎ = 1.33).

Conclusion: BMI training is a useful tool to train the cortical activity necessary for BMI control and to induce some plastic changes in the activity.

Remapping cortical modulation for electrocorticographic brain-computer interfaces: a somatotopy-based approach in individuals with upper-limb paralysis

OBJECTIVE

Brain-computer interface (BCI) technology aims to provide individuals with paralysis a means to restore function. Electrocorticography (ECoG) uses disc electrodes placed on either the surface of the dura or the cortex to record field potential activity. ECoG has been proposed as a viable neural recording modality for BCI systems, potentially providing stable, long-term recordings of cortical activity with high spatial and temporal resolution. Previously we have demonstrated that a subject with spinal cord injury (SCI) could control an ECoG-based BCI system with up to three degrees of freedom (Wang et al 2013 PLoS One). Here, we expand upon these findings by including brain-control results from two additional subjects with upper-limb paralysis due to amyotrophic lateral sclerosis and brachial plexus injury, and investigate the potential of motor and somatosensory cortical areas to enable BCI control.

APPROACH

Individuals were implanted with high-density ECoG electrode grids over sensorimotor cortical areas for less than 30 d. Subjects were trained to control a BCI by employing a somatotopic control strategy where high-gamma activity from attempted arm and hand movements drove the velocity of a cursor.

MAIN RESULTS

Participants were capable of generating robust cortical modulation that was differentiable across attempted arm and hand movements of their paralyzed limb. Furthermore, all subjects were capable of voluntarily modulating this activity to control movement of a computer cursor with up to three degrees of freedom using the somatotopic control strategy. Additionally, for those subjects with electrode coverage of somatosensory cortex, we found that somatosensory cortex was capable of supporting ECoG-based BCI control.

SIGNIFICANCE

These results demonstrate the feasibility of ECoG-based BCI systems for individuals with paralysis as well as highlight some of the key challenges that must be overcome before such systems are translated to the clinical realm. ClinicalTrials.gov Identifier: NCT01393444.

Reconstruction of reaching movement trajectories using electrocorticographic signals in humans

In this study, we used electrocorticographic (ECoG) signals to extract the onset of arm movement as well as the velocity of the hand as a function of time. ECoG recordings were obtained from three individuals while they performed reaching tasks in the left, right and forward directions. The ECoG electrodes were placed over the motor cortex contralateral to the moving arm. Movement onset was detected from gamma activity with near perfect accuracy (> 98%), and a multiple linear regression model was used to predict the trajectory of the reaching task in three-dimensional space with an accuracy exceeding 85%. An adaptive selection of frequency bands was used for movement classification and prediction. This demonstrates the efficacy of developing a real-time brain-machine interface for arm movements with as few as eight ECoG electrodes.

From classic motor imagery to complex movement intention decoding: The noninvasive Graz-BCI approach

In this chapter, we give an overview of the Graz-BCI research, from the classic motor imagery detection to complex movement intentions decoding. We start by describing the classic motor imagery approach, its application in tetraplegic end users, and the significant improvements achieved using coadaptive brain-computer interfaces (BCIs). These strategies have the drawback of not mirroring the way one plans a movement. To achieve a more natural control-and to reduce the training time-the movements decoded by the BCI need to be closely related to the user's intention. Within this natural control, we focus on the kinematic level, where movement direction and hand position or velocity can be decoded from noninvasive recordings. First, we review movement execution decoding studies, where we describe the decoding algorithms, their performance, and associated features. Second, we describe the major findings in movement imagination decoding, where we emphasize the importance of estimating the sources of the discriminative features. Third, we introduce movement target decoding, which could allow the determination of the target without knowing the exact movement-by-movement details. Aside from the kinematic level, we also address the goal level, which contains relevant information on the upcoming action. Focusing on hand-object interaction and action context dependency, we discuss the possible impact of some recent neurophysiological findings in the future of BCI control. Ideally, the goal and the kinematic decoding would allow an appropriate matching of the BCI to the end users' needs, overcoming the limitations of the classic motor imagery approach.

Physiological properties of brain-machine interface input signals

Brain-machine interfaces (BMIs), also called brain-computer interfaces (BCIs), decode neural signals and use them to control some type of external device. Despite many experimental successes and terrific demonstrations in animals and humans, a high-performance, clinically viable device has not yet been developed for widespread usage. There are many factors that impact clinical viability and BMI performance. Arguably, the first of these is the selection of brain signals used to control BMIs. In this review, we summarize the physiological characteristics and performance-including movement-related information, longevity, and stability-of multiple types of input signals that have been used in invasive BMIs to date. These include intracortical spikes as well as field potentials obtained inside the cortex, at the surface of the cortex (electrocorticography), and at the surface of the dura mater (epidural signals). We also discuss the potential for future enhancements in input signal performance, both by improving hardware and by leveraging the knowledge of the physiological characteristics of these signals to improve decoding and stability.

Mapping ECoG channel contributions to trajectory and muscle activity prediction in human sensorimotor cortex

Studies on brain-machine interface techniques have shown that electrocorticography (ECoG) is an effective modality for predicting limb trajectories and muscle activity in humans. Motor control studies have also identified distributions of “extrinsic-like” and “intrinsic-like” neurons in the premotor (PM) and primary motor (M1) cortices. Here, we investigated whether trajectories and muscle activity predicted from ECoG were obtained based on signals derived from extrinsic-like or intrinsic-like neurons. Three participants carried objects of three different masses along the same counterclockwise path on a table. Trajectories of the object and upper arm muscle activity were predicted using a sparse linear regression. Weight matrices for the predictors were then compared to determine if the ECoG channels contributed more information about trajectory or muscle activity. We found that channels over both PM and M1 contributed highly to trajectory prediction, while a channel over M1 was the highest contributor for muscle activity prediction.

Phase locking of β oscillation in electrocorticography (ECoG) in the monkey motor cortex at the onset of EMGs and 3D reaching movements

β oscillations in local field potentials, electrocorticography (ECoG), and electroencephalograms (EEG) are ubiquitous in the motor cortex of monkeys and humans. However, relations between their dynamical properties and behavior have not been well studied. Here, we used ECoG grids to cover large areas of motor cortex and some somatosensory cortex in monkeys while they performed an unconstrained reaching and a lever pulling task in three dimensional space with or without go cue. We used percentage of phase locking (PPL) as a dynamical property of β oscillations aligned to behaviorally relevant events, reaching onsets (RO), lever onset (LO), and holding onset (HO) as well as multiple peaks of electromyography (EMG) recorded from various upper limb muscles. We showed that relative strength of PPL aligned to RO and LO were reserved with or without the external go cues. Furthermore, among the muscles that we recorded EMGs from, β oscillations were not closely phase locked to any of EMG onsets. Therefore, phase locking of β oscillations is related more to the attentive state and external cues as opposed to detailed muscle activities.

3D hand motion trajectory prediction from EEG mu and beta bandpower

A motion trajectory prediction (MTP) - based brain-computer interface (BCI) aims to reconstruct the three-dimensional (3D) trajectory of upper limb movement using electroencephalography (EEG). The most common MTP BCI employs a time series of bandpass-filtered EEG potentials (referred to here as the potential time-series, PTS, model) for reconstructing the trajectory of a 3D limb movement using multiple linear regression. These studies report the best accuracy when a 0.5-2Hz bandpass filter is applied to the EEG. In the present study, we show that spatiotemporal power distribution of theta (4-8Hz), mu (8-12Hz), and beta (12-28Hz) bands are more robust for movement trajectory decoding when the standard PTS approach is replaced with time-varying bandpower values of a specified EEG band, ie, with a bandpower time-series (BTS) model. A comprehensive analysis comprising of three subjects performing pointing movements with the dominant right arm toward six targets is presented. Our results show that the BTS model produces significantly higher MTP accuracy (R~0.45) compared to the standard PTS model (R~0.2). In the case of the BTS model, the highest accuracy was achieved across the three subjects typically in the mu (8-12Hz) and low-beta (12-18Hz) bands. Additionally, we highlight a limitation of the commonly used PTS model and illustrate how this model may be suboptimal for decoding motion trajectory relevant information. Although our results, showing that the mu and beta bands are prominent for MTP, are not in line with other MTP studies, they are consistent with the extensive literature on classical multiclass sensorimotor rhythm-based BCI studies (classification of limbs as opposed to motion trajectory prediction), which report the best accuracy of imagined limb movement classification using power values of mu and beta frequency bands. The methods proposed here provide a positive step toward noninvasive decoding of imagined 3D hand movements for movement-free BCIs.

Electrocorticographic signals comparison in sensorimotor cortex between contralateral and ipsilateral hand movements

Brain machine interfaces (BMIs) have emerged as a technology to restore lost functionality in motor impaired patients. Most BMI systems employed neural signals from contralateral hemisphere. But many studies have also demonstrated the possibility to control hand movement using signals from ipsilateral one. However, the relationship of neural signals in sensorimotor cortex between contralateral and ipsilateral hand movement control is still unclear. In this study, the electrocorticographic signals (ECoG) of sensorimotor cortex were analyzed in two epilepsy participants when they performed a visual guided rock-scissors-paper task by using contralateral and ipsilateral hand respectively. Although typical beta suppression followed increased gamma were observed during the movements of each individual hands, the stronger responses were found in two participants when their contralateral hands were used during the task. We further extracted the power spectrum of high gamma frequency band (70-135Hz) of ECoG signals as neural features to decode the hand movements. The results showed that the classification accuracy of contralateral decoding and ipsilateral decoding were 81% and 78% for participator one (P1) and 84% and 77% for participator two (P2). The accuracy of ipsilateral decoding was only slightly lower than that of contralateral one. The hand movement information contained in ipsilateral sensorimotor cortex suggested that the ipsilateral hemisphere might be also involved in neural modulation as well as contralateral hemisphere did when performing unimanual movement, which would expand the clinical application of BMIs.

Phase and magnitude spatiotemporal dynamics of β oscillation in electrocorticography (ECoG) in the monkey motor cortex at the onset of 3D reaching movements

β oscillations in local field potentials, electro-corticography (ECoG), and electroencephalograms (EEG) are ubiquitous in the motor cortex of monkeys and humans. However due to their lack of contributions, compared to other frequency ranges, to decode effector kinematics especially in ECoG signals, spatiotemporal dynamics of ECoG β oscillations has not been examined despite the larger areas that ECoG arrays can cover than standard intracortical multielectrode arrays. Here, we used ECoG grids to cover large areas of motor cortex and some somatosensory cortex in monkeys while they performed an unconstrained reaching and a lever pulling task at two force levels in three dimensional space. We showed that under the pulling task β power increased around movement onset. However, the β phases were locked around the movement onsets and their peak timings were spatially aligned in the motor cortex. These results may indicate that spatiotemporal dynamics of β oscillation conveys task relevant information and that ECoG arrays will be useful to study larger spatiotemporal patterns in the motor cortex, or any cortical areas in general, than intracortical multielectrode arrays.

Histological evaluation of a chronically-implanted electrocorticographic electrode grid in a non-human primate

OBJECTIVE

Electrocorticography (ECoG), used as a neural recording modality for brain-machine interfaces (BMIs), potentially allows for field potentials to be recorded from the surface of the cerebral cortex for long durations without suffering the host-tissue reaction to the extent that it is common with intracortical microelectrodes. Though the stability of signals obtained from chronically implanted ECoG electrodes has begun receiving attention, to date little work has characterized the effects of long-term implantation of ECoG electrodes on underlying cortical tissue.

APPROACH

We implanted and recorded from a high-density ECoG electrode grid subdurally over cortical motor areas of a Rhesus macaque for 666 d.

MAIN RESULTS

Histological analysis revealed minimal damage to the cortex underneath the implant, though the grid itself was encapsulated in collagenous tissue. We observed macrophages and foreign body giant cells at the tissue-array interface, indicative of a stereotypical foreign body response. Despite this encapsulation, cortical modulation during reaching movements was observed more than 18 months post-implantation.

SIGNIFICANCE

These results suggest that ECoG may provide a means by which stable chronic cortical recordings can be obtained with comparatively little tissue damage, facilitating the development of clinically viable BMI systems.

Single-trial prediction of reaction time variability from MEG brain activity

Neural activity prior to movement onset contains essential information for predictive assistance for humans using brain-machine-interfaces (BMIs). Even though previous studies successfully predicted different goals for upcoming movements, it is unclear whether non-invasive recording signals contain the information to predict trial-by-trial behavioral variability under the same movement. In this paper, we examined the predictability of subsequent short or long reaction times (RTs) from magnetoencephalography (MEG) signals in a delayed-reach task. The difference in RTs was classified significantly above chance from 550 ms before the go-signal onset using the cortical currents in the premotor cortex. Significantly above-chance classification was performed in the lateral prefrontal and the right inferior parietal cortices at the late stage of the delay period. Thus, inter-trial variability in RTs is predictable information. Our study provides a proof-of-concept of the future development of non-invasive BMIs to prevent delayed movements.

Penalized Multi-Way Partial Least Squares for Smooth Trajectory Decoding from Electrocorticographic (ECoG) Recording

In the current paper the decoding algorithms for motor-related BCI systems for continuous upper limb trajectory prediction are considered. Two methods for the smooth prediction, namely Sobolev and Polynomial Penalized Multi-Way Partial Least Squares (PLS) regressions, are proposed. The methods are compared to the Multi-Way Partial Least Squares and Kalman Filter approaches. The comparison demonstrated that the proposed methods combined the prediction accuracy of the algorithms of the PLS family and trajectory smoothness of the Kalman Filter. In addition, the prediction delay is significantly lower for the proposed algorithms than for the Kalman Filter approach. The proposed methods could be applied in a wide range of applications beyond neuroscience.

Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. We analyzed continuous, long-term electrocorticography (ECoG) data recorded over many days from the brain of subjects in a hospital room, with simultaneous audio and video recordings. We discovered coherent clusters in high-dimensional ECoG recordings using hierarchical clustering and automatically annotated them using speech and movement labels extracted from audio and video. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Interpretable behaviors were decoded from ECoG data, including moving, speaking and resting; the results were assessed by comparison with manual annotation. Discovered clusters were projected back onto the brain revealing features consistent with known functional areas, opening the door to automated functional brain mapping in natural settings.

Decoding three-dimensional reaching movements using electrocorticographic signals in humans

OBJECTIVE

Electrocorticography (ECoG) signals have emerged as a potential control signal for brain-computer interface (BCI) applications due to balancing signal quality and implant invasiveness. While there have been numerous demonstrations in which ECoG signals were used to decode motor movements and to develop BCI systems, the extent of information that can be decoded has been uncertain. Therefore, we sought to determine if ECoG signals could be used to decode kinematics (speed, velocity, and position) of arm movements in 3D space.

APPROACH

To investigate this, we designed a 3D center-out reaching task that was performed by five epileptic patients undergoing temporary placement of ECoG arrays. We used the ECoG signals within a hierarchical partial-least squares (PLS) regression model to perform offline prediction of hand speed, velocity, and position.

MAIN RESULTS

The hierarchical PLS regression model enabled us to predict hand speed, velocity, and position during 3D reaching movements from held-out test sets with accuracies above chance in each patient with mean correlation coefficients between 0.31 and 0.80 for speed, 0.27 and 0.54 for velocity, and 0.22 and 0.57 for position. While beta band power changes were the most significant features within the model used to classify movement and rest, the local motor potential and high gamma band power changes, were the most important features in the prediction of kinematic parameters.

SIGNIFICANCE

We believe that this study represents the first demonstration that truly three-dimensional movements can be predicted from ECoG recordings in human patients. Furthermore, this prediction underscores the potential to develop BCI systems with multiple degrees of freedom in human patients using ECoG.

Current Challenges Facing the Translation of Brain Computer Interfaces from Preclinical Trials to Use in Human Patients

Current research in brain computer interface (BCI) technology is advancing beyond preclinical studies, with trials beginning in human patients. To date, these trials have been carried out with several different types of recording interfaces. The success of these devices has varied widely, but different factors such as the level of invasiveness, timescale of recorded information, and ability to maintain stable functionality of the device over a long period of time all must be considered in addition to accuracy in decoding intent when assessing the most practical type of device moving forward. Here, we discuss various approaches to BCIs, distinguishing between devices focusing on control of operations extrinsic to the subject (e.g., prosthetic limbs, computer cursors) and those focusing on control of operations intrinsic to the brain (e.g., using stimulation or external feedback), including closed-loop or adaptive devices. In this discussion, we consider the current challenges facing the translation of various types of BCI technology to eventual human application.