Physiological properties of brain-machine interface input signals

Machine learning-based high-frequency neuronal spike reconstruction from low-frequency and low-sampling-rate recordings

Recording neuronal activity using multiple electrodes has been widely used to understand the functional mechanisms of the brain. Increasing the number of electrodes allows us to decode more variety of functionalities. However, handling massive amounts of multichannel electrophysiological data is still challenging due to the limited hardware resources and unavoidable thermal tissue damage. Here, we present machine learning (ML)-based reconstruction of high-frequency neuronal spikes from subsampled low-frequency band signals. Inspired by the equivalence between high-frequency restoration and super-resolution in image processing, we applied a transformer ML model to neuronal data recorded from both in vitro cultures and in vivo male mouse brains. Even with the x8 downsampled datasets, our trained model reasonably estimated high-frequency information of spiking activity, including spike timing, waveform, and network connectivity. With our ML-based data reduction applicable to existing multichannel recording hardware while achieving neuronal signals of broad bandwidths, we expect to enable more comprehensive analysis and control of brain functions.

Nine decades of electrocorticography: A comparison between epidural and subdural recordings

In recent years, electrocorticography (ECoG) has arisen as a neural signal recording tool in the development of clinically viable neural interfaces. ECoG electrodes are generally placed below the dura mater (subdural) but can also be placed on top of the dura (epidural). In deciding which of these modalities best suits long-term implants, complications and signal quality are important considerations. Conceptually, epidural placement may present a lower risk of complications as the dura is left intact but also a lower signal quality due to the dura acting as a signal attenuator. The extent to which complications and signal quality are affected by the dura, however, has been a matter of debate. To improve our understanding of the effects of the dura on complications and signal quality, we conducted a literature review. We inventorized the effect of the dura on signal quality, decodability and longevity of acute and chronic ECoG recordings in humans and non-human primates. Also, we compared the incidence and nature of serious complications in studies that employed epidural and subdural ECoG. Overall, we found that, even though epidural recordings exhibit attenuated signal amplitude over subdural recordings, particularly for high-density grids, the decodability of epidural recorded signals does not seem to be markedly affected. Additionally, we found that the nature of serious complications was comparable between epidural and subdural recordings. These results indicate that both epidural and subdural ECoG may be suited for long-term neural signal recordings, at least for current generations of clinical and high-density ECoG grids.

Long-term unsupervised recalibration of cursor BCIs

Intracortical brain-computer interfaces (iBCIs) require frequent recalibration to maintain robust performance due to changes in neural activity that accumulate over time. Compensating for this nonstationarity would enable consistently high performance without the need for supervised recalibration periods, where users cannot engage in free use of their device. Here we introduce a hidden Markov model (HMM) to infer what targets users are moving toward during iBCI use. We then retrain the system using these inferred targets, enabling unsupervised adaptation to changing neural activity. Our approach outperforms the state of the art in large-scale, closed-loop simulations over two months and in closed-loop with a human iBCI user over one month. Leveraging an offline dataset spanning five years of iBCI recordings, we further show how recently proposed data distribution-matching approaches to recalibration fail over long time scales; only target-inference methods appear capable of enabling long-term unsupervised recalibration. Our results demonstrate how task structure can be used to bootstrap a noisy decoder into a highly-performant one, thereby overcoming one of the major barriers to clinically translating BCIs.

Potential therapeutic mechanism of deep brain stimulation of the nucleus accumbens in obsessive-compulsive disorder

Deep brain stimulation (DBS) of the nucleus accumbens (NAc) (NAc-DBS) is an effective solution to refractory obsessive-compulsive disorder (OCD). However, evidence for the neurobiological mechanisms of OCD and the effect of NAc-DBS is still lacking. One hypothesis is that the electrophysiological activities in the NAc are modulated by DBS, and another hypothesis is that the activities of neurotransmitters in the NAc are influenced by DBS. To investigate these potential alterations, rats with quinpirole (QNP)- induced OCD were treated with DBS of the core part of NAc. Then, extracellular spikes (SPK) and local field potentials (LFP) in the NAc were recorded, and the levels of relevant neurotransmitters and related proteins were measured. Analysis of SPK revealed that the firing rate was decreased and the firing pattern was changed after NAc-DBS, and analysis of LFP showed that overall power spectral density (PSD) levels were reduced after NAc-DBS. Additionally, we found that the relative powers of the theta band, alpha band and beta band were increased in OCD status, while the relative powers of the delta band and gamma band were decreased. This pathological pattern of power distribution was reformed by NAc-DBS. Furthermore, we found that the local levels of monoamines [dopamine (DA) and serotonin (5-HT)] and amino acids [glutamate (Glu) and gamma-aminobutyric acid (GABA)] in the NAc were increased in OCD status, and that the expression of the two types of DA receptors in the NAc exhibited an opposite change. These abnormalities could be reversed by NAc-DBS. These findings provide a more comprehensive understanding about the function of the NAc in the pathophysiology of OCD and provide more detailed evidence for the potential effect of NAc-DBS.

A Review of Brain Activity and EEG-Based Brain-Computer Interfaces for Rehabilitation Application

Patients with severe CNS injuries struggle primarily with their sensorimotor function and communication with the outside world. There is an urgent need for advanced neural rehabilitation and intelligent interaction technology to provide help for patients with nerve injuries. Recent studies have established the brain-computer interface (BCI) in order to provide patients with appropriate interaction methods or more intelligent rehabilitation training. This paper reviews the most recent research on brain-computer-interface-based non-invasive rehabilitation systems. Various endogenous and exogenous methods, advantages, limitations, and challenges are discussed and proposed. In addition, the paper discusses the communication between the various brain-computer interface modes used between severely paralyzed and locked patients and the surrounding environment, particularly the brain-computer interaction system utilizing exogenous (induced) EEG signals (such as P300 and SSVEP). This discussion reveals with an examination of the interface for collecting EEG signals, EEG components, and signal postprocessing. Furthermore, the paper describes the development of natural interaction strategies, with a focus on signal acquisition, data processing, pattern recognition algorithms, and control techniques.

Noninvasively recorded high-gamma signals improve synchrony of force feedback in a novel neurorehabilitation brain-machine interface for brain injury

OBJECTIVE

Brain injury is the leading cause of long-term disability worldwide, often resulting in impaired hand function. Brain-machine interfaces (BMIs) offer a potential way to improve hand function. BMIs often target replacing lost function, but may also be employed in neurorehabilitation (nrBMI) by facilitating neural plasticity and functional recovery. Here, we report a novel nrBMI capable of acquiring high-γ (70-115 Hz) information through a unique post-TBI hemicraniectomy window model, and delivering sensory feedback that is synchronized with, and proportional to, intended grasp force.

APPROACH

We developed the nrBMI to use electroencephalogram recorded over a hemicraniectomy (hEEG) in individuals with traumatic brain injury (TBI). The nrBMI empowered users to exert continuous, proportional control of applied force, and provided continuous force feedback. We report the results of an initial testing group of three human participants with TBI, along with a control group of three skull- and motor-intact volunteers.

MAIN RESULTS

All participants controlled the nrBMI successfully, with high initial success rates (2 of 6 participants) or performance that improved over time (4 of 6 participants). We observed high-γ modulation with force intent in hEEG but not skull-intact EEG. Most significantly, we found that high-γ control significantly improved the timing synchronization between neural modulation onset and nrBMI output/haptic feedback (compared to low-frequency nrBMI control).

SIGNIFICANCE

These proof-of-concept results show that high-γ nrBMIs can be used by individuals with impaired ability to control force (without immediately resorting to invasive signals like ECoG). Of note, the nrBMI includes a parameter to change the fraction of control shared between decoded intent and volitional force, to adjust for recovery progress. The improved synchrony between neural modulations and force control for high-γ signals is potentially important for maximizing the ability of nrBMIs to induce plasticity in neural circuits. Inducing plasticity is critical to functional recovery after brain injury.

The science and engineering behind sensitized brain-controlled bionic hands

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

A machine learning approach to characterize sequential movement-related states in premotor and motor cortices

Non-human primate (NHP) movement kinematics have been decoded from spikes and local field potentials (LFPs) recorded during motor tasks. However, the potential of LFPs to provide network-like characterizations of neural dynamics during planning and execution of sequential movements requires further exploration. Is the aggregate nature of LFPs suitable to construct informative brain state descriptors of movement preparation and execution? To investigate this, we developed a framework to process LFPs based on machine learning classifiers and analyzed LFP from a primate, implanted with several microelectrode arrays covering the premotor cortex in both hemispheres and the primary motor cortex on one side. The primate performed a reach and grasp task, consisting of five consecutive states, starting from rest until a rewarding target (food) was attained. We use this five-state task to characterize brain activity and connectivity within eight frequency bands, using spectral amplitude and pair-wise correlations across electrodes as features. Our results show that we could best distinguish all five movement-related states using the highest frequency band (200-500Hz), yielding an 87% accuracy with spectral amplitude, and 60% with pair-wise electrode correlation. Further analyses characterized each movement-related state, showing differential neuronal population activity at above-gamma frequencies during the various stages of movement. In summary, our results show that the concerted use of novel machine-learning techniques with coarse grained queue broad signals such as LFPs may be successfully used to track and decode fine reach and grasp movement aspects across several brain regions.

Brain-Computer Interfaces in Neurorecovery and Neurorehabilitation

Recent advances in brain-computer interface technology to restore and rehabilitate neurologic function aim to enable persons with disabling neurologic conditions to communicate, interact with the environment, and achieve other key activities of daily living and personal goals. Here we evaluate the principles, benefits, challenges, and future directions of brain-computer interfaces in the context of neurorehabilitation. We then explore the clinical translation of these technologies and propose an approach to facilitate implementation of brain-computer interfaces for persons with neurologic disease.

Robust and accurate decoding of hand kinematics from entire spiking activity using deep learning

OBJECTIVE

Brain-machine interfaces (BMIs) seek to restore lost motor functions in individuals with neurological disorders by enabling them to control external devices directly with their thoughts. This work aims to improve robustness and decoding accuracy that currently become major challenges in the clinical translation of intracortical BMIs.

APPROACH

We propose entire spiking activity (ESA) -an envelope of spiking activity that can be extracted by a simple, threshold-less, and automated technique- as the input signal. We couple ESA with deep learning-based decoding algorithm that uses quasi-recurrent neural network (QRNN) architecture. We evaluate comprehensively the performance of ESA-driven QRNN decoder for decoding hand kinematics from neural signals chronically recorded from the primary motor cortex area of three non-human primates performing different tasks.

MAIN RESULTS

Our proposed method yields consistently higher decoding performance than any other combinations of the input signal and decoding algorithm previously reported across long term recording sessions. It can sustain high decoding performance even when removing spikes from the raw signals, when using the different number of channels, and when using a smaller amount of training data.

SIGNIFICANCE

Overall results demonstrate exceptionally high decoding accuracy and chronic robustness, which is highly desirable given it is an unresolved challenge in BMIs.

The Representation of Finger Movement and Force in Human Motor and Premotor Cortices

The ability to grasp and manipulate objects requires controlling both finger movement kinematics and isometric force in rapid succession. Previous work suggests that these behavioral modes are controlled separately, but it is unknown whether the cerebral cortex represents them differently. Here, we asked the question of how movement and force were represented cortically, when executed sequentially with the same finger. We recorded high-density electrocorticography (ECoG) from the motor and premotor cortices of seven human subjects performing a movement-force motor task. We decoded finger movement [0.7 ± 0.3 fractional variance accounted for (FVAF)] and force (0.7 ± 0.2 FVAF) with high accuracy, yet found different spatial representations. In addition, we used a state-of-the-art deep learning method to uncover smooth, repeatable trajectories through ECoG state space during the movement-force task. We also summarized ECoG across trials and participants by developing a new metric, the neural vector angle (NVA). Thus, state-space techniques can help to investigate broad cortical networks. Finally, we were able to classify the behavioral mode from neural signals with high accuracy (90 ± 6%). Thus, finger movement and force appear to have distinct representations in motor/premotor cortices. These results inform our understanding of the neural control of movement, as well as the design of grasp brain-machine interfaces (BMIs).

The combination of brain-computer interfaces and artificial intelligence: applications and challenges

Brain-computer interfaces (BCIs) have shown great prospects as real-time bidirectional links between living brains and actuators. Artificial intelligence (AI), which can advance the analysis and decoding of neural activity, has turbocharged the field of BCIs. Over the past decade, a wide range of BCI applications with AI assistance have emerged. These "smart" BCIs including motor and sensory BCIs have shown notable clinical success, improved the quality of paralyzed patients' lives, expanded the athletic ability of common people and accelerated the evolution of robots and neurophysiological discoveries. However, despite technological improvements, challenges remain with regard to the long training periods, real-time feedback, and monitoring of BCIs. In this article, the authors review the current state of AI as applied to BCIs and describe advances in BCI applications, their challenges and where they could be headed in the future.

Electrocorticogram (ECoG) Is Highly Informative in Primate Visual Cortex

Neural signals recorded at different scales contain information about environment and behavior and have been used to control Brain Machine Interfaces with varying degrees of success. However, a direct comparison of their efficacy has not been possible due to different recording setups, tasks, species, etc. To address this, we implanted customized arrays having both microelectrodes and electrocorticogram (ECoG) electrodes in the primary visual cortex of 2 female macaque monkeys, and also recorded electroencephalogram (EEG), while they viewed a variety of naturalistic images and parametric gratings. Surprisingly, ECoG had higher information and decodability than all other signals. Combining a few ECoG electrodes allowed more accurate decoding than combining a much larger number of microelectrodes. Control analyses showed that higher decoding accuracy of ECoG compared with local field potential was not because of differences in low-level visual features captured by them but instead because of larger spatial summation of the ECoG. Information was high in the 30-80 Hz range and at lower frequencies. Information in different frequencies and scales was nonredundant. These results have strong implications for Brain Machine Interface applications and for study of population representation of visual stimuli.SIGNIFICANCE STATEMENT Electrophysiological signals captured across scales by different recording electrodes are regularly used for Brain Machine Interfaces, but the information content varies due to electrode size and location. A systematic comparison of their efficiency for Brain Machine Interfaces is important but technically challenging. Here, we recorded simultaneous signals across four scales: spikes, local field potential, electrocorticogram (ECoG), and EEG, and compared their information and decoding accuracy for a large variety of naturalistic stimuli. We found that ECoGs were highly informative and outperformed other signals in information content and decoding accuracy.

Visual epidural field potentials possess high functional specificity in single trials

Recordings of epidural field potentials (EFPs) allow neuronal activity to be acquired over a large region of cortical tissue with minimal invasiveness. Because electrodes are placed on top of the dura and do not enter the neuronal tissue, EFPs offer intriguing options for both clinical and basic science research. On the other hand, EFPs represent the integrated activity of larger neuronal populations and possess a higher trial-by-trial variability and a reduced signal-to-noise ratio due the additional barrier of the dura. It is thus unclear whether and to what extent EFPs have sufficient spatial selectivity to allow for conclusions about the underlying functional cortical architecture, and whether single EFP trials provide enough information on the short timescales relevant for many clinical and basic neuroscience purposes. We used the high spatial resolution of primary visual cortex to address these issues and investigated the extent to which very short EFP traces allow reliable decoding of spatial information. We briefly presented different visual objects at one of nine closely adjacent locations and recorded neuronal activity with a high-density epidural multielectrode array in three macaque monkeys. With the use of receiver operating characteristics (ROC) to identify the most informative data, machine-learning algorithms provided close-to-perfect classification rates for all 27 stimulus conditions. A binary classifier applying a simple max function on ROC-selected data further showed that single trials might be classified with 100% performance even without advanced offline classifiers. Thus, although highly variable, EFPs constitute an extremely valuable source of information and offer new perspectives for minimally invasive recording of large-scale networks.NEW & NOTEWORTHY Epidural field potential (EFP) recordings provide a minimally invasive approach to investigate large-scale neural networks, but little is known about whether they possess the required specificity for basic and clinical neuroscience. By making use of the spatial selectivity of primary visual cortex, we show that single-trial information can be decoded with close-to-perfect performance, even without using advanced classifiers and based on very few data. This labels EFPs as a highly attractive and widely usable signal.

Speech synthesis from ECoG using densely connected 3D convolutional neural networks

OBJECTIVE

Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech.

APPROACH

Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is well-suited to work with the small amount of data available from each participant.

MAIN RESULTS

In a study with six participants, we achieved correlations up to r  =  0.69 between the reconstructed and original logMel spectrograms. We transfered our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel features that harnessed a much larger, pre-existing data corpus to provide the most natural acoustic output.

SIGNIFICANCE

To the best of our knowledge, this is the first time that high-quality speech has been reconstructed from neural recordings during speech production using deep neural networks.

Hemicraniectomy in Traumatic Brain Injury: A Noninvasive Platform to Investigate High Gamma Activity for Brain Machine Interfaces

Brain-machine interfaces (BMIs) translate brain signals into control signals for an external device, such as a computer cursor or robotic limb. These signals can be obtained either noninvasively or invasively. Invasive recordings, using electrocorticography (ECoG) or intracortical microelectrodes, provide higher bandwidth and more informative signals. Rehabilitative BMIs, which aim to drive plasticity in the brain to enhance recovery after brain injury, have almost exclusively used non-invasive recordings, such electroencephalography (EEG) or magnetoencephalography (MEG), which have limited bandwidth and information content. Invasive recordings provide more information and spatiotemporal resolution, but do incur risk, and thus are not usually investigated in people with stroke or traumatic brain injury (TBI). Here, in this paper, we describe a new BMI paradigm to investigate the use of higher frequency signals in brain-injured subjects without incurring significant risk. We recorded EEG in TBI subjects who required hemicraniectomies (removal of a part of the skull). EEG over the hemicraniectomy (hEEG) contained substantial information in the high gamma frequency range (65-115 Hz). Using this information, we decoded continuous finger flexion force with moderate to high accuracy (variance accounted for 0.06 to 0.52), which at best approaches that using epidural signals. These results indicate that people with hemicraniectomies can provide a useful resource for developing BMI therapies for the treatment of brain injury.

Brain-Machine Interfaces: Powerful Tools for Clinical Treatment and Neuroscientific Investigations

Brain-machine interfaces (BMIs) have exploded in popularity in the past decade. BMIs, also called brain-computer interfaces, provide a direct link between the brain and a computer, usually to control an external device. BMIs have a wide array of potential clinical applications, ranging from restoring communication to people unable to speak due to amyotrophic lateral sclerosis or a stroke, to restoring movement to people with paralysis from spinal cord injury or motor neuron disease, to restoring memory to people with cognitive impairment. Because BMIs are controlled directly by the activity of prespecified neurons or cortical areas, they also provide a powerful paradigm with which to investigate fundamental questions about brain physiology, including neuronal behavior, learning, and the role of oscillations. This article reviews the clinical and neuroscientific applications of BMIs, with a primary focus on motor BMIs.

Cortical Brain Stimulation with Endovascular Electrodes

Electrical stimulation of neural tissue and recording of neural activity are the bases of emerging prostheses and treatments for spinal cord injury, stroke, sensory deficits, and drug-resistant neurological disorders. Safety and efficacy are key aspects for the clinical acceptance of therapeutic neural stimulators. The cortical vasculature has been shown to be a safe site for implantation of electrodes for chronically recording neural activity, requiring no craniotomy to access high-bandwidth, intracranial EEG. This work presents the first characterization of endovascular cortical stimulation measured using cortical subdural surface recordings. Visual stimulation was used to verify electrode viability and cortical activation was compared with electrically evoked activity. Due to direct activation of the neural tissue, the latency of responses to electrical stimulation was shorter than for that of visual stimulation. We also found that the center of neural activation was different for visual and electrical stimulation indicating an ability of the stentrode to provide localized activation of neural tissue.

A New Frontier: The Convergence of Nanotechnology, Brain Machine Interfaces, and Artificial Intelligence

A confluence of technological capabilities is creating an opportunity for machine learning and artificial intelligence (AI) to enable "smart" nanoengineered brain machine interfaces (BMI). This new generation of technologies will be able to communicate with the brain in ways that support contextual learning and adaptation to changing functional requirements. This applies to both invasive technologies aimed at restoring neurological function, as in the case of neural prosthesis, as well as non-invasive technologies enabled by signals such as electroencephalograph (EEG). Advances in computation, hardware, and algorithms that learn and adapt in a contextually dependent way will be able to leverage the capabilities that nanoengineering offers the design and functionality of BMI. We explore the enabling capabilities that these devices may exhibit, why they matter, and the state of the technologies necessary to build them. We also discuss a number of open technical challenges and problems that will need to be solved in order to achieve this.

On-Probe Neural Interface ASIC for Combined Electrical Recording and Optogenetic Stimulation

Neuromodulation technologies are progressing from pacemaking and sensory operations to full closed-loop control. In particular, optogenetics-the genetic modification of light sensitivity into neural tissue allows for simultaneous optical stimulation and electronic recording. This paper presents a neural interface application-specified integrated circuit (ASIC) for intelligent optoelectronic probes. The architecture is designed to enable simultaneous optical neural stimulation and electronic recording. It provides four low noise (2.08  μV) recording channels optimized for recording local field potentials (LFPs) (0.1-300 Hz bandwidth, 5 mV range, sampled 10-bit@4 kHz), which are more stable for chronic applications. For stimulation, it provides six independently addressable optical driver circuits, which can provide both intensity (8-bit resolution across a 1.1 mA range) and pulse-width modulation for high-radiance light emitting diodes (LEDs). The system includes a fully digital interface using a serial peripheral interface (SPI) protocol to allow for use with embedded controllers. The SPI interface is embedded within a finite state machine (FSM), which implements a command interpreter that can send out LFP data whilst receiving instructions to control LED emission. The circuit has been implemented in a commercially available 0.35  μm CMOS technology occupying a 1.95 mm 1.10 mm footprint for mounting onto the head of a silicon probe. Measured results are given for a variety of bench-top, in vitro and in vivo experiments, quantifying system performance and also demonstrating concurrent recording and stimulation within relevant experimental models.