Deep learning with convolutional neural networks for EEG decoding and visualization

Deep learning with convolutional neural networks (deep ConvNets) has revolutionized computer vision through end-to-end learning, that is, learning from the raw data. There is increasing interest in using deep ConvNets for end-to-end EEG analysis, but a better understanding of how to design and train ConvNets for end-to-end EEG decoding and how to visualize the informative EEG features the ConvNets learn is still needed. Here, we studied deep ConvNets with a range of different architectures, designed for decoding imagined or executed tasks from raw EEG. Our results show that recent advances from the machine learning field, including batch normalization and exponential linear units, together with a cropped training strategy, boosted the deep ConvNets decoding performance, reaching at least as good performance as the widely used filter bank common spatial patterns (FBCSP) algorithm (mean decoding accuracies 82.1% FBCSP, 84.0% deep ConvNets). While FBCSP is designed to use spectral power modulations, the features used by ConvNets are not fixed a priori. Our novel methods for visualizing the learned features demonstrated that ConvNets indeed learned to use spectral power modulations in the alpha, beta, and high gamma frequencies, and proved useful for spatially mapping the learned features by revealing the topography of the causal contributions of features in different frequency bands to the decoding decision. Our study thus shows how to design and train ConvNets to decode task-related information from the raw EEG without handcrafted features and highlights the potential of deep ConvNets combined with advanced visualization techniques for EEG-based brain mapping. Hum Brain Mapp 38:5391-5420, 2017. © 2017 Wiley Periodicals, Inc.

Filter Bank Common Spatial Pattern Algorithm on BCI Competition IV Datasets 2a and 2b

The Common Spatial Pattern (CSP) algorithm is an effective and popular method for classifying 2-class motor imagery electroencephalogram (EEG) data, but its effectiveness depends on the subject-specific frequency band. This paper presents the Filter Bank Common Spatial Pattern (FBCSP) algorithm to optimize the subject-specific frequency band for CSP on Datasets 2a and 2b of the Brain-Computer Interface (BCI) Competition IV. Dataset 2a comprised 4 classes of 22 channels EEG data from 9 subjects, and Dataset 2b comprised 2 classes of 3 bipolar channels EEG data from 9 subjects. Multi-class extensions to FBCSP are also presented to handle the 4-class EEG data in Dataset 2a, namely, Divide-and-Conquer (DC), Pair-Wise (PW), and One-Versus-Rest (OVR) approaches. Two feature selection algorithms are also presented to select discriminative CSP features on Dataset 2b, namely, the Mutual Information-based Best Individual Feature (MIBIF) algorithm, and the Mutual Information-based Rough Set Reduction (MIRSR) algorithm. The single-trial classification accuracies were presented using 10 × 10-fold cross-validations on the training data and session-to-session transfer on the evaluation data from both datasets. Disclosure of the test data labels after the BCI Competition IV showed that the FBCSP algorithm performed relatively the best among the other submitted algorithms and yielded a mean kappa value of 0.569 and 0.600 across all subjects in Datasets 2a and 2b respectively.

Review of the BCI Competition IV

The BCI competition IV stands in the tradition of prior BCI competitions that aim to provide high quality neuroscientific data for open access to the scientific community. As experienced already in prior competitions not only scientists from the narrow field of BCI compete, but scholars with a broad variety of backgrounds and nationalities. They include high specialists as well as students. The goals of all BCI competitions have always been to challenge with respect to novel paradigms and complex data. We report on the following challenges: (1) asynchronous data, (2) synthetic, (3) multi-class continuous data, (4) session-to-session transfer, (5) directionally modulated MEG, (6) finger movements recorded by ECoG. As after past competitions, our hope is that winning entries may enhance the analysis methods of future BCIs.

Toward enhanced P300 speller performance

This study examines the effects of expanding the classical P300 feature space on the classification performance of data collected from a P300 speller paradigm [Farwell LA, Donchin E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroenceph Clin Neurophysiol 1988;70:510-23]. Using stepwise linear discriminant analysis (SWLDA) to construct a classifier, the effects of spatial channel selection, channel referencing, data decimation, and maximum number of model features are compared with the intent of establishing a baseline not only for the SWLDA classifier, but for related P300 speller classification methods in general. By supplementing the classical P300 recording locations with posterior locations, online classification performance of P300 speller responses can be significantly improved using SWLDA and the favorable parameters derived from the offline comparative analysis.

A P300-based brain-computer interface for people with amyotrophic lateral sclerosis

OBJECTIVE

The current study evaluates the efficacy of a P300-based brain-computer interface (BCI) communication device for individuals with advanced ALS.

METHODS

Participants attended to one cell of a N x N matrix while the N rows and N columns flashed randomly. Each cell of the matrix contained one character. Every flash of an attended character served as a rare event in an oddball sequence and elicited a P300 response. Classification coefficients derived using a stepwise linear discriminant function were applied to the data after each set of flashes. The character receiving the highest discriminant score was presented as feedback.

RESULTS

In Phase I, six participants used a 6 x 6 matrix on 12 separate days with a mean rate of 1.2 selections/min and mean online and offline accuracies of 62% and 82%, respectively. In Phase II, four participants used either a 6 x 6 or a 7 x 7 matrix to produce novel and spontaneous statements with a mean online rate of 2.1 selections/min and online accuracy of 79%. The amplitude and latency of the P300 remained stable over 40 weeks.

CONCLUSIONS

Participants could communicate with the P300-based BCI and performance was stable over many months.

SIGNIFICANCE

BCIs could provide an alternative communication and control technology in the daily lives of people severely disabled by ALS.

Think to move: a neuromagnetic brain-computer interface (BCI) system for chronic stroke

BACKGROUND AND PURPOSE

Stroke is a leading cause of long-term motor disability among adults. Present rehabilitative interventions are largely unsuccessful in improving the most severe cases of motor impairment, particularly in relation to hand function. Here we tested the hypothesis that patients experiencing hand plegia as a result of a single, unilateral subcortical, cortical or mixed stroke occurring at least 1 year previously, could be trained to operate a mechanical hand orthosis through a brain-computer interface (BCI).

METHODS

Eight patients with chronic hand plegia resulting from stroke (residual finger extension function rated on the Medical Research Council scale=0/5) were recruited from the Stroke Neurorehabilitation Clinic, Human Cortical Physiology Section of the National Institute for Neurological Disorders and Stroke (NINDS) (n=5) and the Clinic of Neurology of the University of Tübingen (n=3). Diagnostic MRIs revealed single, unilateral subcortical, cortical or mixed lesions in all patients. A magnetoencephalography-based BCI system was used for this study. Patients participated in between 13 to 22 training sessions geared to volitionally modulate micro rhythm amplitude originating in sensorimotor areas of the cortex, which in turn raised or lowered a screen cursor in the direction of a target displayed on the screen through the BCI interface. Performance feedback was provided visually in real-time. Successful trials (in which the cursor made contact with the target) resulted in opening/closing of an orthosis attached to the paralyzed hand.

RESULTS

Training resulted in successful BCI control in 6 of 8 patients. This control was associated with increased range and specificity of mu rhythm modulation as recorded from sensors overlying central ipsilesional (4 patients) or contralesional (2 patients) regions of the array. Clinical scales used to rate hand function showed no significant improvement after training.

CONCLUSIONS

These results suggest that volitional control of neuromagnetic activity features recorded over central scalp regions can be achieved with BCI training after stroke, and used to control grasping actions through a mechanical hand orthosis.

fNIRS-based brain-computer interfaces: a review

A brain-computer interface (BCI) is a communication system that allows the use of brain activity to control computers or other external devices. It can, by bypassing the peripheral nervous system, provide a means of communication for people suffering from severe motor disabilities or in a persistent vegetative state. In this paper, brain-signal generation tasks, noise removal methods, feature extraction/selection schemes, and classification techniques for fNIRS-based BCI are reviewed. The most common brain areas for fNIRS BCI are the primary motor cortex and the prefrontal cortex. In relation to the motor cortex, motor imagery tasks were preferred to motor execution tasks since possible proprioceptive feedback could be avoided. In relation to the prefrontal cortex, fNIRS showed a significant advantage due to no hair in detecting the cognitive tasks like mental arithmetic, music imagery, emotion induction, etc. In removing physiological noise in fNIRS data, band-pass filtering was mostly used. However, more advanced techniques like adaptive filtering, independent component analysis (ICA), multi optodes arrangement, etc. are being pursued to overcome the problem that a band-pass filter cannot be used when both brain and physiological signals occur within a close band. In extracting features related to the desired brain signal, the mean, variance, peak value, slope, skewness, and kurtosis of the noised-removed hemodynamic response were used. For classification, the linear discriminant analysis method provided simple but good performance among others: support vector machine (SVM), hidden Markov model (HMM), artificial neural network, etc. fNIRS will be more widely used to monitor the occurrence of neuro-plasticity after neuro-rehabilitation and neuro-stimulation. Technical breakthroughs in the future are expected via bundled-type probes, hybrid EEG-fNIRS BCI, and through the detection of initial dips.

A novel P300-based brain-computer interface stimulus presentation paradigm: moving beyond rows and columns

OBJECTIVE

An electroencephalographic brain-computer interface (BCI) can provide a non-muscular means of communication for people with amyotrophic lateral sclerosis (ALS) or other neuromuscular disorders. We present a novel P300-based BCI stimulus presentation - the checkerboard paradigm (CBP). CBP performance is compared to that of the standard row/column paradigm (RCP) introduced by Farwell and Donchin (1988).

METHODS

Using an 8x9 matrix of alphanumeric characters and keyboard commands, 18 participants used the CBP and RCP in counter-balanced fashion. With approximately 9-12 min of calibration data, we used a stepwise linear discriminant analysis for online classification of subsequent data.

RESULTS

Mean online accuracy was significantly higher for the CBP, 92%, than for the RCP, 77%. Correcting for extra selections due to errors, mean bit rate was also significantly higher for the CBP, 23 bits/min, than for the RCP, 17 bits/min. Moreover, the two paradigms produced significantly different waveforms. Initial tests with three advanced ALS participants produced similar results. Furthermore, these individuals preferred the CBP to the RCP.

CONCLUSIONS

These results suggest that the CBP is markedly superior to the RCP in performance and user acceptability.

SIGNIFICANCE

The CBP has the potential to provide a substantially more effective BCI than the RCP. This is especially important for people with severe neuromuscular disabilities.

A Randomized Controlled Trial of EEG-Based Motor Imagery Brain-Computer Interface Robotic Rehabilitation for Stroke

Electroencephalography (EEG)-based motor imagery (MI) brain-computer interface (BCI) technology has the potential to restore motor function by inducing activity-dependent brain plasticity. The purpose of this study was to investigate the efficacy of an EEG-based MI BCI system coupled with MIT-Manus shoulder-elbow robotic feedback (BCI-Manus) for subjects with chronic stroke with upper-limb hemiparesis. In this single-blind, randomized trial, 26 hemiplegic subjects (Fugl-Meyer Assessment of Motor Recovery After Stroke [FMMA] score, 4-40; 16 men; mean age, 51.4 years; mean stroke duration, 297.4 days), prescreened with the ability to use the MI BCI, were randomly allocated to BCI-Manus or Manus therapy, lasting 18 hours over 4 weeks. Efficacy was measured using upper-extremity FMMA scores at weeks 0, 2, 4 and 12. ElEG data from subjects allocated to BCI-Manus were quantified using the revised brain symmetry index (rBSI) and analyzed for correlation with the improvements in FMMA score. Eleven and 15 subjects underwent BCI-Manus and Manus therapy, respectively. One subject in the Manus group dropped out. Mean total FMMA scores at weeks 0, 2, 4, and 12 weeks improved for both groups: 26.3±10.3, 27.4±12.0, 30.8±13.8, and 31.5±13.5 for BCI-Manus and 26.6±18.9, 29.9±20.6, 32.9±21.4, and 33.9±20.2 for Manus, with no intergroup differences (P=.51). More subjects attained further gains in FMMA scores at week 12 from BCI-Manus (7 of 11 [63.6%]) than Manus (5 of 14 [35.7%]). A negative correlation was found between the rBSI and FMMA score improvement (P=.044). BCI-Manus therapy was well tolerated and not associated with adverse events. In conclusion, BCI-Manus therapy is effective and safe for arm rehabilitation after severe poststroke hemiparesis. Motor gains were comparable to those attained with intensive robotic therapy (1,040 repetitions/session) despite reduced arm exercise repetitions using EEG-based MI-triggered robotic feedback (136 repetitions/session). The correlation of rBSI with motor improvements suggests that the rBSI can be used as a prognostic measure for BCI-based stroke rehabilitation.

A high-speed brain speller using steady-state visual evoked potentials

Implementing a complex spelling program using a steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) remains a challenge due to difficulties in stimulus presentation and target identification. This study aims to explore the feasibility of mixed frequency and phase coding in building a high-speed SSVEP speller with a computer monitor. A frequency and phase approximation approach was developed to eliminate the limitation of the number of targets caused by the monitor refresh rate, resulting in a speller comprising 32 flickers specified by eight frequencies (8-15 Hz with a 1 Hz interval) and four phases (0°, 90°, 180°, and 270°). A multi-channel approach incorporating Canonical Correlation Analysis (CCA) and SSVEP training data was proposed for target identification. In a simulated online experiment, at a spelling rate of 40 characters per minute, the system obtained an averaged information transfer rate (ITR) of 166.91 bits/min across 13 subjects with a maximum individual ITR of 192.26 bits/min, the highest ITR ever reported in electroencephalogram (EEG)-based BCIs. The results of this study demonstrate great potential of a high-speed SSVEP-based BCI in real-life applications.

EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy

Background

Electroencephalography (EEG)-based brain-computer interface (BCI) systems are mainly divided into three major paradigms: motor imagery (MI), event-related potential (ERP), and steady-state visually evoked potential (SSVEP). Here, we present a BCI dataset that includes the three major BCI paradigms with a large number of subjects over multiple sessions. In addition, information about the psychological and physiological conditions of BCI users was obtained using a questionnaire, and task-unrelated parameters such as resting state, artifacts, and electromyography of both arms were also recorded. We evaluated the decoding accuracies for the individual paradigms and determined performance variations across both subjects and sessions. Furthermore, we looked for more general, severe cases of BCI illiteracy than have been previously reported in the literature.

Results

Average decoding accuracies across all subjects and sessions were 71.1% (± 0.15), 96.7% (± 0.05), and 95.1% (± 0.09), and rates of BCI illiteracy were 53.7%, 11.1%, and 10.2% for MI, ERP, and SSVEP, respectively. Compared to the ERP and SSVEP paradigms, the MI paradigm exhibited large performance variations between both subjects and sessions. Furthermore, we found that 27.8% (15 out of 54) of users were universally BCI literate, i.e., they were able to proficiently perform all three paradigms. Interestingly, we found no universally illiterate BCI user, i.e., all participants were able to control at least one type of BCI system.

Conclusions

Our EEG dataset can be utilized for a wide range of BCI-related research questions. All methods for the data analysis in this study are supported with fully open-source scripts that can aid in every step of BCI technology. Furthermore, our results support previous but disjointed findings on the phenomenon of BCI illiteracy.

An MEG-based brain-computer interface (BCI)

Brain-computer interfaces (BCIs) allow for communicating intentions by mere brain activity, not involving muscles. Thus, BCIs may offer patients who have lost all voluntary muscle control the only possible way to communicate. Many recent studies have demonstrated that BCIs based on electroencephalography (EEG) can allow healthy and severely paralyzed individuals to communicate. While this approach is safe and inexpensive, communication is slow. Magnetoencephalography (MEG) provides signals with higher spatiotemporal resolution than EEG and could thus be used to explore whether these improved signal properties translate into increased BCI communication speed. In this study, we investigated the utility of an MEG-based BCI that uses voluntary amplitude modulation of sensorimotor mu and beta rhythms. To increase the signal-to-noise ratio, we present a simple spatial filtering method that takes the geometric properties of signal propagation in MEG into account, and we present methods that can process artifacts specifically encountered in an MEG-based BCI. Exemplarily, six participants were successfully trained to communicate binary decisions by imagery of limb movements using a feedback paradigm. Participants achieved significant mu rhythm self control within 32 min of feedback training. For a subgroup of three participants, we localized the origin of the amplitude modulated signal to the motor cortex. Our results suggest that an MEG-based BCI is feasible and efficient in terms of user training.

Restoring the sense of touch with a prosthetic hand through a brain interface

Our ability to manipulate objects dexterously relies fundamentally on sensory signals originating from the hand. To restore motor function with upper-limb neuroprostheses requires that somatosensory feedback be provided to the tetraplegic patient or amputee. Given the complexity of state-of-the-art prosthetic limbs and, thus, the huge state space they can traverse, it is desirable to minimize the need for the patient to learn associations between events impinging on the limb and arbitrary sensations. Accordingly, we have developed approaches to intuitively convey sensory information that is critical for object manipulation--information about contact location, pressure, and timing--through intracortical microstimulation of primary somatosensory cortex. In experiments with nonhuman primates, we show that we can elicit percepts that are projected to a localized patch of skin and that track the pressure exerted on the skin. In a real-time application, we demonstrate that animals can perform a tactile discrimination task equally well whether mechanical stimuli are delivered to their native fingers or to a prosthetic one. Finally, we propose that the timing of contact events can be signaled through phasic intracortical microstimulation at the onset and offset of object contact that mimics the ubiquitous on and off responses observed in primary somatosensory cortex to complement slowly varying pressure-related feedback. We anticipate that the proposed biomimetic feedback will considerably increase the dexterity and embodiment of upper-limb neuroprostheses and will constitute an important step in restoring touch to individuals who have lost it.

EEG Source Imaging Enhances the Decoding of Complex Right-Hand Motor Imagery Tasks

GOAL

Sensorimotor-based brain-computer interfaces (BCIs) have achieved successful control of real and virtual devices in up to three dimensions; however, the traditional sensor-based paradigm limits the intuitive use of these systems. Many control signals for state-of-the-art BCIs involve imagining the movement of body parts that have little to do with the output command, revealing a cognitive disconnection between the user's intent and the action of the end effector. Therefore, there is a need to develop techniques that can identify with high spatial resolution the self-modulated neural activity reflective of the actions of a helpful output device.

METHODS

We extend previous EEG source imaging (ESI) work to decoding natural hand/wrist manipulations by applying a novel technique to classifying four complex motor imaginations of the right hand: flexion, extension, supination, and pronation.

RESULTS

We report an increase of up to 18.6% for individual task classification and 12.7% for overall classification using the proposed ESI approach over the traditional sensor-based method.

CONCLUSION

ESI is able to enhance BCI performance of decoding complex right-hand motor imagery tasks.

SIGNIFICANCE

This study may lead to the development of BCI systems with naturalistic and intuitive motor imaginations, thus facilitating broad use of noninvasive BCIs.

Brain-computer interface-based robotic end effector system for wrist and hand rehabilitation: results of a three-armed randomized controlled trial for chronic stroke

The objective of this study was to investigate the efficacy of an Electroencephalography (EEG)-based Motor Imagery (MI) Brain-Computer Interface (BCI) coupled with a Haptic Knob (HK) robot for arm rehabilitation in stroke patients. In this three-arm, single-blind, randomized controlled trial; 21 chronic hemiplegic stroke patients (Fugl-Meyer Motor Assessment (FMMA) score 10-50), recruited after pre-screening for MI BCI ability, were randomly allocated to BCI-HK, HK or Standard Arm Therapy (SAT) groups. All groups received 18 sessions of intervention over 6 weeks, 3 sessions per week, 90 min per session. The BCI-HK group received 1 h of BCI coupled with HK intervention, and the HK group received 1 h of HK intervention per session. Both BCI-HK and HK groups received 120 trials of robot-assisted hand grasping and knob manipulation followed by 30 min of therapist-assisted arm mobilization. The SAT group received 1.5 h of therapist-assisted arm mobilization and forearm pronation-supination movements incorporating wrist control and grasp-release functions. In all, 14 males, 7 females, mean age 54.2 years, mean stroke duration 385.1 days, with baseline FMMA score 27.0 were recruited. The primary outcome measure was upper extremity FMMA scores measured mid-intervention at week 3, end-intervention at week 6, and follow-up at weeks 12 and 24. Seven, 8 and 7 subjects underwent BCI-HK, HK and SAT interventions respectively. FMMA score improved in all groups, but no intergroup differences were found at any time points. Significantly larger motor gains were observed in the BCI-HK group compared to the SAT group at weeks 3, 12, and 24, but motor gains in the HK group did not differ from the SAT group at any time point. In conclusion, BCI-HK is effective, safe, and may have the potential for enhancing motor recovery in chronic stroke when combined with therapist-assisted arm mobilization.

Post-stroke Rehabilitation Training with a Motor-Imagery-Based Brain-Computer Interface (BCI)-Controlled Hand Exoskeleton: A Randomized Controlled Multicenter Trial

Repeated use of brain-computer interfaces (BCIs) providing contingent sensory feedback of brain activity was recently proposed as a rehabilitation approach to restore motor function after stroke or spinal cord lesions. However, there are only a few clinical studies that investigate feasibility and effectiveness of such an approach. Here we report on a placebo-controlled, multicenter clinical trial that investigated whether stroke survivors with severe upper limb (UL) paralysis benefit from 10 BCI training sessions each lasting up to 40 min. A total of 74 patients participated: median time since stroke is 8 months, 25 and 75% quartiles [3.0; 13.0]; median severity of UL paralysis is 4.5 points [0.0; 30.0] as measured by the Action Research Arm Test, ARAT, and 19.5 points [11.0; 40.0] as measured by the Fugl-Meyer Motor Assessment, FMMA. Patients in the BCI group (n = 55) performed motor imagery of opening their affected hand. Motor imagery-related brain electroencephalographic activity was translated into contingent hand exoskeleton-driven opening movements of the affected hand. In a control group (n = 19), hand exoskeleton-driven opening movements of the affected hand were independent of brain electroencephalographic activity. Evaluation of the UL clinical assessments indicated that both groups improved, but only the BCI group showed an improvement in the ARAT's grasp score from 0 [0.0; 14.0] to 3.0 [0.0; 15.0] points (p < 0.01) and pinch scores from 0.0 [0.0; 7.0] to 1.0 [0.0; 12.0] points (p < 0.01). Upon training completion, 21.8% and 36.4% of the patients in the BCI group improved their ARAT and FMMA scores respectively. The corresponding numbers for the control group were 5.1% (ARAT) and 15.8% (FMMA). These results suggests that adding BCI control to exoskeleton-assisted physical therapy can improve post-stroke rehabilitation outcomes. Both maximum and mean values of the percentage of successfully decoded imagery-related EEG activity, were higher than chance level. A correlation between the classification accuracy and the improvement in the upper extremity function was found. An improvement of motor function was found for patients with different duration, severity and location of the stroke.

Efficient neuroplasticity induction in chronic stroke patients by an associative brain-computer interface

Brain-computer interfaces (BCIs) have the potential to improve functionality in chronic stoke patients when applied over a large number of sessions. Here we evaluated the effect and the underlying mechanisms of three BCI training sessions in a double-blind sham-controlled design. The applied BCI is based on Hebbian principles of associativity that hypothesize that neural assemblies activated in a correlated manner will strengthen synaptic connections. Twenty-two chronic stroke patients were divided into two training groups. Movement-related cortical potentials (MRCPs) were detected by electroencephalography during repetitions of foot dorsiflexion. Detection triggered a single electrical stimulation of the common peroneal nerve timed so that the resulting afferent volley arrived at the peak negative phase of the MRCP (BCIassociative group) or randomly (BCInonassociative group). Fugl-Meyer motor assessment (FM), 10-m walking speed, foot and hand tapping frequency, diffusion tensor imaging (DTI) data, and the excitability of the corticospinal tract to the target muscle [tibialis anterior (TA)] were quantified. The TA motor evoked potential (MEP) increased significantly after the BCIassociative intervention, but not for the BCInonassociative group. FM scores (0.8 ± 0.46 point difference, P = 0.01), foot (but not finger) tapping frequency, and 10-m walking speed improved significantly for the BCIassociative group, indicating clinically relevant improvements. Corticospinal tract integrity on DTI did not correlate with clinical or physiological changes. For the BCI as applied here, the precise coupling between the brain command and the afferent signal was imperative for the behavioral, clinical, and neurophysiological changes reported. This association may become the driving principle for the design of BCI rehabilitation in the future. Indeed, no available BCIs can match this degree of functional improvement with such a short intervention.

Brain-Computer Interfaces for Communication and Control

The brain's electrical signals enable people without muscle control to physically interact with the world.

Role of the Contralesional vs. Ipsilesional Hemisphere in Stroke Recovery

Following a stroke, the resulting lesion creates contralateral motor impairment and an interhemispheric imbalance involving hyperexcitability of the contralesional hemisphere. Neuronal reorganization may occur on both the ipsilesional and contralesional hemispheres during recovery to regain motor functionality and therefore bilateral activation for the hemiparetic side is often observed. Although ipsilesional hemispheric reorganization is traditionally thought to be most important for successful recovery, definitive conclusions into the role and importance of the contralesional motor cortex remain under debate. Through examining recent research in functional neuroimaging investigating motor cortex changes post-stroke, as well as brain-computer interface (BCI) and transcranial magnetic stimulation (TMS) therapies, this review attempts to clarify the contributions of each hemisphere toward recovery. Several functional magnetic resonance imaging studies suggest that continuation of contralesional hemisphere hyperexcitability correlates with lesser recovery, however a subset of well-recovered patients demonstrate contralesional motor activity and show decreased functional capability when the contralesional hemisphere is inhibited. BCI therapy may beneficially activate either the contralesional or ipsilesional hemisphere, depending on the study design, for chronic stroke patients who are otherwise at a functional plateau. Repetitive TMS used to excite the ipsilesional motor cortex or inhibit the contralesional hemisphere has shown promise in enhancing stroke patients' recovery.

Brain-to-text: decoding spoken phrases from phone representations in the brain

It has long been speculated whether communication between humans and machines based on natural speech related cortical activity is possible. Over the past decade, studies have suggested that it is feasible to recognize isolated aspects of speech from neural signals, such as auditory features, phones or one of a few isolated words. However, until now it remained an unsolved challenge to decode continuously spoken speech from the neural substrate associated with speech and language processing. Here, we show for the first time that continuously spoken speech can be decoded into the expressed words from intracranial electrocorticographic (ECoG) recordings.Specifically, we implemented a system, which we call Brain-To-Text that models single phones, employs techniques from automatic speech recognition (ASR), and thereby transforms brain activity while speaking into the corresponding textual representation. Our results demonstrate that our system can achieve word error rates as low as 25% and phone error rates below 50%. Additionally, our approach contributes to the current understanding of the neural basis of continuous speech production by identifying those cortical regions that hold substantial information about individual phones. In conclusion, the Brain-To-Text system described in this paper represents an important step toward human-machine communication based on imagined speech.

Adaptive Deep Brain Stimulation for Movement Disorders: The Long Road to Clinical Therapy

Continuous high-frequency DBS is an established treatment for essential tremor and Parkinson's disease. Current developments focus on trying to widen the therapeutic window of DBS. Adaptive DBS (aDBS), where stimulation is dynamically controlled by feedback from biomarkers of pathological brain circuit activity, is one such development. Relevant biomarkers may be central, such as local field potential activity, or peripheral, such as inertial tremor data. Moreover, stimulation may be directed by the amplitude or the phase (timing) of the biomarker signal. In this review, we evaluate existing aDBS studies as proof-of-principle, discuss their limitations, most of which stem from their acute nature, and propose what is needed to take aDBS into a chronic setting. © 2017 The Authors. Movement Disorders published by Wiley Periodicals, Inc. on behalf of International Parkinson and Movement Disorder Society.

Contralesional Brain-Computer Interface Control of a Powered Exoskeleton for Motor Recovery in Chronic Stroke Survivors

Supplemental Digital Content is available in the text.

Background and Purpose—

There are few effective therapies to achieve functional recovery from motor-related disabilities affecting the upper limb after stroke. This feasibility study tested whether a powered exoskeleton driven by a brain–computer interface (BCI), using neural activity from the unaffected cortical hemisphere, could affect motor recovery in chronic hemiparetic stroke survivors. This novel system was designed and configured for a home-based setting to test the feasibility of BCI-driven neurorehabilitation in outpatient environments.

Methods—

Ten chronic hemiparetic stroke survivors with moderate-to-severe upper-limb motor impairment (mean Action Research Arm Test=13.4) used a powered exoskeleton that opened and closed the affected hand using spectral power from electroencephalographic signals from the unaffected hemisphere associated with imagined hand movements of the paretic limb. Patients used the system at home for 12 weeks. Motor function was evaluated before, during, and after the treatment.

Results—

Across patients, our BCI-driven approach resulted in a statistically significant average increase of 6.2 points in the Action Research Arm Test. This behavioral improvement significantly correlated with improvements in BCI control. Secondary outcomes of grasp strength, Motricity Index, and the Canadian Occupational Performance Measure also significantly improved.

Conclusions—

The findings demonstrate the therapeutic potential of a BCI-driven neurorehabilitation approach using the unaffected hemisphere in this uncontrolled sample of chronic stroke survivors. They also demonstrate that BCI-driven neurorehabilitation can be effectively delivered in the home environment, thus increasing the probability of future clinical translation.

Clinical Trial Registration—

URL: http://www.clinicaltrials.gov. Unique identifier: NCT02552368.

Feature Extraction and Classification Methods for Hybrid fNIRS-EEG Brain-Computer Interfaces

In this study, a brain-computer interface (BCI) framework for hybrid functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) for locked-in syndrome (LIS) patients is investigated. Brain tasks, channel selection methods, and feature extraction and classification algorithms available in the literature are reviewed. First, we categorize various types of patients with cognitive and motor impairments to assess the suitability of BCI for each of them. The prefrontal cortex is identified as a suitable brain region for imaging. Second, the brain activity that contributes to the generation of hemodynamic signals is reviewed. Mental arithmetic and word formation tasks are found to be suitable for use with LIS patients. Third, since a specific targeted brain region is needed for BCI, methods for determining the region of interest are reviewed. The combination of a bundled-optode configuration and threshold-integrated vector phase analysis turns out to be a promising solution. Fourth, the usable fNIRS features and EEG features are reviewed. For hybrid BCI, a combination of the signal peak and mean fNIRS signals and the highest band powers of EEG signals is promising. For classification, linear discriminant analysis has been most widely used. However, further research on vector phase analysis as a classifier for multiple commands is desirable. Overall, proper brain region identification and proper selection of features will improve classification accuracy. In conclusion, five future research issues are identified, and a new BCI scheme, including brain therapy for LIS patients and using the framework of hybrid fNIRS-EEG BCI, is provided.

EEG control of a virtual helicopter in 3-dimensional space using intelligent control strategies

Films like Firefox, Surrogates, and Avatar have explored the possibilities of using brain-computer interfaces (BCIs) to control machines and replacement bodies with only thought. Real world BCIs have made great progress toward that end. Invasive BCIs have enabled monkeys to fully explore 3-D space using neuroprosthetics. However, noninvasive BCIs have not been able to demonstrate such mastery of 3-D space. Here, we report our work, which demonstrates that human subjects can use a noninvasive BCI to fly a virtual helicopter to any point in a 3-D world. Through use of intelligent control strategies, we have facilitated the realization of controlled flight in 3-D space. We accomplished this through a reductionist approach that assigns subject-specific control signals to the crucial components of 3-D flight. Subject control of the helicopter was comparable when using either the BCI or a keyboard. By using intelligent control strategies, the strengths of both the user and the BCI system were leveraged and accentuated. Intelligent control strategies in BCI systems such as those presented here may prove to be the foundation for complex BCIs capable of doing more than we ever imagined.

Toward a high-throughput auditory P300-based brain-computer interface

OBJECTIVE

Brain-computer interface (BCI) technology can provide severely disabled people with non-muscular communication. For those most severely disabled, limitations in eye mobility or visual acuity may necessitate auditory BCI systems. The present study investigates the efficacy of the use of six environmental sounds to operate a 6x6 P300 Speller.

METHODS

A two-group design was used to ascertain whether participants benefited from visual cues early in training. Group A (N=5) received only auditory stimuli during all 11 sessions, whereas Group AV (N=5) received simultaneous auditory and visual stimuli in initial sessions after which the visual stimuli were systematically removed. Stepwise linear discriminant analysis determined the matrix item that elicited the largest P300 response and thereby identified the desired choice.

RESULTS

Online results and offline analyses showed that the two groups achieved equivalent accuracy. In the last session, eight of 10 participants achieved 50% or more, and four of these achieved 75% or more, online accuracy (2.8% accuracy expected by chance). Mean bit rates averaged about 2 bits/min, and maximum bit rates reached 5.6 bits/min.

CONCLUSIONS

This study indicates that an auditory P300 BCI is feasible, that reasonable classification accuracy and rate of communication are achievable, and that the paradigm should be further evaluated with a group of severely disabled participants who have limited visual mobility.

SIGNIFICANCE

With further development, this auditory P300 BCI could be of substantial value to severely disabled people who cannot use a visual BCI.