A Comparative Study on the Detection of Covert Attention in Event-Related EEG and MEG Signals to Control a BCI

A toolbox for decoding BCI commands based on event-related potentials

Commands in brain-computer interface (BCI) applications often rely on the decoding of event-related potentials (ERP). For instance, the P300 potential is frequently used as a marker of attention to an oddball event. Error-related potentials and the N2pc signal are further examples of ERPs used for BCI control. One challenge in decoding brain activity from the electroencephalogram (EEG) is the selection of the most suitable channels and appropriate features for a particular classification approach. Here we introduce a toolbox that enables ERP-based decoding using the full set of channels, while automatically extracting informative components from relevant channels. The strength of our approach is that it handles sequences of stimuli that encode multiple items using binary classification, such as target vs. nontarget events typically used in ERP-based spellers. We demonstrate examples of application scenarios and evaluate the performance of four openly available datasets: a P300-based matrix speller, a P300-based rapid serial visual presentation (RSVP) speller, a binary BCI based on the N2pc, and a dataset capturing error potentials. We show that our approach achieves performances comparable to those in the original papers, with the advantage that only conventional preprocessing is required by the user, while channel weighting and decoding algorithms are internally performed. Thus, we provide a tool to reliably decode ERPs for BCI use with minimal programming requirements.

Carbohydrate based biomaterials for neural interface applications

Neuroprosthetic devices that record and modulate neural activities have demonstrated immense potential for bypassing or restoring lost neurological functions due to neural injuries and disorders. However, implantable electrical devices interfacing with brain tissue are susceptible to a series of inflammatory tissue responses along with mechanical or electrical failures which can affect the device performance over time. Several biomaterial strategies have been implemented to improve device-tissue integration for high quality and stable performance. Ranging from developing smaller, softer, and more flexible electrode designs to introducing bioactive coatings and drug-eluting layers on the electrode surface, such strategies have shown different degrees of success but with limitations. With their hydrophilic properties and specific bioactivities, carbohydrates offer a potential solution for addressing some of the limitations of the existing biomolecular approaches. In this review, we summarize the role of polysaccharides in the central nervous system, with a primary focus on glycoproteins and proteoglycans, to shed light on their untapped potential as biomaterials for neural implants. Utilization of glycosaminoglycans for neural interface and tissue regeneration applications is comprehensively reviewed to provide the current state of carbohydrate-based biomaterials for neural implants. Finally, we will discuss the challenges and opportunities of applying carbohydrate-based biomaterials for neural tissue interfaces.

A Regional Smoothing Block Sparse Bayesian Learning Method With Temporal Correlation for Channel Selection in P300 Speller

The P300-based brain-computer interfaces (BCIs) enable participants to communicate by decoding the electroencephalography (EEG) signal. Different regions of the brain correspond to various mental activities. Therefore, removing weak task-relevant and noisy channels through channel selection is necessary when decoding a specific type of activity from EEG. It can improve the recognition accuracy and reduce the training time of the subsequent models. This study proposes a novel block sparse Bayesian-based channel selection method for the P300 speller. In this method, we introduce block sparse Bayesian learning (BSBL) into the channel selection of P300 BCI for the first time and propose a regional smoothing BSBL (RSBSBL) by combining the spatial distribution properties of EEG. The RSBSBL can determine the number of channels adaptively. To ensure practicality, we design an automatic selection iteration strategy model to reduce the time cost caused by the inverse operation of the large-size matrix. We verified the proposed method on two public P300 datasets and on our collected datasets. The experimental results show that the proposed method can remove the inferior channels and work with the classifier to obtain high-classification accuracy. Hence, RSBSBL has tremendous potential for channel selection in P300 tasks.

Reducing calibration time in motor imagery-based BCIs by data alignment and empirical mode decomposition

One of the major reasons that limit the practical applications of a brain-computer interface (BCI) is its long calibration time. In this paper, we propose a novel approach to reducing the calibration time of motor imagery (MI)-based BCIs without sacrificing classification accuracy. The approach aims to augment the training set size of a new subject by generating artificial electroencephalogram (EEG) data from a few training trials initially available. The artificial EEG data are obtained by first performing empirical mode decomposition (EMD) and then mixing resulting intrinsic mode functions (IMFs). The original training trials are aligned to common reference point with Euclidean alignment (EA) method prior to EMD and pooled together with artificial trials as the expended training set, which is input into a linear discriminant analysis (LDA) classifier or a logistic regression (LR) classifier. The performance of the proposed algorithm is evaluated on two motor imagery (MI) data sets and compared with that of the algorithm trained with only real EEG data (Baseline) and the algorithm trained with expanded EEG data by EMD without data alignment. The experimental results showed that the proposed algorithm can significantly reduce the amount of training data needed to achieve a given performance level and thus is expected to facilitate the real-world applications of MI-based BCIs.

Impact of Stimulus Features on the Performance of a Gaze-Independent Brain-Computer Interface Based on Covert Spatial Attention Shifts

Regaining communication abilities in patients who are unable to speak or move is one of the main goals in decoding brain waves for brain-computer interface (BCI) control. Many BCI approaches designed for communication rely on attention to visual stimuli, commonly applying an oddball paradigm, and require both eye movements and adequate visual acuity. These abilities may, however, be absent in patients who depend on BCI communication. We have therefore developed a response-based communication BCI, which is independent of gaze shifts but utilizes covert shifts of attention to the left or right visual field. We recorded the electroencephalogram (EEG) from 29 channels and coregistered the vertical and horizontal electrooculogram. Data-driven decoding of small attention-based differences between the hemispheres, also known as N2pc, was performed using 14 posterior channels, which are expected to reflect correlates of visual spatial attention. Eighteen healthy participants responded to 120 statements by covertly directing attention to one of two colored symbols (green and red crosses for "yes" and "no," respectively), presented in the user's left and right visual field, respectively, while maintaining central gaze fixation. On average across participants, 88.5% (std: 7.8%) of responses were correctly decoded online. In order to investigate the potential influence of stimulus features on accuracy, we presented the symbols with different visual angles, by altering symbol size and eccentricity. The offline analysis revealed that stimulus features have a minimal impact on the controllability of the BCI. Hence, we show with our novel approach that spatial attention to a colored symbol is a robust method with which to control a BCI, which has the potential to support severely paralyzed people with impaired eye movements and low visual acuity in communicating with their environment.

Can we accurately predict where we look at paintings?

The objective of this study is to investigate and to simulate the gaze deployment of observers on paintings. For that purpose, we built a large eye tracking dataset composed of 150 paintings belonging to 5 art movements. We observed that the gaze deployment over the proposed paintings was very similar to the gaze deployment over natural scenes. Therefore, we evaluate existing saliency models and propose a new one which significantly outperforms the most recent deep-based saliency models. Thanks to this new saliency model, we can predict very accurately what are the salient areas of a painting. This opens new avenues for many image-based applications such as animation of paintings or transformation of a still painting into a video clip.

Decoding the covert shift of spatial attention from electroencephalographic signals permits reliable control of a brain-computer interface

OBJECTIVE

One of the main goals of brain-computer interfaces (BCI) is to restore communication abilities in patients. BCIs often use event-related potentials (ERPs) like the P300 which signals the presence of a target in a stream of stimuli. The P300 and related approaches, however, are inherently limited, as they require many stimulus presentations to obtain a usable control signal. Many approaches depend on gaze direction to focus the target, which is also not a viable approach in many cases, because eye movements might be impaired in potential users. Here we report on a BCI that avoids both shortcomings by decoding spatial target information, independent of gaze shifts.

APPROACH

We present a new method to decode from the electroencephalogram (EEG) covert shifts of attention to one out of four targets simultaneously presented in the left and right visual field. The task is designed to evoke the N2pc component-a hemisphere lateralized response, elicited over the occipital scalp contralateral to the attended target. The decoding approach involves decoding of the N2pc based on data-driven estimation of spatial filters and a correlation measure.

MAIN RESULTS

Despite variability of decoding performance across subjects, 22 out of 24 subjects performed well above chance level. Six subjects even exceeded 80% (cross-validated: 89%) correct predictions in a four-class discrimination task. Hence, the single-trial N2pc proves to be a component that allows for reliable BCI control. An offline analysis of the EEG data with respect to their dependence on stimulation time and number of classes demonstrates that the present method is also a workable approach for two-class tasks.

SIGNIFICANCE

Our method extends the range of strategies for gaze-independent BCI control. The proposed decoding approach has the potential to be efficient in similar applications intended to decode ERPs.

EEG als Steuersignal: Gehirnaktivität entschlüsseln und effizient als Kommunikationsmittel für Patienten mit motorischen Defiziten nutzen

Ziel der Studie Ereignis-korrelierte Potenziale werden in der Regel in einzelnen EEG-Kanälen ermittelt. Mit einem einzelnen Kanal wird jedoch nur ein Teil des gesamten Hirnprozesses erfasst. Für eine Gehirn-Computer Schnittstelle, die in kurzer Zeit eine Entscheidung treffen muss, ist diese singuläre Gehirnantwort häufig unzureichend wogegen die Information aus mehreren Kanälen häufig redundant ist. Beide Vorgehensweisen sind nicht optimal. Daher ist es unser Ziel, die Kanäle zu wenigen Komponenten zu kombinieren, die die relevantesten Modulationen eines Hirnprozesses erfassen.

Methodik Wir nutzen die kanonische Korrelationsanalyse, um datengetrieben räumliche Filter aus dem EEG zu bestimmen. Mit der Produkt-Moment Korrelation ermitteln wir, auf welche von 12 verschiedenen Stimulussequenzen die Studienteilnehmer geachtet haben.

Ergebnisse Die verdeckte Aufmerksamkeit der Studienteilnehmer konnte mit hoher Genauigkeit (89,3±9,2%) aus dem räumlich gefilterten EEG und signifikant besser als aus einzelnen Kanälen dekodiert werden.

Schlussfolgerung Die aus dem EEG erlernten räumlichen Filter ermöglichen die Extraktion von Komponenten, die einen event-korrelierten Gehirnprozess charakterisieren und eine Gehirn-Computer Schnittstelle effektiv steuern können, was von hoher Relevanz für Patienten ist, die nicht mehr anderweitig kommunizieren können.

Brain-Based Binary Communication Using Spatiotemporal Features of fNIRS Responses

“Locked-in” patients lose their ability to communicate naturally due to motor system dysfunction. Brain-computer interfacing offers a solution for their inability to communicate by enabling motor-independent communication. Straightforward and convenient in-session communication is essential in clinical environments. The present study introduces a functional near-infrared spectroscopy (fNIRS)-based binary communication paradigm that requires limited preparation time and merely nine optodes. Eighteen healthy participants performed two mental imagery tasks, mental drawing and spatial navigation, to answer yes/no questions during one of two auditorily cued time windows. Each of the six questions was answered five times, resulting in five trials per answer. This communication paradigm thus combines both spatial (two different mental imagery tasks, here mental drawing for “yes” and spatial navigation for “no”) and temporal (distinct time windows for encoding a “yes” and “no” answer) fNIRS signal features for information encoding. Participants’ answers were decoded in simulated real-time using general linear model analysis. Joint analysis of all five encoding trials resulted in an average accuracy of 66.67 and 58.33% using the oxygenated (HbO) and deoxygenated (HbR) hemoglobin signal respectively. For half of the participants, an accuracy of 83.33% or higher was reached using either the HbO signal or the HbR signal. For four participants, effective communication with 100% accuracy was achieved using either the HbO or HbR signal. An explorative analysis investigated the differentiability of the two mental tasks based solely on spatial fNIRS signal features. Using multivariate pattern analysis (MVPA) group single-trial accuracies of 58.33% (using 20 training trials per task) and 60.56% (using 40 training trials per task) could be obtained. Combining the five trials per run using a majority voting approach heightened these MVPA accuracies to 62.04 and 75%. Additionally, an fNIRS suitability questionnaire capturing participants’ physical features was administered to explore its predictive value for evaluating general data quality. Obtained questionnaire scores correlated significantly (r = -0.499) with the signal-to-noise of the raw light intensities. While more work is needed to further increase decoding accuracy, this study shows the potential of answer encoding using spatiotemporal fNIRS signal features or spatial fNIRS signal features only.

Effects of MEG-based neurofeedback for hand rehabilitation after tetraplegia: preliminary findings in cortical modulations and grip strength

OBJECTIVE

Neurofeedback (NF) trains people to volitionally modulate their cortical activity to affect a behavioral outcome. We evaluated the feasibility of using NF to improve hand function after chronic cervical-level spinal cord injury (SCI) using biologically-relevant visual feedback of motor-related brain activity and an intuitive control scheme.

APPROACH

The NF system acquired magnetoencephalography (MEG) data in real-time to provide feedback of event-related desynchronization (ERD) measured over the sensorimotor cortex during attempted hand grasping. During brain control, stronger ERD resulting from attempted grasping drove the virtual hand towards a more closed grasp, while less ERD drove the hand more open.

MAIN RESULTS

Eight individuals with partial or complete hand impairment due to chronic SCI controlled the NF to perform a grasping task that increased in difficulty as the participants achieved success. During their first NF session, participants achieved an average success rate of 63.7 ± 6.4% (chance level of 13.9%). After as few as one intervention session, four of the seven individuals evaluated for ERD changes had significantly strengthened ERD and three of the four participants with measurable grip strength prior to NF had increased grip strength. Interestingly, both individuals who participated in a longer-term study (i.e. >8 NF sessions) had improved grip strength and significantly strengthened ERD.

SIGNIFICANCE

This study demonstrates that MEG-based NF training can change brain activity in individuals with hand impairment due to SCI and has the potential to induce acute changes in grip strength. Future studies will evaluate whether neuroplasticity induced with long term NF can improve hand function for those with moderate impairment.

Convolutional neural networks for decoding of covert attention focus and saliency maps for EEG feature visualization

OBJECTIVE

Convolutional neural networks (CNNs) have proven successful as function approximators and have therefore been used for classification problems including electroencephalography (EEG) signal decoding for brain-computer interfaces (BCI). Artificial neural networks, however, are considered black boxes, because they usually have thousands of parameters, making interpretation of their internal processes challenging. Here we systematically evaluate the use of CNNs for EEG signal decoding and investigate a method for visualizing the CNN model decision process.

APPROACH

We developed a CNN model to decode the covert focus of attention from EEG event-related potentials during object selection. We compared the CNN and the commonly used linear discriminant analysis (LDA) classifier performance, applied to datasets with different dimensionality, and analyzed transfer learning capacity. Moreover, we validated the impact of single model components by systematically altering the model. Furthermore, we investigated the use of saliency maps as a tool for visualizing the spatial and temporal features driving the model output.

MAIN RESULTS

The CNN model and the LDA classifier achieved comparable accuracy on the lower-dimensional dataset, but CNN exceeded LDA performance significantly on the higher-dimensional dataset (without hypothesis-driven preprocessing), achieving an average decoding accuracy of 90.7% (chance level  =  8.3%). Parallel convolutions, tanh or ELU activation functions, and dropout regularization proved valuable for model performance, whereas the sequential convolutions, ReLU activation function, and batch normalization components reduced accuracy or yielded no significant difference. Saliency maps revealed meaningful features, displaying the typical spatial distribution and latency of the P300 component expected during this task.

SIGNIFICANCE

Following systematic evaluation, we provide recommendations for when and how to use CNN models in EEG decoding. Moreover, we propose a new approach for investigating the neural correlates of a cognitive task by training CNN models on raw high-dimensional EEG data and utilizing saliency maps for relevant feature extraction.

Spatial and Time Domain Feature of ERP Speller System Extracted via Convolutional Neural Network

Feature of event-related potential (ERP) has not been completely understood and illiteracy problem remains unsolved. To this end, P300 peak has been used as the feature of ERP in most brain–computer interface applications, but subjects who do not show such peak are common. Recent development of convolutional neural network provides a way to analyze spatial and temporal features of ERP. Here, we train the convolutional neural network with 2 convolutional layers whose feature maps represented spatial and temporal features of event-related potential. We have found that nonilliterate subjects' ERP show high correlation between occipital lobe and parietal lobe, whereas illiterate subjects only show correlation between neural activities from frontal lobe and central lobe. The nonilliterates showed peaks in P300, P500, and P700, whereas illiterates mostly showed peaks in around P700. P700 was strong in both subjects. We found that P700 peak may be the key feature of ERP as it appears in both illiterate and nonilliterate subjects.