Riemannian geometric and ensemble learning for decoding cross-session motor imagery electroencephalography signals

Unsupervised and semi-supervised domain adaptation networks considering both global knowledge and prototype-based local class information for Motor Imagery Classification

The non-stationarity of EEG signals results in variability across sessions, impeding model building and data sharing. In this paper, we propose a domain adaptation method called GPL, which simultaneously considers global knowledge and prototype-based local class information to enhance the classification accuracy of motor imagery signals. Depending on the amount of labeled data available in the target domain, the method is implemented in both unsupervised and semi-supervised versions. Specifically, at the global level, we employ the maximum mean difference (MMD) loss to globally constrain the feature space, achieving comprehensive alignment. In the context of class-level operations, we propose two memory banks designed to accommodate class prototypes in each domain and constrain feature embeddings by applying two prototype-based contrastive losses. The source contrastive loss is used to organize source features spatially based on categories, thereby reconciling inter-class and intra-class relationships, while the interactive contrastive loss is employed to facilitate cross-domain information interaction. Simultaneously, in unsupervised scenarios, to mitigate the adverse effects of excessive pseudo-labels, we introduce an entropy-aware strategy that dynamically evaluates the confidence level of target data and personalized constraints on the participation of interactive contrastive loss. To validate our approach, extensive experiments were conducted on a highly regarded public EEG dataset, namely Dataset IIa of the BCI Competition IV, as well as a large-scale EEG dataset called GigaDB. The experiments yielded average classification accuracies of 86.03% and 84.22% respectively. These results demonstrate that our method is an effective EEG decoding model, conducive to advancing the development of motor imagery brain-computer interfaces. The architecture proposed in this study and the code for data partitioning can be found at https://github.com/zhangdx21/GPL.

Improving inter-session performance via relevant session-transfer for multi-session motor imagery classification

Motor imagery (MI)-based brain-computer interfaces (BCIs) using electroencephalography (EEG) have found practical applications in external device control. However, the non-stationary nature of EEG signals remains to obstruct BCI performance across multiple sessions, even for the same user. In this study, we aim to address the impact of non-stationarity, also known as inter-session variability, on multi-session MI classification performance by introducing a novel approach, the relevant session-transfer (RST) method. Leveraging the cosine similarity as a benchmark, the RST method transfers relevant EEG data from the previous session to the current one. The effectiveness of the proposed RST method was investigated through performance comparisons with the self-calibrating method, which uses only the data from the current session, and the whole-session transfer method, which utilizes data from all prior sessions. We validated the effectiveness of these methods using two datasets: a large MI public dataset (Shu Dataset) and our own dataset of gait-related MI, which includes both healthy participants and individuals with spinal cord injuries. Our experimental results revealed that the proposed RST method leads to a 2.29 % improvement (p < 0.001) in the Shu Dataset and up to a 6.37 % improvement in our dataset when compared to the self-calibrating method. Moreover, our method surpassed the performance of the recent highest-performing method that utilized the Shu Dataset, providing further support for the efficacy of the RST method in improving multi-session MI classification performance. Consequently, our findings confirm that the proposed RST method can improve classification performance across multiple sessions in practical MI-BCIs.

Research on shared control of robots based on hybrid brain-computer interface

BACKGROUND

With the arrival of the new generation of artificial intelligence wave, new human-robot interaction technologies continue to emerge. Brain-computer interface (BCI) offers a pathway for state monitoring and interaction control between human and robot. However, the unstable mental state reduce the accuracy of human brain intent decoding, and consequently affects the precision of BCI control.

NEW METHODS

This paper proposes a hybrid BCI-based shared control (HB-SC) method for brain-controlled robot navigation. Hybrid BCI fuses electroencephalogram (EEG) and electromyography (EMG) for mental state monitoring and interactive control to output human perception and decision. The shared control based on multi-sensory fusion integrates the special obstacle information perceived by humans with the regular environmental information perceived by the robot. In this process, valid BCI commands are screened by mental state assessment and output to a layered costmap for fusion.

RESULTS

Eight subjects participated in the navigation experiment with dynamically changing mental state levels to validate the effects of a hybrid brain-computer interface through two shared control modes. The results show that the proposed HB-SC reduces collisions by 37.50 %, improves the success rate of traversing obstacles by 25.00 %, and the navigation trajectory is more consistent with expectations.

CONCLUSIONS

The HB-SC method can dynamically and intelligently adjust command output according to different brain states, helping to reduce errors made by subjects in a unstable mental state, thereby greatly enhancing the system's safety.

Improving classification performance of motor imagery BCI through EEG data augmentation with conditional generative adversarial networks

In brain-computer interface (BCI), building accurate electroencephalogram (EEG) classifiers for specific mental tasks is critical for BCI performance. The classifiers are developed by machine learning (ML) and deep learning (DL) techniques, requiring a large dataset for training to build reliable and accurate models. However, collecting large enough EEG datasets is difficult due to intra-/inter-subject variabilities and experimental costs. This leads to the data scarcity problem, which causes overfitting issues to training samples, resulting in reducing generalization performance. To solve the EEG data scarcity problem and improve the performance of the EEG classifiers, we propose a novel EEG data augmentation (DA) framework using conditional generative adversarial networks (cGANs). An experimental study is implemented with two public EEG datasets, including motor imagery (MI) tasks (BCI competition IV IIa and III IVa), to validate the effectiveness of the proposed EEG DA method for the EEG classifiers. To evaluate the proposed cGAN-based DA method, we tested eight EEG classifiers for the experiment, including traditional MLs and state-of-the-art DLs with three existing EEG DA methods. Experimental results showed that most DA methods with proper DA proportion in the training dataset had higher classification performances than without DA. Moreover, applying the proposed DA method showed superior classification performance improvement than the other DA methods. This shows that the proposed method is a promising EEG DA method for enhancing the performances of the EEG classifiers in MI-based BCIs.

An end-to-end multi-task motor imagery EEG classification neural network based on dynamic fusion of spectral-temporal features

Electroencephalograph (EEG) brain-computer interfaces (BCI) have potential to provide new paradigms for controlling computers and devices. The accuracy of brain pattern classification in EEG BCI is directly affected by the quality of features extracted from EEG signals. Currently, feature extraction heavily relies on prior knowledge to engineer features (for example from specific frequency bands); therefore, better extraction of EEG features is an important research direction. In this work, we propose an end-to-end deep neural network that automatically finds and combines features for motor imagery (MI) based EEG BCI with 4 or more imagery classes (multi-task). First, spectral domain features of EEG signals are learned by compact convolutional neural network (CCNN) layers. Then, gated recurrent unit (GRU) neural network layers automatically learn temporal patterns. Lastly, an attention mechanism dynamically combines (across EEG channels) the extracted spectral-temporal features, reducing redundancy. We test our method using BCI Competition IV-2a and a data set we collected. The average classification accuracy on 4-class BCI Competition IV-2a was 85.1 % ± 6.19 %, comparable to recent work in the field and showing low variability among participants; average classification accuracy on our 6-class data was 64.4 % ± 8.35 %. Our dynamic fusion of spectral-temporal features is end-to-end and has relatively few network parameters, and the experimental results show its effectiveness and potential.

A high-performance general computer cursor control scheme based on a hybrid BCI combining motor imagery and eye-tracking

This study introduces a novel virtual cursor control system designed to empower individuals with neuromuscular disabilities in the digital world. By combining eye-tracking with motor imagery (MI) in a hybrid brain-computer interface (BCI), the system enhances cursor control accuracy and simplicity. Real-time classification accuracy reaches 87.92% (peak of 93.33%), with cursor stability in the gazing state at 96.1%. Integrated into common operating systems, it enables tasks like text entry, online chatting, email, web surfing, and picture dragging, with an average text input rate of 53.2 characters per minute (CPM). This technology facilitates fundamental computing tasks for patients, fostering their integration into the online community and paving the way for future developments in BCI systems.

Neural Correlate and Movement Decoding of Simultaneous-and-Sequential Bimanual Movements Using EEG Signals

Bimanual coordination is important for developing a natural motor brain-computer interface (BCI) from electroencephalogram (EEG) signals, covering the aspects of bilateral arm training for rehabilitation, bimanual coordination for daily-life assistance, and also improving the multidimensional control of BCIs. For the same task targets of both hands, simultaneous and sequential bimanual movements are two different bimanual coordination manners. Planning and performing motor sequences are the fundamental abilities of humans, and it is more natural to execute sequential movements compared to simultaneous movements in many complex tasks. However, to date, for these two different manners in which two hands coordinated to reach the same task targets, the differences in the neural correlate and also the feasibility of movement discrimination have not been explored. In this study, we aimed to investigate these two issues based on a bimanual reaching task for the first time. Finally, neural correlates in the view of the movement-related cortical potentials, event-related oscillations, and source imaging showed unique neural encoding patterns of sequential movements. Besides, for the same task targets of both hands, the simultaneous and sequential bimanual movements were successfully discriminated in both pre-movement and movement execution periods. This study revealed the neural encoding patterns of sequential bimanual movements and presented its values in developing a more natural and good-performance motor BCI.

Session-independent subject-adaptive mental imagery BCI using selective filter-bank adaptive Riemannian features

The brain-computer interfaces (BCIs) facilitate the users to exploit information encoded in neural signals, specifically electroencephalogram (EEG), to control devices and for neural rehabilitation. Mental imagery (MI)-driven BCI predicts the user's pre-meditated mental objectives, which could be deployed as command signals. This paper presents a novel learning-based framework for classifying MI tasks using EEG-based BCI. In particular, our work focuses on the variation in inter-session data and the extraction of multi-spectral user-tailored features for robust performance. Thus, the goal is to create a calibration-free subject-adaptive learning framework for various mental imagery tasks not restricted to motor imagery alone. In this regard, critical spectral bands and the best temporal window are first selected from the EEG training trials of the subject based on the Riemannian user learning distance metric (Dscore) that checks for distinct and stable patterns. The filtered covariance matrices of the EEG trials in each spectral band are then transformed towards a reference covariance matrix using the Riemannian transfer learning, enabling the different sessions to be comparable. The evaluation of our proposed Selective Time-window and Multi-scale Filter-Bank with Adaptive Riemannian (STFB-AR) features on four public datasets, including disabled subjects, showed around 15% and 8% improvement in mean accuracy over baseline and fixed filter-bank models, respectively.

Unraveling motor imagery brain patterns using explainable artificial intelligence based on Shapley values

BACKGROUND AND OBJECTIVE

Motor imagery (MI) based brain-computer interfaces (BCIs) are widely used in rehabilitation due to the close relationship that exists between MI and motor execution (ME). However, the underlying brain mechanisms of MI remain not well understood. Most MI-BCIs use the sensorimotor rhythms elicited in the primary motor cortex (M1) and somatosensory cortex (S1), which consist of an event-related desynchronization followed by an event-related synchronization. Consequently, this has resulted in systems that only record signals around M1 and S1. However, MI could involve a more complex network including sensory, association, and motor areas. In this study, we hypothesize that the superior accuracies achieved by new deep learning (DL) models applied to MI decoding rely on focusing on a broader MI activation of the brain. Parallel to the success of DL, the field of explainable artificial intelligence (XAI) has seen continuous development to provide explanations for DL networks success. The goal of this study is to use XAI in combination with DL to extract information about MI brain activation patterns from non-invasive electroencephalography (EEG) signals.

METHODS

We applied an adaptation of Shapley additive explanations (SHAP) to EEGSym, a state-of-the-art DL network with exceptional transfer learning capabilities for inter-subject MI classification. We obtained the SHAP values from two public databases comprising 171 users generating left and right hand MI instances with and without real-time feedback.

RESULTS

We found that EEGSym based most of its prediction on the signal of the frontal electrodes, i.e. F7 and F8, and on the first 1500 ms of the analyzed imagination period. We also found that MI involves a broad network not only based on M1 and S1, but also on the prefrontal cortex (PFC) and the posterior parietal cortex (PPC). We further applied this knowledge to select a 8-electrode configuration that reached inter-subject accuracies of 86.5% ± 10.6% on the Physionet dataset and 88.7% ± 7.0% on the Carnegie Mellon University's dataset.

CONCLUSION

Our results demonstrate the potential of combining DL and SHAP-based XAI to unravel the brain network involved in producing MI. Furthermore, SHAP values can optimize the requirements for out-of-laboratory BCI applications involving real users.