EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy

Cross-session SSVEP brainprint recognition using attentive multi-sub-band depth identity embedding learning network

Brainprint recognition technology, regarded as a promising biometric technology, encounters challenges stemming from the time-varied, low signal-to-noise ratio of brain signals, such as electroencephalogram (EEG). Steady-state visual evoked potentials (SSVEP) exhibit high signal-to-noise ratio and frequency locking, making them a promising paradigm for brainprint recognition. Consequently, the extraction of time-invariant identity information from SSVEP EEG signals is essential. In this paper, we propose an Attentive Multi-sub-band Depth Identity Embedding Learning Network for stable cross-session SSVEP brainprint recognition. To address the issue of low recognition accuracy across sessions, we introduce the Sub-band Attentive Frequency mechanism, which integrates the frequency-domain relevant characteristics of the SSVEP paradigm and focuses on exploring depth-frequency identity embedding information. Also, we employ Attentive Statistic Pooling to enhance the stability of frequency domain feature distributions across sessions. Extensive experimentation and validation were conducted on two multi-session SSVEP benchmark datasets. The experimental results show that our approach outperforms other state-of-art models on 2-second samples across sessions and has the potential to serve as a benchmark in multi-subject biometric recognition systems.

Enhancing motor imagery EEG classification with a Riemannian geometry-based spatial filtering (RSF) method

Motor imagery (MI) refers to the mental simulation of movements without physical execution, and it can be captured using electroencephalography (EEG). This area has garnered significant research interest due to its substantial potential in brain-computer interface (BCI) applications, especially for individuals with physical disabilities. However, accurate classification of MI EEG signals remains a major challenge due to their non-stationary nature, low signal-to-noise ratio, and sensitivity to both external and physiological noise. Traditional classification methods, such as common spatial pattern (CSP), often assume that the data is stationary and Gaussian, which limits their applicability in real-world scenarios where these assumptions do not hold. These challenges highlight the need for more robust methods to improve classification accuracy in MI-BCI systems. To address these issues, this study introduces a Riemannian geometry-based spatial filtering (RSF) method that projects EEG signals into a lower-dimensional subspace, maximizing the Riemannian distance between covariance matrices from different classes. By leveraging the inherent geometric properties of EEG data, RSF enhances the discriminative power of the features while maintaining robustness against noise. The performance of RSF was evaluated in combination with ten commonly used MI decoding algorithms, including CSP with linear discriminant analysis (CSP-LDA), Filter Bank CSP (FBCSP), Minimum Distance to Riemannian Mean (MDM), Tangent Space Mapping (TSM), EEGNet, ShallowConvNet (sCNN), DeepConvNet (dCNN), FBCNet, Graph-CSPNet, and LMDA-Net, using six publicly available MI-BCI datasets. The results demonstrate that RSF significantly improves classification accuracy and reduces computational time, particularly for deep learning models with high computational complexity. These findings underscore the potential of RSF as an effective spatial filtering approach for MI EEG classification, providing new insights and opportunities for the development of robust MI-BCI systems. The code for this research is available at https://github.com/PLC-TJU/RSF.

A subject transfer neural network fuses Generator and Euclidean alignment for EEG-based motor imagery classification

BACKGROUND

Brain-computer interface (BCI) facilitates the connection between human brain and computer, enabling individuals to control external devices indirectly through cognitive processes. Although it has great development prospects, the significant difference in EEG signals among individuals hinders users from further utilizing the BCI system.

NEW METHOD

Addressing this difference and improving BCI classification accuracy remain key challenges. In this paper, we propose a transfer learning model based on deep learning to transfer the data distribution from the source domain to the target domain, named a subject transfer neural network combining the Generator with Euclidean alignment (ST-GENN). It consists of three parts: 1) Align the original EEG signals in the Euclidean space; 2) Send the aligned data to the Generator to obtain the transferred features; 3) Utilize the Convolution-attention-temporal (CAT) classifier to classify the transferred features.

RESULTS

The model is validated on BCI competition IV 2a, BCI competition IV 2b and SHU datasets to evaluate its classification performance, and the results are 82.85 %, 86.28 % and 67.2 % for the three datasets, respectively.

COMPARISON WITH EXISTING METHODS

The results have been shown to be robust to subject variability, with the average accuracy of the proposed method outperforming baseline algorithms by ranging from 2.03 % to 15.43 % on the 2a dataset, from 0.86 % to 10.16 % on the 2b dataset and from 3.3 % to 17.9 % on the SHU dataset.

CONCLUSIONS FOR RESEARCH ARTICLES

The advantage of our model lies in its ability to effectively transfer the experience and knowledge of the source domain data to the target domain, thus bridging the gap between them. Our method can improve the practicability of MI-BCI systems.

SSVEP-based BCI using ultra-low-frequency and high-frequency peripheral flickers

Objective. existing steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) systems predominantly employ a flicker frequency range of 8-20 Hz, which often induces visual fatigue in users, thereby compromising system performance. Considering that, this study introduces an innovative paradigm to enhance the user experience of SSVEP-based BCIs while maintaining the performance.Approach. the system encodes 12 targets by integrating ultra-low-frequency (2.00-3.32 Hz) and high-frequency (34.00-35.32 Hz) flickers with peripheral stimulation, and task-related component analysis is employed for SSVEP signal identification.Main results. the feasibility of the ultra-low-frequency peripheral stimulation paradigm was validated through online experiments, achieving an average accuracy of 89.03 ± 9.95% and an information transfer rate (ITR) of 66.74 ± 15.44 bits min-1. For the high-frequency peripheral stimulation paradigm, only the stimulation frequency changed, the paradigm, the signal processing algorithm and the step of frequency and phase were unchanged. The online experiments demonstrated an average accuracy of 93.55 ± 3.02% and an ITR of 51.88 ± 3.74 bits min-1.Significance. the performance of the proposed system has reached a relatively high level among the current user-friendly SSVEP-based BCI systems. This study successfully innovates the paradigm for SSVEP-based BCIs, offering new insights into the development of user-friendly systems that balance high performance and user comfort.

A multi-modal dataset of electroencephalography and functional near-infrared spectroscopy recordings for motor imagery of multi-types of joints from unilateral upper limb

As one of the important brain-computer interface (BCI) paradigms, motor imagery (MI) enables the control of external devices via identification of motor intention by decoding the features of Electroencephalography (EEG). Movement imagination of multi-types of joints from the same limb allows the development of more accurate and intuitive BCI systems. In this work, we reported an open dataset including EEG and functional near-infrared spectroscopy (fNIRS) recordings from 18 subjects performing eight MI tasks from four types of joints including hand open/close, wrist flexion/extension, wrist abduction/adduction, elbow pronation/supination, elbow flexion/extension, shoulder pronation/supination, shoulder abduction/adduction, and shoulder flexion/extension, resulting in a total of 5760 trials. The validity of multi-modal data was verified both from the EEG/fNIRS activation patterns and the classification performance. It is expected that this dataset will facilitate the development and innovation of decoding algorithms for MI of multi-types of joints based on multi-modal EEG-fNIRS data.

Graph convolution network-based eeg signal analysis: a review

With the advancement of artificial intelligence technology, more and more effective methods are being used to identify and classify Electroencephalography (EEG) signals to address challenges in healthcare and brain-computer interface fields. The applications and major achievements of Graph Convolution Network (GCN) techniques in EEG signal analysis are reviewed in this paper. Through an exhaustive search of the published literature, a module-by-module discussion is carried out for the first time to address the current research status of GCN. An exhaustive classification of methods and a systematic analysis of key modules, such as brain map construction, node feature extraction, and GCN architecture design, are presented. In addition, we pay special attention to several key research issues related to GCN. This review enhances the understanding of the future potential of GCN in the field of EEG signal analysis. At the same time, several valuable development directions are sorted out for researchers in related fields, such as analysing the applicability of different GCN layers, building task-oriented GCN models, and improving adaptation to limited data.

Data augmentation using masked principal component representation for deep learning-based SSVEP-BCIs

Objective.Data augmentation has been demonstrated to improve the classification accuracy of deep learning models in steady-state visual evoked potential-based brain-computer interfaces (BCIs), particularly when dealing with limited electroencephalography (EEG) data. However, current data augmentation methods often rely on signal-level manipulations, which may lead to significant distortion of EEG signals. To overcome this limitation, this study proposes a component-level data augmentation method called masked principal component representation (MPCR).Approach.MPCR utilizes a principal component-based reconstruction approach, integrating a random masking strategy applied to principal component representations. Specifically, certain principal components are randomly selected and set to zero, thereby introducing random perturbations in the reconstructed samples. Furthermore, reconstructing samples via linear combinations of the remaining components effectively preserves the primary inherent structure of EEG signals. By expanding the input space covered by training samples, MPCR helps the trained model learn more robust features. To validate the efficacy of MPCR, experiments are performed on two widely utilized public datasets.Main results.Experimental results indicate that MPCR substantially enhances classification accuracy across diverse deep learning models. Additionally, in comparison to various state-of-the-art data augmentation approaches, MPCR demonstrates both greater performance and high compatibility.Significance.This study proposes a simple yet effective component-level data augmentation method, contributing valuable insights for advancing data augmentation methods in EEG-based BCIs.

Virtual Electroencephalogram Acquisition: A Review on Electroencephalogram Generative Methods

Driven by the remarkable capabilities of machine learning, brain-computer interfaces (BCIs) are carving out an ever-expanding range of applications across a multitude of diverse fields. Notably, electroencephalogram (EEG) signals have risen to prominence as the most prevalently utilized signals within BCIs, owing to their non-invasive essence, exceptional portability, cost-effectiveness, and high temporal resolution. However, despite the significant strides made, the paucity of EEG data has emerged as the main bottleneck, preventing generalization of decoding algorithms. Taking inspiration from the resounding success of generative models in computer vision and natural language processing arenas, the generation of synthetic EEG data from limited recorded samples has recently garnered burgeoning attention. This paper undertakes a comprehensive and thorough review of the techniques and methodologies underpinning the generative models of the general EEG, namely the variational autoencoder (VAE), the generative adversarial network (GAN), and the diffusion model. Special emphasis is placed on their practical utility in augmenting EEG data. The structural designs and performance metrics of the different generative approaches in various application domains have been meticulously dissected and discussed. A comparative analysis of the strengths and weaknesses of each existing model has been carried out, and prospective avenues for future enhancement and refinement have been put forward.

Cross-Subject Motor Imagery Electroencephalogram Decoding with Domain Generalization

Decoding motor imagery (MI) electroencephalogram (EEG) signals in the brain-computer interface (BCI) can assist patients in accelerating motor function recovery. To realize the implementation of plug-and-play functionality for MI-BCI applications, cross-subject models are employed to alleviate time-consuming calibration and avoid additional model training for target subjects by utilizing EEG data from source subjects. However, the diversity in data distribution among subjects limits the model's robustness. In this study, we investigate a cross-subject MI-EEG decoding model with domain generalization based on a deep learning neural network that extracts domain-invariant features from source subjects. Firstly, a knowledge distillation framework is adopted to obtain the internally invariant representations based on spectral features fusion. Then, the correlation alignment approach aligns mutually invariant representations between each pair of sub-source domains. In addition, we use distance regularization on two kinds of invariant features to enhance generalizable information. To assess the effectiveness of our approach, experiments are conducted on the BCI Competition IV 2a and the Korean University dataset. The results demonstrate that the proposed model achieves 8.93% and 4.4% accuracy improvements on two datasets, respectively, compared with current state-of-the-art models, confirming that the proposed approach can effectively extract invariant features from source subjects and generalize to the unseen target distribution, hence paving the way for effective implementation of the plug-and-play functionality in MI-BCI applications.

Universal semantic feature extraction from EEG signals: a task-independent framework

Objective.Extracting universal, task-independent semantic features from electroencephalography (EEG) signals remains an open challenge. Traditional approaches are often task-specific, limiting their generalization across different EEG paradigms. This study aims to develop a robust, unsupervised framework for learning high-level, task-independent neural representations.Approach.We propose a novel framework integrating convolutional neural networks, AutoEncoders, and Transformers to extract both low-level spatiotemporal patterns and high-level semantic features from EEG signals. The model is trained in an unsupervised manner to ensure adaptability across diverse EEG paradigms, including motor imagery (MI), steady-state visually evoked potentials (SSVEPs), and event-related potentials (ERPs, specifically P300). Extensive analyses, including clustering, correlation, and ablation studies, are conducted to validate the quality and interpretability of the extracted features.Main results.Our method achieves state-of-the-art performance, with average classification accuracies of 83.50% and 84.84% on MI datasets (BCICIV_2a and BCICIV_2b), 98.41% and 99.66% on SSVEP datasets (Lee2019-SSVEP and Nakanishi2015), and an average AUC of 91.80% across eight ERP datasets. t-distributed stochastic neighbor embedding and clustering analyses reveal that the extracted features exhibit enhanced separability and structure compared to raw EEG data. Correlation studies confirm the framework's ability to balance universal and subject-specific features, while ablation results highlight the near-optimality of the selected model configuration.Significance.This work establishes a universal framework for task-independent semantic feature extraction from EEG signals, bridging the gap between conventional feature engineering and modern deep learning methods. By providing robust, generalizable representations across diverse EEG paradigms, this approach lays the foundation for advanced brain-computer interface applications, cross-task EEG analysis, and future developments in semantic EEG processing.

EEGUnity: Open-Source Tool in Facilitating Unified EEG Datasets Toward Large-Scale EEG Model

The increasing number of dispersed EEG dataset publications and the advancement of large-scale Electroencephalogram (EEG) models have increased the demand for practical tools to manage diverse EEG datasets. However, the inherent complexity of EEG data, characterized by variability in content data, metadata, and data formats, poses challenges for integrating multiple datasets and conducting large-scale EEG model research. To tackle the challenges, this paper introduces EEGUnity, an open-source tool that incorporates modules of "EEG Parser", "Correction", "Batch Processing", and "Large Language Model Boost". Leveraging the functionality of such modules, EEGUnity facilitates the efficient management of multiple EEG datasets, such as intelligent data structure inference, data cleaning, and data unification. In addition, the capabilities of EEGUnity ensure high data quality and consistency, providing a reliable foundation for large-scale EEG data research. EEGUnity is evaluated across 25 EEG datasets from different sources, offering several typical batch processing workflows. The results demonstrate the high performance and flexibility of EEGUnity in parsing and data processing. The project code is publicly available at github.com/Baizhige/EEGUnity.

Effects of Different Preprocessing Pipelines on Motor Imagery-Based Brain-Computer Interfaces

In recent years, brain-computer interfaces (BCIs) leveraging electroencephalography (EEG) signals for the control of external devices have garnered increasing attention. The information transfer rate of BCI has been significantly improved by a lot of cutting-edge methods. The exploration of effective preprocessing in brain-computer interfaces, particularly in terms of identifying suitable preprocessing methods and determining the optimal sequence for their application, remains an area ripe for further investigation. To address this gap, this study explores a range of preprocessing techniques, including but not limited to independent component analysis, surface Laplacian, bandpass filtering, and baseline correction, examining their potential contributions and synergies in the context of BCI applications. In this extensive research, a variety of preprocessing pipelines were rigorously tested across four EEG data sets, all of which were pertinent to motor imagery-based BCIs. These tests incorporated five EEG machine learning models, working in tandem with the preprocessing methods discussed earlier. The study's results highlighted that baseline correction and bandpass filtering consistently provided the most beneficial preprocessing effects. From the perspective of online deployment, after testing and time complexity analysis, this study recommends baseline correction, bandpass filtering and surface Laplace as more suitable for online implementation. An interesting revelation of the study was the enhanced effectiveness of the surface Laplacian algorithm when used alongside algorithms that focus on spatial information. Using appropriate processing algorithms, we can even achieve results (92.91% and 88.11%) that exceed the SOTA feature extraction methods in some cases. Such findings are instrumental in offering critical insights for the selection of effective preprocessing pipelines in EEG signal decoding. This, in turn, contributes to the advancement and refinement of brain-computer interface technologies.

SMANet: A Model Combining SincNet, Multi-Branch Spatial-Temporal CNN, and Attention Mechanism for Motor Imagery BCI

Building a brain-computer interface (BCI) based on motor imagery (MI) requires accurately decoding MI tasks, which poses a significant challenge due to individual discrepancy among subjects and low signal-to-noise ratio of EEG signals. We propose an end-to-end deep learning model, Sinc-multibranch-attention network (SMANet), which combines a SincNet, a multibranch spatial-temporal convolutional neural network (MBSTCNN), and an attention mechanism for MI-BCI classification. Firstly, Sinc convolution is utilized as a band-pass filter bank for data filtering; Second, pointwise convolution facilitates the effective integration of feature information across different frequency ranges, thereby enhancing the overall feature expression capability; Next, the resulting data are fed into the MBSTCNN to learn a deep feature representation. Thereafter, the outputs of the MBSTCNN are concatenated and then passed through an efficient channel attention (ECA) module to enhance local channel feature extraction and calibrate feature mapping. Ultimately, the feature maps yielded by ECA are classified using a fully connected layer. This model SMANet enhances discriminative features through a multi-objective optimization scheme that integrates cross-entropy loss and central loss. The experimental outcomes reveal that our model attains an average accuracy of 80.21% on the four-class MI dataset (BCI Competition IV 2a), 84.02% on the two-class MI dataset (BCI Competition IV 2b), and 72.70% on the two-class MI dataset (OpenBMI). These results are superior to those of the current state-of-the-art methods. The SMANet is capable to effectively decoding the spatial-spectral-temporal information of EEG signals, thereby enhancing the performance of MI-BCIs.

MCL-SWT: Mirror Contrastive Learning with Sliding Window Transformer for Subject-Independent EEG Recognition

Background: In brain-computer interfaces (BCIs), transformer-based models have found extensive application in motor imagery (MI)-based EEG signal recognition. However, for subject-independent EEG recognition, these models face challenges: low sensitivity to spatial dynamics of neural activity and difficulty balancing high temporal resolution features with manageable computational complexity. The overarching objective is to address these critical issues. Methods: We introduce Mirror Contrastive Learning with Sliding Window Transformer (MCL-SWT). Inspired by left/right hand motor imagery inducing event-related desynchronization (ERD) in the contralateral sensorimotor cortex, we develop a mirror contrastive loss function. It segregates feature spaces of EEG signals from contralateral ERD locations while curtailing variability in signals sharing similar ERD locations. The Sliding Window Transformer computes self-attention scores over high temporal resolution features, enabling efficient capture of global temporal dependencies. Results: Evaluated on benchmark datasets for subject-independent MI EEG recognition, MCL-SWT achieves classification accuracies of 66.48% and 75.62%, outperforming State-of-the-Art models by 2.82% and 2.17%, respectively. Ablation studies validate the efficacy of both the mirror contrastive loss and sliding window mechanism. Conclusions: These findings underscore MCL-SWT's potential as a robust, interpretable framework for subject-independent EEG recognition. By addressing existing challenges, MCL-SWT could significantly advance BCI technology development.

Multi-view collaborative ensemble classification for EEG signals based on 3D second-order difference plot and CSP

Objective.EEG signal analysis methods based on electrical source imaging (ESI) technique have significantly improved classification accuracy and response time. However, for the refined and informative source signals, the current studies have not fully considered their dynamic variability in feature extraction and lacked an effective integration of their dynamic variability and spatial characteristics. Additionally, the adaptability and complementarity of classifiers have not been considered comprehensively. These two aspects lead to the issue of insufficient decoding of source signals, which still limits the application of brain-computer interface (BCI). To address these challenges, this paper proposes a multi-view collaborative ensemble classification method for EEG signals based on three-dimensional second-order difference plot (3D SODP) and common spatial pattern.Approach.First, EEG signals are mapped to the source domain using the ESI technique, and then the source signals in the region of interest are obtained. Next, features from three viewpoints of the source signals are extracted, including 3D SODP features, spatial features, and the weighted fusion of both. Finally, the extracted multi-view features are integrated with subject-specific sub-classifier combination, and a voting mechanism is used to determine the final classification.Main results.The results show that the proposed method achieves classification accuracy of 81.3% and 82.6% respectively in two sessions of the OpenBMI dataset, which is nearly 5% higher than the state-of-the-art method, and maintains the analysis response time required for online BCI.Significance.This paper employs multi-view feature extraction to fully capture the characteristics of the source signals and enhances feature utilization through collaborative ensemble classification. The results demonstrate high accuracy and robust performance, providing a novel approach for online BCI.

Multi-scale convolutional transformer network for motor imagery brain-computer interface

Brain-computer interface (BCI) systems allow users to communicate with external devices by translating neural signals into real-time commands. Convolutional neural networks (CNNs) have been effectively utilized for decoding motor imagery electroencephalography (MI-EEG) signals in BCIs. However, traditional CNN-based methods face challenges such as individual variability in EEG signals and the limited receptive fields of CNNs. This study presents the Multi-Scale Convolutional Transformer (MSCFormer) model that integrates multiple CNN branches for multi-scale feature extraction and a Transformer module to capture global dependencies, followed by a fully connected layer for classification. The multi-branch multi-scale CNN structure effectively addresses individual variability in EEG signals, enhancing the model's generalization capabilities, while the Transformer encoder strengthens global feature integration and improves decoding performance. Extensive experiments on the BCI IV-2a and IV-2b datasets show that MSCFormer achieves average accuracies of 82.95% (BCI IV-2a) and 88.00% (BCI IV-2b), with kappa values of 0.7726 and 0.7599 in five-fold cross-validation, surpassing several state-of-the-art methods. These results highlight MSCFormer's robustness and accuracy, underscoring its potential in EEG-based BCI applications. The code has been released in https://github.com/snailpt/MSCFormer .

Nature documentaries vs. quiet rest: no evidence for an impact on event-related desynchronization during motor imagery and neurofeedback

Motor imagery (MI) in combination with neurofeedback (NF) has emerged as a promising approach in motor neurorehabilitation, facilitating brain activity modulation and promoting motor learning. Although MI-NF has been demonstrated to enhance motor performance and cortical plasticity, its efficacy varies considerably across individuals. Various context factors have been identified as influencing neurophysiological outcomes in motor execution and MI, however, their specific impact on event-related desynchronization (ERD), a key neurophysiological marker in NF, remains insufficiently understood. Previous research suggested that declarative interference following MI-NF may serve as a context factor hindering the progression of ERD. Yet, no significant changes in ERD within the mu and beta (8-30 Hz) frequency bands were observed across blocks in either a declarative interference or a control condition. This raises the question of whether the absence of ERD modulation could be attributed to the break task that was common to both declarative interference and control condition: watching nature documentaries immediately after MI blocks. To investigate this, we conducted a follow-up study replicating the original methodology while collecting new data. We compared NF-MI-ERD between groups with and without nature documentaries as a post-MI condition. Participants completed three sessions of kinesthetic MI-NF training involving a finger-tapping task over two consecutive days, with quiet rest as the post-MI condition (group quiet rest). 64-channel EEG data were analyzed from 17 healthy participants (8 females, 18-35 years, M and SD: 25.2 ± 4.2 years). Data were compared to a previously recorded dataset (group documentaries), in which 17 participants (10 females, 23-32 years, M and SD: 25.8 ± 2.5 years) watched nature documentaries after MI blocks. The results showed no significant main effects for blocks or group, though a session-by-group interaction was observed. Post-hoc tests, however, did not reveal significant differences in ERD development between the groups across individual blocks. These findings do not provide evidence that nature documentaries used as a post-MI condition negatively affect across-block development of NF-MI-ERD. This study highlights the importance of exploring additional context factors in MI-NF training to better understand their influence on ERD development.

The Riemannian Means Field Classifier for EEG-Based BCI Data

: A substantial amount of research has demonstrated the robustness and accuracy of the Riemannian minimum distance to mean (MDM) classifier for all kinds of EEG-based brain-computer interfaces (BCIs). This classifier is simple, fully deterministic, robust to noise, computationally efficient, and prone to transfer learning. Its training is very simple, requiring just the computation of a geometric mean of a symmetric positive-definite (SPD) matrix per class. We propose an improvement of the MDM involving a number of power means of SPD matrices instead of the sole geometric mean. By the analysis of 20 public databases, 10 for the motor-imagery BCI paradigm and 10 for the P300 BCI paradigm, comprising 587 individuals in total, we show that the proposed classifier clearly outperforms the MDM, approaching the state-of-the art in terms of performance while retaining the simplicity and the deterministic behavior. In order to promote reproducible research, our code will be released as open source.

EEG-DG: A Multi-Source Domain Generalization Framework for Motor Imagery EEG Classification

Motorimagery EEG classification plays a crucial role in non-invasive Brain-Computer Interface (BCI) research. However, the performance of classification is affected by the non-stationarity and individual variations of EEG signals. Simply pooling EEG data with different statistical distributions to train a classification model can severely degrade the generalization performance. To address this issue, the existing methods primarily focus on domain adaptation, which requires access to the test data during training. This is unrealistic and impractical in many EEG application scenarios. In this paper, we propose a novel multi-source domain generalization framework called EEG-DG, which leverages multiple source domains with different statistical distributions to build generalizable models on unseen target EEG data. We optimize both the marginal and conditional distributions to ensure the stability of the joint distribution across source domains and extend it to a multi-source domain generalization framework to achieve domain-invariant feature representation, thereby alleviating calibration efforts. Systematic experiments conducted on a simulative dataset, BCI competition IV 2a, 2b, and OpenBMI datasets, demonstrate the superiority and competitive performance of our proposed framework over other state-of-the-art methods. Specifically, EEG-DG achieves average classification accuracies of 81.79% and 87.12% on datasets IV-2a and IV-2b, respectively, and 78.37% and 76.94% for inter-session and inter-subject evaluations on dataset OpenBMI, which even outperforms some domain adaptation methods.

Flexible Patched Brain Transformer model for EEG decoding

Decoding the human brain using non-invasive methods is a significant challenge. This study aims to enhance electroencephalography (EEG) decoding by developing of machine learning methods. Specifically, we propose the novel, attention-based Patched Brain Transformer model to achieve this goal. The model exhibits flexibility regarding the number of EEG channels and recording duration, enabling effective pre-training across diverse datasets. We investigate the effect of data augmentation methods and pre-training on the training process. To gain insights into the training behavior, we incorporate an inspection of the architecture. We compare our model with state-of-the-art models and demonstrate superior performance using only a fraction of the parameters. The results are achieved with supervised pre-training, coupled with time shifts as data augmentation for multi-participant classification on motor imagery datasets.

A multi-day and high-quality EEG dataset for motor imagery brain-computer interface

A key challenge in developing a robust electroencephalography (EEG)-based brain-computer interface (BCI) is obtaining reliable classification performance across multiple days. In particular, EEG-based motor imagery (MI) BCI faces large variability and low signal-to-noise ratio. To address these issues, collecting a large and reliable dataset is critical for learning of cross-session and cross-subject patterns while mitigating EEG signals inherent instability. In this study, we obtained a comprehensive MI dataset from the 2019 World Robot Conference Contest-BCI Robot Contest. We collected EEG data from 62 healthy participants across three recording sessions. This experiment includes two paradigms: (1) two-class tasks: left and right hand-grasping, (2) three-class tasks: left and right hand-grasping, and foot-hooking. The dataset comprises raw data, and preprocessed data. For the two-class data, an average classification accuracy of 85.32% was achieved using EEGNet, while the three-class data achieved an accuracy of 76.90% using deepConvNet. Different researchers can reuse the dataset according to their needs. We hope that this dataset will significantly advance MI-BCI research, particularly in addressing cross-session and cross-subject challenges.

A New Canonical Log-Euclidean Kernel for Symmetric Positive Definite Matrices for EEG Analysis (Oct 2024)

OBJECTIVE

Working with the Riemannian manifold of symmetric positive-definite (SPD) matrices has become popular in electroencephalography (EEG) analysis. Frequently selected for its speed property is the manifold geometry provided by the log-euclidean Riemannian metric. However, the kernels used in the log-euclidean framework are not canonically based on the underlying geometry. Therefore, we introduce a new canonical log-euclidean (CLE) kernel.

METHODS

We used the log-euclidean metric tensor on the SPD manifold to derive the CLE kernel. We compared it with existing kernels, namely the affine-invariant, log-euclidean, and Gaussian log-euclidean kernel. For comparison, we tested the kernels on two paradigms: classification and dimensionality reduction. Each paradigm was evaluated on five open-access brain-computer interface datasets with motor-imagery tasks across multiple sessions. Performance was measured as balanced classification accuracy using a leave-one-session-out cross-validation. Dimensionality reduction performance was measured using AUClogRNX.

RESULTS

The CLE kernel itself is simple and easily turned into code, which is provided in addition to all the analytical solutions to relevant equations in the log-euclidean framework. The CLE kernel significantly outperformed existing log-euclidean kernels in classification tasks and was several times faster than the affine-invariant kernel for most datasets.

CONCLUSION

We found that adhering to the geometrical structure significantly improves the accuracy over two commonly used log-euclidean kernels while keeping the speed advantages of the log-euclidean framework.

SIGNIFICANCE

The CLE provides a good choice as a kernel in time-critical applications and fills a gap in the kernel methods of the log-euclidean framework.

Effect of natural and synthetic noise data augmentation on physical action classification by brain-computer interface and deep learning

Analysis of electroencephalography (EEG) signals gathered by brain-computer interface (BCI) recently demonstrated that deep neural networks (DNNs) can be effectively used for investigation of time sequences for physical actions (PA) classification. In this study, the relatively simple DNN with fully connected network (FCN) components and convolutional neural network (CNN) components was considered to classify finger-palm-hand manipulations each from the grasp-and-lift (GAL) dataset. The main aim of this study was to imitate and investigate environmental influence by the proposed noise data augmentation (NDA) of two kinds: (i) natural NDA by inclusion of noise EEG data from neighboring regions by increasing the sampling size N and the different offset values for sample labeling and (ii) synthetic NDA by adding the generated Gaussian noise. The natural NDA by increasing N leads to the higher micro and macro area under the curve (AUC) for receiver operating curve values for the bigger N values than usage of synthetic NDA. The detrended fluctuation analysis (DFA) was applied to investigate the fluctuation properties and calculate the correspondent Hurst exponents H for the quantitative characterization of the fluctuation variability. H values for the low time window scales (< 2 s) are higher in comparison with ones for the bigger time window scales. For example, H more than 2-3 times higher for some PAs, i.e., it means that the shorter EEG fragments (< 2 s) demonstrate the scaling behavior of the higher complexity than the longer fragments. As far as these results were obtained by the relatively small DNN with the low resource requirements, this approach can be promising for porting such models to Edge Computing infrastructures on devices with the very limited computational resources.

Latent alignment in deep learning models for EEG decoding

Objective. Brain-computer interfaces (BCIs) face a significant challenge due to variability in electroencephalography (EEG) signals across individuals. While recent approaches have focused on standardizing input signal distributions, we propose that aligning distributions in the deep learning model's feature space is more effective for classification.Approach. We introduce the Latent Alignment method, which won the Benchmarks for EEG Transfer Learning competition. This method can be formulated as a deep set architecture applied to trials from a given subject, introducing deep sets to EEG decoding for the first time. We compare Latent Alignment to recent statistical domain adaptation techniques, carefully considering class-discriminative artifacts and the impact of class distributions on classification performance.Main results. Our experiments across motor imagery, sleep stage classification, and P300 event-related potential tasks validate Latent Alignment's effectiveness. We identify a trade-off between improved classification accuracy when alignment is performed at later modeling stages and increased susceptibility to class imbalance in the trial set used for statistical computation.Significance. Latent Alignment offers consistent improvements to subject-independent deep learning models for EEG decoding when relevant practical considerations are addressed. This work advances our understanding of statistical alignment techniques in EEG decoding and provides insights for their effective implementation in real-world BCI applications, potentially facilitating broader use of BCIs in healthcare, assistive technologies, and beyond. The model code is available athttps://github.com/StylianosBakas/LatentAlignment.

TFTL: A Task-Free Transfer Learning Strategy for EEG-Based Cross-Subject and Cross-Dataset Motor Imagery BCI

OBJECTIVE

Motor imagery-based brain-computer interfaces (MI-BCIs) have been playing an increasingly vital role in neural rehabilitation. However, the long-term task-based calibration required for enhanced model performance leads to an unfriendly user experience, while the inadequacy of EEG data hinders the performance of deep learning models. To address these challenges, a task-free transfer learning strategy (TFTL) for EEG-based cross-subject & cross-dataset MI-BCI is proposed for calibration time reduction and multi-center data co-modeling.

METHODS

TFTL strategy consists of data alignment, shared feature extractor, and specific classifiers, in which the label predictor for MI tasks classification, as well as domain and dataset discriminator for inter-subject variability reduction are concurrently optimized for knowledge transfer from subjects across different datasets to the target subject. Moreover, only resting data of the target subject is used for subject-specific model construction to achieve task-free.

RESULTS

We employed three deep learning methods (ShallowConvNet, EEGNet, and TCNet-Fusion) as baseline approaches to evaluate the effectiveness of the proposed strategy on five datasets (BCIC IV Dataset 2a, Dataset 1, Physionet MI, Dreyer 2023, and OpenBMI). The results demonstrate a significant improvement with the inclusion of the TFTL strategy compared to the baseline methods, reaching a maximum enhancement of 15.67% with a statistical significance (p = 2.4e-5 < 0.05). Moreover, task-free resulted in MI trials needed for calibration being 0 for all datasets, which significantly alleviated the calibration burden for patients before usage.

CONCLUSION/SIGNIFICANCE

The proposed TFTL strategy effectively addresses challenges posed by prolonged calibration periods and insufficient EEG data, thus promoting MI-BCI from laboratory to clinical application.

A protocol for trustworthy EEG decoding with neural networks

Deep learning solutions have rapidly emerged for EEG decoding, achieving state-of-the-art performance on a variety of decoding tasks. Despite their high performance, existing solutions do not fully address the challenge posed by the introduction of many hyperparameters, defining data pre-processing, network architecture, network training, and data augmentation. Automatic hyperparameter search is rarely performed and limited to network-related hyperparameters. Moreover, pipelines are highly sensitive to performance fluctuations due to random initialization, hindering their reliability. Here, we design a comprehensive protocol for EEG decoding that explores the hyperparameters characterizing the entire pipeline and that includes multi-seed initialization for providing robust performance estimates. Our protocol is validated on 9 datasets about motor imagery, P300, SSVEP, including 204 participants and 26 recording sessions, and on different deep learning models. We accompany our protocol with extensive experiments on the main aspects influencing it, such as the number of participants used for hyperparameter search, the split into sequential simpler searches (multi-step search), the use of informed vs. non-informed search algorithms, and the number of random seeds for obtaining stable performance. The best protocol included 2-step hyperparameter search via an informed search algorithm, with the final training and evaluation performed using 10 random initializations. The optimal trade-off between performance and computational time was achieved by using a subset of 3-5 participants for hyperparameter search. Our protocol consistently outperformed baseline state-of-the-art pipelines, widely across datasets and models, and could represent a standard approach for neuroscientists for decoding EEG in a trustworthy and reliable way.

Aperiodic Component Analysis in Quantification of Steady-State Visually Evoked Potentials

OBJECTIVE

In this study, we aimed to investigate whether and how the aperiodic component in electroencephalograms affects different quantitative processes of steady-state visually evoked potentials and the performance of corresponding brain-computer interfaces.

METHODS

We applied the Fitting Oscillations & One-Over-F method to parameterize power spectra as a combination of periodic oscillations and an aperiodic component. Electroencephalographic responses and system performance were measured and compared using four prevailing methods: power spectral density analysis, canonical correlation analysis, filter bank canonical correlation analysis and the state-of-the-art method, task discriminant component analysis.

RESULTS

We found that controlling for the aperiodic component prominently downgraded the performance of brain-computer interfaces measured by canonical correlation analysis (94.9% to 82.8%), filter bank canonical correlation analysis (94.1% to 87.6%), and task discriminant component analysis (96.5% to 70.3%). However, it had almost no effect on that measured by power spectral density analysis (80.4% to 78.7%). This was accompanied by a differential aperiodic impact between power spectral density analysis and the other three methods on the differentiation of the target and non-target stimuli.

CONCLUSION

The aperiodic component distinctly impacts the quantification of steady-state visually evoked potentials and the performance of corresponding brain-computer interfaces.

SIGNIFICANCE

Our work underscores the significance of taking into account the dynamic nature of aperiodic activities in research related to the quantification of steady-state visually evoked potentials.

A multi-branch, multi-scale, and multi-view CNN with lightweight temporal attention mechanism for EEG-based motor imagery decoding

Convolutional neural networks (CNNs) have been widely utilized for decoding motor imagery (MI) from electroencephalogram (EEG) signals. However, extracting discriminative spatial-temporal-spectral features from low signal-to-noise ratio EEG signals remains challenging. This paper proposes MBMSNet , a multi-branch, multi-scale, and multi-view CNN with a lightweight temporal attention mechanism for EEG-Based MI decoding. Specifically, MBMSNet first extracts multi-view representations from raw EEG signals, followed by independent branches to capture spatial, spectral, temporal-spatial, and temporal-spectral features. Each branch includes a domain-specific convolutional layer, a variance layer, and a temporal attention layer. Finally, the features derived from each branch are concatenated with weights and classified through a fully connected layer. Experiments demonstrate MBMSNet outperforms state-of-the-art models, achieving accuracies of 84.60% on BCI Competition IV 2a, 87.80% on 2b, and 74.58% on OpenBMI, showcasing its potential for robust BCI applications.

Supervised Contrastive Learning-Based Domain Generalization Network for Cross-Subject Motor Decoding

Developing an electroencephalogram (EEG)-based motor imagery and motor execution (MI/ME) decoding system that is both highly accurate and calibration-free for cross-subject applications remains challenging due to domain shift problem inherent in such scenario. Recent research has increasingly embraced transfer learning strategies, especially domain adaptation techniques. However, domain adaptation becomes impractical when the target subject data is either difficult to obtain or unknown. To address this issue, we propose a supervised contrastive learning-based domain generalization network (SCLDGN) for cross-subject MI/ME decoding. Firstly, the feature encoder is purposefully designed to learn the EEG discriminative feature representations. Secondly, the domain alignment based on deep correlation alignment constrains the representations distance across various domains to learn domain-invariant features. In addition, the class regularization block is proposed, where the supervised contrastive learning with domain-agnostic mixup is established to learn the class-relevant features and achieve class-level alignment. Finally, in the latent space, clusters of domain-agnostic representations from the same class are mapped closer together. Consequently, SCLDGN is capable of learning domain-invariant and class-relevant discriminative representations, which are essential for effective cross-subject decoding. Extensive experiments conducted on six MI/ME datasets demonstrate the effectiveness of the proposed method in comparison with other state-of-the-art approaches. Furthermore, ablation study and visualization analyses explain the generalization mechanism of the proposed method and also show neurophysiologically meaningful patterns related to MI/ME.

Improving classification performance of motor imagery BCI through EEG data augmentation with conditional generative adversarial networks

In brain-computer interface (BCI), building accurate electroencephalogram (EEG) classifiers for specific mental tasks is critical for BCI performance. The classifiers are developed by machine learning (ML) and deep learning (DL) techniques, requiring a large dataset for training to build reliable and accurate models. However, collecting large enough EEG datasets is difficult due to intra-/inter-subject variabilities and experimental costs. This leads to the data scarcity problem, which causes overfitting issues to training samples, resulting in reducing generalization performance. To solve the EEG data scarcity problem and improve the performance of the EEG classifiers, we propose a novel EEG data augmentation (DA) framework using conditional generative adversarial networks (cGANs). An experimental study is implemented with two public EEG datasets, including motor imagery (MI) tasks (BCI competition IV IIa and III IVa), to validate the effectiveness of the proposed EEG DA method for the EEG classifiers. To evaluate the proposed cGAN-based DA method, we tested eight EEG classifiers for the experiment, including traditional MLs and state-of-the-art DLs with three existing EEG DA methods. Experimental results showed that most DA methods with proper DA proportion in the training dataset had higher classification performances than without DA. Moreover, applying the proposed DA method showed superior classification performance improvement than the other DA methods. This shows that the proposed method is a promising EEG DA method for enhancing the performances of the EEG classifiers in MI-based BCIs.

Attention-Based Multimodal tCNN for Classification of Steady-State Visual Evoked Potentials and Its Application to Gripper Control

The classification problem for short time-window steady-state visual evoked potentials (SSVEPs) is important in practical applications because shorter time-window often means faster response speed. By combining the advantages of the local feature learning ability of convolutional neural network (CNN) and the feature importance distinguishing ability of attention mechanism, a novel network called AttentCNN is proposed to further improve the classification performance for short time-window SSVEP. Considering the frequency-domain features extracted from short time-window signals are not obvious, this network starts with the time-domain feature extraction module based on the filter bank (FB). The FB consists of four sixth-order Butterworth filters with different bandpass ranges. Then extracted multimodal features are aggregated together. The second major module is a set of residual squeeze and excitation blocks (RSEs) that has the ability to improve the quality of extracted features by learning the interdependence between features. The final major module is time-domain CNN (tCNN) that consists of four CNNs for further feature extraction and followed by a fully connected (FC) layer for output. Our designed networks are validated over two large public datasets, and necessary comparisons are given to verify the effectiveness and superiority of the proposed network. In the end, in order to demonstrate the application potential of the proposed strategy in the medical rehabilitation field, we design a novel five-finger bionic hand and connect it to our trained network to achieve the control of bionic hand by human brain signals directly. Our source codes are available on Github: https://github.com/JiannanChen/AggtCNN.git.

Multiband Convolutional Riemannian Network With Band-Wise Riemannian Triplet Loss for Motor Imagery Classification

This paper presents a novel motor imagery classification algorithm that uses an overlapping multiscale multiband convolutional Riemannian network with band-wise Riemannian triplet loss to improve classification performance. Despite the superior performance of the Riemannian approach over the common spatial pattern filter approach, deep learning methods that generalize the Riemannian approach have received less attention. The proposed algorithm develops a state-of-the-art multiband Riemannian network that reduces the potential overfitting problem of Riemannian networks, a drawback of Riemannian networks due to their inherent large feature dimension from covariance matrix, by using fewer subbands with discriminative frequency diversity, by inserting convolutional layers before computing the subband covariance matrix, and by regularizing subband networks with Riemannian triplet loss. The proposed method is evaluated using the publicly available datasets, the BCI Competition IV dataset 2a and the OpenBMI dataset. The experimental results confirm that the proposed method improves performance, in particular achieving state-of-the-art classification accuracy among the currently studied Riemannian networks.

Attention Induced Dual Convolutional-Capsule Network (AIDC-CN): A deep learning framework for motor imagery classification

In recent times, Electroencephalography (EEG)-based motor imagery (MI) decoding has garnered significant attention due to its extensive applicability in healthcare, including areas such as assistive robotics and rehabilitation engineering. Nevertheless, the decoding of EEG signals presents considerable challenges owing to their inherent complexity, non-stationary characteristics, and low signal-to-noise ratio. Notably, deep learning-based classifiers have emerged as a prominent focus for addressing the EEG signal decoding process. This study introduces a novel deep learning classifier named the Attention Induced Dual Convolutional-Capsule Network (AIDC-CN) with the specific aim of accurately categorizing various motor imagination class labels. To enhance the classifier's performance, a dual feature extraction approach leveraging spectrogram and brain connectivity networks has been employed, diversifying the feature set in the classification task. The main highlights of the proposed AIDC-CN classifier includes the introduction of a dual convolution layer to handle the brain connectivity and spectrogram features, addition of a novel self-attention module (SAM) to accentuate the relevant parts of the convolved spectrogram features, introduction of a new cross-attention module (CAM) to refine the outputs obtained from the dual convolution layers and incorporation of a Gaussian Error Linear Unit (GELU) based dynamic routing algorithm to strengthen the coupling among the primary and secondary capsule layers. Performance analysis undertaken on four public data sets depict the superior performance of the proposed model with respect to the state-of-the-art techniques. The code for this model is available at https://github.com/RiteshSurChowdhury/AIDC-CN.

MSVTNet: Multi-Scale Vision Transformer Neural Network for EEG-Based Motor Imagery Decoding

OBJECT

Transformer-based neural networks have been applied to the electroencephalography (EEG) decoding for motor imagery (MI). However, most networks focus on applying the self-attention mechanism to extract global temporal information, while the cross-frequency coupling features between different frequencies have been neglected. Additionally, effectively integrating different neural networks poses challenges for the advanced design of decoding algorithms.

METHODS

This study proposes a novel end-to-end Multi-Scale Vision Transformer Neural Network (MSVTNet) for MI-EEG classification. MSVTNet first extracts local spatio-temporal features at different filtered scales through convolutional neural networks (CNNs). Then, these features are concatenated along the feature dimension to form local multi-scale spatio-temporal feature tokens. Finally, Transformers are utilized to capture cross-scale interaction information and global temporal correlations, providing more distinguishable feature embeddings for classification. Moreover, auxiliary branch loss is leveraged for intermediate supervision to ensure the effective integration of CNNs and Transformers.

RESULTS

The performance of MSVTNet was assessed through subject-dependent (session-dependent and session-independent) and subject-independent experiments on three MI datasets, i.e., the BCI competition IV 2a, 2b and OpenBMI datasets. The experimental results demonstrate that MSVTNet achieves state-of-the-art performance in all analyses.

CONCLUSION

MSVTNet shows superiority and robustness in enhancing MI decoding performance.

MP: A steady-state visual evoked potential dataset based on multiple paradigms

In the field of steady-state visual evoked potential (SSVEP), stimulus paradigms are regularly arranged or mimic the style of a keyboard with the same size. However, stimulation paradigms have important effects on the performance of SSVEP systems, which correlate with the electroencephalogram (EEG) signal amplitude and recognition accuracy. This paper provides MP dataset that was acquired using a 12-target BCI speller. MP dataset contains 9-channel EEG signals from the occipital region of 24 subjects under 5 stimulation paradigms with different stimulus sizes and arrangements. Stimuli were encoded using joint frequency and phase modulation (JFPM) method. Subjects completed an offline prompted spelling task using a speller under 5 paradigms. Each experiment contains 8 blocks, and each block contains 12 trials. Designers can use this dataset to test the performance of algorithms considering "stimulus size" and "stimulus arrangement". EEG data showed SSVEP features through amplitude-frequency analysis. FBCCA and TRCA confirmed its suitability.

Motor task-to-task transfer learning for motor imagery brain-computer interfaces

Motor imagery (MI) is one of the popular control paradigms in the non-invasive brain-computer interface (BCI) field. MI-BCI generally requires users to conduct the imagination of movement (e.g., left or right hand) to collect training data for generating a classification model during the calibration phase. However, this calibration phase is generally time-consuming and tedious, as users conduct the imagination of hand movement several times without being given feedback for an extended period. This obstacle makes MI-BCI non user-friendly and hinders its use. On the other hand, motor execution (ME) and motor observation (MO) are relatively easier tasks, yield lower fatigue than MI, and share similar neural mechanisms to MI. However, few studies have integrated these three tasks into BCIs. In this study, we propose a new task-to-task transfer learning approach of 3-motor tasks (ME, MO, and MI) for building a better user-friendly MI-BCI. For this study, 28 subjects participated in 3-motor tasks experiment, and electroencephalography (EEG) was acquired. User opinions regarding the 3-motor tasks were also collected through questionnaire survey. The 3-motor tasks showed a power decrease in the alpha rhythm, known as event-related desynchronization, but with slight differences in the temporal patterns. In the classification analysis, the cross-validated accuracy (within-task) was 67.05 % for ME, 65.93 % for MI, and 73.16 % for MO on average. Consistently with the results, the subjects scored MI (3.16) as the most difficult task compared with MO (1.42) and ME (1.41), with p < 0.05. In the analysis of task-to-task transfer learning, where training and testing are performed using different task datasets, the ME-trained model yielded an accuracy of 65.93 % (MI test), which is statistically similar to the within-task accuracy (p > 0.05). The MO-trained model achieved an accuracy of 60.82 % (MI test). On the other hand, combining two datasets yielded interesting results. ME and 50 % of the MI-trained model (50-shot) classified MI with a 69.21 % accuracy, which outperformed the within-task accuracy (p < 0.05), and MO and 50 % of the MI-trained model showed an accuracy of 66.75 %. Of the low performers with a within-task accuracy of 70 % or less, 90 % (n = 21) of the subjects improved in training with ME, and 76.2 % (n = 16) improved in training with MO on the MI test at 50-shot. These results demonstrate that task-to-task transfer learning is possible and could be a promising approach to building a user-friendly training protocol in MI-BCI.

Unsupervised and semi-supervised domain adaptation networks considering both global knowledge and prototype-based local class information for Motor Imagery Classification

The non-stationarity of EEG signals results in variability across sessions, impeding model building and data sharing. In this paper, we propose a domain adaptation method called GPL, which simultaneously considers global knowledge and prototype-based local class information to enhance the classification accuracy of motor imagery signals. Depending on the amount of labeled data available in the target domain, the method is implemented in both unsupervised and semi-supervised versions. Specifically, at the global level, we employ the maximum mean difference (MMD) loss to globally constrain the feature space, achieving comprehensive alignment. In the context of class-level operations, we propose two memory banks designed to accommodate class prototypes in each domain and constrain feature embeddings by applying two prototype-based contrastive losses. The source contrastive loss is used to organize source features spatially based on categories, thereby reconciling inter-class and intra-class relationships, while the interactive contrastive loss is employed to facilitate cross-domain information interaction. Simultaneously, in unsupervised scenarios, to mitigate the adverse effects of excessive pseudo-labels, we introduce an entropy-aware strategy that dynamically evaluates the confidence level of target data and personalized constraints on the participation of interactive contrastive loss. To validate our approach, extensive experiments were conducted on a highly regarded public EEG dataset, namely Dataset IIa of the BCI Competition IV, as well as a large-scale EEG dataset called GigaDB. The experiments yielded average classification accuracies of 86.03% and 84.22% respectively. These results demonstrate that our method is an effective EEG decoding model, conducive to advancing the development of motor imagery brain-computer interfaces. The architecture proposed in this study and the code for data partitioning can be found at https://github.com/zhangdx21/GPL.

SpeechBrain-MOABB: An open-source Python library for benchmarking deep neural networks applied to EEG signals

Deep learning has revolutionized EEG decoding, showcasing its ability to outperform traditional machine learning models. However, unlike other fields, EEG decoding lacks comprehensive open-source libraries dedicated to neural networks. Existing tools (MOABB and braindecode) prevent the creation of robust and complete decoding pipelines, as they lack support for hyperparameter search across the entire pipeline, and are sensitive to fluctuations in results due to network random initialization. Furthermore, the absence of a standardized experimental protocol exacerbates the reproducibility crisis in the field. To address these limitations, we introduce SpeechBrain-MOABB, a novel open-source toolkit carefully designed to facilitate the development of a comprehensive EEG decoding pipeline based on deep learning. SpeechBrain-MOABB incorporates a complete experimental protocol that standardizes critical phases, such as hyperparameter search and model evaluation. It natively supports multi-step hyperparameter search for finding the optimal hyperparameters in a high-dimensional space defined by the entire pipeline, and multi-seed training and evaluation for obtaining performance estimates robust to the variability caused by random initialization. SpeechBrain-MOABB outperforms other libraries, including MOABB and braindecode, with accuracy improvements of 14.9% and 25.2% (on average), respectively. By enabling easy-to-use and easy-to-share decoding pipelines, our toolkit can be exploited by neuroscientists for decoding EEG with neural networks in a replicable and trustworthy way.

IMH-Net: a convolutional neural network for end-to-end EEG motor imagery classification

As the main component of Brain-computer interface (BCI) technology, the classification algorithm based on EEG has developed rapidly. The previous algorithms were often based on subject-dependent settings, resulting in BCI needing to be calibrated for new users. In this work, we propose IMH-Net, an end-to-end subject-independent model. The model first uses Inception blocks extracts the frequency domain features of the data, then further compresses the feature vectors to extract the spatial domain features, and finally learns the global information and classification through Multi-Head Attention mechanism. On the OpenBMI dataset, IMH-Net obtained 73.90 ± 13.10% accuracy and 73.09 ± 14.99% F1-score in subject-independent manner, which improved the accuracy by 1.96% compared with the comparison model. On the BCI competition IV dataset 2a, this model also achieved the highest accuracy and F1-score in subject-dependent manner. The IMH-Net model we proposed can improve the accuracy of subject-independent Motor Imagery (MI), and the robustness of the algorithm is high, which has strong practical value in the field of BCI.

Leveraging temporal dependency for cross-subject-MI BCIs by contrastive learning and self-attention

Brain-computer interfaces (BCIs) built based on motor imagery paradigm have found extensive utilization in motor rehabilitation and the control of assistive applications. However, traditional MI-BCI systems often exhibit suboptimal classification performance and require significant time for new users to collect subject-specific training data. This limitation diminishes the user-friendliness of BCIs and presents significant challenges in developing effective subject-independent models. In response to these challenges, we propose a novel subject-independent framework for learning temporal dependency for motor imagery BCIs by Contrastive Learning and Self-attention (CLS). In CLS model, we incorporate self-attention mechanism and supervised contrastive learning into a deep neural network to extract important information from electroencephalography (EEG) signals as features. We evaluate the CLS model using two large public datasets encompassing numerous subjects in a subject-independent experiment condition. The results demonstrate that CLS outperforms six baseline algorithms, achieving a mean classification accuracy improvement of 1.3 % and 4.71 % than the best algorithm on the Giga dataset and OpenBMI dataset, respectively. Our findings demonstrate that CLS can effectively learn invariant discriminative features from training data obtained from non-target subjects, thus showcasing its potential for building models for new users without the need for calibration.

Short-length SSVEP data extension by a novel generative adversarial networks based framework

Steady-state visual evoked potentials (SSVEPs) based brain-computer interface (BCI) has received considerable attention due to its high information transfer rate (ITR) and available quantity of targets. However, the performance of frequency identification methods heavily hinges on the amount of user calibration data and data length, which hinders the deployment in real-world applications. Recently, generative adversarial networks (GANs)-based data generation methods have been widely adopted to create synthetic electroencephalography data, holds promise to address these issues. In this paper, we proposed a GAN-based end-to-end signal transformation network for Time-window length Extension, termed as TEGAN. TEGAN transforms short-length SSVEP signals into long-length artificial SSVEP signals. Additionally, we introduced a two-stage training strategy and the LeCam-divergence regularization term to regularize the training process of GAN during the network implementation. The proposed TEGAN was evaluated on two public SSVEP datasets (a 4-class and 12-class dataset). With the assistance of TEGAN, the performance of traditional frequency recognition methods and deep learning-based methods have been significantly improved under limited calibration data. And the classification performance gap of various frequency recognition methods has been narrowed. This study substantiates the feasibility of the proposed method to extend the data length for short-time SSVEP signals for developing a high-performance BCI system. The proposed GAN-based methods have the great potential of shortening the calibration time and cutting down the budget for various real-world BCI-based applications.

EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts

Understanding emotional states is pivotal for the development of next-generation human-machine interfaces. Human behaviors in social interactions have resulted in psycho-physiological processes influenced by perceptual inputs. Therefore, efforts to comprehend brain functions and human behavior could potentially catalyze the development of AI models with human-like attributes. In this study, we introduce a multimodal emotion dataset comprising data from 30-channel electroencephalography (EEG), audio, and video recordings from 42 participants. Each participant engaged in a cue-based conversation scenario, eliciting five distinct emotions: neutral, anger, happiness, sadness, and calmness. Throughout the experiment, each participant contributed 200 interactions, which encompassed both listening and speaking. This resulted in a cumulative total of 8,400 interactions across all participants. We evaluated the baseline performance of emotion recognition for each modality using established deep neural network (DNN) methods. The Emotion in EEG-Audio-Visual (EAV) dataset represents the first public dataset to incorporate three primary modalities for emotion recognition within a conversational context. We anticipate that this dataset will make significant contributions to the modeling of the human emotional process, encompassing both fundamental neuroscience and machine learning viewpoints.

Decoding Multi-Class Motor Imagery From Unilateral Limbs Using EEG Signals

The EEG is a widely utilized neural signal source, particularly in motor imagery-based brain-computer interface (MI-BCI), offering distinct advantages in applications like stroke rehabilitation. Current research predominantly concentrates on the bilateral limbs paradigm and decoding, but the use scenarios for stroke rehabilitation are typically for unilateral upper limbs. There is a significant challenge to decoding unilateral MI of multitasks due to the overlapped spatial neural activities of the tasks. This study aims to formulate a novel MI-BCI experimental paradigm for unilateral limbs with multitasks. The paradigm encompasses four imagined movement directions: top-bottom, left-right, top right-bottom left, and top left-bottom right. Forty-six healthy subjects participated in this experiment. Commonly used machine learning techniques, such as FBCSP, EEGNet, deepConvNet, and FBCNet, were employed for evaluation. To improve decoding accuracy, we propose an MVCA method that introduces temporal convolution and attention mechanism to effectively capture temporal features from multiple perspectives. With the MVCA model, we have achieved 40.6% and 64.89% classification accuracies for the four-class and two-class scenarios (top right-bottom left and top left-bottom right), respectively. Conclusion: This is the first study demonstrating that motor imagery of multiple directions in unilateral limbs can be decoded. In particular, decoding two directions, right top to left bottom and left top to right bottom, provides the best accuracy, which sheds light on future studies. This study advances the development of the MI-BCI paradigm, offering preliminary evidence for the feasibility of decoding multiple directional information from EEG. This, in turn, enhances the dimensions of MI control commands.

Optimization of stimulus properties for SSVEP-based BMI system with a heads-up display to control in-vehicle features

Over the years, the driver-vehicle interface has been improved, but interacting with in-vehicle features can still increase distraction and affect road safety. This study aims to introduce brain-machine interface (BMI)- based solution to potentially enhance road safety. To achieve this goal, we evaluated visual stimuli properties (SPs) for a steady state visually evoked potentials (SSVEP)-based BMI system. We used a heads-up display (HUD) as the primary screen to present icons for controlling in-vehicle functions such as music, temperature, settings, and navigation. We investigated the effect of various SPs on SSVEP detection performance including the duty cycle and signal-to-noise ratio of visual stimuli, the size, color, and frequency of the icons, and array configuration and location. The experiments were conducted with 10 volunteers and the signals were analyzed using the canonical correlation analysis (CCA), filter bank CCA (FBCCA), and power spectral density analysis (PSDA). Our experimental results suggest that stimuli with a green color, a duty cycle of 50%, presented at a central location, with a size of 36 cm2 elicit a significantly stronger SSVEP response and enhanced SSVEP detection time. We also observed that lower SNR stimuli significantly affect SSVEP detection performance. There was no statistically significant difference observed in SSVEP response between the use of an LCD monitor and a HUD.

Efficient Multi-View Graph Convolutional Network with Self-Attention for Multi-Class Motor Imagery Decoding

Research on electroencephalogram-based motor imagery (MI-EEG) can identify the limbs of subjects that generate motor imagination by decoding EEG signals, which is an important issue in the field of brain-computer interface (BCI). Existing deep-learning-based classification methods have not been able to entirely employ the topological information among brain regions, and thus, the classification performance needs further improving. In this paper, we propose a multi-view graph convolutional attention network (MGCANet) with residual learning structure for multi-class MI decoding. Specifically, we design a multi-view graph convolution spatial feature extraction method based on the topological relationship of brain regions to achieve more comprehensive information aggregation. During the modeling, we build an adaptive weight fusion (Awf) module to adaptively merge feature from different brain views to improve classification accuracy. In addition, the self-attention mechanism is introduced for feature selection to expand the receptive field of EEG signals to global dependence and enhance the expression of important features. The proposed model is experimentally evaluated on two public MI datasets and achieved a mean accuracy of 78.26% (BCIC IV 2a dataset) and 73.68% (OpenBMI dataset), which significantly outperforms representative comparative methods in classification accuracy. Comprehensive experiment results verify the effectiveness of our proposed method, which can provide novel perspectives for MI decoding.

Iteratively Calibratable Network for Reliable EEG-Based Robotic Arm Control

Robotic arms are increasingly being utilized in shared workspaces, which necessitates the accurate interpretation of human intentions for both efficiency and safety. Electroencephalogram (EEG) signals, commonly employed to measure brain activity, offer a direct communication channel between humans and robotic arms. However, the ambiguous and unstable characteristics of EEG signals, coupled with their widespread distribution, make it challenging to collect sufficient data and hinder the calibration performance for new signals, thereby reducing the reliability of EEG-based applications. To address these issues, this study proposes an iteratively calibratable network aimed at enhancing the reliability and efficiency of EEG-based robotic arm control systems. The proposed method integrates feature inputs with network expansion techniques. This integration allows a network trained on an extensive initial dataset to adapt effectively to new users during calibration. Additionally, our approach combines motor imagery and speech imagery datasets to increase not only its intuitiveness but also the number of command classes. The evaluation is conducted in a pseudo-online manner, with a robotic arm operating in real-time to collect data, which is then analyzed offline. The evaluation results demonstrated that the proposed method outperformed the comparison group in 10 sessions and demonstrated competitive results when the two paradigms were combined. Therefore, it was confirmed that the network can be calibrated and personalized using only the new data from new users.

Connectivity study on resting-state EEG between motor imagery BCI-literate and BCI-illiterate groups

Objective.Although motor imagery-based brain-computer interface (MI-BCI) holds significant potential, its practical application faces challenges such as BCI-illiteracy. To mitigate this issue, researchers have attempted to predict BCI-illiteracy by using the resting state, as this was found to be associated with BCI performance. As connectivity's significance in neuroscience has grown, BCI researchers have applied connectivity to it. However, the issues of connectivity have not been considered fully. First, although various connectivity metrics exist, only some have been used to predict BCI-illiteracy. This is problematic because each metric has a distinct hypothesis and perspective to estimate connectivity, resulting in different outcomes according to the metric. Second, the frequency range affects the connectivity estimation. In addition, it is still unknown whether each metric has its own optimal frequency range. Third, the way that estimating connectivity may vary depending upon the dataset has not been investigated. Meanwhile, we still do not know a great deal about how the resting state electroencephalography (EEG) network differs between BCI-literacy and -illiteracy.Approach.To address the issues above, we analyzed three large public EEG datasets using three functional connectivity and three effective connectivity metrics by employing diverse graph theory measures. Our analysis revealed that the appropriate frequency range to predict BCI-illiteracy varies depending upon the metric. The alpha range was found to be suitable for the metrics of the frequency domain, while alpha + theta were found to be appropriate for multivariate Granger causality. The difference in network efficiency between BCI-literate and -illiterate groups was constant regardless of the metrics and datasets used. Although we observed that BCI-literacy had stronger connectivity, no other significant constructional differences were found.Significance.Based upon our findings, we predicted MI-BCI performance for the entire dataset. We discovered that combining several graph features could improve the prediction's accuracy.

A novel feature extraction method PSS-CSP for binary motor imagery - based brain-computer interfaces

In order to improve the performance of binary motor imagery (MI) - based brain-computer interfaces (BCIs) using electroencephalography (EEG), a novel method (PSS-CSP) is proposed, which combines spectral subtraction with common spatial pattern. Spectral subtraction is an effective denoising method which is initially adopted to process MI-based EEG signals for binary BCIs in this work. On this basis, we proposed a novel feature extraction method called power spectral subtraction-based common spatial pattern (PSS-CSP) , which calculates the differences in power spectrum between binary classes of EEG signals and uses the differences in the feature extraction process. Additionally, support vector machine (SVM) algorithm is used for signal classification. Results show the proposed method (PSS-CSP) outperforms certain existing methods, achieving a classification accuracy of 76.8% on the BCIIV dataset 2b, and 76.25% and 77.38% on the OpenBMI dataset session 1 and session 2, respectively.

Channel- and Label-Flip Data Augmentation for Motor Imagery-Based Brain-Computer Interfaces

Achieving high classification accuracy in motor-imagery-based brain-computer interfaces (BCIs) requires substantial amounts of training data. A challenge arises because of the impracticality of measuring large amounts of data from users. Data augmentation (DA) has emerged as a promising solution for this challenge. We propose a novel DA method called channel&label-flip DA that involves not only flipping channels but also flipping class labels. This method is based on the neuroscience finding that motor imageries of left- and right-hand movements are roughly symmetrical. The efficiency of the proposed method was evaluated using the OpenBMI dataset, which comprises electroencephalograms collected from 54 participants engaged in left- and right-hand motor imagery tasks. To compare the impact on classifiers, we employed three classical machine learning models utilizing filter bank common spatial pattern features, along with a deep learning-based model that uses raw signal input. As a result, the channel&label-flip DA improved the classification accuracy on average, whereas simple flipping of the channels reduced the classification accuracy compared to the case without DA.