Deep learning with convolutional neural networks for EEG decoding and visualization

Online continual decoding of streaming EEG signal with a balanced and informative memory buffer

Electroencephalography (EEG) based Brain Computer Interface (BCI) systems play a significant role in facilitating how individuals with neurological impairments effectively interact with their environment. In real world applications of BCI system for clinical assistance and rehabilitation training, the EEG classifier often needs to learn on sequentially arriving subjects in an online manner. As patterns of EEG signals can be significantly different for different subjects, the EEG classifier can easily erase knowledge of learnt subjects after learning on later ones as it performs decoding in online streaming scenario, namely catastrophic forgetting. In this work, we tackle this problem with a memory-based approach, which considers the following conditions: (1) subjects arrive sequentially in an online manner, with no large scale dataset available for joint training beforehand, (2) data volume from the different subjects could be imbalanced, (3) decoding difficulty of the sequential streaming signal vary, (4) continual classification for a long time is required. This online sequential EEG decoding problem is more challenging than classic cross subject EEG decoding as there is no large-scale training data from the different subjects available beforehand. The proposed model keeps a small balanced memory buffer during sequential learning, with memory data dynamically selected based on joint consideration of data volume and informativeness. Furthermore, for the more general scenarios where subject identity is unknown to the EEG decoder, aka. subject agnostic scenario, we propose a kernel based subject shift detection method that identifies underlying subject changes on the fly in a computationally efficient manner. We develop challenging benchmarks of streaming EEG data from sequentially arriving subjects with both balanced and imbalanced data volumes, and performed extensive experiments with a detailed ablation study on the proposed model. The results show the effectiveness of our proposed approach, enabling the decoder to maintain performance on all previously seen subjects over a long period of sequential decoding. The model demonstrates the potential for real-world applications.

Single-Trial Detection and Classification of Event-Related Optical Signals for a Brain-Computer Interface Application

Event-related optical signals (EROS) measure fast modulations in the brain's optical properties related to neuronal activity. EROS offer a high spatial and temporal resolution and can be used for brain-computer interface (BCI) applications. However, the ability to classify single-trial EROS remains unexplored. This study evaluates the performance of neural network methods for single-trial classification of motor response-related EROS. EROS activity was obtained from a high-density recording montage covering the motor cortex during a two-choice reaction time task involving responses with the left or right hand. This study utilized a convolutional neural network (CNN) approach to extract spatiotemporal features from EROS data and perform classification of left and right motor responses. Subject-specific classifiers trained on EROS phase data outperformed those trained on intensity data, reaching an average single-trial classification accuracy of around 63%. Removing low-frequency noise from intensity data is critical for achieving discriminative classification results with this measure. Our results indicate that deep learning with high-spatial-resolution signals, such as EROS, can be successfully applied to single-trial classifications.

HiRENet: Novel convolutional neural network architecture using Hilbert-transformed and raw electroencephalogram (EEG) for subject-independent emotion classification

BACKGROUND AND OBJECTIVES

Convolutional neural networks (CNNs) are the most widely used deep-learning framework for decoding electroencephalograms (EEGs) due to their exceptional ability to extract hierarchical features from high-dimensional EEG data. Traditionally, CNNs have primarily utilized multi-channel raw EEG data as the input tensor; however, the performance of CNN-based EEG decoding may be enhanced by incorporating phase information alongside amplitude information.

METHODS

This study introduces a novel CNN architecture called the Hilbert-transformed (HT) and raw EEG network (HiRENet), which incorporates both raw and HT EEG as inputs. This concurrent use of HT and raw EEG aims to integrate phase information with existing amplitude information, potentially offering a more comprehensive reflection of functional connectivity across various brain regions. The HiRENet model was developed using two CNN frameworks: ShallowFBCSPNet and a CNN with a residual block (ResCNN). The performance of the HiRENet model was assessed using a lab-made EEG database to classify human emotions, comparing three input modalities: raw EEG, HT EEG, and a combination of both signals. Additionally, the computational complexity was evaluated to validate the computational efficiency of the ResCNN design.

RESULTS

The HiRENet model based on ResCNN achieved the highest classification accuracy, with 86.03% for valence and 84.01% for arousal classifications, surpassing traditional CNN methodologies. Considering computational efficiency, ResCNN demonstrated superiority over ShallowFBCSPNet in terms of speed and inference time, despite having a higher parameter count.

CONCLUSION

Our experimental results showed that the proposed HiRENet can be potentially used as a new option to improve the overall performance for deep learning-based EEG decoding problems.

EEG-Based Motor Imagery Recognition Framework via Multisubject Dynamic Transfer and Iterative Self-Training

A robust decoding model that can efficiently deal with the subject and period variation is urgently needed to apply the brain-computer interface (BCI) system. The performance of most electroencephalogram (EEG) decoding models depends on the characteristics of specific subjects and periods, which require calibration and training with annotated data prior to application. However, this situation will become unacceptable as it would be difficult for subjects to collect data for an extended period, especially in the rehabilitation process of disability based on motor imagery (MI). To address this issue, we propose an unsupervised domain adaptation framework called iterative self-training multisubject domain adaptation (ISMDA) that focuses on the offline MI task. First, the feature extractor is purposefully designed to map the EEG to a latent space of discriminative representations. Second, the attention module based on dynamic transfer matches the source domain and target domain samples with a higher coincidence degree in latent space. Then, an independent classifier oriented to the target domain is employed in the first stage of the iterative training process to cluster the samples of the target domain through similarity. Finally, a pseudolabel algorithm based on certainty and confidence is employed in the second stage of the iterative training process to adequately calibrate the error between prediction and empirical probabilities. To evaluate the effectiveness of the model, extensive testing has been performed on three publicly available MI datasets, the BCI IV IIa, the High gamma dataset, and Kwon et al. datasets. The proposed method achieved 69.51%, 82.38%, and 90.98% cross-subject classification accuracy on the three datasets, which outperforms the current state-of-the-art offline algorithms. Meanwhile, all results demonstrated that the proposed method could address the main challenges of the offline MI paradigm.

Image classification and reconstruction from low-density EEG

Recent advances in visual decoding have enabled the classification and reconstruction of perceived images from the brain. However, previous approaches have predominantly relied on stationary, costly equipment like fMRI or high-density EEG, limiting the real-world availability and applicability of such projects. Additionally, several EEG-based paradigms have utilized artifactual, rather than stimulus-related information yielding flawed classification and reconstruction results. Our goal was to reduce the cost of the decoding paradigm, while increasing its flexibility. Therefore, we investigated whether the classification of an image category and the reconstruction of the image itself is possible from the visually evoked brain activity measured by a portable, 8-channel EEG. To compensate for the low electrode count and to avoid flawed predictions, we designed a theory-guided EEG setup and created a new experiment to obtain a dataset from 9 subjects. We compared five contemporary classification models with our setup reaching an average accuracy of 34.4% for 20 image classes on hold-out test recordings. For the reconstruction, the top-performing model was used as an EEG-encoder which was combined with a pretrained latent diffusion model via double-conditioning. After fine-tuning, we reconstructed images from the test set with a 1000 trial 50-class top-1 accuracy of 35.3%. While not reaching the same performance as MRI-based paradigms on unseen stimuli, our approach greatly improved the affordability and mobility of the visual decoding technology.

Functional Near-Infrared Spectroscopy-Based Computer-Aided Diagnosis of Major Depressive Disorder Using Convolutional Neural Network with a New Channel Embedding Layer Considering Inter-Hemispheric Asymmetry in Prefrontal Hemodynamic Responses

Background

Functional near-infrared spectroscopy (fNIRS) is being extensively explored as a potential primary screening tool for major depressive disorder (MDD) because of its portability, cost-effectiveness, and low susceptibility to motion artifacts. However, the fNIRS-based computer-aided diagnosis (CAD) of MDD using deep learning methods has rarely been studied. In this study, we propose a novel deep learning framework based on a convolutional neural network (CNN) for the fNIRS-based CAD of MDD with high accuracy.

Materials and Methods

The fNIRS data of participants-48 patients with MDD and 68 healthy controls (HCs)-were obtained while they performed a Stroop task. The hemodynamic responses calculated from the preprocessed fNIRS data were used as inputs to the proposed CNN model with an ensemble CNN architecture, comprising three 1D depth-wise convolutional layers specifically designed to reflect interhemispheric asymmetry in hemodynamic responses between patients with MDD and HCs, which is known to be a distinct characteristic in previous MDD studies. The performance of the proposed model was evaluated using a leave-one-subject-out cross-validation strategy and compared with those of conventional machine learning and CNN models.

Results

The proposed model exhibited a high accuracy, sensitivity, and specificity of 84.48%, 83.33%, and 85.29%, respectively. The accuracies of conventional machine learning algorithms-shrinkage linear discriminator analysis, regularized support vector machine, EEGNet, and ShallowConvNet-were 73.28%, 74.14%, 62.93%, and 62.07%, respectively.

Conclusions

In conclusion, the proposed deep learning model can differentiate between the patients with MDD and HCs more accurately than the conventional models, demonstrating its applicability in fNIRS-based CAD systems.

MutaPT: A Multi-Task Pre-Trained Transformer for Classifying State of Disorders of Consciousness Using EEG Signal

Deep learning (DL) has been demonstrated to be a valuable tool for classifying state of disorders of consciousness (DOC) using EEG signals. However, the performance of the DL-based DOC state classification is often challenged by the limited size of EEG datasets. To overcome this issue, we introduce multiple open-source EEG datasets to increase data volume and train a novel multi-task pre-training Transformer model named MutaPT. Furthermore, we propose a cross-distribution self-supervised (CDS) pre-training strategy to enhance the model's generalization ability, addressing data distribution shifts across multiple datasets. An EEG dataset of DOC patients is used to validate the effectiveness of our methods for the task of classifying DOC states. Experimental results show the superiority of our MutaPT over several DL models for EEG classification.

Harnessing Few-Shot Learning for EEG signal classification: a survey of state-of-the-art techniques and future directions

This paper presents a systematic literature review, providing a comprehensive taxonomy of Data Augmentation (DA), Transfer Learning (TL), and Self-Supervised Learning (SSL) techniques within the context of Few-Shot Learning (FSL) for EEG signal classification. EEG signals have shown significant potential in various paradigms, including Motor Imagery, Emotion Recognition, Visual Evoked Potentials, Steady-State Visually Evoked Potentials, Rapid Serial Visual Presentation, Event-Related Potentials, and Mental Workload. However, challenges such as limited labeled data, noise, and inter/intra-subject variability have impeded the effectiveness of traditional machine learning (ML) and deep learning (DL) models. This review methodically explores how FSL approaches, incorporating DA, TL, and SSL, can address these challenges and enhance classification performance in specific EEG paradigms. It also delves into the open research challenges related to these techniques in EEG signal classification. Specifically, the review examines the identification of DA strategies tailored to various EEG paradigms, the creation of TL architectures for efficient knowledge transfer, and the formulation of SSL methods for unsupervised representation learning from EEG data. Addressing these challenges is crucial for enhancing the efficacy and robustness of FSL-based EEG signal classification. By presenting a structured taxonomy of FSL techniques and discussing the associated research challenges, this systematic review offers valuable insights for future investigations in EEG signal classification. The findings aim to guide and inspire researchers, promoting advancements in applying FSL methodologies for improved EEG signal analysis and classification in real-world settings.

AttentionCARE: replicability of a BCI for the clinical application of augmented reality-guided EEG-based attention modification for adolescents at high risk for depression

Affect-biased attention is the phenomenon of prioritizing attention to emotionally salient stimuli and away from goal-directed stimuli. It is thought that affect-biased attention to emotional stimuli is a driving factor in the development of depression. This effect has been well-studied in adults, but research shows that this is also true during adolescence, when the severity of depressive symptoms are correlated with the magnitude of affect-biased attention to negative emotional stimuli. Prior studies have shown that trainings to modify affect-biased attention may ameliorate depression in adults, but this research has also been stymied by concerns about reliability and replicability. This study describes a clinical application of augmented reality-guided EEG-based attention modification ("AttentionCARE") for adolescents who are at highest risk for future depressive disorders (i.e., daughters of depressed mothers). Our results (n = 10) indicated that the AttentionCARE protocol can reliably and accurately provide neurofeedback about adolescent attention to negative emotional distractors that detract from attention to a primary task. Through several within and cross-study replications, our work addresses concerns about the lack of reliability and reproducibility in brain-computer interface applications, offering insights for future interventions to modify affect-biased attention in high-risk adolescents.

A Dynamic Multi-Scale Convolution Model for Face Recognition Using Event-Related Potentials

With the development of data mining technology, the analysis of event-related potential (ERP) data has evolved from statistical analysis of time-domain features to data-driven techniques based on supervised and unsupervised learning. However, there are still many challenges in understanding the relationship between ERP components and the representation of familiar and unfamiliar faces. To address this, this paper proposes a model based on Dynamic Multi-Scale Convolution for group recognition of familiar and unfamiliar faces. This approach uses generated weight masks for cross-subject familiar/unfamiliar face recognition using a multi-scale model. The model employs a variable-length filter generator to dynamically determine the optimal filter length for time-series samples, thereby capturing features at different time scales. Comparative experiments are conducted to evaluate the model's performance against SOTA models. The results demonstrate that our model achieves impressive outcomes, with a balanced accuracy rate of 93.20% and an F1 score of 88.54%, outperforming the methods used for comparison. The ERP data extracted from different time regions in the model can also provide data-driven technical support for research based on the representation of different ERP components.

An auto-segmented multi-time window dual-scale neural network for brain-computer interfaces based on event-related potentials

Objective.Event-related potentials (ERPs) are cerebral responses to cognitive processes, also referred to as cognitive potentials. Accurately decoding ERPs can help to advance research on brain-computer interfaces (BCIs). The spatial pattern of ERP varies with time. In recent years, convolutional neural networks (CNNs) have shown promising results in electroencephalography (EEG) classification, specifically for ERP-based BCIs.Approach.This study proposes an auto-segmented multi-time window dual-scale neural network (AWDSNet). The combination of a multi-window design and a lightweight base network gives AWDSNet good performance at an acceptable cost of computing. For each individual, we create a time window set by calculating the correlation of signedR-squared values, which enables us to determine the length and number of windows automatically. The signal data are segmented based on the obtained window sets in sub-plus-global mode. Then, the multi-window data are fed into a dual-scale CNN model, where the sizes of the convolution kernels are determined by the window sizes. The use of dual-scale spatiotemporal convolution focuses on feature details while also having a large enough receptive length, and the grouping parallelism undermines the increase in the number of parameters that come with dual scaling.Main results.We evaluated the performance of AWDSNet on a public dataset and a self-collected dataset. A comparison was made with four popular methods including EEGNet, DeepConvNet, EEG-Inception, and PPNN. The experimental results show that AWDSNet has excellent classification performance with acceptable computational complexity.Significance.These results indicate that AWDSNet has great potential for applications in ERP decoding.

Baseline-Guided Representation Learning for Noise-Robust EEG Signal Classification

Brain-computer interfaces (BCIs) suffer from limited accuracy due to noisy electroencephalography (EEG) signals. Existing denoising methods often remove artifacts such as eye movement or use techniques such as linear detrending, which inadvertently discard crucial task-relevant information. To address this issue, we present BGNet, a novel deep learning framework that leverages underutilized baseline EEG signals for dynamic noise mitigation and robust feature extraction to improve motor imagery (MI) EEG classification. Our approach employs data augmentation to strengthen model robustness, an autoencoder to extract features from baseline and MI signals, a feature alignment module to separate specific task and noise, and a classifier. We achieve state-of-the-art performance, an improvement of 5.9% and 3.7% on the BCIC IV 2a and 2b datasets, respectively. The qualitative analysis of our learned features proves superior representational power over baseline models, a critical aspect in dealing with noisy EEG signals. Our findings demonstrate the efficacy of readily available baseline signals in enhancing performance, opening possibilities for simplified BCI systems in brain-based communication applications.

Benchmarking motor imagery algorithms for pediatric users of brain-computer interfaces

Brain-computer interfaces (BCIs) can enable opportunities for self-expression and life participation for children with severe neurological disabilities. Unfortunately, the development and evaluation of state-of-the-art algorithms has largely neglected pediatric users. This work tests 12 state-of-the-art algorithms for motor imagery classification on three datasets of typically developing pediatric users (n=94 ages 5-17). When all datasets were combined, there were no significant differences between most non-deep learning algorithms, with all having a mean AUC score of 0.64 or 0.65. All the non-deep learning algorithms significantly outperformed the deep learning algorithms, which can be partially attributed to a lack of hyperparameter tuning. The best of the deep learning algorithms was ShallowConvNet, with a mean AUC score of 0.57. Of the algorithms tested, only the filter bank common spatial pattern (FBCSP) and ShallowConvNet exhibited significant age effects. This general lack of age effects, combined with examples of children as young as 6 having AUC scores as high as 0.8, provides evidence that young children are capable of producing measurable motor imagery activations. The age effects that were present for some algorithms suggest that the changing EEG patterns associated with development could have a measurable impact on classification algorithm outcomes, and such algorithms should be evaluated to ensure that they are not performing disproportionately poorly for younger children. This work serves as a first step towards ensuring that the state-of-the-art improvements in BCI classification can be evaluated, and where necessary, adapted to meet the needs of pediatric users.

Direct lingam and visibility graphs for analyzing brain connectivity in BCI

The brain-computer interface (BCI) is a direct pathway of communication between the electrical activity of the brain and an external device. The present paper was aimed to investigate directed connectivity between different areas of the brain during motor imagery (MI)-based BCI. For this purpose, two methods were implemented including, Limited Penetrable Horizontal Visibility Graph (LPHVG) and Direct Lingam. The visibility graph (VG) is a robust algorithm for analyzing complex systems such as the brain. Direct Lingam uses a non-Gaussian model to extract causal links which is appropriate for analyzing large-scale connectivity. First, LPHVG map MI-EEG (electroencephalogram) signals into networks. After extracting the topological features of the networks, a support vector machine classifier was applied to categorize multi-classes MI. The network of all classes was found to be different from one another, and the kappa value of classification was 0.68. The degree sequence of LPHVG was calculated for each channel in order to obtain the direction of brain information flow. Transfer entropy (TE) is used to compute the relations of the channel degree sequence. Therefore, the directed graph between channels was formed. This method is called LPHVG_TE directed graph. The Bayesian network, also known as the Direct LiNGAM model, was implemented for the second method. Finally, images of the LPHVG and Direct Lingam were classified by convolutional neural network (CNN). In this study, Data sets 2a of BCI competition IV was used. The outcomes reveal that the brain network developed by LPHVG (92.7%) might be more effective to distinguish 4 classes of MI than the Direct Lingam (90.6%) and it was shown that graph theory has the potential to get better efficiency of BCI.

A novel feature extraction method PSS-CSP for binary motor imagery - based brain-computer interfaces

In order to improve the performance of binary motor imagery (MI) - based brain-computer interfaces (BCIs) using electroencephalography (EEG), a novel method (PSS-CSP) is proposed, which combines spectral subtraction with common spatial pattern. Spectral subtraction is an effective denoising method which is initially adopted to process MI-based EEG signals for binary BCIs in this work. On this basis, we proposed a novel feature extraction method called power spectral subtraction-based common spatial pattern (PSS-CSP) , which calculates the differences in power spectrum between binary classes of EEG signals and uses the differences in the feature extraction process. Additionally, support vector machine (SVM) algorithm is used for signal classification. Results show the proposed method (PSS-CSP) outperforms certain existing methods, achieving a classification accuracy of 76.8% on the BCIIV dataset 2b, and 76.25% and 77.38% on the OpenBMI dataset session 1 and session 2, respectively.

LGGNet: Learning From Local-Global-Graph Representations for Brain-Computer Interface

Neuropsychological studies suggest that co-operative activities among different brain functional areas drive high-level cognitive processes. To learn the brain activities within and among different functional areas of the brain, we propose local-global-graph network (LGGNet), a novel neurologically inspired graph neural network (GNN), to learn local-global-graph (LGG) representations of electroencephalography (EEG) for brain-computer interface (BCI). The input layer of LGGNet comprises a series of temporal convolutions with multiscale 1-D convolutional kernels and kernel-level attentive fusion. It captures temporal dynamics of EEG which then serves as input to the proposed local- and global-graph-filtering layers. Using a defined neurophysiologically meaningful set of local and global graphs, LGGNet models the complex relations within and among functional areas of the brain. Under the robust nested cross-validation settings, the proposed method is evaluated on three publicly available datasets for four types of cognitive classification tasks, namely the attention, fatigue, emotion, and preference classification tasks. LGGNet is compared with state-of-the-art (SOTA) methods, such as DeepConvNet, EEGNet, R2G-STNN, TSception, regularized graph neural network (RGNN), attention-based multiscale convolutional neural network-dynamical graph convolutional network (AMCNN-DGCN), hierarchical recurrent neural network (HRNN), and GraphNet. The results show that LGGNet outperforms these methods, and the improvements are statistically significant ( ) in most cases. The results show that bringing neuroscience prior knowledge into neural network design yields an improvement of classification performance. The source code can be found at https://github.com/yi-ding-cs/LGG.

Motor Imagery Decoding from EEG under Visual Distraction via Feature Map Attention EEGNet

The investigation of motor imagery (MI)-based brain-computer interface (BCI) is vital to the domains of human-computer interaction and rehabilitation. Few existing studies on electroencephalogram(EEG) signals decoding based on MI consider any distractions. However, it is difficult for users to do a single MI task in real life, which is especially affected by visual distraction. In this paper, we aim to investigate the effects of visual distraction on MI decoding performance. We first design a noval MI paradigm under visual distraction and observe distinct patterns of event-related desynchronization (ERD) and event-related synchronization (ERS) in MI under visual distraction. Then, we propose a robust decoding method of MI under visual distraction from EEG signals by using the feature map attention EEGNet (named FMA-EEGNet) and use EEG data under conditions without and with distraction to compare the decoding performance of five methods (including the proposed method and other methods). The results demonstrate that FMA-EEGNet achieved mean accuracy of 89.1% and 82.2% without and with visual distraction, respectively, indicating superior performance compared to other methods while exhibiting minimal degradation in performance. This work contributes significantly to the advancement of practical applications in MI-BCI technology.

The Impact of Cross-Validation Schemes for EEG-Based Auditory Attention Detection with Deep Neural Networks

This study assesses the performance of different cross-validation splits for brain-signal-based Auditory Attention Decoding (AAD) using deep neural networks on three publicly available Electroencephalography datasets. We investigate the effect of trial-specific knowledge during training and assess adaptability to diverse scenarios with a trial-independent split. Introducing a causal time-series split, and simulating online decoding, our results demonstrate a consistent performance increase for auditory attention classification. These positive outcomes provide valuable insights for the development of future brain-signal-based AAD systems, emphasizing the potential for practical, person-dependent AAD applications. The results highlight the importance of diverse evaluation methodologies for enhancing generalizability in developing effective neurofeedback systems and assistive technologies for auditory processing disorders under more real-life conditions.

Dynamic Multi-modal Fusion for Biosignal-based Motion Sickness Prediction in Vehicles

With the advent of autonomous vehicles, motion sickness (MS) has emerged as a significant issue, affecting the comfort and safety of drivers and passengers. However, traditional MS research, often confined to simulations or manual data analysis, does not fully capture real-world complexities. Therefore, the significance of multi-biosignal fusion, which can reflect the complexity and utility of data, is underscored. In this study, we propose a novel dynamic multi-modal fusion for the MS classification (DMFMS) framework. DMFMS adaptively focuses on significant samples by evaluating data quality in noisy environments. It includes confidence-aware learning to estimate the reliability of modalities, a dynamic gating mechanism that adjusts based on each modality's contribution to the features, and a spatial-temporal attention module (STAM) that focuses on relevant information while filtering out the extraneous. We conducted extensive experiments on a multi-biosignal dataset from real driving scenarios, involving data from 13 subjects. The results show that DMFMS outperforms conventional MS prediction models, showing that the proposed dynamic fusion approach is the superior solution for detecting MS in a real-world driving environment.

CRE-TSCAE: A Novel Classification Model Based on Stacked Convolutional Autoencoder for Dual-Target RSVP-BCI Tasks

OBJECTIVE

The Rapid Serial Visual Presentation (RSVP) paradigm facilitates target identification in a rapid picture stream, which is applied extensively in military target surveillance and police monitoring. Most researchers concentrate on the single target RSVP-BCI whereas the study of dual-target is scarcely conducted, limiting RSVP application considerably.

METHODS

This paper proposed a novel classification model named Common Representation Extraction-Targeted Stacked Convolutional Autoencoder (CRE-TSCAE) to detect two targets with one nontarget in RSVP tasks. CRE generated a common representation for each target class to reduce variability from different trials of the same class and distinguish the difference between two targets better. TSCAE aimed to control uncertainty in the training process while requiring less target training data. The model learned a compact and discriminative feature through the training from several learning tasks so as to distinguish each class effectively.

RESULTS

It was validated on the World Robot Contest 2021 and 2022 ERP datasets. Experimental results showed that CRE-TSCAE outperformed the state-of-the-art RSVP decoding algorithms and the Average ACC was 71.25%, improving 6.5% at least over the rest.

CONCLUSION

It demonstrated that CRE-TSCAE showed a strong ability to extract discriminative latent features in detecting the differences among two targets with nontarget, which guaranteed increased classification accuracy.

SIGNIFICANCE

CRE-TSCAE provided an innovative and effective classification model for dual-target RSVP-BCI tasks and some insights into the neurophysiological distinction between different targets.

Reconstruction of Continuous Hand Grasp Movement from EEG Using Deep Learning

Brain-Computer Interface (BCI) is a promising neu-rotechnology offering non-muscular control of external devices, such as neuroprostheses and robotic exoskeletons. A new yet under-explored BCI control paradigm is Motion Trajectory Prediction (MTP). While MTP provides continuous control signals suitable for high-precision tasks, its feasibility and applications are challenged by the low signal-to-noise ratio, especially in noninvasive settings. Previous research has predominantly focused on kinematic reconstruction of upper (e.g., arm reaching) and lower limbs (e.g., gait). However, finger movements have received much less attention, despite their crucial role in daily activities. To address this gap, our study explores the potential of noninvasive Electroencephalography (EEG) for reconstructing finger movements, specifically during hand grasping actions. A new experimental paradigm to collect multichannel EEG data from 20 healthy subjects, while performing full, natural hand opening and closing movements, was designed. Employing state-of-the-art deep learning algorithms, continuous decoding models were constructed for eight key finger joints. The Convolutional Neural Network with Attention approach achieved an average decoding performance of r=0.63. Furthermore, a post-hoc metric was proposed for hand grasp cycle detection, and 83.5% of hand grasps were successfully detected from the reconstructed motion signals, which can potentially serve as a new BCI command. Explainable AI algorithm was also applied to analyze the topographical relevance of trained features. Our findings demonstrate the feasibility of using EEG to reconstruct hand joint movements and highlight the potential of MTP-BCI in control and rehabilitation applications.

MASA-TCN: Multi-Anchor Space-Aware Temporal Convolutional Neural Networks for Continuous and Discrete EEG Emotion Recognition

Emotion recognition from electroencephalogram (EEG) signals is a critical domain in biomedical research with applications ranging from mental disorder regulation to human-computer interaction. In this paper, we address two fundamental aspects of EEG emotion recognition: continuous regression of emotional states and discrete classification of emotions. While classification methods have garnered significant attention, regression methods remain relatively under-explored. To bridge this gap, we introduce MASA-TCN, a novel unified model that leverages the spatial learning capabilities of Temporal Convolutional Networks (TCNs) for EEG emotion regression and classification tasks. The key innovation lies in the introduction of a space-aware temporal layer, which empowers TCN to capture spatial relationships among EEG electrodes, enhancing its ability to discern nuanced emotional states. Additionally, we design a multi-anchor block with attentive fusion, enabling the model to adaptively learn dynamic temporal dependencies within the EEG signals. Experiments on two publicly available datasets show that MASA-TCN achieves higher results than the state-of-the-art methods for both EEG emotion regression and classification tasks.

Interpretable SincNet-Based Spatiotemporal Neural Network for Seizure Prediction

Spatiotemporal convolutional neural networks (CNNs) have emerged as potent tools for seizure prediction (SP) using electroencephalogram (EEG) signals, probing spatiotemporal biomarkers in epileptic brains. Nevertheless, it poses significant challenges for clinical practice due to the poor interpretability of learned features and the numerous trainable parameters in existing CNNs. To improve the interpretability and performance, this study proposed an interpretable SincNet-based architecture for spatiotemporal CNNs, encompassing EEGNet-8,2, ShallowConvNet, DeepConvNet, and EEGWaveNet, enabling direct visualization of the bandpass temporal filter range using a sinc-convolution layer. Furthermore, we also constructed a visualization analysis method to demonstrate the crucial spatiotemporal features learned by the proposed optimal CNN. Results on the CHB-MIT dataset revealed that both ShallowConvNet and EEGWaveNet had significantly improved performance with more lightweight parameters. Notably, the architecture enabled ShallowConvNet to achieve an average accuracy of 87.2%, sensitivity of 88.3%, weighted F1-score of 87.1%, and AUC of 92.7% for 21 epilepsy patients. Besides, the visualization outcomes underscored the ability of the optimal model to extract statistically significant spatiospectral energy differences within high-frequency EEG bands for SP classification tasks.

Domain-Incremental Learning Framework for Continual Motor Imagery EEG Classification Task

Due to inter-subject variability in electroencephalogram (EEG) signals, the generalization ability of many existing brain-computer interface (BCI) models is significantly limited. Although transfer learning (TL) offers a temporary solution, in scenarios requiring sustained knowledge transfer, the performance of TL-based models gradually declines as the number of transfers increases-a phenomenon known as catastrophic forgetting. To address this issue, we introduce a novel domain-incremental learning framework for the continual motor imagery (MI) EEG classification. Specifically, to learn and retain common features between subjects, we separate latent representations into subject-invariant and subject-specific features through adversarial training, while also proposing an extensible architecture to preserve features that are easily forgotten. Additionally, we incorporate a memory replay mechanism to reinforce previously acquired knowledge. Through extensive experiments, we demonstrate our framework's effectiveness in mitigating forgetting within the continual MI-EEG classification task.

Enhancing Word-Level Imagined Speech BCI Through Heterogeneous Transfer Learning

In this study, we proposed a novel heterogeneous transfer learning approach named Focused Speech Feature Transfer Learning (FSFTL), aimed at enhancing the performance of electroencephalogram (EEG)-based word-level Imagined Speech (IS) Brain-Computer Interface (BCI). In IS BCI, the classification accuracy for imagining specific words is relatively low due to the inherent complexity in high-level feature variations. However, the binary classification accuracy for IS/rest is significantly higher. FSFTL leverages the refined feature focusing capability of the binary IS/Rest classification task to effectively locate relevant features for the word-level task. The feature extractor in the IS/Rest model demonstrates robust decoding ability for low-level IS features in EEG signals. We applied this high-performance yet low-resolution feature extractor to a public dataset for five-word IS task. The classifier was retrained to handle an increased number of classification categories, and the feature extractor was further fine-tuned to accommodate higher-level classification tasks. Before the experiment, we aligned the data from the two datasets to maintain the versatility of the feature extractor. Our proposed FSFTL approach was compared with existing EEG models, showing a significant improvement. The FSFTL approach outperformed the backbone strategy with a 6% increase in mean accuracy across all fifteen subjects. This study highlights the commonality of features in EEG data of IS and their transferability across various datasets and tasks, which is beneficial for improving the decoding ability of word-level IS BCI.

Bi-Stream Adaptation Network for Motor Imagery Decoding

Neural activities in distinct brain regions variably contribute to the formation of motor imagery (MI). Utilizing the hidden contextual information can thereby enhance network performance by having a comprehensive understanding of MI. Besides, due to the non-stationarity of EEG, the global and local distributions of cross-session EEG from an individual vary in applications. Based on these ideas, a novel Bi-Stream Adaptation Network (BSAN) is proposed to generate multi-scale context dependencies and to bridge the cross-session discrepancies in MI classification. Specifically, a Bi-attention module is proposed to cultivate multi-scale temporal dependencies and figure out the predominant brain regions. After features extraction, a Bi-discriminator is trained to implement the task of domain adaptation both globally and locally. To validate the proposed BSAN, extensive experiments were conducted based on two public MI datasets. The results revealed that the proposed BSAN improved the performance and robustness of MI classification and outperformed several state-of-the-art methods.

Intended Speech Classification with EEG Signals Based on a Temporal Attention Mechanism: A Study of Mandarin Vowels

Speech brain-machine interfaces (BCIs) offer an effective means for patients with voice disorders to communicate, and research on decoding electroencephalography (EEG) signals related to intended speech can help to understand the mechanisms of language production in the brain. This study classified the intended speech EEG signals of four Chinese vowels, utilizing a dataset collected from 10 participants. A proposed TA-EEGNet model was employed, incorporating a temporal attention module. The model achieved an accuracy of 49.47%, surpassing other prevalent EEG classification models. The average accuracy in the binary classification of vowels was 69.83%. The vowels /u/ and /ü/ were classified with the lowest accuracy, suggesting difficulties in classifying vowels with similar articulatory movements based on intended speech EEG signals. Furthermore, the research analyzed the classification performance using data of different brain regions. The results showed that the auditory cortex, Broca's and Wernicke's areas, prefrontal cortex, and motor cortex outperformed the sensory cortex, indicating their contributions in the intended speech process of Mandarin vowels. Results also showed left hemisphere dominance. These findings contribute to the study of the neural mechanisms underlying speech production and articulatory movements, emphasizing the potential of speech BCIs to improve communication for people with speech disorders.

Bi-hemisphere Interaction Convolutional Neural Network for Motor Imagery Classification

Decoding EEG-based, Motor Imagery Brain-Computer Interfaces (MI-BCI) in a subject-independent manner is very challenging due to high dimensionality of the EEG signal, and high inter-subject variability. In recent years, Convolutional neural networks (CNNs) have significantly enhanced decoding accuracy. Nevertheless, the majority of these CNN designs did not explicitly incorporate the inter-hemisphere functional connections, omitting crucial spatial information. Notably, in binary MI decoding of the left-hand versus right-hand, the Event-Related Desynchronization is observed in the contralateral hemisphere. Building upon this concept and various Neuroscience research, we have designed a CNN architecture that forges a functional connection between the two hemispheres. Specifically, we applied the Channel Average Referencing to one hemisphere and compared the output with all channels of the opposite hemisphere. Then, we utilized the cosine similarity to identify the most correlated channels and combined with them the original hemisphere for spatial filtering to learn the inter-hemispheric connections. This innovative technique aligns more closely with the actual brain functionality. Our method has demonstrated superior results on the Cho2017 and OpenBMI datasets, underscoring its effectiveness.

Unsupervised Hybrid Deep Feature Encoder for Robust Feature Learning from Resting-State EEG Data

EEG classification is a challenging task due to the nonstationary nature of EEG data and the covariance shift induced by cross-subject variance. Recently, various machine learning and deep learning models have been developed to learn robust features for inter-subject EEG classification tasks. However, current existing models are designed based on active task-related EEG, with a lack of investigation into learning robust feature representation from resting-state EEG data. Given the differences in the nature of brain activities captured by resting-state and active task-related EEG, existing models might not be applicable to resting-state EEG. This study proposed an unsupervised hybrid deep feature encoder to learn robust feature representation in resting-state EEG data. It involves using a Variational Autoencoder (VAE) to learn latent feature representation, followed by a further feature selection conducted through a non-task-related sample-level proximity classification using K-means clustering. We demonstrate the efficiency of our proposed model through significantly improved classification accuracies compared to benchmark models, as well as the high between-subject separability manifested by the learned feature representation.

Multiband task related components enhance rapid cognition decoding for both small and similar objects

The cortically-coupled target recognition system based on rapid serial visual presentation (RSVP) has a wide range of applications in brain computer interface (BCI) fields such as medical and military. However, in the complex natural environment backgrounds, the identification of event-related potentials (ERP) of both small and similar objects that are quickly presented is a research challenge. Therefore, we designed corresponding experimental paradigms and proposed a multi-band task related components matching (MTRCM) method to improve the rapid cognitive decoding of both small and similar objects. We compared the areas under the receiver operating characteristic curve (AUC) between MTRCM and other 9 methods under different numbers of training sample using RSVP-ERP data from 50 subjects. The results showed that MTRCM maintained an overall superiority and achieved the highest average AUC (0.6562 ± 0.0091). We also optimized the frequency band and the time parameters of the method. The verification on public data sets further showed the necessity of designing MTRCM method. The MTRCM method provides a new approach for neural decoding of both small and similar RSVP objects, which is conducive to promote the further development of RSVP-BCI.

Toward EEG-Based Objective Assessment of Emotion Intensity

Understanding the temporal dynamics of emotion poses a significant challenge due to the lack of methods to measure them objectively. In this study, we propose a novel approach to tracking intensity (EI) based on electroencephalogram (EEG) during continuous exposure to affective stimulation. We design selective sampling strategies to validate the association between the prediction outcome of an EEG-based emotion recognition model and the prominence of emotion-related EEG patterns, evidenced by the improvement in the classification task of discriminating arousal and valence by 2.01% and 1.71%, respectively. This study constitutes a breakthrough in the objective evaluation of the temporal dynamics of emotions, proposing a promising avenue to refine EEG-based emotion recognition models through intensity-selective sampling. Furthermore, our findings can contribute to future affective studies by providing a reliable and objective measurement method to profile emotion dynamics.

Robust EEG-based Emotion Recognition Using an Inception and Two-sided Perturbation Model

Automated emotion recognition using electroencephalogram (EEG) signals has gained substantial attention. Although deep learning approaches exhibit strong performance, they often suffer from vulnerabilities to various perturbations, like environmental noise and adversarial attacks. In this paper, we propose an Inception feature generator and two-sided perturbation (INC-TSP) approach to enhance emotion recognition in brain-computer interfaces. INC-TSP integrates the Inception module for EEG data analysis and employs two-sided perturbation (TSP) as a defensive mechanism against input perturbations. TSP introduces worst-case perturbations to the model's weights and inputs, reinforcing the model's elasticity against adversarial attacks. The proposed approach addresses the challenge of maintaining accurate emotion recognition in the presence of input uncertainties. We validate INC-TSP in a subject-independent three-class emotion recognition scenario, demonstrating robust performance.

A Dual-Adversarial Model for Cross-Time and Cross-Subject Cognitive Workload Decoding

Electroencephalogram (EEG) signals are widely utilized in the field of cognitive workload decoding (CWD). However, when the recognition scenario is shifted from subject-dependent to subject-independent or spans a long period, the accuracy of CWD deteriorates significantly. Current solutions are either dependent on extensive training datasets or fail to maintain clear distinctions between categories, additionally lacking a robust feature extraction mechanism. In this paper, we tackle these issues by proposing a Bi-Classifier Joint Domain Adaptation (BCJDA) model for EEG-based cross-time and cross-subject CWD. Specifically, the model consists of a feature extractor, a domain discriminator, and a Bi-Classifier, containing two sets of adversarial processes for domain-wise alignment and class-wise alignment. In the adversarial domain adaptation, the feature extractor is forced to learn the common domain features deliberately. The Bi-Classifier also fosters the feature extractor to retain the category discrepancies of the unlabeled domain, so that its classification boundary is consistent with the labeled domain. Furthermore, different adversarial distance functions of the Bi-Classifier are adopted and evaluated in this model. We conduct classification experiments on a publicly available BCI competition dataset for recognizing low, medium, and high cognitive workload levels. The experimental results demonstrate that our proposed BCJDA model based on cross-gradient difference maximization achieves the best performance.

The Effect of Stress on a Personal Identification System Based on Electroencephalographic Signals

Personal identification systems based on electroencephalographic (EEG) signals have their own strengths and limitations. The stability of EEG signals strongly affects such systems. The human emotional state is one of the important factors that affects EEG signals' stability. Stress is a major emotional state that affects individuals' capability to perform day-to-day tasks. The main objective of this work is to study the effect of mental and emotional stress on such systems. Two experiments have been performed. In the first, we used hand-crafted features (time domain, frequency domain, and non-linear features), followed by a machine learning classifier. In the second, raw EEG signals were used as an input for the deep learning approaches. Different types of mental and emotional stress have been examined using two datasets, SAM 40 and DEAP. The proposed experiments proved that performing enrollment in a relaxed or calm state and identification in a stressed state have a negative effect on the identification system's performance. The best achieved accuracy for the DEAP dataset was 99.67% in the calm state and 96.67% in the stressed state. For the SAM 40 dataset, the best accuracy was 99.67%, 93.33%, 92.5%, and 91.67% for the relaxed state and stress caused by identifying mirror images, the Stroop color-word test, and solving arithmetic operations, respectively.

A Cascade xDAWN EEGNet Structure for Unified Visual-Evoked Related Potential Detection

Visual-based brain-computer interface (BCI) enables people to communicate with others by spelling words from the brain and helps professionals recognize targets in large numbers of images. P300 signals evoked by different types of stimuli, such as words or images, may vary significantly in terms of both amplitude and latency. A unified approach is required to detect variable P300 signals, which facilitates BCI applications, as well as deepens the understanding of the P300 generation mechanism. In this study, our proposed approach involves a cascade network structure that combines xDAWN and classical EEGNet techniques. This network is designed to classify target and non-target stimuli in both P300 speller and rapid serial visual presentation (RSVP) paradigms. The proposed approach is capable of recognizing more symbols with fewer repetitions (up to 5 rounds) compared to other models while possessing a better information transfer rate (ITR) as demonstrated on Dataset II (17.22 bits/min in the second repetition round) of BCI Competition III. Additionally, our approach has the highest unweighted average recall (UAR) performance for both 5 Hz ( 0.8134±0.0259 ) and 20 Hz ( 0.6527±0.0321 ) RSVP. The results show that the cascade network structure has better performance between both the P300 Speller and RSVP paradigms, manifesting that such a cascade structure is robust enough for dealing with P300-related signals (source code is available at https://github.com/embneural/Cascade-xDAWN-EEGNet-for-ERP-Detection).

EEGGAN-Net: enhancing EEG signal classification through data augmentation

Background

Emerging brain-computer interface (BCI) technology holds promising potential to enhance the quality of life for individuals with disabilities. Nevertheless, the constrained accuracy of electroencephalography (EEG) signal classification poses numerous hurdles in real-world applications.

Methods

In response to this predicament, we introduce a novel EEG signal classification model termed EEGGAN-Net, leveraging a data augmentation framework. By incorporating Conditional Generative Adversarial Network (CGAN) data augmentation, a cropped training strategy and a Squeeze-and-Excitation (SE) attention mechanism, EEGGAN-Net adeptly assimilates crucial features from the data, consequently enhancing classification efficacy across diverse BCI tasks.

Results

The EEGGAN-Net model exhibits notable performance metrics on the BCI Competition IV-2a and IV-2b datasets. Specifically, it achieves a classification accuracy of 81.3% with a kappa value of 0.751 on the IV-2a dataset, and a classification accuracy of 90.3% with a kappa value of 0.79 on the IV-2b dataset. Remarkably, these results surpass those of four other CNN-based decoding models.

Conclusions

In conclusion, the amalgamation of data augmentation and attention mechanisms proves instrumental in acquiring generalized features from EEG signals, ultimately elevating the overall proficiency of EEG signal classification.

EEG decoding with spatiotemporal convolutional neural network for visualization and closed-loop control of sensorimotor activities: A simultaneous EEG-fMRI study

Closed-loop neurofeedback training utilizes neural signals such as scalp electroencephalograms (EEG) to manipulate specific neural activities and the associated behavioral performance. A spatiotemporal filter for high-density whole-head scalp EEG using a convolutional neural network can overcome the ambiguity of the signaling source because each EEG signal includes information on the remote regions. We simultaneously acquired EEG and functional magnetic resonance images in humans during the brain-computer interface (BCI) based neurofeedback training and compared the reconstructed and modeled hemodynamic responses of the sensorimotor network. Filters constructed with a convolutional neural network captured activities in the targeted network with spatial precision and specificity superior to those of the EEG signals preprocessed with standard pipelines used in BCI-based neurofeedback paradigms. The middle layers of the trained model were examined to characterize the neuronal oscillatory features that contributed to the reconstruction. Analysis of the layers for spatial convolution revealed the contribution of distributed cortical circuitries to reconstruction, including the frontoparietal and sensorimotor areas, and those of temporal convolution layers that successfully reconstructed the hemodynamic response function. Employing a spatiotemporal filter and leveraging the electrophysiological signatures of the sensorimotor excitability identified in our middle layer analysis would contribute to the development of a further effective neurofeedback intervention.

A diagonal masking self-attention-based multi-scale network for motor imagery classification

Objective. Electroencephalography (EEG)-based motor imagery (MI) is a promising paradigm for brain-computer interface (BCI), but the non-stationarity and low signal-to-noise ratio of EEG signals make it a challenging task.Approach. To achieve high-precision MI classification, we propose a Diagonal Masking Self-Attention-based Multi-Scale Network (DMSA-MSNet) to fully develop, extract, and emphasize features from different scales. First, for local features, a multi-scale temporal-spatial block is proposed to extract features from different receptive fields. Second, an adaptive branch fusion block is specifically designed to bridge the semantic gap between these coded features from different scales. Finally, in order to analyze global information over long ranges, a diagonal masking self-attention block is introduced, which highlights the most valuable features in the data.Main results. The proposed DMSA-MSNet outperforms state-of-the-art models on the BCI Competition IV 2a and the BCI Competition IV 2b datasets.Significance. Our study achieves rich information extraction from EEG signals and provides an effective solution for MI classification.

A systematic evaluation of Euclidean alignment with deep learning for EEG decoding

Objective:Electroencephalography signals are frequently used for various Brain-Computer interface (BCI) tasks. While deep learning (DL) techniques have shown promising results, they are hindered by the substantial data requirements. By leveraging data from multiple subjects, transfer learning enables more effective training of DL models. A technique that is gaining popularity is Euclidean alignment (EA) due to its ease of use, low computational complexity, and compatibility with DL models. However, few studies evaluate its impact on the training performance of shared and individual DL models. In this work, we systematically evaluate the effect of EA combined with DL for decoding BCI signals.Approach:We used EA as a pre-processing step to train shared DL models with data from multiple subjects and evaluated their transferability to new subjects.Main results:Our experimental results show that it improves decoding in the target subject by 4.33% and decreases convergence time by more than 70%. We also trained individual models for each subject to use as a majority-voting ensemble classifier. In this scenario, using EA improved the 3-model ensemble accuracy by 3.71%. However, when compared to the shared model with EA, the ensemble accuracy was 3.62% lower.Significance:EA succeeds in the task of improving transfer learning performance with DL models and, could be used as a standard pre-processing technique.

Motor Imagery EEG Signal Classification Using Distinctive Feature Fusion with Adaptive Structural LASSO

A motor imagery brain-computer interface connects the human brain and computers via electroencephalography (EEG). However, individual differences in the frequency ranges of brain activity during motor imagery tasks pose a challenge, limiting the manual feature extraction for motor imagery classification. To extract features that match specific subjects, we proposed a novel motor imagery classification model using distinctive feature fusion with adaptive structural LASSO. Specifically, we extracted spatial domain features from overlapping and multi-scale sub-bands of EEG signals and mined discriminative features by fusing the task relevance of features with spatial information into the adaptive LASSO-based feature selection. We evaluated the proposed model on public motor imagery EEG datasets, demonstrating that the model has excellent performance. Meanwhile, ablation studies and feature selection visualization of the proposed model further verified the great potential of EEG analysis.

A panoramic driving perception fusion algorithm based on multi-task learning

With the rapid development of intelligent connected vehicles, there is an increasing demand for hardware facilities and onboard systems of driver assistance systems. Currently, most vehicles are constrained by the hardware resources of onboard systems, which mainly process single-task and single-sensor data. This poses a significant challenge in achieving complex panoramic driving perception technology. While the panoramic driving perception algorithm YOLOP has achieved outstanding performance in multi-task processing, it suffers from poor adaptability of feature map pooling operations and loss of details during downsampling. To address these issues, this paper proposes a panoramic driving perception fusion algorithm based on multi-task learning. The model training involves the introduction of different loss functions and a series of processing steps for lidar point cloud data. Subsequently, the perception information from lidar and vision sensors is fused to achieve synchronized processing of multi-task and multi-sensor data, thereby effectively improving the performance and reliability of the panoramic driving perception system. To evaluate the performance of the proposed algorithm in multi-task processing, the BDD100K dataset is used. The results demonstrate that, compared to the YOLOP model, the multi-task learning network performs better in lane detection, drivable area detection, and vehicle detection tasks. Specifically, the lane detection accuracy improves by 11.6%, the mean Intersection over Union (mIoU) for drivable area detection increases by 2.1%, and the mean Average Precision at 50% IoU (mAP50) for vehicle detection improves by 3.7%.

Optimizing motor imagery BCI models with hard trials removal and model refinement

Deep learning models have demonstrated remarkable performance in the classification of motor imagery BCI systems. However, these models exhibit sensitivity to challenging trials, often called hard trials, leading to performance degradation. In this paper, we address this issue by proposing two novel methods for identifying and mitigating the impact of hard trials on model performance. The first method leverages model prediction scores to discern hard trials. The second approach employs a quantitative explainable artificial intelligence (XAI) approach, enabling a more transparent and interpretable means of hard trial identification. The identified hard trials are removed from the entire motor imagery training and validation dataset, and the deep learning model is further re-trained using the dataset without hard trials. To evaluate the efficacy of these proposed methods, experiments were conducted on the Open BMI dataset. The results for hold-out analysis show that the proposed quantitative XAI- based hard trial removal method has statistically improved the average classification accuracy of the baseline deep CNN model from 63.77% to 68.70%, withp-value =7.66e-11for the subject-specific MI classification. Additionally, analyzing the scalp map representing the average relevance scores of correctly classified trials compared to a misclassified trial provides a deeper insight into identifying hard trials. The results indicate that the proposed quantitaive-based XAI approach outperformes the prediction-score-based approach in hard trial identification.

Multi-source deep domain adaptation ensemble framework for cross-dataset motor imagery EEG transfer learning

Objective. Electroencephalography (EEG) is an important kind of bioelectric signal for measuring physiological activities of the brain, and motor imagery (MI) EEG has significant clinical application prospects. Convolutional neural network has become a mainstream algorithm for MI EEG classification, however lack of subject-specific data considerably restricts its decoding accuracy and generalization performance. To address this challenge, a novel transfer learning (TL) framework using auxiliary dataset to improve the MI EEG classification performance of target subject is proposed in this paper.Approach. We developed a multi-source deep domain adaptation ensemble framework (MSDDAEF) for cross-dataset MI EEG decoding. The proposed MSDDAEF comprises three main components: model pre-training, deep domain adaptation, and multi-source ensemble. Moreover, for each component, different designs were examined to verify the robustness of MSDDAEF.Main results. Bidirectional validation experiments were performed on two large public MI EEG datasets (openBMI and GIST). The highest average classification accuracy of MSDDAEF reaches 74.28% when openBMI serves as target dataset and GIST serves as source dataset. While the highest average classification accuracy of MSDDAEF is 69.85% when GIST serves as target dataset and openBMI serves as source dataset. In addition, the classification performance of MSDDAEF surpasses several well-established studies and state-of-the-art algorithms.Significance. The results of this study show that cross-dataset TL is feasible for left/right-hand MI EEG decoding, and further indicate that MSDDAEF is a promising solution for addressing MI EEG cross-dataset variability.

Adaptive Multimodel Knowledge Transfer Matrix Machine for EEG Classification

The emerging matrix learning methods have achieved promising performances in electroencephalogram (EEG) classification by exploiting the structural information between the columns or rows of feature matrices. Due to the intersubject variability of EEG data, these methods generally need to collect a large amount of labeled individual EEG data, which would cause fatigue and inconvenience to the subjects. Insufficient subject-specific EEG data will weaken the generalization capability of the matrix learning methods in neural pattern decoding. To overcome this dilemma, we propose an adaptive multimodel knowledge transfer matrix machine (AMK-TMM), which can selectively leverage model knowledge from multiple source subjects and capture the structural information of the corresponding EEG feature matrices. Specifically, by incorporating least-squares (LS) loss with spectral elastic net regularization, we first present an LS support matrix machine (LS-SMM) to model the EEG feature matrices. To boost the generalization capability of LS-SMM in scenarios with limited EEG data, we then propose a multimodel adaption method, which can adaptively choose multiple correlated source model knowledge with a leave-one-out cross-validation strategy on the available target training data. We extensively evaluate our method on three independent EEG datasets. Experimental results demonstrate that our method achieves promising performances on EEG classification.

A Spatiotemporal Deep Learning Framework for Scalp EEG-Based Automated Pain Assessment in Children

OBJECTIVE

Common pain assessment approaches such as self-evaluation and observation scales are inappropriate for children as they require patients to have reasonable communication ability. Subjective, inconsistent, and discontinuous pain assessment in children may reduce therapeutic effectiveness and thus affect their later life.

METHODS

To address the need for suitable assessment measures, this paper proposes a spatiotemporal deep learning framework for scalp electroencephalogram (EEG)-based automated pain assessment in children. The dataset comprises scalp EEG data recorded from 33 pediatric patients with an arterial puncture as a pain stimulus. Two electrode reduction plans in line with clinical findings are proposed. Combining three-dimensional hand-crafted features and preprocessed raw signals, the proposed transformer-based pain assessment network (STPA-Net) integrates both spatial and temporal information.

RESULTS

STPA-Net achieves superior performance with a subject-independent accuracy of 87.83% for pain recognition, and outperforms other state-of-the-art approaches. The effectiveness of electrode combinations is explored to analyze pain-related cortical activities and correspondingly reduce cost. The two proposed electrode reduction plans both demonstrate competitive pain assessment performance qualitatively and quantitatively.

CONCLUSION AND SIGNIFICANCE

This study is the first to develop a scalp EEG-based automated pain assessment for children adopting a method that is objective, standardized, and consistent. The findings provide a potential reference for future clinical research.

Semi-supervised multi-source transfer learning for cross-subject EEG motor imagery classification

Electroencephalogram (EEG) motor imagery (MI) classification refers to the use of EEG signals to identify and classify subjects' motor imagery activities; this task has received increasing attention with the development of brain-computer interfaces (BCIs). However, the collection of EEG data is usually time-consuming and labor-intensive, which makes it difficult to obtain sufficient labeled data from the new subject to train a new model. Moreover, the EEG signals of different individuals exhibit significant differences, leading to a significant drop in the performance of a model trained on the existing subjects when directly classifying EEG signals acquired from new subjects. Therefore, it is crucial to make full use of the EEG data of the existing subjects and the unlabeled EEG data of the new target subject to improve the MI classification performance achieved for the target subject. This research study proposes a semi-supervised multi-source transfer (SSMT) learning model to address the above problems; the model learns informative and domain-invariant representations to address cross-subject MI-EEG classification tasks. In particular, a dynamic transferred weighting schema is presented to obtain the final predictions by integrating the weighted features derived from multi-source domains. The average accuracies achieved on two publicly available EEG datasets reach 83.57 % and 85.09 % , respectively, validating the effectiveness of the SSMT process. The SSMT process reveals the importance of informative and domain-invariant representations in MI classification tasks, as they make full use of the domain-invariant information acquired from each subject.

GCNs-Net: A Graph Convolutional Neural Network Approach for Decoding Time-Resolved EEG Motor Imagery Signals

Toward the development of effective and efficient brain-computer interface (BCI) systems, precise decoding of brain activity measured by an electroencephalogram (EEG) is highly demanded. Traditional works classify EEG signals without considering the topological relationship among electrodes. However, neuroscience research has increasingly emphasized network patterns of brain dynamics. Thus, the Euclidean structure of electrodes might not adequately reflect the interaction between signals. To fill the gap, a novel deep learning (DL) framework based on the graph convolutional neural networks (GCNs) is presented to enhance the decoding performance of raw EEG signals during different types of motor imagery (MI) tasks while cooperating with the functional topological relationship of electrodes. Based on the absolute Pearson's matrix of overall signals, the graph Laplacian of EEG electrodes is built up. The GCNs-Net constructed by graph convolutional layers learns the generalized features. The followed pooling layers reduce dimensionality, and the fully-connected (FC) softmax layer derives the final prediction. The introduced approach has been shown to converge for both personalized and groupwise predictions. It has achieved the highest averaged accuracy, 93.06% and 88.57% (PhysioNet dataset), 96.24% and 80.89% (high gamma dataset), at the subject and group level, respectively, compared with existing studies, which suggests adaptability and robustness to individual variability. Moreover, the performance is stably reproducible among repetitive experiments for cross-validation. The excellent performance of our method has shown that it is an important step toward better BCI approaches. To conclude, the GCNs-Net filters EEG signals based on the functional topological relationship, which manages to decode relevant features for brain MI. A DL library for EEG task classification including the code for this study is open source at https://github.com/SuperBruceJia/ EEG-DL for scientific research.

A two-stage transformer based network for motor imagery classification

Brain-computer interfaces (BCIs) are used to understand brain functioning and develop therapies for neurological and neurodegenerative disorders. Therefore, BCIs are crucial in rehabilitating motor dysfunction and advancing motor imagery applications. For motor imagery, electroencephalogram (EEG) signals are used to classify the subject's intention of moving a body part without actually moving it. This paper presents a two-stage transformer-based architecture that employs handcrafted features and deep learning techniques to enhance the classification performance on benchmarked EEG signals. Stage-1 is built on parallel convolution based EEGNet, multi-head attention, and separable temporal convolution networks for spatiotemporal feature extraction. Further, for enhanced classification, in stage-2, additional features and embeddings extracted from stage-1 are used to train TabNet. In addition, a novel channel cluster swapping data augmentation technique is also developed to handle the issue of limited samples for training deep learning architectures. The developed two-stage architecture offered an average classification accuracy of 88.5 % and 88.3 % on the BCI Competition IV-2a and IV-2b datasets, respectively, which is approximately 3.0 % superior over similar recent reported works.

Model and Data Dual-Driven Double-Point Observation Network for Ultra-Short MI EEG Classification

Although deep networks have succeeded in various signal classification tasks, the time sequence samples used to train the deep models are usually required to reach a certain length. Especially, in brain computer interface (BCI) research, around 3.5s-long motor imagery (MI) Electroencephalography (EEG) samples are needed to obtain satisfactory classification performance. This time-span requirement of the training samples makes real-time MI BCI systems impossible to implement based on deep networks, which restricts the related researches within laboratory and makes practical application hard to accomplish. To address this issue, a double-point observation deep network (DoNet) is developed to classify ultra-short samples buried in noise. First, an analytical solution is developed theoretically to perform ultra-short signal classification based on double-point couples. Then, a signal-noise model is constructed to study the interference of noise on classification based on double-point couples. Based on which, an independent identical distribution condition is utilized to improve the classification accuracy in a data-driven manner. Combining the theoretical model and data-driven mechanism, DoNet can construct a steady data-distribution for the double-point couples of the samples with the same label. Therefore, the conditional probability of each double-point couple of a test sample can be obtained. With a voting strategy, the samples can be accurately classified by fusing these conditional probabilities. Meanwhile, the noise interference can be suppressed. DoNet has been evaluated on two public EEG datasets. Compared to most state-of-the-art methods, the 1s-long EEG signal classification accuracy has been improved by more than 3%.