Deep learning with convolutional neural networks for EEG decoding and visualization

Adaptive filter of frequency bands based coordinate attention network for EEG-based motor imagery classification

Purpose

In the brain-computer interface (BCI), motor imagery (MI) could be defined as the Electroencephalogram (EEG) signals through imagined movements, and ultimately enabling individuals to control external devices. However, the low signal-to-noise ratio, multiple channels and non-linearity are the essential challenges of accurate MI classification. To tackle these issues, we investigate the role of adaptive frequency bands selection and spatial-temporal feature learning in decoding motor imagery.

Methods

We propose an Adaptive Filter of Frequency Bands based Coordinate Attention Network (AFFB-CAN) to improve the performance of MI classification. Specifically, we design the AFFB to adaptively obtain the upper and lower limits of frequency bands in order to alleviate information loss caused by manual selection. Next, we propose the CAN-based network to emphasize the key brain regions and temporal segments. And we design a multi-scale module to enhance temporal context learning.

Results

The conducted experiments on the BCI Competition IV-2a and 2b datasets reveal that our approach achieves an outstanding average accuracy, kappa values, and Macro F1-Score with 0.7825, 0.7104, and 0.7486 respectively. Similarly, for the BCI Competition IV-2b dataset, the average accuracy, kappa values, and F1-Score obtained are 0.8879, 0.7427, and 0.8734, respectively.

Conclusion

The proposed AFFB-CAN method improves the performance of MI classification. In addition, our study confirms previous findings that motor imagery is mainly associated with µ and β rhythms. Moreover, we also find that γ rhythms also play an important role in MI classification.

Unsupervised and semi-supervised domain adaptation networks considering both global knowledge and prototype-based local class information for Motor Imagery Classification

The non-stationarity of EEG signals results in variability across sessions, impeding model building and data sharing. In this paper, we propose a domain adaptation method called GPL, which simultaneously considers global knowledge and prototype-based local class information to enhance the classification accuracy of motor imagery signals. Depending on the amount of labeled data available in the target domain, the method is implemented in both unsupervised and semi-supervised versions. Specifically, at the global level, we employ the maximum mean difference (MMD) loss to globally constrain the feature space, achieving comprehensive alignment. In the context of class-level operations, we propose two memory banks designed to accommodate class prototypes in each domain and constrain feature embeddings by applying two prototype-based contrastive losses. The source contrastive loss is used to organize source features spatially based on categories, thereby reconciling inter-class and intra-class relationships, while the interactive contrastive loss is employed to facilitate cross-domain information interaction. Simultaneously, in unsupervised scenarios, to mitigate the adverse effects of excessive pseudo-labels, we introduce an entropy-aware strategy that dynamically evaluates the confidence level of target data and personalized constraints on the participation of interactive contrastive loss. To validate our approach, extensive experiments were conducted on a highly regarded public EEG dataset, namely Dataset IIa of the BCI Competition IV, as well as a large-scale EEG dataset called GigaDB. The experiments yielded average classification accuracies of 86.03% and 84.22% respectively. These results demonstrate that our method is an effective EEG decoding model, conducive to advancing the development of motor imagery brain-computer interfaces. The architecture proposed in this study and the code for data partitioning can be found at https://github.com/zhangdx21/GPL.

Classification of Internal and External Distractions in an Educational VR Environment Using Multimodal Features

Virtual reality (VR) can potentially enhance student engagement and memory retention in the classroom. However, distraction among participants in a VR-based classroom is a significant concern. Several factors, including mind wandering, external noise, stress, etc., can cause students to become internally and/or externally distracted while learning. To detect distractions, single or multi-modal features can be used. A single modality is found to be insufficient to detect both internal and external distractions, mainly because of individual variability. In this work, we investigated multi-modal features: eye tracking and EEG data, to classify the internal and external distractions in an educational VR environment. We set up our educational VR environment and equipped it for multi-modal data collection. We implemented different machine learning (ML) methods, including k-nearest-neighbors (kNN), Random Forest (RF), one-dimensional convolutional neural network - long short-term memory (1 D-CNN-LSTM), and two-dimensional convolutional neural networks (2D-CNN) to classify participants' internal and external distraction states using the multi-modal features. We performed cross-subject, cross-session, and gender-based grouping tests to evaluate our models. We found that the RF classifier achieves the highest accuracy over 83% in the cross-subject test, around 68% to 78% in the cross-session test, and around 90% in the gender-based grouping test compared to other models. SHAP analysis of the extracted features illustrated greater contributions from the occipital and prefrontal regions of the brain, as well as gaze angle, gaze origin, and head rotation features from the eye tracking data.

Session-independent subject-adaptive mental imagery BCI using selective filter-bank adaptive Riemannian features

The brain-computer interfaces (BCIs) facilitate the users to exploit information encoded in neural signals, specifically electroencephalogram (EEG), to control devices and for neural rehabilitation. Mental imagery (MI)-driven BCI predicts the user's pre-meditated mental objectives, which could be deployed as command signals. This paper presents a novel learning-based framework for classifying MI tasks using EEG-based BCI. In particular, our work focuses on the variation in inter-session data and the extraction of multi-spectral user-tailored features for robust performance. Thus, the goal is to create a calibration-free subject-adaptive learning framework for various mental imagery tasks not restricted to motor imagery alone. In this regard, critical spectral bands and the best temporal window are first selected from the EEG training trials of the subject based on the Riemannian user learning distance metric (Dscore) that checks for distinct and stable patterns. The filtered covariance matrices of the EEG trials in each spectral band are then transformed towards a reference covariance matrix using the Riemannian transfer learning, enabling the different sessions to be comparable. The evaluation of our proposed Selective Time-window and Multi-scale Filter-Bank with Adaptive Riemannian (STFB-AR) features on four public datasets, including disabled subjects, showed around 15% and 8% improvement in mean accuracy over baseline and fixed filter-bank models, respectively.

Shuffled ECA-Net for stress detection from multimodal wearable sensor data

BACKGROUND

Recently, stress has been recognized as a key factor in the emergence of individual and social issues. Numerous attempts have been made to develop sensor-augmented psychological stress detection techniques, although existing methods are often impractical or overly subjective. To overcome these limitations, we acquired a dataset utilizing both wireless wearable multimodal sensors and salivary cortisol tests for supervised learning. We also developed a novel deep neural network (DNN) model that maximizes the benefits of sensor fusion.

METHOD

We devised a DNN involving a shuffled efficient channel attention (ECA) module called a shuffled ECA-Net, which achieves advanced feature-level sensor fusion by considering inter-modality relationships. Through an experiment involving salivary cortisol tests on 26 participants, we acquired multiple bio-signals including electrocardiograms, respiratory waveforms, and electrogastrograms in both relaxed and stressed mental states. A training dataset was generated from the obtained data. Using the dataset, our proposed model was optimized and evaluated ten times through five-fold cross-validation, while varying a random seed.

RESULTS

Our proposed model achieved acceptable performance in stress detection, showing 0.916 accuracy, 0.917 sensitivity, 0.916 specificity, 0.914 F1-score, and 0.964 area under the receiver operating characteristic curve (AUROC). Furthermore, we demonstrated that combining multiple bio-signals with a shuffled ECA module can more accurately detect psychological stress.

CONCLUSIONS

We believe that our proposed model, coupled with the evidence for the viability of multimodal sensor fusion and a shuffled ECA-Net, would significantly contribute to the resolution of stress-related issues.

Cross-Modal Guiding Neural Network for Multimodal Emotion Recognition From EEG and Eye Movement Signals

Multimodal emotion recognition research is gaining attention because of the emerging trend of integrating information from different sensory modalities to improve performance. Electroencephalogram (EEG) signals are considered objective indicators of emotions and provide precise insights despite their complex data collection. In contrast, eye movement signals are more susceptible to environmental and individual differences but offer convenient data collection. Conventional emotion recognition methods typically use separate models for different modalities, potentially overlooking their inherent connections. This study introduces a cross-modal guiding neural network designed to fully leverage the strengths of both modalities. The network includes a dual-branch feature extraction module that simultaneously extracts features from EEG and eye movement signals. In addition, the network includes a feature guidance module that uses EEG features to direct eye movement feature extraction, reducing the impact of subjective factors. This study also introduces a feature reweighting module to explore emotion-related features within eye movement signals, thereby improving emotion classification accuracy. The empirical findings from both the SEED-IV dataset and our collected dataset substantiate the commendable performance of the model, thereby confirming its efficacy.

Cross-Channel Specific-Mutual Feature Transfer Learning for Motor Imagery EEG Signals Decoding

In recent years, with the rapid development of deep learning, various deep learning frameworks have been widely used in brain-computer interface (BCI) research for decoding motor imagery (MI) electroencephalogram (EEG) signals to understand brain activity accurately. The electrodes, however, record the mixed activities of neurons. If different features are directly embedded in the same feature space, the specific and mutual features of different neuron regions are not considered, which will reduce the expression ability of the feature itself. We propose a cross-channel specific-mutual feature transfer learning (CCSM-FT) network model to solve this problem. The multibranch network extracts the specific and mutual features of brain's multiregion signals. Effective training tricks are used to maximize the distinction between the two kinds of features. Suitable training tricks can also improve the effectiveness of the algorithm compared with novel models. Finally, we transfer two kinds of features to explore the potential of mutual and specific features to enhance the expressive power of the feature and use the auxiliary set to improve identification performance. The experimental results show that the network has a better classification effect in the BCI Competition IV-2a and the HGD datasets.

STaRNet: A spatio-temporal and Riemannian network for high-performance motor imagery decoding

Brain-computer interfaces (BCIs), representing a transformative form of human-computer interaction, empower users to interact directly with external environments through brain signals. In response to the demands for high accuracy, robustness, and end-to-end capabilities within BCIs based on motor imagery (MI), this paper introduces STaRNet, a novel model that integrates multi-scale spatio-temporal convolutional neural networks (CNNs) with Riemannian geometry. Initially, STaRNet integrates a multi-scale spatio-temporal feature extraction module that captures both global and local features, facilitating the construction of Riemannian manifolds from these comprehensive spatio-temporal features. Subsequently, a matrix logarithm operation transforms the manifold-based features into the tangent space, followed by a dense layer for classification. Without preprocessing, STaRNet surpasses state-of-the-art (SOTA) models by achieving an average decoding accuracy of 83.29% and a kappa value of 0.777 on the BCI Competition IV 2a dataset, and 95.45% accuracy with a kappa value of 0.939 on the High Gamma Dataset. Additionally, a comparative analysis between STaRNet and several SOTA models, focusing on the most challenging subjects from both datasets, highlights exceptional robustness of STaRNet. Finally, the visualizations of learned frequency bands demonstrate that temporal convolutions have learned MI-related frequency bands, and the t-SNE analyses of features across multiple layers of STaRNet exhibit strong feature extraction capabilities. We believe that the accurate, robust, and end-to-end capabilities of the STaRNet will facilitate the advancement of BCIs.

Leveraging temporal dependency for cross-subject-MI BCIs by contrastive learning and self-attention

Brain-computer interfaces (BCIs) built based on motor imagery paradigm have found extensive utilization in motor rehabilitation and the control of assistive applications. However, traditional MI-BCI systems often exhibit suboptimal classification performance and require significant time for new users to collect subject-specific training data. This limitation diminishes the user-friendliness of BCIs and presents significant challenges in developing effective subject-independent models. In response to these challenges, we propose a novel subject-independent framework for learning temporal dependency for motor imagery BCIs by Contrastive Learning and Self-attention (CLS). In CLS model, we incorporate self-attention mechanism and supervised contrastive learning into a deep neural network to extract important information from electroencephalography (EEG) signals as features. We evaluate the CLS model using two large public datasets encompassing numerous subjects in a subject-independent experiment condition. The results demonstrate that CLS outperforms six baseline algorithms, achieving a mean classification accuracy improvement of 1.3 % and 4.71 % than the best algorithm on the Giga dataset and OpenBMI dataset, respectively. Our findings demonstrate that CLS can effectively learn invariant discriminative features from training data obtained from non-target subjects, thus showcasing its potential for building models for new users without the need for calibration.

Deep Neural Network-Based Empirical Mode Decomposition for Motor Imagery EEG Classification

Motor imagery refers to the brain's response during the mental simulation of physical activities, which can be detected through electroencephalogram (EEG) signals. However, EEG signals exhibit a low signal-to-noise ratio (SNR) due to various artifacts originating from other physiological sources. To enhance the classification performance of motor imagery tasks by increasing the SNR of EEG signals, several signal decomposition approaches have been proposed. Empirical mode decomposition (EMD) has shown promising results in extracting EEG components associated with motor imagery tasks more effectively than traditional linear decomposition algorithms such as Fourier and wavelet methods. Nevertheless, the EMD-based algorithm suffers from a significant challenge known as mode mixing, where frequency components intertwine with the intrinsic mode functions obtained through EMD. This issue severely hampers the accuracy of motor imagery classification. Despite numerous algorithms proposed, mode mixing remains a persistent issue. In this paper, we propose the Deep-EMD algorithm, a deep neural network-based approach to mode mixing problem. We employ two datasets to compare the motor imagery classification and mode mixing improvement achieved by the conventional EMD algorithms. Our experimental results demonstrate that the Deep-EMD algorithm effectively mitigates the mode mixing problem in decomposed EEG components, leading to improved motor imagery classification performance.

Temporal aware Mixed Attention-based Convolution and Transformer Network for cross-subject EEG emotion recognition

Emotion recognition is crucial for human-computer interaction, and electroencephalography (EEG) stands out as a valuable tool for capturing and reflecting human emotions. In this study, we propose a hierarchical hybrid model called Mixed Attention-based Convolution and Transformer Network (MACTN). This model is designed to collectively capture both local and global temporal information and is inspired by insights from neuroscientific research on the temporal dynamics of emotions. First, we introduce depth-wise temporal convolution and separable convolution to extract local temporal features. Then, a self-attention-based transformer is used to integrate the sparse global emotional features. Besides, channel attention mechanism is designed to identify the most task-relevant channels, facilitating the capture of relationships between different channels and emotional states. Extensive experiments are conducted on three public datasets under both offline and online evaluation modes. In the multi-class cross-subject online evaluation using the THU-EP dataset, MACTN demonstrates an approximate 8% enhancement in 9-class emotion recognition accuracy in comparison to state-of-the-art methods. In the multi-class cross-subject offline evaluation using the DEAP and SEED datasets, a comparable performance is achieved solely based on the raw EEG signals, without the need for prior knowledge or transfer learning during the feature extraction and learning process. Furthermore, ablation studies have shown that integrating self-attention and channel-attention mechanisms improves classification performance. This method won the Emotional BCI Competition's final championship in the World Robot Contest. The source code is available at https://github.com/ThreePoundUniverse/MACTN.

Multiclass motor imagery classification with Riemannian geometry and temporal-spectral selection

Motor imagery (MI) based brain-computer interfaces (BCIs) decode the users' intentions from electroencephalography (EEG) to achieve information control and interaction between the brain and external devices. In this paper, firstly, we apply Riemannian geometry to the covariance matrix extracted by spatial filtering to obtain robust and distinct features. Then, a multiscale temporal-spectral segmentation scheme is developed to enrich the feature dimensionality. In order to determine the optimal feature configurations, we utilize a linear learning-based temporal window and spectral band (TWSB) selection method to evaluate the feature contributions, which efficiently reduces the redundant features and improves the decoding efficiency without excessive loss of accuracy. Finally, support vector machines are used to predict the classification labels based on the selected MI features. To evaluate the performance of our model, we test it on the publicly available BCI Competition IV dataset 2a and 2b. The results show that the method has an average accuracy of 79.1% and 83.1%, which outperforms other existing methods. Using TWSB feature selection instead of selecting all features improves the accuracy by up to about 6%. Moreover, the TWSB selection method can effectively reduce the computational burden. We believe that the framework reveals more interpretable feature information of motor imagery EEG signals, provides neural responses discriminative with high accuracy, and facilitates the performance of real-time MI-BCI.

An Intersubject Brain-Computer Interface Based on Domain-Adversarial Training of Convolutional Neural Network

OBJECTIVE

Attention decoding plays a vital role in daily life, where electroencephalography (EEG) has been widely involved. However, training a universally effective model for everyone is impractical due to substantial interindividual variability in EEG signals. To tackle the above challenge, we propose an end-to-end brain-computer interface (BCI) framework, including temporal and spatial one-dimensional (1D) convolutional neural network and domain-adversarial training strategy, namely DA-TSnet.

METHOD

Specifically, DA-TSnet extracts temporal and spatial features of EEG, while it is jointly supervised by task loss and domain loss. During training, DA-TSnet aims to maximize the domain loss while simultaneously minimizing the task loss. We conduct an offline analysis, simulate online experiments on a self-collected dataset of 85 subjects, and real online experiments on 22 subjects.

MAIN RESULTS

DA-TSnet achieves a leave-one-subject-out (LOSO) cross-validation (CV) classification accuracy of 89.40% ± 9.96%, outperforming several state-of-the-art attention EEG decoding methods. In simulated online experiments, DA-TSnet achieves an outstanding accuracy of 88.07% ± 11.22%. In real online experiments, it achieves an average accuracy surpassing 86%.

SIGNIFICANCE

An end-to-end network framework does not rely on elaborate preprocessing and feature extraction steps, which saves time and human workload. Moreover, our framework utilizes domain-adversarial training neural network (DANN) to tackle the challenge posed by the high interindividual variability in EEG signals, which has significant reference value for handling other EEG signal decoding issues. Last, the performance of the DA-TSnet framework in offline and online experiments underscores its potential to facilitate more reliable applications.

Concept-based AI interpretability in physiological time-series data: Example of abnormality detection in electroencephalography

BACKGROUND AND OBJECTIVE

Despite recent performance advancements, deep learning models are not yet adopted in clinical practice on a wide scale. The intrinsic intransparency of such systems is commonly cited as one major reason for this reluctance. This has motivated methods that aim to provide explanations of model functioning. Known limitations of feature-based explanations have led to an increased interest in concept-based interpretability. Testing with Concept Activation Vectors (TCAV) employs human-understandable, abstract concepts to explain model behavior. The method has previously been applied to the medical domain in the context of electronic health records, retinal fundus images and magnetic resonance imaging.

METHODS

We explore the usage of TCAV for building interpretable models on physiological time series, using an example of abnormality detection in electroencephalography (EEG). For this purpose, we adopt the XceptionTime model, which is suitable for multi-channel physiological data of variable sizes. The model provides state-of-the-art performance on raw EEG data and is publically available. We propose and test several ideas regarding concept definition through metadata mining, using additional labeled EEG data and extracting interpretable signal characteristics in the form of frequencies. By including our own hospital data with analog labeling, we further evaluate the robustness of our approach.

RESULTS

The tested concepts show a TCAV score distribution that is in line with the clinical expectations, i.e. concepts known to have strong links with EEG pathologies (such as epileptiform discharges) received higher scores than the neutral concepts (e.g. sex). The scores were consistent across the applied concept generation strategies.

CONCLUSIONS

TCAV has the potential to improve interpretability of deep learning applied to multi-channel signals as well as to detect possible biases in the data. Still, further work on developing the strategies for concept definition and validation on clinical physiological time series is needed to better understand how to extract clinically relevant information from the concept sensitivity scores.

EEG-VTTCNet: A loss joint training model based on the vision transformer and the temporal convolution network for EEG-based motor imagery classification

Brain-computer interface (BCI) is a technology that directly connects signals between the human brain and a computer or other external device. Motor imagery electroencephalographic (MI-EEG) signals are considered a promising paradigm for BCI systems, with a wide range of potential applications in medical rehabilitation, human-computer interaction, and virtual reality. Accurate decoding of MI-EEG signals poses a significant challenge due to issues related to the quality of the collected EEG data and subject variability. Therefore, developing an efficient MI-EEG decoding network is crucial and warrants research. This paper proposes a loss joint training model based on the vision transformer (VIT) and the temporal convolutional network (EEG-VTTCNet) to classify MI-EEG signals. To take advantage of multiple modules together, the EEG-VTTCNet adopts a shared convolution strategy and a dual-branching strategy. The dual-branching modules perform complementary learning and jointly train shared convolutional modules with better performance. We conducted experiments on the BCI Competition IV-2a and IV-2b datasets, and the proposed network outperformed the current state-of-the-art techniques with an accuracy of 84.58% and 90.94%, respectively, for the subject-dependent mode. In addition, we used t-SNE to visualize the features extracted by the proposed network, further demonstrating the effectiveness of the feature extraction framework. We also conducted extensive ablation and hyperparameter tuning experiments to construct a robust network architecture that can be well generalized.

OS-SSVEP: One-shot SSVEP classification

It is extremely challenging to classify steady-state visual evoked potentials (SSVEPs) in scenarios characterized by a huge scarcity of calibration data where only one calibration trial is available for each stimulus target. To address this challenge, we introduce a novel approach named OS-SSVEP, which combines a dual domain cross-subject fusion network (CSDuDoFN) with the task-related and task-discriminant component analysis (TRCA and TDCA) based on data augmentation. The CSDuDoFN framework is designed to comprehensively transfer information from source subjects, while TRCA and TDCA are employed to exploit the information from the single available calibration trial of the target subject. Specifically, CSDuDoFN uses multi-reference least-squares transformation (MLST) to map data from both the source subjects and the target subject into the domain of sine-cosine templates, thereby reducing cross-subject domain gap and benefiting transfer learning. In addition, CSDuDoFN is fed with both transformed and original data, with an adequate fusion of their features occurring at different network layers. To capitalize on the calibration trial of the target subject, OS-SSVEP utilizes source aliasing matrix estimation (SAME)-based data augmentation to incorporate into the training process of the ensemble TRCA (eTRCA) and TDCA models. Ultimately, the outputs of CSDuDoFN, eTRCA, and TDCA are combined for the SSVEP classification. The effectiveness of our proposed approach is comprehensively evaluated on three publicly available SSVEP datasets, achieving the best performance on two datasets and competitive performance on the third. Further, it is worth noting that our method follows a different technical route from the current state-of-the-art (SOTA) method and the two are complementary. The performance is significantly improved when our method is combined with the SOTA method. This study underscores the potential to integrate the SSVEP-based brain-computer interface (BCI) into daily life. The corresponding source code is accessible at https://github.com/Sungden/One-shot-SSVEP-classification.

DOCTer: a novel EEG-based diagnosis framework for disorders of consciousness

Objective. Accurately diagnosing patients with disorders of consciousness (DOC) is challenging and prone to errors. Recent studies have demonstrated that EEG (electroencephalography), a non-invasive technique of recording the spontaneous electrical activity of brains, offers valuable insights for DOC diagnosis. However, some challenges remain: (1) the EEG signals have not been fully used; and (2) the data scale in most existing studies is limited. In this study, our goal is to differentiate between minimally conscious state (MCS) and unresponsive wakefulness syndrome (UWS) using resting-state EEG signals, by proposing a new deep learning framework.Approach. We propose DOCTer, an end-to-end framework for DOC diagnosis based on EEG. It extracts multiple pertinent features from the raw EEG signals, including time-frequency features and microstates. Meanwhile, it takes clinical characteristics of patients into account, and then combines all the features together for the diagnosis. To evaluate its effectiveness, we collect a large-scale dataset containing 409 resting-state EEG recordings from 128 UWS and 187 MCS cases.Main results. Evaluated on our dataset, DOCTer achieves the state-of-the-art performance, compared to other methods. The temporal/spectral features contributes the most to the diagnosis task. The cerebral integrity is important for detecting the consciousness level. Meanwhile, we investigate the influence of different EEG collection duration and number of channels, in order to help make the appropriate choices for clinics.Significance. The DOCTer framework significantly improves the accuracy of DOC diagnosis, helpful for developing appropriate treatment programs. Findings derived from the large-scale dataset provide valuable insights for clinics.

OxcarNet: sinc convolutional network with temporal and channel attention for prediction of oxcarbazepine monotherapy responses in patients with newly diagnosed epilepsy

Objective.Monotherapy with antiepileptic drugs (AEDs) is the preferred strategy for the initial treatment of epilepsy. However, an inadequate response to the initially prescribed AED is a significant indicator of a poor long-term prognosis, emphasizing the importance of precise prediction of treatment outcomes with the initial AED regimen in patients with epilepsy.Approach. We introduce OxcarNet, an end-to-end neural network framework developed to predict treatment outcomes in patients undergoing oxcarbazepine monotherapy. The proposed predictive model adopts a Sinc Module in its initial layers for adaptive identification of discriminative frequency bands. The derived feature maps are then processed through a Spatial Module, which characterizes the scalp distribution patterns of the electroencephalography (EEG) signals. Subsequently, these features are fed into an attention-enhanced Temporal Module to capture temporal dynamics and discrepancies. A channel module with an attention mechanism is employed to reveal inter-channel dependencies within the output of the Temporal Module, ultimately achieving response prediction. OxcarNet was rigorously evaluated using a proprietary dataset of retrospectively collected EEG data from newly diagnosed epilepsy patients at Nanjing Drum Tower Hospital. This dataset included patients who underwent long-term EEG monitoring in a clinical inpatient setting.Main results.OxcarNet demonstrated exceptional accuracy in predicting treatment outcomes for patients undergoing Oxcarbazepine monotherapy. In the ten-fold cross-validation, the model achieved an accuracy of 97.27%, and in the validation involving unseen patient data, it maintained an accuracy of 89.17%, outperforming six conventional machine learning methods and three generic neural decoding networks. These findings underscore the model's effectiveness in accurately predicting the treatment responses in patients with newly diagnosed epilepsy. The analysis of features extracted by the Sinc filters revealed a predominant concentration of predictive frequencies in the high-frequency range of the gamma band.Significance. The findings of our study offer substantial support and new insights into tailoring early AED selection, enhancing the prediction accuracy for the responses of AEDs.

EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts

Understanding emotional states is pivotal for the development of next-generation human-machine interfaces. Human behaviors in social interactions have resulted in psycho-physiological processes influenced by perceptual inputs. Therefore, efforts to comprehend brain functions and human behavior could potentially catalyze the development of AI models with human-like attributes. In this study, we introduce a multimodal emotion dataset comprising data from 30-channel electroencephalography (EEG), audio, and video recordings from 42 participants. Each participant engaged in a cue-based conversation scenario, eliciting five distinct emotions: neutral, anger, happiness, sadness, and calmness. Throughout the experiment, each participant contributed 200 interactions, which encompassed both listening and speaking. This resulted in a cumulative total of 8,400 interactions across all participants. We evaluated the baseline performance of emotion recognition for each modality using established deep neural network (DNN) methods. The Emotion in EEG-Audio-Visual (EAV) dataset represents the first public dataset to incorporate three primary modalities for emotion recognition within a conversational context. We anticipate that this dataset will make significant contributions to the modeling of the human emotional process, encompassing both fundamental neuroscience and machine learning viewpoints.

Unconscious classification of quantitative electroencephalogram features from propofol versus propofol combined with etomidate anesthesia using one-dimensional convolutional neural network

Objective

Establishing a convolutional neural network model for the recognition of characteristic raw electroencephalogram (EEG) signals is crucial for monitoring consciousness levels and guiding anesthetic drug administration.

Methods

This trial was conducted from December 2023 to March 2024. A total of 40 surgery patients were randomly divided into either a propofol group (1% propofol injection, 10 mL: 100 mg) (P group) or a propofol-etomidate combination group (1% propofol injection, 10 mL: 100 mg, and 0.2% etomidate injection, 10 mL: 20 mg, mixed at a 2:1 volume ratio) (EP group). In the P group, target-controlled infusion (TCI) was employed for sedation induction, with an initial effect site concentration set at 5-6 μg/mL. The EP group received an intravenous push with a dosage of 0.2 mL/kg. Six consciousness-related EEG features were extracted from both groups and analyzed using four prediction models: support vector machine (SVM), Gaussian Naive Bayes (GNB), artificial neural network (ANN), and one-dimensional convolutional neural network (1D CNN). The performance of the models was evaluated based on accuracy, precision, recall, and F1-score.

Results

The power spectral density (94%) and alpha/beta ratio (72%) demonstrated higher accuracy as indicators for assessing consciousness. The classification accuracy of the 1D CNN model for anesthesia-induced unconsciousness (97%) surpassed that of the SVM (83%), GNB (81%), and ANN (83%) models, with a significance level of p < 0.05. Furthermore, the mean and mean difference ± standard error of the primary power values for the EP and P groups during the induced period were as follows: delta (23.85 and 16.79, 7.055 ± 0.817, p < 0.001), theta (10.74 and 8.743, 1.995 ± 0.7045, p < 0.02), and total power (24.31 and 19.72, 4.588 ± 0.7107, p < 0.001).

Conclusion

Large slow-wave oscillations, power spectral density, and the alpha/beta ratio are effective indicators of changes in consciousness during intravenous anesthesia with a propofol-etomidate combination. These indicators can aid anesthesiologists in evaluating the depth of anesthesia and adjusting dosages accordingly. The 1D CNN model, which incorporates consciousness-related EEG features, represents a promising tool for assessing the depth of anesthesia.

Clinical Trial Registration

https://www.chictr.org.cn/index.html.

Decoding Multi-Class Motor Imagery From Unilateral Limbs Using EEG Signals

The EEG is a widely utilized neural signal source, particularly in motor imagery-based brain-computer interface (MI-BCI), offering distinct advantages in applications like stroke rehabilitation. Current research predominantly concentrates on the bilateral limbs paradigm and decoding, but the use scenarios for stroke rehabilitation are typically for unilateral upper limbs. There is a significant challenge to decoding unilateral MI of multitasks due to the overlapped spatial neural activities of the tasks. This study aims to formulate a novel MI-BCI experimental paradigm for unilateral limbs with multitasks. The paradigm encompasses four imagined movement directions: top-bottom, left-right, top right-bottom left, and top left-bottom right. Forty-six healthy subjects participated in this experiment. Commonly used machine learning techniques, such as FBCSP, EEGNet, deepConvNet, and FBCNet, were employed for evaluation. To improve decoding accuracy, we propose an MVCA method that introduces temporal convolution and attention mechanism to effectively capture temporal features from multiple perspectives. With the MVCA model, we have achieved 40.6% and 64.89% classification accuracies for the four-class and two-class scenarios (top right-bottom left and top left-bottom right), respectively. Conclusion: This is the first study demonstrating that motor imagery of multiple directions in unilateral limbs can be decoded. In particular, decoding two directions, right top to left bottom and left top to right bottom, provides the best accuracy, which sheds light on future studies. This study advances the development of the MI-BCI paradigm, offering preliminary evidence for the feasibility of decoding multiple directional information from EEG. This, in turn, enhances the dimensions of MI control commands.

Improving inter-session performance via relevant session-transfer for multi-session motor imagery classification

Motor imagery (MI)-based brain-computer interfaces (BCIs) using electroencephalography (EEG) have found practical applications in external device control. However, the non-stationary nature of EEG signals remains to obstruct BCI performance across multiple sessions, even for the same user. In this study, we aim to address the impact of non-stationarity, also known as inter-session variability, on multi-session MI classification performance by introducing a novel approach, the relevant session-transfer (RST) method. Leveraging the cosine similarity as a benchmark, the RST method transfers relevant EEG data from the previous session to the current one. The effectiveness of the proposed RST method was investigated through performance comparisons with the self-calibrating method, which uses only the data from the current session, and the whole-session transfer method, which utilizes data from all prior sessions. We validated the effectiveness of these methods using two datasets: a large MI public dataset (Shu Dataset) and our own dataset of gait-related MI, which includes both healthy participants and individuals with spinal cord injuries. Our experimental results revealed that the proposed RST method leads to a 2.29 % improvement (p < 0.001) in the Shu Dataset and up to a 6.37 % improvement in our dataset when compared to the self-calibrating method. Moreover, our method surpassed the performance of the recent highest-performing method that utilized the Shu Dataset, providing further support for the efficacy of the RST method in improving multi-session MI classification performance. Consequently, our findings confirm that the proposed RST method can improve classification performance across multiple sessions in practical MI-BCIs.

Efficient Multi-View Graph Convolutional Network with Self-Attention for Multi-Class Motor Imagery Decoding

Research on electroencephalogram-based motor imagery (MI-EEG) can identify the limbs of subjects that generate motor imagination by decoding EEG signals, which is an important issue in the field of brain-computer interface (BCI). Existing deep-learning-based classification methods have not been able to entirely employ the topological information among brain regions, and thus, the classification performance needs further improving. In this paper, we propose a multi-view graph convolutional attention network (MGCANet) with residual learning structure for multi-class MI decoding. Specifically, we design a multi-view graph convolution spatial feature extraction method based on the topological relationship of brain regions to achieve more comprehensive information aggregation. During the modeling, we build an adaptive weight fusion (Awf) module to adaptively merge feature from different brain views to improve classification accuracy. In addition, the self-attention mechanism is introduced for feature selection to expand the receptive field of EEG signals to global dependence and enhance the expression of important features. The proposed model is experimentally evaluated on two public MI datasets and achieved a mean accuracy of 78.26% (BCIC IV 2a dataset) and 73.68% (OpenBMI dataset), which significantly outperforms representative comparative methods in classification accuracy. Comprehensive experiment results verify the effectiveness of our proposed method, which can provide novel perspectives for MI decoding.

A Spatio-Temporal Capsule Neural Network with Self-Correlation Routing for EEG Decoding of Semantic Concepts of Imagination and Perception Tasks

Decoding semantic concepts for imagination and perception tasks (SCIP) is important for rehabilitation medicine as well as cognitive neuroscience. Electroencephalogram (EEG) is commonly used in the relevant fields, because it is a low-cost noninvasive technique with high temporal resolution. However, as EEG signals contain a high noise level resulting in a low signal-to-noise ratio, it makes decoding EEG-based semantic concepts for imagination and perception tasks (SCIP-EEG) challenging. Currently, neural network algorithms such as CNN, RNN, and LSTM have almost reached their limits in EEG signal decoding due to their own short-comings. The emergence of transformer methods has improved the classification performance of neural networks for EEG signals. However, the transformer model has a large parameter set and high complexity, which is not conducive to the application of BCI. EEG signals have high spatial correlation. The relationship between signals from different electrodes is more complex. Capsule neural networks can effectively model the spatial relationship between electrodes through vector representation and a dynamic routing mechanism. Therefore, it achieves more accurate feature extraction and classification. This paper proposes a spatio-temporal capsule network with a self-correlation routing mechaninsm for the classification of semantic conceptual EEG signals. By improving the feature extraction and routing mechanism, the model is able to more effectively capture the highly variable spatio-temporal features from EEG signals and establish connections between capsules, thereby enhancing classification accuracy and model efficiency. The performance of the proposed model was validated using the publicly accessible semantic concept dataset for imagined and perceived tasks from Bath University. Our model achieved average accuracies of 94.9%, 93.3%, and 78.4% in the three sensory modalities (pictorial, orthographic, and audio), respectively. The overall average accuracy across the three sensory modalities is 88.9%. Compared to existing advanced algorithms, the proposed model achieved state-of-the-art performance, significantly improving classification accuracy. Additionally, the proposed model is more stable and efficient, making it a better decoding solution for SCIP-EEG decoding.

Improving non-invasive trajectory decoding via neural correlates of continuous erroneous feedback processing

Objective. Over the last decades, error-related potentials (ErrPs) have repeatedly proven especially useful as corrective mechanisms in invasive and non-invasive brain-computer interfaces (BCIs). However, research in this context exclusively investigated the distinction of discrete events intocorrectorerroneousto the present day. Due to this predominant formulation as a binary classification problem, classical ErrP-based BCIs fail to monitor tasks demanding quantitative information on error severity rather than mere qualitative decisions on error occurrence. As a result, fine-tuned and natural feedback control based on continuously perceived deviations from an intended target remains beyond the capabilities of previously used BCI setups.Approach.To address this issue for future BCI designs, we investigated the feasibility of regressing rather than classifying error-related activity non-invasively from the brain.Main results.Using pre-recorded data from ten able-bodied participants in three sessions each and a multi-output convolutional neural network, we demonstrated the above-chance regression of ongoing target-feedback discrepancies from brain signals in a pseudo-online fashion. In a second step, we used this inferred information about the target deviation to correct the initially displayed feedback accordingly, reporting significant improvements in correlations between corrected feedback and target trajectories across feedback conditions.Significance.Our results indicate that continuous information on target-feedback discrepancies can be successfully regressed from cortical activity, paving the way to increasingly naturalistic, fine-tuned correction mechanisms for future BCI applications.

SpeechBrain-MOABB: An open-source Python library for benchmarking deep neural networks applied to EEG signals

Deep learning has revolutionized EEG decoding, showcasing its ability to outperform traditional machine learning models. However, unlike other fields, EEG decoding lacks comprehensive open-source libraries dedicated to neural networks. Existing tools (MOABB and braindecode) prevent the creation of robust and complete decoding pipelines, as they lack support for hyperparameter search across the entire pipeline, and are sensitive to fluctuations in results due to network random initialization. Furthermore, the absence of a standardized experimental protocol exacerbates the reproducibility crisis in the field. To address these limitations, we introduce SpeechBrain-MOABB, a novel open-source toolkit carefully designed to facilitate the development of a comprehensive EEG decoding pipeline based on deep learning. SpeechBrain-MOABB incorporates a complete experimental protocol that standardizes critical phases, such as hyperparameter search and model evaluation. It natively supports multi-step hyperparameter search for finding the optimal hyperparameters in a high-dimensional space defined by the entire pipeline, and multi-seed training and evaluation for obtaining performance estimates robust to the variability caused by random initialization. SpeechBrain-MOABB outperforms other libraries, including MOABB and braindecode, with accuracy improvements of 14.9% and 25.2% (on average), respectively. By enabling easy-to-use and easy-to-share decoding pipelines, our toolkit can be exploited by neuroscientists for decoding EEG with neural networks in a replicable and trustworthy way.

Social anxiety prediction based on ERP features: A deep learning approach

BACKGROUND

Social Anxiety Disorder is traditionally diagnosed using subjective scales that may lack accuracy. Recently, EEG technology has gained importance for anxiety detection due to its ability to capture stable and objective neurophysiological activities. However, existing methods mainly focus on extracting EEG features during resting states, with limited use of psychologically features like Event-Related Potential (ERP) in task-related states for anxiety detection in deep learning frameworks.

METHODS

We collected EEG data from 63 participants exposed to four facial expressions and extracted task-relevant features. Using the EEGNet model, we predicted social anxiety and evaluated its performance using metrics such as accuracy, F1 score, sensitivity, and specificity. We compared EEGNet's performance with Deep Convolutional Neural Network (DeepConvNet), ShallowConvNet, Bi-directional Long Short-Term Memory (BiLSTM), and SVM. To assess the generalizability of the results, we carried out the same procedure on our prior dataset.

RESULTS

EEGNet outperformed other models, achieving 99.16 % accuracy with Late Positive Potential (LPP). ERP components yielded higher accuracy than time-domain and frequency-domain features for social anxiety recognition. Accuracy was better for neutral and negative facial stimuli. Consistency across two datasets indicates stability of findings.

LIMITATIONS

Due to limited publicly available task-state datasets, only our own were used. Future studies could assess generalizability on larger datasets from different sources.

CONCLUSIONS

We conducted the first test of ERP features in anxiety recognition tasks. Results show ERP features have greater potential in social anxiety recognition, with LPP exhibiting high stability and accuracy. Outcomes indicate recognizing social anxiety with negative or neutral facial stimuli is more useful.

EEG-based motor imagery channel selection and classification using hybrid optimization and two-tier deep learning

Brain-computer interface (BCI) technology holds promise for individuals with profound motor impairments, offering the potential for communication and control. Motor imagery (MI)-based BCI systems are particularly relevant in this context. Despite their potential, achieving accurate and robust classification of MI tasks using electroencephalography (EEG) data remains a significant challenge. In this paper, we employed the Minimum Redundancy Maximum Relevance (MRMR) algorithm to optimize channel selection. Furthermore, we introduced a hybrid optimization approach that combines the War Strategy Optimization (WSO) and Chimp Optimization Algorithm (ChOA). This hybridization significantly enhances the classification model's overall performance and adaptability. A two-tier deep learning architecture is proposed for classification, consisting of a Convolutional Neural Network (CNN) and a modified Deep Neural Network (M-DNN). The CNN focuses on capturing temporal correlations within EEG data, while the M-DNN is designed to extract high-level spatial characteristics from selected EEG channels. Integrating optimal channel selection, hybrid optimization, and the two-tier deep learning methodology in our BCI framework presents an enhanced approach for precise and effective BCI control. Our model got 95.06% accuracy with high precision. This advancement has the potential to significantly impact neurorehabilitation and assistive technology applications, facilitating improved communication and control for individuals with motor impairments.

Design and application of pneumatic rehabilitation glove system based on brain-computer interface

Stroke has been the second leading cause of death and disability worldwide. With the innovation of therapeutic schedules, its death rate has decreased significantly but still guides chronic movement disorders. Due to the lack of independent activities and minimum exercise standards, the traditional rehabilitation means of occupational therapy and constraint-induced movement therapy pose challenges in stroke patients with severe impairments. Therefore, specific and effective rehabilitation methods seek innovation. To address the overlooked limitation, we design a pneumatic rehabilitation glove system. Specially, we developed a pneumatic glove, which utilizes ElectroEncephaloGram (EEG) acquisition to gain the EEG signals. A proposed EEGTran model is inserted into the system to distinguish the specific motor imagination behavior, thus, the glove can perform specific activities according to the patient's imagination, facilitating the patients with severe movement disorders and promoting the rehabilitation technology. The experimental results show that the proposed EEGTrans reached an accuracy of 87.3% and outperformed that of competitors. It demonstrates that our pneumatic rehabilitation glove system contributes to the rehabilitation training of stroke patients.

CTNet: a convolutional transformer network for EEG-based motor imagery classification

Brain-computer interface (BCI) technology bridges the direct communication between the brain and machines, unlocking new possibilities for human interaction and rehabilitation. EEG-based motor imagery (MI) plays a pivotal role in BCI, enabling the translation of thought into actionable commands for interactive and assistive technologies. However, the constrained decoding performance of brain signals poses a limitation to the broader application and development of BCI systems. In this study, we introduce a convolutional Transformer network (CTNet) designed for EEG-based MI classification. Firstly, CTNet employs a convolutional module analogous to EEGNet, dedicated to extracting local and spatial features from EEG time series. Subsequently, it incorporates a Transformer encoder module, leveraging a multi-head attention mechanism to discern the global dependencies of EEG's high-level features. Finally, a straightforward classifier module comprising fully connected layers is followed to categorize EEG signals. In subject-specific evaluations, CTNet achieved remarkable decoding accuracies of 82.52% and 88.49% on the BCI IV-2a and IV-2b datasets, respectively. Furthermore, in the challenging cross-subject assessments, CTNet achieved recognition accuracies of 58.64% on the BCI IV-2a dataset and 76.27% on the BCI IV-2b dataset. In both subject-specific and cross-subject evaluations, CTNet holds a leading position when compared to some of the state-of-the-art methods. This underscores the exceptional efficacy of our approach and its potential to set a new benchmark in EEG decoding.

Improving classification performance of motor imagery BCI through EEG data augmentation with conditional generative adversarial networks

In brain-computer interface (BCI), building accurate electroencephalogram (EEG) classifiers for specific mental tasks is critical for BCI performance. The classifiers are developed by machine learning (ML) and deep learning (DL) techniques, requiring a large dataset for training to build reliable and accurate models. However, collecting large enough EEG datasets is difficult due to intra-/inter-subject variabilities and experimental costs. This leads to the data scarcity problem, which causes overfitting issues to training samples, resulting in reducing generalization performance. To solve the EEG data scarcity problem and improve the performance of the EEG classifiers, we propose a novel EEG data augmentation (DA) framework using conditional generative adversarial networks (cGANs). An experimental study is implemented with two public EEG datasets, including motor imagery (MI) tasks (BCI competition IV IIa and III IVa), to validate the effectiveness of the proposed EEG DA method for the EEG classifiers. To evaluate the proposed cGAN-based DA method, we tested eight EEG classifiers for the experiment, including traditional MLs and state-of-the-art DLs with three existing EEG DA methods. Experimental results showed that most DA methods with proper DA proportion in the training dataset had higher classification performances than without DA. Moreover, applying the proposed DA method showed superior classification performance improvement than the other DA methods. This shows that the proposed method is a promising EEG DA method for enhancing the performances of the EEG classifiers in MI-based BCIs.

Classification of Known and Unknown Study Items in a Memory Task Using Single-Trial Event-Related Potentials and Convolutional Neural Networks

This study examines the feasibility of using event-related potentials (ERPs) obtained from electroencephalographic (EEG) recordings as biomarkers for long-term memory item classification. Previous studies have identified old/new effects in memory paradigms associated with explicit long-term memory and familiarity. Recent advancements in convolutional neural networks (CNNs) have enabled the classification of ERP trials under different conditions and the identification of features related to neural processes at the single-trial level. We employed this approach to compare three CNN models with distinct architectures using experimental data. Participants (N = 25) performed an association memory task while recording ERPs that were used for training and validation of the CNN models. The EEGNET-based model achieved the most reliable performance in terms of precision, recall, and specificity compared with the shallow and deep convolutional approaches. The classification accuracy of this model reached 62% for known items and 66% for unknown items. Good overall accuracy requires a trade-off between recall and specificity and depends on the architecture of the model and the dataset size. These results suggest the possibility of integrating ERP and CNN into online learning tools and identifying the underlying processes related to long-term memorization.

Evaluation of perceived urgency from single-trial EEG data elicited by upper-body vibration feedback using deep learning

Notification systems that convey urgency without adding cognitive burden are crucial in human-computer interaction. Haptic feedback systems, particularly those utilizing vibration feedback, have emerged as a compelling solution, capable of providing desirable levels of urgency depending on the application. High-risk applications require an evaluation of the urgency level elicited during critical notifications. Traditional evaluations of perceived urgency rely on subjective self-reporting and performance metrics, which, while useful, are not real-time and can be distracting from the task at hand. In contrast, EEG technology offers a direct, non-intrusive method of assessing the user's cognitive state. Leveraging deep learning, this study introduces a novel approach to evaluate perceived urgency from single-trial EEG data, induced by vibration stimuli on the upper body, utilizing our newly collected urgency-via-vibration dataset. The proposed model combines a 2D convolutional neural network with a temporal convolutional network to capture spatial and temporal EEG features, outperforming several established EEG models. The proposed model achieves an average classification accuracy of 83% through leave-one-subject-out cross-validation across three urgency classes (not urgent, urgent, and very urgent) from a single trial of EEG data. Furthermore, explainability analysis showed that the prefrontal brain region, followed by the central brain region, are the most influential in predicting the urgency level. A follow-up neural statistical analysis revealed an increase in event-related synchronization (ERS) in the theta frequency band (4-7 Hz) with the increased level of urgency, which is associated with high arousal and attention in the neuroscience literature. A limitation of this study is that the proposed model's performance was tested only the urgency-via-vibration dataset, which may affect the generalizability of the findings.

Independent Vector Analysis for Feature Extraction in Motor Imagery Classification

Independent vector analysis (IVA) can be viewed as an extension of independent component analysis (ICA) to multiple datasets. It exploits the statistical dependency between different datasets through mutual information. In the context of motor imagery classification based on electroencephalogram (EEG) signals for the brain-computer interface (BCI), several methods have been proposed to extract features efficiently, mainly based on common spatial patterns, filter banks, and deep learning. However, most methods use only one dataset at a time, which may not be sufficient for dealing with a multi-source retrieving problem in certain scenarios. From this perspective, this paper proposes an original approach for feature extraction through multiple datasets based on IVA to improve the classification of EEG-based motor imagery movements. The IVA components were used as features to classify imagined movements using consolidated classifiers (support vector machines and K-nearest neighbors) and deep classifiers (EEGNet and EEGInception). The results show an interesting performance concerning the clustering of MI-based BCI patients, and the proposed method reached an average accuracy of 86.7%.

Inter-participant transfer learning with attention based domain adversarial training for P300 detection

A Brain-computer interface (BCI) system establishes a novel communication channel between the human brain and a computer. Most event related potential-based BCI applications make use of decoding models, which requires training. This training process is often time-consuming and inconvenient for new users. In recent years, deep learning models, especially participant-independent models, have garnered significant attention in the domain of ERP classification. However, individual differences in EEG signals hamper model generalization, as the ERP component and other aspects of the EEG signal vary across participants, even when they are exposed to the same stimuli. This paper proposes a novel One-source domain transfer learning method based Attention Domain Adversarial Neural Network (OADANN) to mitigate data distribution discrepancies for cross-participant classification tasks. We train and validate our proposed model on both a publicly available OpenBMI dataset and a Self-collected dataset, employing a leave one participant out cross validation scheme. Experimental results demonstrate that the proposed OADANN method achieves the highest and most robust classification performance and exhibits significant improvements when compared to baseline methods (CNN, EEGNet, ShallowNet, DeepCovNet) and domain generalization methods (ERM, Mixup, and Groupdro). These findings underscore the efficacy of our proposed method.

Benchmarking brain-computer interface algorithms: Riemannian approaches vs convolutional neural networks

Objective.To date, a comprehensive comparison of Riemannian decoding methods with deep convolutional neural networks for EEG-based brain-computer interfaces remains absent from published work. We address this research gap by using MOABB, The Mother Of All BCI Benchmarks, to compare novel convolutional neural networks to state-of-the-art Riemannian approaches across a broad range of EEG datasets, including motor imagery, P300, and steady-state visual evoked potentials paradigms.Approach.We systematically evaluated the performance of convolutional neural networks, specifically EEGNet, shallow ConvNet, and deep ConvNet, against well-established Riemannian decoding methods using MOABB processing pipelines. This evaluation included within-session, cross-session, and cross-subject methods, to provide a practical analysis of model effectiveness and to find an overall solution that performs well across different experimental settings.Main results.We find no significant differences in decoding performance between convolutional neural networks and Riemannian methods for within-session, cross-session, and cross-subject analyses.Significance.The results show that, when using traditional Brain-Computer Interface paradigms, the choice between CNNs and Riemannian methods may not heavily impact decoding performances in many experimental settings. These findings provide researchers with flexibility in choosing decoding approaches based on factors such as ease of implementation, computational efficiency or individual preferences.

MOVING: A Multi-Modal Dataset of EEG Signals and Virtual Glove Hand Tracking

Brain-computer interfaces (BCIs) are pivotal in translating neural activities into control commands for external assistive devices. Non-invasive techniques like electroencephalography (EEG) offer a balance of sensitivity and spatial-temporal resolution for capturing brain signals associated with motor activities. This work introduces MOVING, a Multi-Modal dataset of EEG signals and Virtual Glove Hand Tracking. This dataset comprises neural EEG signals and kinematic data associated with three hand movements-open/close, finger tapping, and wrist rotation-along with a rest period. The dataset, obtained from 11 subjects using a 32-channel dry wireless EEG system, also includes synchronized kinematic data captured by a Virtual Glove (VG) system equipped with two orthogonal Leap Motion Controllers. The use of these two devices allows for fast assembly (∼1 min), although introducing more noise than the gold standard devices for data acquisition. The study investigates which frequency bands in EEG signals are the most informative for motor task classification and the impact of baseline reduction on gesture recognition. Deep learning techniques, particularly EEGnetV4, are applied to analyze and classify movements based on the EEG data. This dataset aims to facilitate advances in BCI research and in the development of assistive devices for people with impaired hand mobility. This study contributes to the repository of EEG datasets, which is continuously increasing with data from other subjects, which is hoped to serve as benchmarks for new BCI approaches and applications.

Iteratively Calibratable Network for Reliable EEG-Based Robotic Arm Control

Robotic arms are increasingly being utilized in shared workspaces, which necessitates the accurate interpretation of human intentions for both efficiency and safety. Electroencephalogram (EEG) signals, commonly employed to measure brain activity, offer a direct communication channel between humans and robotic arms. However, the ambiguous and unstable characteristics of EEG signals, coupled with their widespread distribution, make it challenging to collect sufficient data and hinder the calibration performance for new signals, thereby reducing the reliability of EEG-based applications. To address these issues, this study proposes an iteratively calibratable network aimed at enhancing the reliability and efficiency of EEG-based robotic arm control systems. The proposed method integrates feature inputs with network expansion techniques. This integration allows a network trained on an extensive initial dataset to adapt effectively to new users during calibration. Additionally, our approach combines motor imagery and speech imagery datasets to increase not only its intuitiveness but also the number of command classes. The evaluation is conducted in a pseudo-online manner, with a robotic arm operating in real-time to collect data, which is then analyzed offline. The evaluation results demonstrated that the proposed method outperformed the comparison group in 10 sessions and demonstrated competitive results when the two paradigms were combined. Therefore, it was confirmed that the network can be calibrated and personalized using only the new data from new users.

Evaluating the real-world usability of BCI control systems with augmented reality: a user study protocol

Brain-computer interfaces (BCI) enable users to control devices through their brain activity. Motor imagery (MI), the neural activity resulting from an individual imagining performing a movement, is a common control paradigm. This study introduces a user-centric evaluation protocol for assessing the performance and user experience of an MI-based BCI control system utilizing augmented reality. Augmented reality is employed to enhance user interaction by displaying environment-aware actions, and guiding users on the necessary imagined movements for specific device commands. One of the major gaps in existing research is the lack of comprehensive evaluation methodologies, particularly in real-world conditions. To address this gap, our protocol combines quantitative and qualitative assessments across three phases. In the initial phase, the BCI prototype's technical robustness is validated. Subsequently, the second phase involves a performance assessment of the control system. The third phase introduces a comparative analysis between the prototype and an alternative approach, incorporating detailed user experience evaluations through questionnaires and comparisons with non-BCI control methods. Participants engage in various tasks, such as object sorting, picking and placing, and playing a board game using the BCI control system. The evaluation procedure is designed for versatility, intending applicability beyond the specific use case presented. Its adaptability enables easy customization to meet the specific user requirements of the investigated BCI control application. This user-centric evaluation protocol offers a comprehensive framework for iterative improvements to the BCI prototype, ensuring technical validation, performance assessment, and user experience evaluation in a systematic and user-focused manner.

Reaching the ceiling? Empirical scaling behaviour for deep EEG pathology classification

Machine learning techniques, particularly deep convolutional neural networks (ConvNets), are increasingly being used to automate clinical EEG analysis, with the potential to reduce the clinical burden and improve patient care. However, further research is required before they can be used in clinical settings, particularly regarding the impact of the number of training samples and model parameters on their testing error. To address this, we present a comprehensive study of the empirical scaling behaviour of ConvNets for EEG pathology classification. We analysed the testing error with increasing the training samples and model size for four different ConvNet architectures. The focus of our experiments is width scaling, and we have increased the number of parameters to up to 1.8 million. Our evaluation was based on two publicly available datasets: the Temple University Hospital (TUH) Abnormal EEG Corpus and the TUH Abnormal Expansion Balanced EEG Corpus, which together contain 10,707 training samples. The results show that the testing error follows a saturating power-law with both model and dataset size. This pattern is consistent across different datasets and ConvNet architectures. Furthermore, empirically observed accuracies saturate at 85%-87%, which may be due to an imperfect inter-rater agreement on the clinical labels. The empirical scaling behaviour of the test performance with dataset and model size has significant implications for deep EEG pathology classification research and practice. Our findings highlight the potential of deep ConvNets for high-performance EEG pathology classification, and the identified scaling relationships provide valuable recommendations for the advancement of automated EEG diagnostics.

Exploring the frontier: Transformer-based models in EEG signal analysis for brain-computer interfaces

This review systematically explores the application of transformer-based models in EEG signal processing and brain-computer interface (BCI) development, with a distinct focus on ensuring methodological rigour and adhering to empirical validations within the existing literature. By examining various transformer architectures, such as the Temporal Spatial Transformer Network (TSTN) and EEG Conformer, this review delineates their capabilities in mitigating challenges intrinsic to EEG data, such as noise and artifacts, and their subsequent implications on decoding and classification accuracies across disparate mental tasks. The analytical scope extends to a meticulous examination of attention mechanisms within transformer models, delineating their role in illuminating critical temporal and spatial EEG features and facilitating interpretability in model decision-making processes. The discourse additionally encapsulates emerging works that substantiate the efficacy of transformer models in noise reduction of EEG signals and diversifying applications beyond the conventional motor imagery paradigm. Furthermore, this review elucidates evident gaps and propounds exploratory avenues in the applications of pre-trained transformers in EEG analysis and the potential expansion into real-time and multi-task BCI applications. Collectively, this review distils extant knowledge, navigates through the empirical findings, and puts forward a structured synthesis, thereby serving as a conduit for informed future research endeavours in transformer-enhanced, EEG-based BCI systems.

Machine learning of brain-specific biomarkers from EEG

BACKGROUND

Electroencephalography (EEG) has a long history as a clinical tool to study brain function, and its potential to derive biomarkers for various applications is far from exhausted. Machine learning (ML) can guide future innovation by harnessing the wealth of complex EEG signals to isolate relevant brain activity. Yet, ML studies in EEG tend to ignore physiological artefacts, which may cause problems for deriving biomarkers specific to the central nervous system (CNS).

METHODS

We present a framework for conceptualising machine learning from CNS versus peripheral signals measured with EEG. A signal representation based on Morlet wavelets allowed us to define traditional brain activity features (e.g. log power) and alternative inputs used by state-of-the-art ML approaches based on covariance matrices. Using more than 2600 EEG recordings from large public databases (TUAB, TDBRAIN), we studied the impact of peripheral signals and artefact removal techniques on ML models in age and sex prediction analyses.

FINDINGS

Across benchmarks, basic artefact rejection improved model performance, whereas further removal of peripheral signals using ICA decreased performance. Our analyses revealed that peripheral signals enable age and sex prediction. However, they explained only a fraction of the performance provided by brain signals.

INTERPRETATION

We show that brain signals and body signals, both present in the EEG, allow for prediction of personal characteristics. While these results may depend on specific applications, our work suggests that great care is needed to separate these signals when the goal is to develop CNS-specific biomarkers using ML.

FUNDING

All authors have been working for F. Hoffmann-La Roche Ltd.

Online continual decoding of streaming EEG signal with a balanced and informative memory buffer

Electroencephalography (EEG) based Brain Computer Interface (BCI) systems play a significant role in facilitating how individuals with neurological impairments effectively interact with their environment. In real world applications of BCI system for clinical assistance and rehabilitation training, the EEG classifier often needs to learn on sequentially arriving subjects in an online manner. As patterns of EEG signals can be significantly different for different subjects, the EEG classifier can easily erase knowledge of learnt subjects after learning on later ones as it performs decoding in online streaming scenario, namely catastrophic forgetting. In this work, we tackle this problem with a memory-based approach, which considers the following conditions: (1) subjects arrive sequentially in an online manner, with no large scale dataset available for joint training beforehand, (2) data volume from the different subjects could be imbalanced, (3) decoding difficulty of the sequential streaming signal vary, (4) continual classification for a long time is required. This online sequential EEG decoding problem is more challenging than classic cross subject EEG decoding as there is no large-scale training data from the different subjects available beforehand. The proposed model keeps a small balanced memory buffer during sequential learning, with memory data dynamically selected based on joint consideration of data volume and informativeness. Furthermore, for the more general scenarios where subject identity is unknown to the EEG decoder, aka. subject agnostic scenario, we propose a kernel based subject shift detection method that identifies underlying subject changes on the fly in a computationally efficient manner. We develop challenging benchmarks of streaming EEG data from sequentially arriving subjects with both balanced and imbalanced data volumes, and performed extensive experiments with a detailed ablation study on the proposed model. The results show the effectiveness of our proposed approach, enabling the decoder to maintain performance on all previously seen subjects over a long period of sequential decoding. The model demonstrates the potential for real-world applications.

Convolutional neural networks can identify brain interactions involved in decoding spatial auditory attention

Human listeners have the ability to direct their attention to a single speaker in a multi-talker environment. The neural correlates of selective attention can be decoded from a single trial of electroencephalography (EEG) data. In this study, leveraging the source-reconstructed and anatomically-resolved EEG data as inputs, we sought to employ CNN as an interpretable model to uncover task-specific interactions between brain regions, rather than simply to utilize it as a black box decoder. To this end, our CNN model was specifically designed to learn pairwise interaction representations for 10 cortical regions from five-second inputs. By exclusively utilizing these features for decoding, our model was able to attain a median accuracy of 77.56% for within-participant and 65.14% for cross-participant classification. Through ablation analysis together with dissecting the features of the models and applying cluster analysis, we were able to discern the presence of alpha-band-dominated inter-hemisphere interactions, as well as alpha- and beta-band dominant interactions that were either hemisphere-specific or were characterized by a contrasting pattern between the right and left hemispheres. These interactions were more pronounced in parietal and central regions for within-participant decoding, but in parietal, central, and partly frontal regions for cross-participant decoding. These findings demonstrate that our CNN model can effectively utilize features known to be important in auditory attention tasks and suggest that the application of domain knowledge inspired CNNs on source-reconstructed EEG data can offer a novel computational framework for studying task-relevant brain interactions.

HiRENet: Novel convolutional neural network architecture using Hilbert-transformed and raw electroencephalogram (EEG) for subject-independent emotion classification

BACKGROUND AND OBJECTIVES

Convolutional neural networks (CNNs) are the most widely used deep-learning framework for decoding electroencephalograms (EEGs) due to their exceptional ability to extract hierarchical features from high-dimensional EEG data. Traditionally, CNNs have primarily utilized multi-channel raw EEG data as the input tensor; however, the performance of CNN-based EEG decoding may be enhanced by incorporating phase information alongside amplitude information.

METHODS

This study introduces a novel CNN architecture called the Hilbert-transformed (HT) and raw EEG network (HiRENet), which incorporates both raw and HT EEG as inputs. This concurrent use of HT and raw EEG aims to integrate phase information with existing amplitude information, potentially offering a more comprehensive reflection of functional connectivity across various brain regions. The HiRENet model was developed using two CNN frameworks: ShallowFBCSPNet and a CNN with a residual block (ResCNN). The performance of the HiRENet model was assessed using a lab-made EEG database to classify human emotions, comparing three input modalities: raw EEG, HT EEG, and a combination of both signals. Additionally, the computational complexity was evaluated to validate the computational efficiency of the ResCNN design.

RESULTS

The HiRENet model based on ResCNN achieved the highest classification accuracy, with 86.03% for valence and 84.01% for arousal classifications, surpassing traditional CNN methodologies. Considering computational efficiency, ResCNN demonstrated superiority over ShallowFBCSPNet in terms of speed and inference time, despite having a higher parameter count.

CONCLUSION

Our experimental results showed that the proposed HiRENet can be potentially used as a new option to improve the overall performance for deep learning-based EEG decoding problems.

Single-Trial Detection and Classification of Event-Related Optical Signals for a Brain-Computer Interface Application

Event-related optical signals (EROS) measure fast modulations in the brain's optical properties related to neuronal activity. EROS offer a high spatial and temporal resolution and can be used for brain-computer interface (BCI) applications. However, the ability to classify single-trial EROS remains unexplored. This study evaluates the performance of neural network methods for single-trial classification of motor response-related EROS. EROS activity was obtained from a high-density recording montage covering the motor cortex during a two-choice reaction time task involving responses with the left or right hand. This study utilized a convolutional neural network (CNN) approach to extract spatiotemporal features from EROS data and perform classification of left and right motor responses. Subject-specific classifiers trained on EROS phase data outperformed those trained on intensity data, reaching an average single-trial classification accuracy of around 63%. Removing low-frequency noise from intensity data is critical for achieving discriminative classification results with this measure. Our results indicate that deep learning with high-spatial-resolution signals, such as EROS, can be successfully applied to single-trial classifications.

EEG-Based Motor Imagery Recognition Framework via Multisubject Dynamic Transfer and Iterative Self-Training

A robust decoding model that can efficiently deal with the subject and period variation is urgently needed to apply the brain-computer interface (BCI) system. The performance of most electroencephalogram (EEG) decoding models depends on the characteristics of specific subjects and periods, which require calibration and training with annotated data prior to application. However, this situation will become unacceptable as it would be difficult for subjects to collect data for an extended period, especially in the rehabilitation process of disability based on motor imagery (MI). To address this issue, we propose an unsupervised domain adaptation framework called iterative self-training multisubject domain adaptation (ISMDA) that focuses on the offline MI task. First, the feature extractor is purposefully designed to map the EEG to a latent space of discriminative representations. Second, the attention module based on dynamic transfer matches the source domain and target domain samples with a higher coincidence degree in latent space. Then, an independent classifier oriented to the target domain is employed in the first stage of the iterative training process to cluster the samples of the target domain through similarity. Finally, a pseudolabel algorithm based on certainty and confidence is employed in the second stage of the iterative training process to adequately calibrate the error between prediction and empirical probabilities. To evaluate the effectiveness of the model, extensive testing has been performed on three publicly available MI datasets, the BCI IV IIa, the High gamma dataset, and Kwon et al. datasets. The proposed method achieved 69.51%, 82.38%, and 90.98% cross-subject classification accuracy on the three datasets, which outperforms the current state-of-the-art offline algorithms. Meanwhile, all results demonstrated that the proposed method could address the main challenges of the offline MI paradigm.

Image classification and reconstruction from low-density EEG

Recent advances in visual decoding have enabled the classification and reconstruction of perceived images from the brain. However, previous approaches have predominantly relied on stationary, costly equipment like fMRI or high-density EEG, limiting the real-world availability and applicability of such projects. Additionally, several EEG-based paradigms have utilized artifactual, rather than stimulus-related information yielding flawed classification and reconstruction results. Our goal was to reduce the cost of the decoding paradigm, while increasing its flexibility. Therefore, we investigated whether the classification of an image category and the reconstruction of the image itself is possible from the visually evoked brain activity measured by a portable, 8-channel EEG. To compensate for the low electrode count and to avoid flawed predictions, we designed a theory-guided EEG setup and created a new experiment to obtain a dataset from 9 subjects. We compared five contemporary classification models with our setup reaching an average accuracy of 34.4% for 20 image classes on hold-out test recordings. For the reconstruction, the top-performing model was used as an EEG-encoder which was combined with a pretrained latent diffusion model via double-conditioning. After fine-tuning, we reconstructed images from the test set with a 1000 trial 50-class top-1 accuracy of 35.3%. While not reaching the same performance as MRI-based paradigms on unseen stimuli, our approach greatly improved the affordability and mobility of the visual decoding technology.

MutaPT: A Multi-Task Pre-Trained Transformer for Classifying State of Disorders of Consciousness Using EEG Signal

Deep learning (DL) has been demonstrated to be a valuable tool for classifying state of disorders of consciousness (DOC) using EEG signals. However, the performance of the DL-based DOC state classification is often challenged by the limited size of EEG datasets. To overcome this issue, we introduce multiple open-source EEG datasets to increase data volume and train a novel multi-task pre-training Transformer model named MutaPT. Furthermore, we propose a cross-distribution self-supervised (CDS) pre-training strategy to enhance the model's generalization ability, addressing data distribution shifts across multiple datasets. An EEG dataset of DOC patients is used to validate the effectiveness of our methods for the task of classifying DOC states. Experimental results show the superiority of our MutaPT over several DL models for EEG classification.

Harnessing Few-Shot Learning for EEG signal classification: a survey of state-of-the-art techniques and future directions

This paper presents a systematic literature review, providing a comprehensive taxonomy of Data Augmentation (DA), Transfer Learning (TL), and Self-Supervised Learning (SSL) techniques within the context of Few-Shot Learning (FSL) for EEG signal classification. EEG signals have shown significant potential in various paradigms, including Motor Imagery, Emotion Recognition, Visual Evoked Potentials, Steady-State Visually Evoked Potentials, Rapid Serial Visual Presentation, Event-Related Potentials, and Mental Workload. However, challenges such as limited labeled data, noise, and inter/intra-subject variability have impeded the effectiveness of traditional machine learning (ML) and deep learning (DL) models. This review methodically explores how FSL approaches, incorporating DA, TL, and SSL, can address these challenges and enhance classification performance in specific EEG paradigms. It also delves into the open research challenges related to these techniques in EEG signal classification. Specifically, the review examines the identification of DA strategies tailored to various EEG paradigms, the creation of TL architectures for efficient knowledge transfer, and the formulation of SSL methods for unsupervised representation learning from EEG data. Addressing these challenges is crucial for enhancing the efficacy and robustness of FSL-based EEG signal classification. By presenting a structured taxonomy of FSL techniques and discussing the associated research challenges, this systematic review offers valuable insights for future investigations in EEG signal classification. The findings aim to guide and inspire researchers, promoting advancements in applying FSL methodologies for improved EEG signal analysis and classification in real-world settings.