EEG-Based Cross-Subject Driver Drowsiness Recognition With an Interpretable Convolutional Neural Network

Exploration of an intrinsically explainable self-attention based model for prototype generation on single-channel EEG sleep stage classification

Prototype-based methods in deep learning offer interpretable explanations for decisions by comparing inputs to typical representatives in the data. This study explores the adaptation of SESM, a self-attention-based prototype method successful in electrocardiogram (ECG) tasks, for electroencephalogram (EEG) signals. The architecture is evaluated on sleep stage classification, exploring its efficacy in predicting stages with single-channel EEG. The model achieves comparable test accuracy compared to EEGNet, a state-of-the-art black-box architecture for EEG classification. The generated prototypical components are exaimed qualitatively and using the area over the perterbation curve (AOPC) indicate some alignment with expected bio-markers for different sleep stages such as alpha spindles and slow waves in non-REM sleep, but the results are severely limited by the model's ability to only extract and present information in the time-domain. Ablation studies are used to explore the impact of kernel size, number of heads, and diversity threshold on model performance and explainability. This study represents the first application of a self-attention based prototype method to EEG data and provides a step forward in explainable AI for EEG data analysis.

Systematic review of experimental paradigms and deep neural networks for electroencephalography-based cognitive workload detection

This article summarizes a systematic literature review of deep neural network-based cognitive workload (CWL) estimation from electroencephalographic (EEG) signals. The focus of this article can be delineated into two main elements: first is the identification of experimental paradigms prevalently employed for CWL induction, and second, is an inquiry about the data structure and input formulations commonly utilized in deep neural networks (DNN)-based CWL detection. The survey revealed several experimental paradigms that can reliably induce either graded levels of CWL or a desired cognitive state due to sustained induction of CWL. This article has characterized them with respect to the number of distinct CWL levels, cognitive states, experimental environment, and agents in focus. Further, this literature analysis found that DNNs can successfully detect distinct levels of CWL despite the inter-subject and inter-session variability typically observed in EEG signals. Several methodologies were found using EEG signals in its native representation of a two-dimensional matrix as input to the classification algorithm, bypassing traditional feature selection steps. More often than not, researchers used DNNs as black-box type models, and only a few studies employed interpretable or explainable DNNs for CWL detection. However, these algorithms were mostly post hoc data analysis and classification schemes, and only a few studies adopted real-time CWL estimation methodologies. Further, it has been suggested that using interpretable deep learning methodologies may shed light on EEG correlates of CWL, but this remains mostly an unexplored area. This systematic review suggests using networks sensitive to temporal dependencies and appropriate input formulations for each type of DNN architecture to achieve robust classification performance. An additional suggestion is to utilize transfer learning methods to achieve high generalizability across tasks (task-independent classifiers), while simple cross-subject data pooling may achieve the same for subject-independent classifiers.

CT-Net: an interpretable CNN-Transformer fusion network for fNIRS classification

Functional near-infrared spectroscopy (fNIRS), an optical neuroimaging technique, has been widely used in the field of brain activity recognition and brain-computer interface. Existing works have proposed deep learning-based algorithms for the fNIRS classification problem. In this paper, a novel approach based on convolutional neural network and Transformer, named CT-Net, is established to guide the deep modeling for the classification of mental arithmetic (MA) tasks. We explore the effect of data representations, and design a temporal-level combination of two raw chromophore signals to improve the data utilization and enrich the feature learning of the model. We evaluate our model on two open-access datasets and achieve the classification accuracy of 98.05% and 77.61%, respectively. Moreover, we explain our model by the gradient-weighted class activation mapping, which presents a high consistent between the contributing value of features learned by the model and the mapping of brain activity in the MA task. The results suggest the feasibility and interpretability of CT-Net for decoding MA tasks.

Cross-Channel Specific-Mutual Feature Transfer Learning for Motor Imagery EEG Signals Decoding

In recent years, with the rapid development of deep learning, various deep learning frameworks have been widely used in brain-computer interface (BCI) research for decoding motor imagery (MI) electroencephalogram (EEG) signals to understand brain activity accurately. The electrodes, however, record the mixed activities of neurons. If different features are directly embedded in the same feature space, the specific and mutual features of different neuron regions are not considered, which will reduce the expression ability of the feature itself. We propose a cross-channel specific-mutual feature transfer learning (CCSM-FT) network model to solve this problem. The multibranch network extracts the specific and mutual features of brain's multiregion signals. Effective training tricks are used to maximize the distinction between the two kinds of features. Suitable training tricks can also improve the effectiveness of the algorithm compared with novel models. Finally, we transfer two kinds of features to explore the potential of mutual and specific features to enhance the expressive power of the feature and use the auxiliary set to improve identification performance. The experimental results show that the network has a better classification effect in the BCI Competition IV-2a and the HGD datasets.

Efficient Generalized Electroencephalography-Based Drowsiness Detection Approach with Minimal Electrodes

Drowsiness is a main factor for various costly defects, even fatal accidents in areas such as construction, transportation, industry and medicine, due to the lack of monitoring vigilance in the mentioned areas. The implementation of a drowsiness detection system can greatly help to reduce the defects and accident rates by alerting individuals when they enter a drowsy state. This research proposes an electroencephalography (EEG)-based approach for detecting drowsiness. EEG signals are passed through a preprocessing chain composed of artifact removal and segmentation to ensure accurate detection followed by different feature extraction methods to extract the different features related to drowsiness. This work explores the use of various machine learning algorithms such as Support Vector Machine (SVM), the K nearest neighbor (KNN), the Naive Bayes (NB), the Decision Tree (DT), and the Multilayer Perceptron (MLP) to analyze EEG signals sourced from the DROZY database, carefully labeled into two distinct states of alertness (awake and drowsy). Segmentation into 10 s intervals ensures precise detection, while a relevant feature selection layer enhances accuracy and generalizability. The proposed approach achieves high accuracy rates of 99.84% and 96.4% for intra (subject by subject) and inter (cross-subject) modes, respectively. SVM emerges as the most effective model for drowsiness detection in the intra mode, while MLP demonstrates superior accuracy in the inter mode. This research offers a promising avenue for implementing proactive drowsiness detection systems to enhance occupational safety across various industries.

Ultra-Wide Band Radar Empowered Driver Drowsiness Detection with Convolutional Spatial Feature Engineering and Artificial Intelligence

Driving while drowsy poses significant risks, including reduced cognitive function and the potential for accidents, which can lead to severe consequences such as trauma, economic losses, injuries, or death. The use of artificial intelligence can enable effective detection of driver drowsiness, helping to prevent accidents and enhance driver performance. This research aims to address the crucial need for real-time and accurate drowsiness detection to mitigate the impact of fatigue-related accidents. Leveraging ultra-wideband radar data collected over five minutes, the dataset was segmented into one-minute chunks and transformed into grayscale images. Spatial features are retrieved from the images using a two-dimensional Convolutional Neural Network. Following that, these features were used to train and test multiple machine learning classifiers. The ensemble classifier RF-XGB-SVM, which combines Random Forest, XGBoost, and Support Vector Machine using a hard voting criterion, performed admirably with an accuracy of 96.6%. Additionally, the proposed approach was validated with a robust k-fold score of 97% and a standard deviation of 0.018, demonstrating significant results. The dataset is augmented using Generative Adversarial Networks, resulting in improved accuracies for all models. Among them, the RF-XGB-SVM model outperformed the rest with an accuracy score of 99.58%.

A robust deep learning system for screening of obstructive sleep apnea using T-F spectrum of ECG signals

Obstructive sleep apnea (OSA) is a non-communicable sleep-related medical condition marked by repeated disruptions in breathing during sleep. It may induce various cardiovascular and neurocognitive complications. Electrocardiography (ECG) is a useful method for detecting numerous health-related disorders. ECG signals provide a less complex and non-invasive solution for the screening of OSA. Automated and accurate detection of OSA may enhance diagnostic performance and reduce the clinician's workload. Traditional machine learning methods typically involve several labor-intensive manual procedures, including signal decomposition, feature evaluation, selection, and categorization. This article presents the time-frequency (T-F) spectrum classification of de-noised ECG data for the automatic screening of OSA patients using deep convolutional neural networks (DCNNs). At first, a filter-fusion algorithm is used to eliminate the artifacts from the raw ECG data. Stock-well transform (S-T) is employed to change filtered time-domain ECG into T-F spectrums. To discriminate between apnea and normal ECG signals, the obtained T-F spectrums are categorized using benchmark Alex-Net and Squeeze-Net, along with a less complex DCNN. The superiority of the presented system is measured by computing the sensitivity, specificity, accuracy, negative predicted value, precision, F1-score, and Fowlkes-Mallows index. The results of comparing all three utilized DCNNs reveal that the proposed DCNN requires fewer learning parameters and provides higher accuracy. An average accuracy of 95.31% is yielded using the proposed system. The presented deep learning system is lightweight and faster than Alex-Net and Squeeze-Net as it utilizes fewer learnable parameters, making it simple and reliable.

SPARK: A High-Efficiency Black-Box Domain Adaptation Framework for Source Privacy-Preserving Drowsiness Detection

Developing an effective and efficient electroencephalography (EEG)-based drowsiness monitoring system is crucial for enhancing road safety and reducing the risk of accidents. For general usage, cross-subject evaluation is indispensable. Despite progress in unsupervised domain adaptation (UDA) and source-free domain adaptation (SFDA) methods, these often rely on the availability of labeled source data or white-box source models, posing potential privacy risks. This study explores a more challenging setting of UDA for EEG-based drowsiness detection, termed black-box domain adaptation (BBDA). In BBDA, adaptation in the target domain relies solely on a black-box source model, without access to the source data or parameters of the source model. Specifically, we propose a framework called Self-distillation and Pseudo-labelling for Ensemble Deep Random Vector Functional Link (edRVFL)-based Black-box Knowledge Adaptation (SPARK). SPARK employs entropy-based selection of high-confidence samples, which are then pseudo-labeled to train a student edRVFL network. Subsequently, ensemble self-distillation is performed to extract knowledge by training the edRVFL using refined labels introduced by ensemble learning. This process further improves the robustness of the student edRVFL network. The features of the edRVFL are beneficial for improving the computational efficiency of the framework, making it more suitable for tasks involving small datasets. The proposed SPARK framework is evaluated on two publicly available driver drowsiness datasets. Experimental results demonstrate its superior performance over strong baselines, while significantly reducing training time. These findings underscore the potential for practical integration of the proposed framework into drowsiness monitoring systems, thereby contributing substantially to the privacy preservation of source subjects.

A novel deep-learning model based on τ-shaped convolutional network (τNet) with long short-term memory (LSTM) for physiological fatigue detection from EEG and EOG signals

In recent years, fatigue driving has become the main cause of traffic accidents, leading to increased attention towards fatigue detection systems. However, the pooling and strided convolutional operations in fatigue detection algorithm based on traditional deep learning methods may led to the loss of some useful information. This paper proposed a novel τ -shaped convolutional network ( τ Net ) aiming to address this issue. Unlike traditional network structures, τ Net incorporates the operations of upsampling features and concatenating high- and low-level features, enabling full utilization of useful information. Moreover, considering that the fatigue state is a mental state involving temporal evolution, we proposed the novel long short-term memory (LSTM)- τ -shaped convolutional network (LSTM- τ Net ), a parallel structure composed of LSTM and τ Net for fatigue detection, where τ Net extracts time-invariant features with location information, and LSTM extracts long temporal dependencies. We compared LSTM- τ Net with six competing methods based on two datasets. Results showed that the proposed algorithm achieved higher classification accuracy than the other methods, with 94.25% on EEG data (binary classification) and 82.19% on EOG data (triple classification). Additionally, the proposed algorithm exhibits low computational cost, good training stability, and robustness against insufficient training. Therefore, it is promising for further implementation of fatigue online detection systems.

EEGminer: discovering interpretable features of brain activity with learnable filters

Objective. The patterns of brain activity associated with different brain processes can be used to identify different brain states and make behavioural predictions. However, the relevant features are not readily apparent and accessible. Our aim is to design a system for learning informative latent representations from multichannel recordings of ongoing EEG activity.Approach: We propose a novel differentiable decoding pipeline consisting of learnable filters and a pre-determined feature extraction module. Specifically, we introduce filters parameterized by generalized Gaussian functions that offer a smooth derivative for stable end-to-end model training and allow for learning interpretable features. For the feature module, we use signal magnitude and functional connectivity estimates.Main results.We demonstrate the utility of our model on a new EEG dataset of unprecedented size (i.e. 721 subjects), where we identify consistent trends of music perception and related individual differences. Furthermore, we train and apply our model in two additional datasets, specifically for emotion recognition on SEED and workload classification on simultaneous task EEG workload. The discovered features align well with previous neuroscience studies and offer new insights, such as marked differences in the functional connectivity profile between left and right temporal areas during music listening. This agrees with the specialisation of the temporal lobes regarding music perception proposed in the literature.Significance. The proposed method offers strong interpretability of learned features while reaching similar levels of accuracy achieved by black box deep learning models. This improved trustworthiness may promote the use of deep learning models in real world applications. The model code is available athttps://github.com/SMLudwig/EEGminer/.

CSF-GTNet: A Novel Multi-Dimensional Feature Fusion Network Based on Convnext-GeLU- BiLSTM for EEG-Signals-Enabled Fatigue Driving Detection

Electroencephalography (EEG) signal has been recognized as an effective fatigue detection method, which can intuitively reflect the drivers' mental state. However, the research on multi-dimensional features in existing work could be much better. The instability and complexity of EEG signals will increase the difficulty of extracting data features. More importantly, most current work only treats deep learning models as classifiers. They ignored the features of different subjects learned by the model. Aiming at the above problems, this paper proposes a novel multi-dimensional feature fusion network, CSF-GTNet, based on time and space-frequency domains for fatigue detection. Specifically, it comprises Gaussian Time Domain Network (GTNet) and Pure Convolutional Spatial Frequency Domain Network (CSFNet). The experimental results show that the proposed method effectively distinguishes between alert and fatigue states. The accuracy rates are 85.16% and 81.48% on the self-made and SEED-VIG datasets, respectively, which are higher than the state-of-the-art methods. Moreover, we analyze the contribution of each brain region for fatigue detection through the brain topology map. In addition, we explore the changing trend of each frequency band and the significance between different subjects in the alert state and fatigue state through the heat map. Our research can provide new ideas in brain fatigue research and play a specific role in promoting the development of this field.

Explainable cross-task adaptive transfer learning for motor imagery EEG classification

Objective. In the field of motor imagery (MI) electroencephalography (EEG)-based brain-computer interfaces, deep transfer learning (TL) has proven to be an effective tool for solving the problem of limited availability in subject-specific data for the training of robust deep learning (DL) models. Although considerable progress has been made in the cross-subject/session and cross-device scenarios, the more challenging problem of cross-task deep TL remains largely unexplored.Approach. We propose a novel explainable cross-task adaptive TL method for MI EEG decoding. Firstly, similarity analysis and data alignment are performed for EEG data of motor execution (ME) and MI tasks. Afterwards, the MI EEG decoding model is obtained via pre-training with extensive ME EEG data and fine-tuning with partial MI EEG data. Finally, expected gradient-based post-hoc explainability analysis is conducted for the visualization of important temporal-spatial features.Main results. Extensive experiments are conducted on one large ME EEG High-Gamma dataset and two large MI EEG datasets (openBMI and GIST). The best average classification accuracy of our method reaches 80.00% and 72.73% for OpenBMI and GIST respectively, which outperforms several state-of-the-art algorithms. In addition, the results of the explainability analysis further validate the correlation between ME and MI EEG data and the effectiveness of ME/MI cross-task adaptation.Significance. This paper confirms that the decoding of MI EEG can be well facilitated by pre-existing ME EEG data, which largely relaxes the constraint of training samples for MI EEG decoding and is important in a practical sense.

Illuminating the Neural Landscape of Pilot Mental States: A Convolutional Neural Network Approach with Shapley Additive Explanations Interpretability

Predicting pilots' mental states is a critical challenge in aviation safety and performance, with electroencephalogram data offering a promising avenue for detection. However, the interpretability of machine learning and deep learning models, which are often used for such tasks, remains a significant issue. This study aims to address these challenges by developing an interpretable model to detect four mental states-channelised attention, diverted attention, startle/surprise, and normal state-in pilots using EEG data. The methodology involves training a convolutional neural network on power spectral density features of EEG data from 17 pilots. The model's interpretability is enhanced via the use of SHapley Additive exPlanations values, which identify the top 10 most influential features for each mental state. The results demonstrate high performance in all metrics, with an average accuracy of 96%, a precision of 96%, a recall of 94%, and an F1 score of 95%. An examination of the effects of mental states on EEG frequency bands further elucidates the neural mechanisms underlying these states. The innovative nature of this study lies in its combination of high-performance model development, improved interpretability, and in-depth analysis of the neural correlates of mental states. This approach not only addresses the critical need for effective and interpretable mental state detection in aviation but also contributes to our understanding of the neural underpinnings of these states. This study thus represents a significant advancement in the field of EEG-based mental state detection.

Characterisation of Cognitive Load Using Machine Learning Classifiers of Electroencephalogram Data

A high cognitive load can overload a person, potentially resulting in catastrophic accidents. It is therefore important to ensure the level of cognitive load associated with safety-critical tasks (such as driving a vehicle) remains manageable for drivers, enabling them to respond appropriately to changes in the driving environment. Although electroencephalography (EEG) has attracted significant interest in cognitive load research, few studies have used EEG to investigate cognitive load in the context of driving. This paper presents a feasibility study on the simulation of various levels of cognitive load through designing and implementing four driving tasks. We employ machine learning-based classification techniques using EEG recordings to differentiate driving conditions. An EEG dataset containing these four driving tasks from a group of 20 participants was collected to investigate whether EEG can be used as an indicator of changes in cognitive load. The collected dataset was used to train four Deep Neural Networks and four Support Vector Machine classification models. The results showed that the best model achieved a classification accuracy of 90.37%, utilising statistical features from multiple frequency bands in 24 EEG channels. Furthermore, the Gamma and Beta bands achieved higher classification accuracy than the Alpha and Theta bands during the analysis. The outcomes of this study have the potential to enhance the Human-Machine Interface of vehicles, contributing to improved safety.

Latent Space Coding Capsule Network for Mental Workload Classification

Mental workload can be monitored in real time, which helps us improve work efficiency by maintaining an appropriate workload level. Based on previous studies, we have known that features, such as band power and brain connectivity, can be utilized to classify the levels of mental workload. As band power and brain connectivity represent different but complementary information related to mental workload, it is helpful to integrate them together for workload classification. Although deep learning models have been utilized for workload classification based on EEG, the classification performance is not satisfactory. This is because the current models cannot well tackle variances in the features extracted from non-stationary EEG. In order to address this problem, we, in this study, proposed a novel deep learning model, called latent space coding capsule network (LSCCN). The features of band power and brain connectivity were fused and then modelled in a latent space. The subsequent convolutional and capsule modules were used for workload classification. The proposed LSCCN was compared to the state-of-the-art methods. The results demonstrated that the proposed LSCCN was superior to the compared methods. LSCCN achieved a higher testing accuracy with a relatively smaller standard deviation, indicating a more reliable classification across participants. In addition, we explored the distribution of the features and found that top discriminative features were localized in the frontal, parietal, and occipital regions. This study not only provides a novel deep learning model but also informs further studies in workload classification and promotes practical usage of workload monitoring.

Automatic Detection and Classification of Epileptic Seizures from EEG Data: Finding Optimal Acquisition Settings and Testing Interpretable Machine Learning Approach

Deep learning (DL) is emerging as a successful technique for automatic detection and differentiation of spontaneous seizures that may otherwise be missed or misclassified. Herein, we propose a system architecture based on top-performing DL models for binary and multigroup classifications with the non-overlapping window technique, which we tested on the TUSZ dataset. The system accurately detects seizure episodes (87.7% Sn, 91.16% Sp) and carefully distinguishes eight seizure types (95-100% Acc). An increase in EEG sampling rate from 50 to 250 Hz boosted model performance: the precision of seizure detection rose by 5%, and seizure differentiation by 7%. A low sampling rate is a reasonable solution for training reliable models with EEG data. Decreasing the number of EEG electrodes from 21 to 8 did not affect seizure detection but worsened seizure differentiation significantly: 98.24 ± 0.17 vs. 85.14 ± 3.14% recall. In detecting epileptic episodes, all electrodes provided equally informative input, but in seizure differentiation, their informative value varied. We improved model explainability with interpretable ML. Activation maximization highlighted the presence of EEG patterns specific to eight seizure types. Cortical projection of epileptic sources depicted differences between generalized and focal seizures. Interpretable ML techniques confirmed that our system recognizes biologically meaningful features as indicators of epileptic activity in EEG.

Towards best practice of interpreting deep learning models for EEG-based brain computer interfaces

Introduction

As deep learning has achieved state-of-the-art performance for many tasks of EEG-based BCI, many efforts have been made in recent years trying to understand what have been learned by the models. This is commonly done by generating a heatmap indicating to which extent each pixel of the input contributes to the final classification for a trained model. Despite the wide use, it is not yet understood to which extent the obtained interpretation results can be trusted and how accurate they can reflect the model decisions.

Methods

We conduct studies to quantitatively evaluate seven different deep interpretation techniques across different models and datasets for EEG-based BCI.

Results

The results reveal the importance of selecting a proper interpretation technique as the initial step. In addition, we also find that the quality of the interpretation results is inconsistent for individual samples despite when a method with an overall good performance is used. Many factors, including model structure and dataset types, could potentially affect the quality of the interpretation results.

Discussion

Based on the observations, we propose a set of procedures that allow the interpretation results to be presented in an understandable and trusted way. We illustrate the usefulness of our method for EEG-based BCI with instances selected from different scenarios.

An Efficient Hybrid Model for Patient-Independent Seizure Prediction Using Deep Learning

Recently, many researchers have deployed different deep learning techniques to predict epileptic seizure, using electroencephalogram signals. However, most of this research requires very large amounts of memory and complicated feature extraction algorithms. In addition, they could not precisely examine EEG signal characteristics, which led to poor prediction performance. In this research, a non-patient-specific epileptic seizure prediction approach is proposed. The proposed model integrates Wavelet-based EEG signal processing with deep learning architectures for efficient prediction of pre-ictal and inter-ictal signals. The proposed system uses different models of one-dimensional convolutional neural networks to discriminate between inter-ictal signal and pre-ictal signals in order to enhance prediction performance. Experiments have been carried out on a benchmark dataset to validate the robustness of the proposed model. The experimental results showed that the proposed approach achieved 93.4% for 16 patients and 97.87% for 6 patients. Experiments showed that the proposed model can predict epileptic seizures effectively, which can have remarkable potential in clinical applications.

Coupled Projection Transfer Metric Learning for Cross-Session Emotion Recognition from EEG

Distribution discrepancies between different sessions greatly degenerate the performance of video-evoked electroencephalogram (EEG) emotion recognition. There are discrepancies since the EEG signal is weak and non-stationary and these discrepancies are manifested in different trails in each session and even in some trails which belong to the same emotion. To this end, we propose a Coupled Projection Transfer Metric Learning (CPTML) model to jointly complete domain alignment and graph-based metric learning, which is a unified framework to simultaneously minimize cross-session and cross-trial divergences. By experimenting on the SEED_IV emotional dataset, we show that (1) CPTML exhibits a significantly better performance than several other approaches; (2) the cross-session distribution discrepancies are minimized and emotion metric graph across different trials are optimized in the CPTML-induced subspace, indicating the effectiveness of data alignment and metric exploration; and (3) critical EEG frequency bands and channels for emotion recognition are automatically identified from the learned projection matrices, providing more insights into the occurrence of the effect.