Subject-Specific Cognitive Workload Classification Using EEG-Based Functional Connectivity and Deep Learning

Reduction of cognitive fatigue and improved performance at a VR-based driving simulator using tRNS

Cognitive fatigue (CF) increases accident risk reducing performance, especially during complex tasks such as driving. We evaluated whether transcranial random noise stimulation (tRNS) could mitigate CF and improve driving performance. In a double-blind study, thirty participants performed a virtual reality truck driving task during real (n = 15) or sham (n = 15) tRNS applied bilaterally on the "anti-fatigue network". They completed two 30-min driving sessions while their driving performances were constantly monitored; heart rate was also monitored to evaluate arousal (Root-Mean-Square of successive R-R difference). tRNS was applied only during the first driving session to evaluate both online and offline stimulation effects. The primary outcome was CF reduction and performance improvement in the second (non-stimulated) driving session. Real tRNS significantly improved driving performances in the second driving session and reduced perceived CF. These results might also lead to the use of tRNS in those neurological disorders characterized by fatigue.

Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks

The detection of the cognitive tasks performed by a subject during data acquisition of a neuroimaging method has a wide range of applications: functioning of brain-computer interface (BCI), detection of neuronal disorders, neurorehabilitation for disabled patients, and many others. Recent studies show that the combination or fusion of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) demonstrates improved classification and detection performance compared to sole-EEG and sole-fNIRS. Deep learning (DL) networks are suitable for the classification of large volume time-series data like EEG and fNIRS. This study performs the decision fusion of EEG and fNIRS. The classification of EEG, fNIRS, and decision-fused EEG-fNIRSinto cognitive task labels is performed by DL networks. Two different open-source datasets of simultaneously recorded EEG and fNIRS are examined in this study. Dataset 01 is comprised of 26 subjects performing 3 cognitive tasks: n-back, discrimination or selection response (DSR), and word generation (WG). After data acquisition, fNIRS is converted to oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (HbR) in Dataset 01. Dataset 02 is comprised of 29 subjects who performed 2 tasks: motor imagery and mental arithmetic. The classification procedure of EEG and fNIRS (or HbO2, HbR) are carried out by 7 DL classifiers: convolutional neural network (CNN), long short-term memory network (LSTM), gated recurrent unit (GRU), CNN-LSTM, CNN-GRU, LSTM-GRU, and CNN-LSTM-GRU. After the classification of single modalities, their prediction scores or decisions are combined to obtain the decision-fused modality. The classification performance is measured by overall accuracy and area under the ROC curve (AUC). The highest accuracy and AUC recorded in Dataset 01 are 96% and 100% respectively; both by the decision fusion modality using CNN-LSTM-GRU. For Dataset 02, the highest accuracy and AUC are 82.76% and 90.44% respectively; both by the decision fusion modality using CNN-LSTM. The experimental result shows that decision-fused EEG-HbO2-HbR and EEG-fNIRSdeliver higher performances compared to their constituent unimodalities in most cases. For DL classifiers, CNN-LSTM-GRU in Dataset 01 and CNN-LSTM in Dataset 02 yield the highest performance.

A Dual-Adversarial Model for Cross-Time and Cross-Subject Cognitive Workload Decoding

Electroencephalogram (EEG) signals are widely utilized in the field of cognitive workload decoding (CWD). However, when the recognition scenario is shifted from subject-dependent to subject-independent or spans a long period, the accuracy of CWD deteriorates significantly. Current solutions are either dependent on extensive training datasets or fail to maintain clear distinctions between categories, additionally lacking a robust feature extraction mechanism. In this paper, we tackle these issues by proposing a Bi-Classifier Joint Domain Adaptation (BCJDA) model for EEG-based cross-time and cross-subject CWD. Specifically, the model consists of a feature extractor, a domain discriminator, and a Bi-Classifier, containing two sets of adversarial processes for domain-wise alignment and class-wise alignment. In the adversarial domain adaptation, the feature extractor is forced to learn the common domain features deliberately. The Bi-Classifier also fosters the feature extractor to retain the category discrepancies of the unlabeled domain, so that its classification boundary is consistent with the labeled domain. Furthermore, different adversarial distance functions of the Bi-Classifier are adopted and evaluated in this model. We conduct classification experiments on a publicly available BCI competition dataset for recognizing low, medium, and high cognitive workload levels. The experimental results demonstrate that our proposed BCJDA model based on cross-gradient difference maximization achieves the best performance.

A Convolutional Neural Network for SSVEP Identification by Using a Few-Channel EEG

The application of wearable electroencephalogram (EEG) devices is growing in brain-computer interfaces (BCI) owing to their good wearability and portability. Compared with conventional devices, wearable devices typically support fewer EEG channels. Devices with few-channel EEGs have been proven to be available for steady-state visual evoked potential (SSVEP)-based BCI. However, fewer-channel EEGs can cause the BCI performance to decrease. To address this issue, an attention-based complex spectrum-convolutional neural network (atten-CCNN) is proposed in this study, which combines a CNN with a squeeze-and-excitation block and uses the spectrum of the EEG signal as the input. The proposed model was assessed on a wearable 40-class dataset and a public 12-class dataset under subject-independent and subject-dependent conditions. The results show that whether using a three-channel EEG or single-channel EEG for SSVEP identification, atten-CCNN outperformed the baseline models, indicating that the new model can effectively enhance the performance of SSVEP-BCI with few-channel EEGs. Therefore, this SSVEP identification algorithm based on a few-channel EEG is particularly suitable for use with wearable EEG devices.

The effects of spatial leakage correction on the reliability of EEG-based functional connectivity networks

Electroencephalography (EEG) functional connectivity (FC) estimates are confounded by the volume conduction problem. This effect can be greatly reduced by applying FC measures insensitive to instantaneous, zero-lag dependencies (corrected measures). However, numerous studies showed that FC measures sensitive to volume conduction (uncorrected measures) exhibit higher reliability and higher subject-level identifiability. We tested how source reconstruction contributed to the reliability difference of EEG FC measures on a large (n = 201) resting-state data set testing eight FC measures (including corrected and uncorrected measures). We showed that the high reliability of uncorrected FC measures in resting state partly stems from source reconstruction: idiosyncratic noise patterns define a baseline resting-state functional network that explains a significant portion of the reliability of uncorrected FC measures. This effect remained valid for template head model-based, as well as individual head model-based source reconstruction. Based on our findings we made suggestions how to best use spatial leakage corrected and uncorrected FC measures depending on the main goals of the study.

Investigation of the neural correlation with task performance and its effect on cognitive load level classification

Electroencephalogram (EEG)-based cognitive load assessment is now an important assignment in psychological research. This type of research work is conducted by providing some mental task to the participants and their responses are counted through their EEG signal. In general assumption, it is considered that during different tasks, the cognitive workload is increased. This paper has investigated this specific idea and showed that the conventional hypothesis is not correct always. This paper showed that cognitive load can be varied according to the performance of the participants. In this paper, EEG data of 36 participants are taken against their resting and task (mental arithmetic) conditions. The features of the signal were extracted using the empirical mode decomposition (EMD) method and classified using the support vector machine (SVM) model. Based on the classification accuracy, some hypotheses are built upon the impact of subjects' performance on cognitive load. Based on some statistical consideration and graphical justification, it has been shown how the hypotheses are valid. This result will help to construct the machine learning-based model in predicting the cognitive load assessment more appropriately in a subject-independent approach.

Raw Electroencephalogram-Based Cognitive Workload Classification Using Directed and Nondirected Functional Connectivity Analysis and Deep Learning

With the phenomenal rise in internet-of-things devices, the use of electroencephalogram (EEG) based brain-computer interfaces (BCIs) can empower individuals to control equipment with thoughts. These allow BCI to be used and pave the way for pro-active health management and the development of internet-of-medical-things architecture. However, EEG-based BCIs have low fidelity, high variance, and EEG signals are very noisy. These challenges compel researchers to design algorithms that can process big data in real-time while being robust to temporal variations and other variations in the data. Another issue in designing a passive BCI is the regular change in user's cognitive state (measured through cognitive workload). Though considerable amount of research has been conducted on this front, methods that could withstand high variability in EEG data and still reflect the neuronal dynamics of cognitive state variations are lacking and much needed in literature. In this research, we evaluate the efficacy of a combination of functional connectivity algorithms and state-of-the-art deep learning algorithms for the classification of three different levels of cognitive workload. We acquire 64-channel EEG data from 23 participants executing the n-back task at three different levels; 1-back (low-workload condition), 2-back (medium-workload condition), and 3-back (high-workload condition). We compared two different functional connectivity algorithms, namely phase transfer entropy (PTE) and mutual information (MI). PTE is a directed functional connectivity algorithm, whereas MI is non-directed. Both methods are suitable for extracting functional connectivity matrices in real-time, which could eventually be used for rapid, robust, and efficient classification. For classification, we use the recently proposed BrainNetCNN deep learning model, designed specifically to classify functional connectivity matrices. Results reveal a classification accuracy of 92.81% with MI and BrainNetCNN and a staggering 99.50% with PTE and BrainNetCNN on test data. PTE can yield a higher classification accuracy due to its robustness to linear mixing of the data and its ability to detect functional connectivity across a range of analysis lags.

Special Issue "Emotion Intelligence Based on Smart Sensing"

Emotional intelligence is essential to maintaining human relationships in communities, organizations, and societies [...].

Fronto-parietal single-trial brain connectivity benefits successful memory recognition

Successful recognition has been known to produce distinct patterns of neural activity. Many studies have used spectral power or event-related potentials of single recognition-specific regions as classification features. However, this does not accurately reflect the mechanisms behind recognition, in that recognition requires multiple brain regions to work together. Hence, classification accuracy of subsequent memory performance could be improved by using functional connectivity within memory-related brain networks instead of using local brain activity as classifiers. In this study, we examined electroencephalography (EEG) signals while performing a word recognition memory task. Recorded EEG signals were collected using a 32-channel cap. Connectivity measures related to the left hemispheric fronto-parietal connectivity (P3 and F3) were found to contribute to the accurate recognition of previously studied memory items. Classification of subsequent memory outcome using connectivity features revealed that the classifier with support vector machine achieved the highest classification accuracy of 86.79 ± 5.93% (mean ± standard deviation) by using theta (3-8 Hz) connectivity during successful recognition trials. The results strongly suggest that highly accurate classification of subsequent memory outcome can be achieved by using single-trial functional connectivity.