Towards an effective cross-task mental workload recognition model using electroencephalography based on feature selection and support vector machine regression

Combined Effects of Moderate Hypoxia and Sleep Restriction on Mental Workload

Aircraft pilots face a high mental workload (MW) under environmental constraints induced by high altitude and sometimes sleep restriction (SR). Our aim was to assess the combined effects of hypoxia and sleep restriction on cognitive and physiological responses to different MW levels using the Multi-Attribute Test Battery (MATB)-II with an additional auditory Oddball-like task. Seventeen healthy subjects were subjected in random order to three 12-min periods of increased MW level (low, medium, and high): sleep restriction (SR, <3 h of total sleep time (TST)) vs. habitual sleep (HS, >6 h TST), hypoxia (HY, 2 h, FIO2 = 13.6%, ~3500 m vs. normoxia, NO, FIO2 = 21%). Following each MW level, participants completed the NASA-TLX subjective MW scale. Increasing MW decreases performance on the MATB-II Tracking task (p = 0.001, MW difficulty main effect) and increases NASA-TLX (p = 0.001). In the combined HY/SR condition, MATB-II performance was lower, and the NASA-TLX score was higher compared with the NO/HS condition, while no effect of hypoxia alone was observed. In the accuracy of the auditory task, there is a significant interaction between hypoxia and MW difficulty (F(2-176) = 3.14, p = 0.04), with lower values at high MW under hypoxic conditions. Breathing rate, pupil size, and amplitude of pupil dilation response (PDR) to auditory stimuli are associated with increased MW. These parameters are the best predictors of increased MW, independently of physiological constraints. Adding ECG, SpO2, or electrodermal conductance does not improve model performance. In conclusion, hypoxia and sleep restriction have an additive effect on MW. Physiological and electrophysiological responses must be taken into account when designing a MW predictive model and cross-validation.

Reproducible machine learning research in mental workload classification using EEG

This study addresses concerns about reproducibility in scientific research, focusing on the use of electroencephalography (EEG) and machine learning to estimate mental workload. We established guidelines for reproducible machine learning research using EEG and used these to assess the current state of reproducibility in mental workload modeling. We first started by summarizing the current state of reproducibility efforts in machine learning and in EEG. Next, we performed a systematic literature review on Scopus, Web of Science, ACM Digital Library, and Pubmed databases to find studies about reproducibility in mental workload prediction using EEG. All of this previous work was used to formulate guidelines, which we structured along the widely recognized Cross-Industry Standard Process for Data Mining (CRISP-DM) framework. By using these guidelines, researchers can ensure transparency and comprehensiveness of their methodologies, therewith enhancing collaboration and knowledge-sharing within the scientific community, and enhancing the reliability, usability and significance of EEG and machine learning techniques in general. A second systematic literature review extracted machine learning studies that used EEG to estimate mental workload. We evaluated the reproducibility status of these studies using our guidelines. We highlight areas studied and overlooked and identify current challenges for reproducibility. Our main findings include limitations on reporting performance on unseen test data, open sharing of data and code, and reporting of resources essential for training and inference processes.

Research on Mental Workload of Deep-Sea Oceanauts Driving Operation Tasks from EEG Data

A person's present mental state is closely associated with the frequency and temporal domain features of spontaneous electroencephalogram (EEG) impulses, which directly reflect neurophysiological signals of brain activity. EEG signals are employed in this study to measure the mental workload of drivers while they are operating a vehicle. A technique based on the quantum genetic algorithm (QGA) is suggested for improving the kernel function parameters of the multi-class support vector machine (MSVM). The performance of the algorithm based on the quantum genetic algorithm is found to be superior to that of other ways when other methods and the quantum genetic algorithm are evaluated for the parameter optimization of kernel function via simulation. A multi-classification support vector machine based on the quantum genetic algorithm (QGA-MSVM) is applied to identify the mental workload of oceanauts through the collection and feature extraction of EEG signals during driving simulation operation experiments in a sea basin area, a seamount area, and a hydrothermal area. Even with a limited data set, QGA-MSVM is able to accurately identify the cognitive burden experienced by ocean sailors, with an overall accuracy of 91.8%.

Altered functional connectivity during visual working memory state in patients with mild cognitive impairment

Patients with mild cognitive impairment (MCI) suffer from severe memory function impairment, especially working memory [1]. Based on Electroencephalogram (EEG), this study used power spectral density and large-scale network analysis to reveal the frequency changes of brain areas and the diverse network patterns during the visual WM coding stages between MCI and normal controls (NC). The results showed, compared to NC, the left and right prefrontal lobes and central regions has significant synchronization in the θ frequency band, while the left temporal lobe has significant desynchronization during the working memory coding state among MCI. Brain network analysis revealed the diverse network patterns during the WM coding stage between two group. Compared with the NC, the brain of MCI patients reduced the top-down visual network connection flow derived from frontal lobe to parietal lobe, and recruited a more up-down mechanism with a much stronger information flow from frontal lobe to occipital lobe during the WM coding state. This result provides a new perspective for the neural mechanism of WM deficits in MCI patients.Clinical Relevance-Abnormal EEG rhythm and connectivity of brain regions may be important indicators of WM disorders in patients with MCI.

Cross-task cognitive workload recognition using a dynamic residual network with attention mechanism based on neurophysiological signals

BACKGROUND AND OBJECTIVE

Evaluation of human cognitive workload (CW) helps improve the user experience of human-centered systems. To provide a continuous estimation of the CW, we built a CW recognizer that maps human electroencephalograms (EEGs) to discrete CW levels with deep learning tools. However, the EEG distribution varies when humans perform different cognitive tasks. There is thus a question on the capacity for generalizing the CW recognizer across tasks. In this study, we examined the CW's performance when it was trained and tested on two EEG databases corresponding to different human-machine tasks.

METHODS

A novel deep neural network-based EEG recognizer, dynamic residual network with attention mechanism (DRNA-Net), is proposed in the present study. By taking advantage of recurrent networks, the DRNA-Net further incorporates a self-attention mechanism in discovering robust EEG patterns across different cognitive tasks.

RESULTS

We designed an experiment that applied a multidimensional N-back task to induce the CW that consists of visual and auditory memory tasks. We validated the cross-task generalization capability of the DRNA-Net based on the EEG features extracted from the N-back task and a public database. The results show that the DRNA-Net achieves classification accuracy and Macro-F1 values are 0.6055 and 0.6067, respectively.

CONCLUSIONS

The performance of the DRNA-Net indicates that it has a certain ability of cross-task cognitive workload classification, which outperforms several shallow learners and deep convolutional neural networks under various conditions of the feature subsets.

Evaluation of a New Lightweight EEG Technology for Translational Applications of Passive Brain-Computer Interfaces

Technologies like passive brain-computer interfaces (BCI) can enhance human-machine interaction. Anyhow, there are still shortcomings in terms of easiness of use, reliability, and generalizability that prevent passive-BCI from entering real-life situations. The current work aimed to technologically and methodologically design a new gel-free passive-BCI system for out-of-the-lab employment. The choice of the water-based electrodes and the design of a new lightweight headset met the need for easy-to-wear, comfortable, and highly acceptable technology. The proposed system showed high reliability in both laboratory and realistic settings, performing not significantly different from the gold standard based on gel electrodes. In both cases, the proposed system allowed effective discrimination (AUC > 0.9) between low and high levels of workload, vigilance, and stress even for high temporal resolution (<10 s). Finally, the generalizability of the proposed system has been tested through a cross-task calibration. The system calibrated with the data recorded during the laboratory tasks was able to discriminate the targeted human factors during the realistic task reaching AUC values higher than 0.8 at 40 s of temporal resolution in case of vigilance and workload, and 20 s of temporal resolution for the stress monitoring. These results pave the way for ecologic use of the system, where calibration data of the realistic task are difficult to obtain.

Cross-task Cognitive Workload Recognition Based on EEG and Domain Adaptation

Cognitive workload recognition is pivotal to maintain the operator's health and prevent accidents in the human-robot interaction condition. So far, the focus of workload research is mostly restricted to a single task, yet cross-task cognitive workload recognition has remained a challenge. Furthermore, when extending to a new workload condition, the discrepancy of electroencephalogram (EEG) signals across various cognitive tasks limits the generalization of the existed model. To tackle this problem, we propose to construct the EEG-based cross-task cognitive workload recognition models using domain adaptation methods in a leave-one-task-out cross-validation setting, where we view any task of each subject as a domain. Specifically, we first design a fine-grained workload paradigm including working memory and mathematic addition tasks. Then, we explore four domain adaptation methods to bridge the discrepancy between the two different tasks. Finally, based on the supporting vector machine classifier, we conduct experiments to classify the low and high workload levels on a private EEG dataset. Experimental results demonstrate that our proposed task transfer framework outperforms the non-transfer classifier with improvements of 3% to 8% in terms of mean accuracy, and the transfer joint matching (TJM) consistently achieves the best performance.

Cross-Task Consistency of Electroencephalography-Based Mental Workload Indicators: Comparisons Between Power Spectral Density and Task-Irrelevant Auditory Event-Related Potentials

Mental workload (MWL) estimators based on ongoing electroencephalography (EEG) and event-related potentials (ERPs) have shown great potentials to build adaptive aiding systems for human-machine systems by estimating MWL in real time. However, extracting EEG features which are consistent in indicating MWL across different tasks is still one of the critical challenges. This study attempts to compare the cross-task consistency in indexing MWL variations between two commonly used EEG-based MWL indicators, power spectral density (PSD) of ongoing EEG and task-irrelevant auditory ERPs (tir-aERPs). The verbal N-back and the multi-attribute task battery (MATB), both with two difficulty levels, were employed in the experiment, along with task-irrelevant auditory probes. EEG was recorded from 17 subjects when they were performing the tasks. The tir-aERPs elicited by the auditory probes and the relative PSDs of ongoing EEG between two consecutive auditory probes were extracted and statistically analyzed to reveal the effects of MWL and task type. Discriminant analysis and support vector machine were employed to examine the generalization of tir-aERP and PSD features in indexing MWL variations across different tasks. The results showed that the amplitudes of tir-aERP components, N1, early P3a, late P3a, and the reorienting negativity, significantly decreased with the increasing MWL in both N-back and MATB. Task type had no obvious influence on the amplitudes and topological layout of the MWL-sensitive tir-aERP features. The relative PSDs in θ, α, and low β bands were also sensitive to MWL variations. However, the MWL-sensitive PSD features and their topological patterns were significantly affected by task type. The cross-task classification results based on tir-aERP features also significantly outperformed the PSD features. These results suggest that the tir-aERPs should be potentially more consistent MWL indicators across very different task types when compared to PSD. The current study may provide new insights to our understanding of the common and distinctive neuropsychological essences of MWL across different tasks.

Evaluation of Mental Workload in Working Memory Tasks with Different Information Types Based on EEG

To explore the effectiveness of using Electro- encephalogram (EEG) spectral power and multiscale sample entropy for accessing mental workload in different tasks, working memory tasks with different information types (verbal, object and spatial) and various mental loads were designed based on the N-Back paradigm. Subjective scores, accuracy and response time were used to verify the rationality of the tasks. EEGs from 18 normal adults were acquired when tasks were being performed, an independent component analysis (ICA) based artifact removal method were applied to get clean data. Linear (relative power in Theta and Alpha band, etc.) and nonlinear (multiscale sample entropy) features of EEGs were then extracted. Indices that can effectively reflect mental workload levels were selected by using multivariate analysis of variance statistical approach. Results showed that with the increment of task load, power of frontal Theta, Theta/Alpha ratio and sample entropies at scale more than 10 in parietal regions increased significantly first and decreased slightly then, while the power of central-parietal Alpha decreased significantly first and increased slightly then. Considering the difference between task types, no difference in power of frontal Theta, central-parietal Alpha and sample entropies at scales more than 10 of parietal regions were found between verbal and object tasks, as well as between two spatial tasks. No difference of frontal Theta/Alpha ratio was found in all the four tasks. The results can provide evidence for the mental workload evaluation in tasks with different information types.

Recognition of cognitive load with a stacking network ensemble of denoising autoencoders and abstracted neurophysiological features

The safety of human-machine systems can be indirectly evaluated based on operator's cognitive load levels at each temporal instant. However, relevant features of cognitive states are hidden behind in multiple sources of cortical neural responses. In this study, we developed a novel neural network ensemble, SE-SDAE, based on stacked denoising autoencoders (SDAEs) which identify different levels of cognitive load by electroencephalography (EEG) signals. To improve the generalization capability of the ensemble framework, a stacking-based approach is adopted to fuse the abstracted EEG features from activations of deep-structured hidden layers. In particular, we also combine multiple K-nearest neighbor and naive Bayesian classifiers with SDAEs to generate a heterogeneous classification committee to enhance ensemble's diversity. Finally, we validate the proposed SE-SDAE by comparing its performance with mainstream pattern classifiers for cognitive load evaluation to show its effectiveness.

Study of mental workload imposed by different tasks based on teleoperation

To explore mental workload and methods for dynamically monitoring mental workload imposed by complex tasks, this study constructed a virtual operating environment according to three cognitive steps: perception, judgment-making and action execution. Dynamic characteristics of mental workload were then analyzed employing subjective questionnaires, performance data and electroencephalography (EEG) characteristics. The analysis of non-linear dynamic characteristics of EEG signals showed that the fractal box dimension features of EEG signals are quite sensitive to the level of mental workload, exercising a significant impact on the four brain areas. The sample entropy is also quite sensitive to the level of mental workload, exercising a significant impact on the frontal, central and occipital areas. Based on this study, operational tasks can be dynamically assigned according to the state of personnel load and the safety and efficiency of the operation of the human-machine system can be ensured.

Adaptive Neuro-Fuzzy Fusion of Multi-Sensor Data for Monitoring a Pilot's Workload Condition

To realize an early warning of unbalanced workload in the aircraft cockpit, it is required to monitor the pilot’s real-time workload condition. For the purpose of building the mapping relationship from physiological and flight data to workload, a multi-source data fusion model is proposed based on a fuzzy neural network, mainly structured using a principal components extraction layer, fuzzification layer, fuzzy rules matching layer, and normalization layer. Aiming at the high coupling characteristic variables contributing to workload, principal component analysis reconstructs the feature data by reducing its dimension. Considering the uncertainty for a single variable to reflect overall workload, a fuzzy membership function and fuzzy control rules are defined to abstract the inference process. An error feedforward algorithm based on gradient descent is utilized for parameter learning. Convergence speed and accuracy can be adjusted by controlling the gradient descent rate and error tolerance threshold. Combined with takeoff and initial climbing tasks of a Boeing 737–800 aircraft, crucial performance indicators—including pitch angle, heading, and airspeed—as well as physiological indicators—including electrocardiogram (ECG), respiration, and eye movements—were featured. The mapping relationship between multi-source data and the comprehensive workload level synthesized using the NASA task load index was established. Experimental results revealed that the predicted workload corresponding to different flight phases and difficulty levels showed clear distinctions, thereby proving the validity of data fusion.

Sensor Networks for Aerospace Human-Machine Systems

Intelligent automation and trusted autonomy are being introduced in aerospace cyber-physical systems to support diverse tasks including data processing, decision-making, information sharing and mission execution. Due to the increasing level of integration/collaboration between humans and automation in these tasks, the operational performance of closed-loop human-machine systems can be enhanced when the machine monitors the operator's cognitive states and adapts to them in order to maximise the effectiveness of the Human-Machine Interfaces and Interactions (HMI²). Technological developments have led to neurophysiological observations becoming a reliable methodology to evaluate the human operator's states using a variety of wearable and remote sensors. The adoption of sensor networks can be seen as an evolution of this approach, as there are notable advantages if these sensors collect and exchange data in real-time, while their operation is controlled remotely and synchronised. This paper discusses recent advances in sensor networks for aerospace cyber-physical systems, focusing on Cognitive HMI² (CHMI²) implementations. The key neurophysiological measurements used in this context and their relationship with the operator's cognitive states are discussed. Suitable data analysis techniques based on machine learning and statistical inference are also presented, as these techniques allow processing both neurophysiological and operational data to obtain accurate cognitive state estimations. Lastly, to support the development of sensor networks for CHMI² applications, the paper addresses the performance characterisation of various state-of-the-art sensors and the propagation of measurement uncertainties through a machine learning-based inference engine. Results show that a proper sensor selection and integration can support the implementation of effective human-machine systems for various challenging aerospace applications, including Air Traffic Management (ATM), commercial airliner Single-Pilot Operations (SIPO), one-to-many Unmanned Aircraft Systems (UAS), and space operations management.

Assessing cognitive mental workload via EEG signals and an ensemble deep learning classifier based on denoising autoencoders

To estimate the reliability and cognitive states of operator performance in a human-machine collaborative environment, we propose a novel human mental workload (MW) recognizer based on deep learning principles and utilizing the features of the electroencephalogram (EEG). To determine personalized properties in high dimensional EEG indicators, we introduce a feature mapping layer in stacked denoising autoencoder (SDAE) that is capable of preserving the local information in EEG dynamics. The ensemble classifier is then built via the subject-specific integrated deep learning committee, and adapts to the cognitive properties of a specific human operator and alleviates inter-subject feature variations. We validate our algorithms and the ensemble SDAE classifier with local information preservation (denoted by EL-SDAE) on an EEG database collected during the execution of complex human-machine tasks. The classification performance indicates that the EL-SDAE outperforms several classical MW estimators when its optimal network architecture has been identified.

Learning Spatial-Spectral-Temporal EEG Features With Recurrent 3D Convolutional Neural Networks for Cross-Task Mental Workload Assessment

Mental workload assessment is essential for maintaining human health and preventing accidents. Most research on this issue is limited to a single task. However, cross-task assessment is indispensable for extending a pre-trained model to new workload conditions. Because brain dynamics are complex across different tasks, it is difficult to propose efficient human-designed features based on prior knowledge. Therefore, this paper proposes a concatenated structure of deep recurrent and 3D convolutional neural networks (R3DCNNs) to learn EEG features across different tasks without prior knowledge. First, this paper adds frequency and time dimensions to EEG topographic maps based on a Morlet wavelet transformation. Then, R3DCNN is proposed to simultaneously learn EEG features from the spatial, spectral, and temporal dimensions. The proposed model is validated based on the EEG signals collected from 20 subjects. This paper employs a binary classification of low and high mental workload across spatial n-back and arithmetic tasks. The results show that the R3DCNN achieves an average accuracy of 88.9%, which is a significant increase compared with that of the state-of-the-art methods. In addition, the visualization of the convolutional layers demonstrates that the deep neural network can extract detailed features. These results indicate that R3DCNN is capable of identifying the mental workload levels for cross-task conditions.

Feature Weight Driven Interactive Mutual Information Modeling for Heterogeneous Bio-Signal Fusion to Estimate Mental Workload

Many people suffer from high mental workload which may threaten human health and cause serious accidents. Mental workload estimation is especially important for particular people such as pilots, soldiers, crew and surgeons to guarantee the safety and security. Different physiological signals have been used to estimate mental workload based on the n-back task which is capable of inducing different mental workload levels. This paper explores a feature weight driven signal fusion method and proposes interactive mutual information modeling (IMIM) to increase the mental workload classification accuracy. We used EEG and ECG signals to validate the effectiveness of the proposed method for heterogeneous bio-signal fusion. The experiment of mental workload estimation consisted of signal recording, artifact removal, feature extraction, feature weight calculation, and classification. Ten subjects were invited to take part in easy, medium and hard tasks for the collection of EEG and ECG signals in different mental workload levels. Therefore, heterogeneous physiological signals of different mental workload states were available for classification. Experiments reveal that ECG can be utilized as a supplement of EEG to optimize the fusion model and improve mental workload estimation. Classification results show that the proposed bio-signal fusion method IMIM can increase the classification accuracy in both feature level and classifier level fusion. This study indicates that multi-modal signal fusion is promising to identify the mental workload levels and the fusion strategy has potential application of mental workload estimation in cognitive activities during daily life.

Task-Independent Mental Workload Classification Based Upon Common Multiband EEG Cortical Connectivity

Efficient classification of mental workload, an important issue in neuroscience, is limited, so far to single task, while cross-task classification remains a challenge. Furthermore, network approaches have emerged as a promising direction for studying the complex organization of the brain, enabling easier interpretation of various mental states. In this paper, using two mental tasks (N-back and mental arithmetic), we present a framework for cross- as well as within-task workload discrimination by utilizing multiband electroencephalography (EEG) cortical brain connectivity. In detail, we constructed functional networks in EEG source space in different frequency bands and considering the individual functional connections as classification features, we identified salient feature subsets based on a sequential feature selection algorithm. These connectivity subsets were able to provide accuracy of 87% for cross-task, 88% for N-back task, and 86% for mental arithmetic task. In conclusion, our method achieved to detect a small number of discriminative interactions among brain areas, leading to high accuracy in both within-task and cross-task classifications. In addition, the identified functional connectivity features, the majority of which were detected in frontal areas in theta and beta frequency bands, helped delineate the shared as well as the distinct neural mechanisms of the two mental tasks.