Development of an EEG-based workload measurement method in nuclear power plants

Measuring cognitive workload in the nuclear control room: a review

Despite the substantial literature and human factors guidance, evaluators report challenges in selecting cognitive workload measures for the evaluation of complex human-technology systems. A review of 32 articles found that self-report measures and secondary tasks were systematically sensitive to human-system interface conditions and correlated with physiological measures. Therefore, including a self-report measure of cognitive workload is recommended when evaluating human-system interfaces. Physiological measures were mainly used in method studies, and future research must demonstrate the utility of these measures for human-system evaluation in complex work settings. However, indexes of physiological measures showed promise for cognitive workload assessment. The review revealed a limited focus on the measurement of excessive cognitive workload, although this is a key topic in nuclear process control. To support human-system evaluation of adequate cognitive workload, future research on behavioural measures may be useful in the identification and analysis of underload and overload.

Mental Workload Classification and Tasks Detection in Multitasking: Deep Learning Insights from EEG Study

While the term task load (TL) refers to external task demands, the amount of work, or the number of tasks to be performed, mental workload (MWL) refers to the individual's effort, mental capacity, or cognitive resources utilized while performing a task. MWL in multitasking scenarios is often closely linked with the quantity of tasks a person is handling within a given timeframe. In this study, we challenge this hypothesis from the perspective of electroencephalography (EEG) using a deep learning approach. We conducted an EEG experiment with 50 participants performing NASA Multi-Attribute Task Battery II (MATB-II) under 4 different task load levels. We designed a convolutional neural network (CNN) to help with two distinct classification tasks. In one setting, the CNN was used to classify EEG segments based on their task load level. In another setting, the same CNN architecture was trained again to detect the presence of individual MATB-II subtasks. Results show that, while the model successfully learns to detect whether a particular subtask is active in a given segment (i.e., to differentiate between different subtasks-related EEG patterns), it struggles to differentiate between the two highest levels of task load (i.e., to distinguish MWL-related EEG patterns). We speculate that the challenge comes from two factors: first, the experiment was designed in a way that these two highest levels differed only in the quantity of work within a given timeframe; and second, the participants' effective adaptation to increased task demands, as evidenced by low error rates. Consequently, this indicates that under such conditions in multitasking, EEG may not reflect distinct enough patterns to differentiate higher levels of task load.

Domain- and task-analytic workload (DTAW) method: a methodology for predicting mental workload during severe accidents in nuclear power plants

Excessive mental workload reduces operators' performance and threatens the safety of nuclear power plants (NPPs) in severe accident management (SAM). Given the lack of suitable mental workload measurement methods for SAM tasks, we proposed a Domain- and Task-Analytic Workload (DTAW) method to predict SAM workload. The DTAW method is developed in three stages: scenario construction based on work domain analysis, task analysis, and workload estimation with eight workload components scored through task-analytic and projective methods. To demonstrate its utility, we applied the method to construct two SAM scenarios and predict the mental workload demand of operators in these scenarios as compared to two design basis accident scenarios. With statistical analysis, the DTAW method can predict the overall subjective workload rated by NPP operators, be used to identify high-load tasks, cluster tasks with similar workload patterns, and provide direct implications for improving SAM strategies and supporting systems.Practitioner summary: To predict mental workload in severe accident management (SAM) scenarios in nuclear power plants, we proposed an analytic method and applied it to estimate mental workload in two SAM scenarios and two design basis accident (DBA) scenarios. We found that the workload pattern in SAM scenarios is different from that in DBA scenarios.

Predictions of task using neural modeling

Introduction

A well-designed brain-computer interface (BCI) can make accurate and reliable predictions of a user's state through the passive assessment of their brain activity; in turn, BCI can inform an adaptive system (such as artificial intelligence, or AI) to intelligently and optimally aid the user to maximize the human-machine team (HMT) performance. Various groupings of spectro-temporal neural features have shown to predict the same underlying cognitive state (e.g., workload) but vary in their accuracy to generalize across contexts, experimental manipulations, and beyond a single session. In our work we address an outstanding challenge in neuroergonomic research: we quantify if (how) identified neural features and a chosen modeling approach will generalize to various manipulations defined by the same underlying psychological construct, (multi)task cognitive workload.

Methods

To do this, we train and test 20 different support vector machine (SVM) models, each given a subset of neural features as recommended from previous research or matching the capabilities of commercial devices. We compute each model's accuracy to predict which (monitoring, communications, tracking) and how many (one, two, or three) task(s) were completed simultaneously. Additionally, we investigate machine learning model accuracy to predict task(s) within- vs. between-sessions, all at the individual-level.

Results

Our results indicate gamma activity across all recording locations consistently outperformed all other subsets from the full model. Our work demonstrates that modelers must consider multiple types of manipulations which may each influence a common underlying psychological construct.

Discussion

We offer a novel and practical modeling solution for system designers to predict task through brain activity and suggest next steps in expanding our framework to further contribute to research and development in the neuroergonomics community. Further, we quantified the cost in model accuracy should one choose to deploy our BCI approach using a mobile EEG-systems with fewer electrodes-a practical recommendation from our work.

Identification of human mental workload levels in a language comprehension task with imbalance neurophysiological data

BACKGROUND AND OBJECTIVE

Operator's capability for accurately comprehending verbal commands is critically important to maintain the performance of human-machine interaction. It can be evaluated by human mental workload measured with electroencephalography (EEG). However, the time duration of different workload conditions within a task session is unequal due to varied psychophysiological processes across individuals. It leads to data imbalance of the EEG for training workload classifiers.

METHODS

In this study, we propose an EEG feature oversampling technique, Gaussian-SMOTE based feature ensemble (GSMOTE-FE), for workload recognition with imbalanced classes. First, artificial EEG instances are drawn from a Gaussian distribution in the margin between the minority and majority workload classes. Tomek links are detected as clues to remove redundant feature vectors. Then, we embed a feature selection module based on the GINI importance while an ensemble classifier committee with bootstrap aggregating is used to further enhance classification performance.

RESULTS

We validate the GSMOTE-FE framework based on an experiment that simulates operators to understand the correct meaning of the instructions in the Chinese language. Participants' EEG signals and reaction time data were both recorded to validate the proposed workload classifier. Workload classification accuracy and Macro-F1 values are 0.6553 and 0.5862, respectively. Corresponding G-mean and AUC achieve at 0.5757 and 0.5958, respectively.

CONCLUSIONS

The performance of the GSMOTE-FE is demonstrated to be comparable with the advanced oversampling techniques. The workload classifier has the capability to indicate low and high levels of the task demand of the Chinese language understanding task.

An Evaluation of the EEG Alpha-to-Theta and Theta-to-Alpha Band Ratios as Indexes of Mental Workload

Many research works indicate that EEG bands, specifically the alpha and theta bands, have been potentially helpful cognitive load indicators. However, minimal research exists to validate this claim. This study aims to assess and analyze the impact of the alpha-to-theta and the theta-to-alpha band ratios on supporting the creation of models capable of discriminating self-reported perceptions of mental workload. A dataset of raw EEG data was utilized in which 48 subjects performed a resting activity and an induced task demanding exercise in the form of a multitasking SIMKAP test. Band ratios were devised from frontal and parietal electrode clusters. Building and model testing was done with high-level independent features from the frequency and temporal domains extracted from the computed ratios over time. Target features for model training were extracted from the subjective ratings collected after resting and task demand activities. Models were built by employing Logistic Regression, Support Vector Machines and Decision Trees and were evaluated with performance measures including accuracy, recall, precision and f1-score. The results indicate high classification accuracy of those models trained with the high-level features extracted from the alpha-to-theta ratios and theta-to-alpha ratios. Preliminary results also show that models trained with logistic regression and support vector machines can accurately classify self-reported perceptions of mental workload. This research contributes to the body of knowledge by demonstrating the richness of the information in the temporal, spectral and statistical domains extracted from the alpha-to-theta and theta-to-alpha EEG band ratios for the discrimination of self-reported perceptions of mental workload.

Development of an Eye Responses-Based Mental Workload Evaluation Method

This study proposed an eye responses-based mental workload (E-MWL) evaluation method in nuclear power plants (NPPs) when performing the task via a user interface control. The fuzzy theory was used to combine four eye response indices using the entropy weight method. Then, the E-MWL method was validated through experiments by comparison with the NASA-TLX rating and performance measures indices in two different tasks of the State Oriented Procedure (SOP) in NPP. The correlation analysis results between the NASA-TLX and eye response indices showed that four eye response indices used in this study were correlated significantly with the NASA-TLX, indicating that these indices may develop the E-MWL method. The E-MWL score results indicated that it is highly correlated with NASA-TLX and performance measures indices in two different tasks of SOP in NPP. This has proved that E-MWL is an objective method suitable for evaluating and predicting human mental workload (MWL) for interface control task in NPPs.

Biosignal-Based Attention Monitoring to Support Nuclear Operator Safety-Relevant Tasks

In the main control room (MCR) of a nuclear power plant (NPP), the quality of an operator's performance can depend on their level of attention to the task. Insufficient operator attention accounted for more than 26% of the total causes of human errors and is the highest category for errors. It is therefore necessary to check whether operators are sufficiently attentive either as supervisors or peers during reactor operation. Recently, digital control technologies have been introduced to the operating environment of an NPP MCR. These upgrades are expected to enhance plant and operator performance. At the same time, because personal computers are used in the advanced MCR, the operators perform more cognitive works than physical work. However, operators may not consciously check fellow operators' attention in this environment indicating potentially higher importance of the role of operator attention. Therefore, remote measurement of an operator's attention in real time would be a useful tool, providing feedback to supervisors. The objective of this study is to investigate the development of quantitative indicators that can identify an operator's attention, to diagnose or detect a lack of operator attention thus preventing potential human errors in advanced MCRs. To establish a robust baseline of operator attention, this study used two of the widely used biosignals: electroencephalography (EEG) and eye movement. We designed an experiment to collect EEG and eye movements of the subjects who were monitoring and diagnosing nuclear operator safety-relevant tasks. There was a statistically significant difference between biosignals with and without appropriate attention. Furthermore, an average classification accuracy of about 90% was obtained by the k-nearest neighbors and support vector machine classifiers with a few EEG and eye movements features. Potential applications of EEG and eye movement measures in monitoring and diagnosis tasks in an NPP MCR are also discussed.

Recognition of cognitive load with a stacking network ensemble of denoising autoencoders and abstracted neurophysiological features

The safety of human-machine systems can be indirectly evaluated based on operator's cognitive load levels at each temporal instant. However, relevant features of cognitive states are hidden behind in multiple sources of cortical neural responses. In this study, we developed a novel neural network ensemble, SE-SDAE, based on stacked denoising autoencoders (SDAEs) which identify different levels of cognitive load by electroencephalography (EEG) signals. To improve the generalization capability of the ensemble framework, a stacking-based approach is adopted to fuse the abstracted EEG features from activations of deep-structured hidden layers. In particular, we also combine multiple K-nearest neighbor and naive Bayesian classifiers with SDAEs to generate a heterogeneous classification committee to enhance ensemble's diversity. Finally, we validate the proposed SE-SDAE by comparing its performance with mainstream pattern classifiers for cognitive load evaluation to show its effectiveness.

Evaluating effectiveness of information visualizations using cognitive fit theory: A neuroergonomics approach

Information visualizations may be evaluated from the perspective of how they match tasks that must be performed with them, a cognitive fit perspective. However, there is a gap between the high-level references made to cognitive fit and the low-level ability to identify and measure it during human interaction with visualizations. We bridge this gap by using an electroencephalography metric derived from frontal midline theta power and parietal alpha power, known as the task load index, to determine if cognitive effort measured at the level of cortical activity is less when cognitive fit is present compared to when cognitive fit is not. We found that when there is cognitive fit between the type of problem to be solved and the information displayed by a system, the task load index is lower compared to when cognitive fit is not present. We support this finding with subjective (NASA task load index) and performance (response time and accuracy) measures. Our approach, using electroencephalography, provides supplemental information to self-report and performance measures. Findings from this study are important because they (1) provide more validity to the cognitive fit theory using a neurophysiological measure, and (2) use the electroencephalography task load index metric as a means to assess cognitive workload and effort in general.

A Systematic Review of Physiological Measures of Mental Workload

Mental workload (MWL) can affect human performance and is considered critical in the design and evaluation of complex human-machine systems. While numerous physiological measures are used to assess MWL, there appears no consensus on their validity as effective agents of MWL. This study was conducted to provide a comprehensive understanding of the use of physiological measures of MWL and to synthesize empirical evidence on the validity of the measures to discriminate changes in MWL. A systematical literature search was conducted with four electronic databases for empirical studies measuring MWL with physiological measures. Ninety-one studies were included for analysis. We identified 78 physiological measures, which were distributed in cardiovascular, eye movement, electroencephalogram (EEG), respiration, electromyogram (EMG) and skin categories. Cardiovascular, eye movement and EEG measures were the most widely used across varied research domains, with 76%, 66%, and 71% of times reported a significant association with MWL, respectively. While most physiological measures were found to be able to discriminate changes in MWL, they were not universally valid in all task scenarios. The use of physiological measures and their validity for MWL assessment also varied across different research domains. Our study offers insights into the understanding and selection of appropriate physiological measures for MWL assessment in varied human-machine systems.

Assessing cognitive mental workload via EEG signals and an ensemble deep learning classifier based on denoising autoencoders

To estimate the reliability and cognitive states of operator performance in a human-machine collaborative environment, we propose a novel human mental workload (MW) recognizer based on deep learning principles and utilizing the features of the electroencephalogram (EEG). To determine personalized properties in high dimensional EEG indicators, we introduce a feature mapping layer in stacked denoising autoencoder (SDAE) that is capable of preserving the local information in EEG dynamics. The ensemble classifier is then built via the subject-specific integrated deep learning committee, and adapts to the cognitive properties of a specific human operator and alleviates inter-subject feature variations. We validate our algorithms and the ensemble SDAE classifier with local information preservation (denoted by EL-SDAE) on an EEG database collected during the execution of complex human-machine tasks. The classification performance indicates that the EL-SDAE outperforms several classical MW estimators when its optimal network architecture has been identified.