Dynamic assessment of control room operator's cognitive workload using Electroencephalography (EEG)

Recognizing the situation awareness of forklift operators based on EEG techniques in a field experiment

Lack of situation awareness (SA) is the primary cause of human errors when operating forklifts, so determining the SA level of the forklift operator is crucial to the safety of forklift operations. An EEG recognition approach of forklift operator SA in actual settings was presented in order to address the issues with invasiveness, subjectivity, and intermittency of existing measuring methods. In this paper, we conducted a field experiment that mimicked a typical forklift operation scenario to verify the differences in EEG states of forklift operators with different SA levels and investigate the correlation of multi-band combination features of each brain region of forklift operators with SA. Based on the sensitive EEG combination indexes, Support Vector Mechanism was used to construct a forklift operator SA recognition model. The results revealed that there were differences between forklift operators with high and low SA in the θ, α, and β frequency bands in zones F, C, P, and O; combined EEG indicators θ/β, (α + θ)/(α + β), and θ/(α + β) in zones F, P, and C were significantly correlated with SA; the recognition accuracy of the model reached 88.64% in the case of combined EEG indicators of zones C & F & P as input. It could provide a reference for SA measurement, contributing to the improvement of SA.

A Neurophysiological Evaluation of Cognitive Load during Augmented Reality Interactions in Various Industrial Maintenance and Assembly Tasks

Augmented reality (AR) has been shown to improve productivity in industry, but its adverse effects (e.g., headaches, eye strain, nausea, and mental workload) on users warrant further investigation. The objective of this study is to investigate the effects of different instruction methods (i.e., HoloLens AR-based and paper-based instructions) and task complexity (low and high-demanding tasks) on cognitive workloads and performance. Twenty-eight healthy males with a mean age of 32.12 (SD 2.45) years were recruited in this study and were randomly divided into two groups. The first group performed the experiment using AR-based instruction, and the second group used paper-based instruction. Performance was measured using total task time (TTT). The cognitive workload was measured using the power of electroencephalograph (EEG) features and the NASA task load index (NASA TLX). The results showed that using AR instructions resulted in a reduction in maintenance times and an increase in mental workload compared to paper instructions, particularly for the more demanding tasks. With AR instruction, 0.45% and 14.94% less time was spent on low- and high-demand tasks, respectively, as compared to paper instructions. According to the EEG features, employing AR to guide employees during highly demanding maintenance tasks increased information processing, which could be linked with an increased germane cognitive load. Increased germane cognitive load means participants can better facilitate long-term knowledge and skill acquisition. These results suggested that AR is superior and recommended for highly demanding maintenance tasks since it speeds up maintenance times and increases the possibility that information is stored in long-term memory and encrypted for recalls.

Analysis of Physiological Response of Attention and Stress States in Teleoperation Performance of Social Tasks

Some studies addressed monitoring mental states by physiological responses analysis in robots' teleoperation in traditional applications such as inspection and exploration; however, no study analyzed the physiological response during teleoperated social tasks to the best of our knowledge. We analyzed the physiological response of attention and stress mental states by computing the correlation between multimodal biomarkers and performance, pleasure-arousal scale, and workload. Physiological data were recorded during simulated teleoperated social tasks to induce mental states, such as normal, attention, and stress. The results showed that task performance and workload subscales achieved moderate correlations with some multimodal biomarkers. The correlations depended on the induced state. The cognitive workload was related to brain biomarkers of attention in the frontal and frontal-central regions. These regions were close to the frontopolar region, which is commonly reported in attentional studies. Thus, some multimodal biomarkers of attention and stress mental states could monitor or predict metrics related to the performance in teleoperation of social tasks.

Experimental Analysis of Handcart Pushing and Pulling Safety in an Industrial Environment by Using IoT Force and EMG Sensors: Relationship with Operators' Psychological Status and Pain Syndromes

Non-ergonomic execution of repetitive physical tasks represents a major cause of work-related musculoskeletal disorders (WMSD). This study was focused on the pushing and pulling (P&P) of an industrial handcart (which is a generic physical task present across many industries), with the aim to investigate the dependence of P&P execution on the operators' psychological status and the presence of pain syndromes of the upper limbs and spine. The developed acquisition system integrated two three-axis force sensors (placed on the left and right arm) and six electromyography (EMG) electrodes (placed on the chest, back, and hand flexor muscles). The conducted experiment involved two groups of participants (with and without increased psychological scores and pain syndromes). Ten force parameters (for both left and right side), one EMG parameter (for three different muscles, both left and right side), and two time-domain parameters were extracted from the acquired signals. Data analysis showed intergroup differences in the examined parameters, especially in force integral values and EMG mean absolute values. To the best of our knowledge, this is the first study that evaluated the composite effects of pain syndromes, spine mobility, and psychological status of the participants on the execution of P&P tasks-concluding that they have a significant impact on the P&P task execution and potentially on the risk of WMSD. The future work will be directed towards the development of a personalized risk assessment system by considering more muscle groups, supplementary data derived from operators' poses (extracted with computer vision algorithms), and cognitive parameters (extracted with EEG sensors).

Identification of human mental workload levels in a language comprehension task with imbalance neurophysiological data

BACKGROUND AND OBJECTIVE

Operator's capability for accurately comprehending verbal commands is critically important to maintain the performance of human-machine interaction. It can be evaluated by human mental workload measured with electroencephalography (EEG). However, the time duration of different workload conditions within a task session is unequal due to varied psychophysiological processes across individuals. It leads to data imbalance of the EEG for training workload classifiers.

METHODS

In this study, we propose an EEG feature oversampling technique, Gaussian-SMOTE based feature ensemble (GSMOTE-FE), for workload recognition with imbalanced classes. First, artificial EEG instances are drawn from a Gaussian distribution in the margin between the minority and majority workload classes. Tomek links are detected as clues to remove redundant feature vectors. Then, we embed a feature selection module based on the GINI importance while an ensemble classifier committee with bootstrap aggregating is used to further enhance classification performance.

RESULTS

We validate the GSMOTE-FE framework based on an experiment that simulates operators to understand the correct meaning of the instructions in the Chinese language. Participants' EEG signals and reaction time data were both recorded to validate the proposed workload classifier. Workload classification accuracy and Macro-F1 values are 0.6553 and 0.5862, respectively. Corresponding G-mean and AUC achieve at 0.5757 and 0.5958, respectively.

CONCLUSIONS

The performance of the GSMOTE-FE is demonstrated to be comparable with the advanced oversampling techniques. The workload classifier has the capability to indicate low and high levels of the task demand of the Chinese language understanding task.

"Context-Dependent Cognitive Workload Monitoring using Pupillometry for Control Room Operators to Prevent Overload"

Occupational ApplicationModernization of process plants has replaced traditional machine control with sophisticated computerized control, with the work required of control room operators changing from predominantly physical to cognitive. As a result, excessive cognitive workload during an abnormal condition is not an uncommon situation. Almost 70% of accidents in process industries are the result of human error, which is largely modulated by high cognitive workload. A context-dependent methodology using pupillometry was adopted in this study for workload assessment. We found that workload has a significant effect on task performance and subsequent success in handling an abnormal condition. Incorporating contextual information along with pupillometry-based measures was useful to explore the dynamics of variations in workload in depth. In addition, the proposed methodology is a more reliable method for a dynamic environment in which the situation evolves with operator interactions with the process.Technical Abstract:Background: Operator workload monitoring has become an important aspect of system engineering due to ever increasing cognitive demands in the control room. Higher cognitive workload, beyond capability, is directly associated with human error in plant operations. Even a small increase in workload during peak demand may result in human error, which can endanger the safety of a process plant.Purpose: Subjective methods are of limited use due to the requirement for near real-time evaluation of cognitive workload. Applications such as process plant control room require the operator to reiterate and re-represent the situation based on the contextual cues from environment. Hence, our objective was to develop and test a methodology that incorporates contextual information in workload measurement using pupillometry.Method: Participants (n = 44) performed simulated process control tasks. Pupil size was recorded using an eye-tracking device. Recorded pupil size was analyzed based on a methodology that incorporates contextual information from the human-machine interface. Events that present new information about the current state of the process were categorized as critical events. Pupil size variation was adopted to provide an estimate of cognitive workload between consecutive critical events.Results: Cognitive workload was associated with process behavior and operator actions. An increase in cognitive workload for specific events was observed when participant actions deteriorated process conditions. We also observed that overall task performance was associated with the frequency of certain events and corresponding variations in pupillary behavior.Conclusion: Incorporating contextual information provides more detailed insights into the dynamics of variations in cognitive workload. The mental representation of a current situation may vary depending on expertise level. The proposed methodology assesses such dynamic mental representations in the form of frequency of critical events and corresponding changes in the cognitive workload that ultimately determines the likelihood of an operator successfully completing a task.