The effects of dynamic workload and experience on commercially available EEG cognitive state metrics in a high-fidelity air traffic control environment

Physiological measures of operators' mental state in supervisory process control tasks: a scoping review

Physiological measures are often used to assess the mental state of human operators in supervisory process control tasks. However, the diversity of research approaches creates a heterogeneous landscape of empirical evidence. To map existing evidence and provide guidance to researchers and practitioners, this paper systematically reviews 109 empirical studies that report relationships between peripheral nervous system measures and mental state dimensions (e.g. mental workload, mental fatigue, stress, and vigilance) of interest. Ocular and electrocardiac measures were the most prominent measures across application fields. Most studies sought to validate such measures for reliable assessments of cognitive task demands and time on task, with measures of pupil size receiving the most empirical support. In comparison, less research examined the utility of physiological measures in predicting human task performance. This approach is discussed as an opportunity to focus on operators' individual response to cognitive task demands and to advance the state of research.

A video processing and machine learning based method for evaluating safety-critical operator engagement in a motorway control room

In safety-critical automatic systems, safety can be compromised if operators lack engagement. Effective detection of undesirable engagement states can inform the design of interventions for enhancing engagement. However, the existing engagement measurement methods suffer from several limitations which damage their effectiveness in the work environment. A novel engagement evaluation methodology, which adopts Artificial Intelligence (AI) technologies, has been proposed. It was developed using motorway control room operators as subjects. Openpose and Open Source Computer Vision Library (OpenCV) were used to estimate the body postures of operators, then a Support Vector Machine (SVM) was utilised to build the engagement evaluation model based on discrete states of operator engagement. The average accuracy of the evaluation results reached 0.89 and the weighted average precision, recall, and F1-score were all above 0.84. This study emphasises the importance of specific data labelling when measuring typical engagement states, forming the basis for potential control room improvements.Practitioner summary: This study demonstrates an automatic, real-time, objective, and relatively unobtrusive method for measuring dynamic operator engagement states. Computer vision technologies were used to estimate body posture, then machine learning (ML) was utilised to build the engagement evaluation model. The overall evaluation shows the effectiveness of this framework.Abbreviations: AI: Artificial Intelligence; OpenCV: Open Source Computer Vision Library; SVM: Support Vector Machine; UWES: Utrecht Work Engagement Scale; ISA Engagement Scale: Intellectual, Social, Affective Engagement Scale; DSSQ: Dundee Stress State Questionnaire; SSSQ: Short Stress State Questionnaire; EEG: electroencephalography; ECG: Electrocardiography; VMOE: Video-based Measurement for Operator Engagement; CMU: Carnegie Mellon University; CNN: Convolutional Neural Network; 2D: two dimensional; ML: Machine learning.

Detection of Pilot's Mental Workload Using a Wireless EEG Headset in Airfield Traffic Pattern Tasks

Elevated mental workload (MWL) experienced by pilots can result in increased reaction times or incorrect actions, potentially compromising flight safety. This study aims to develop a functional system to assist administrators in identifying and detecting pilots' real-time MWL and evaluate its effectiveness using designed airfield traffic pattern tasks within a realistic flight simulator. The perceived MWL in various situations was assessed and labeled using NASA Task Load Index (NASA-TLX) scores. Physiological features were then extracted using a fast Fourier transformation with 2-s sliding time windows. Feature selection was conducted by comparing the results of the Kruskal-Wallis (K-W) test and Sequential Forward Floating Selection (SFFS). The results proved that the optimal input was all PSD features. Moreover, the study analyzed the effects of electroencephalography (EEG) features from distinct brain regions and PSD changes across different MWL levels to further assess the proposed system's performance. A 10-fold cross-validation was performed on six classifiers, and the optimal accuracy of 87.57% was attained using a multi-class K-Nearest Neighbor (KNN) classifier for classifying different MWL levels. The findings indicate that the wireless headset-based system is reliable and feasible. Consequently, numerous wireless EEG device-based systems can be developed for application in diverse real-driving scenarios. Additionally, the current system contributes to future research on actual flight conditions.

A Survey on Affective and Cognitive VR

In Virtual Reality (VR), users can be immersed in emotionally intense and cognitively engaging experiences. Yet, despite strong interest from scholars and a large amount of work associating VR and Affective and Cognitive States (ACS), there is a clear lack of structured and systematic form in which this research can be classified. We define "Affective and Cognitive VR" to relate to works which (1) induce ACS, (2) recognize ACS, or (3) exploit ACS by adapting virtual environments based on ACS measures. This survey clarifies the different models of ACS, presents the methods for measuring them with their respective advantages and drawbacks in VR, and showcases Affective and Cognitive VR studies done in an Immersive Virtual Environment (IVE) in a non-clinical context. Our article covers the main research lines in Affective and Cognitive VR. We provide a comprehensive list of references with the analysis of 63 research articles and summarize future works directions.

Scan-based eye tracking measures are predictive of workload transition performance

Given there is no unifying theory or design guidance for workload transitions, this work investigated how visual attention allocation patterns could inform both topics, by understanding if scan-based eye tracking metrics could predict workload transition performance trends in a context-relevant domain. The eye movements of sixty Naval flight students were tracked as workload transitioned at a slow, medium, and fast pace in an unmanned aerial vehicle testbed. Four scan-based metrics were significant predictors across the different growth curve models of response time and accuracy. Stationary gaze entropy (a measure of how dispersed visual attention transitions are across tasks) was predictive across all three transition rates. The other three predictive scan-based metrics captured different aspects of visual attention, including its spread, directness, and duration. The findings specify several missing details in both theory and design guidance, which is unprecedented, and serves as a basis of future workload transition research.

Identification of human mental workload levels in a language comprehension task with imbalance neurophysiological data

BACKGROUND AND OBJECTIVE

Operator's capability for accurately comprehending verbal commands is critically important to maintain the performance of human-machine interaction. It can be evaluated by human mental workload measured with electroencephalography (EEG). However, the time duration of different workload conditions within a task session is unequal due to varied psychophysiological processes across individuals. It leads to data imbalance of the EEG for training workload classifiers.

METHODS

In this study, we propose an EEG feature oversampling technique, Gaussian-SMOTE based feature ensemble (GSMOTE-FE), for workload recognition with imbalanced classes. First, artificial EEG instances are drawn from a Gaussian distribution in the margin between the minority and majority workload classes. Tomek links are detected as clues to remove redundant feature vectors. Then, we embed a feature selection module based on the GINI importance while an ensemble classifier committee with bootstrap aggregating is used to further enhance classification performance.

RESULTS

We validate the GSMOTE-FE framework based on an experiment that simulates operators to understand the correct meaning of the instructions in the Chinese language. Participants' EEG signals and reaction time data were both recorded to validate the proposed workload classifier. Workload classification accuracy and Macro-F1 values are 0.6553 and 0.5862, respectively. Corresponding G-mean and AUC achieve at 0.5757 and 0.5958, respectively.

CONCLUSIONS

The performance of the GSMOTE-FE is demonstrated to be comparable with the advanced oversampling techniques. The workload classifier has the capability to indicate low and high levels of the task demand of the Chinese language understanding task.

Cognitive performance and electroencephalographic variations in air traffic controllers under various mental workload and time of day

The aim of this study was to investigate the effects of mental workload (MWL) and time of day on cognitive performance and electroencephalographic (EEG) parameters of air traffic controllers. EEG signals recorded while 20 professional air traffic controllers performed cognitive tasks [A-X Continuous Performance Test (AX-CPT) and 3-back working memory task] after they were exposed to two levels of task difficulty (high and low MWL) in the morning and afternoon. Significant decreases in cognitive performance were found when the levels of task difficulty increased in both tasks. The results confirmed the sensitivity of the theta and beta activities to levels of task difficulty in the 3-back task, while they were not affected in the AX-CPT. Theta and beta activities were influenced by time of day in the AX-CPT. The findings provide guidance for application of changes in EEG parameters when MWL level is manipulated during the day that could be implemented in future for the development of real-time monitoring systems to improve aviation safety.

Task difficulty and physiological measures of mental workload in air traffic control: a scoping review

This study provides a systematic synthesis of empirical research on mental workload (MWL) in air traffic control (ATC). MWL is a key concept in research on innovative technologies, because the assessment of MWL is crucial to the evaluation of such technologies. Our specific focus was on physiological measures of MWL. The used search strategy identified 39 peer-reviewed publications that analysed ATC tasks, examined different levels of difficulty of the ATC task, and considered at least one physiological measure of MWL. Positive relations between measures of MWL and task difficulty were observed most frequently, indicating that the measures indeed allowed the assessment of MWL. The most commonly used physiological measures were brain measures (EEG and fNIR) and heart rate measures. The review revealed a need for more precise descriptions of crucial experimental parameters in order to permit a transition of the field towards more interactive and dynamic types of analysis. Practitioner summary: Research on innovative technology in air traffic control (ATC) depends on assessments of mental workload (MWL). We reviewed empirical research on MWL in ATC. Brain and heart measures often allow assessments of MWL. Better descriptions of experiments are needed to allow comparisons among studies and more dynamic and interactive analyses.

Using IoT devices for sensor-based monitoring of employees' mental workload: Investigating managers' expectations and concerns

Although the objective assessment of mental workload has been a focus of human factors research, few studies have investigated stakeholders' attitudes towards its implementation in real workplaces. The present study addresses this research gap by surveying N = 702 managers in three European countries (Germany, United Kingdom, Spain) about their expectations and concerns regarding sensor-based monitoring of employee mental workload. The data confirm the relevance of expectations regarding improvements of workplace design and employee well-being, as well as concerns about restrictions of employees' privacy and sovereignty, for the implementation of workload monitoring. Furthermore, Bayesian regression models show that the examined expectations have a substantial positive association with managers' willingness to support workload monitoring in their company. Privacy concerns are identified as a significant barrier to the acceptance of workload monitoring, both in terms of their prevalence among managers and their strong negative relationship with monitoring support.

Performance degrades less under increased workload with the addition of speech control in a dynamic environment

This research empirically evaluates the introduction of speech to existing keyboard and mouse input modalities in an application used to control aircraft in a simulated, complex and dynamic environment. Task performance and task performance degradation are assessed for three levels of workload. Previous studies have evaluated task performance using these modalities however, only a couple have evaluated task performance under varying workload. Even though speech is a common addition to modern control interfaces, the effect of varying workload on this combination of control modalities has not yet been reported. Thirty-six participants commanded simulated aircraft through generated obstacle courses to reach a Combat Air Patrol (CAP) point while also responding to a secondary task. There were nine conditions that varied the control modality (Keyboard and Mouse (KM), Voice (V), and Keyboard, Mouse and Voice (KMV)), and workload by varying the number of aircraft being controlled (low, medium and high). Results showed that KM outperformed KMV and V for the low and medium workload levels. However, task performance with KMV was found to degrade the least as workload increased. KMV and KM were found to enable significantly more correct responses to the secondary task which was delivered aurally. Participants reported a preference for the combined modalities (KMV), self-assessing that KMV most reduced their workload. This research suggests that the addition of a speech interface to existing keyboard and mouse modalities, for control of aircraft in a simulation, may help manage cognitive load and may assist in controlling more aircraft under higher workloads.

Workplace design-related stress effects on prefrontal cortex connectivity and neurovascular coupling

This study aims to evaluate the effect of workstation type on the neural and vascular networks of the prefrontal cortex (PFC) underlying the cognitive activity involved during mental stress. Workstation design has been reported to affect the physical and mental health of employees. However, while the functional effects of ergonomic workstations have been documented, there is little research on the influence of workstation design on the executive function of the brain. In this study, 23 healthy volunteers in ergonomic and non-ergonomic workstations completed the Montreal imaging stress task, while their brain activity was recorded using the synchronized measurement of electroencephalography and functional near-infrared spectroscopy. The results revealed desynchronization in alpha rhythms and oxygenated hemoglobin, as well as decreased functional connectivity in the PFC networks at the non-ergonomic workstations. Additionally, a significant increase in salivary alpha-amylase activity was observed in all participants at the non-ergonomic workstations, confirming the presence of induced stress. These findings suggest that workstation design can significantly impact cognitive functioning and human capabilities at work. Therefore, the use of functional neuroimaging in workplace design can provide critical information on the causes of workplace-related stress.

A Sound Prediction: EEG-Based Neural Synchrony Predicts Online Music Streams

Neuroforecasting predicts population-wide choices based on neural data of individuals and can be used, for example, in neuromarketing to estimate campaign successes. To deliver true value, the brain activity metrics should deliver predictive value above and beyond traditional stated preferences. Evidence from movie trailer research has proposed neural synchrony, which compares the similarity of brain responses across participants and has shown to be a promising tool in neuroforecasting for movie popularity. The music industry might also benefit from these increasingly accurate success predictors, but only one study has been forecasting music popularity, using functional magnetic resonance imaging measures. Current research validates the strength of neural synchrony as a predictive measure for popularity of music, making use of electroencephalogram to capture moment-to-moment neural similarity between respondents while they listen to music. Neural synchrony is demonstrated to be a significant predictor for public appreciation on Spotify 3 weeks and 10 months after the release of the albums, especially when combined with the release of a single. On an individual level, other brain measures were shown to relate to individual subjective likeability ratings, including Frontal Alpha Asymmetry and engagement when combined with the factors artist and single release. Our results show the predictive value of brain activity measures outperforms stated preferences. Especially, neural synchrony carries high predictive value for the popularity on Spotify, providing the music industry with an essential asset for efficient decision making and investments, in addition to other practical implications that include neuromarketing and advertising industries.

The Index of Cognitive Activity - Eligibility for task-evoked informational strain and robustness towards visual influences

Various researchers have proposed pupillometric indicators to assess a person's cognitive strain. However, to distinguish the variation of pupil light response from psychosensory pupil response in experimental field conditions is a challenge. The Index of Cognitive Activity (ICA) addresses this problem by wavelet separation. This research investigates the ICA's sensitivity for multiple level task-evoked cognitive activity and visual influences concerning informational work tasks. Objective and subjective measures assessed cognitive strain of participants (N = 22) during various tasks. In a first experiment, mental arithmetic tasks were used to induce different levels of cognitive activity. In a second experiment, influences of screen polarity and presentation of information were investigated (N = 18). The results indicate that eye metrics are rarely sensitive to slight variations in task difficulty. Moreover, the ICA is likely to be sensitive towards constant screen illumination and shows tendencies regarding changes in displayed information. Possible ramifications for the objective assessment of cognitive strain are discussed.

Electro-Encephalography and Electro-Oculography in Aeronautics: A Review Over the Last Decade (2010-2020)

Electro-encephalography (EEG) and electro-oculography (EOG) are methods of electrophysiological monitoring that have potentially fruitful applications in neuroscience, clinical exploration, the aeronautical industry, and other sectors. These methods are often the most straightforward way of evaluating brain oscillations and eye movements, as they use standard laboratory or mobile techniques. This review describes the potential of EEG and EOG systems and the application of these methods in aeronautics. For example, EEG and EOG signals can be used to design brain-computer interfaces (BCI) and to interpret brain activity, such as monitoring the mental state of a pilot in determining their workload. The main objectives of this review are to, (i) offer an in-depth review of literature on the basics of EEG and EOG and their application in aeronautics; (ii) to explore the methodology and trends of research in combined EEG-EOG studies over the last decade; and (iii) to provide methodological guidelines for beginners and experts when applying these methods in environments outside the laboratory, with a particular focus on human factors and aeronautics. The study used databases from scientific, clinical, and neural engineering fields. The review first introduces the characteristics and the application of both EEG and EOG in aeronautics, undertaking a large review of relevant literature, from early to more recent studies. We then built a novel taxonomy model that includes 150 combined EEG-EOG papers published in peer-reviewed scientific journals and conferences from January 2010 to March 2020. Several data elements were reviewed for each study (e.g., pre-processing, extracted features and performance metrics), which were then examined to uncover trends in aeronautics and summarize interesting methods from this important body of literature. Finally, the review considers the advantages and limitations of these methods as well as future challenges.

Examining Positivity Effect and Working Memory in Young-Old and Very Old Adults Using EEG-Derived Cognitive State Metrics

The positivity effect among older adults is a tendency to process more positive and/or less negative emotional stimuli compared to younger adults, with unknown upper age boundaries. Cognitive and emotional working memory were assessed in young-old adults (60-75) and very old adults (VOAs; 80+) to determine whether emotional working memory declines similar to the age-related decline of cognitive working memory. The moderating role of valence on the link between age and emotional working memory was examined to identify change in positivity effect with advanced age. Electroencephalography (EEG) markers of cognitive workload and engagement were obtained to test the theory of cognitive resource allocation in older adults' emotional stimuli processing. EEG recordings were collected during cognitive memory task and emotional working memory tasks that required rating emotional intensity of images pairs. Results indicate a positivity effect among VOAs that does not require additional cognitive effort and is not likely to diminish with age.