Metrics for individual differences in EEG response to cognitive workload: Optimizing performance prediction

A numerical evaluation of real-time workloads for ramp controller through optimization of multi-type feature combinations derived from eye tracker, respiratory, and fatigue patterns

Ramp controllers are required to manage their workloads effectively while handling complex operational tasks, a crucial part of improving aviation safety. The ability to detect their instantaneous workload is vital for ensuring operational effectiveness and preventing hazardous incidents. This paper introduces a novel methodology aimed at enhancing the evaluation of the ramp controller's cumulative workload by incorporating and optimizing the feature combination from eye movement, respiratory, and fatigue characteristics. Specifically, a 90-minute simulated experiment related to ramp control tasks, using real data from Shanghai Hongqiao Airport, is conducted to collect multi-type data from 8 controllers. Following data construction and the extraction of multi-type, the workloads of all samples are categorized through unsupervised learning. Subsequently, supervised learning techniques are used to calculate feature weights and train classifiers after data alignment. The optimal feature combination is established by calculating feature weights, and the best classification accuracy is over 98%, achieved by the KNN classifier. Furthermore, numerical evaluation and threshold calculations for different workload levels are interpreted. It is promising to provide insights into future works towards human-centered data construction, processing, and interpretation to promote the progress of workload assessment within the aviation industry.

An analysis of physiological responses as indicators of driver takeover readiness in conditionally automated driving

By the year 2045, it is projected that Autonomous Vehicles (AVs) will make up half of the new vehicle market. Successful adoption of AVs can reduce drivers' stress and fatigue, curb traffic congestion, and improve safety, mobility, and economic efficiency. Due to the limited intelligence in relevant technologies, human-in-the-loop modalities are still necessary to ensure the safety of AVs at current or near future stages, because the vehicles may not be able to handle all emergencies. Therefore, it is important to know the takeover readiness of the drivers to ensure the takeover quality and avoid any potential accidents. To achieve this, a comprehensive understanding of the drivers' physiological states is crucial. However, there is a lack of systematic analysis of the correlation between different human physiological responses and takeover behaviors which could serve as important references for future studies to determine the types of data to use. This paper provides a comprehensive analysis of the effects of takeover behaviors on the common physiological indicators. A program for conditional automation was developed based on a game engine and applied to a driving simulator. The experiment incorporated three types of secondary tasks, three takeover events, and two traffic densities. Brain signals, Skin Conductance Level (SCL), and Heart Rate (HR) of the participants were collected while they were performing the driving simulations. The Frontal Asymmetry Index (FAI) (as an indicator of engagement) and Mental Workload (MWL) were calculated from the brain signals to indicate the mental states of the participants. The results revealed that the FAI of the drivers would slightly decrease after the takeover alerts were issued when they were doing secondary tasks prior to the takeover activities, and the higher difficulty of the secondary tasks could lead to lower overall FAI during the takeover periods. In contrast, The MWL and SCL increased during the takeover periods. The HR also increased rapidly at the beginning of the takeover period but dropped back to a normal level quickly. It was found that a fake takeover alert would lead to lower overall HR, slower increase, and lower peak of SCL during the takeover periods. Moreover, the higher traffic density scenarios were associated with higher MWL, and a more difficult secondary task would lead to higher MWL and HR during the takeover activities. A preliminary discussion of the correlation between the physiological data, takeover scenario, and vehicle data (that relevant to takeover readiness) was then conducted, revealing that although takeover event, SCL, and HR had slightly higher correlations with the maximum acceleration and reaction time, none of them dominated the takeover readiness. In addition, the analysis of the data across different participants was conducted, which emphasized the importance of considering standardization or normalization of the data when they were further used as input features for estimating takeover readiness. Overall, the results presented in this paper offer profound insights into the patterns of physiological data changes during takeover periods. These findings can be used as benchmarks for utilizing these variables as indicators of takeover preparedness and performance in future research endeavors.

EEG-based hierarchical classification of level of demand and modality of auditory and visual sensory processing

Objective.To date, most research on electroencephalography (EEG)-based mental workload detection for passive brain-computer interface (pBCI) applications has focused on identifying the overall level of cognitive resources required, such as whether the workload is high or low. We propose, however, that being able to determine the specific type of cognitive resources being used, such as visual or auditory, would also be useful. This would enable the pBCI to take more appropriate action to reduce the overall level of cognitive demand on the user. For example, if a high level of workload was detected and it is determined that the user is primarily engaged in visual information processing, then the pBCI could cause some information to be presented aurally instead. In our previous work we showed that EEG could be used to differentiate visual from auditory processing tasks when the level of processing is high, but the two modalities could not be distinguished when the level of cognitive processing demand was very low. The current study aims to build on this work and move toward the overall objective of developing a pBCI that is capable of predicting both the level and the type of cognitive resources being used.Approach.Fifteen individuals undertook carefully designed visual and auditory tasks while their EEG data was being recorded. In this study, we incorporated a more diverse range of sensory processing conditions including not only single-modality conditions (i.e. those requiring one of either visual or auditory processing) as in our previous study, but also dual-modality conditions (i.e. those requiring both visual and auditory processing) and no-task/baseline conditions (i.e. when the individual is not engaged in either visual or auditory processing).Main results.Using regularized linear discriminant analysis within a hierarchical classification algorithm, the overall cognitive demand was predicted with an accuracy of more than 86%, while the presence or absence of visual and auditory sensory processing were each predicted with an accuracy of approximately 70%.Significance.The findings support the feasibility of establishing a pBCI that can determine both the level and type of attentional resources required by the user at any given moment. This pBCI could assist in enhancing safety in hazardous jobs by triggering the most effective and efficient adaptation strategies when high workload conditions are detected.

Nonlinear dynamics of EEG responses to unmanned vehicle visual detection with different levels of task difficulty

The main objective of this study was to examine the presence of chaos in the EEG recordings of brain activity under simulated unmanned ground vehicle visual detection scenarios with different levels of task difficulty. One hundred and fifty people participated in the experiment and completed four visual detection task scenarios: (1) change detection, (2) a threat detection task, (3) a dual-task with different change detection task rates, and (4) a dual-task with different threat detection task rates. We used the largest Lyapunov exponent and correlation dimension of the EEG data and performed 0-1 tests on the EEG data. The results revealed a change in the level of nonlinearity in the EEG data corresponding to different levels of cognitive task difficulty. The differences in EEG nonlinearity measures among the studied levels of task difficulty, as well as between a single task scenario and a dual-task scenario, have also been assessed. The results increase our understanding of the nature of unmanned systems' operational requirements.

Cross-subject aesthetic preference recognition of Chinese dance posture using EEG

Due to the differences in knowledge, experience, background, and social influence, people have subjective characteristics in the process of dance aesthetic cognition. To explore the neural mechanism of the human brain in the process of dance aesthetic preference, and to find a more objective determining criterion for dance aesthetic preference, this paper constructs a cross-subject aesthetic preference recognition model of Chinese dance posture. Specifically, Dai nationality dance (a classic Chinese folk dance) was used to design dance posture materials, and an experimental paradigm for aesthetic preference of Chinese dance posture was built. Then, 91 subjects were recruited for the experiment, and their EEG signals were collected. Finally, the transfer learning method and convolutional neural networks were used to identify the aesthetic preference of the EEG signals. Experimental results have shown the feasibility of the proposed model, and the objective aesthetic measurement in dance appreciation has been implemented. Based on the classification model, the accuracy of aesthetic preference recognition is 79.74%. Moreover, the recognition accuracies of different brain regions, different hemispheres, and different model parameters were also verified by the ablation study. Additionally, the experimental results reflected the following two facts: (1) in the visual aesthetic processing of Chinese dance posture, the occipital and frontal lobes are more activated and participate in dance aesthetic preference; (2) the right brain is more involved in the visual aesthetic processing of Chinese dance posture, which is consistent with the common knowledge that the right brain is responsible for processing artistic activities.

Fatigue and Secondary Media Impacts in the Automated Vehicle: A Multidimensional State Perspective

Safety researchers increasingly recognize the impacts of task-induced fatigue on vehicle driving behavior. The current study (N = 180) explored the use of a multidimensional fatigue measure, the Driver Fatigue Questionnaire (DFQ), to test the impacts of vehicle automation, secondary media use, and driver personality on fatigue states and performance in a driving simulator. Secondary media included a trivia game and a cellphone conversation. Simulated driving induced large-magnitude fatigue states in participants, including tiredness, confusion, coping through self-comforting, and muscular symptoms. Consistent with previous laboratory and field studies, dispositional fatigue proneness predicted increases in state fatigue during the drive, especially tiredness, irrespective of automation level and secondary media. Similar to previous studies, automation slowed braking response to the emergency event following takeover but did not affect fatigue. Secondary media use relieved subjective fatigue and improved lateral control but did not affect emergency braking. Confusion was, surprisingly, associated with faster braking, and tiredness was associated with impaired control of lateral position of the vehicle. These associations were not moderated by the experimental factors. Overall, data support the use of multidimensional assessments of both fatigue symptoms and information-processing components for evaluating safety impacts of interventions for fatigue.

Bilingualism modulates neural efficiency at rest through alpha reactivity

The aim of the present study was to investigate how resting state EEG rhythms reflect attentional processes and bilingual experience. We compared alpha and beta rhythms for monolingual and bilingual young adults in eyes open and eyes closed conditions using EEG measures of frequency power, reactivity, and coherence. Power shows the amount of brain activity at a given frequency band; reactivity indexes the desynchronization of neuronal activity when individuals open their eyes at rest; and coherence indicates the brain regions that have correlated activity. The results showed that bilinguals had similar alpha power as monolinguals in both resting conditions but less alpha reactivity across the whole scalp. There was also more focused activation for bilinguals expressed as more coherence in posterior electrodes, particularly when eyes were opened to direct attention. For beta, there were no group differences in power or reactivity, but there was higher coherence for monolinguals than bilinguals, a pattern consistent with previous literature showing that beta frequency was related to language learning and native language proficiency. These results are in line with a neural efficiency theory and suggest that bilinguals have a more efficient brain for attentional mechanisms than monolinguals at rest.

Investigating Methods for Cognitive Workload Estimation for Assistive Robots

Robots interacting with humans in assistive contexts have to be sensitive to human cognitive states to be able to provide help when it is needed and not overburden the human when the human is busy. Yet, it is currently still unclear which sensing modality might allow robots to derive the best evidence of human workload. In this work, we analyzed and modeled data from a multi-modal simulated driving study specifically designed to evaluate different levels of cognitive workload induced by various secondary tasks such as dialogue interactions and braking events in addition to the primary driving task. Specifically, we performed statistical analyses of various physiological signals including eye gaze, electroencephalography, and arterial blood pressure from the healthy volunteers and utilized several machine learning methodologies including k-nearest neighbor, naive Bayes, random forest, support-vector machines, and neural network-based models to infer human cognitive workload levels. Our analyses provide evidence for eye gaze being the best physiological indicator of human cognitive workload, even when multiple signals are combined. Specifically, the highest accuracy (in %) of binary workload classification based on eye gaze signals is 80.45 ∓ 3.15 achieved by using support-vector machines, while the highest accuracy combining eye gaze and electroencephalography is only 77.08 ∓ 3.22 achieved by a neural network-based model. Our findings are important for future efforts of real-time workload estimation in the multimodal human-robot interactive systems given that eye gaze is easy to collect and process and less susceptible to noise artifacts compared to other physiological signal modalities.

The Effect of Time Window Length on EEG-Based Emotion Recognition

Various lengths of time window have been used in feature extraction for electroencephalogram (EEG) signal processing in previous studies. However, the effect of time window length on feature extraction for the downstream tasks such as emotion recognition has not been well examined. To this end, we investigate the effect of different time window (TW) lengths on human emotion recognition to find the optimal TW length for extracting electroencephalogram (EEG) emotion signals. Both power spectral density (PSD) features and differential entropy (DE) features are used to evaluate the effectiveness of different TW lengths based on the SJTU emotion EEG dataset (SEED). Different lengths of TW are then processed with an EEG feature-processing approach, namely experiment-level batch normalization (ELBN). The processed features are used to perform emotion recognition tasks in the six classifiers, the results of which are then compared with the results without ELBN. The recognition accuracies indicate that a 2-s TW length has the best performance on emotion recognition and is the most suitable to be used in EEG feature extraction for emotion recognition. The deployment of ELBN in the 2-s TW can further improve the emotion recognition performances by 21.63% and 5.04% when using an SVM based on PSD and DE features, respectively. These results provide a solid reference for the selection of TW length in analyzing EEG signals for applications in intelligent systems.

Dual Passive Reactive Brain-Computer Interface: A Novel Approach to Human-Machine Symbiosis

The present study proposes a novel concept of neuroadaptive technology, namely a dual passive-reactive Brain-Computer Interface (BCI), that enables bi-directional interaction between humans and machines. We have implemented such a system in a realistic flight simulator using the NextMind classification algorithms and framework to decode pilots' intention (reactive BCI) and to infer their level of attention (passive BCI). Twelve pilots used the reactive BCI to perform checklists along with an anti-collision radar monitoring task that was supervised by the passive BCI. The latter simulated an automatic avoidance maneuver when it detected that pilots missed an incoming collision. The reactive BCI reached 100% classification accuracy with a mean reaction time of 1.6 s when exclusively performing the checklist task. Accuracy was up to 98.5% with a mean reaction time of 2.5 s when pilots also had to fly the aircraft and monitor the anti-collision radar. The passive BCI achieved a F1-score of 0.94. This first demonstration shows the potential of a dual BCI to improve human-machine teaming which could be applied to a variety of applications.

Measuring the Domain Specificity of Workload Using EEG: Auditory and Visual Domains in Rotary-Wing Simulated Flight

OBJECTIVE

The overarching objective was to evaluate whether workload sensory-domain specificity could be identified through electroencephalogram (EEG) recordings during simulated rotary-wing operations.

BACKGROUND

Rotary-wing aviators experience workload from different sensory domains, although predominantly through auditory and visual domains. Development of real-time monitoring tools using psychophysiological indices, such as EEG recordings, could enable identification of aviator overload in real time.

METHOD

Two studies were completed, both of which recorded EEG, task performance, and self-report data. In Study 1, 16 individuals completed a basic auditory and a basic visual laboratory task where workload was manipulated. In Study 2, 23 Army aviators completed simulated aviation flights where workload was manipulated within auditory and visual sensory domains.

RESULTS

Results from Study 1 found differences in frontal alpha activity during the auditory task, and that alpha and beta activities were associated with perceived workload. Frontal theta activity was found to differ during the visual task while frontal alpha was associated with perceived workload. Study 2 found support for frontal beta activity and the ratio of beta to alpha + theta to differentiate level of workload within the auditory domain.

CONCLUSION

There is likely a role of frontal alpha and beta activities in response to workload manipulations within the auditory domain; however, this role becomes more equivocal when examined in a multifaceted flight scenario.

APPLICATION

Results from this study provide a basis for understanding changes in EEG activity when workload is manipulated in sensory domains that can be used in furthering the development of real-time monitoring tools.

Quantifying Physiological Biomarkers of a Microwave Brain Stimulation Device

Physiological signals are immediate and sensitive to neural and cardiovascular change resulting from brain stimulation, and are considered as a quantifying tool with which to evaluate the association between brain stimulation and cognitive performance. Brain stimulation outside a highly equipped, clinical setting requires the use of a low-cost, ambulatory miniature system. The purpose of this double-blind, randomized, sham-controlled study is to quantify the physiological biomarkers of the neural and cardiovascular systems induced by a microwave brain stimulation (MBS) device. We investigated the effect of an active MBS and a sham device on the cardiovascular and neurological responses of ten volunteers (mean age 26.33 years, 70% male). Electroencephalography (EEG) and electrocardiography (ECG) were recorded in the initial resting-state, intermediate state, and the final state at half-hour intervals using a portable sensing device. During the experiment, the participants were engaged in a cognitive workload. In the active MBS group, the power of high-alpha, high-beta, and low-beta bands in the EEG increased, and the power of low-alpha and theta waves decreased, relative to the sham group. RR Interval and QRS interval showed a significant association with MBS stimulation. Heart rate variability features showed no significant difference between the two groups. A wearable MBS modality may be feasible for use in biomedical research; the MBS can modulate the neurological and cardiovascular responses to cognitive workload.

A Systematic Review of Physiological Measures of Mental Workload

Mental workload (MWL) can affect human performance and is considered critical in the design and evaluation of complex human-machine systems. While numerous physiological measures are used to assess MWL, there appears no consensus on their validity as effective agents of MWL. This study was conducted to provide a comprehensive understanding of the use of physiological measures of MWL and to synthesize empirical evidence on the validity of the measures to discriminate changes in MWL. A systematical literature search was conducted with four electronic databases for empirical studies measuring MWL with physiological measures. Ninety-one studies were included for analysis. We identified 78 physiological measures, which were distributed in cardiovascular, eye movement, electroencephalogram (EEG), respiration, electromyogram (EMG) and skin categories. Cardiovascular, eye movement and EEG measures were the most widely used across varied research domains, with 76%, 66%, and 71% of times reported a significant association with MWL, respectively. While most physiological measures were found to be able to discriminate changes in MWL, they were not universally valid in all task scenarios. The use of physiological measures and their validity for MWL assessment also varied across different research domains. Our study offers insights into the understanding and selection of appropriate physiological measures for MWL assessment in varied human-machine systems.

Driving Style Recognition Based on Electroencephalography Data From a Simulated Driving Experiment

Driving style is a very important indicator and a crucial measurement of a driver's performance and ability to drive in a safe and protective manner. A dangerous driving style would possibly result in dangerous behaviors. If the driving styles can be recognized by some appropriate classification methods, much attention could be paid to the drivers with dangerous driving styles. The driving style recognition module can be integrated into the advanced driving assistance system (ADAS), which integrates different modules to improve driving automation, safety and comfort, and then the driving safety could be enhanced by pre-warning the drivers or adjusting the vehicle's controlling parameters when the dangerous driving style is detected. In most previous studies, driver's questionnaire data and vehicle's objective driving data were utilized to recognize driving styles. And promising results were obtained. However, these methods were indirect or subjective in driving style evaluation. In this paper a method based on objective driving data and electroencephalography (EEG) data was presented to classify driving styles. A simulated driving system was constructed and the EEG data and the objective driving data were collected synchronously during the simulated driving. The driving style of each participant was classified by clustering the driving data via K-means. Then the EEG data was denoised and the amplitude and the Power Spectral Density (PSD) of four frequency bands were extracted as the EEG features by Fast Fourier transform and Welch. Finally, the EEG features, combined with the classification results of the driving data were used to train a Support Vector Machine (SVM) model and a leave-one-subject-out cross validation was utilized to evaluate the performance. The SVM classification accuracy was about 80.0%. Conservative drivers showed higher PSDs in the parietal and occipital areas in the alpha and beta bands, aggressive drivers showed higher PSD in the temporal area in the delta and theta bands. These results imply that different driving styles were related with different driving strategies and mental states and suggest the feasibility of driving style recognition from EEG patterns.

Workload and Performance: Associations, Insensitivities, and Dissociations

OBJECTIVE

The aim of this study was to distill and define those influences under which change in objective performance level and the linked cognitive workload reflections of subjective experience and physiological variation either associate, dissociate, or are insensitive, one to another.

BACKGROUND

Human factors/ergonomics frequently employs users' self-reports of their own conscious experience, as well as their physiological reactivity, to augment the understanding of changing performance capacity. Under some circumstances, these latter workload responses are the only available assessment information to hand. How such perceptions and physiological responses match, fail to match, or are insensitive to the change in primary-task performance can prove critical to operational success. The reasons underlying these associations, dissociations, and insensitivities are central to the success of future effective human-machine interaction.

METHOD

Using extant research on the relations between differing methods of workload assessment, factors influencing their association, dissociation, and insensitivity are identified.

RESULTS

Dissociations and insensitivities occur more frequently than extant explanatory theories imply. Methodological and conceptual reasons for these patterns of incongruity are identified and evaluated.

APPLICATION

We often seek convergence of results in order to provide coherent explanations as bases for future prediction and practical design implementation. Identifying and understanding the causes as to why different reflections of workload diverge can help practitioners toward operational success.

Metacognition, Hardiness, and Grit as Resilience Factors in Unmanned Aerial Systems (UAS) Operations: A Simulation Study

Operators of Unmanned Aerial Systems (UAS) face a variety of stress factors resulting from both the cognitive demands of the work and its broader social context. Dysfunctional metacognitions including those concerning worry may increase stress vulnerability, whereas personality traits including hardiness and grit may confer resilience. The present study utilized a simulation of UAS operation requiring control of multiple vehicles. Two stressors were manipulated independently in a within-subjects design: cognitive demands and negative evaluative feedback. Stress response was assessed using both subjective measures and a suite of psychophysiological sensors, including the electroencephalogram (EEG), electrocardiogram (ECG), and hemodynamic sensors. Both stress manipulations elevated subjective distress and elicited greater high-frequency activity in the EEG. However, predictors of stress response varied across the two stressors. The Anxious Thoughts Inventory (AnTI: Wells, 1994) was generally associated with higher state worry in both control and stressor conditions. It also predicted stress reactivity indexed by EEG and worry responses in the negative feedback condition. Measures of hardiness and grit were associated with somewhat different patterns of stress response. In addition, within the negative feedback condition, the AnTI meta-worry scale moderated relationships between state worry and objective performance and psychophysiological outcome measures. Under high state worry, AnTI meta-worry was associated with lower frontal oxygen saturation, but higher spectral power in high-frequency EEG bands. High meta-worry may block adaptive compensatory effort otherwise associated with worry. Findings support both the metacognitive theory of anxiety and negative emotions (Wells and Matthews, 2015), and the Trait-Stressor-Outcome (TSO: Matthews et al., 2017a) framework for resilience.

Monitoring Pilot's Mental Workload Using ERPs and Spectral Power with a Six-Dry-Electrode EEG System in Real Flight Conditions

Recent technological progress has allowed the development of low-cost and highly portable brain sensors such as pre-amplified dry-electrodes to measure cognitive activity out of the laboratory. This technology opens promising perspectives to monitor the "brain at work" in complex real-life situations such as while operating aircraft. However, there is a need to benchmark these sensors in real operational conditions. We therefore designed a scenario in which twenty-two pilots equipped with a six-dry-electrode EEG system had to perform one low load and one high load traffic pattern along with a passive auditory oddball. In the low load condition, the participants were monitoring the flight handled by a flight instructor, whereas they were flying the aircraft in the high load condition. At the group level, statistical analyses disclosed higher P300 amplitude for the auditory target (Pz, P4 and Oz electrodes) along with higher alpha band power (Pz electrode), and higher theta band power (Oz electrode) in the low load condition as compared to the high load one. Single trial classification accuracy using both event-related potentials and event-related frequency features at the same time did not exceed chance level to discriminate the two load conditions. However, when considering only the frequency features computed over the continuous signal, classification accuracy reached around 70% on average. This study demonstrates the potential of dry-EEG to monitor cognition in a highly ecological and noisy environment, but also reveals that hardware improvement is still needed before it can be used for everyday flight operations.

Does social presence or the potential for interaction reduce social gaze in online social scenarios? Introducing the "live lab" paradigm

Research has shown that people's gaze is biased away from faces in the real world but towards them when they are viewed onscreen. Non-equivalent stimulus conditions may have represented a confound in this research, however, as participants viewed onscreen stimuli as pre-recordings where interaction was not possible compared with real-world stimuli which were viewed in real time where interaction was possible. We assessed the independent contributions of online social presence and ability for interaction on social gaze by developing the "live lab" paradigm. Participants in three groups ( N = 132) viewed a confederate as (1) a live webcam stream where interaction was not possible (one-way), (2) a live webcam stream where an interaction was possible (two-way), or (3) a pre-recording. Potential for interaction, rather than online social presence, was the primary influence on gaze behaviour: participants in the pre-recorded and one-way conditions looked more to the face than those in the two-way condition, particularly, when the confederate made "eye contact." Fixation durations to the face were shorter when the scene was viewed live, particularly, during a bid for eye contact. Our findings support the dual function of gaze but suggest that online social presence alone is not sufficient to activate social norms of civil inattention. Implications for the reinterpretation of previous research are discussed.