Electrocardiographic features for the measurement of drivers' mental workload

Reliable but multi-dimensional cognitive demand in operating partially automated vehicles: implications for real-world automation research

The reliability of cognitive demand measures in controlled laboratory settings is well-documented; however, limited research has directly established their stability under real-life and high-stakes conditions, such as operating automated technology on actual highways. Partially automated vehicles have advanced to become an everyday mode of transportation, and research on driving these advanced vehicles requires reliable tools for evaluating the cognitive demand on motorists to sustain optimal engagement in the driving process. This study examined the reliability of five cognitive demand measures, while participants operated partially automated vehicles on real roads across four occasions. Seventy-one participants (aged 18-64 years) drove on actual highways while their heart rate, heart rate variability, electroencephalogram (EEG) alpha power, and behavioral performance on the Detection Response Task were measured simultaneously. Findings revealed that EEG alpha power had excellent test-retest reliability, heart rate and its variability were good, and Detection Response Task reaction time and hit-rate had moderate reliabilities. Thus, the current study addresses concerns regarding the reliability of these measures in assessing cognitive demand in real-world automation research, as acceptable test-retest reliabilities were found across all measures for drivers across occasions. Despite the high reliability of each measure, low intercorrelations among measures were observed, and internal consistency was better when cognitive demand was estimated as a multi-factorial construct. This suggests that they tap into different aspects of cognitive demand while operating automation in real life. The findings highlight that a combination of psychophysiological and behavioral methods can reliably capture multi-faceted cognitive demand in real-world automation research.

Evaluation of physiological responses to mental workload in n-back and arithmetic tasks

Working memory tasks, such as n-back and arithmetic tasks, are frequently used in studying mental workload. The present study investigated and compared the sensitivity of several physiological measures at three levels of difficulty of n-back and arithmetic tasks. The results showed significant differences in fixation duration and pupil diameter among three task difficulty levels for both n-back and arithmetic tasks. Pupil diameters increase with increasing mental workload, whereas fixation duration decreases. Blink duration and heart rate (HR) were significantly increased as task difficulty increased in the n-back task, while root mean square of successive differences (RMSSD) and standard deviation of R-R intervals (SDNN) were significantly decreased in the arithmetic task. On the other hand, blink rate and Galvanic Skin Response (GSR) were not sensitive enough to assess the differences in task difficulty for both tasks. All significant physiological measures yielded significant differences between low and high task difficulty except for SDNN.Practitioner summary: This study aimed to assess the sensitivity levels of several physiological measures of mental workload in n-back and arithmetic tasks. It showed that pupil diameter was the most sensitive in both tasks. This study also found that most physiological indices are sensitive to an extreme change in task difficulty levels.

Predicting Driver's mental workload using physiological signals: A functional data analysis approach

This study investigates the impact of advanced driver-assistance systems on drivers' mental workload. Using a combination of physiological signals including ECG, EMG, EDA, EEG (af4 and fc6 channels from the theta band), and eye diameter data, this study aims to predict and categorize drivers' mental workload into low, adequate, and high levels. Data were collected from five different driving situations with varying cognitive demands. A functional linear regression model was employed for prediction, and the accuracy rate was calculated. Among the 31 tested combinations of physiological variables, 9 combinations achieved the highest accuracy result of 90%. These results highlight the potential benefits of utilizing raw physiological signal data and employing functional data analysis methods to understand and assess driver mental workload. The findings of this study have implications for the design and improvement of driver-assistance systems to optimize safety and performance.

Speed behaviour and mental workload of small-spacing expressway interchanges based on field driving test

Many small-spacing interchanges (SSI) appear with the improvement of the expressway network. To investigate the speed and mental workload characteristics in the SSI and acquire the mechanism of the influence of speed on the drivers' workload, 37 participants were recruited to perform a field driving test. Each driver performed four driving conditions (i.e. ramp-mainline, mainline-ramp, mainline driving, and auxiliary lane driving). The speed and drivers' electrocardiogram (ECG) data were collected using SpeedBox speed acquisition equipment and PhysioLAB physiological instrument. The heart rate increase (HRI) index was used to analyse the drivers' mental workload regularity. The relationship model between speed and HRI was developed to examine the impact of speed on HRI. The results show that the speed variation in the SSI displayed two patterns: 'decrease - increase and continuous decrease.' The drivers' HRI variation presented four patterns: 'convex curve, continuously increasing, continuously decreasing and concave curve'. SSI's influenced area length is given based on the speed and HRI variation regularity. HRI is significantly higher when driving in the ramp-mainline condition in the SSI than when driving in other conditions, indicating that drivers are more nervous when merging with the mainline traffic. HRI increases significantly in the first 50% of the weaving area in four driving conditions, indicating that vehicle weaving greatly influences the drivers' mental workload. A positive correlation exists between vehicle speed and drivers' HRI without interference from other vehicles and road alignment.

Detecting and Predicting Pilot Mental Workload Using Heart Rate Variability: A Systematic Review

Measuring pilot mental workload (MWL) is crucial for enhancing aviation safety. However, MWL is a multi-dimensional construct that could be affected by multiple factors. Particularly, in the context of a more automated cockpit setting, the traditional methods of assessing pilot MWL may face challenges. Heart rate variability (HRV) has emerged as a potential tool for detecting pilot MWL during real-flight operations. This review aims to investigate the relationship between HRV and pilot MWL and to assess the performance of machine-learning-based MWL detection systems using HRV parameters. A total of 29 relevant papers were extracted from three databases for review based on rigorous eligibility criteria. We observed significant variability across the reviewed studies, including study designs and measurement methods, as well as machine-learning techniques. Inconsistent results were observed regarding the differences in HRV measures between pilots under varying levels of MWL. Furthermore, for studies that developed HRV-based MWL detection systems, we examined the diverse model settings and discovered that several advanced techniques could be used to address specific challenges. This review serves as a practical guide for researchers and practitioners who are interested in employing HRV indicators for evaluating MWL and wish to incorporate cutting-edge techniques into their MWL measurement approaches.

Assessment of Drivers' Mental Workload by Multimodal Measures during Auditory-Based Dual-Task Driving Scenarios

Assessing drivers' mental workload is crucial for reducing road accidents. This study examined drivers' mental workload in a simulated auditory-based dual-task driving scenario, with driving tasks as the main task, and auditory-based N-back tasks as the secondary task. A total of three levels of mental workload (i.e., low, medium, high) were manipulated by varying the difficulty levels of the secondary task (i.e., no presence of secondary task, 1-back, 2-back). Multimodal measures, including a set of subjective measures, physiological measures, and behavioral performance measures, were collected during the experiment. The results showed that an increase in task difficulty led to increased subjective ratings of mental workload and a decrease in task performance for the secondary N-back tasks. Significant differences were observed across the different levels of mental workload in multimodal physiological measures, such as delta waves in EEG signals, fixation distance in eye movement signals, time- and frequency-domain measures in ECG signals, and skin conductance in EDA signals. In addition, four driving performance measures related to vehicle velocity and the deviation of pedal input and vehicle position also showed sensitivity to the changes in drivers' mental workload. The findings from this study can contribute to a comprehensive understanding of effective measures for mental workload assessment in driving scenarios and to the development of smart driving systems for the accurate recognition of drivers' mental states.

Assessing the physiological effect of non-driving-related task performance and task modality in conditionally automated driving systems: A systematic review and meta-analysis

In conditionally automated driving, the driver is free to disengage from controlling the vehicle, but they are expected to resume driving in response to certain situations or events that the system is not equipped to respond to. As the level of vehicle automation increases, drivers often engage in non-driving-related tasks (NDRTs), defined as any secondary task unrelated to the primary task of driving. This engagement can have a detrimental effect on the driver's situation awareness and attentional resources. NDRTs with resource demands that overlap with the driving task, such as visual or manual tasks, may be particularly deleterious. Therefore, monitoring the driver's state is an important safety feature for conditionally automated vehicles, and physiological measures constitute a promising means of doing this. The present systematic review and meta-analysis synthesises findings from 32 studies concerning the effect of NDRTs on drivers' physiological responses, in addition to the effect of NDRTs with a visual or a manual modality. Evidence was found that NDRT engagement led to higher physiological arousal, indicated by increased heart rate, electrodermal activity and a decrease in heart rate variability. There was mixed evidence for an effect of both visual and manual NDRT modalities on all physiological measures. Understanding the relationship between task performance and arousal during automated driving is of critical importance to the development of driver monitoring systems and improving the safety of this technology.

A systematic review on the influence factors, measurement, and effect of driver workload

Driver workload (DWL) is an important factor that needs to be considered in the study of traffic safety. The research focus on DWL has undergone certain shifts with the rapid development of scientific and technological advancements in the field of transportation in recent years. This study aims to grasp the state of research on DWL by both bibliometric analysis and individual critical literature review. The knowledge structure and development trend are described using bibliometric analysis. The knowledge mapping method is applied to mine the available literature in depth. It is discovered that one of the current research focus on DWL has shifted towards investigating its application in the field of autonomous driving. Subjective questionnaires and experimental tests (including both simulation technology and field study) are the main approaches to analyze DWL. An individual critical literature review of the influencing factors, measurement, and performance of DWL is provided. Research findings have shown that DWL was highly impacted by both intrinsic (e.g., age, temperament, driving experience) and external factors (e.g., vehicles, roads, tasks, environments). Scholars are actively exploring the combined effects of various factors and the level of vehicle automation on DWL. In addition to assess DWL by using subjective measures or physiological parameter measures separately, studies have started to improve classification accuracy by combining multiple measurement methods. Safety thresholds of DWL are not sufficiently studied due to the various interference items corresponding to different scenarios, but it is expected to quantify the DWL and find the threshold by establishing assessment models considering these intrinsic and external multiple-factors simultaneously. Driver or vehicle performance indicators are controversial to measure DWL directly, but they were suitable to reflect the impact of DWL in different driving conditions.

Persistent effects of mobile phone conversation while driving after disconnect: Physiological evidence and driving performance

Cognitive workload has been known as a key factor in traffic accidents, which can be highly increased by talking on the phone while driving. A wide range of studies around the world investigated the effects of mobile phone conversations on driving performance and traffic accidents. But less noticed is the durability of cognitive effects of mobile phone conversations. This study aimed to determine the effects of different types of mobile phone conversations on physiological response and driving performance during and after the conversation. Heart rate, heart rate variability (physiological response), Standard deviation of lane position (SDLP), and the relative distance between two cars (driving performance) of 34 samples (male and female) in the driving simulator were recorded. In this study, three types of conversations (neutral, cognitive, and arousal) were used. Neutral conversation did not pursue specific purpose questions. Cognitive conversations were simple mathematical problem-solving questions and arousal conversations aimed at arousing participant emotions. Each conversation was used as a secondary task in a condition. The study had three conditions; in each condition the participant drove for 15 min. Each condition consisted of 5 min of driving (Background), 5 min of driving and conversation (dual tasks) and 5 min of driving after conversation to trace the effects of the conversation. Vehicle speed was 110 km/h in each of the three conditions using car-following scenario. The results showed that neutral conversations had no significant effects on physiological response. Though, arousal conversations had significant effects on physiological responsiveness and driving performance during conversations, where it was even more significant after disconnection. Therefore, the content of the conversation determines the amount of cognitive load imposed on the driver. Considering the persistence of cognitive effects caused by conversation, the risk of traffic accidents is still high even after disconnection.

A Systematic Review of In-Vehicle Physiological Indices and Sensor Technology for Driver Mental Workload Monitoring

The concept of vehicle automation ceases to seem futuristic with the current advancement of the automotive industry. With the introduction of conditional automated vehicles, drivers are no longer expected to focus only on driving activities but are still required to stay alert to resume control. However, fluctuations in driving demands are known to alter the driver's mental workload (MWL), which might affect the driver's vehicle take-over capabilities. Driver mental workload can be specified as the driver's capacity for information processing for task performance. This paper summarizes the literature that relates to analysing driver mental workload through various in-vehicle physiological sensors focusing on cardiovascular and respiratory measures. The review highlights the type of study, hardware, method of analysis, test variable, and results of studies that have used physiological indices for MWL analysis in the automotive context.

Cognitive Analyses for Interface Design Using Dual N-Back Tasks for Mental Workload (MWL) Evaluation

In the manufacturing environments of today, human-machine systems are constituted with complex and advanced technology, which demands workers' considerable mental workload. This work aims to design and evaluate a Graphical User Interface developed to induce mental workload based on Dual N-Back tasks for further analysis of human performance. This study's contribution lies in developing proper cognitive analyses of the graphical user interface, identifying human error when the Dual N-Back tasks are presented in an interface, and seeking better user-system interaction. Hierarchical task analysis and the Task Analysis Method for Error Identification were used for the cognitive analysis. Ten subjects participated voluntarily in the study, answering the NASA-TLX questionnaire at the end of the task. The NASA-TLX results determined the subjective participants' mental workload proving that the subjects were induced to different levels of mental workload (Low, Medium, and High) based on the ANOVA statistical results using the mean scores obtained and cognitive analysis identified redesign opportunities for graphical user interface improvement.

Classification of Drivers' Mental Workload Levels: Comparison of Machine Learning Methods Based on ECG and Infrared Thermal Signals

Mental workload (MW) represents the amount of brain resources required to perform concurrent tasks. The evaluation of MW is of paramount importance for Advanced Driver-Assistance Systems, given its correlation with traffic accidents risk. In the present research, two cognitive tests (Digit Span Test-DST and Ray Auditory Verbal Learning Test-RAVLT) were administered to participants while driving in a simulated environment. The tests were chosen to investigate the drivers' response to predefined levels of cognitive load to categorize the classes of MW. Infrared (IR) thermal imaging concurrently with heart rate variability (HRV) were used to obtain features related to the psychophysiology of the subjects, in order to feed machine learning (ML) classifiers. Six categories of models have been compared basing on unimodal IR/unimodal HRV/multimodal IR + HRV features. The best classifier performances were reached by the multimodal IR + HRV features-based classifiers (DST: accuracy = 73.1%, sensitivity = 0.71, specificity = 0.69; RAVLT: accuracy = 75.0%, average sensitivity = 0.75, average specificity = 0.87). The unimodal IR features based classifiers revealed high performances as well (DST: accuracy = 73.1%, sensitivity = 0.73, specificity = 0.73; RAVLT: accuracy = 71.1%, average sensitivity = 0.71, average specificity = 0.85). These results demonstrated the possibility to assess drivers' MW levels with high accuracy, also using a completely non-contact and non-invasive technique alone, representing a key advancement with respect to the state of the art in traffic accident prevention.

Multi-Level Classification of Driver Drowsiness by Simultaneous Analysis of ECG and Respiration Signals Using Deep Neural Networks

The high number of fatal crashes caused by driver drowsiness highlights the need for developing reliable drowsiness detection methods. An ideal driver drowsiness detection system should estimate multiple levels of drowsiness accurately without intervening in the driving task. This paper proposes a multi-level drowsiness detection system by a deep neural network-based classification system using a combination of electrocardiogram and respiration signals. The proposed method is based on a combination of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks for classifying drowsiness by concurrently using heart rate variability (HRV), power spectral density of HRV, and respiration rate signal as inputs. Two models, a CNN-based model and a hybrid CNN-LSTM-based model were used for multi-level classifications. The performance of the proposed method was evaluated on experimental data collected from 30 subjects in a simulated driving environment. The performance and the results of both models are presented and compared. The best performance for both three-level and five-level drowsiness classifications was achieved by the CNN-LSTM model. The results indicate that the three-level and five-level classifications of drowsiness can be achieved with 91 and 67% accuracy, respectively.

Machine-Learning Based Monitoring of Cognitive Workload in Rescue Missions with Drones

In search and rescue missions, drone operations are challenging and cognitively demanding. High levels of cognitive workload can affect rescuers' performance, leading to failure with catastrophic outcomes. To face this problem, we propose a machine learning algorithm for real-time cognitive workload monitoring to understand if a search and rescue operator has to be replaced or if more resources are required. Our multimodal cognitive workload monitoring model combines the information of 25 features extracted from physiological signals, such as respiration, electrocardiogram, photoplethysmogram, and skin temperature, acquired in a noninvasive way. To reduce both subject and day inter-variability of the signals, we explore different feature normalization techniques, and introduce a novel weighted-learning method based on support vector machines suitable for subject-specific optimizations. On an unseen test set acquired from 34 volunteers, our proposed subject-specific model is able to distinguish between low and high cognitive workloads with an average accuracy of 87.3% and 91.2% while controlling a drone simulator using both a traditional controller and a new-generation controller, respectively.

Quantitative Study on Road Traffic Environment Complexity under Car-Following Condition

With the development of the drive of electronic communication technology, the driving assistance system that perceives the external traffic environment has developed rapidly. However, when quantifying the complexity of the road traffic environment without fully considering the driving characteristics and subjective feelings, the false alarm rate of the driving warning system increases and affects the early warning effect. In order to more accurately quantify the complexity of the road traffic environment, we analyzed the impact of road traffic environment changes on drivers under the condition of car-following. Firstly, we selected the influencing factors of the traffic environment complexity, such as the driving operation indicators, the vehicle driving status indicators and the road environmental indicators. The weight calculation model of each influence factor is established based on the principal component analysis method. Secondly, the driver’s reaction time during car-following is used as the quantitative index of road traffic environment complexity. The quantitative model of road traffic environment complexity is constructed combined with the weight of road traffic environment complexity. Finally, the driving simulation experiment is designed to verify the complexity quantification model of the road traffic environment. The road traffic environment complexity value calculated in our study is better than the TTC, and the early-warning threshold is raised by 2–5%. The research conclusion can provide a basis for the design of the car alarm system.

Evaluating mental workload during multitasking in simulated flight

BACKGROUND

Pilots must process multiple streams of information simultaneously. Mental workload is one of the main issues in man-machine interactive mode when dealing with multiple tasks. This study aimed to combine functional near-infrared spectroscopy (fNIRS) and electrocardiogram (ECG) to detect changes in mental workload during multitasking in a simulated flight.

METHODS

Twenty-six participants performed three multitasking tasks at different mental workload levels. These mental workload levels were set by varying the number of subtasks. fNIRS and ECG signals were recorded during tasks. Participants filled in the national aeronautics and space administration task load index (NASA-TLX) scale after each task. The effects of mental workload on scores of NASA-TLX, performance of tasks, heart rate (HR), heart rate variability (HRV), and the prefrontal cortex (PFC) activation were analyzed.

RESULTS

Compared to multitasking in lower mental workload conditions, participants exhibited higher scores of NASA-TLX, HR, and PFC activation when multitasking in high mental workload conditions. Their performance was worse during the high mental workload multitasking condition, as evidenced by the higher average tracking distance, smaller number of response times, and longer response time of the meter. The standard deviation of the RR intervals (SDNN) was negatively correlated with subjective mental workload in the low task load condition and PFC activation was positively correlated with HR and subjective mental workload in the medium task load condition.

CONCLUSION

HR and PFC activation can be used to detect changes in mental workload during simulated flight multitasking tasks.

Unravelling the Physiological Correlates of Mental Workload Variations in Tracking and Collision Prediction Tasks

Modern work environments have extensive interactions with technology and greater cognitive complexity of the tasks, which results in human operators experiencing increased mental workload. Air traffic control operators routinely work in such complex environments, and we designed tracking and collision prediction tasks to emulate their elementary tasks. The physiological response to the workload variations in these tasks was elucidated to untangle the impact of workload variations experienced by operators. Electroencephalogram (EEG), eye activity, and heart rate variability (HRV) data were recorded from 24 participants performing tracking and collision prediction tasks with three levels of difficulty. Our findings indicate that variations in task load in both these tasks are sensitively reflected in EEG, eye activity and HRV data. Multiple regression results also show that operators' performance in both tasks can be predicted using the corresponding EEG, eye activity and HRV data. The results also demonstrate that the brain dynamics during each of these tasks can be estimated from the corresponding eye activity, HRV and performance data. Furthermore, the markedly distinct neurometrics of workload variations in the tracking and collision prediction tasks indicate that neurometrics can provide insights on the type of mental workload. These findings have applicability to the design of future mental workload adaptive systems that integrate neurometrics in deciding not just "when" but also "what" to adapt. Our study provides compelling evidence in the viability of developing intelligent closed-loop mental workload adaptive systems that ensure efficiency and safety in complex work environments.

Recognition of the Mental Workloads of Pilots in the Cockpit Using EEG Signals

The commercial flightdeck is a naturally multi-tasking work environment, one in which interruptions are frequent come in various forms, contributing in many cases to aviation incident reports. Automatic characterization of pilots’ workloads is essential to preventing these kind of incidents. In addition, minimizing the physiological sensor network as much as possible remains both a challenge and a requirement. Electroencephalogram (EEG) signals have shown high correlations with specific cognitive and mental states, such as workload. However, there is not enough evidence in the literature to validate how well models generalize in cases of new subjects performing tasks with workloads similar to the ones included during the model’s training. In this paper, we propose a convolutional neural network to classify EEG features across different mental workloads in a continuous performance task test that partly measures working memory and working memory capacity. Our model is valid at the general population level and it is able to transfer task learning to pilot mental workload recognition in a simulated operational environment.

Let Complexity Bring Clarity: A Multidimensional Assessment of Cognitive Load Using Physiological Measures

The effects of cognitive load on driver behavior and traffic safety are unclear and in need of further investigation. Reliable measures of cognitive load for use in research and, subsequently, in the development and implementation of driver monitoring systems are therefore sought. Physiological measures are of interest since they can provide continuous recordings of driver state. Currently, however, a few issues related to their use in this context are not usually taken into consideration, despite being well-known. First, cognitive load is a multidimensional construct consisting of many mental responses (cognitive load components) to added task demand. Yet, researchers treat it as unidimensional. Second, cognitive load does not occur in isolation; rather, it is part of a complex response to task demands in a specific operational setting. Third, physiological measures typically correlate with more than one mental state, limiting the inferences that can be made from them individually. We suggest that acknowledging these issues and studying multiple mental responses using multiple physiological measures and independent variables will lead to greatly improved measurability of cognitive load. To demonstrate the potential of this approach, we used data from a driving simulator study in which a number of physiological measures (heart rate, heart rate variability, breathing rate, skin conductance, pupil diameter, eye blink rate, eye blink duration, EEG alpha power, and EEG theta power) were analyzed. Participants performed a cognitively loading n-back task at two levels of difficulty while driving through three different traffic scenarios, each repeated four times. Cognitive load components and other coinciding mental responses were assessed by considering response patterns of multiple physiological measures in relation to multiple independent variables. With this approach, the construct validity of cognitive load is improved, which is important for interpreting results accurately. Also, the use of multiple measures and independent variables makes the measurements (when analyzed jointly) more diagnostic-that is, better able to distinguish between different cognitive load components. This in turn improves the overall external validity. With more detailed, diagnostic, and valid measures of cognitive load, the effects of cognitive load on traffic safety can be better understood, and hence possibly mitigated.

Neural Decoding of EEG Signals with Machine Learning: A Systematic Review

Electroencephalography (EEG) is a non-invasive technique used to record the brain's evoked and induced electrical activity from the scalp. Artificial intelligence, particularly machine learning (ML) and deep learning (DL) algorithms, are increasingly being applied to EEG data for pattern analysis, group membership classification, and brain-computer interface purposes. This study aimed to systematically review recent advances in ML and DL supervised models for decoding and classifying EEG signals. Moreover, this article provides a comprehensive review of the state-of-the-art techniques used for EEG signal preprocessing and feature extraction. To this end, several academic databases were searched to explore relevant studies from the year 2000 to the present. Our results showed that the application of ML and DL in both mental workload and motor imagery tasks has received substantial attention in recent years. A total of 75% of DL studies applied convolutional neural networks with various learning algorithms, and 36% of ML studies achieved competitive accuracy by using a support vector machine algorithm. Wavelet transform was found to be the most common feature extraction method used for all types of tasks. We further examined the specific feature extraction methods and end classifier recommendations discovered in this systematic review.

Evaluation of Mental Workload in Working Memory Tasks with Different Information Types Based on EEG

To explore the effectiveness of using Electro- encephalogram (EEG) spectral power and multiscale sample entropy for accessing mental workload in different tasks, working memory tasks with different information types (verbal, object and spatial) and various mental loads were designed based on the N-Back paradigm. Subjective scores, accuracy and response time were used to verify the rationality of the tasks. EEGs from 18 normal adults were acquired when tasks were being performed, an independent component analysis (ICA) based artifact removal method were applied to get clean data. Linear (relative power in Theta and Alpha band, etc.) and nonlinear (multiscale sample entropy) features of EEGs were then extracted. Indices that can effectively reflect mental workload levels were selected by using multivariate analysis of variance statistical approach. Results showed that with the increment of task load, power of frontal Theta, Theta/Alpha ratio and sample entropies at scale more than 10 in parietal regions increased significantly first and decreased slightly then, while the power of central-parietal Alpha decreased significantly first and increased slightly then. Considering the difference between task types, no difference in power of frontal Theta, central-parietal Alpha and sample entropies at scales more than 10 of parietal regions were found between verbal and object tasks, as well as between two spatial tasks. No difference of frontal Theta/Alpha ratio was found in all the four tasks. The results can provide evidence for the mental workload evaluation in tasks with different information types.

Using in-seat electrical potential sensors for non-contact monitoring of heart rate, heart rate variability, and heart rate recovery

Detecting transient changes in heart rate and heart rate variability during experimental simulated autonomous driving scenarios can indicate participant arousal and cognitive load, providing valuable insights into the relationship between human and vehicle autonomy. Successfully detecting such parameters unobtrusively may assist these experimental situations as well as naturalistic driver monitoring systems within an autonomous vehicle. However, non-contact sensors must collect reliable and accurate signals. This study aims to compare the in-seat, non-contact Plessey EPIC sensor to the gold standard, contact Biopac sensor. Thirty participants took part in five-minute simulated autonomous vehicle journeys in a city environment and a rural environment, and a five-minute resting condition. To ensure the seat sensor was sensitive to elevated heart rate values, heart rate was also collected following the energetic Harvard Step Test. Lin concordance coefficients and Bland-Altman analyses were employed to assess the level of agreement between the non-contact Plessey EPIC sensor and the contact Biopac sensor for heart rate and heart rate variability. Analyses revealed a high level of agreement (r_c > 0.96) between both sensors for one-minute averaged heart rate and five-minute averaged heart rate variability during simulated autonomous driving and rest, and one-minute averaged heart rate following the Harvard Step Test. In addition, the non-contact sensor was sensitive to significant differences during tasks. This proof of principle study demonstrates the feasibility of using the non-contact Plessey EPIC sensor to accurately detect heart rate and heart rate variability during simulated autonomous driving environments.

Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems

With increasingly higher levels of automation in aerospace decision support systems, it is imperative that the human operator maintains a high level of situational awareness in different operational conditions and a central role in the decision-making process. While current aerospace systems and interfaces are limited in their adaptability, a Cognitive Human Machine System (CHMS) aims to perform dynamic, real-time system adaptation by estimating the cognitive states of the human operator. Nevertheless, to reliably drive system adaptation of current and emerging aerospace systems, there is a need to accurately and repeatably estimate cognitive states, particularly for Mental Workload (MWL), in real-time. As part of this study, two sessions were performed during a Multi-Attribute Task Battery (MATB) scenario, including a session for offline calibration and validation and a session for online validation of eleven multimodal inference models of MWL. The multimodal inference model implemented included an Adaptive Neuro Fuzzy Inference System (ANFIS), which was used in different configurations to fuse data from an Electroencephalogram (EEG) model’s output, four eye activity features and a control input feature. The results from the online validation of the ANFIS models demonstrated that five of the ANFIS models (containing different feature combinations of eye activity and control input features) all demonstrated good results, while the best performing model (containing all four eye activity features and the control input feature) showed an average Mean Absolute Error (MAE) = 0.67 ± 0.18 and Correlation Coefficient (CC) = 0.71 ± 0.15. The remaining six ANFIS models included data from the EEG model’s output, which had an offset discrepancy. This resulted in an equivalent offset for the online multimodal fusion. Nonetheless, the efficacy of these ANFIS models could be seen with the pairwise correlation with the task level, where one model demonstrated a CC = 0.77 ± 0.06, which was the highest among all the ANFIS models tested. Hence, this study demonstrates the ability for online multimodal fusion from features extracted from EEG signals, eye activity and control inputs to produce an accurate and repeatable inference of MWL.

No Difference in Arousal or Cognitive Demands Between Manual and Partially Automated Driving: A Multi-Method On-Road Study

INTRODUCTION

Partial driving automation is not always reliable and requires that drivers maintain readiness to take over control and manually operate the vehicle. Little is known about differences in drivers' arousal and cognitive demands under partial automation and how it may make it difficult for drivers to transition from automated to manual modes. This research examined whether there are differences in drivers' arousal and cognitive demands during manual versus partial automation driving.

METHOD

We compared arousal (using heart rate) and cognitive demands (using the root mean square of successive differences in normal heartbeats; RMSSD, and Detection Response Task; DRT) while 39 younger (M = 28.82 years) and 32 late-middle-aged (M = 52.72 years) participants drove four partially automated vehicles (Cadillac, Nissan Rogue, Tesla, and Volvo) on interstate highways. If compared to manual driving, drivers' arousal and cognitive demands were different under partial automation, then corresponding differences in heart rate, RMSSD, and DRT would be expected. Alternatively, if drivers' arousal and cognitive demands were similar in manual and partially automated driving, no difference in the two driving modes would be expected.

RESULTS

Results suggest no significant differences in heart rate, RMSSD, or DRT reaction time performance between manual and partially automated modes of driving for either younger or late-middle-aged adults across the four test vehicles. A Bayes Factor analysis suggested that heart rate, RMSSD, and DRT data showed extreme evidence in favor of the null hypothesis.

CONCLUSION

This novel study conducted on real roads with a representative sample provides important evidence of no difference in arousal and cognitive demands. Younger and late-middle-aged motorists who are new to partial automation are able to maintain arousal and cognitive demands comparable to manual driving while using the partially automated technology. Drivers who are more experienced with partially automated technology may respond differently than those with limited prior experience.

Real-time prediction of short-timescale fluctuations in cognitive workload

Human operators often experience large fluctuations in cognitive workload over seconds timescales that can lead to sub-optimal performance, ranging from overload to neglect. Adaptive automation could potentially address this issue, but to do so it needs to be aware of real-time changes in operators' spare cognitive capacity, so it can provide help in times of peak demand and take advantage of troughs to elicit operator engagement. However, it is unclear whether rapid changes in task demands are reflected in similarly rapid fluctuations in spare capacity, and if so what aspects of responses to those demands are predictive of the current level of spare capacity. We used the ISO standard detection response task (DRT) to measure cognitive workload approximately every 4 s in a demanding task requiring monitoring and refueling of a fleet of simulated unmanned aerial vehicles (UAVs). We showed that the DRT provided a valid measure that can detect differences in workload due to changes in the number of UAVs. We used cross-validation to assess whether measures related to task performance immediately preceding the DRT could predict detection performance as a proxy for cognitive workload. Although the simple occurrence of task events had weak predictive ability, composite measures that tapped operators' situational awareness with respect to fuel levels were much more effective. We conclude that cognitive workload does vary rapidly as a function of recent task events, and that real-time predictive models of operators' cognitive workload provide a potential avenue for automation to adapt without an ongoing need for intrusive workload measurements.

Assessment of mental workload based on multi-physiological signals

BACKGROUND:

Mental workload is one of the contributing factors to human errors in road accidents or other potentially adverse incidents.

OBJECTIVE:

This research probes the effects of mental workload on the electroencephalographic (EEG) and electrocardiogram (ECG) of subjects in visual monitoring tasks, based on which a comprehensive evaluation model for mental workload is established effectively.

METHODS:

Three degrees of mental workload were obtained by monitoring tasks with different levels of difficulty. 20 healthy subjects were selected to take part in the research.

RESULTS:

The subjective scores showed a significant increase with the increase of task difficulty, meanwhile the reaction time (RT) increased and the accuracy decreased significantly, which proved the validity of three degrees of mental workload induced. For the EEG parameters, a significant decrease of θ energy was found in Frontal, Parietal and Occipital with the increase of level of mental workload, as well as a significant decrease of α energy in Frontal, Central and Occipital, meanwhile a significant increase of β energy occurred in Frontal and Occipital. There was a significant decrease of α/θ in Occipital, and significant increases of θ/β and (α+β)/θ in Frontal, Central and Occipital, meanwhile (α+θ)/β and WPE decreased significantly in Frontal and Occipital. Among the ECG parameters, it was shown that Mean RR, RMSSD, HF_norm and SampEn decreased significantly with the increase of task difficulty, while LF_norm and LF/HF showed significant increases. These EEG indictors in Occipital and ECG indictors were chosen and constituted a multidimensional original sample. Principal Component Analysis (PCA) was used to extract the principal elements and decreased the dimension of sample space in order to simplify the calculation, based on which an effective classification model with accuracy of 80% was achieved by support vector machine (SVM).

CONCLUSION:

This study demonstrates that the proposed algorithm can be applied to mental workload monitoring.

Investigating the relationship between eye movements and situation awareness in weather forecasting

Physiological indicators, including eye tracking measures, may provide insight into human decision making and cognition in many domains, including weather forecasting. Situation awareness (SA), a critical component of forecast decision making, is commonly conceptualized as the degree to which information is perceived, understood, and projected into a future context. Drawing upon recent applications of eye tracking in the study of forecaster decision making, we investigate the relationship among eye movement measures, automation, and SA assessed through a freeze probe assessment method. In addition, we explore the relationship between an automated forecasting decision aid use and information seeking behavior. In this study, a sample of professional weather forecasters completed a series of tasks, informed by a set of forecasting decision aids, and with variable access to an experimental automated tool, while an eye tracking system captured data related to eye movements and information usage. At the end of each forecasting task, participants responded to a set of questions related to the environmental situation in the framework of a survey-based assessment technique in order to assess their level of situation awareness. Regression analysis revealed a moderate relationship between the SA measure and eye tracking metrics, supporting the hypothesis that eye tracking may have utility in assessing SA. The results support the use of eye tracking in the assessment of specific and measurable attributes of the decision-making process in weather forecasting. The findings are discussed in light of potential benefits that eye tracking could bring to human performance assessment as well as decision-making research in the forecasting domain.

A Systematic Review of Physiological Measures of Mental Workload

Mental workload (MWL) can affect human performance and is considered critical in the design and evaluation of complex human-machine systems. While numerous physiological measures are used to assess MWL, there appears no consensus on their validity as effective agents of MWL. This study was conducted to provide a comprehensive understanding of the use of physiological measures of MWL and to synthesize empirical evidence on the validity of the measures to discriminate changes in MWL. A systematical literature search was conducted with four electronic databases for empirical studies measuring MWL with physiological measures. Ninety-one studies were included for analysis. We identified 78 physiological measures, which were distributed in cardiovascular, eye movement, electroencephalogram (EEG), respiration, electromyogram (EMG) and skin categories. Cardiovascular, eye movement and EEG measures were the most widely used across varied research domains, with 76%, 66%, and 71% of times reported a significant association with MWL, respectively. While most physiological measures were found to be able to discriminate changes in MWL, they were not universally valid in all task scenarios. The use of physiological measures and their validity for MWL assessment also varied across different research domains. Our study offers insights into the understanding and selection of appropriate physiological measures for MWL assessment in varied human-machine systems.

Assessing cognitive mental workload via EEG signals and an ensemble deep learning classifier based on denoising autoencoders

To estimate the reliability and cognitive states of operator performance in a human-machine collaborative environment, we propose a novel human mental workload (MW) recognizer based on deep learning principles and utilizing the features of the electroencephalogram (EEG). To determine personalized properties in high dimensional EEG indicators, we introduce a feature mapping layer in stacked denoising autoencoder (SDAE) that is capable of preserving the local information in EEG dynamics. The ensemble classifier is then built via the subject-specific integrated deep learning committee, and adapts to the cognitive properties of a specific human operator and alleviates inter-subject feature variations. We validate our algorithms and the ensemble SDAE classifier with local information preservation (denoted by EL-SDAE) on an EEG database collected during the execution of complex human-machine tasks. The classification performance indicates that the EL-SDAE outperforms several classical MW estimators when its optimal network architecture has been identified.

A Review of Psychophysiological Measures to Assess Cognitive States in Real-World Driving

As driving functions become increasingly automated, motorists run the risk of becoming cognitively removed from the driving process. Psychophysiological measures may provide added value not captured through behavioral or self-report measures alone. This paper provides a selective review of the psychophysiological measures that can be utilized to assess cognitive states in real-world driving environments. First, the importance of psychophysiological measures within the context of traffic safety is discussed. Next, the most commonly used physiology-based indices of cognitive states are considered as potential candidates relevant for driving research. These include: electroencephalography and event-related potentials, optical imaging, heart rate and heart rate variability, blood pressure, skin conductance, electromyography, thermal imaging, and pupillometry. For each of these measures, an overview is provided, followed by a discussion of the methods for measuring it in a driving context. Drawing from recent empirical driving and psychophysiology research, the relative strengths and limitations of each measure are discussed to highlight each measures' unique value. Challenges and recommendations for valid and reliable quantification from lab to (less predictable) real-world driving settings are considered. Finally, we discuss measures that may be better candidates for a near real-time assessment of motorists' cognitive states that can be utilized in applied settings outside the lab. This review synthesizes the literature on in-vehicle psychophysiological measures to advance the development of effective human-machine driving interfaces and driver support systems.

Respiration and Heart Rate Modulation Due to Competing Cognitive Tasks While Driving

Research works on operator monitoring underline the benefit of taking into consideration several signal modalities to improve accuracy for an objective mental state diagnosis. Heart rate (HR) is one of the most utilized systemic measures to assess cognitive workload (CW), whereas, respiration parameters are hardly utilized. This study aims at verifying the contribution of analyzing respiratory signals to extract features to evaluate driver’s activity and CW variations in driving. Eighteen subjects participated in the study. The participants carried out two different cognitive tasks requiring different CW demands, a single task as well as a competing cognitive task realized while driving in a simulator. Our results confirm that both HR and breathing rate (BR) increase in driving and are sensitive to CW. However, HR and BR are differently modulated by the CW variations in driving. Specifically, HR is affected by both driving activity and CW, whereas, BR is suitable to evidence a variation of CW only when driving is not required. On the other hand, spectral features characterizing respiratory signal could be also used similarly to HR variability indices to detect high CW episodes. These results hint the use of respiration as an alternative to HR to monitor the driver mental state in autonomic vehicles in order to predict the available cognitive resources if the user has to take over the vehicle.

The effects of physical vibration on heart rate variability as a measure of drowsiness

We investigated the effects of low frequency whole body vibration on heart rate variability (HRV), a measure of autonomic nervous system activation that differentiates between stress and drowsiness. Fifteen participants underwent two simulated driving tasks for 60 min each: one involved whole-body 4-7 Hz vibration delivered through the car seat, and one involved no vibration. The Karolinska Sleepiness Scale (KSS), a subjective measure of drowsiness, demonstrated a significant increase in drowsiness during the task. Within 15-30 min of exposure to vibration, autonomic (sympathetic) activity increased (p < .01) in response to the stress of maintaining alertness and performance when drowsy, and peaked at 60 min (p < .001). Changes in three other HRV domains [higher LF/HF ratios, lower RMSSD (ms) and pNN50 (%) values] were consistent with increased sympathetic activation. These findings have implications for the future development of equivalent drowsiness contours leading to improvements in road safety. Practitioner summary: The effects of physical vibration on driver drowsiness have not been well investigated. This laboratory-controlled study found characteristic changes in heart rate variability (HRV) domains that indicated progressively increasing neurological effort in maintaining alertness in response to low frequency vibration, which becomes significant within 30 min.

ABBREVIATIONS

ANS: autonomic nervous system; Ctrl: control; EEG: electroencephalography; HF: the power in high frequency range (0.15 Hz-0.4Hz) in the PSD relected parasympathetic activity only; HRV: heart rate variability; KSS: karolinska sleepiness scale; LF: the power in low frequency range (0.04 Hz-0.15Hz) in the PSD reflected both sympathetic and parasympathetic activity of the autonomic nervous system; LF/HF ratio: the ratio of LF to HF indicated the balance between sympathetic and parasympathetic activity; RMSSD: the root mean square of difference of adjacent RR interval; pNN50: the number of successive RR interval pairs that differed by more than 50 ms divided by the total number of RR intervals; RR interval: the differences between successive R-wave occurrence times; PSD: power spectral density; RTP: research training program; SD: standard deviation; SEM: standard error of the Mean; Vib: vibration.

Cardiovascular correlates of emotional state, cognitive workload and time-on-task effect during a realistic flight simulation

In aviation, emotion and cognitive workload can considerably increase the probability of human error. An accurate online physiological monitoring of pilot's mental state could prevent accidents. The heart rate (HR) and heart rate variability (HRV) of 21 private pilots were analysed during two realistic flight simulator scenarios. Emotion was manipulated by a social stressor and cognitive workload with the difficulty of a secondary task. Our results confirmed the sensitivity of the HR to cognitive demand and training effects, with increased HR when the task was more difficult and decreased HR with training (time-on-task). Training was also associated with an increased HRV, with increased values along the flight scenario time course. Finally, the social stressor seemed to provoke an emotional reaction that enhanced motivation and performance on the secondary task. However, this was not reflected by the cardiovascular activity.

Effect of examination stress on intraocular pressure in university students

Intraocular pressure (IOP) has been investigated as a possible objective index of mental stressors. Here, we assessed the effect of examination stress on IOP in 33 university students. A repeated-measures design was used with two experimental conditions (examination and control) and two points of measurements (pre- and post-sessions). Also, the cardiovascular response, subjective perceived stress, as well as calculated ocular perfusion pressure and blood-pulse pressure were determined. A Bayesian statistical analysis showed higher IOP values in the examination in comparison to the control condition (BF₀₁ < 0.001). A similar pattern was found for the cardiovascular indices (diastolic and systolic blood pressure, and heart rate), and these findings were corroborated by subjective reports (BF₀₁ < 0.001 in all cases). Our data incorporates evidence in relation to the utility of IOP as an objective marker of examination stress, and it may help in the assessment and management of stress in applied scenarios.

Driver's mental workload prediction model based on physiological indices

Developing an early warning model to predict the driver's mental workload (MWL) is critical and helpful, especially for new or less experienced drivers. The present study aims to investigate the correlation between new drivers' MWL and their work performance, regarding the number of errors. Additionally, the group method of data handling is used to establish the driver's MWL predictive model based on subjective rating (NASA task load index [NASA-TLX]) and six physiological indices. The results indicate that the NASA-TLX and the number of errors are positively correlated, and the predictive model shows the validity of the proposed model with an R² value of 0.745. The proposed model is expected to provide a reference value for the new drivers of their MWL by providing the physiological indices, and the driving lesson plans can be proposed to sustain an appropriate MWL as well as improve the driver's work performance.