A Novel Method for Classifying Driver Mental Workload Under Naturalistic Conditions With Information From Near-Infrared Spectroscopy

EEG and fNIRS are associated with situation awareness (hazard) prediction during a driving task

Situation awareness (SA) is important in many demanding tasks (e.g. driving). Assessing SA during training can indicate whether someone is ready to perform in the real world. SA is typically assessed by interrupting the task to ask questions about the situation or asking questions after task completion, assessing only momentary SA. An objective and continuous means of detecting SA is needed. We examined whether neurophysiological sensors are useful to objectively measure Level 3 SA (projection of events into the future) during a driving task. We measured SA by the speed at which participants responded to SA questions and the accuracy of responses. For EEG, beta and theta power were most sensitive to SA response time. For fNIRS, oxygenated haemoglobin (HbO) was most sensitive to accuracy. This is the first evidence to our knowledge that neurophysiological measures are useful for assessing Level 3 SA during an ecologically valid task.

A systematic review on the influence factors, measurement, and effect of driver workload

Driver workload (DWL) is an important factor that needs to be considered in the study of traffic safety. The research focus on DWL has undergone certain shifts with the rapid development of scientific and technological advancements in the field of transportation in recent years. This study aims to grasp the state of research on DWL by both bibliometric analysis and individual critical literature review. The knowledge structure and development trend are described using bibliometric analysis. The knowledge mapping method is applied to mine the available literature in depth. It is discovered that one of the current research focus on DWL has shifted towards investigating its application in the field of autonomous driving. Subjective questionnaires and experimental tests (including both simulation technology and field study) are the main approaches to analyze DWL. An individual critical literature review of the influencing factors, measurement, and performance of DWL is provided. Research findings have shown that DWL was highly impacted by both intrinsic (e.g., age, temperament, driving experience) and external factors (e.g., vehicles, roads, tasks, environments). Scholars are actively exploring the combined effects of various factors and the level of vehicle automation on DWL. In addition to assess DWL by using subjective measures or physiological parameter measures separately, studies have started to improve classification accuracy by combining multiple measurement methods. Safety thresholds of DWL are not sufficiently studied due to the various interference items corresponding to different scenarios, but it is expected to quantify the DWL and find the threshold by establishing assessment models considering these intrinsic and external multiple-factors simultaneously. Driver or vehicle performance indicators are controversial to measure DWL directly, but they were suitable to reflect the impact of DWL in different driving conditions.

Benchmarking framework for machine learning classification from fNIRS data

Background

While efforts to establish best practices with functional near infrared spectroscopy (fNIRS) signal processing have been published, there are still no community standards for applying machine learning to fNIRS data. Moreover, the lack of open source benchmarks and standard expectations for reporting means that published works often claim high generalisation capabilities, but with poor practices or missing details in the paper. These issues make it hard to evaluate the performance of models when it comes to choosing them for brain-computer interfaces.

Methods

We present an open-source benchmarking framework, BenchNIRS, to establish a best practice machine learning methodology to evaluate models applied to fNIRS data, using five open access datasets for brain-computer interface (BCI) applications. The BenchNIRS framework, using a robust methodology with nested cross-validation, enables researchers to optimise models and evaluate them without bias. The framework also enables us to produce useful metrics and figures to detail the performance of new models for comparison. To demonstrate the utility of the framework, we present a benchmarking of six baseline models [linear discriminant analysis (LDA), support-vector machine (SVM), k-nearest neighbours (kNN), artificial neural network (ANN), convolutional neural network (CNN), and long short-term memory (LSTM)] on the five datasets and investigate the influence of different factors on the classification performance, including: number of training examples and size of the time window of each fNIRS sample used for classification. We also present results with a sliding window as opposed to simple classification of epochs, and with a personalised approach (within subject data classification) as opposed to a generalised approach (unseen subject data classification).

Results and discussion

Results show that the performance is typically lower than the scores often reported in literature, and without great differences between models, highlighting that predicting unseen data remains a difficult task. Our benchmarking framework provides future authors, who are achieving significant high classification scores, with a tool to demonstrate the advances in a comparable way. To complement our framework, we contribute a set of recommendations for methodology decisions and writing papers, when applying machine learning to fNIRS data.

Assessment of senior drivers' internal state in the event of simulated unexpected vehicle motion based on near-infrared spectroscopy

OBJECTIVE

A driver's internal state is a critical factor influencing driving performance, especially in cases of surprise or shock in response to unexpected incidents while driving. This study was designed to investigate the brain activity of a senior driver in response to simulated unexpected vehicle motion, compared with a relaxed state and normal driving.

METHODS

To accomplish this, we created a driving simulator paradigm wherein participants were involved in one of the following three scenarios: sitting down and relaxing, normal driving around the city with traffic signals and other vehicles, and the exiting of a parking area. In the scenario where the driver was to exit the parking area, the gear was switched automatically by the CarMaker software without the driver being notified, leading to an unexpected condition. The driver's internal states were classified by artificial intelligence, based on information obtained through four-channel near-infrared spectroscopy.

RESULTS

Significant differences were observed between the hemodynamic responses obtained in the three conditions. Ultimately, this method can be used to update advanced driver assistance systems, with a view to preventing future traffic accidents, by activating in-vehicle safety functions based on the driver's condition.

CONCLUSIONS

A driver's internal states in a panic situation while driving can be detected using near-infrared spectroscopy and artificial intelligence.

Optimized electroencephalogram and functional near-infrared spectroscopy-based mental workload detection method for practical applications

Background

Mental workload is a critical consideration in complex man–machine systems design. Among various mental workload detection techniques, multimodal detection techniques integrating electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals have attracted considerable attention. However, existing EEG–fNIRS-based mental workload detection methods have certain defects, such as complex signal acquisition channels and low detection accuracy, which restrict their practical application.

Methods

The signal acquisition configuration was optimized by analyzing the feature importance in mental workload recognition model and a more accurate and convenient EEG–fNIRS-based mental workload detection method was constructed. A classical Multi-Task Attribute Battery (MATB) task was conducted with 20 participating volunteers. Subjective scale data, 64-channel EEG data, and two-channel fNIRS data were collected.

Results

A higher number of EEG channels correspond to higher detection accuracy. However, there is no obvious improvement in accuracy once the number of EEG channels reaches 26, with a four-level mental workload detection accuracy of 76.25 ± 5.21%. Partial results of physiological analysis verify the results of previous studies, such as that the θ power of EEG and concentration of O₂Hb in the prefrontal region increase while the concentration of HHb decreases with task difficulty. It was further observed, for the first time, that the energy of each band of EEG signals was significantly different in the occipital lobe region, and the power of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta_{1}$$\end{document}β1 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta_{2}$$\end{document}β2 bands in the occipital region increased significantly with task difficulty. The changing range and the mean amplitude of O₂Hb in high-difficulty tasks were significantly higher compared with those in low-difficulty tasks.

Conclusions

The channel configuration of EEG–fNIRS-based mental workload detection was optimized to 26 EEG channels and two frontal fNIRS channels. A four-level mental workload detection accuracy of 76.25 ± 5.21% was obtained, which is higher than previously reported results. The proposed configuration can promote the application of mental workload detection technology in military, driving, and other complex human–computer interaction systems.

Relevant Physiological Indicators for Assessing Workload in Conditionally Automated Driving, Through Three-Class Classification and Regression

In future conditionally automated driving, drivers may be asked to take over control of the car while it is driving autonomously. Performing a non-driving-related task could degrade their takeover performance, which could be detected by continuous assessment of drivers' mental load. In this regard, three physiological signals from 80 subjects were collected during 1 h of conditionally automated driving in a simulator. Participants were asked to perform a non-driving cognitive task (N-back) for 90 s, 15 times during driving. The modality and difficulty of the task were experimentally manipulated. The experiment yielded a dataset of drivers' physiological indicators during the task sequences, which was used to predict drivers' workload. This was done by classifying task difficulty (three classes) and regressing participants' reported level of subjective workload after each task (on a 0–20 scale). Classification of task modality was also studied. For each task, the effect of sensor fusion and task performance were studied. The implemented pipeline consisted of a repeated cross validation approach with grid search applied to three machine learning algorithms. The results showed that three different levels of mental load could be classified with a f1-score of 0.713 using the skin conductance and respiration signals as inputs of a random forest classifier. The best regression model predicted the subjective level of workload with a mean absolute error of 3.195 using the three signals. The accuracy of the model increased with participants' task performance. However, classification of task modality (visual or auditory) was not successful. Some physiological indicators such as estimates of respiratory sinus arrhythmia, respiratory amplitude, and temporal indices of heart rate variability were found to be relevant measures of mental workload. Their use should be preferred for ongoing assessment of driver workload in automated driving.

Functional near-infrared spectroscopy as a potential objective evaluation technique in neurocognitive disorders after traumatic brain injury

Most patients with neurocognitive disorders after traumatic brain injury (TBI) show executive dysfunction, in which the pre-frontal cortex (PFC) plays an important role. However, less objective evaluation technique could be used to assess the executive dysfunction in these patients. Functional near-infrared spectroscopy (fNIRS), which is a non-invasive technique, has been widely used in the study of psychiatric disorders, cognitive dysfunction, etc. The present study aimed to explore whether fNIRS could be a technique to assess the damage degree of executive function in patients with neurocognitive disorders after TBI by using the Stroop and N-back tasks in PFC areas. We enrolled 37 patients with neurocognitive disorders after TBI and 60 healthy controls. A 22-channel fNIRS device was used to record HbO during Stroop, 1-back and 2-back tasks. The results showed that patients made significantly more errors and had longer response times than healthy controls. There were statistically significant differences in HbO level variation in bilateral frontopolar, bilateral inferior frontal gyrus and left middle temporal gyrus during Stroop color word consistency tasks and in left frontopolar during Stroop color word inconsistency tasks. During 2-back tasks, there were also statistically significant differences in HbO level variation in bilateral frontopolar, bilateral inferior frontal gyrus, bilateral dorsolateral pre-frontal cortex. According to brain activation maps, the patients exhibited lower but more widespread activation during the 2-back and Stroop color word consistency tasks. The fNIRS could identify executive dysfunction in patients with neurocognitive disorders after TBI by detecting HbO levels, which suggested that fNIRS could be a potential objective evaluation technique in neurocognitive disorders after TBI.

Predicting Driver Stress Levels with a Sensor-Equipped Steering Wheel and a Quality-Aware Heart Rate Measurement Algorithm

Unobtrusive monitoring of driver mental states has been regarded as an important element in improving the safety of existing transportation systems. While many solutions exist relying on camera-based systems for e.g., drowsiness detection, these can be sensitive to varying lighting conditions and to driver facial accessories, such as eye/sunglasses. In this work, we evaluate the use of physiological signals derived from sensors embedded directly into the steering wheel. In particular, we are interested in monitoring driver stress levels. To achieve this goal, we first propose a modulation spectral signal representation to reliably extract electrocardiogram (ECG) signals from the steering wheel sensors, thus allowing for heart rate and heart rate variability features to be computed. When input to a simple logistic regression classifier, we show that up to 72% accuracy can be achieved when discriminating between stressful and non-stressful driving conditions. In particular, the proposed modulation spectral signal representation allows for direct quality assessment of the obtained heart rate information, thus can provide additional intelligence to autonomous driver monitoring systems.

A Methodological Review of fNIRS in Driving Research: Relevance to the Future of Autonomous Vehicles

As automobile manufacturers have begun to design, engineer, and test autonomous driving systems of the future, brain imaging with functional near-infrared spectroscopy (fNIRS) can provide unique insights about cognitive processes associated with evolving levels of autonomy implemented in the automobile. Modern fNIRS devices provide a portable, relatively affordable, and robust form of functional neuroimaging that allows researchers to investigate brain function in real-world environments. The trend toward "naturalistic neuroscience" is evident in the growing number of studies that leverage the methodological flexibility of fNIRS, and in doing so, significantly expand the scope of cognitive function that is accessible to observation via functional brain imaging (i.e., from the simulator to on-road scenarios). While more than a decade's worth of study in this field of fNIRS driving research has led to many interesting findings, the number of studies applying fNIRS during autonomous modes of operation is limited. To support future research that directly addresses this lack in autonomous driving research with fNIRS, we argue that a cogent distillation of the methods used to date will help facilitate and streamline this research of tomorrow. To that end, here we provide a methodological review of the existing fNIRS driving research, with the overarching goal of highlighting the current diversity in methodological approaches. We argue that standardization of these approaches will facilitate greater overlap of methods by researchers from all disciplines, which will, in-turn, allow for meta-analysis of future results. We conclude by providing recommendations for advancing the use of such fNIRS technology in furthering understanding the adoption of safe autonomous vehicle technology.

Real-time prediction of short-timescale fluctuations in cognitive workload

Human operators often experience large fluctuations in cognitive workload over seconds timescales that can lead to sub-optimal performance, ranging from overload to neglect. Adaptive automation could potentially address this issue, but to do so it needs to be aware of real-time changes in operators' spare cognitive capacity, so it can provide help in times of peak demand and take advantage of troughs to elicit operator engagement. However, it is unclear whether rapid changes in task demands are reflected in similarly rapid fluctuations in spare capacity, and if so what aspects of responses to those demands are predictive of the current level of spare capacity. We used the ISO standard detection response task (DRT) to measure cognitive workload approximately every 4 s in a demanding task requiring monitoring and refueling of a fleet of simulated unmanned aerial vehicles (UAVs). We showed that the DRT provided a valid measure that can detect differences in workload due to changes in the number of UAVs. We used cross-validation to assess whether measures related to task performance immediately preceding the DRT could predict detection performance as a proxy for cognitive workload. Although the simple occurrence of task events had weak predictive ability, composite measures that tapped operators' situational awareness with respect to fuel levels were much more effective. We conclude that cognitive workload does vary rapidly as a function of recent task events, and that real-time predictive models of operators' cognitive workload provide a potential avenue for automation to adapt without an ongoing need for intrusive workload measurements.

Unsupervised fNIRS feature extraction with CAE and ESN autoencoder for driver cognitive load classification

Objective. Understanding the cognitive load of drivers is crucial for road safety. Brain sensing has the potential to provide an objective measure of driver cognitive load. We aim to develop an advanced machine learning framework for classifying driver cognitive load using functional near-infrared spectroscopy (fNIRS).Approach. We conducted a study using fNIRS in a driving simulator with theN-back task used as a secondary task to impart structured cognitive load on drivers. To classify different driver cognitive load levels, we examined the application of convolutional autoencoder (CAE) and Echo State Network (ESN) autoencoder for extracting features from fNIRS.Main results. By using CAE, the accuracies for classifying two and four levels of driver cognitive load with the 30 s window were 73.25% and 47.21%, respectively. The proposed ESN autoencoder achieved state-of-art classification results for group-level models without window selection, with accuracies of 80.61% and 52.45% for classifying two and four levels of driver cognitive load.Significance. This work builds a foundation for using fNIRS to measure driver cognitive load in real-world applications. Also, the results suggest that the proposed ESN autoencoder can effectively extract temporal information from fNIRS data and can be useful for other fNIRS data classification tasks.

Classification of Drivers' Workload Using Physiological Signals in Conditional Automation

The use of automation in cars is increasing. In future vehicles, drivers will no longer be in charge of the main driving task and may be allowed to perform a secondary task. However, they might be requested to regain control of the car if a hazardous situation occurs (i.e., conditionally automated driving). Performing a secondary task might increase drivers' mental workload and consequently decrease the takeover performance if the workload level exceeds a certain threshold. Knowledge about the driver's mental state might hence be useful for increasing safety in conditionally automated vehicles. Measuring drivers' workload continuously is essential to support the driver and hence limit the number of accidents in takeover situations. This goal can be achieved using machine learning techniques to evaluate and classify the drivers' workload in real-time. To evaluate the usefulness of physiological data as an indicator for workload in conditionally automated driving, three physiological signals from 90 subjects were collected during 25 min of automated driving in a fixed-base simulator. Half of the participants performed a verbal cognitive task to induce mental workload while the other half only had to monitor the environment of the car. Three classifiers, sensor fusion and levels of data segmentation were compared. Results show that the best model was able to successfully classify the condition of the driver with an accuracy of 95%. In some cases, the model benefited from sensors' fusion. Increasing the segmentation level (e.g., size of the time window to compute physiological indicators) increased the performance of the model for windows smaller than 4 min, but decreased for windows larger than 4 min. In conclusion, the study showed that a high level of drivers' mental workload can be accurately detected while driving in conditional automation based on 4-min recordings of respiration and skin conductance.

Sensor-based indicators of performance changes between sessions during robotic surgery training

Training of surgeons is essential for safe and effective use of robotic surgery, yet current assessment tools for learning progression are limited. The objective of this study was to measure changes in trainees' cognitive and behavioral states as they progressed in a robotic surgeon training curriculum at a medical institution. Seven surgical trainees in urology who had no formal robotic training experience participated in the simulation curriculum. They performed 12 robotic skills exercises with varying levels of difficulty repetitively in separate sessions. EEG (electroencephalogram) activity and eye movements were measured throughout to calculate three metrics: engagement index (indicator of task engagement), pupil diameter (indicator of mental workload) and gaze entropy (indicator of randomness in gaze pattern). Performance scores (completion of task goals) and mental workload ratings (NASA-Task Load Index) were collected after each exercise. Changes in performance scores between training sessions were calculated. Analysis of variance, repeated measures correlation, and machine learning classification were used to diagnose how cognitive and behavioral states associate with performance increases or decreases between sessions. The changes in performance were correlated with changes in engagement index (rrm=-.25,p<.001) and gaze entropy (rrm=-.37,p<.001). Changes in cognitive and behavioral states were able to predict training outcomes with 72.5% accuracy. Findings suggest that cognitive and behavioral metrics correlate with changes in performance between sessions. These measures can complement current feedback tools used by medical educators and learners for skills assessment in robotic surgery training.

A Three-Class Classification of Cognitive Workload Based on EEG Spectral Data

Evaluation of cognitive workload finds its application in many areas, from educational program assessment through professional driver health examination to monitoring the mental state of people carrying out jobs of high responsibility, such as pilots or airline traffic dispatchers. Estimation of multilevel cognitive workload is a task usually realized in a subject-dependent way, while the present research is focused on developing the procedure of subject-independent evaluation of cognitive workload level. The aim of the paper is to estimate cognitive workload level in accordance with subject-independent approach, applying classical machine learning methods combined with feature selection techniques. The procedure of data acquisition was based on registering the EEG signal of the person performing arithmetical tasks divided into six intervals of advancement. The analysis included the stages of preprocessing, feature extraction, and selection, while the final step covered multiclass classification performed with several models. The results discussed show high maximal accuracies achieved: ~91% for both the validation dataset and for the cross-validation approach for kNN model.

A Review of Psychophysiological Measures to Assess Cognitive States in Real-World Driving

As driving functions become increasingly automated, motorists run the risk of becoming cognitively removed from the driving process. Psychophysiological measures may provide added value not captured through behavioral or self-report measures alone. This paper provides a selective review of the psychophysiological measures that can be utilized to assess cognitive states in real-world driving environments. First, the importance of psychophysiological measures within the context of traffic safety is discussed. Next, the most commonly used physiology-based indices of cognitive states are considered as potential candidates relevant for driving research. These include: electroencephalography and event-related potentials, optical imaging, heart rate and heart rate variability, blood pressure, skin conductance, electromyography, thermal imaging, and pupillometry. For each of these measures, an overview is provided, followed by a discussion of the methods for measuring it in a driving context. Drawing from recent empirical driving and psychophysiology research, the relative strengths and limitations of each measure are discussed to highlight each measures' unique value. Challenges and recommendations for valid and reliable quantification from lab to (less predictable) real-world driving settings are considered. Finally, we discuss measures that may be better candidates for a near real-time assessment of motorists' cognitive states that can be utilized in applied settings outside the lab. This review synthesizes the literature on in-vehicle psychophysiological measures to advance the development of effective human-machine driving interfaces and driver support systems.

Demonstrating Brain-Level Interactions Between Visuospatial Attentional Demands and Working Memory Load While Driving Using Functional Near-Infrared Spectroscopy

Driving is a complex task concurrently drawing on multiple cognitive resources. Yet, there is a lack of studies investigating interactions at the brain-level among different driving subtasks in dual-tasking. This study investigates how visuospatial attentional demands related to increased driving difficulty interacts with different working memory load (WML) levels at the brain level. Using multichannel whole-head high density functional near-infrared spectroscopy (fNIRS) brain activation measurements, we aimed to predict driving difficulty level, both separate for each WML level and with a combined model. Participants drove for approximately 60 min on a highway with concurrent traffic in a virtual reality driving simulator. In half of the time, the course led through a construction site with reduced lane width, increasing visuospatial attentional demands. Concurrently, participants performed a modified version of the n-back task with five different WML levels (from 0-back up to 4-back), forcing them to continuously update, memorize, and recall the sequence of the previous 'n' speed signs and adjust their speed accordingly. Using multivariate logistic ridge regression, we were able to correctly predict driving difficulty in 75.0% of the signal samples (1.955 Hz sampling rate) across 15 participants in an out-of-sample cross-validation of classifiers trained on fNIRS data separately for each WML level. There was a significant effect of the WML level on the driving difficulty prediction accuracies [range 62.2-87.1%; χ2(4) = 19.9, p < 0.001, Kruskal-Wallis H test] with highest prediction rates at intermediate WML levels. On the contrary, training one classifier on fNIRS data across all WML levels severely degraded prediction performance (mean accuracy of 46.8%). Activation changes in the bilateral dorsal frontal (putative BA46), bilateral inferior parietal (putative BA39), and left superior parietal (putative BA7) areas were most predictive to increased driving difficulty. These discriminative patterns diminished at higher WML levels indicating that visuospatial attentional demands and WML involve interacting underlying brain processes. The changing pattern of driving difficulty related brain areas across WML levels could indicate potential changes in the multitasking strategy with level of WML demand, in line with the multiple resource theory.