EEG-Based Mental Workload Neurometric to Evaluate the Impact of Different Traffic and Road Conditions in Real Driving Settings

Speed behaviour and mental workload of small-spacing expressway interchanges based on field driving test

Many small-spacing interchanges (SSI) appear with the improvement of the expressway network. To investigate the speed and mental workload characteristics in the SSI and acquire the mechanism of the influence of speed on the drivers' workload, 37 participants were recruited to perform a field driving test. Each driver performed four driving conditions (i.e. ramp-mainline, mainline-ramp, mainline driving, and auxiliary lane driving). The speed and drivers' electrocardiogram (ECG) data were collected using SpeedBox speed acquisition equipment and PhysioLAB physiological instrument. The heart rate increase (HRI) index was used to analyse the drivers' mental workload regularity. The relationship model between speed and HRI was developed to examine the impact of speed on HRI. The results show that the speed variation in the SSI displayed two patterns: 'decrease - increase and continuous decrease.' The drivers' HRI variation presented four patterns: 'convex curve, continuously increasing, continuously decreasing and concave curve'. SSI's influenced area length is given based on the speed and HRI variation regularity. HRI is significantly higher when driving in the ramp-mainline condition in the SSI than when driving in other conditions, indicating that drivers are more nervous when merging with the mainline traffic. HRI increases significantly in the first 50% of the weaving area in four driving conditions, indicating that vehicle weaving greatly influences the drivers' mental workload. A positive correlation exists between vehicle speed and drivers' HRI without interference from other vehicles and road alignment.

The moving wave: Applications of the mobile EEG approach to study human attention

Although historically confined to traditional research laboratories, electroencephalography (EEG) paradigms are now being applied to study a wide array of behaviors, from daily activities to specialized tasks in diverse fields such as sports science, neurorehabilitation, and education. This transition from traditional to real-world mobile research can provide new tools for understanding attentional processes as they occur naturally. Early mobile EEG research has made progress, despite the large size and wired connections. Recent developments in hardware and software have expanded the possibilities of mobile EEG, enabling a broader range of applications. Despite these advancements, limitations influencing mobile EEG remain that must be overcome to achieve adequate reliability and validity. In this review, we first assess the feasibility of mobile paradigms, including electrode selection, artifact correction techniques, and methodological considerations. This review underscores the importance of ecological, construct, and predictive validity in ensuring the trustworthiness and applicability of mobile EEG findings. Second, we explore studies on attention in naturalistic settings, focusing on replicating classic P3 component studies in mobile paradigms like stationary biking in our lab, and activities such as walking, cycling, and dual-tasking outside of the lab. We emphasize how the mobile approach complements traditional laboratory paradigms and the types of insights gained in naturalistic research settings. Third, we discuss promising applications of portable EEG in workplace safety and other areas including road safety, rehabilitation medicine, and brain-computer interfaces. In summary, this review explores the expanding possibilities of mobile EEG while recognizing the existing challenges in fully realizing its potential.

EEG Dataset Collection for Mental Workload Predictions in Flight-Deck Environment

High mental workload reduces human performance and the ability to correctly carry out complex tasks. In particular, aircraft pilots enduring high mental workloads are at high risk of failure, even with catastrophic outcomes. Despite progress, there is still a lack of knowledge about the interrelationship between mental workload and brain functionality, and there is still limited data on flight-deck scenarios. Although recent emerging deep-learning (DL) methods using physiological data have presented new ways to find new physiological markers to detect and assess cognitive states, they demand large amounts of properly annotated datasets to achieve good performance. We present a new dataset of electroencephalogram (EEG) recordings specifically collected for the recognition of different levels of mental workload. The data were recorded from three experiments, where participants were induced to different levels of workload through tasks of increasing cognition demand. The first involved playing the N-back test, which combines memory recall with arithmetical skills. The second was playing Heat-the-Chair, a serious game specifically designed to emphasize and monitor subjects under controlled concurrent tasks. The third was flying in an Airbus320 simulator and solving several critical situations. The design of the dataset has been validated on three different levels: (1) correlation of the theoretical difficulty of each scenario to the self-perceived difficulty and performance of subjects; (2) significant difference in EEG temporal patterns across the theoretical difficulties and (3) usefulness for the training and evaluation of AI models.

An analysis of physiological responses as indicators of driver takeover readiness in conditionally automated driving

By the year 2045, it is projected that Autonomous Vehicles (AVs) will make up half of the new vehicle market. Successful adoption of AVs can reduce drivers' stress and fatigue, curb traffic congestion, and improve safety, mobility, and economic efficiency. Due to the limited intelligence in relevant technologies, human-in-the-loop modalities are still necessary to ensure the safety of AVs at current or near future stages, because the vehicles may not be able to handle all emergencies. Therefore, it is important to know the takeover readiness of the drivers to ensure the takeover quality and avoid any potential accidents. To achieve this, a comprehensive understanding of the drivers' physiological states is crucial. However, there is a lack of systematic analysis of the correlation between different human physiological responses and takeover behaviors which could serve as important references for future studies to determine the types of data to use. This paper provides a comprehensive analysis of the effects of takeover behaviors on the common physiological indicators. A program for conditional automation was developed based on a game engine and applied to a driving simulator. The experiment incorporated three types of secondary tasks, three takeover events, and two traffic densities. Brain signals, Skin Conductance Level (SCL), and Heart Rate (HR) of the participants were collected while they were performing the driving simulations. The Frontal Asymmetry Index (FAI) (as an indicator of engagement) and Mental Workload (MWL) were calculated from the brain signals to indicate the mental states of the participants. The results revealed that the FAI of the drivers would slightly decrease after the takeover alerts were issued when they were doing secondary tasks prior to the takeover activities, and the higher difficulty of the secondary tasks could lead to lower overall FAI during the takeover periods. In contrast, The MWL and SCL increased during the takeover periods. The HR also increased rapidly at the beginning of the takeover period but dropped back to a normal level quickly. It was found that a fake takeover alert would lead to lower overall HR, slower increase, and lower peak of SCL during the takeover periods. Moreover, the higher traffic density scenarios were associated with higher MWL, and a more difficult secondary task would lead to higher MWL and HR during the takeover activities. A preliminary discussion of the correlation between the physiological data, takeover scenario, and vehicle data (that relevant to takeover readiness) was then conducted, revealing that although takeover event, SCL, and HR had slightly higher correlations with the maximum acceleration and reaction time, none of them dominated the takeover readiness. In addition, the analysis of the data across different participants was conducted, which emphasized the importance of considering standardization or normalization of the data when they were further used as input features for estimating takeover readiness. Overall, the results presented in this paper offer profound insights into the patterns of physiological data changes during takeover periods. These findings can be used as benchmarks for utilizing these variables as indicators of takeover preparedness and performance in future research endeavors.

A systematic review on the influence factors, measurement, and effect of driver workload

Driver workload (DWL) is an important factor that needs to be considered in the study of traffic safety. The research focus on DWL has undergone certain shifts with the rapid development of scientific and technological advancements in the field of transportation in recent years. This study aims to grasp the state of research on DWL by both bibliometric analysis and individual critical literature review. The knowledge structure and development trend are described using bibliometric analysis. The knowledge mapping method is applied to mine the available literature in depth. It is discovered that one of the current research focus on DWL has shifted towards investigating its application in the field of autonomous driving. Subjective questionnaires and experimental tests (including both simulation technology and field study) are the main approaches to analyze DWL. An individual critical literature review of the influencing factors, measurement, and performance of DWL is provided. Research findings have shown that DWL was highly impacted by both intrinsic (e.g., age, temperament, driving experience) and external factors (e.g., vehicles, roads, tasks, environments). Scholars are actively exploring the combined effects of various factors and the level of vehicle automation on DWL. In addition to assess DWL by using subjective measures or physiological parameter measures separately, studies have started to improve classification accuracy by combining multiple measurement methods. Safety thresholds of DWL are not sufficiently studied due to the various interference items corresponding to different scenarios, but it is expected to quantify the DWL and find the threshold by establishing assessment models considering these intrinsic and external multiple-factors simultaneously. Driver or vehicle performance indicators are controversial to measure DWL directly, but they were suitable to reflect the impact of DWL in different driving conditions.

TDLNet: Transfer Data Learning Network for Cross-Subject Classification Based on Multiclass Upper Limb Motor Imagery EEG

The limited number of brain-computer interface based on motor imagery (MI-BCI) instruction sets for different movements of single limbs makes it difficult to meet practical application requirements. Therefore, designing a single-limb, multi-category motor imagery (MI) paradigm and effectively decoding it is one of the important research directions in the future development of MI-BCI. Furthermore, one of the major challenges in MI-BCI is the difficulty of classifying brain activity across different individuals. In this article, the transfer data learning network (TDLNet) is proposed to achieve the cross-subject intention recognition for multiclass upper limb motor imagery. In TDLNet, the Transfer Data Module (TDM) is used to process cross-subject electroencephalogram (EEG) signals in groups and then fuse cross-subject channel features through two one-dimensional convolutions. The Residual Attention Mechanism Module (RAMM) assigns weights to each EEG signal channel and dynamically focuses on the EEG signal channels most relevant to a specific task. Additionally, a feature visualization algorithm based on occlusion signal frequency is proposed to qualitatively analyze the proposed TDLNet. The experimental results show that TDLNet achieves the best classification results on two datasets compared to CNN-based reference methods and transfer learning method. In the 6-class scenario, TDLNet obtained an accuracy of 65%±0.05 on the UML6 dataset and 63%±0.06 on the GRAZ dataset. The visualization results demonstrate that the proposed framework can produce distinct classifier patterns for multiple categories of upper limb motor imagery through signals of different frequencies. The ULM6 dataset is available at https://dx.doi.org/10.21227/8qw6-f578.

A Neuroergonomic Approach Fostered by Wearable EEG for the Multimodal Assessment of Drivers Trainees

When assessing trainees' progresses during a driving training program, instructors can only rely on the evaluation of a trainee's explicit behavior and their performance, without having any insight about the training effects at a cognitive level. However, being able to drive does not imply knowing how to drive safely in a complex scenario such as the road traffic. Indeed, the latter point involves mental aspects, such as the ability to manage and allocate one's mental effort appropriately, which are difficult to assess objectively. In this scenario, this study investigates the validity of deploying an electroencephalographic neurometric of mental effort, obtained through a wearable electroencephalographic device, to improve the assessment of the trainee. The study engaged 22 young people, without or with limited driving experience. They were asked to drive along five different but similar urban routes, while their brain activity was recorded through electroencephalography. Moreover, driving performance, subjective and reaction times measures were collected for a multimodal analysis. In terms of subjective and performance measures, no driving improvement could be detected either through the driver's subjective measures or through their driving performance. On the other side, through the electroencephalographic neurometric of mental effort, it was possible to catch their improvement in terms of mental performance, with a decrease in experienced mental demand after three repetitions of the driving training tasks. These results were confirmed by the analysis of reaction times, that significantly improved from the third repetition as well. Therefore, being able to measure when a task is less mentally demanding, and so more automatic, allows to deduce the degree of users training, becoming capable of handling additional tasks and reacting to unexpected events.

Neurophysiological Assessment of An Innovative Maritime Safety System in Terms of Ship Operators' Mental Workload, Stress, and Attention in the Full Mission Bridge Simulator

The current industrial environment relies heavily on maritime transportation. Despite the continuous technological advances for the development of innovative safety software and hardware systems, there is a consistent gap in the scientific literature regarding the objective evaluation of the performance of maritime operators. The human factor is profoundly affected by changes in human performance or psychological state. The difficulty lies in the fact that the technology, tools, and protocols for investigating human performance are not fully mature or suitable for experimental investigation. The present research aims to integrate these two concepts by (i) objectively characterizing the psychological state of mariners, i.e., mental workload, stress, and attention, through their electroencephalographic (EEG) signal analysis, and (ii) validating an innovative safety framework countermeasure, defined as Human Risk-Informed Design (HURID), through the aforementioned neurophysiological approach. The proposed study involved 26 mariners within a high-fidelity bridge simulator while encountering collision risk in congested waters with and without the HURID. Subjective, behavioral, and neurophysiological data, i.e., EEG, were collected throughout the experimental activities. The results showed that the participants experienced a statistically significant higher mental workload and stress while performing the maritime activities without the HURID, while their attention level was statistically lower compared to the condition in which they performed the experiments with the HURID (all p < 0.05). Therefore, the presented study confirmed the effectiveness of the HURID during maritime operations in critical scenarios and led the way to extend the neurophysiological evaluation of the HFs of maritime operators during the performance of critical and/or standard shipboard tasks.

Time-Dependent Analysis of Human Neurophysiological Activities during an Ecological Olfactory Experience

It has been demonstrated that odors could affect humans at the psychophysiological level. Significant research has been done on odor perception and physiological mechanisms; however, this research was mainly performed in highly controlled conditions in order to highlight the perceptive phenomena and the correlated physiological responses in the time frame of milliseconds. The present study explored how human physiological activity evolves in response to different odor conditions during an ecological olfactory experience on a broader time scale (from 1 to 90 s). Two odors, vanilla and menthol, together with a control condition (blank) were employed as stimuli. Electroencephalographic (EEG) activity in four frequency bands of interest, theta, alpha, low beta, and high beta, and the electrodermal activity (EDA) of the skin conductance level and response (SCL and SCR) were investigated at five time points taken during: (i) the first ten seconds of exposure (short-term analysis) and (ii) throughout the entire exposure to each odor (90 s, long-term analysis). The results revealed significant interactions between the odor conditions and the time periods in the short-term analysis for the overall frontal activity in the theta (p = 0.03), alpha (p = 0.005), and low beta (p = 0.0067) bands, the frontal midline activity in the alpha (p = 0.015) and low beta (p = 0.02) bands, and the SCR component (p = 0.024). For the long-term effects, instead, only one EEG parameter, frontal alpha asymmetry, was significantly sensitive to the considered dimensions (p = 0.037). In conclusion, the present research determined the physiological response to different odor conditions, also demonstrating the sensitivity of the employed parameters in characterizing the dynamic of such response during the time. As an exploratory study, this work points out the relevance of considering the effects of continuous exposure instead of short stimulation when evaluating the human olfactory experience, providing insights for future studies in the field.

Using machine learning methods and EEG to discriminate aircraft pilot cognitive workload during flight

Pilots of aircraft face varying degrees of cognitive workload even during normal flight operations. Periods of low cognitive workload may be followed by periods of high cognitive workload and vice versa. During such changing demands, there exists potential for increased error on behalf of the pilots due to periods of boredom or excessive cognitive task demand. To further understand cognitive workload in aviation, the present study involved collection of electroencephalogram (EEG) data from ten (10) collegiate aviation students in a live-flight environment in a single-engine aircraft. Each pilot possessed a Federal Aviation Administration (FAA) commercial pilot certificate and either FAA class I or class II medical certificate. Each pilot flew a standardized flight profile representing an average instrument flight training sequence. For data analysis, we used four main sub-bands of the recorded EEG signals: delta, theta, alpha, and beta. Power spectral density (PSD) and log energy entropy of each sub-band across 20 electrodes were computed and subjected to two feature selection algorithms (recursive feature elimination (RFE) and lasso cross-validation (LassoCV), and a stacking ensemble machine learning algorithm composed of support vector machine, random forest, and logistic regression. Also, hyperparameter optimization and tenfold cross-validation were used to improve the model performance, reliability, and generalization. The feature selection step resulted in 15 features that can be considered an indicator of pilots' cognitive workload states. Then these features were applied to the stacking ensemble algorithm, and the highest results were achieved using the selected features by the RFE algorithm with an accuracy of 91.67% (± 0.11), a precision of 93.89% (± 0.09), recall of 91.67% (± 0.11), F-score of 91.22% (± 0.12), and the mean ROC-AUC of 0.93 (± 0.06). The achieved results indicated that the combination of PSD and log energy entropy, along with well-designed machine learning algorithms, suggest the potential for the use of EEG to discriminate periods of the low, medium, and high workload to augment aircraft system design, including flight automation features to improve aviation safety.

Reliability of Mental Workload Index Assessed by EEG with Different Electrode Configurations and Signal Pre-Processing Pipelines

BACKGROUND AND OBJECTIVE

Mental workload (MWL) is a relevant construct involved in all cognitively demanding activities, and its assessment is an important goal in many research fields. This paper aims at evaluating the reproducibility and sensitivity of MWL assessment from EEG signals considering the effects of different electrode configurations and pre-processing pipelines (PPPs).

METHODS

Thirteen young healthy adults were enrolled and were asked to perform 45 min of Simon's task to elicit a cognitive demand. EEG data were collected using a 32-channel system with different electrode configurations (fronto-parietal; Fz and Pz; Cz) and analyzed using different PPPs, from the simplest bandpass filtering to the combination of filtering, Artifact Subspace Reconstruction (ASR) and Independent Component Analysis (ICA). The reproducibility of MWL indexes estimation and the sensitivity of their changes were assessed using Intraclass Correlation Coefficient and statistical analysis.

RESULTS

MWL assessed with different PPPs showed reliability ranging from good to very good in most of the electrode configurations (average consistency > 0.87 and average absolute agreement > 0.92). Larger fronto-parietal electrode configurations, albeit being more affected by the choice of PPPs, provide better sensitivity in the detection of MWL changes if compared to a single-electrode configuration (18 vs. 10 statistically significant differences detected, respectively).

CONCLUSIONS

The most complex PPPs have been proven to ensure good reliability (>0.90) and sensitivity in all experimental conditions. In conclusion, we propose to use at least a two-electrode configuration (Fz and Pz) and complex PPPs including at least the ICA algorithm (even better including ASR) to mitigate artifacts and obtain reliable and sensitive MWL assessment during cognitive tasks.

EEG-based analysis for pilots' at-risk cognitive competency identification using RF-CNN algorithm

Cognitive competency is an essential complement to the existing ship pilot screening system that should be focused on. Situation awareness (SA), as the cognitive foundation of unsafe behaviors, is susceptible to influencing piloting performance. To address this issue, this paper develops an identification model based on random forest- convolutional neural network (RF-CNN) method for detecting at-risk cognitive competency (i.e., low SA level) using wearable EEG signal acquisition technology. In the poor visibility scene, the pilots' SA levels were correlated with EEG frequency metrics in frontal (F) and central (C) regions, including α/β (p = 0.071 < 0.1 in F and p = 0.042 < 0.05 in C), θ/(α + θ) (p = 0.048 < 0.05 in F and p = 0.026 < 0.05 in C) and (α + θ)/β (p = 0.046 < 0.05 in F and p = 0.012 < 0.05 in C), and then a total of 12 correlation features were obtained based on a 5 s sliding time window. Using the RF algorithm developed by principal component analysis (PCA) for further feature combination, these salient combinations are used as input sets to obtain the CNN algorithm with optimal parameters for identification. The comparative results of the proposed RF-CNN (accuracy is 84.8%) against individual RF (accuracy is 78.1%) and CNN (accuracy is 81.6%) methods demonstrate that the RF-CNN with feature optimization provides the best identification of at-risk cognitive competency (accuracy increases 6.7%). Overall, the results of this paper provide key technical support for the development of an adaptive evaluation system of pilots' cognitive competency based on intelligent technology, and lay the foundation and framework for monitoring the cognitive process and competency of ship piloting operation in China.

Wearable electroencephalography and multi-modal mental state classification: A systematic literature review

BACKGROUND

Wearable multi-modal time-series classification applications outperform their best uni-modal counterparts and hold great promise. A modality that directly measures electrical correlates from the brain is electroencephalography. Due to varying noise sources, different key brain regions, key frequency bands, and signal characteristics like non-stationarity, techniques for data pre-processing and classification algorithms are task-dependent.

METHOD

Here, a systematic literature review on mental state classification for wearable electroencephalography is presented. Four search terms in different combinations were used for an in-title search. The search was executed on the 29th of June 2022, across Google Scholar, PubMed, IEEEXplore, and ScienceDirect. 76 most relevant publications were set into context as the current state-of-the-art in mental state time-series classification.

RESULTS

Pre-processing techniques, features, and time-series classification models were analyzed. Across publications, a window length of one second was mainly chosen for classification and spectral features were utilized the most. The achieved performance per time-series classification model is analyzed, finding linear discriminant analysis, decision trees, and k-nearest neighbors models outperform support-vector machines by a factor of up to 1.5. A historical analysis depicts future trends while under-reported aspects relevant to practical applications are discussed.

CONCLUSIONS

Five main conclusions are given, covering utilization of available area for electrode placement on the head, most often or scarcely utilized features and time-series classification model architectures, baseline reporting practices, as well as explainability and interpretability of Deep Learning. The importance of a 'test battery' assessing the influence of data pre-processing and multi-modality on time-series classification performance is emphasized.

EMCI: A Novel EEG-Based Mental Workload Assessment Index of Mild Cognitive Impairment

As aging deepens, early detection of mild cognitive impairment (MCI) is increasingly important to prevent Alzheimer Dementia (AD) and improve the quality of life of older adults. In recent years, a large number of studies focus on the abnormal brain cognitive function of MCI, while ignoring the quantitative evaluation of MCI's mental workload. In this study, we propose a workload index for MCI screening, named EMCI, which is a linear discriminant cumulative estimate of subjects' electroencephalography (EEG) power spectra in α and β rhythms. Then, we design a matched prototype system to verify the effectiveness of EMCI. The results show that the EMCI is sensitive to changes of subjects' mental workload, and is significantly lower in MCI than in HC (Health control), which may be precisely caused by cognitive dysfunction. The proposed EMCI index can be used for online assessment of mental workload in older adults, which can help achieve quick screening of MCI and provide a critical window for clinical treatment interventions.

Classification of Drivers' Mental Workload Levels: Comparison of Machine Learning Methods Based on ECG and Infrared Thermal Signals

Mental workload (MW) represents the amount of brain resources required to perform concurrent tasks. The evaluation of MW is of paramount importance for Advanced Driver-Assistance Systems, given its correlation with traffic accidents risk. In the present research, two cognitive tests (Digit Span Test-DST and Ray Auditory Verbal Learning Test-RAVLT) were administered to participants while driving in a simulated environment. The tests were chosen to investigate the drivers' response to predefined levels of cognitive load to categorize the classes of MW. Infrared (IR) thermal imaging concurrently with heart rate variability (HRV) were used to obtain features related to the psychophysiology of the subjects, in order to feed machine learning (ML) classifiers. Six categories of models have been compared basing on unimodal IR/unimodal HRV/multimodal IR + HRV features. The best classifier performances were reached by the multimodal IR + HRV features-based classifiers (DST: accuracy = 73.1%, sensitivity = 0.71, specificity = 0.69; RAVLT: accuracy = 75.0%, average sensitivity = 0.75, average specificity = 0.87). The unimodal IR features based classifiers revealed high performances as well (DST: accuracy = 73.1%, sensitivity = 0.73, specificity = 0.73; RAVLT: accuracy = 71.1%, average sensitivity = 0.71, average specificity = 0.85). These results demonstrated the possibility to assess drivers' MW levels with high accuracy, also using a completely non-contact and non-invasive technique alone, representing a key advancement with respect to the state of the art in traffic accident prevention.

Evaluation of a New Lightweight EEG Technology for Translational Applications of Passive Brain-Computer Interfaces

Technologies like passive brain-computer interfaces (BCI) can enhance human-machine interaction. Anyhow, there are still shortcomings in terms of easiness of use, reliability, and generalizability that prevent passive-BCI from entering real-life situations. The current work aimed to technologically and methodologically design a new gel-free passive-BCI system for out-of-the-lab employment. The choice of the water-based electrodes and the design of a new lightweight headset met the need for easy-to-wear, comfortable, and highly acceptable technology. The proposed system showed high reliability in both laboratory and realistic settings, performing not significantly different from the gold standard based on gel electrodes. In both cases, the proposed system allowed effective discrimination (AUC > 0.9) between low and high levels of workload, vigilance, and stress even for high temporal resolution (<10 s). Finally, the generalizability of the proposed system has been tested through a cross-task calibration. The system calibrated with the data recorded during the laboratory tasks was able to discriminate the targeted human factors during the realistic task reaching AUC values higher than 0.8 at 40 s of temporal resolution in case of vigilance and workload, and 20 s of temporal resolution for the stress monitoring. These results pave the way for ecologic use of the system, where calibration data of the realistic task are difficult to obtain.

Real-time Pilot Crew's Mental Workload and Arousal Assessment During Simulated Flights for Training Evaluation: a Case Study

Training assessment is usually done by evaluating information derived from instructor's supervision related to the pilot's operational performance and behavior. However, this approach lacks objective measures, especially regarding the pilots' mental states while accomplishing the flight training tasks. The study therefore aimed at developing and testing a method for gathering and analyzing in real-time pilots' brain activity and skin conductance to improve the training evaluation. In this regard, Novice pilots' neurophysiological signals were acquired throughout multi-crew training sessions. The results demonstrated how the methodology proposed was able to endow real-time pilots' mental workload and arousal assessment for i) better evaluating training progress and operational behavior during the training session, and ii) for objectively comparing different training sessions.

Human Mental Workload: A Survey and a Novel Inclusive Definition

Human mental workload is arguably the most invoked multidimensional construct in Human Factors and Ergonomics, getting momentum also in Neuroscience and Neuroergonomics. Uncertainties exist in its characterization, motivating the design and development of computational models, thus recently and actively receiving support from the discipline of Computer Science. However, its role in human performance prediction is assured. This work is aimed at providing a synthesis of the current state of the art in human mental workload assessment through considerations, definitions, measurement techniques as well as applications, Findings suggest that, despite an increasing number of associated research works, a single, reliable and generally applicable framework for mental workload research does not yet appear fully established. One reason for this gap is the existence of a wide swath of operational definitions, built upon different theoretical assumptions which are rarely examined collectively. A second reason is that the three main classes of measures, which are self-report, task performance, and physiological indices, have been used in isolation or in pairs, but more rarely in conjunction all together. Multiple definitions complement each another and we propose a novel inclusive definition of mental workload to support the next generation of empirical-based research. Similarly, by comprehensively employing physiological, task-performance, and self-report measures, more robust assessments of mental workload can be achieved.

EEG-Based Index for Timely Detecting User's Drowsiness Occurrence in Automotive Applications

Human errors are widely considered among the major causes of road accidents. Furthermore, it is estimated that more than 90% of vehicle crashes causing fatal and permanent injuries are directly related to mental tiredness, fatigue, and drowsiness of the drivers. In particular, driving drowsiness is recognized as a crucial aspect in the context of road safety, since drowsy drivers can suddenly lose control of the car. Moreover, the driving drowsiness episodes mostly appear suddenly without any prior behavioral evidence. The present study aimed at characterizing the onset of drowsiness in car drivers by means of a multimodal neurophysiological approach to develop a synthetic electroencephalographic (EEG)-based index, able to detect drowsy events. The study involved 19 participants in a simulated scenario structured in a sequence of driving tasks under different situations and traffic conditions. The experimental conditions were designed to induce prominent mental drowsiness in the final part. The EEG-based index, so-called “MDrow index”, was developed and validated to detect the driving drowsiness of the participants. The MDrow index was derived from the Global Field Power calculated in the Alpha EEG frequency band over the parietal brain sites. The results demonstrated the reliability of the proposed MDrow index in detecting the driving drowsiness experienced by the participants, resulting also more sensitive and timely sensible with respect to more conventional autonomic parameters, such as the EyeBlinks Rate and the Heart Rate Variability, and to subjective measurements (self-reports).

Research on Effective Recognition of Alarm Signals in Human-Machine System Based on Cognitive Neural Experiments

The reasonable design of the alarm signal in the man-machine system is one of the important factors that determine the occurrence of safety accidents. Neuroergonomics provides a new perspective for the study of the cognitive process of alarm signals, which can reveal the mechanism of human perception of visual alarm signals from the cognitive level of the brain, thereby identifying the effectiveness of alarm signals. The article's research simulated the human-machine system for heat dissipation of new energy vehicles, used the automatic control interface of the cooling water system as the stimulus material, and used the event-related potential technology in cognitive neuroscience for experimental verification. The experimental results showed that: three kinds of alarm signals (color, color + shape, color + orientation) all induce visual mismatch waves, and the effective response of human to the alarm signal is color + orientation, color + shape, color from small to large, which provides a reference for the design of the alarm signal of the man-machine system.

Design of Proactive Interaction for In-Vehicle Robots Based on Transparency

Based on the transparency theory, this study investigates the appropriate amount of transparency information expressed by the in-vehicle robot under two channels of voice and visual in a proactive interaction scenario. The experiments are to test and evaluate different transparency levels and combinations of information in different channels of the in-vehicle robot, based on a driving simulator to collect subjective and objective data, which focuses on users’ safety, usability, trust, and emotion dimensions under driving conditions. The results show that appropriate transparency expression is able to improve drivers’ driving control and subjective evaluation and that drivers need a different amount of transparency information in different types of tasks.

An Evaluation of the EEG Alpha-to-Theta and Theta-to-Alpha Band Ratios as Indexes of Mental Workload

Many research works indicate that EEG bands, specifically the alpha and theta bands, have been potentially helpful cognitive load indicators. However, minimal research exists to validate this claim. This study aims to assess and analyze the impact of the alpha-to-theta and the theta-to-alpha band ratios on supporting the creation of models capable of discriminating self-reported perceptions of mental workload. A dataset of raw EEG data was utilized in which 48 subjects performed a resting activity and an induced task demanding exercise in the form of a multitasking SIMKAP test. Band ratios were devised from frontal and parietal electrode clusters. Building and model testing was done with high-level independent features from the frequency and temporal domains extracted from the computed ratios over time. Target features for model training were extracted from the subjective ratings collected after resting and task demand activities. Models were built by employing Logistic Regression, Support Vector Machines and Decision Trees and were evaluated with performance measures including accuracy, recall, precision and f1-score. The results indicate high classification accuracy of those models trained with the high-level features extracted from the alpha-to-theta ratios and theta-to-alpha ratios. Preliminary results also show that models trained with logistic regression and support vector machines can accurately classify self-reported perceptions of mental workload. This research contributes to the body of knowledge by demonstrating the richness of the information in the temporal, spectral and statistical domains extracted from the alpha-to-theta and theta-to-alpha EEG band ratios for the discrimination of self-reported perceptions of mental workload.

Unravelling the Physiological Correlates of Mental Workload Variations in Tracking and Collision Prediction Tasks

Modern work environments have extensive interactions with technology and greater cognitive complexity of the tasks, which results in human operators experiencing increased mental workload. Air traffic control operators routinely work in such complex environments, and we designed tracking and collision prediction tasks to emulate their elementary tasks. The physiological response to the workload variations in these tasks was elucidated to untangle the impact of workload variations experienced by operators. Electroencephalogram (EEG), eye activity, and heart rate variability (HRV) data were recorded from 24 participants performing tracking and collision prediction tasks with three levels of difficulty. Our findings indicate that variations in task load in both these tasks are sensitively reflected in EEG, eye activity and HRV data. Multiple regression results also show that operators' performance in both tasks can be predicted using the corresponding EEG, eye activity and HRV data. The results also demonstrate that the brain dynamics during each of these tasks can be estimated from the corresponding eye activity, HRV and performance data. Furthermore, the markedly distinct neurometrics of workload variations in the tracking and collision prediction tasks indicate that neurometrics can provide insights on the type of mental workload. These findings have applicability to the design of future mental workload adaptive systems that integrate neurometrics in deciding not just "when" but also "what" to adapt. Our study provides compelling evidence in the viability of developing intelligent closed-loop mental workload adaptive systems that ensure efficiency and safety in complex work environments.

Validation of a Light EEG-Based Measure for Real-Time Stress Monitoring during Realistic Driving

Driver’s stress affects decision-making and the probability of risk occurrence, and it is therefore a key factor in road safety. This suggests the need for continuous stress monitoring. This work aims at validating a stress neurophysiological measure—a Neurometric—for out-of-the-lab use obtained from lightweight EEG relying on two wet sensors, in real-time, and without calibration. The Neurometric was tested during a multitasking experiment and validated with a realistic driving simulator. Twenty subjects participated in the experiment, and the resulting stress Neurometric was compared with the Random Forest (RF) model, calibrated by using EEG features and both intra-subject and cross-task approaches. The Neurometric was also compared with a measure based on skin conductance level (SCL), representing one of the physiological parameters investigated in the literature mostly correlated with stress variations. We found that during both multitasking and realistic driving experiments, the Neurometric was able to discriminate between low and high levels of stress with an average Area Under Curve (AUC) value higher than 0.9. Furthermore, the stress Neurometric showed higher AUC and stability than both the SCL measure and the RF calibrated with a cross-task approach. In conclusion, the Neurometric proposed in this work proved to be suitable for out-of-the-lab monitoring of stress levels.

Let Complexity Bring Clarity: A Multidimensional Assessment of Cognitive Load Using Physiological Measures

The effects of cognitive load on driver behavior and traffic safety are unclear and in need of further investigation. Reliable measures of cognitive load for use in research and, subsequently, in the development and implementation of driver monitoring systems are therefore sought. Physiological measures are of interest since they can provide continuous recordings of driver state. Currently, however, a few issues related to their use in this context are not usually taken into consideration, despite being well-known. First, cognitive load is a multidimensional construct consisting of many mental responses (cognitive load components) to added task demand. Yet, researchers treat it as unidimensional. Second, cognitive load does not occur in isolation; rather, it is part of a complex response to task demands in a specific operational setting. Third, physiological measures typically correlate with more than one mental state, limiting the inferences that can be made from them individually. We suggest that acknowledging these issues and studying multiple mental responses using multiple physiological measures and independent variables will lead to greatly improved measurability of cognitive load. To demonstrate the potential of this approach, we used data from a driving simulator study in which a number of physiological measures (heart rate, heart rate variability, breathing rate, skin conductance, pupil diameter, eye blink rate, eye blink duration, EEG alpha power, and EEG theta power) were analyzed. Participants performed a cognitively loading n-back task at two levels of difficulty while driving through three different traffic scenarios, each repeated four times. Cognitive load components and other coinciding mental responses were assessed by considering response patterns of multiple physiological measures in relation to multiple independent variables. With this approach, the construct validity of cognitive load is improved, which is important for interpreting results accurately. Also, the use of multiple measures and independent variables makes the measurements (when analyzed jointly) more diagnostic-that is, better able to distinguish between different cognitive load components. This in turn improves the overall external validity. With more detailed, diagnostic, and valid measures of cognitive load, the effects of cognitive load on traffic safety can be better understood, and hence possibly mitigated.

Urban Mid-Block Bicycle Crossings: The Effects of Red Colored Pavement and Portal Overhead Bicycle Crossing Sign

This paper aims to investigate the effectiveness of some mid-block bicycle crossing elements by analyzing the drivers’ behavior, when approaching the bicycle crossings in a real road experiments with 18 participants. The eye-tracking instrument has been used to monitor the driver’s visual behavior during the test in an instrumented vehicle with GPS (global positioning system) and an inertial measurement unit (IMU). In particular, the drivers’ gaze was investigated frame by frame while approaching the mid-block bicycle crossings. The results showed that the red colored pavement increased the visibility of the mid-block crossing zone to 65.3% with respect to zebra crossing 59.6%. The drivers’ visual field was also narrowed by the portal overhead bicycle crossing sign and, consequently, drivers reduced their velocity and looked more to the vertical signs by 28%. The drivers’ speed reduction helped drivers to see the mid-block crossing elements from a greater distance with a higher fixation duration.

Continuous Monitoring of Mental Load During Virtual Simulator Training for Laparoscopic Surgery Reflects Laparoscopic Dexterity: A Comparative Study Using a Novel Wireless Device

Introduction

Cognitive Load Theory (CLT) relates to the efficiency with which individuals manipulate the limited capacity of working memory load. Repeated training generally results in individual performance increase and cognitive load decrease, as measured by both behavioral and neuroimaging methods. One of the known biomarkers for cognitive load is frontal theta band, measured by an EEG. Simulation-based training is an effective tool for acquiring practical skills, specifically to train new surgeons in a controlled and hazard-free environment. Measuring the cognitive load of young surgeons undergoing such training can help to determine whether they are ready to take part in a real surgery. In this study, we measured the performance of medical students and interns in a surgery simulator, while their brain activity was monitored by a single-channel EEG.

Methods

A total of 38 medical students and interns were divided into three groups and underwent three experiments examining their behavioral performances. The participants were performing a task while being monitored by the Simbionix LAP MENTOR™. Their brain activity was simultaneously measured using a single-channel EEG with novel signal processing (Aurora by Neurosteer^®). Each experiment included three trials of a simulator task performed with laparoscopic hands. The time retention between the tasks was different in each experiment, in order to examine changes in performance and cognitive load biomarkers that occurred during the task or as a result of nighttime sleep consolidation.

Results

The participants’ behavioral performance improved with trial repetition in all three experiments. In Experiments 1 and 2, delta band and the novel VC9 biomarker (previously shown to correlate with cognitive load) exhibited a significant decrease in activity with trial repetition. Additionally, delta, VC9, and, to some extent, theta activity decreased with better individual performance.

Discussion

In correspondence with previous research, EEG markers delta, VC9, and theta (partially) decreased with lower cognitive load and higher performance; the novel biomarker, VC9, showed higher sensitivity to lower cognitive load levels. Together, these measurements may be used for the neuroimaging assessment of cognitive load while performing simulator laparoscopic tasks. This can potentially be expanded to evaluate the efficacy of different medical simulations to provide more efficient training to medical staff and measure cognitive and mental loads in real laparoscopic surgeries.

EEG and behavioral correlates of attentional processing while walking and navigating naturalistic environments

The capacity to regulate one's attention in accordance with fluctuating task demands and environmental contexts is an essential feature of adaptive behavior. Although the electrophysiological correlates of attentional processing have been extensively studied in the laboratory, relatively little is known about the way they unfold under more variable, ecologically-valid conditions. Accordingly, this study employed a 'real-world' EEG design to investigate how attentional processing varies under increasing cognitive, motor, and environmental demands. Forty-four participants were exposed to an auditory oddball task while (1) sitting in a quiet room inside the lab, (2) walking around a sports field, and (3) wayfinding across a university campus. In each condition, participants were instructed to either count or ignore oddball stimuli. While behavioral performance was similar across the lab and field conditions, oddball count accuracy was significantly reduced in the campus condition. Moreover, event-related potential components (mismatch negativity and P3) elicited in both 'real-world' settings differed significantly from those obtained under laboratory conditions. These findings demonstrate the impact of environmental factors on attentional processing during simultaneously-performed motor and cognitive tasks, highlighting the value of incorporating dynamic and unpredictable contexts within naturalistic designs.

Driving-Induced Neurological Biomarkers in an Advanced Driver-Assistance System

Physiological signals are immediate and sensitive to neurological changes resulting from the mental workload induced by various driving environments and are considered a quantifying tool for understanding the association between neurological outcomes and driving cognitive workloads. Neurological assessment, outside of a highly-equipped clinical setting, requires an ambulatory electroencephalography (EEG) headset. This study aimed to quantify neurological biomarkers during a resting state and two different scenarios of driving states in a virtual driving environment. We investigated the neurological responses of seventeen healthy male drivers. EEG data were measured in an initial resting state, city-roadways driving state, and expressway driving state using a portable EEG headset in a driving simulator. During the experiment, the participants drove while experiencing cognitive workloads due to various driving environments, such as road traffic conditions, lane changes of surrounding vehicles, the speed limit, etc. The power of the beta and gamma bands decreased, and the power of the delta waves, theta, and frontal theta asymmetry increased in the driving state relative to the resting state. Delta-alpha ratio (DAR) and delta-theta ratio (DTR) showed a strong correlation with a resting state, city-roadways driving state, and expressway driving state. Binary machine-learning (ML) classification models showed a near-perfect accuracy between the resting state and driving state. Moderate classification performances were observed between the resting state, city-roadways state, and expressway state in multi-class classification. An EEG-based neurological state prediction approach may be utilized in an advanced driver-assistance system (ADAS).

Measuring Correlates of Mental Workload During Simulated Driving Using cEEGrid Electrodes: A Test-Retest Reliability Analysis

The EEG reflects mental processes, especially modulations in the alpha and theta frequency bands are associated with attention and the allocation of mental resources. EEG has also been used to study mental processes while driving, both in real environments and in virtual reality. However, conventional EEG methods are of limited use outside of controlled laboratory settings. While modern EEG technologies offer hardly any restrictions for the user, they often still have limitations in measurement reliability. We recently showed that low-density EEG methods using film-based round the ear electrodes (cEEGrids) are well-suited to map mental processes while driving a car in a driving simulator. In the present follow-up study, we explored aspects of ecological and internal validity of the cEEGrid measurements. We analyzed longitudinal data of 127 adults, who drove the same driving course in a virtual environment twice at intervals of 12-15 months while the EEG was recorded. Modulations in the alpha and theta frequency bands as well as within behavioral parameters (driving speed and steering wheel angular velocity) which were highly consistent over the two measurement time points were found to reflect the complexity of the driving task. At the intraindividual level, small to moderate (albeit significant) correlations were observed in about 2/3 of the participants, while other participants showed significant deviations between the two measurements. Thus, the test-retest reliability at the intra-individual level was rather low and challenges the value of the application for diagnostic purposes. However, across all participants the reliability and ecological validity of cEEGrid electrodes were satisfactory in the context of driving-related parameters.

The Sample Size Matters: To What Extent the Participant Reduction Affects the Outcomes of a Neuroscientific Research. A Case-Study in Neuromarketing Field

The sample size is a crucial concern in scientific research and even more in behavioural neurosciences, where besides the best practice it is not always possible to reach large experimental samples. In this study we investigated how the outcomes of research change in response to sample size reduction. Three indices computed during a task involving the observations of four videos were considered in the analysis, two related to the brain electroencephalographic (EEG) activity and one to autonomic physiological measures, i.e., heart rate and skin conductance. The modifications of these indices were investigated considering five subgroups of sample size (32, 28, 24, 20, 16), each subgroup consisting of 630 different combinations made by bootstrapping n (n = sample size) out of 36 subjects, with respect to the total population (i.e., 36 subjects). The correlation analysis, the mean squared error (MSE), and the standard deviation (STD) of the indexes were studied at the participant reduction and three factors of influence were considered in the analysis: the type of index, the task, and its duration (time length). The findings showed a significant decrease of the correlation associated to the participant reduction as well as a significant increase of MSE and STD (p < 0.05). A threshold of subjects for which the outcomes remained significant and comparable was pointed out. The effects were to some extents sensitive to all the investigated variables, but the main effect was due to the task length. Therefore, the minimum threshold of subjects for which the outcomes were comparable increased at the reduction of the spot duration.

Developing personalised braking and steering thresholds for driver support systems from SHRP2 NDS data

Examining the relationships between the factors associated with the crash development enabled the realisation of driver support systems aiming to proactively avert and control crash causation at various points within the crash sequence. Developing such systems requires new insights in personalised pre-crash driver behaviour with respect to braking and steering to develop crash prevention strategies. Therefore, the current study utilises Strategic Highway Research Program 2 Naturalistic Driving Studies (SHRP2 NDS) data to investigate personalised steering and braking thresholds by examining the last stage of a crash sequence. More specifically, this paper carried out an in-depth examination of braking and steering manoeuvres observed in the final 30 s prior to safety critical events. Two algorithms were developed to extract braking and steering events by examining deceleration and yaw rate and another developed and applied to determine the sequence of the manoeuvres. Based on the analysis, thresholds for detecting emerging situations were recommended. The investigation of driver behaviour before the safety critical events, provides valuable insights into the transition from normal driving to safety critical scenarios. The results indicate that 20% of the drivers did not react to the impending event suggesting that they were not aware of the imminent safety critical situation. Future development of Advanced Driver Assistance Systems (ADAS) can focus on individual drivers' needs with tailored activation thresholds. The developed algorithms can facilitate driver behaviour and safety analysis for NDS while the thresholds recommended could be exploited for the design of new driver support systems.

Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces

In industrial or surgical settings, to achieve many tasks successfully, at least two people are needed. To this end, robotic assistance could be used to enable a single person to perform such tasks alone, with the help of robots through direct, shared, or autonomous control. We are interested in four-arm manipulation scenarios, where both feet are used to control two robotic arms via bi-pedal haptic interfaces. The robotic arms complement the tasks of the biological arms, for instance, in supporting and moving an object while working on it (using both hands). To reduce fatigue, cognitive workload, and to ease the execution of the foot manipulation, we propose two types of assistance that can be enabled upon contact with the object (i.e., based on the interaction forces): autonomous-contact force generation and auto-coordination of the robotic arms. The latter relates to controlling both arms with a single foot, once the object is grasped. We designed four (shared) control strategies that are derived from the combinations (absence/presence) of both assistance modalities, and we compared them through a user study (with 12 participants) on a four-arm manipulation task. The results show that force assistance positively improves human–robot fluency in the four-arm task, the ease of use and usefulness; it also reduces the fatigue. Finally, to make the dual-assistance approach the preferred and most successful among the proposed control strategies, delegating the grasping force to the robotic arms is a crucial factor when controlling them both with a single foot.

Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems

With increasingly higher levels of automation in aerospace decision support systems, it is imperative that the human operator maintains a high level of situational awareness in different operational conditions and a central role in the decision-making process. While current aerospace systems and interfaces are limited in their adaptability, a Cognitive Human Machine System (CHMS) aims to perform dynamic, real-time system adaptation by estimating the cognitive states of the human operator. Nevertheless, to reliably drive system adaptation of current and emerging aerospace systems, there is a need to accurately and repeatably estimate cognitive states, particularly for Mental Workload (MWL), in real-time. As part of this study, two sessions were performed during a Multi-Attribute Task Battery (MATB) scenario, including a session for offline calibration and validation and a session for online validation of eleven multimodal inference models of MWL. The multimodal inference model implemented included an Adaptive Neuro Fuzzy Inference System (ANFIS), which was used in different configurations to fuse data from an Electroencephalogram (EEG) model’s output, four eye activity features and a control input feature. The results from the online validation of the ANFIS models demonstrated that five of the ANFIS models (containing different feature combinations of eye activity and control input features) all demonstrated good results, while the best performing model (containing all four eye activity features and the control input feature) showed an average Mean Absolute Error (MAE) = 0.67 ± 0.18 and Correlation Coefficient (CC) = 0.71 ± 0.15. The remaining six ANFIS models included data from the EEG model’s output, which had an offset discrepancy. This resulted in an equivalent offset for the online multimodal fusion. Nonetheless, the efficacy of these ANFIS models could be seen with the pairwise correlation with the task level, where one model demonstrated a CC = 0.77 ± 0.06, which was the highest among all the ANFIS models tested. Hence, this study demonstrates the ability for online multimodal fusion from features extracted from EEG signals, eye activity and control inputs to produce an accurate and repeatable inference of MWL.

Applications of brain imaging methods in driving behaviour research

Applications of neuroimaging methods have substantially contributed to the scientific understanding of human factors during driving by providing a deeper insight into the neuro-cognitive aspects of driver brain. This has been achieved by conducting simulated (and occasionally, field) driving experiments while collecting driver brain signals of various types. Here, this sector of studies is comprehensively reviewed at both macro and micro scales. At the macro scale, bibliometric aspects of these studies are analysed. At the micro scale, different themes of neuroimaging driving behaviour research are identified and the findings within each theme are synthesised. The surveyed literature has reported on applications of four major brain imaging methods. These include Functional Magnetic Resonance Imaging (fMRI), Electroencephalography (EEG), Functional Near-Infrared Spectroscopy (fNIRS) and Magnetoencephalography (MEG), with the first two being the most common methods in this domain. While collecting driver fMRI signal has been particularly instrumental in studying neural correlates of intoxicated driving (e.g. alcohol or cannabis) or distracted driving, the EEG method has been predominantly utilised in relation to the efforts aiming at development of automatic fatigue/drowsiness detection systems, a topic to which the literature on neuro-ergonomics of driving particularly has shown a spike of interest within the last few years. The survey also reveals that topics such as driver brain activity in semi-automated settings or neural activity of drivers with brain injuries or chronic neurological conditions have by contrast been investigated to a very limited extent. Potential topics in driving behaviour research are identified that could benefit from the adoption of neuroimaging methods in future studies. In terms of practicality, while fMRI and MEG experiments have proven rather invasive and technologically challenging for adoption in driving behaviour research, EEG and fNIRS applications have been more diverse. They have even been tested beyond simulated driving settings, in field driving experiments. Advantages and limitations of each of these four neuroimaging methods in the context of driving behaviour experiments are outlined in the paper.

An EEG-Based Transfer Learning Method for Cross-Subject Fatigue Mental State Prediction

Fatigued driving is one of the main causes of traffic accidents. The electroencephalogram (EEG)-based mental state analysis method is an effective and objective way of detecting fatigue. However, as EEG shows significant differences across subjects, effectively “transfering” the EEG analysis model of the existing subjects to the EEG signals of other subjects is still a challenge. Domain-Adversarial Neural Network (DANN) has excellent performance in transfer learning, especially in the fields of document analysis and image recognition, but has not been applied directly in EEG-based cross-subject fatigue detection. In this paper, we present a DANN-based model, Generative-DANN (GDANN), which combines Generative Adversarial Networks (GAN) to enhance the ability by addressing the issue of different distribution of EEG across subjects. The comparative results show that in the analysis of cross-subject tasks, GDANN has a higher average accuracy of 91.63% in fatigue detection across subjects than those of traditional classification models, which is expected to have much broader application prospects in practical brain–computer interaction (BCI).

The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence

Real-world experience is typically multimodal. Evidence indicates that the facilitation in the detection of multisensory stimuli is modulated by the perceptual load, the amount of information involved in the processing of the stimuli. Here, we used a realistic virtual reality environment while concomitantly acquiring Electroencephalography (EEG) and Galvanic Skin Response (GSR) to investigate how multisensory signals impact target detection in two conditions, high and low perceptual load. Different multimodal stimuli (auditory and vibrotactile) were presented, alone or in combination with the visual target. Results showed that only in the high load condition, multisensory stimuli significantly improve performance, compared to visual stimulation alone. Multisensory stimulation also decreases the EEG-based workload. Instead, the perceived workload, according to the "NASA Task Load Index" questionnaire, was reduced only by the trimodal condition (i.e., visual, auditory, tactile). This trimodal stimulation was more effective in enhancing the sense of presence, that is the feeling of being in the virtual environment, compared to the bimodal or unimodal stimulation. Also, we show that in the high load task, the GSR components are higher compared to the low load condition. Finally, the multimodal stimulation (Visual-Audio-Tactile-VAT and Visual-Audio-VA) induced a significant decrease in latency, and a significant increase in the amplitude of the P300 potentials with respect to the unimodal (visual) and visual and tactile bimodal stimulation, suggesting a faster and more effective processing and detection of stimuli if auditory stimulation is included. Overall, these findings provide insights into the relationship between multisensory integration and human behavior and cognition.

EEG in motion: Using an oddball task to explore motor interference in active skateboarding

Recent advancements in portable computer devices have opened new avenues in the study of human cognition outside research laboratories. This flexibility in methodology has led to the publication of several electroencephalography studies recording brain responses in real-world scenarios such as cycling and walking outside. In the present study, we tested the classic auditory oddball task while participants moved around an indoor running track using an electric skateboard. This novel approach allows for the study of attention in motion while virtually removing body movement. Using the skateboard auditory oddball paradigm, we found reliable and expected standard-target differences in the P3 and MMN/N2b event-related potentials. We also recorded baseline electroencephalography activity and found that, compared to this baseline, alpha power is attenuated in frontal and parietal regions during skateboarding. In order to explore the influence of motor interference in cognitive resources during skateboarding, we compared participants' preferred riding stance (baseline level of riding difficulty) versus their non-preferred stance (increased level of riding difficulty). We found that an increase in riding difficulty did not modulate the P3 and tonic alpha amplitude during skateboard motion. These results suggest that increases in motor demands might not lead to reductions in cognitive resources as shown in previous literature.

A Video-Based Technique for Heart Rate and Eye Blinks Rate Estimation: A Potential Solution for Telemonitoring and Remote Healthcare

Current telemedicine and remote healthcare applications foresee different interactions between the doctor and the patient relying on the use of commercial and medical wearable sensors and internet-based video conferencing platforms. Nevertheless, the existing applications necessarily require a contact between the patient and sensors for an objective evaluation of the patient’s state. The proposed study explored an innovative video-based solution for monitoring neurophysiological parameters of potential patients and assessing their mental state. In particular, we investigated the possibility to estimate the heart rate (HR) and eye blinks rate (EBR) of participants while performing laboratory tasks by mean of facial—video analysis. The objectives of the study were focused on: (i) assessing the effectiveness of the proposed technique in estimating the HR and EBR by comparing them with laboratory sensor-based measures and (ii) assessing the capability of the video—based technique in discriminating between the participant’s resting state (Nominal condition) and their active state (Non-nominal condition). The results demonstrated that the HR and EBR estimated through the facial—video technique or the laboratory equipment did not statistically differ (p > 0.1), and that these neurophysiological parameters allowed to discriminate between the Nominal and Non-nominal states (p < 0.02).

Forefront Users' Experience Evaluation by Employing Together Virtual Reality and Electroencephalography: A Case Study on Cognitive Effects of Scents

Scents have the ability to affect peoples' mental states and task performance with to different extents. It has been widely demonstrated that the lemon scent, included in most all-purpose cleaners, elicits stimulation and activation, while the lavender scent elicits relaxation and sedative effects. The present study aimed at investigating and fostering a novel approach to evaluate users' experience with respect to scents' effects through the joint employment of Virtual Reality and users' neurophysiological monitoring, in particular Electroencephalography. In particular, this study, involving 42 participants, aimed to compare the effects of lemon and lavender scents on the deployment of cognitive resources during a daily life experience consisting in a train journey carried out in virtual reality. Our findings showed a significant higher request of cognitive resources during the processing of an informative message for subjects exposed to the lavender scent with respect to the lemon exposure. No differences were found between lemon and lavender conditions on the self-reported items of pleasantness and involvement; as this study demonstrated, the employment of the lavender scent preserves the quality of the customer experience to the same extent as the more widely used lemon scent.

Evaluation of mental workload during automobile driving using one-class support vector machine with eye movement data

The aim of this study is to investigate the usefulness of the anomaly detection method by one-class support vector machine (OCSVM) for the evaluation of mental workload (MWL) during automobile driving. Twelve students (six males and six females) participated. The participants performed driving tasks with a driving simulator (DS) and the N-back task that was used to control their MWL. The N-back task had five difficulty levels from "none" to "3-back." Eye and head movements were measured during the DS driving. Results showed that the standard deviation (SD) of the gaze angle, SD of eyeball rotation angle, share rate of head movement, and blink frequency had significant correlations with the task difficulty. The decision boundary of OCSVM could detect 95% of high MWL state (i.e., "3-back" state). In addition, the absolute value of the distance from the decision boundary increased with the task difficulty from "0-back" to "3-back."

A Novel Mutual Information Based Feature Set for Drivers' Mental Workload Evaluation Using Machine Learning

Analysis of physiological signals, electroencephalography more specifically, is considered a very promising technique to obtain objective measures for mental workload evaluation, however, it requires a complex apparatus to record, and thus, with poor usability in monitoring in-vehicle drivers’ mental workload. This study proposes a methodology of constructing a novel mutual information-based feature set from the fusion of electroencephalography and vehicular signals acquired through a real driving experiment and deployed in evaluating drivers’ mental workload. Mutual information of electroencephalography and vehicular signals were used as the prime factor for the fusion of features. In order to assess the reliability of the developed feature set mental workload score prediction, classification and event classification tasks were performed using different machine learning models. Moreover, features extracted from electroencephalography were used to compare the performance. In the prediction of mental workload score, expert-defined scores were used as the target values. For classification tasks, true labels were set from contextual information of the experiment. An extensive evaluation of every prediction tasks was carried out using different validation methods. In predicting the mental workload score from the proposed feature set lowest mean absolute error was 0.09 and for classifying mental workload highest accuracy was 94%. According to the outcome of the study, it can be stated that the novel mutual information based features developed through the proposed approach can be employed to classify and monitor in-vehicle drivers’ mental workload.

A brain-computer interface for the continuous, real-time monitoring of working memory load in real-world environments

We developed a brain-computer interface (BCI) able to continuously monitor working memory (WM) load in real-time (considering the last 2.5 s of brain activity). The BCI is based on biomarkers derived from spectral properties of non-invasive electroencephalography (EEG), subsequently classified by a linear discriminant analysis classifier. The BCI was trained on a visual WM task, tested in a real-time visual WM task, and further validated in a real-time cross task (mental arithmetic). Throughout each trial of the cross task, subjects were given real or sham feedback about their WM load. At the end of the trial, subjects were asked whether the feedback provided was real or sham. The high rate of correct answers provided by the subjects validated not only the global behaviour of the WM-load feedback, but also its real-time dynamics. On average, subjects were able to provide a correct answer 82% of the time, with one subject having 100% accuracy. Possible cognitive and motor confounding factors were disentangled to support the claim that our EEG-based markers correspond indeed to WM.

A multimodal and signals fusion approach for assessing the impact of stressful events on Air Traffic Controllers

Stress is a word used to describe human reactions to emotionally, cognitively and physically challenging experiences. A hallmark of the stress response is the activation of the autonomic nervous system, resulting in the "fight-freeze-flight" response to a threat from a dangerous situation. Consequently, the capability to objectively assess and track a controller's stress level while dealing with air traffic control (ATC) activities would make it possible to better tailor the work shift and maintain high safety levels, as well as to preserve the operator's health. In this regard, sixteen controllers were asked to perform a realistic air traffic management (ATM) simulation during which subjective data (i.e. stress perception) and neurophysiological data (i.e. brain activity, heart rate, and galvanic skin response) were collected with the aim of accurately characterising the controller's stress level experienced in the various experimental conditions. In addition, external supervisors regularly evaluated the controllers in terms of manifested stress, safety, and efficiency throughout the ATM scenario. The results demonstrated 1) how the stressful events caused both supervisors and controllers to underestimate the experienced stress level, 2) the advantage of taking into account both cognitive and hormonal processes in order to define a reliable stress index, and 3) the importance of the points in time at which stress is measured owing to the potential transient effect once the stressful events have ceased.

Warmer Environments Increase Implicit Mental Workload Even If Learning Efficiency Is Enhanced

Climate change is one of the most important issues for humanity. To defuse this problem, it is considered necessary to improve energy efficiency, make energy sources cleaner, and reduce energy consumption in urban areas. The Japanese government has recommended an air conditioner setting of 28°C in summer and 20°C in winter since 2005. The aim of this setting is to save energy by keeping room temperatures constant. However, it is unclear whether this is an appropriate temperature for workers and students. This study examined whether thermal environments influence task performance over time. To examine whether the relationship between task performance and thermal environments influences the psychological states of participants, we recorded their subjective rating of mental workload along with their working memory score, electroencephalogram (EEG), heart rate variability, skin conductance level (SCL), and tympanum temperature during the task and compared the results among different conditions. In this experiment, participants were asked to read some texts and answer questions related to those texts. Room temperature (18, 22, 25, or 29°C) and humidity (50%) were manipulated during the task and participants performed the task at these temperatures. The results of this study showed that the temporal cost of task and theta power of EEG, which is an index for concentration, decreased over time. However, subjective mental workload increased with time. Moreover, the low frequency to high frequency ratio and SCL increased with time and heat (25 and 29°C). These results suggest that mental workload, especially implicit mental workload, increases in warmer environments, even if learning efficiency is facilitated. This study indicates integrated evidence for relationships among task performance, psychological state, and thermal environment by analyzing behavioral, subjective, and physiological indexes multidirectionally.

EEG Theta Power Activity Reflects Workload among Army Combat Drivers: An Experimental Study

We aimed to evaluate the effects of mental workload variations, as a function of the road environment, on the brain activity of army drivers performing combat and non-combat scenarios in a light multirole vehicle dynamic simulator. Forty-one non-commissioned officers completed three standardized driving exercises with different terrain complexities (low, medium, and high) while we recorded their electroencephalographic (EEG) activity. We focused on variations in the theta EEG power spectrum, a well-known index of mental workload. We also assessed performance and subjective ratings of task load. The theta EEG power spectrum in the frontal, temporal, and occipital areas were higher during the most complex scenarios. Performance (number of engine stops) and subjective data supported these findings. Our findings strengthen previous results found in civilians on the relationship between driver mental workload and the theta EEG power spectrum. This suggests that EEG activity can give relevant insight into mental workload variations in an objective, unbiased fashion, even during real training and/or operations. The continuous monitoring of the warfighter not only allows instantaneous detection of over/underload but also might provide online feedback to the system (either automated equipment or the crew) to take countermeasures and prevent fatal errors.

Brain-Computer Interfaces: Toward a Daily Life Employment

Recent publications in the Electroencephalogram (EEG)-based brain–computer interface field suggest that this technology could be ready to go outside the research labs and enter the market as a new consumer product. This assumption is supported by the recent advantages obtained in terms of front-end graphical user interfaces, back-end classification algorithms, and technology improvement in terms of wearable devices and dry EEG sensors. This editorial paper aims at mentioning these aspects, starting from the review paper “Brain–Computer Interface Spellers: A Review” (Rezeika et al., 2018), published within the Brain Sciences journal, and citing other relevant review papers that discussed these points.

A Novel Classification Method for a Driver's Cognitive Stress Level by Transferring Interbeat Intervals of the ECG Signal to Pictures

In this study, a novel classification method for a driver's cognitive stress level was proposed, whereby the interbeat intervals extracted from an electrocardiogram (ECG) signal were transferred to pictures, and a convolution neural network (CNN) was used to train the pictures to classify a driver's cognitive stress level. First, we defined three levels of tasks and collected the ECG signal of the driver at different cognitive stress levels by designing and performing a driving simulation experiment. We extracted the interbeat intervals and converted them to pictures according to the number of consecutive interbeat intervals in each picture. Second, the CNN model was used to train the data set to recognize the cognitive stress levels. Classification accuracies of 100%, 91.6% and 92.8% were obtained for the training set, validation set and test set, respectively, and were compared with those the BP neural network. Last, we discussed the influence of the number of interbeat intervals in each picture on the performance of the proposed classification method. The results showed that the performance initially improved with an increase in the number of interbeat intervals. A downward trend was observed when the number exceeded 40, and when the number was 40, the model performed best with the highest accuracy (98.79%) and a relatively low relative standard deviation (0.019).

Tracing the physiological response and behavioral performance of drivers at different levels of mental workload using driving simulators

INTRODUCTION

The use of mobile phones while driving is known to be a distraction factor and a cause of accidents. The way in which different kinds of conversations affect the behavioral performance of the driver as well as the persistence of the effects are not yet fully understood.

METHOD

In this study, in addition to comparing brain function and behavioral function in dual task conditions in three conversations types, the persistent effects of these types of conversations have also been traced.

RESULTS

The results show that the content of the mobile phone conversation while driving is the cause of the persistent changes in behavioral and brain functions. Increased time headway and lane departure was observed during and up to 5 min after the emotional conversation was finished. EEG bands also varied in different types of conversations. Cognitive conversations caused an increase in the activity of the alpha and beta bands while emotional conversations enhanced the rate of gamma and beta bands. A meaningful correlation was found between changes in the theta and alpha bands and changes in behavioral performance both during the dual task condition and after the conversation was finished, was also observed.

CONCLUSIONS

The content of the conversation is one of the most important factors that increase the risk of road accidents. This can also deteriorate the behavioral performance of the driver and can have persistent effects on behavioral performance and the brain. Practical applications: The findings of this study provide a basis to measure and tracing drivers' cognitive distractions induced by different levels of mental workload through physiological and behavioral performances.