Brain Electrodynamic and Hemodynamic Signatures Against Fatigue During Driving

A cross-scenario and cross-subject domain adaptation method for driving fatigue detection

Objective.The scarcity of electroencephalogram (EEG) data, coupled with individual and scenario variations, leads to considerable challenges in real-world EEG-based driver fatigue detection. We propose a domain adaptation method that utilizes EEG data collected from a laboratory to supplement real-world EEG data and constructs a cross-scenario and cross-subject driver fatigue detection model for real-world scenarios.Approach.First, we collect EEG data from subjects participating in a driving experiment conducted in both laboratory and real-world scenarios. To address the issue of data scarcity, we build a real-world fatigued driving detection model by integrating the real-world data with the laboratory data. Then, we propose a method named cross-scenario and cross-subject domain adaptation (CS2DA), which aims to eliminate the domain shift problem caused by individual variances and scenario differences. Adversarial learning is adopted to extract the common features observed across different subjects within the same scenario. The multikernel maximum mean discrepancy (MK-MMD) method is applied to further minimize scenario differences. Additionally, we propose a conditional MK-MMD constraint to better utilize label information. Finally, we use seven rules to fuse the predicted labels.Main results.We evaluate the CS2DA method through extensive experiments conducted on the two EEG datasets created in this work: the SEED-VLA and the SEED-VRW datasets. Different domain adaptation methods are used to construct a real-world fatigued driving detection model using data from laboratory and real-world scenarios, as well as a combination of both. Our findings show that the proposed CS2DA method outperforms the existing traditional and adversarial learning-based domain adaptation approaches. We also find that combining data from both laboratory and real-world scenarios improves the performance of the model.Significance.This study contributes two EEG-based fatigue driving datasets and demonstrates that the proposed CS2DA method can effectively enhance the performance of a real-world fatigued driving detection model.

Relationship between physical fatigue and mental fatigue based on multimodal measurement under different load levels

Based on multimodal measurement methods of NASA task load index (NASA-TLX), task performance, surface electromyography (sEMG), heart rate (HR), and functional near-infrared spectroscopy (fNIRS), this study conducted experimental measurements and analyses under 16 different load levels of physical fatigue and mental fatigue combination conditions. This study observed the interaction between physical fatigue and mental fatigue at different levels, and at the subjective level, the effect of physical fatigue on mental fatigue was greater than that of mental fatigue on physical fatigue. Secondly, the results of fNIRS analysis showed that the premotor cortex is affected by physical fatigue, and the dorsolateral prefrontal cortex is affected by mental fatigue. Finally, this study constructed a fatigue classification model with an accuracy of 95.3%, which takes multimodal physiological data as input and 16 fatigue states as output. The research results will provide a basis for fatigue analysis, evaluation, and improvement in complex working situations.

Evaluation of driver drowsiness based on respiratory metrics

BACKGROUND

The transition from alertness to drowsiness can cause considerable changes in the respiratory system, providing an opportunity to detect driver drowsiness.

OBJECTIVE

The aim of this study was to determine which respiratory features indicate driver drowsiness and then use these features to classify the level of drowsiness and alertness.

METHODS

Twenty male students (mean age 25.6±2.41 years) participated in the study using a driving simulator, and eight features, including expiration duration (ED), inspiration duration (ID), peak-to-peak amplitude (PA), inspiration-to-expiration time ratio (I/E ratio), driving, timing, respiration rate (RR), and yawning, were extracted from the respiratory signal generated by abdominal motions using a belt equipped with a force sensor.

RESULTS

All eight features were statistically significant at the significance level of 0.05. Drowsiness can be detected using respiratory features with 88% accuracy, 82% precision, 86% recall, and an 90% F1 score.

CONCLUSION

The findings of this study may be useful in the development of driver drowsiness monitoring systems based on less intrusive respiratory signal analysis, particularly for specific process automation applications when vehicle control is not in the hands of the driver.

Utilizing EEG and fNIRS for the detection of sleep-deprivation-induced fatigue and its inhibition using colored light stimulation

Drowsy driving is a common, but underestimated phenomenon in terms of associated risks as it often results in crashes causing fatalities and serious injuries. It is a challenging task to alert or reduce the driver's drowsy state using non-invasive techniques. In this study, a drowsiness reduction strategy has been developed and analyzed using exposure to different light colors and recording the corresponding electrical and biological brain activities. 31 subjects were examined by dividing them into 2 classes, a control group, and a healthy group. Fourteen EEG and 42 fNIRS channels were used to gather neurological data from two brain regions (prefrontal and visual cortices). Experiments shining 3 different colored lights have been carried out on them at certain times when there is a high probability to get drowsy. The results of this study show that there is a significant increase in HbO of a sleep-deprived participant when he is exposed to blue light. Similarly, the beta band of EEG also showed an increased response. However, the study found that there is no considerable increase in HbO and beta band power in the case of red and green light exposures. In addition to that, values of other physiological signals acquired such as heart rate, eye blinking, and self-reported Karolinska Sleepiness Scale scores validated the findings predicted by the electrical and biological signals. The statistical significance of the signals achieved has been tested using repeated measures ANOVA and t-tests. Correlation scores were also calculated to find the association between the changes in the data signals with the corresponding changes in the alertness level.

Assessing the development of mental fatigue during simulated flights with concurrent EEG-fNIRS measurement

Mental fatigue (MF) can impair pilots' performance and reactions to unforeseen events and is therefore an important concept within aviation. The physiological measurement of MF, especially with EEG and, in recent years, fNIRS, has gained much attention. However, a systematic investigation and comparison of the measurements is seldomly done. We induced MF via time on task during a 90-min simulated flight task and collected concurrent EEG-fNIRS, performance and self-report data from 31 participants. While their subjective MF increased linearly, the participants were able to keep their performance stable over the course of the experiment. EEG data showed an early increase and levelling in parietal alpha power and a slower, but steady increase in frontal theta power. No consistent trend could be observed in the fNIRS data. Thus, more research on fNIRS is needed to understand its possibilities and limits for MF assessment, and a combination with EEG is advisable to compare and validate results. Until then, EEG remains the better choice for continuous MF assessment in cockpit applications because of its high sensitivity to a transition from alert to fatigued, even before performance is impaired.

Fatigue performance in patients with chronic insomnia

Insomnia is associated with fatigue and poor driving performance, thus increasing the risk of traffic accidents. This study aimed to evaluate the effect of fatigue on driving in patients with chronic insomnia in a free-flow traffic scenario and car-following scenario, and to investigate the relationships between driving performance, cognitive function, and insomnia. The Trail Making Test (TMT), Stroop Color and Word Test (SCWT), Symbol Digit Modalities Test (SDMT), and Digit Span Test (DST) of 15 participants with mild-to-moderate chronic insomnia and 16 healthy participants were assessed. During the fatigue driving task, drivers completed simulated driving tasks under free-flow traffic and car-following scenarios. The mean speed (MS), mean acceleration (MA), mean lateral position (MLP), and standard deviation of lateral position (SDLP) were measured to assess driving performance. During fatigued tasks, the MA and MLP in the free-driving scenario were higher than those in the car-following scenario (P < 0.01), the SDLP was higher in the insomnia group than in the healthy group (P = 0.02), and the interaction effect was significantly different for MLP between the groups (P = 0.03). MS was negatively correlated with TMT score, SDMT score, and DST score, and positively correlated with time to complete TMT, errors in SCWT, and time to complete SCWT. SDLP was negatively correlated with DST score and positively correlated with time to complete SCWT. Furthermore, the insomnia group had poorer lateral vehicle control ability than the healthy group. The insomnia group had a more impaired driving performance in the free-driving scenario than in the car-following scenario. Drivers with impaired cognitive function exhibited impaired driving performance.

Slowed reaction times in cognitive fatigue are not attributable to declines in motor preparation

Cognitive fatigue (CF) can result from sustained mental effort, is characterized by subjective feelings of exhaustion and cognitive performance deficits, and is associated with slowed simple reaction time (RT). This study determined whether declines in motor preparation underlie this RT effect. Motor preparation level was indexed using simple RT and the StartReact effect, wherein a prepared movement is involuntarily triggered at short latency by a startling acoustic stimulus (SAS). It was predicted that if decreased motor preparation underlies CF-associated RT increases, then an attenuated StartReact effect would be observed following cognitive task completion. Subjective fatigue assessment and a simple RT task were performed before and after a cognitively fatiguing task or non-fatiguing control intervention. On 25% of RT trials, a SAS replaced the go-signal to assess the StartReact effect. CF inducement was verified by significant declines in cognitive performance (p = 0.003), along with increases in subjective CF (p < 0.001) and control RT (p = 0.018) following the cognitive fatigue intervention, but not the control intervention. No significant pre-to-post-test changes in SAS RT were observed, indicating that RT increases resulting from CF are not substantially associated with declines in motor preparation, and instead may be attributable to other stages of processing during a simple RT task.

Feasibility study of personalized speed adaptation method based on mental state for teleoperated robots

The teleoperated robotic system can support humans to complete tasks in high-risk, high-precision and difficult special environments. Because this kind of special working environment is easy to cause stress, high mental workload, fatigue and other mental states of the operator, which will reduce the quality of operation and even cause safety accidents, so the mental state of the people in this system has received extensive attention. However, the existence of individual differences and mental state diversity is often ignored, so that most of the existing adjustment strategy is out of a match between mental state and adaptive decision, which cannot effectively improve operational quality and safety. Therefore, a personalized speed adaptation (PSA) method based on policy gradient reinforcement learning was proposed in this paper. It can use electroencephalogram and electro-oculogram to accurately perceive the operator’s mental state, and adjust the speed of the robot individually according to the mental state of different operators, in order to perform teleoperation tasks efficiently and safely. The experimental results showed that the PSA method learns the mapping between the mental state and the robot’s speed regulation action by means of rewards and punishments, and can adjust the speed of the robot individually according to the mental state of different operators, thereby improving the operating quality of the system. And the feasibility and superiority of this method were proved. It is worth noting that the PSA method was validated on 6 real subjects rather than a simulation model. To the best of our knowledge, the PSA method is the first implementation of online reinforcement learning control of teleoperated robots involving human subjects.

Redesigning navigational aids using virtual global landmarks to improve spatial knowledge retrieval

Although beacon- and map-based spatial strategies are the default strategies for navigation activities, today's navigational aids mostly follow a beacon-based design where one is provided with turn-by-turn instructions. Recent research, however, shows that our reliance on these navigational aids is causing a decline in our spatial skills. We are processing less of our surrounding environment and relying too heavily on the instructions given. To reverse this decline, we need to engage more in map-based learning, which encourages the user to process and integrate spatial knowledge into a cognitive map built to benefit flexible and independent spatial navigation behaviour. In an attempt to curb our loss of skills, we proposed a navigation assistant to support map-based learning during active navigation. Called the virtual global landmark (VGL) system, this augmented reality (AR) system is based on the kinds of techniques used in traditional orienteering. Specifically, a notable landmark is always present in the user's sight, allowing the user to continuously compute where they are in relation to that specific location. The efficacy of the unit as a navigational aid was tested in an experiment with 27 students from the University of Technology Sydney via a comparison of brain dynamics and behaviour. From an analysis of behaviour and event-related spectral perturbation, we found that participants were encouraged to process more spatial information with a map-based strategy where a silhouette of the compass-like landmark was perpetually in view. As a result of this technique, they consistently navigated with greater efficiency and better accuracy.

Investigating mental workload-induced changes in cortical oxygenation and frontal theta activity during simulated flights

Monitoring pilots' cognitive states becomes increasingly important in aviation. Physiological measurement can detect increased mental workload (MWL) even before performance declines. Yet, changes in MWL are rarely varied systematically and few studies control for confounding effects of other cognitive states. The present study targets these shortcomings by analysing the effects of stepwise increased MWL on cortical activation, while controlling for mental fatigue (MF). 35 participants conducted a simulated flight with an incorporated adapted n-back and monitoring task. We recorded cortical activation with concurrent EEG and fNIRS measurement, performance, self-reported MWL and MF. Our results show the successful manipulation of MWL without confounding effects of MF. Higher task difficulty elicited higher subjective MWL ratings, performance decline, higher frontal theta activity and reduced frontal deoxyhaemoglobin (Hbr) concentration. Using both EEG and fNIRS, we could discriminate all induced MWL levels. fNIRS was more sensitive to tasks with low difficulty, and EEG to tasks with high difficulty. Our findings further suggest a plateau effect for high MWL that could present an upper boundary to individual cognitive capacity. Our results highlight the benefits of physiological measurement in aviation, both for assessment of cognitive states and as a data source for adaptive assistance systems.

Viewing neurovascular coupling through the lens of combined EEG-fNIRS: A systematic review of current methods

Neurovascular coupling is a key physiological mechanism that occurs in the healthy human brain, and understanding this process has implications for understanding the aging and neuropsychiatric populations. Combined electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) has emerged as a promising, noninvasive tool for probing neurovascular interactions in humans. However, the utility of this approach critically depends on the methodological quality used for multimodal integration. Despite a growing number of combined EEG-fNIRS applications reported in recent years, the methodological rigor of past studies remains unclear, limiting the accurate interpretation of reported findings and hindering the translational application of this multimodal approach. To fill this knowledge gap, we critically evaluated various methodological aspects of previous combined EEG-fNIRS studies performed in healthy individuals. A literature search was conducted using PubMed and PsycINFO on June 28, 2021. Studies involving concurrent EEG and fNIRS measurements in awake and healthy individuals were selected. After screening and eligibility assessment, 96 studies were included in the methodological evaluation. Specifically, we critically reviewed various aspects of participant sampling, experimental design, signal acquisition, data preprocessing, outcome selection, data analysis, and results presentation reported in these studies. Altogether, we identified several notable strengths and limitations of the existing EEG-fNIRS literature. In light of these limitations and the features of combined EEG-fNIRS, recommendations are made to improve and standardize research practices to facilitate the use of combined EEG-fNIRS when studying healthy neurovascular coupling processes and alterations in neurovascular coupling among various populations.

Research on the Identification of Pilots’ Fatigue Status Based on Functional Near-Infrared Spectroscopy

Fatigue can lead to sluggish responses, misjudgments, flight illusions and other problems for pilots, which could easily bring about serious flight accidents. In this paper, a wearable functional near-infrared spectroscopy (fNIRS) device was used to record the changes of hemoglobin concentration of pilots during flight missions. The data was pre-processed, and 1080 valid samples were determined. Then, mean value, variance, standard deviation, kurtosis, skewness, coefficient of variation, peak value, and range of oxyhemoglobin (HbO2) in each channel were extracted. These indexes were regarded as the input of a stacked denoising autoencoder (SDAE) and were used to train the identification model of pilots’ fatigue state. The identification model of pilots’ fatigue status was established. The identification accuracy of the SDAE model was 91.32%, which was 23.26% and 15.97% higher than that of linear discriminant analysis (LDA) models and support vector machines (SVM) models, respectively. Results show that the SDAE model established in our study has high identification accuracy, which can accurately identify different fatigue states of pilots. Identification of pilots’ fatigue status based on fNIRS has important practical significance for reducing flight accidents caused by pilot fatigue.

A Driving Performance Forecasting System Based on Brain Dynamic State Analysis Using 4-D Convolutional Neural Networks

Vehicle accidents are the primary cause of fatalities worldwide. Most often, experiencing fatigue on the road leads to operator errors and behavioral lapses. Thus, there is a need to predict the cognitive state of drivers, particularly their fatigue level. Electroencephalography (EEG) has been demonstrated to be effective for monitoring changes in the human brain state and behavior. Thirty-seven subjects participated in this driving experiment and performed a perform lane-keeping task in a visual-reality environment. Three domains, namely, frequency, temporal, and 2-D spatial information, of the EEG channel location were comprehensively considered. A 4-D convolutional neural-network (4-D CNN) algorithm was then proposed to associate all information from the EEG signals and the changes in the human state and behavioral performance. A 4-D CNN achieves superior forecasting performance over 2-D CNN, 3-D CNN, and shallow networks. The results showed a 3.82% improvement in the root mean-square error, a 3.45% improvement in the error rate, and a 11.98% improvement in the correlation coefficient with 4-D CNN compared with 3-D CNN. The 4-D CNN algorithm extracts the significant theta and alpha activations in the frontal and posterior cingulate cortices under distinct fatigue levels. This work contributes to enhancing our understanding of deep learning methods in the analysis of EEG signals. We even envision that deep learning might serve as a bridge between translation neuroscience and further real-world applications.

Research on Driving Fatigue Alleviation Using Interesting Auditory Stimulation Based on VMD-MMSE

The accurate detection and alleviation of driving fatigue are of great significance to traffic safety. In this study, we tried to apply the modified multi-scale entropy (MMSE) approach, based on variational mode decomposition (VMD), to driving fatigue detection. Firstly, the VMD was used to decompose EEG into multiple intrinsic mode functions (IMFs), then the best IMFs and scale factors were selected using the least square method (LSM). Finally, the MMSE features were extracted. Compared with the traditional sample entropy (SampEn), the VMD-MMSE method can identify the characteristics of driving fatigue more effectively. The VMD-MMSE characteristics combined with a subjective questionnaire (SQ) were used to analyze the change trends of driving fatigue under two driving modes: normal driving mode and interesting auditory stimulation mode. The results show that the interesting auditory stimulation method adopted in this paper can effectively relieve driving fatigue. In addition, the interesting auditory stimulation method, which simply involves playing interesting auditory information on the vehicle-mounted player, can effectively relieve driving fatigue. Compared with traditional driving fatigue-relieving methods, such as sleeping and drinking coffee, this interesting auditory stimulation method can relieve fatigue in real-time when the driver is driving normally.

EEG-Based Brain-Computer Interfaces (BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications

Brain-Computer interfaces (BCIs) enhance the capability of human brain activities to interact with the environment. Recent advancements in technology and machine learning algorithms have increased interest in electroencephalographic (EEG)-based BCI applications. EEG-based intelligent BCI systems can facilitate continuous monitoring of fluctuations in human cognitive states under monotonous tasks, which is both beneficial for people in need of healthcare support and general researchers in different domain areas. In this review, we survey the recent literature on EEG signal sensing technologies and computational intelligence approaches in BCI applications, compensating for the gaps in the systematic summary of the past five years. Specifically, we first review the current status of BCI and signal sensing technologies for collecting reliable EEG signals. Then, we demonstrate state-of-the-art computational intelligence techniques, including fuzzy models and transfer learning in machine learning and deep learning algorithms, to detect, monitor, and maintain human cognitive states and task performance in prevalent applications. Finally, we present a couple of innovative BCI-inspired healthcare applications and discuss future research directions in EEG-based BCI research.

Applications of brain imaging methods in driving behaviour research

Applications of neuroimaging methods have substantially contributed to the scientific understanding of human factors during driving by providing a deeper insight into the neuro-cognitive aspects of driver brain. This has been achieved by conducting simulated (and occasionally, field) driving experiments while collecting driver brain signals of various types. Here, this sector of studies is comprehensively reviewed at both macro and micro scales. At the macro scale, bibliometric aspects of these studies are analysed. At the micro scale, different themes of neuroimaging driving behaviour research are identified and the findings within each theme are synthesised. The surveyed literature has reported on applications of four major brain imaging methods. These include Functional Magnetic Resonance Imaging (fMRI), Electroencephalography (EEG), Functional Near-Infrared Spectroscopy (fNIRS) and Magnetoencephalography (MEG), with the first two being the most common methods in this domain. While collecting driver fMRI signal has been particularly instrumental in studying neural correlates of intoxicated driving (e.g. alcohol or cannabis) or distracted driving, the EEG method has been predominantly utilised in relation to the efforts aiming at development of automatic fatigue/drowsiness detection systems, a topic to which the literature on neuro-ergonomics of driving particularly has shown a spike of interest within the last few years. The survey also reveals that topics such as driver brain activity in semi-automated settings or neural activity of drivers with brain injuries or chronic neurological conditions have by contrast been investigated to a very limited extent. Potential topics in driving behaviour research are identified that could benefit from the adoption of neuroimaging methods in future studies. In terms of practicality, while fMRI and MEG experiments have proven rather invasive and technologically challenging for adoption in driving behaviour research, EEG and fNIRS applications have been more diverse. They have even been tested beyond simulated driving settings, in field driving experiments. Advantages and limitations of each of these four neuroimaging methods in the context of driving behaviour experiments are outlined in the paper.

Vector Phase Analysis Approach for Sleep Stage Classification: A Functional Near-Infrared Spectroscopy-Based Passive Brain-Computer Interface

A passive brain-computer interface (BCI) based upon functional near-infrared spectroscopy (fNIRS) brain signals is used for earlier detection of human drowsiness during driving tasks. This BCI modality acquired hemodynamic signals of 13 healthy subjects from the right dorsolateral prefrontal cortex (DPFC) of the brain. Drowsiness activity is recorded using a continuous-wave fNIRS system and eight channels over the right DPFC. During the experiment, sleep-deprived subjects drove a vehicle in a driving simulator while their cerebral oxygen regulation (CORE) state was continuously measured. Vector phase analysis (VPA) was used as a classifier to detect drowsiness state along with sleep stage-based threshold criteria. Extensive training and testing with various feature sets and classifiers are done to justify the adaptation of threshold criteria for any subject without requiring recalibration. Three statistical features (mean oxyhemoglobin, signal peak, and the sum of peaks) along with six VPA features (trajectory slopes of VPA indices) were used. The average accuracies for the five classifiers are 90.9% for discriminant analysis, 92.5% for support vector machines, 92.3% for nearest neighbors, 92.4% for both decision trees, and ensembles over all subjects' data. Trajectory slopes of CORE vector magnitude and angle: m(|R|) and m(∠R) are the best-performing features, along with ensemble classifier with the highest accuracy of 95.3% and minimum computation time of 40 ms. The statistical significance of the results is validated with a p-value of less than 0.05. The proposed passive BCI scheme demonstrates a promising technique for online drowsiness detection using VPA along with sleep stage classification.

A Methodological Review of fNIRS in Driving Research: Relevance to the Future of Autonomous Vehicles

As automobile manufacturers have begun to design, engineer, and test autonomous driving systems of the future, brain imaging with functional near-infrared spectroscopy (fNIRS) can provide unique insights about cognitive processes associated with evolving levels of autonomy implemented in the automobile. Modern fNIRS devices provide a portable, relatively affordable, and robust form of functional neuroimaging that allows researchers to investigate brain function in real-world environments. The trend toward "naturalistic neuroscience" is evident in the growing number of studies that leverage the methodological flexibility of fNIRS, and in doing so, significantly expand the scope of cognitive function that is accessible to observation via functional brain imaging (i.e., from the simulator to on-road scenarios). While more than a decade's worth of study in this field of fNIRS driving research has led to many interesting findings, the number of studies applying fNIRS during autonomous modes of operation is limited. To support future research that directly addresses this lack in autonomous driving research with fNIRS, we argue that a cogent distillation of the methods used to date will help facilitate and streamline this research of tomorrow. To that end, here we provide a methodological review of the existing fNIRS driving research, with the overarching goal of highlighting the current diversity in methodological approaches. We argue that standardization of these approaches will facilitate greater overlap of methods by researchers from all disciplines, which will, in-turn, allow for meta-analysis of future results. We conclude by providing recommendations for advancing the use of such fNIRS technology in furthering understanding the adoption of safe autonomous vehicle technology.

Mid-Task Physical Exercise Keeps Your Mind Vigilant: Evidences From Behavioral Performance and EEG Functional Connectivity

Accumulating efforts have been made to discover effective solutions for fatigue recovery with the ultimate aim of reducing adverse consequences of mental fatigue in real life. The previously-reported behavioral benefits of physical exercise on mental fatigue recovery prompted us to investigate the restorative effect and reveal the underlying neural mechanisms. Specifically, we introduced an empirical method to investigate the beneficial effect of physical exercise on the reorganization of EEG functional connectivity (FC) in a two-session experiment where one session including a successive 30-min psychomotor vigilance task (PVT) (No-intervention session) compared to an insertion of a mid-task 15-min cycling exercise (Intervention session). EEG FC was obtained from 21 participants and quantitatively assessed via graph theoretical analysis and a classification framework. The findings demonstrated the effectiveness of exercise intervention on behavioral performance as shown in improved reaction time and response accuracy. Although we found significantly altered network alterations towards the end of experiment in both sessions, no significant differences between the two sessions and no interaction between session and time were found in EEG network topology. Further interrogation of functional connectivity through classification analysis showed decreased FC in distributed brain areas, which may lead to the significant reduction of network efficiency in both sessions. Moreover, we showed distinct patterns of FC alterations between the two sessions, indicating different information processing strategies adopted in the intervention session. In sum, these results provide some of the first quantitative insights into the complex neural mechanism of exercise intervention for fatigue recovery and lead a new direction for further application research in real-world situations.

Stochastic Multichannel Ranking with Brain Dynamics Preferences

A driver's cognitive state of mental fatigue significantly affects his or her driving performance and more important, public safety. Previous studies have leveraged reaction time (RT) as the metric for mental fatigue and aim at estimating the exact value of RT using electroencephalogram (EEG) signals within a regression model. However, due to the easily corrupted and also nonsmooth properties of RTs during data collection, methods focusing on predicting the exact value of a noisy measurement, RT generally suffer from poor generalization performance. Considering that human RT is the reflection of brain dynamics preference (BDP) rather than a single regression output of EEG signals, we propose a novel channel-reliability-aware ranking (CArank) model for the multichannel ranking problem. CArank learns from BDPs using EEG data robustly and aims at preserving the ordering corresponding to RTs. In particular, we introduce a transition matrix to characterize the reliability of each channel used in the EEG data, which helps in learning with BDPs only from informative EEG channels. To handle large-scale EEG signals, we propose a stochastic-generalized expectation maximum (SGEM) algorithm to update CArank in an online fashion. Comprehensive empirical analysis on EEG signals from 40 participants shows that our CArank achieves substantial improvements in reliability while simultaneously detecting noisy or less informative EEG channels.

Methodological Approaches and Recommendations for Functional Near-Infrared Spectroscopy Applications in HF/E Research

OBJECTIVE

The objective of this study was to systematically document current methods and protocols employed when using functional near-infrared spectroscopy (fNIRS) techniques in human factors and ergonomics (HF/E) research and generate recommendations for conducting and reporting fNIRS findings in HF/E applications.

METHOD

A total of 1,687 articles were identified through Ovid-MEDLINE, PubMed, Web of Science, and Scopus databases, of which 37 articles were included in the review based on review inclusion/exclusion criteria.

RESULTS

A majority of the HF/E fNIRS investigations were found in transportation, both ground and aviation, and in assessing cognitive (e.g., workload, working memory) over physical constructs. There were large variations pertaining to data cleaning, processing, and analysis approaches across the studies that warrant standardization of methodological approaches. The review identified major challenges in transparency and reporting of important fNIRS data collection and analyses specifications that diminishes study replicability, introduces potential biases, and increases likelihood of inaccurate results. As such, results reported in existing fNIRS studies need to be cautiously approached.

CONCLUSION

To improve the quality of fNIRS investigations and/or to facilitate its adoption and integration in different HF/E applications, such as occupational ergonomics and rehabilitation, recommendations for fNIRS data collection, processing, analysis, and reporting are provided.

EEG & Eye Tracking User Experiments for Spatial Memory Task on Maps

The aim of this research is to evaluate the use of ET and EEG for studying the cognitive processes of expert and novice map users and to explore these processes by comparing two types of spatial memory experiments through cognitive load measurements. The first experiment consisted of single trials and participants were instructed to study a map stimulus without any time constraints in order to draw a sketch map afterwards. According to the ET metrics (i.e., average fixation duration and the number of fixations per second), no statistically significant differences emerged between experts and novices. A similar result was also obtained with EEG Frontal Alpha Asymmetry calculations. On the contrary, in terms of alpha power across all electrodes, novices exhibited significantly lower alpha power, indicating a higher cognitive load. In the second experiment, a larger number of stimuli were used to study the effect of task difficulty. The same ET metrics used in the first experiment indicated that the difference between these user groups was not statistically significant. The cognitive load was also extracted using EEG event-related spectral power changes at alpha and theta frequency bands. Preliminary data exploration mostly suggested an increase in theta power and a decrease in alpha power.

Exploring the Brain Responses to Driving Fatigue Through Simultaneous EEG and fNIRS Measurements

Fatigue is one problem with driving as it can lead to difficulties with sustaining attention, behavioral lapses, and a tendency to ignore vital information or operations. In this research, we explore multimodal physiological phenomena in response to driving fatigue through simultaneous functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) recordings with the aim of investigating the relationships between hemodynamic and electrical features and driving performance. Sixteen subjects participated in an event-related lane-deviation driving task while measuring their brain dynamics through fNIRS and EEGs. Three performance groups, classified as Optimal, Suboptimal, and Poor, were defined for comparison. From our analysis, we find that tonic variations occur before a deviation, and phasic variations occur afterward. The tonic results show an increased concentration of oxygenated hemoglobin (HbO₂) and power changes in the EEG theta, alpha, and beta bands. Both dynamics are significantly correlated with deteriorated driving performance. The phasic EEG results demonstrate event-related desynchronization associated with the onset of steering vehicle in all power bands. The concentration of phasic HbO₂ decreased as performance worsened. Further, the negative correlations between tonic EEG delta and alpha power and HbO₂ oscillations suggest that activations in HbO₂ are related to mental fatigue. In summary, combined hemodynamic and electrodynamic activities can provide complete knowledge of the brain's responses as evidence of state changes during fatigue driving.

An oculometrics-based biofeedback system to impede fatigue development during computer work: A proof-of-concept study

A biofeedback system may objectively identify fatigue and provide an individualized timing plan for micro-breaks. We developed and implemented a biofeedback system based on oculometrics using continuous recordings of eye movements and pupil dilations to moderate fatigue development in its early stages. Twenty healthy young participants (10 males and 10 females) performed a cyclic computer task for 31–35 min over two sessions: 1) self-triggered micro-breaks (manual sessions), and 2) biofeedback-triggered micro-breaks (automatic sessions). The sessions were held with one-week inter-session interval and in a counterbalanced order across participants. Each session involved 180 cycles of the computer task and after each 20 cycles (a segment), the task paused for 5-s to acquire perceived fatigue using Karolinska Sleepiness Scale (KSS). Following the pause, a 25-s micro-break involving seated exercises was carried out whether it was triggered by the biofeedback system following the detection of fatigue (KSS≥5) in the automatic sessions or by the participants in the manual sessions. National Aeronautics and Space Administration Task Load Index (NASA-TLX) was administered after sessions. The functioning core of the biofeedback system was based on a Decision Tree Ensemble model for fatigue classification, which was developed using an oculometrics dataset previously collected during the same computer task. The biofeedback system identified fatigue with a mean accuracy of approx. 70%. Perceived workload obtained from NASA-TLX was significantly lower in the automatic sessions compared with the manual sessions, p = 0.01 Cohen’s d_z = 0.89. The results give support to the effectiveness of integrating oculometrics-based biofeedback in timing plan of micro-breaks to impede fatigue development during computer work.

Finger Temperature: A Psychophysiological Assessment of the Attentional State

Attention is a key cognitive phenomenon that is studied to understand cognitive disorders or even to estimate workloads to prevent accidents. Usually, it is studied using brain activity, even though it has many psychophysiological correlates. In the present study, we aim to evaluate if finger temperature, as a surrogate of peripheral vasoconstriction, can be used to obtain similar and complementary information to electroencephalography (EEG) brain activity measurements. To conduct this, 34 participants were recruited and submitted to performing four tasks-one as a baseline, and three attentional tasks. These three attentional tasks measured sustained attention, resilience to distractors, and attentional resources. During the tasks, the room, forehead, tympanic, and finger temperatures were measured. Furthermore, we included a 32-channel EEG recording. Our results showed a strong monotonic association between the finger temperature and the Alpha and Beta EEG spectral bands. When predicting attentional performance, the finger temperature was complementary to the EEG spectral measurements, through the prediction of aspects of attentional performance that had not been assessed by spectral EEG activity, or through the improvement of the model's fit. We also found that during the baseline task (non-goal-oriented task), the spectral EEG activity has an inverted correlation, as compared to a goal-oriented task. Our current results suggest that the psychophysiological assessment of attention is complementary to classic EEG approach, while also having the advantage of easy implementation of analysis tools in environments of reducing control (workplaces, student classrooms).

Multi-channel EEG recordings during a sustained-attention driving task

We describe driver behaviour and brain dynamics acquired from a 90-minute sustained-attention task in an immersive driving simulator. The data included 62 sessions of 32-channel electroencephalography (EEG) data for 27 subjects driving on a four-lane highway who were instructed to keep the car cruising in the centre of the lane. Lane-departure events were randomly induced to cause the car to drift from the original cruising lane towards the left or right lane. A complete trial included events with deviation onset, response onset, and response offset. The next trial, in which the subject was instructed to drive back to the original cruising lane, began 5-10 seconds after finishing the previous trial. We believe that this dataset will lead to the development of novel neural processing methodology that can be used to index brain cortical dynamics and detect driving fatigue and drowsiness. This publicly available dataset will be beneficial to the neuroscience and brain-computer interface communities.

Brain Complex Network Characteristic Analysis of Fatigue during Simulated Driving Based on Electroencephalogram Signals

Fatigued driving is one of the major causes of traffic accidents. Frequent repetition of driving behavior for a long time may lead to driver fatigue, which is closely related to the central nervous system. In the present work, we designed a fatigue driving simulation experiment and collected the electroencephalogram (EEG) signals. Complex network theory was introduced to study the evolution of brain dynamics under different rhythms of EEG signals during several periods of the simulated driving. The results show that as the fatigue degree deepened, the functional connectivity and the clustering coefficients increased while the average shortest path length decreased for the delta rhythm. In addition, there was a significant increase of the degree centrality in partial channels on the right side of the brain for the delta rhythm. Therefore, it can be concluded that driving fatigue can cause brain complex network characteristics to change significantly for certain brain regions and certain rhythms. This exploration may provide a theoretical basis for further finding objective and effective indicators to evaluate the degree of driving fatigue and to help avoid fatigue driving.

How Physical Activities Affect Mental Fatigue Based on EEG Energy, Connectivity, and Complexity

Many studies have verified that there is an interaction between physical activities and mental fatigue. However, few studies are focused on the effect of physical activities on mental fatigue. This study was to analyze the states of mental fatigue based on electroencephalography (EEG) and investigate how physical activities affect mental fatigue. Fourteen healthy participants participated in an experiment including a 2-back mental task (the control) and the same mental task with cycling simultaneously (physical-mental task). Each experiment consisted of three 20 min fatigue-inducing sessions repeatedly (mental fatigue for mental tasks or mental fatigue plus physical activities for physical-mental tasks). During the evaluation sessions (before and after the fatigue-inducing sessions), the states of the participants were assessed by EEG parameters. Wavelet Packet Energy (WPE), Spectral Coherence Value (SCV), and Lempel-Ziv Complexity (LZC) were used to indicate mental fatigue from the perspectives of activation, functional connectivity, and complexity of the brain. The indices are the beta band energy E_β, the energy ratio E_α/β, inter-hemispheric SCV of beta band SCV_β and LZC. The statistical analysis shows that mental fatigue was detected by E_β, E_α/β, SCV_β, and LZC in physical-mental task. The slopes of the linear fit on these indices verified that the mental fatigue increased more fast during physical-mental task. It is concluded form the result that physical activities can enhance the mental fatigue and speed up the fatigue process based on brain activation, functional connection, and complexity. This result differs from the traditional opinion that physical activities have no influence on mental fatigue, and finds that physical activities can increase mental fatigue. This finding helps fatigue management through exercise instruction.

Prefrontal Cortex Activity Predicts Mental Fatigue in Young and Elderly Men During a 2 h "Go/NoGo" Task

Background: Although the effects of mental fatigue on cognitive–motor function and psychological state in young adults are well-documented, its effects in the elderly are not completely understood. The aim of this study was to estimate the effect of prolonged cognitive load on the indicators of psychological, cognitive, and motor functions.

Methods: Fifteen young and 15 elderly men were asked to perform a 2 h “Go/NoGo” task. Psychological state (mood and motivation), cognitive (prefrontal cortex activity and cognitive performance), and motor (motor cortex excitability and grip strength) functions were measured before and after the task. During the 2 h task, both groups had a significantly similar increase in the number of “Incorrect NoGo” errors. Only in young men reaction time (RT) of “Incorrect NoGo” and intraindividual variability of RT of “Incorrect NoGo” significantly increased during task. After the task, handgrip strength decreased for the young men, whereas latency of motor evoked potentials prolonged both groups. Nevertheless, both groups indicated that they felt fatigue after the 2 h task; we observed that mental demand increased, whereas intrinsic motivation and mood decreased only in young men. Prolonged task decreased the switching/rest ratio of oxygenated hemoglobin for the young and the elderly men; however, greater for elderly than young men. Interestingly, the more the prefrontal cortex was activated before the 2 h task during the switching task, the fewer of “Incorrect NoGo” errors made by the young men and the greater the number of errors made by the elderly men.

Conclusion: Because of the greater mental load and (possibly) greater activation of prefrontal cortex during the 2 h “Go/NoGo” task, there was greater mental and neuromuscular performance fatigue in young men than in elderly men.

Identifying the Causes of Drivers' Hazardous States Using Driver Characteristics, Vehicle Kinematics, and Physiological Measurements

Drivers’ hazardous physical and mental states (e.g., distraction, fatigue, stress, and high workload) have a major effect on driving performance and strongly contribute to 25–50% of all traffic accidents. They are caused by numerous factors, such as cell phone use or lack of sleep. However, while significant research has been done on detecting hazardous states, most studies have not tried to identify the causes of the hazardous states. Such information would be very useful, as it would allow intelligent vehicles to better respond to a detected hazardous state. Thus, this study examined whether the cause of a driver’s hazardous state can be automatically identified using a combination of driver characteristics, vehicle kinematics, and physiological measures. Twenty-one healthy participants took part in four 45-min sessions of simulated driving, of which they were mildly sleep-deprived for two sessions. Within each session, there were eight different scenarios with different weather (sunny or snowy), traffic density and cell phone usage (with or without cell phone). During each scenario, four physiological (respiration, electrocardiogram, skin conductance, and body temperature) and eight vehicle kinematics measures were monitored. Additionally, three self-reported driver characteristics were obtained: personality, stress level, and mood. Three feature sets were formed based on driver characteristics, vehicle kinematics, and physiological signals. All possible combinations of the three feature sets were used to classify sleep deprivation (drowsy vs. alert), traffic density (low vs. high), cell phone use, and weather conditions (foggy/snowy vs. sunny) with highest accuracies of 98.8%, 91.4%, 82.3%, and 71.5%, respectively. Vehicle kinematics were most useful for classification of weather and traffic density while physiology and driver characteristics were useful for classification of sleep deprivation and cell phone use. Furthermore, a second classification scheme was tested that also incorporates information about whether or not other causes of hazardous states are present, though this did not result in higher classification accuracy. In the future, these classifiers could be used to identify both the presence and cause of a driver’s hazardous state, which could serve as the basis for more intelligent intervention systems.