Driving Fatigue Detection from EEG Using a Modified PCANet Method

Reduction of cognitive fatigue and improved performance at a VR-based driving simulator using tRNS

Cognitive fatigue (CF) increases accident risk reducing performance, especially during complex tasks such as driving. We evaluated whether transcranial random noise stimulation (tRNS) could mitigate CF and improve driving performance. In a double-blind study, thirty participants performed a virtual reality truck driving task during real (n = 15) or sham (n = 15) tRNS applied bilaterally on the "anti-fatigue network". They completed two 30-min driving sessions while their driving performances were constantly monitored; heart rate was also monitored to evaluate arousal (Root-Mean-Square of successive R-R difference). tRNS was applied only during the first driving session to evaluate both online and offline stimulation effects. The primary outcome was CF reduction and performance improvement in the second (non-stimulated) driving session. Real tRNS significantly improved driving performances in the second driving session and reduced perceived CF. These results might also lead to the use of tRNS in those neurological disorders characterized by fatigue.

EEG-based driving intuition and collision anticipation using joint temporal-frequency multi-layer dynamic brain network

Intuition plays a crucial role in human driving decision-making, and this rapid and unconscious cognitive process is essential for improving traffic safety. We used the first proposed multi-layer network analysis method, "Joint Temporal-Frequency Multi-layer Dynamic Brain Network" (JTF-MDBN), to study the EEG data from the initial and advanced phases of driving intuition training in the theta, alpha, and beta bands. Additionally, we conducted a comparative study between these two phases using multi-layer metrics as well as local and global metrics of single layers. The results show that brain region activity is more stable in the advanced phase of intuition training compared to the initial phase. Particularly in the alart state task, the JTF-MDBN demonstrated stronger connection strength. Multi-layer network analysis indicates that modularity is significantly higher for the non-alert state task than the alert state task in the alpha and beta bands. In the W4 time window (1 second before a collision), we identified significant features that can differentiate situations where a car collision is imminent from those where no collision occurs. Single-layer network analysis also revealed statistical differences in node strength and local efficiency for some EEG channels in the alpha and beta bands during the W4 and W5 time windows. Using these biomarkers to predict vehicle collision risk, the classification accuracy of a linear kernel SVM reached up to 87.5%, demonstrating the feasibility of predicting driving collisions through brain network biomarkers. These findings are important for the study of human intuition and the development of brain-computer interface-based intelligent driving hazard perception assistance systems.

A Hybrid EEG-Based Stress State Classification Model Using Multi-Domain Transfer Entropy and PCANet

This paper proposes a new hybrid model for classifying stress states using EEG signals, combining multi-domain transfer entropy (TrEn) with a two-dimensional PCANet (2D-PCANet) approach. The aim is to create an automated system for identifying stress levels, which is crucial for early intervention and mental health management. A major challenge in this field lies in extracting meaningful emotional information from the complex patterns observed in EEG. Our model addresses this by initially applying independent component analysis (ICA) to purify the EEG signals, enhancing the clarity for further analysis. We then leverage the adaptability of the fractional Fourier transform (FrFT) to represent the EEG data in time, frequency, and time-frequency domains. This multi-domain representation allows for a more nuanced understanding of the brain's activity in response to stress. The subsequent stage involves the deployment of a two-layer 2D-PCANet network designed to autonomously distill EEG features associated with stress. These features are then classified by a support vector machine (SVM) to determine the stress state. Moreover, stress induction and data acquisition experiments are designed. We employed two distinct tasks known to trigger stress responses. Other stress-inducing elements that enhance the stress response were included in the experimental design, such as time limits and performance feedback. The EEG data collected from 15 participants were retained. The proposed algorithm achieves an average accuracy of over 92% on this self-collected dataset, enabling stress state detection under different task-induced conditions.

Driving fatigue detection based on brain source activity and ARMA model

Fatigue among drivers is a significant issue in society, and according to organizational reports, it substantially contributes to accidents. So accurate fatigue detection in drivers plays a crucial role in reducing the number of people fatalities or injured resulting from accidents. Several methods are proposed for fatigue driver recognition among which electroencephalography (EEG) is one. This paper proposed a method for fatigue recognition by EEG signals with extracted features from source and sensor spaces. The proposed method starts with preprocessing by applying filtering and artifact rejection. Then source localization methods are applied to EEG signals for active source extraction. A multivariate autoregressive (MVAR) model is fitted to selected sources, and a dual Kalman filter is applied to estimate the source activity and their relationships. Then multivariate autoregressive moving average (ARMA) is fitted between EEG and source activity signals. Features are extracted from model parameters, source relationship matrix, and wavelet transform of EEG and source activity signals. The novelty of this approach is the use of ARMA model between source activities (as input) and EEG signals (as output) and feature extraction from source relations. Relevant features are selected using a combination of RelifF and neighborhood component analysis (NCA) methods. Three classifiers, namely k-nearest neighbor (KNN), support vector machine (SVM), and naive Bayesian (NB) classifiers, are employed to classify drivers. To improve performance, the final label for fatigue detection is calculated by combining these classifiers using the voting method. The results demonstrate that the proposed method accurately recognizes and classifies fatigued drivers with the ensemble classifiers in comparison with other methods.

Driver drowsiness detection methods using EEG signals: a systematic review

Electroencephalography (EEG) is a complex signal that may require several years of training, advanced signal processing, and feature extraction methodologies to interpret correctly. Recently, many methods have been used to extract and classify EEG data. This study reviews 62 papers that used EEG signals to detect driver drowsiness, published between January 2018 and 2022. We extract trends and highlight interesting approaches from this large body of literature to inform future research and formulate recommendations. To find relevant papers published in scientific journals, conferences, and electronic preprint repositories, researchers searched major databases covering the domains of science and engineering. For each investigation, many data items about (1) the data, (2) the channels used, (3) the extraction and classification procedure, and (4) the outcomes were extracted. These items were then analyzed one by one to uncover trends. Our analysis reveals that the amount of EEG data used across studies varies. We saw that more than half the studies used simulation driving experimental. About 21% of the studies used support vector machine (SVM), while 19% used convolutional neural networks (CNN). Overall, we can conclude that drowsiness and fatigue impair driving performance, resulting in drivers who are more exposed to risky situations.

A regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based cross-subject fatigue detection

Fatigue, one of the most important factors affecting road safety, has attracted many researchers' attention. Most existing fatigue detection methods are based on feature engineering and classification models. The feature engineering is greatly influenced by researchers' domain knowledge, which will lead to a poor performance in fatigue detection, especially in cross-subject experiment design. In addition, fatigue detection is often simplified as a classification problem of several discrete states. Models based on deep learning can realize automatic feature extraction without the limitation of researcher's domain knowledge. Therefore, this paper proposes a regression model combined convolutional neural network and recurrent neural network for electroencephalogram-based (EEG-based) cross-subject fatigue detection. At the same time, a twofold random-offset zero-overlapping sampling method is proposed to train a bigger model and reduce overfitting. Compared with existing results, the proposed method achieves a much better result of 0.94 correlation coefficient (COR) and 0.09 root mean square error (RMSE) in a within-subject experiment design. What is more, there is no misclassification between awake and drowsy states. For cross-subject experiment design, the COR and RMSE are 0.79 and 0.15, respectively, which are close to the existing within-subject results and better than similar cross-subject results. The cross-subject regression model is very important for fatigue detection application since the fatigue indication is more precise than several discrete states and no model calibration is required for a new user. The twofold random-offset zero-overlapping sampling method can also be used as a reference by other EEG-based deep learning research.

Electroencephalogram based brain-computer interface: Applications, challenges, and opportunities

Brain-Computer Interfaces (BCI) is an exciting and emerging research area for researchers and scientists. It is a suitable combination of software and hardware to operate any device mentally. This review emphasizes the significant stages in the BCI domain, current problems, and state-of-the-art findings. This article also covers how current results can contribute to new knowledge about BCI, an overview of BCI from its early developments to recent advancements, BCI applications, challenges, and future directions. The authors pointed to unresolved issues and expressed how BCI is valuable for analyzing the human brain. Humans' dependence on machines has led humankind into a new future where BCI can play an essential role in improving this modern world.

A real-time driver fatigue identification method based on GA-GRNN

It is of great practical and theoretical significance to identify driver fatigue state in real time and accurately and provide active safety warning in time. In this paper, a non-invasive and low-cost method of fatigue driving state identification based on genetic algorithm optimization of generalized regression neural network model is proposed. The specific work is as follows: (1) design simulated driving experiment and real driving experiment, determine the fatigue state of drivers according to the binary Karolinska Sleepiness Scale (KSS), and establish the fatigue driving sample database. (2) Improved Multi-Task Cascaded Convolutional Networks (MTCNN) and applied to face detection. Dlib library was used to extract the coordinate values of face feature points, collect the characteristic parameters of driver's eyes and mouth, and calculate the Euler Angle parameters of head posture. A fatigue identification model was constructed by using multiple characteristic parameters. (3) Genetic Algorithm (GA) was used to find the optimal smooth factor of Generalized Regression Neural Network (GRNN) and construct GA-GRNN fatigue driving identification model. Compared with K-Nearest Neighbor (KNN), Random Forest (RF), and GRNN fatigue driving identification algorithms. GA-GRNN has the best generalization ability and high stability, with an accuracy of 93.3%. This study provides theoretical and technical support for the application of driver fatigue identification.

Study on the Effect of Judgment Excitation Mode to Relieve Driving Fatigue Based on MF-DFA

Driving fatigue refers to a phenomenon in which a driver’s physiological and psychological functions become unbalanced after a long period of continuous driving, and their driving skills decline objectively. The hidden dangers of driving fatigue to traffic safety should not be underestimated. In this work, we propose a judgment excitation mode (JEM), which adds secondary cognitive tasks to driving behavior through dual-channel human–computer interaction, so as to delay the occurrence of driving fatigue. We used multifractal detrended fluctuation analysis (MF-DFA) to study the dynamic properties of subjects’ EEG, and analyzed the effect of JEM on fatigue retardation by Hurst exponent value and multifractal spectrum width value. The results show that the multifractal properties of the two driving modes (normal driving mode and JEM) are significantly different. The JEM we propose can effectively delay the occurrence of driving fatigue, and has good prospects for future practical applications.

Feasibility study of personalized speed adaptation method based on mental state for teleoperated robots

The teleoperated robotic system can support humans to complete tasks in high-risk, high-precision and difficult special environments. Because this kind of special working environment is easy to cause stress, high mental workload, fatigue and other mental states of the operator, which will reduce the quality of operation and even cause safety accidents, so the mental state of the people in this system has received extensive attention. However, the existence of individual differences and mental state diversity is often ignored, so that most of the existing adjustment strategy is out of a match between mental state and adaptive decision, which cannot effectively improve operational quality and safety. Therefore, a personalized speed adaptation (PSA) method based on policy gradient reinforcement learning was proposed in this paper. It can use electroencephalogram and electro-oculogram to accurately perceive the operator’s mental state, and adjust the speed of the robot individually according to the mental state of different operators, in order to perform teleoperation tasks efficiently and safely. The experimental results showed that the PSA method learns the mapping between the mental state and the robot’s speed regulation action by means of rewards and punishments, and can adjust the speed of the robot individually according to the mental state of different operators, thereby improving the operating quality of the system. And the feasibility and superiority of this method were proved. It is worth noting that the PSA method was validated on 6 real subjects rather than a simulation model. To the best of our knowledge, the PSA method is the first implementation of online reinforcement learning control of teleoperated robots involving human subjects.

Concurrent fNIRS and EEG for Brain Function Investigation: A Systematic, Methodology-Focused Review

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) stand as state-of-the-art techniques for non-invasive functional neuroimaging. On a unimodal basis, EEG has poor spatial resolution while presenting high temporal resolution. In contrast, fNIRS offers better spatial resolution, though it is constrained by its poor temporal resolution. One important merit shared by the EEG and fNIRS is that both modalities have favorable portability and could be integrated into a compatible experimental setup, providing a compelling ground for the development of a multimodal fNIRS-EEG integration analysis approach. Despite a growing number of studies using concurrent fNIRS-EEG designs reported in recent years, the methodological reference of past studies remains unclear. To fill this knowledge gap, this review critically summarizes the status of analysis methods currently used in concurrent fNIRS-EEG studies, providing an up-to-date overview and guideline for future projects to conduct concurrent fNIRS-EEG studies. A literature search was conducted using PubMed and Web of Science through 31 August 2021. After screening and qualification assessment, 92 studies involving concurrent fNIRS-EEG data recordings and analyses were included in the final methodological review. Specifically, three methodological categories of concurrent fNIRS-EEG data analyses, including EEG-informed fNIRS analyses, fNIRS-informed EEG analyses, and parallel fNIRS-EEG analyses, were identified and explained with detailed description. Finally, we highlighted current challenges and potential directions in concurrent fNIRS-EEG data analyses in future research.

ADRIS: The new open-source accessible driving simulator for training and evaluation of driving abilities

BACKGROUND AND OBJECTIVE

Independent living and transportation are crucial aspects for people living with a disability. After an injury, it is important to assess driving ability, in terms of physical and psychological conditions, and to test the effects of prescribed drugs. Within this framework, driving simulators are suitable tools for training driving skills; however, available tools are expensive or lack appropriate sets of behavioral measures to fully characterize the drivers' ability.

METHODS

This work presents the first step toward the development of ADRIS, a new open-source, accessible, realistic virtual reality simulator for training and testing driving skills of people with sensory-motor disability. This includes a prototype based on an open-source simulator for autonomous driving research (CARLA), with the addition of customized features such as adaptable driving controllers, a virtual reality headset, and the possibility to collect behavioral and physiological data. Also, the new system allows to set different environmental conditions, to include and control the timing of potentially dangerous situations, and to set scenarios with various difficulty levels.

RESULTS

Tests on 17 healthy participants demonstrated that the simulator is well tolerated in terms of discomfort, physical fatigue, and mental effort. Also, the system is easy to use and is capable of providing a realistic driving experience, allowing the extraction of reliable behavioral parameters.

CONCLUSIONS

ADRIS combines a high-fidelity virtual world, with customizable features specifically designed for the training and testing of people living with a disability, thus making it usable in many contexts such as home training, rehabilitation, education, and research.

Assessment of Combination of Automated Pupillometry and Heart Rate Variability to Detect Driving Fatigue

Objectives

Approximately 20~30% of all traffic accidents are caused by fatigue driving. However, limited practicability remains a barrier for the real application of available techniques to detect driving fatigue. Use of pupillary light reflex (PLR) may be potentially effective for driving fatigue detection.

Methods

A 90 min monotonous simulated driving task was utilized to induce driving fatigue. During the task, PLR measurements were performed at baseline and at an interval of 30 min. Subjective rating scales, heart rate variability (HRV) were monitored simultaneously.

Results

Thirty-two healthy volunteers in China participated in our study. Based on the results of subjective evaluation and behavioral performances, driving fatigue was verified to be successfully induced by a simulated driving task. Significant variations of PLR and HRV parameters were observed, which also showed significant relevance with the change in Karolinska Sleepiness Scale at several timepoints (|r| = 0.55 ~ 0.72, P < 0.001). Furthermore, PLR variations had excellent ability to detect driving fatigue with high sensitivity and specificity, of which maximum constriction velocity variations achieved a sensitivity of 85.00% and specificity of 72.34% for driving fatigue detection, vs. 82.50 and 78.72% with a combination of HRV variations, a nonsignificant difference (AUC = 0.835, 0.872, P > 0.05).

Conclusions

Pupillary light reflex variation may be a potential indicator in the detection of driving fatigue, achieving a comparative performance compared with the combination with heart rate variability. Further work may be involved in developing a commercialized driving fatigue detection system based on pupillary parameters.

Types, Risk Factors, Consequences, and Detection Methods of Train Driver Fatigue and Distraction

Train drivers' inattention, including fatigue and distraction, impairs their ability to drive and is the major risk factor for human-caused train accidents. Many experts have undertaken numerous studies on train driver exhaustion and distraction, but a systematic study is still missing. Through a systematic review, this work aims to outline the types, risk factors, consequences, and detection methods of train driver fatigue and distraction. The effects of central nervous fatigue and cognitive distraction in train drivers during driving are caused by rest and sleep schedules, workload, automation levels, and mobile phones. Furthermore, train drivers' fatigue and distraction can cause loss of concentration and slow reaction, resulting in dangerous driving behaviour such as speeding and SPAD. Researchers have combined subjective reporting, physiological parameters, and physical factors to construct detection algorithms with good results to detect train driver fatigue and distraction. This review offers recommendations for researchers looking into train driver fatigue and distraction. And it can also make valuable recommendations for future studies about railway traffic safety.

A Novel Prototype for Safe Driving Using Embedded Smart Box System

Every day, vehicle accidents occur and many of them might be avoided if the drivers demonstrated excellent driving without mistakes. This paper presents a novel prototype applied in a real transportation system, particularly for buses, to avoid accidents, which may involve numerous victims, and even occasionally cause death. This system consists of a wearable device and embedded system with several sensors connected via Bluetooth, similar to the Internet of Things (IoT). Wearable devices are made to monitor the driver's heart rate and alert the driver if they are in a state of sleep deprivation to prevent any potential accidents. The embedded system includes a Global Positioning System (GPS), accelerometers, and gyroscopes attached to a Smart Box mounted on the bus. The embedded system alert function will be triggered if an accident occurs and automatically sends the geolocation of the accident to the registered phone number through a message using a mobile phone. The results for all scenarios were significant when measured by an automatic accident trigger via the smart box if the value of measured values in each axis exceeded 583. In conclusion, the implementation of this innovative solution at the system-level was shown to be satisfactory in terms of the safety mechanism used by the nominated drivers.

Multi-Parameter Physiological State Monitoring in Target Detection Under Real-World Settings

Mental state changes induced by stimuli under experimental settings or by daily events in real life affect task performance and are entwined with physical and mental health. In this study, we developed a physiological state indicator with five parameters that reflect the subject's real-time physiological states based on online EEG signal processing. These five parameters are attention, fatigue, stress, and the brain activity shifts of the left and right hemispheres. We designed a target detection experiment modified by a cognitive attention network test for validating the effectiveness of the proposed indicator, as such conditions would better approximate a real chaotic environment. Results demonstrated that attention levels while performing the target detection task were significantly higher than during rest periods, but also exhibited a decay over time. In contrast, the fatigue level increased gradually and plateaued by the third rest period. Similar to attention levels, the stress level decreased as the experiment proceeded. These parameters are therefore shown to be highly correlated to different stages of the experiment, suggesting their usage as primary factors in passive brain-computer interfaces (BCI). In addition, the left and right brain activity indexes reveal the EEG neural modulations of the corresponding hemispheres, which set a feasible reference of activation for an active BCI control system, such as one executing motor imagery tasks. The proposed indicator is applicable to potential passive and active BCI applications for monitoring the subject's physiological state change in real-time, along with providing a means of evaluating the associated signal quality to enhance the BCI performance.

Advanced e-Call Support Based on Non-Intrusive Driver Condition Monitoring for Connected and Autonomous Vehicles

BACKGROUND

The growth of the number of vehicles in traffic has led to an exponential increase in the number of road accidents with many negative consequences, such as loss of lives and pollution.

METHODS

This article focuses on using a new technology in automotive electronics by equipping a semi-autonomous vehicle with a complex sensor structure that is able to provide centralized information regarding the physiological signals (Electro encephalogram-EEG, electrocardiogram-ECG) of the driver/passengers and their location along with indoor temperature changes, employing the Internet of Things (IoT) technology. Thus, transforming the vehicle into a mobile sensor connected to the internet will help highlight and create a new perspective on the cognitive and physiological conditions of passengers, which is useful for specific applications, such as health management and a more effective intervention in case of road accidents. These sensor structures mounted in vehicles will allow for a higher detection rate of potential dangers in real time. The approach uses detection, recording, and transmission of relevant health information in the event of an incident as support for e-Call or other emergency services, including telemedicine.

RESULTS

The novelty of the research is based on the design of specialized non-invasive sensors for the acquisition of EEG and ECG signals installed in the headrest and backrest of car seats, on the algorithms used for data analysis and fusion, but also on the implementation of an IoT temperature measurement system in several points that simultaneously uses sensors based on MEMS technology. The solution can also be integrated with an e-Call system for telemedicine emergency assistance.

CONCLUSION

The research presents both positive and negative results of field experiments, with possible further developments. In this context, the solution has been developed based on state-of-the-art technical devices, methods, and technologies for monitoring vital functions of the driver/passengers (degree of fatigue, cognitive state, heart rate, blood pressure). The purpose is to reduce the risk of accidents for semi-autonomous vehicles and to also monitor the condition of passengers in the case of autonomous vehicles for providing first aid in a timely manner. Reported abnormal values of vital parameters (critical situations) will allow interveneing in a timely manner, saving the patient's life, with the support of the e-Call system.

Toward practical driving fatigue detection using three frontal EEG channels: a proof-of-concept study

Objective. Although various driving fatigue detection strategies have been introduced, the limited practicability is still an obstacle for the real application of these technologies. This study is based on the newly proposed non-hair-bearing (NHB) method to achieve practical driving fatigue detection with fewer channels from NHB areas and more efficient electroencephalogram (EEG) features.Approach. EEG data were recorded from 20 healthy subjects (15 males, age = 22.2 ± 3.2 years) in a 90 min simulated driving task using a remote wireless cap. Behaviorally, subjects demonstrated a salient fatigue effect, as reflected by a monotonic increase in reaction time. Using a sliding-window approach, we determined the vigilant and fatigued states at individual level to reduce the inter-subject differences in behavioral impairment and brain activity. Multiple EEG features, including power-spectrum density (PSD), functional connectivity (FC), and entropy, were estimated in a pairwise manner, which were set as input for fatigue classification.Main results. Intriguingly, this data-driven approach showed that the best classification performance was achieved using three EEG channel pairs located in the NHB area. The mixed features of the frontal NHB area lead to the high within-subject detection rate of driving fatigue (92.7% ± 0.92%) with satisfactory generalizability for fatigue classification across different subjects (77.13% ± 0.85%). Moreover, we found the most prominent contributing features were PSD of different frequency bands within the frontal NHB area and FC within the frontal NHB area and between frontal and parietal areas.Significance. In summary, the current work provided objective evidence to support the effectiveness of the NHB method and further improved the performance, thereby moving a step forward towards practical driving fatigue detection in real-world scenarios.

Applications of brain imaging methods in driving behaviour research

Applications of neuroimaging methods have substantially contributed to the scientific understanding of human factors during driving by providing a deeper insight into the neuro-cognitive aspects of driver brain. This has been achieved by conducting simulated (and occasionally, field) driving experiments while collecting driver brain signals of various types. Here, this sector of studies is comprehensively reviewed at both macro and micro scales. At the macro scale, bibliometric aspects of these studies are analysed. At the micro scale, different themes of neuroimaging driving behaviour research are identified and the findings within each theme are synthesised. The surveyed literature has reported on applications of four major brain imaging methods. These include Functional Magnetic Resonance Imaging (fMRI), Electroencephalography (EEG), Functional Near-Infrared Spectroscopy (fNIRS) and Magnetoencephalography (MEG), with the first two being the most common methods in this domain. While collecting driver fMRI signal has been particularly instrumental in studying neural correlates of intoxicated driving (e.g. alcohol or cannabis) or distracted driving, the EEG method has been predominantly utilised in relation to the efforts aiming at development of automatic fatigue/drowsiness detection systems, a topic to which the literature on neuro-ergonomics of driving particularly has shown a spike of interest within the last few years. The survey also reveals that topics such as driver brain activity in semi-automated settings or neural activity of drivers with brain injuries or chronic neurological conditions have by contrast been investigated to a very limited extent. Potential topics in driving behaviour research are identified that could benefit from the adoption of neuroimaging methods in future studies. In terms of practicality, while fMRI and MEG experiments have proven rather invasive and technologically challenging for adoption in driving behaviour research, EEG and fNIRS applications have been more diverse. They have even been tested beyond simulated driving settings, in field driving experiments. Advantages and limitations of each of these four neuroimaging methods in the context of driving behaviour experiments are outlined in the paper.

A dynamic center and multi threshold point based stable feature extraction network for driver fatigue detection utilizing EEG signals

Driver fatigue is the one of the main reasons of the traffic accidents. The human brain is a complex structure, whose function can be evaluated with electroencephalogram (EEG). Automated driver fatigue detection utilizing EEG decreases the incidence probability of related traffic accidents. Therefore, devising an appropriate feature extraction technique and selecting a competent classification method can be considered as the crucial part of the effective driver fatigue detection. Therefore, in this study, an EEG-based intelligent system was devised for driver fatigue detection. The proposed framework includes a new feature generation network, which is implemented by using texture descriptors, for fatigue detection. The proposed scheme contains pre-processing, feature generation, informative features selection and classification with shallow classifiers phases. In the pre-processing, discrete cosine transform and fast Fourier transform are used together. Moreover, dynamic center based binary pattern and multi threshold ternary pattern are utilized together to create a new feature generation network. To improve the detection performance, we utilized discrete wavelet transform as a pooling method, in which the functional brain network-based feature describing the relationship between fatigue and brain network organization. In the feature selection phase, a hybrid three layered feature selection method is presented, and benchmark classifiers are used in the classification phase to demonstrate the strength of the proposed method. In the experiments, the proposed framework achieved 97.29% classification accuracy for fatigue detection using EEG signals. This result reveals that the proposed framework can be utilized effectively for driver fatigue detection.

Rhythm-Dependent Multilayer Brain Network for the Detection of Driving Fatigue

Fatigue driving has attracted a great deal of attention for its huge influence on automobile accidents. Recognizing driving fatigue provides a primary but significant way for addressing this problem. In this paper, we first conduct the simulated driving experiments to acquire the EEG signals in alert and fatigue states. Then, for multi-channel EEG signals without pre-processing, a novel rhythm-dependent multilayer brain network (RDMB network) is developed and analyzed for driving fatigue detection. We find that there exists a significant difference between alert and fatigue states from the view of network science. Further, key sub-RDMB network based on closeness centrality are extracted. We calculate six network measures from the key sub-RDMB network and construct feature vectors to classify the alert and fatigue states. The results show that our method can respectively achieve the average accuracy of 95.28% (with sample length of 5 s), 90.25% (2 s), and 87.69% (1 s), significantly higher than compared methods. All these validate the effectiveness of RDMB network for reliable driving fatigue detection via EEG.

Convolutional Neural Network for Drowsiness Detection Using EEG Signals

Drowsiness detection (DD) has become a relevant area of active research in biomedical signal processing. Recently, various deep learning (DL) researches based on the EEG signals have been proposed to detect fatigue conditions. The research presented in this paper proposes an EEG classification system for DD based on DL networks. However, the proposed DD system is mainly realized into two procedures; (i) data acquisition and (ii) model analysis. For the data acquisition procedure, two key steps are considered, which are the signal collection using a wearable Emotiv EPOC+ headset to record 14 channels of EEG, and the signal annotation. Furthermore, a data augmentation (DA) step has been added to the proposed system to overcome the problem of over-fitting and to improve accuracy. As regards the model analysis, a comparative study is also introduced in this paper to argue the choice of DL architecture and frameworks used in our DD system. In this sense, The proposed DD protocol makes use of a convolutional neural network (CNN) architecture implemented using the Keras library. The results showed a high accuracy value (90.42%) in drowsy/awake discrimination and revealed the efficiency of the proposed DD system compared to other research works.

Driver sleepiness detection with deep neural networks using electrophysiological data

OBJECTIVE

The objective of this paper is to present a driver sleepiness detection model based on electrophysiological data and a neural network consisting of Convolutional Neural Networks and a Long Short Term Memory architecture.

APPROACH

The model was developed and evaluated on data from 12 different experiments with 269 drivers and 1187 driving sessions during daytime (low sleepiness condition) and night-time (high sleepiness condition), collected during naturalistic driving conditions on real roads in Sweden or in an advanced moving-base driving simulator. Electrooculographic and electroencephalographic time series data, split up in 16634 2.5-minute data segments was used as input to the deep neural network. This probably constitutes the largest labelled driver sleepiness dataset in the world. The model outputs a binary decision as alert (defined as ≤6 on the Karolinska Sleepiness Scale, KSS) or sleepy (KSS≥8) or a regression output corresponding to KSS ϵ [1-5,6,7,8,9].

MAIN RESULTS

The subject-independent mean absolute error (MAE) was 0.78. Binary classification accuracy for the regression model was 82.6% as compared to 82.0% for a model that was trained specifically for the binary classification task. Data from the eyes were more informative than data from the brain. A combined input improved performance for some models, but the gain was very limited.

SIGNIFICANCE

Improved classification results were achieved with the regression model compared to the classification model. This suggests that the implicit order of the KSS ratings, i.e. the progression from alert to sleepy, provides important information for robust modelling of driver sleepiness, and that class labels should not simply be aggregated into an alert and a sleepy class. Furthermore, the model consistently showed better results than a model trained on manually extracted features based on expert knowledge, indicating that the model can detect sleepiness that is not covered by traditional algorithms.

EEG-Based Driving Fatigue Detection Using a Two-Level Learning Hierarchy Radial Basis Function

Electroencephalography (EEG)-based driving fatigue detection has gained increasing attention recently due to the non-invasive, low-cost, and potable nature of the EEG technology, but it is still challenging to extract informative features from noisy EEG signals for driving fatigue detection. Radial basis function (RBF) neural network has drawn lots of attention as a promising classifier due to its linear-in-the-parameters network structure, strong non-linear approximation ability, and desired generalization property. The RBF network performance heavily relies on network parameters such as the number of the hidden nodes, number of the center vectors, width, and output weights. However, global optimization methods that directly optimize all the network parameters often result in high evaluation cost and slow convergence. To enhance the accuracy and efficiency of EEG-based driving fatigue detection model, this study aims to develop a two-level learning hierarchy RBF network (RBF-TLLH) which allows for global optimization of the key network parameters. Experimental EEG data were collected, at both fatigue and alert states, from six healthy participants in a simulated driving environment. Principal component analysis was first utilized to extract features from EEG signals, and the proposed RBF-TLLH was then employed for driving status (fatigue vs. alert) classification. The results demonstrated that the proposed RBF-TLLH approach achieved a better classification performance (mean accuracy: 92.71%; area under the receiver operating curve: 0.9199) compared to other widely used artificial neural networks. Moreover, only three core parameters need to be determined using the training datasets in the proposed RBF-TLLH classifier, which increases its reliability and applicability. The findings demonstrate that the proposed RBF-TLLH approach can be used as a promising framework for reliable EEG-based driving fatigue detection.

Single-trial EEG emotion recognition using Granger Causality/Transfer Entropy analysis

BACKGROUND

Emotion recognition has been studied for decades, but the classification accuracy needs to be improved.

NEW METHOD

In this study, a novel emotional classification approach is proposed by combining the Histogram of Oriented Gradient (HOG) method with the Granger Causality (GC) or Transfer Entropy (TE) methods. HOG extracts local valid information from the GC/TE relationship matrices and then the Support Vector Machine (SVM) is employed to classify the emotional states of stress and calm.

RESULTS

Compared with previous studies, the classification accuracy has been greatly improved. The results of this study show that the classification based on GC or TE with HOG offers an average accuracy 88.93 % and 95.21 %, respectively. The achieved accuracy is about 12 % higher than that achieved without using HOG feature extraction.

COMPARISON WITH EXISTING METHOD(S)

Numerous efforts have been devoted to classify emotional states by extracting EEG characteristics on a single channel basis, the method developed in this study utilizes information interaction between brain channels as a feature to classify emotional states. Furthermore, this study combines HOG and relation matrices for the first time and uses image processing to extract EEG features.

CONCLUSION

Our results demonstrate the feasibility of combining TE with HOG for emotion recognition with improved classification accuracy by taking advantage of both network and gradient features. More specific features can be selected to improve classification accuracy by taking advantage of information exchanges between EEG channels directly or the extracted property of the relationship matrix based on information interactions.

Partial directed coherence based graph convolutional neural networks for driving fatigue detection

The mental state of a driver can be accurately and reliably evaluated by detecting the driver's electroencephalogram (EEG) signals. However, traditional machine learning and deep learning methods focus on the single electrode feature analysis and ignore the functional connection of the brain. In addition, the recent brain function connection network method needs to manually extract substantial brain network features, which results in cumbersome operation. For this reason, this paper introduces graph convolution combined with brain function connection theory into the study of mental fatigue and proposes a method for driving fatigue detection based on the partial directed coherence graph convolutional neural network (PDC-GCNN), which can analyze the characteristics of single electrodes while automatically extracting the topological features of the brain network. We designed a fatigue driving simulation experiment and collected the EEG signals. In the present work, the PDC method constructs the adjacency matrix to describe the relationship between EEG channels, and the GCNN combines single-electrode local brain area information and brain area connection information to further improve the performance of detecting fatigue states. Based on the features of differential entropy (DE) and power spectral density (PSD), the average recognition accuracy of ten-fold cross validation is 84.32% and 83.84%, respectively. For further experiments on each subject, the average recognition results are 95.24%/5.10% (PSD) and 96.01%/3.81% (DE). This research can be embedded in the vehicle driving fatigue detection system, which has practical application value.

Deep Learning in Physiological Signal Data: A Survey

Deep Learning (DL), a successful promising approach for discriminative and generative tasks, has recently proved its high potential in 2D medical imaging analysis; however, physiological data in the form of 1D signals have yet to be beneficially exploited from this novel approach to fulfil the desired medical tasks. Therefore, in this paper we survey the latest scientific research on deep learning in physiological signal data such as electromyogram (EMG), electrocardiogram (ECG), electroencephalogram (EEG), and electrooculogram (EOG). We found 147 papers published between January 2018 and October 2019 inclusive from various journals and publishers. The objective of this paper is to conduct a detailed study to comprehend, categorize, and compare the key parameters of the deep-learning approaches that have been used in physiological signal analysis for various medical applications. The key parameters of deep-learning approach that we review are the input data type, deep-learning task, deep-learning model, training architecture, and dataset sources. Those are the main key parameters that affect system performance. We taxonomize the research works using deep-learning method in physiological signal analysis based on: (1) physiological signal data perspective, such as data modality and medical application; and (2) deep-learning concept perspective such as training architecture and dataset sources.

Classification of Drowsiness Levels Based on a Deep Spatio-Temporal Convolutional Bidirectional LSTM Network Using Electroencephalography Signals

Non-invasive brain-computer interfaces (BCI) have been developed for recognizing human mental states with high accuracy and for decoding various types of mental conditions. In particular, accurately decoding a pilot's mental state is a critical issue as more than 70% of aviation accidents are caused by human factors, such as fatigue or drowsiness. In this study, we report the classification of not only two mental states (i.e., alert and drowsy states) but also five drowsiness levels from electroencephalogram (EEG) signals. To the best of our knowledge, this approach is the first to classify drowsiness levels in detail using only EEG signals. We acquired EEG data from ten pilots in a simulated night flight environment. For accurate detection, we proposed a deep spatio-temporal convolutional bidirectional long short-term memory network (DSTCLN) model. We evaluated the classification performance using Karolinska sleepiness scale (KSS) values for two mental states and five drowsiness levels. The grand-averaged classification accuracies were 0.87 (±0.01) and 0.69 (±0.02), respectively. Hence, we demonstrated the feasibility of classifying five drowsiness levels with high accuracy using deep learning.