1
|
Gu T, Yao W, Wang F, Fu R. Research on low-power driving fatigue monitoring method based on spiking neural network. Exp Brain Res 2024; 242:2457-2471. [PMID: 39177685 DOI: 10.1007/s00221-024-06911-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2024] [Accepted: 08/18/2024] [Indexed: 08/24/2024]
Abstract
Fatigue driving is one of the leading causes of traffic accidents, and the rapid and accurate detection of driver fatigue is of paramount importance for enhancing road safety. However, the application of deep learning models in fatigue driving detection has long been constrained by high computational costs and power consumption. To address this issue, this study proposes an approach that combines Self-Organizing Map (SOM) and Spiking Neural Networks (SNN) to develop a low-power model capable of accurately recognizing the driver's mental state. Initially, spatial features are extracted from electroencephalogram (EEG) signals using the SOM network. Subsequently, the extracted weight vectors are encoded and fed into the SNN for fatigue driving classification. The research results demonstrate that the proposed method effectively considers the spatiotemporal characteristics of EEG signals, achieving efficient fatigue detection. Simultaneously, this approach successfully reduces the model's power consumption. When compared to traditional artificial neural networks, our method reduces energy consumption by approximately 12.21-42.59%.
Collapse
Affiliation(s)
- Tianshu Gu
- School of Mechanic Engineering, Northeast Electric Power University, Jilin, 132012, China
| | - Wanchao Yao
- School of Mechanic Engineering, Northeast Electric Power University, Jilin, 132012, China
| | - Fuwang Wang
- School of Mechanic Engineering, Northeast Electric Power University, Jilin, 132012, China.
| | - Rongrong Fu
- College of Electrical Engineering, Yanshan University, Qinhuangdao, 066004, China
| |
Collapse
|
2
|
Imran MAA, Nasirzadeh F, Karmakar C. Designing a practical fatigue detection system: A review on recent developments and challenges. JOURNAL OF SAFETY RESEARCH 2024; 90:100-114. [PMID: 39251269 DOI: 10.1016/j.jsr.2024.05.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/08/2023] [Revised: 02/11/2024] [Accepted: 05/29/2024] [Indexed: 09/11/2024]
Abstract
INTRODUCTION Fatigue is considered to have a life-threatening effect on human health and it has been an active field of research in different sectors. Deploying wearable physiological sensors helps to detect the level of fatigue objectively without any concern of bias in subjective assessment and interfering with work. METHODS This paper provides an in-depth review of fatigue detection approaches using physiological signals to pinpoint their main achievements, identify research gaps, and recommend avenues for future research. The review results are presented under three headings, including: signal modality, experimental environments, and fatigue detection models. Fatigue detection studies are first divided based on signal modality into uni-modal and multi-modal approaches. Then, the experimental environments utilized for fatigue data collection are critically analyzed. At the end, the machine learning models used for the classification of fatigue state are reviewed. PRACTICAL APPLICATIONS The directions for future research are provided based on critical analysis of past studies. Finally, the challenges of objective fatigue detection in the real-world scenario are discussed.
Collapse
Affiliation(s)
- Md Abdullah Al Imran
- School of Information Technology, Faculty of Science Engineering & Built Environment, Deakin University, Australia.
| | - Farnad Nasirzadeh
- School of Architecture & Built Environment, Faculty of Science Engineering & Built Environment, Deakin University, Australia.
| | - Chandan Karmakar
- School of Information Technology, Faculty of Science Engineering & Built Environment, Deakin University, Australia.
| |
Collapse
|
3
|
Nadalizadeh F, Rajabioun M, Feyzi A. Driving fatigue detection based on brain source activity and ARMA model. Med Biol Eng Comput 2024; 62:1017-1030. [PMID: 38117429 DOI: 10.1007/s11517-023-02983-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2023] [Accepted: 11/28/2023] [Indexed: 12/21/2023]
Abstract
Fatigue among drivers is a significant issue in society, and according to organizational reports, it substantially contributes to accidents. So accurate fatigue detection in drivers plays a crucial role in reducing the number of people fatalities or injured resulting from accidents. Several methods are proposed for fatigue driver recognition among which electroencephalography (EEG) is one. This paper proposed a method for fatigue recognition by EEG signals with extracted features from source and sensor spaces. The proposed method starts with preprocessing by applying filtering and artifact rejection. Then source localization methods are applied to EEG signals for active source extraction. A multivariate autoregressive (MVAR) model is fitted to selected sources, and a dual Kalman filter is applied to estimate the source activity and their relationships. Then multivariate autoregressive moving average (ARMA) is fitted between EEG and source activity signals. Features are extracted from model parameters, source relationship matrix, and wavelet transform of EEG and source activity signals. The novelty of this approach is the use of ARMA model between source activities (as input) and EEG signals (as output) and feature extraction from source relations. Relevant features are selected using a combination of RelifF and neighborhood component analysis (NCA) methods. Three classifiers, namely k-nearest neighbor (KNN), support vector machine (SVM), and naive Bayesian (NB) classifiers, are employed to classify drivers. To improve performance, the final label for fatigue detection is calculated by combining these classifiers using the voting method. The results demonstrate that the proposed method accurately recognizes and classifies fatigued drivers with the ensemble classifiers in comparison with other methods.
Collapse
Affiliation(s)
- Fahimeh Nadalizadeh
- Faculty of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran
| | - Mehdi Rajabioun
- Department of Engineering, Mamaghan Branch, Islamic Azad University, Mamaghan, Iran.
| | - Amirreza Feyzi
- Department of Electrical and Computer Engineering, Tabriz University, Tabriz, Iran
| |
Collapse
|
4
|
Wang F, Chen D, Yao W, Fu R. Real driving environment EEG-based detection of driving fatigue using the wavelet scattering network. J Neurosci Methods 2023; 400:109983. [PMID: 37838152 DOI: 10.1016/j.jneumeth.2023.109983] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Revised: 09/29/2023] [Accepted: 10/11/2023] [Indexed: 10/16/2023]
Abstract
BACKGROUND Driving fatigue is one of the main factors leading to traffic accidents. So, it is necessary to detect driver fatigue accurately and quickly. NEW METHOD To precisely detect driving fatigue in a real driving environment, this paper adopts a classification method for driving fatigue based on the wavelet scattering network (WSN). Firstly, electroencephalogram (EEG) signals of 12 subjects in the real driving environment are collected and categorized into two states: fatigue and awake. Secondly, the WSN algorithm extracts wavelet scattering coefficients of EEG signals, and these coefficients are used as input in support vector machine (SVM) as feature vectors for classification. RESULTS The results showed that the average classification accuracy of 12 subjects reached 99.33%; the average precision rate reached 99.28%; the average recall rate reached 98.27%; the average F1 score reached 98.74%; and the average classification accuracy of the public data set SEED-VIG reached 99.39%. The average precision, recall rate and F1 score reached 99.27%, 98.41% and 98.83% respectively. COMPARISON WITH EXISTING METHODS In addition, the WSN algorithm is compared with traditional convolutional neural network (CNN), Sparse-deep belief networks (SDBN), Spatio-temporal convolutional neural networks (STCNN), Long short-term memory (LSTM), and other methods, and it is found that WSN has higher classification accuracy. CONCLUSION Furthermore, this method has good versatility, providing excellent recognition effect on small sample data sets, and fast running time, making it convenient for real-time online monitoring of driver fatigue. Therefore, the WSN algorithm is promising in efficiently detecting driving fatigue state of drivers in real environments, contributing to improved traffic safety.
Collapse
Affiliation(s)
- Fuwang Wang
- Northeast Electric Power University, School of Mechanic Engineering, Jilin 132012, China.
| | - Daping Chen
- Northeast Electric Power University, School of Mechanic Engineering, Jilin 132012, China
| | - Wanchao Yao
- Northeast Electric Power University, School of Mechanic Engineering, Jilin 132012, China
| | - Rongrong Fu
- Yanshan University, College of Electrical Engineering, Qinhuangdao 066004, China
| |
Collapse
|
5
|
Xu L, Li J, Feng D. Miner Fatigue Detection from Electroencephalogram-Based Relative Power Spectral Topography Using Convolutional Neural Network. SENSORS (BASEL, SWITZERLAND) 2023; 23:9055. [PMID: 38005443 PMCID: PMC10675395 DOI: 10.3390/s23229055] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/05/2023] [Revised: 10/04/2023] [Accepted: 10/24/2023] [Indexed: 11/26/2023]
Abstract
Fatigue of miners is caused by intensive workloads, long working hours, and shift-work schedules. It is one of the major factors increasing the risk of safety problems and work mistakes. Examining the detection of miner fatigue is important because it can potentially prevent work accidents and improve working efficiency in underground coal mines. Many previous studies have introduced feature-based machine-learning methods to estimate miner fatigue. This work proposes a method that uses electroencephalogram (EEG) signals to generate topographic maps containing frequency and spatial information. It utilizes a convolutional neural network (CNN) to classify the normal state, critical state, and fatigue state of miners. The topographic maps are generated from the EEG signals and contrasted using power spectral density (PSD) and relative power spectral density (RPSD). These two feature extraction methods were applied to feature recognition and four representative deep-learning methods. The results showthat RPSD achieves better performance than PSD in classification accuracy with all deep-learning methods. The CNN achieved superior results to the other deep-learning methods, with an accuracy of 94.5%, precision of 97.0%, sensitivity of 94.8%, and F1 score of 96.3%. Our results also show that the RPSD-CNN method outperforms the current state of the art. Thus, this method might be a useful and effective miner fatigue detection tool for coal companies in the near future.
Collapse
Affiliation(s)
- Lili Xu
- College of Economics and Management, Taiyuan University of Technology, Taiyuan 030024, China;
- College of Coal Engineering, Shanxi Datong University, Datong 037009, China
| | - Jizu Li
- College of Economics and Management, Taiyuan University of Technology, Taiyuan 030024, China;
| | - Ding Feng
- College of Data Science, Taiyuan University of Technology, Taiyuan 030024, China;
| |
Collapse
|
6
|
Hussein RM, Miften FS, George LE. Driver drowsiness detection methods using EEG signals: a systematic review. Comput Methods Biomech Biomed Engin 2023; 26:1237-1249. [PMID: 35983784 DOI: 10.1080/10255842.2022.2112574] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2022] [Revised: 07/21/2022] [Accepted: 08/08/2022] [Indexed: 11/03/2022]
Abstract
Electroencephalography (EEG) is a complex signal that may require several years of training, advanced signal processing, and feature extraction methodologies to interpret correctly. Recently, many methods have been used to extract and classify EEG data. This study reviews 62 papers that used EEG signals to detect driver drowsiness, published between January 2018 and 2022. We extract trends and highlight interesting approaches from this large body of literature to inform future research and formulate recommendations. To find relevant papers published in scientific journals, conferences, and electronic preprint repositories, researchers searched major databases covering the domains of science and engineering. For each investigation, many data items about (1) the data, (2) the channels used, (3) the extraction and classification procedure, and (4) the outcomes were extracted. These items were then analyzed one by one to uncover trends. Our analysis reveals that the amount of EEG data used across studies varies. We saw that more than half the studies used simulation driving experimental. About 21% of the studies used support vector machine (SVM), while 19% used convolutional neural networks (CNN). Overall, we can conclude that drowsiness and fatigue impair driving performance, resulting in drivers who are more exposed to risky situations.
Collapse
Affiliation(s)
- Raed Mohammed Hussein
- Iraqi Commission for Computers and Informatics, Informatics Institute of Postgraduate Studies, Baghdad, Iraq
| | - Firas Sabar Miften
- College of Education for Pure Science, University of Thi-Qar, Nasiriyah, Iraq
| | - Loay E George
- University of Information Technology & Communication, Baghdad, Iraq
| |
Collapse
|
7
|
Kargarnovin S, Hernandez C, Farahani FV, Karwowski W. Evidence of Chaos in Electroencephalogram Signatures of Human Performance: A Systematic Review. Brain Sci 2023; 13:813. [PMID: 37239285 PMCID: PMC10216576 DOI: 10.3390/brainsci13050813] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2023] [Revised: 05/09/2023] [Accepted: 05/15/2023] [Indexed: 05/28/2023] Open
Abstract
(1) Background: Chaos, a feature of nonlinear dynamical systems, is well suited for exploring biological time series, such as heart rates, respiratory records, and particularly electroencephalograms. The primary purpose of this article is to review recent studies using chaos theory and nonlinear dynamical methods to analyze human performance in different brain processes. (2) Methods: Several studies have examined chaos theory and related analytical tools for describing brain dynamics. The present study provides an in-depth analysis of the computational methods that have been proposed to uncover brain dynamics. (3) Results: The evidence from 55 articles suggests that cognitive function is more frequently assessed than other brain functions in studies using chaos theory. The most frequently used techniques for analyzing chaos include the correlation dimension and fractal analysis. Approximate, Kolmogorov and sample entropy account for the largest proportion of entropy algorithms in the reviewed studies. (4) Conclusions: This review provides insights into the notion of the brain as a chaotic system and the successful use of nonlinear methods in neuroscience studies. Additional studies of brain dynamics would aid in improving our understanding of human cognitive performance.
Collapse
Affiliation(s)
- Shaida Kargarnovin
- Computational Neuroergonomics Laboratory, Department of Industrial Engineering and Management Systems, University of Central Florida, Orlando, FL 32816, USA; (C.H.); (F.V.F.); (W.K.)
| | - Christopher Hernandez
- Computational Neuroergonomics Laboratory, Department of Industrial Engineering and Management Systems, University of Central Florida, Orlando, FL 32816, USA; (C.H.); (F.V.F.); (W.K.)
| | - Farzad V. Farahani
- Computational Neuroergonomics Laboratory, Department of Industrial Engineering and Management Systems, University of Central Florida, Orlando, FL 32816, USA; (C.H.); (F.V.F.); (W.K.)
- Department of Biostatistics, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Waldemar Karwowski
- Computational Neuroergonomics Laboratory, Department of Industrial Engineering and Management Systems, University of Central Florida, Orlando, FL 32816, USA; (C.H.); (F.V.F.); (W.K.)
| |
Collapse
|
8
|
Vehicle Driving Risk Prediction Model by Reverse Artificial Intelligence Neural Network. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:3100509. [PMID: 36248936 PMCID: PMC9568302 DOI: 10.1155/2022/3100509] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/29/2022] [Revised: 08/30/2022] [Accepted: 09/03/2022] [Indexed: 11/17/2022]
Abstract
The popularity of private cars has brought great convenience to citizens’ travel. However, the number of private cars in society is increasing yearly, and the traffic pressure on the road is also increasing. The number of traffic accidents is increasing yearly, and the vast majority are caused by small private cars. Therefore, it is necessary to improve the traffic safety awareness of drivers and help car manufacturers to design traffic risk prediction systems. The Backpropagation neural network (BPNN) algorithm is used as the technical basis, combined with the MATLAB operation program, to simulate the driving process of the car. Dynamic predictive models are built to predict and analyze vehicle safety risks. Multiple experiments found that: (1) in various simulations, the simulation driving process of MATLAB is more in line with the actual car driving process; (2) the error between BPNN and the actual driving prediction is within 0.4, which can meet the actual needs. Predictive models are optimized to deploy and predict in various traffic situations. The model can effectively prompt risk accidents, reduce the probability of traffic accidents, provide a certain degree of protection for the lives of drivers and passengers, and significantly improve the safety of traffic roads.
Collapse
|
9
|
Cai Q, An JP, Li HY, Guo JY, Gao ZK. Cross-subject emotion recognition using visibility graph and genetic algorithm-based convolution neural network. CHAOS (WOODBURY, N.Y.) 2022; 32:093110. [PMID: 36182360 DOI: 10.1063/5.0098454] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/09/2022] [Accepted: 08/01/2022] [Indexed: 06/16/2023]
Abstract
An efficient emotion recognition model is an important research branch in electroencephalogram (EEG)-based brain-computer interfaces. However, the input of the emotion recognition model is often a whole set of EEG channels obtained by electrodes placed on subjects. The unnecessary information produced by redundant channels affects the recognition rate and depletes computing resources, thereby hindering the practical applications of emotion recognition. In this work, we aim to optimize the input of EEG channels using a visibility graph (VG) and genetic algorithm-based convolutional neural network (GA-CNN). First, we design an experiment to evoke three types of emotion states using movies and collect the multi-channel EEG signals of each subject under different emotion states. Then, we construct VGs for each EEG channel and derive nonlinear features representing each EEG channel. We employ the genetic algorithm (GA) to find the optimal subset of EEG channels for emotion recognition and use the recognition results of the CNN as fitness values. The experimental results show that the recognition performance of the proposed method using a subset of EEG channels is superior to that of the CNN using all channels for each subject. Last, based on the subset of EEG channels searched by the GA-CNN, we perform cross-subject emotion recognition tasks employing leave-one-subject-out cross-validation. These results demonstrate the effectiveness of the proposed method in recognizing emotion states using fewer EEG channels and further enrich the methods of EEG classification using nonlinear features.
Collapse
Affiliation(s)
- Qing Cai
- School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
| | - Jian-Peng An
- School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
| | - Hao-Yu Li
- School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
| | - Jia-Yi Guo
- School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
| | - Zhong-Ke Gao
- School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
| |
Collapse
|
10
|
Automatic Detection of Driver Fatigue Based on EEG Signals Using a Developed Deep Neural Network. ELECTRONICS 2022. [DOI: 10.3390/electronics11142169] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Abstract
In recent years, detecting driver fatigue has been a significant practical necessity and issue. Even though several investigations have been undertaken to examine driver fatigue, there are relatively few standard datasets on identifying driver fatigue. For earlier investigations, conventional methods relying on manual characteristics were utilized to assess driver fatigue. In any case study, such approaches need previous information for feature extraction, which could raise computing complexity. The current work proposes a driver fatigue detection system, which is a fundamental necessity to minimize road accidents. Data from 11 people are gathered for this purpose, resulting in a comprehensive dataset. The dataset is prepared in accordance with previously published criteria. A deep convolutional neural network–long short-time memory (CNN–LSTM) network is conceived and evolved to extract characteristics from raw EEG data corresponding to the six active areas A, B, C, D, E (based on a single channel), and F. The study’s findings reveal that the suggested deep CNN–LSTM network could learn features hierarchically from raw EEG data and attain a greater precision rate than previous comparative approaches for two-stage driver fatigue categorization. The suggested approach may be utilized to construct automatic fatigue detection systems because of their precision and high speed.
Collapse
|
11
|
Motion Fatigue State Detection Based on Neural Networks. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:9602631. [PMID: 35330594 PMCID: PMC8940542 DOI: 10.1155/2022/9602631] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/12/2022] [Revised: 02/11/2022] [Accepted: 02/17/2022] [Indexed: 11/29/2022]
Abstract
Aiming at the problem of fatigue state detection at the back of sports, a cascade deep learning detection system structure is designed, and a convolutional neural network fatigue state detection model based on multiscale pooling is proposed. Firstly, face detection is carried out by a deep learning model MTCNN to extract eye and mouth regions. Aiming at the problem of eye and mouth state representation and recognition, a multiscale pooling model (MSP) based on RESNET is proposed to train the eye and mouth state. In real-time detection, the state of the eye and mouth region is recognized through the trained convolution neural network model. Finally, the athlete's fatigue is determined based on PERCLOS and the proposed mouth opening and closing frequency (FOM). The experimental results show that in the training process, we set the batch_size = 100 and the initial learning rate = 0.01. When the evaluation index is no longer improved, the learning rate is reduced by 10 times to 0.001, and a total of 50 epochs are trained. The precision and recall of the system are high. Compared with the infrared image simulating the night state, the RGB image taken by the ordinary camera in the daytime has higher precision and recall. It is proven that the neural network has high detection accuracy, meets the real-time requirements, and has high robustness in complex environments.
Collapse
|
12
|
Developing a Deep Neural Network for Driver Fatigue Detection Using EEG Signals Based on Compressed Sensing. SUSTAINABILITY 2022. [DOI: 10.3390/su14052941] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
In recent years, driver fatigue has become one of the main causes of road accidents. As a result, fatigue detection systems have been developed to warn drivers, and, among the available methods, EEG signal analysis is recognized as the most reliable method for detecting driver fatigue. This study presents an automated system for a two-stage classification of driver fatigue, using a combination of compressed sensing (CS) theory and deep neural networks (DNNs), that is based on EEG signals. First, CS theory is used to compress the recorded EEG data in order to reduce the computational load. Then, the compressed EEG data is fed into the proposed deep convolutional neural network for automatic feature extraction/selection and classification purposes. The proposed network architecture includes seven convolutional layers together with three long short-term memory (LSTM) layers. For compression rates of 40, 50, 60, 70, 80, and 90, the simulation results for a single-channel recording show accuracies of 95, 94.8, 94.6, 94.4, 94.4, and 92%, respectively. Furthermore, by comparing the results to previous methods, the accuracy of the proposed method for the two-stage classification of driver fatigue has been improved and can be used to effectively detect driver fatigue.
Collapse
|
13
|
Ye C, Yin Z, Zhao M, Tian Y, Sun Z. Identification of mental fatigue levels in a language understanding task based on multi-domain EEG features and an ensemble convolutional neural network. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103360] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
14
|
Wriessnegger SC, Raggam P, Kostoglou K, Müller-Putz GR. Mental State Detection Using Riemannian Geometry on Electroencephalogram Brain Signals. Front Hum Neurosci 2021; 15:746081. [PMID: 34899215 PMCID: PMC8663761 DOI: 10.3389/fnhum.2021.746081] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2021] [Accepted: 10/12/2021] [Indexed: 11/16/2022] Open
Abstract
The goal of this study was to implement a Riemannian geometry (RG)-based algorithm to detect high mental workload (MWL) and mental fatigue (MF) using task-induced electroencephalogram (EEG) signals. In order to elicit high MWL and MF, the participants performed a cognitively demanding task in the form of the letter n-back task. We analyzed the time-varying characteristics of the EEG band power (BP) features in the theta and alpha frequency band at different task conditions and cortical areas by employing a RG-based framework. MWL and MF were considered as too high, when the Riemannian distances of the task-run EEG reached or surpassed the threshold of the baseline EEG. The results of this study showed a BP increase in the theta and alpha frequency bands with increasing experiment duration, indicating elevated MWL and MF that impedes/hinders the task performance of the participants. High MWL and MF was detected in 8 out of 20 participants. The Riemannian distances also showed a steady increase toward the threshold with increasing experiment duration, with the most detections occurring toward the end of the experiment. To support our findings, subjective ratings (questionnaires concerning fatigue and workload levels) and behavioral measures (performance accuracies and response times) were also considered.
Collapse
Affiliation(s)
- Selina C Wriessnegger
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria.,BioTechMed-Graz, Graz, Austria
| | - Philipp Raggam
- Research Group Neuroinformatics, Faculty of Computer Science, University of Vienna, Vienna, Austria.,Department of Neurology and Stroke, Hertie Institute for Clinical Brain Research, University of Tübingen, Tübingen, Germany
| | - Kyriaki Kostoglou
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria
| | - Gernot R Müller-Putz
- Institute of Neural Engineering, Graz University of Technology, Graz, Austria.,BioTechMed-Graz, Graz, Austria
| |
Collapse
|
15
|
A Novel Fatigue Driving State Recognition and Warning Method Based on EEG and EOG Signals. JOURNAL OF HEALTHCARE ENGINEERING 2021; 2021:7799793. [PMID: 34853672 PMCID: PMC8629631 DOI: 10.1155/2021/7799793] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/16/2021] [Revised: 10/29/2021] [Accepted: 11/05/2021] [Indexed: 11/17/2022]
Abstract
Traffic accidents are easily caused by tired driving. If the fatigue state of the driver can be identified in time and a corresponding early warning can be provided, then the occurrence of traffic accidents could be avoided to a large extent. At present, the recognition of fatigue driving states is mostly based on recognition accuracy. Fatigue state is currently recognized by combining different features, such as facial expressions, electroencephalogram (EEG) signals, yawning, and the percentage of eyelid closure over the pupil over time (PERCLoS). The combination of these features increases the recognition time and lacks real-time performance. In addition, some features will increase error in the recognition result, such as yawning frequently with the onset of a cold or frequent blinking with dry eyes. On the premise of ensuring the recognition accuracy and improving the realistic feasibility and real-time recognition performance of fatigue driving states, a fast support vector machine (FSVM) algorithm based on EEGs and electrooculograms (EOGs) is proposed to recognize fatigue driving states. First, the collected EEG and EOG modal data are preprocessed. Second, multiple features are extracted from the preprocessed EEGs and EOGs. Finally, FSVM is used to classify and recognize the data features to obtain the recognition result of the fatigue state. Based on the recognition results, this paper designs a fatigue driving early warning system based on Internet of Things (IoT) technology. When the driver shows symptoms of fatigue, the system not only sends a warning signal to the driver but also informs other nearby vehicles using this system through IoT technology and manages the operation background.
Collapse
|
16
|
Evaluation of a Fast Test Based on Biometric Signals to Assess Mental Fatigue at the Workplace-A Pilot Study. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph182211891. [PMID: 34831645 PMCID: PMC8621458 DOI: 10.3390/ijerph182211891] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 09/05/2021] [Revised: 10/19/2021] [Accepted: 11/02/2021] [Indexed: 01/29/2023]
Abstract
Non-pathological mental fatigue is a recurring, but undesirable condition among people in the fields of office work, industry, and education. This type of mental fatigue can often lead to negative outcomes, such as performance reduction and cognitive impairment in education; loss of focus and burnout syndrome in office work; and accidents leading to injuries or death in the transportation and manufacturing industries. Reliable mental fatigue assessment tools are promising in the improvement of performance, mental health and safety of students and workers, and at the same time, in the reduction of risks, accidents and the associated economic loss (e.g., medical fees and equipment reparations). The analysis of biometric (brain, cardiac, skin conductance) signals has proven to be effective in discerning different stages of mental fatigue; however, many of the reported studies in the literature involve the use of long fatigue-inducing tests and subject-specific models in their methodologies. Recent trends in the modeling of mental fatigue suggest the usage of non subject-specific (general) classifiers and a time reduction of calibration procedures and experimental setups. In this study, the evaluation of a fast and short-calibration mental fatigue assessment tool based on biometric signals and inter-subject modeling, using multiple linear regression, is presented. The proposed tool does not require fatigue-inducing tests, which allows fast setup and implementation. Electroencephalography, photopletismography, electrodermal activity, and skin temperature from 17 subjects were recorded, using an OpenBCI helmet and an Empatica E4 wristband. Correlations to self-reported mental fatigue levels (using the fatigue assessment scale) were calculated to find the best mental fatigue predictors. Three-class mental fatigue models were evaluated, and the best model obtained an accuracy of 88% using three features, β/θ (C3), and the α/θ (O2 and C3) ratios, from one minute of electroencephalography measurements. The results from this pilot study show the feasibility and potential of short-calibration procedures and inter-subject classifiers in mental fatigue modeling, and will contribute to the use of wearable devices for the development of tools oriented to the well-being of workers and students, and also in daily living activities.
Collapse
|
17
|
Putty MS. A Whirlwind Tour of Complex Systems. J Indian Inst Sci 2021; 101:297-302. [PMID: 34642556 PMCID: PMC8496620 DOI: 10.1007/s41745-021-00264-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/19/2021] [Indexed: 11/29/2022]
|
18
|
Quddus A, Shahidi Zandi A, Prest L, Comeau FJE. Using long short term memory and convolutional neural networks for driver drowsiness detection. ACCIDENT; ANALYSIS AND PREVENTION 2021; 156:106107. [PMID: 33848710 DOI: 10.1016/j.aap.2021.106107] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/23/2019] [Revised: 07/19/2020] [Accepted: 03/27/2021] [Indexed: 06/12/2023]
Abstract
Fatigue negatively affects the safety and performance of drivers on the road. In fact, drowsiness and fatigue are the cause of a substantial number of motor vehicle accidents. Drowsiness among the drivers can be detected using variety of modalities, including electroencephalogram (EEG), eye movement, and vehicle driving dynamics. Among these EEG is highly accurate but very intrusive and cumbersome. On the other hand, vehicle driving dynamics are very easy to acquire but accuracy is not very high. Eye movement based approach is very attractive in terms of balance between these two extremes. However, eye movement based techniques normally require an eye tracking device which consists of high speed camera with sophisticated algorithm to extract eye movement related parameters such as blinking, eye closure, saccades, fixation etc. This makes eye tracking based drowsiness detection difficult to implement as a practical system, especially on an embedded platform. In this paper, authors propose to use eye images from camera directly without the need for expensive eye-tracking system. Here, eye related movements are captured by Recurrent Neural Network (RNN) to detect the drowsiness. Long Short Term Memory (LSTM) is a class of RNN which has several advantages over vanilla RNNs. In this work an array of LSTM cells are utilized to model the eye movements. Two types of LSTMs were employed: 1-D LSTM (R-LSTM) which is used as baseline and the convolutional LSTM (C-LSTM) which facilitates using 2-D images directly. Patches of size 48 × 48 around each eye were extracted from 38 subjects, participating in a simulated driving experiment. The state of vigilance among the subjects were independently assessed by power spectral analysis of multichannel electroencephalogram (EEG) signals, recorded simultaneously, and binary labels of alert and drowsy (baseline) were generated. Results show high efficacy of the proposed system. R-LSTM based approach resulted in accuracy around 82 % and C-LSTM based approach resulted in accuracy in the range of 95%-97%. Comparison is also provided with a recently published eye-tracking based approach, showing the proposed LSTM technique outperform with a wide margin.
Collapse
Affiliation(s)
| | - Ali Shahidi Zandi
- Alcohol Countermeasure Systems Corp. (ACS), 60 International Boulevard, Toronto, ON, Canada.
| | - Laura Prest
- Alcohol Countermeasure Systems Corp. (ACS), 60 International Boulevard, Toronto, ON, Canada.
| | - Felix J E Comeau
- Alcohol Countermeasure Systems Corp. (ACS), 60 International Boulevard, Toronto, ON, Canada.
| |
Collapse
|
19
|
Haghani M, Bliemer MCJ, Farooq B, Kim I, Li Z, Oh C, Shahhoseini Z, MacDougall H. Applications of brain imaging methods in driving behaviour research. ACCIDENT; ANALYSIS AND PREVENTION 2021; 154:106093. [PMID: 33770719 DOI: 10.1016/j.aap.2021.106093] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/21/2020] [Revised: 01/14/2021] [Accepted: 03/15/2021] [Indexed: 06/12/2023]
Abstract
Applications of neuroimaging methods have substantially contributed to the scientific understanding of human factors during driving by providing a deeper insight into the neuro-cognitive aspects of driver brain. This has been achieved by conducting simulated (and occasionally, field) driving experiments while collecting driver brain signals of various types. Here, this sector of studies is comprehensively reviewed at both macro and micro scales. At the macro scale, bibliometric aspects of these studies are analysed. At the micro scale, different themes of neuroimaging driving behaviour research are identified and the findings within each theme are synthesised. The surveyed literature has reported on applications of four major brain imaging methods. These include Functional Magnetic Resonance Imaging (fMRI), Electroencephalography (EEG), Functional Near-Infrared Spectroscopy (fNIRS) and Magnetoencephalography (MEG), with the first two being the most common methods in this domain. While collecting driver fMRI signal has been particularly instrumental in studying neural correlates of intoxicated driving (e.g. alcohol or cannabis) or distracted driving, the EEG method has been predominantly utilised in relation to the efforts aiming at development of automatic fatigue/drowsiness detection systems, a topic to which the literature on neuro-ergonomics of driving particularly has shown a spike of interest within the last few years. The survey also reveals that topics such as driver brain activity in semi-automated settings or neural activity of drivers with brain injuries or chronic neurological conditions have by contrast been investigated to a very limited extent. Potential topics in driving behaviour research are identified that could benefit from the adoption of neuroimaging methods in future studies. In terms of practicality, while fMRI and MEG experiments have proven rather invasive and technologically challenging for adoption in driving behaviour research, EEG and fNIRS applications have been more diverse. They have even been tested beyond simulated driving settings, in field driving experiments. Advantages and limitations of each of these four neuroimaging methods in the context of driving behaviour experiments are outlined in the paper.
Collapse
Affiliation(s)
- Milad Haghani
- Institute of Transport and Logistics Studies, The University of Sydney Business School, The University of Sydney, NSW, Australia; Centre for Spatial Data Infrastructure and Land Administration (CSDILA), School of Electrical, Mechanical and Infrastructure Engineering, The University of Melbourne, Australia.
| | - Michiel C J Bliemer
- Institute of Transport and Logistics Studies, The University of Sydney Business School, The University of Sydney, NSW, Australia
| | - Bilal Farooq
- Laboratory of Innovations in Transportation, Ryerson University, Toronto, Canada
| | - Inhi Kim
- Institute of Transport Studies, Department of Civil Engineering, Monash University, VIC, Australia; Department of Civil and Environmental Engineering, Kongju National University, Cheonan, Republic of Korea
| | - Zhibin Li
- School of Transportation, Southeast University, Nanjing, China
| | - Cheol Oh
- Department of Transportation and Logistics Engineering, Hanyang University, Republic of Korea
| | | | - Hamish MacDougall
- School of Psychology, Faculty of Science, The University of Sydney, Sydney, Australia
| |
Collapse
|
20
|
Wang H, Xu L, Bezerianos A, Chen C, Zhang Z. Linking Attention-Based Multiscale CNN With Dynamical GCN for Driving Fatigue Detection. IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 2021; 70:1-11. [PMID: 0 DOI: 10.1109/tim.2020.3047502] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/24/2023]
|
21
|
Kachhara S, Ambika G. Multiplex recurrence networks from multi-lead ECG data. CHAOS (WOODBURY, N.Y.) 2020; 30:123106. [PMID: 33380014 DOI: 10.1063/5.0026954] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/26/2020] [Accepted: 11/06/2020] [Indexed: 06/12/2023]
Abstract
We present an integrated approach to analyze the multi-lead electrocardiogram (ECG) data using the framework of multiplex recurrence networks (MRNs). We explore how their intralayer and interlayer topological features can capture the subtle variations in the recurrence patterns of the underlying spatio-temporal dynamics of the cardiac system. We find that MRNs from ECG data of healthy cases are significantly more coherent with high mutual information and less divergence between respective degree distributions. In cases of diseases, significant differences in specific measures of similarity between layers are seen. The coherence is affected most in the cases of diseases associated with localized abnormality such as bundle branch block. We note that it is important to do a comprehensive analysis using all the measures to arrive at disease-specific patterns. Our approach is very general and as such can be applied in any other domain where multivariate or multi-channel data are available from highly complex systems.
Collapse
Affiliation(s)
- Sneha Kachhara
- Indian Institute of Science Education and Research (IISER) Tirupati, Tirupati 517507, India
| | - G Ambika
- Indian Institute of Science Education and Research (IISER) Tirupati, Tirupati 517507, India
| |
Collapse
|
22
|
Abstract
Background and objective: Driving a car is a complex activity which involves movements of the whole body. Many studies on drivers’ behavior are conducted to improve road traffic safety. Such studies involve the registration and processing of multiple signals, such as electroencephalography (EEG), electrooculography (EOG) and the images of the driver’s face. In our research, we attempt to develop a classifier of scenarios related to learning to drive based on the data obtained in real road traffic conditions via smart glasses. In our approach, we try to minimize the number of signals which can be used to recognize the activities performed while driving a car. Material and methods: We attempt to evaluate the drivers’ activities using both electrooculography (EOG) and a deep learning approach. To acquire data we used JINS MEME smart glasses furnished with 3-point EOG electrodes, 3-axial accelerometer and 3-axial gyroscope. Sensor data were acquired on 20 drivers (ten experienced and ten learner drivers) on the same 28.7 km route under real road conditions in southern Poland. The drivers performed several tasks while wearing the smart glasses and the tasks were linked to the signal during the drive. For the recognition of four activities (parking, driving through a roundabout, city traffic and driving through an intersection), we used one-dimensional convolutional neural network (1D CNN). Results: The maximum accuracy was 95.6% on validation set and 99.8% on training set. The results prove that the model based on 1D CNN can classify the actions performed by drivers accurately. Conclusions: We have proved the feasibility of recognizing drivers’ activity based solely on EOG data, regardless of the driving experience and style. Our findings may be useful in the objective assessment of driving skills and thus, improving driving safety.
Collapse
|
23
|
Zhang W, Wang F, Wu S, Xu Z, Ping J, Jiang Y. Partial directed coherence based graph convolutional neural networks for driving fatigue detection. THE REVIEW OF SCIENTIFIC INSTRUMENTS 2020; 91:074713. [PMID: 32752838 DOI: 10.1063/5.0008434] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/22/2020] [Accepted: 07/05/2020] [Indexed: 05/18/2023]
Abstract
The mental state of a driver can be accurately and reliably evaluated by detecting the driver's electroencephalogram (EEG) signals. However, traditional machine learning and deep learning methods focus on the single electrode feature analysis and ignore the functional connection of the brain. In addition, the recent brain function connection network method needs to manually extract substantial brain network features, which results in cumbersome operation. For this reason, this paper introduces graph convolution combined with brain function connection theory into the study of mental fatigue and proposes a method for driving fatigue detection based on the partial directed coherence graph convolutional neural network (PDC-GCNN), which can analyze the characteristics of single electrodes while automatically extracting the topological features of the brain network. We designed a fatigue driving simulation experiment and collected the EEG signals. In the present work, the PDC method constructs the adjacency matrix to describe the relationship between EEG channels, and the GCNN combines single-electrode local brain area information and brain area connection information to further improve the performance of detecting fatigue states. Based on the features of differential entropy (DE) and power spectral density (PSD), the average recognition accuracy of ten-fold cross validation is 84.32% and 83.84%, respectively. For further experiments on each subject, the average recognition results are 95.24%/5.10% (PSD) and 96.01%/3.81% (DE). This research can be embedded in the vehicle driving fatigue detection system, which has practical application value.
Collapse
Affiliation(s)
- Weiwei Zhang
- Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China
| | - Fei Wang
- Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China
| | - Shichao Wu
- Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China
| | - Zongfeng Xu
- College of Information Science and Engineering, Northeastern University, Shenyang, China
| | - Jingyu Ping
- Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China
| | - Yang Jiang
- Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China
| |
Collapse
|
24
|
Tang Y, Kurths J, Lin W, Ott E, Kocarev L. Introduction to Focus Issue: When machine learning meets complex systems: Networks, chaos, and nonlinear dynamics. CHAOS (WOODBURY, N.Y.) 2020; 30:063151. [PMID: 32611112 DOI: 10.1063/5.0016505] [Citation(s) in RCA: 27] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/04/2020] [Accepted: 06/05/2020] [Indexed: 06/11/2023]
Affiliation(s)
- Yang Tang
- Key Laboratory of Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology, Shanghai, China
| | - Jürgen Kurths
- Potsdam Institute for Climate Impact Research, Potsdam 14473, Germany
| | - Wei Lin
- Center for Computational Systems Biology of ISTBI and Research Institute of Intelligent Complex Systems, Fudan University, Shanghai 200433, China
| | - Edward Ott
- Department of Physics, University of Maryland, College Park, Maryland 20742, USA
| | - Ljupco Kocarev
- Macedonian Academy of Sciences and Arts, 1000 Skopje, Macedonia
| |
Collapse
|