Detection of driver drowsiness level using a hybrid learning model based on ECG signals

OBJECTIVES

Fatigue has a considerable impact on the driver's vehicle and even the driver's own operating ability.

METHODS

An intelligent algorithm is proposed for the problem that it is difficult to classify the degree of drowsiness generated by the driver during the driving process. By studying the driver's electrocardiogram (ECG) during driving, two models were established to jointly classify the ECG signals as awake, stress, and fatigue or drowsiness states for drowsiness levels. Firstly, the deep learning method was used to establish the model_1 to predict the drowsiness of the original ECG, and model_2 was developed using the combination of principal component analysis (PCA) and weighted K-nearest neighbor (WKNN) algorithm to classify the heart rate variability characteristics. Then, the drowsiness prediction results of the two models were weighted according to certain rules, and the hybrid learning model combining dilated convolution and bidirectional long short-term memory network with PCA and WKNN algorithm was established, and the mixed model was denoted as DiCNN-BiLSTM and PCA-WKNN (DBPW). Finally, the validity of the DBPW model was verified by simulation of the public database.

RESULTS

The experimental results show that the average accuracy, sensitivity and F1 score of the test model in the dataset containing multiple drivers are 98.79, 98.81, and 98.79 % respectively, and the recognition accuracy for drowsiness or drowsiness state is 99.33 %.

CONCLUSIONS

Using the proposed algorithm, it is possible to identify driver anomalies and provide new ideas for the development of intelligent vehicles.

Detection of Drowsiness among Drivers Using Novel Deep Convolutional Neural Network Model

Detecting drowsiness among drivers is critical for ensuring road safety and preventing accidents caused by drowsy or fatigued driving. Research on yawn detection among drivers has great significance in improving traffic safety. Although various studies have taken place where deep learning-based approaches are being proposed, there is still room for improvement to develop better and more accurate drowsiness detection systems using behavioral features such as mouth and eye movement. This study proposes a deep neural network architecture for drowsiness detection employing a convolutional neural network (CNN) for driver drowsiness detection. Experiments involve using the DLIB library to locate key facial points to calculate the mouth aspect ratio (MAR). To compensate for the small dataset, data augmentation is performed for the 'yawning' and 'no_yawning' classes. Models are trained and tested involving the original and augmented dataset to analyze the impact on model performance. Experimental results demonstrate that the proposed CNN model achieves an average accuracy of 96.69%. Performance comparison with existing state-of-the-art approaches shows better performance of the proposed model.

Real-Time Deep Learning-Based Drowsiness Detection: Leveraging Computer-Vision and Eye-Blink Analyses for Enhanced Road Safety

Drowsy driving can significantly affect driving performance and overall road safety. Statistically, the main causes are decreased alertness and attention of the drivers. The combination of deep learning and computer-vision algorithm applications has been proven to be one of the most effective approaches for the detection of drowsiness. Robust and accurate drowsiness detection systems can be developed by leveraging deep learning to learn complex coordinate patterns using visual data. Deep learning algorithms have emerged as powerful techniques for drowsiness detection because of their ability to learn automatically from given inputs and feature extractions from raw data. Eye-blinking-based drowsiness detection was applied in this study, which utilized the analysis of eye-blink patterns. In this study, we used custom data for model training and experimental results were obtained for different candidates. The blinking of the eye and mouth region coordinates were obtained by applying landmarks. The rate of eye-blinking and changes in the shape of the mouth were analyzed using computer-vision techniques by measuring eye landmarks with real-time fluctuation representations. An experimental analysis was performed in real time and the results proved the existence of a correlation between yawning and closed eyes, classified as drowsy. The overall performance of the drowsiness detection model was 95.8% accuracy for drowsy-eye detection, 97% for open-eye detection, 0.84% for yawning detection, 0.98% for right-sided falling, and 100% for left-sided falling. Furthermore, the proposed method allowed a real-time eye rate analysis, where the threshold served as a separator of the eye into two classes, the "Open" and "Closed" states.

Driver Drowsiness Detection: A Machine Learning Approach on Skin Conductance

The majority of car accidents worldwide are caused by drowsy drivers. Therefore, it is important to be able to detect when a driver is starting to feel drowsy in order to warn them before a serious accident occurs. Sometimes, drivers are not aware of their own drowsiness, but changes in their body signals can indicate that they are getting tired. Previous studies have used large and intrusive sensor systems that can be worn by the driver or placed in the vehicle to collect information about the driver's physical status from a variety of signals that are either physiological or vehicle-related. This study focuses on the use of a single wrist device that is comfortable for the driver to wear and appropriate signal processing to detect drowsiness by analyzing only the physiological skin conductance (SC) signal. To determine whether the driver is drowsy, the study tests three ensemble algorithms and finds that the Boosting algorithm is the most effective in detecting drowsiness with an accuracy of 89.4%. The results of this study show that it is possible to identify when a driver is drowsy using only signals from the skin on the wrist, and this encourages further research to develop a real-time warning system for early detection of drowsiness.

A CNN-Based Wearable System for Driver Drowsiness Detection

Drowsiness poses a serious challenge to road safety and various in-cabin sensing technologies have been experimented with to monitor driver alertness. Cameras offer a convenient means for contactless sensing, but they may violate user privacy and require complex algorithms to accommodate user (e.g., sunglasses) and environmental (e.g., lighting conditions) constraints. This paper presents a lightweight convolution neural network that measures eye closure based on eye images captured by a wearable glass prototype, which features a hot mirror-based design that allows the camera to be installed on the glass temples. The experimental results showed that the wearable glass prototype, with the neural network in its core, was highly effective in detecting eye blinks. The blink rate derived from the glass output was highly consistent with an industrial gold standard EyeLink eye-tracker. As eye blink characteristics are sensitive measures of driver drowsiness, the glass prototype and the lightweight neural network presented in this paper would provide a computationally efficient yet viable solution for real-world applications.

An Electro-Oculogram (EOG) Sensor's Ability to Detect Driver Hypovigilance Using Machine Learning

Driving safely is crucial to avoid death, injuries, or financial losses that can be sustained in an accident. Thus, a driver's physical state should be monitored to prevent accidents, rather than vehicle-based or behavioral measurements, and provide reliable information in this regard. Electrocardiography (ECG), electroencephalography (EEG), electrooculography (EOG), and surface electromyography (sEMG) signals are used to monitor a driver's physical state during a drive. The purpose of this study was to detect driver hypovigilance (drowsiness, fatigue, as well as visual and cognitive inattention) using signals collected from 10 drivers while they were driving. EOG signals from the driver were preprocessed to remove noise, and 17 features were extracted. ANOVA (analysis of variance) was used to select statistically significant features that were then loaded into a machine learning algorithm. We then reduced the features by using principal component analysis (PCA) and trained three classifiers: support vector machine (SVM), k-nearest neighbor (KNN), and ensemble. A maximum accuracy of 98.7% was obtained for the classification of normal and cognitive classes under the category of two-class detection. Upon considering hypovigilance states as five-class, a maximum accuracy of 90.9% was achieved. In this case, the number of detection classes increased, resulting in a reduction in the accuracy of detecting more driver states. However, with the possibility of incorrect identification and the presence of issues, the ensemble classifier's performance produced an enhanced accuracy when compared to others.

Multi-Level Classification of Driver Drowsiness by Simultaneous Analysis of ECG and Respiration Signals Using Deep Neural Networks

The high number of fatal crashes caused by driver drowsiness highlights the need for developing reliable drowsiness detection methods. An ideal driver drowsiness detection system should estimate multiple levels of drowsiness accurately without intervening in the driving task. This paper proposes a multi-level drowsiness detection system by a deep neural network-based classification system using a combination of electrocardiogram and respiration signals. The proposed method is based on a combination of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks for classifying drowsiness by concurrently using heart rate variability (HRV), power spectral density of HRV, and respiration rate signal as inputs. Two models, a CNN-based model and a hybrid CNN-LSTM-based model were used for multi-level classifications. The performance of the proposed method was evaluated on experimental data collected from 30 subjects in a simulated driving environment. The performance and the results of both models are presented and compared. The best performance for both three-level and five-level drowsiness classifications was achieved by the CNN-LSTM model. The results indicate that the three-level and five-level classifications of drowsiness can be achieved with 91 and 67% accuracy, respectively.

Drowsiness Detection System Based on PERCLOS and Facial Physiological Signal

Accidents caused by fatigue occur frequently, and numerous scholars have devoted tremendous efforts to investigate methods to reduce accidents caused by fatigued driving. Accordingly, the assessment of the spirit status of the driver through the eyes blinking frequency and the measurement of physiological signals have emerged as effective methods. In this study, a drowsiness detection system is proposed to combine the detection of LF/HF ratio from heart rate variability (HRV) of photoplethysmographic imaging (PPGI) and percentage of eyelid closure over the pupil over time (PERCLOS), and to utilize the advantages of both methods to improve the accuracy and robustness of drowsiness detection. The proposed algorithm performs three functions, including LF/HF ratio from HRV status judgment, eye state detection, and drowsiness judgment. In addition, this study utilized a near-infrared webcam to obtain a facial image to achieve non-contact measurement, alleviate the inconvenience of using a contact wearable device, and for use in a dark environment. Furthermore, we selected the appropriate RGB channel under different light sources to obtain LF/HF ratio from HRV of PPGI. The main drowsiness judgment basis of the proposed drowsiness detection system is the use of algorithm to obtain sympathetic/parasympathetic nervous balance index and percentage of eyelid closure. In the experiment, there are 10 awake samples and 30 sleepy samples. The sensitivity is 88.9%, the specificity is 93.5%, the positive predictive value is 80%, and the system accuracy is 92.5%. In addition, an electroencephalography signal was used as a contrast to validate the reliability of the proposed method.

EEG Signal Multichannel Frequency-Domain Ratio Indices for Drowsiness Detection Based on Multicriteria Optimization

Drowsiness is a risk to human lives in many occupations and activities where full awareness is essential for the safe operation of systems and vehicles, such as driving a car or flying an airplane. Although it is one of the main causes of many road accidents, there is still no reliable definition of drowsiness or a system to reliably detect it. Many researchers have observed correlations between frequency-domain features of the EEG signal and drowsiness, such as an increase in the spectral power of the theta band or a decrease in the spectral power of the beta band. In addition, features calculated as ratio indices between these frequency-domain features show further improvements in detecting drowsiness compared to frequency-domain features alone. This work aims to develop novel multichannel ratio indices that take advantage of the diversity of frequency-domain features from different brain regions. In contrast to the state-of-the-art, we use an evolutionary metaheuristic algorithm to find the nearly optimal set of features and channels from which the indices are calculated. Our results show that drowsiness is best described by the powers in delta and alpha bands. Compared to seven existing single-channel ratio indices, our two novel six-channel indices show improvements in (1) statistically significant differences observed between wakefulness and drowsiness segments, (2) precision of drowsiness detection and classification accuracy of the XGBoost algorithm and (3) model performance by saving time and memory during classification. Our work suggests that a more precise definition of drowsiness is needed, and that accurate early detection of drowsiness should be based on multichannel frequency-domain features.

Non-Invasive Driver Drowsiness Detection System

Drowsiness when in command of a vehicle leads to a decline in cognitive performance that affects driver behavior, potentially causing accidents. Drowsiness-related road accidents lead to severe trauma, economic consequences, impact on others, physical injury and/or even death. Real-time and accurate driver drowsiness detection and warnings systems are necessary schemes to reduce tiredness-related driving accident rates. The research presented here aims at the classification of drowsy and non-drowsy driver states based on respiration rate detection by non-invasive, non-touch, impulsive radio ultra-wideband (IR-UWB) radar. Chest movements of 40 subjects were acquired for 5 m using a lab-placed IR-UWB radar system, and respiration per minute was extracted from the resulting signals. A structured dataset was obtained comprising respiration per minute, age and label (drowsy/non-drowsy). Different machine learning models, namely, Support Vector Machine, Decision Tree, Logistic regression, Gradient Boosting Machine, Extra Tree Classifier and Multilayer Perceptron were trained on the dataset, amongst which the Support Vector Machine shows the best accuracy of 87%. This research provides a ground truth for verification and assessment of UWB to be used effectively for driver drowsiness detection based on respiration.

A Review of EEG Signal Features and their Application in Driver Drowsiness Detection Systems

Detecting drowsiness in drivers, especially multi-level drowsiness, is a difficult problem that is often approached using neurophysiological signals as the basis for building a reliable system. In this context, electroencephalogram (EEG) signals are the most important source of data to achieve successful detection. In this paper, we first review EEG signal features used in the literature for a variety of tasks, then we focus on reviewing the applications of EEG features and deep learning approaches in driver drowsiness detection, and finally we discuss the open challenges and opportunities in improving driver drowsiness detection based on EEG. We show that the number of studies on driver drowsiness detection systems has increased in recent years and that future systems need to consider the wide variety of EEG signal features and deep learning approaches to increase the accuracy of detection.

A novel convolutional neural network method for subject-independent driver drowsiness detection based on single-channel data and EEG alpha spindles

A significant number of fatal accidents are caused by drowsy drivers worldwide. Driver drowsiness detection based on electroencephalography (EEG) signals has high accuracy and is known as a reference method for evaluating drowsiness. Among brain waves, EEG alpha spindle activity is a silent feature of decreasing alertness levels. In this paper, based on the detection of EEG alpha spindles, a novel driver drowsiness detection method is presented. The EEG spindles were detected using Continuous Wavelet Transform (CWT) analysis and the Morlet function. To do so, the signal is divided into 30-s epochs, and the observer rating of drowsiness determines the drowsiness level in each epoch. Tests were conducted on 17 healthy males in a driving simulator with a monotonous driving scenario. The Convolutional Neural Network (CNN) is used for classifying EEG signals and automatically learns features of the early drowsy state. The subject-independent classification results for single-channel P4 show 94% accuracy.

Vector Phase Analysis Approach for Sleep Stage Classification: A Functional Near-Infrared Spectroscopy-Based Passive Brain-Computer Interface

A passive brain-computer interface (BCI) based upon functional near-infrared spectroscopy (fNIRS) brain signals is used for earlier detection of human drowsiness during driving tasks. This BCI modality acquired hemodynamic signals of 13 healthy subjects from the right dorsolateral prefrontal cortex (DPFC) of the brain. Drowsiness activity is recorded using a continuous-wave fNIRS system and eight channels over the right DPFC. During the experiment, sleep-deprived subjects drove a vehicle in a driving simulator while their cerebral oxygen regulation (CORE) state was continuously measured. Vector phase analysis (VPA) was used as a classifier to detect drowsiness state along with sleep stage-based threshold criteria. Extensive training and testing with various feature sets and classifiers are done to justify the adaptation of threshold criteria for any subject without requiring recalibration. Three statistical features (mean oxyhemoglobin, signal peak, and the sum of peaks) along with six VPA features (trajectory slopes of VPA indices) were used. The average accuracies for the five classifiers are 90.9% for discriminant analysis, 92.5% for support vector machines, 92.3% for nearest neighbors, 92.4% for both decision trees, and ensembles over all subjects' data. Trajectory slopes of CORE vector magnitude and angle: m(|R|) and m(∠R) are the best-performing features, along with ensemble classifier with the highest accuracy of 95.3% and minimum computation time of 40 ms. The statistical significance of the results is validated with a p-value of less than 0.05. The proposed passive BCI scheme demonstrates a promising technique for online drowsiness detection using VPA along with sleep stage classification.

Convolutional Neural Network for Drowsiness Detection Using EEG Signals

Drowsiness detection (DD) has become a relevant area of active research in biomedical signal processing. Recently, various deep learning (DL) researches based on the EEG signals have been proposed to detect fatigue conditions. The research presented in this paper proposes an EEG classification system for DD based on DL networks. However, the proposed DD system is mainly realized into two procedures; (i) data acquisition and (ii) model analysis. For the data acquisition procedure, two key steps are considered, which are the signal collection using a wearable Emotiv EPOC+ headset to record 14 channels of EEG, and the signal annotation. Furthermore, a data augmentation (DA) step has been added to the proposed system to overcome the problem of over-fitting and to improve accuracy. As regards the model analysis, a comparative study is also introduced in this paper to argue the choice of DL architecture and frameworks used in our DD system. In this sense, The proposed DD protocol makes use of a convolutional neural network (CNN) architecture implemented using the Keras library. The results showed a high accuracy value (90.42%) in drowsy/awake discrimination and revealed the efficiency of the proposed DD system compared to other research works.

A Systemic Review of Available Low-Cost EEG Headsets Used for Drowsiness Detection

Drowsiness is a leading cause of traffic and industrial accidents, costing lives and productivity. Electroencephalography (EEG) signals can reflect awareness and attentiveness, and low-cost consumer EEG headsets are available on the market. The use of these devices as drowsiness detectors could increase the accessibility of safety and productivity-enhancing devices for small businesses and developing countries. We conducted a systemic review of currently available, low-cost, consumer EEG-based drowsiness detection systems. We sought to determine whether consumer EEG headsets could be reliably utilized as rudimentary drowsiness detection systems. We included documented cases describing successful drowsiness detection using consumer EEG-based devices, including the Neurosky MindWave, InteraXon Muse, Emotiv Epoc, Emotiv Insight, and OpenBCI. Of 46 relevant studies, ~27 reported an accuracy score. The lowest of these was the Neurosky Mindwave, with a minimum of 31%. The second lowest accuracy reported was 79.4% with an OpenBCI study. In many cases, algorithmic optimization remains necessary. Different methods for accuracy calculation, system calibration, and different definitions of drowsiness made direct comparisons problematic. However, even basic features, such as the power spectra of EEG bands, were able to consistently detect drowsiness. Each specific device has its own capabilities, tradeoffs, and limitations. Widely used spectral features can achieve successful drowsiness detection, even with low-cost consumer devices; however, reliability issues must still be addressed in an occupational context.

A Portable Fuzzy Driver Drowsiness Estimation System

The adequate automatic detection of driver fatigue is a very valuable approach for the prevention of traffic accidents. Devices that can determine drowsiness conditions accurately must inherently be portable, adaptable to different vehicles and drivers, and robust to conditions such as illumination changes or visual occlusion. With the advent of a new generation of computationally powerful embedded systems such as the Raspberry Pi, a new category of real-time and low-cost portable drowsiness detection systems could become standard tools. Usually, the proposed solutions using this platform are limited to the definition of thresholds for some defined drowsiness indicator or the application of computationally expensive classification models that limits their use in real-time. In this research, we propose the development of a new portable, low-cost, accurate, and robust drowsiness recognition device. The proposed device combines complementary drowsiness measures derived from a temporal window of eyes (PERCLOS, ECD) and mouth (AOT) states through a fuzzy inference system deployed in a Raspberry Pi with the capability of real-time response. The system provides three degrees of drowsiness (Low-Normal State, Medium-Drowsy State, and High-Severe Drowsiness State), and was assessed in terms of its computational performance and efficiency, resulting in a significant accuracy of 95.5% in state recognition that demonstrates the feasibility of the approach.

CoDR: Correlation-based Data Reduction Scheme for Efficient Gathering of Heterogeneous Driving Data

A variety of deep learning techniques are actively employed for advanced driver assistance systems, which in turn require gathering lots of heterogeneous driving data, such as traffic conditions, driver behavior, vehicle status and location information. However, these different types of driving data easily become more than tens of GB per day, forming a significant hurdle due to the storage and network cost. To address this problem, this paper proposes a novel scheme, called CoDR, which can reduce data volume by considering the correlations among heterogeneous driving data. Among heterogeneous datasets, CoDR first chooses one set as a pivot data. Then, according to the objective of data collection, it identifies data ranges relevant to the objective from the pivot dataset. Finally, it investigates correlations among sets, and reduces data volume by eliminating irrelevant data from not only the pivot set but also other remaining datasets. CoDR gathers four heterogeneous driving datasets: two videos for front view and driver behavior, OBD-II and GPS data. We show that CoDR decreases data volume by up to 91%. We also present diverse analytical results that reveal the correlations among the four datasets, which can be exploited usefully for edge computing to reduce data volume on the spot.

Assessment of the Potential of Wrist-Worn Wearable Sensors for Driver Drowsiness Detection

Drowsy driving imposes a high safety risk. Current systems often use driving behavior parameters for driver drowsiness detection. The continuous driving automation reduces the availability of these parameters, therefore reducing the scope of such methods. Especially, techniques that include physiological measurements seem to be a promising alternative. However, in a dynamic environment such as driving, only non- or minimal intrusive methods are accepted, and vibrations from the roadbed could lead to degraded sensor technology. This work contributes to driver drowsiness detection with a machine learning approach applied solely to physiological data collected from a non-intrusive retrofittable system in the form of a wrist-worn wearable sensor. To check accuracy and feasibility, results are compared with reference data from a medical-grade ECG device. A user study with 30 participants in a high-fidelity driving simulator was conducted. Several machine learning algorithms for binary classification were applied in user-dependent and independent tests. Results provide evidence that the non-intrusive setting achieves a similar accuracy as compared to the medical-grade device, and high accuracies (>92%) could be achieved, especially in a user-dependent scenario. The proposed approach offers new possibilities for human-machine interaction in a car and especially for driver state monitoring in the field of automated driving.