Heart Rate Variability-Based Driver Drowsiness Detection and Its Validation With EEG

Multi-Index Driver Drowsiness Detection Method Based on Driver's Facial Recognition Using Haar Features and Histograms of Oriented Gradients

It is estimated that 10% to 20% of road accidents are related to fatigue, with accidents caused by drowsiness up to twice as deadly as those caused by other factors. In order to reduce these numbers, strategies such as advertising campaigns, the implementation of driving recorders in vehicles used for road transport of goods and passengers, or the use of drowsiness detection systems in cars have been implemented. Within the scope of the latter area, the technologies used are diverse. They can be based on the measurement of signals such as steering wheel movement, vehicle position on the road, or driver monitoring. Driver monitoring is a technology that has been exploited little so far and can be implemented in many different approaches. This work addresses the evaluation of a multidimensional drowsiness index based on the recording of facial expressions, gaze direction, and head position and studies the feasibility of its implementation in a low-cost electronic package. Specifically, the aim is to determine the driver's state by monitoring their facial expressions, such as the frequency of blinking, yawning, eye-opening, gaze direction, and head position. For this purpose, an algorithm capable of detecting drowsiness has been developed. Two approaches are compared: Facial recognition based on Haar features and facial recognition based on Histograms of Oriented Gradients (HOG). The implementation has been carried out on a Raspberry Pi, a low-cost device that allows the creation of a prototype that can detect drowsiness and interact with peripherals such as cameras or speakers. The results show that the proposed multi-index methodology performs better in detecting drowsiness than algorithms based on one-index detection.

Enhancing Road Safety: Fast and Accurate Noncontact Driver HRV Detection Based on Huber-Kalman and Autocorrelation Algorithms

Enhancing road safety by monitoring a driver's physical condition is critical in both conventional and autonomous driving contexts. Our research focuses on a wireless intelligent sensor system that utilizes millimeter-wave (mmWave) radar to monitor heart rate variability (HRV) in drivers. By assessing HRV, the system can detect early signs of drowsiness and sudden medical emergencies, such as heart attacks, thereby preventing accidents. This is particularly vital for fully self-driving (FSD) systems, as it ensures control is not transferred to an impaired driver. The proposed system employs a 60 GHz frequency-modulated continuous wave (FMCW) radar placed behind the driver's seat. This article mainly describes how advanced signal processing methods, including the Huber-Kalman filtering algorithm, are applied to mitigate the impact of respiration on heart rate detection. Additionally, the autocorrelation algorithm enables fast detection of vital signs. Intensive experiments demonstrate the system's effectiveness in accurately monitoring HRV, highlighting its potential to enhance safety and reliability in both traditional and autonomous driving environments.

Cardiac indices of driver fatigue across in-lab and on-road studies

Driver fatigue is a major contributor to road accidents. Therefore, driver assistance systems (DAS) that would monitor drivers' states may contribute to road safety. Such monitoring can potentially be achieved with input from ECG indices (e.g., heart rate). We reviewed the empirical literature on responses of cardiac measures to driver fatigue and on detecting fatigue with cardiac indices and classification algorithms. We used meta-analytical methods to explore the pooled effect sizes of different cardiac indices of fatigue, their heterogeneity, and the consistency of their responses across studies. Our large pool of studies (N = 39) allowed us to stratify the results across on-road and simulator studies. We found that despite the large heterogeneity of the effect sizes between the studies, many indices had significant pooled effect sizes across the studies, and more frequently across the on-road studies. We also found that most indices showed consistent responses across both on-road and simulator studies. Regarding the detection accuracy, we found that even on-road classification could have been as accurate as 70% with only 2-min of data. However, we could only find two on-road studies that employed fatigue classification algorithms. Overall, our findings are encouraging with respect to the prospect of using cardiac measures for detecting driver fatigue. Yet, to fully explore this possibility, there is a need for additional on-road studies that would employ a similar set of cardiac indices and detection algorithms, a unified definition of fatigue, and additional levels of fatigue than the two fatigue vs alert states.

Fatigue at the wheel: A non-visual approach to truck driver fatigue detection by multi-feature fusion

BACKGROUND

Monitoring of long-haul truck driver fatigue state has attracted considerable interest. Conventional fatigue driving detection methods based on the physiological and visual features are scarcely applicable, due to the intrusiveness, reliability, and cost-effectiveness concerns.

METHODS

We elaborately developed a fatigue driving detection method by fusion of non-visual features derived from the customized wristbands, vehicle-mounted equipment, and trip logs. To capture the spatiotemporal information within the sequential data, the bidirectional long short-term memory network with attention mechanism was proposed to determine whether the truck driver was fatigued within a fine-grained episode of one minute. The model was validated using a natural driving dataset with nine truck drivers on real-world roads in Guiyang, China during June and July 2021.

RESULTS

Our approach yielded 99.21 %, 84.44 %, 82.01 %, 99.63 %, and 83.21 % in accuracy, precision, recall, specificity, and F1-score, respectively. Compared with the mainstream visual-based methods, our approach outperformed particularly in terms of precision and recall. Photoplethysmogram stood out as the most important feature for truck driver fatigue state detection. Vehicle load, driving forward angle, cumulative driving time, midnight, and recent working hours were found to be positively associated with the probability of fatigue driving, while the galvanic skin response, vehicle acceleration, current time, and recent rest hours had a negative relationship. Specifically, truck drivers were more likely to fatigue when driving at 20-40 km/h, braking abruptly at 5-10 m/s2, with vehicle loads over 70 tons, and driving more than 100 min consecutively.

CONCLUSIONS

Our study is among the first to harness the natural driving dataset to delve into the real-life fatigue pattern of long-haul truck drivers without disruptions on routine driving tasks. The proposed method holds pragmatic prospects by providing a privacy-preserving, robust, real-time, and non-intrusive technical pathway for truck driver fatigue monitoring.

Association of Visual-Based Signals with Electroencephalography Patterns in Enhancing the Drowsiness Detection in Drivers with Obstructive Sleep Apnea

Individuals with obstructive sleep apnea (OSA) face increased accident risks due to excessive daytime sleepiness. PERCLOS, a recognized drowsiness detection method, encounters challenges from image quality, eyewear interference, and lighting variations, impacting its performance, and requiring validation through physiological signals. We propose visual-based scoring using adaptive thresholding for eye aspect ratio with OpenCV for face detection and Dlib for eye detection from video recordings. This technique identified 453 drowsiness (PERCLOS ≥ 0.3 || CLOSDUR ≥ 2 s) and 474 wakefulness episodes (PERCLOS < 0.3 and CLOSDUR < 2 s) among fifty OSA drivers in a 50 min driving simulation while wearing six-channel EEG electrodes. Applying discrete wavelet transform, we derived ten EEG features, correlated them with visual-based episodes using various criteria, and assessed the sensitivity of brain regions and individual EEG channels. Among these features, theta-alpha-ratio exhibited robust mapping (94.7%) with visual-based scoring, followed by delta-alpha-ratio (87.2%) and delta-theta-ratio (86.7%). Frontal area (86.4%) and channel F4 (75.4%) aligned most episodes with theta-alpha-ratio, while frontal, and occipital regions, particularly channels F4 and O2, displayed superior alignment across multiple features. Adding frontal or occipital channels could correlate all episodes with EEG patterns, reducing hardware needs. Our work could potentially enhance real-time drowsiness detection reliability and assess fitness to drive in OSA drivers.

Physiological measurements for driving drowsiness: A comparative study of multi-modality feature fusion and selection

A large number of traffic accidents were caused by drowsiness while driving. In-vehicle alert system based on physiological signals was one of the most promising solutions to monitor driving fatigue. However, different physiological modalities can be used, and many relative studies compared different modalities without considering the implementation feasibility of portable or wearable devices. Moreover, evaluations of each modality in previous studies were based on inconsistent choices of fatigue label and signal features, making it hard to compare the results of different studies. Therefore, the modality comparison and fusion for continuous drowsiness estimation while driving was still unclear. This work sought to comprehensively compare widely-used physiological modalities, including forehead electroencephalogram (EEG), electrooculogram (EOG), R-R intervals (RRI) and breath, in a hardware setting feasible for portable or wearable devices to monitor driving fatigue. Moreover, a more general conclusion on modality comparison and fusion was reached based on the regression of features or their combinations and the awake-to-drowsy transition. Finally, the feature subset of fused modalities was produced by feature selection method, to select the optimal feature combination and reduce computation consumption. Considering practical feasibility, the most effective combination with the highest correlation coefficient was using forehead EEG or EOG, along with RRI and RRI-derived breath. If more comfort and convenience was required, the combination of RRI and RRI-derived breath was also promising.

Neurophysiological mental fatigue assessment for developing user-centered Artificial Intelligence as a solution for autonomous driving

The human factor plays a key role in the automotive field since most accidents are due to drivers' unsafe and risky behaviors. The industry is now pursuing two main solutions to deal with this concern: in the short term, there is the development of systems monitoring drivers' psychophysical states, such as inattention and fatigue, and in the medium-long term, there is the development of fully autonomous driving. This second solution is promoted by recent technological progress in terms of Artificial Intelligence and sensing systems aimed at making vehicles more and more accurately aware of their "surroundings." However, even with an autonomous vehicle, the driver should be able to take control of the vehicle when needed, especially during the current transition from the lower (SAE < 3) to the highest level (SAE = 5) of autonomous driving. In this scenario, the vehicle has to be aware not only of its "surroundings" but also of the driver's psychophysical state, i.e., a user-centered Artificial Intelligence. The neurophysiological approach is one the most effective in detecting improper mental states. This is particularly true if considering that the more automatic the driving will be, the less available the vehicular data related to the driver's driving style. The present study aimed at employing a holistic approach, considering simultaneously several neurophysiological parameters, in particular, electroencephalographic, electrooculographic, photopletismographic, and electrodermal activity data to assess the driver's mental fatigue in real time and to detect the onset of fatigue increasing. This would ideally work as an information/trigger channel for the vehicle AI. In all, 26 professional drivers were engaged in a 45-min-lasting realistic driving task in simulated conditions, during which the previously listed biosignals were recorded. Behavioral (reaction times) and subjective measures were also collected to validate the experimental design and to support the neurophysiological results discussion. Results showed that the most sensitive and timely parameters were those related to brain activity. To a lesser extent, those related to ocular parameters were also sensitive to the onset of mental fatigue, but with a delayed effect. The other investigated parameters did not significantly change during the experimental session.

Electroencephalogram based brain-computer interface: Applications, challenges, and opportunities

Brain-Computer Interfaces (BCI) is an exciting and emerging research area for researchers and scientists. It is a suitable combination of software and hardware to operate any device mentally. This review emphasizes the significant stages in the BCI domain, current problems, and state-of-the-art findings. This article also covers how current results can contribute to new knowledge about BCI, an overview of BCI from its early developments to recent advancements, BCI applications, challenges, and future directions. The authors pointed to unresolved issues and expressed how BCI is valuable for analyzing the human brain. Humans' dependence on machines has led humankind into a new future where BCI can play an essential role in improving this modern world.

A New ECG Data Processing Approach to Developing an Accurate Driving Fatigue Detection Framework with Heart Rate Variability Analysis and Ensemble Learning

More than 1.3 million people are killed in traffic accidents annually. Road traffic accidents are mostly caused by human error. Therefore, an accurate driving fatigue detection system is required for drivers. Most driving fatigue detection studies concentrated on improving feature engineering and classification methods. We propose a novel driving fatigue detection framework concentrating on the development of the preprocessing, feature extraction, and classification stages to improve the classification accuracy of fatigue states. The proposed driving fatigue detection framework measures fatigue using a two-electrode ECG. The resampling method and heart rate variability analysis were used to extract features from the ECG data, and an ensemble learning model was utilized to classify fatigue states. To achieve the best model performance, 40 possible scenarios were applied: a combination of 5 resampling scenarios, 2 feature extraction scenarios, and 4 classification model scenarios. It was discovered that the combination of a resampling method with a window duration of 300 s and an overlap of 270 s, 54 extracted features, and AdaBoost yielded an optimum accuracy of 98.82% for the training dataset and 81.82% for the testing dataset. Furthermore, the preprocessing resampling method had the greatest impact on the model’s performance; it is a new approach presented in this study.

KPCA-WRF-prediction of heart rate using deep feature fusion and machine learning classification with tuned weighted hyper-parameter

The rapid advancement of technology such as stream processing technologies, deep-learning approaches, and artificial intelligence plays a prominent and vital role, to detect heart rate using a prediction model. However, the existing methods could not handle high -dimensional datasets, and deep feature learning to improvise the performance. Therefore, this work proposed a real-time heart rate prediction model, using K-nearest neighbour (KNN) adhered to the principle component analysis algorithm (PCA) and weighted random forest algorithm for feature fusion (KPCA-WRF) approach and deep CNN feature learning framework. The feature selection, from the fused features, was optimized by ant colony optimization (ACO) and particle swarm optimization (PSO) algorithm to enhance the selected fused features from deep CNN. The optimized features were reduced to low dimensions using the PCA algorithm. The significant straight heart rate features are plotted by capturing out nearest similar data point values using the algorithm. The fused features were then classified for aiding the training process. The weighted values are assigned to those tuned hyper parameters (feature matrix forms). The optimal path and continuity of the weighted feature representations are moved using the random forest algorithm, in K-fold validation iterations.

Two-Stream Spatial-Temporal Graph Convolutional Networks for Driver Drowsiness Detection

Convolutional neural networks (CNNs) have achieved remarkable performance in driver drowsiness detection based on the extraction of deep features of drivers' faces. However, the performance of driver drowsiness detection methods decreases sharply when complications, such as illumination changes in the cab, occlusions and shadows on the driver's face, and variations in the driver's head pose, occur. In addition, current driver drowsiness detection methods are not capable of distinguishing between driver states, such as talking versus yawning or blinking versus closing eyes. Therefore, technical challenges remain in driver drowsiness detection. In this article, we propose a novel and robust two-stream spatial-temporal graph convolutional network (2s-STGCN) for driver drowsiness detection to solve the above-mentioned challenges. To take advantage of the spatial and temporal features of the input data, we use a facial landmark detection method to extract the driver's facial landmarks from real-time videos and then obtain the driver drowsiness detection result by 2s-STGCN. Unlike existing methods, our proposed method uses videos rather than consecutive video frames as processing units. This is the first effort to exploit these processing units in the field of driver drowsiness detection. Moreover, the two-stream framework not only models both the spatial and temporal features but also models both the first-order and second-order information simultaneously, thereby notably improving driver drowsiness detection. Extensive experiments have been performed on the yawn detection dataset (YawDD) and the National TsingHua University drowsy driver detection (NTHU-DDD) dataset. The experimental results validate the feasibility of the proposed method. This method achieves an average accuracy of 93.4% on the YawDD dataset and an average accuracy of 92.7% on the evaluation set of the NTHU-DDD dataset.

Detecting driver fatigue using heart rate variability: A systematic review

Driver fatigue detection systems have potential to improve road safety by preventing crashes and saving lives. Conventional driver monitoring systems based on driving performance and facial features may be challenged by the application of automated driving systems. This limitation could potentially be overcome by monitoring systems based on physiological measurements. Heart rate variability (HRV) is a physiological marker of interest for detecting driver fatigue that can be measured during real life driving. This systematic review investigates the relationship between HRV measures and driver fatigue, as well as the performance of HRV based fatigue detection systems. With the applied eligibility criteria, 18 articles were identified in this review. Inconsistent results can be found within the studies that investigated differences of HRV measures between alert and fatigued drivers. For studies that developed HRV based fatigue detection systems, the detection performance showed a large variation, where the detection accuracy ranged from 44% to 100%. The inconsistency and variation of the results can be caused by differences in several key aspects in the study designs. Progress in this field is needed to determine the relationship between HRV and different fatigue causal factors and its connection to driver performance. To be deployed, HRV-based fatigue detection systems need to be thoroughly tested in real life conditions with good coverage of relevant driving scenarios and a sufficient number of participants.

Validation of Wearable Device Consisting of a Smart Shirt with Built-In Bioelectrodes and a Wireless Transmitter for Heart Rate Monitoring in Light to Moderate Physical Work

Real-time monitoring of heart rate is useful for monitoring workers. Wearable heart rate monitors worn on the upper body are less susceptible to artefacts caused by arm and wrist movements than popular wristband-type sensors using the photoplethysmography method. Therefore, they are considered suitable for stable and accurate measurement for various movements. In this study, we conducted an experiment to verify the accuracy of our developed and commercially available wearable heart rate monitor consisting of a smart shirt with bioelectrodes and a transmitter, assuming a real-world work environment with physical loads. An exercise protocol was designed to light to moderate intensity according to international standards because no standard exercise protocol for the validation simulating these works has been reported. This protocol includes worker-specific movements such as applying external vibration and lifting and lowering loads. In the experiment, we simultaneously measured the instantaneous heart rate with the above wearable device and a Holter monitor as a reference to evaluate mean absolute percentage error (MAPE). The MAPE was 0.92% or less for all exercise protocols conducted. This value indicates that the accuracy of the wearable device is high enough for use in real-world cases of physical load in light to moderate intensity tasks such as those in our experimental protocol. In addition, the experimental protocol and measurement data devised in this study can be used as a benchmark for other wearable heart rate monitors for use for similar purposes.

Quantifying Cognitive Workload Using a Non-Contact Magnetocardiography (MCG) Wearable Sensor

Quantifying cognitive workload, i.e., the level of mental effort put forth by an individual in response to a cognitive task, is relevant for healthcare, training and gaming applications. However, there is currently no technology available that can readily and reliably quantify the cognitive workload of an individual in a real-world environment at a seamless way and affordable price. In this work, we overcome these limitations and demonstrate the feasibility of a magnetocardiography (MCG) sensor to reliably classify high vs. low cognitive workload while being non-contact, fully passive and low-cost, with the potential to have a wearable form factor. The operating principle relies on measuring the naturally emanated magnetic fields from the heart and subsequently analyzing the heart rate variability (HRV) matrix in three time-domain parameters: standard deviation of RR intervals (SDRR); root mean square of successive differences between heartbeats (RMSSD); and mean values of adjacent R-peaks in the cardiac signals (MeanRR). A total of 13 participants were recruited, two of whom were excluded due to low signal quality. The results show that SDRR and RMSSD achieve a 100% success rate in classifying high vs. low cognitive workload, while MeanRR achieves a 91% success rate. Tests for the same individual yield an intra-subject classification accuracy of 100% for all three HRV parameters. Future studies should leverage machine learning and advanced digital signal processing to achieve automated classification of cognitive workload and reliable operation in a natural environment.

Drowsiness Detection System Based on PERCLOS and Facial Physiological Signal

Accidents caused by fatigue occur frequently, and numerous scholars have devoted tremendous efforts to investigate methods to reduce accidents caused by fatigued driving. Accordingly, the assessment of the spirit status of the driver through the eyes blinking frequency and the measurement of physiological signals have emerged as effective methods. In this study, a drowsiness detection system is proposed to combine the detection of LF/HF ratio from heart rate variability (HRV) of photoplethysmographic imaging (PPGI) and percentage of eyelid closure over the pupil over time (PERCLOS), and to utilize the advantages of both methods to improve the accuracy and robustness of drowsiness detection. The proposed algorithm performs three functions, including LF/HF ratio from HRV status judgment, eye state detection, and drowsiness judgment. In addition, this study utilized a near-infrared webcam to obtain a facial image to achieve non-contact measurement, alleviate the inconvenience of using a contact wearable device, and for use in a dark environment. Furthermore, we selected the appropriate RGB channel under different light sources to obtain LF/HF ratio from HRV of PPGI. The main drowsiness judgment basis of the proposed drowsiness detection system is the use of algorithm to obtain sympathetic/parasympathetic nervous balance index and percentage of eyelid closure. In the experiment, there are 10 awake samples and 30 sleepy samples. The sensitivity is 88.9%, the specificity is 93.5%, the positive predictive value is 80%, and the system accuracy is 92.5%. In addition, an electroencephalography signal was used as a contrast to validate the reliability of the proposed method.

Assessment of Combination of Automated Pupillometry and Heart Rate Variability to Detect Driving Fatigue

Objectives

Approximately 20~30% of all traffic accidents are caused by fatigue driving. However, limited practicability remains a barrier for the real application of available techniques to detect driving fatigue. Use of pupillary light reflex (PLR) may be potentially effective for driving fatigue detection.

Methods

A 90 min monotonous simulated driving task was utilized to induce driving fatigue. During the task, PLR measurements were performed at baseline and at an interval of 30 min. Subjective rating scales, heart rate variability (HRV) were monitored simultaneously.

Results

Thirty-two healthy volunteers in China participated in our study. Based on the results of subjective evaluation and behavioral performances, driving fatigue was verified to be successfully induced by a simulated driving task. Significant variations of PLR and HRV parameters were observed, which also showed significant relevance with the change in Karolinska Sleepiness Scale at several timepoints (|r| = 0.55 ~ 0.72, P < 0.001). Furthermore, PLR variations had excellent ability to detect driving fatigue with high sensitivity and specificity, of which maximum constriction velocity variations achieved a sensitivity of 85.00% and specificity of 72.34% for driving fatigue detection, vs. 82.50 and 78.72% with a combination of HRV variations, a nonsignificant difference (AUC = 0.835, 0.872, P > 0.05).

Conclusions

Pupillary light reflex variation may be a potential indicator in the detection of driving fatigue, achieving a comparative performance compared with the combination with heart rate variability. Further work may be involved in developing a commercialized driving fatigue detection system based on pupillary parameters.

Contactless Vital Sign Monitoring System for In-Vehicle Driver Monitoring Using a Near-Infrared Time-of-Flight Camera

We demonstrate a Contactless Vital Sign Monitoring (CVSM) system and road-test the system for in-cabin driver monitoring using a near-infrared indirect Time-of-Flight (ToF) camera. The CVSM measures both heart rate (HR) and respiration rate (RR) by leveraging the simultaneously measured grayscale and depth information from a ToF camera. For a camera-based driver monitoring system (DMS), key challenges from varying background illumination and motion-induced artifacts need to be addressed. In this study, active illumination and depth-based motion compensation are used to mitigate these two challenges. For HR measurements, active illumination allows the system to work under various lighting conditions, while our depth-based motion compensation has the advantage of directly measuring the motion of the driver without making prior assumptions about the motion artifacts. In addition, we can extract RR directly from the chest wall motion, circumventing the challenge of acquiring RR from the near-infrared photoplethysmography (PPG) signal of low signal quality. We investigate the system’s performance in various scenarios, including monitoring both drivers and passengers while driving on highways and local roads. Our results show that our CVSM system is ambient light agnostic, and the success rates of HR measurements on the highway are 82% and 71.9% for the passenger and driver, respectively. At the same time, we show that the system can measure RR on users driving on a highway with a mean deviation of −1.4 breaths per minute (BPM). With reliable HR and RR measurement in the vehicle, the CVSM system could one day be a key enabler to sudden sickness or drowsiness detection in DMS.

A Review of Recent Developments in Driver Drowsiness Detection Systems

Continuous advancements in computing technology and artificial intelligence in the past decade have led to improvements in driver monitoring systems. Numerous experimental studies have collected real driver drowsiness data and applied various artificial intelligence algorithms and feature combinations with the goal of significantly enhancing the performance of these systems in real-time. This paper presents an up-to-date review of the driver drowsiness detection systems implemented over the last decade. The paper illustrates and reviews recent systems using different measures to track and detect drowsiness. Each system falls under one of four possible categories, based on the information used. Each system presented in this paper is associated with a detailed description of the features, classification algorithms, and used datasets. In addition, an evaluation of these systems is presented, in terms of the final classification accuracy, sensitivity, and precision. Furthermore, the paper highlights the recent challenges in the area of driver drowsiness detection, discusses the practicality and reliability of each of the four system types, and presents some of the future trends in the field.

Driver drowsiness estimation using EEG signals with a dynamical encoder-decoder modeling framework

Drowsiness is a leading cause of accidents on the road as it negatively affects the driver’s ability to safely operate a vehicle. Neural activity recorded by EEG electrodes is a widely used physiological correlate of driver drowsiness. This paper presents a novel dynamical modeling solution to estimate the instantaneous level of the driver drowsiness using EEG signals, where the PERcentage of eyelid CLOSure (PERCLOS) is employed as the ground truth of driver drowsiness. Applying our proposed modeling framework, we find neural features present in EEG data that encode PERCLOS. In the decoding phase, we use a Bayesian filtering solution to estimate the PERCLOS level over time. A data set that comprises 18 driving tests, conducted by 13 drivers, has been used to investigate the performance of the proposed framework. The modeling performance in estimation of PERCLOS provides robust and repeatable results in tests with manual and automated driving modes by an average RMSE of 0.117 (at a PERCLOS range of 0 to 1) and average High Probability Density percentage of 62.5%. We further hypothesized that there are biomarkers that encode the PERCLOS across different driving tests and participants. Using this solution, we identified possible biomarkers such as Theta and Delta powers. Results show that about 73% and 66% of the Theta and Delta powers which are selected as biomarkers are increasing as PERCLOS grows during the driving test. We argue that the proposed method is a robust and reliable solution to estimate drowsiness in real-time which opens the door in utilizing EEG-based measures in driver drowsiness detection systems.

Driver Monitoring of Automated Vehicles by Classification of Driver Drowsiness Using a Deep Convolutional Neural Network Trained by Scalograms of ECG Signals

Driver drowsiness is one of the leading causes of traffic accidents. This paper proposes a new method for classifying driver drowsiness using deep convolution neural networks trained by wavelet scalogram images of electrocardiogram (ECG) signals. Three different classes were defined for drowsiness based on video observation of driving tests performed in a simulator for manual and automated modes. The Bayesian optimization method is employed to optimize the hyperparameters of the designed neural networks, such as the learning rate and the number of neurons in every layer. To assess the results of the deep network method, heart rate variability (HRV) data is derived from the ECG signals, some features are extracted from this data, and finally, random forest and k-nearest neighbors (KNN) classifiers are used as two traditional methods to classify the drowsiness levels. Results show that the trained deep network achieves balanced accuracies of about 77% and 79% in the manual and automated modes, respectively. However, the best obtained balanced accuracies using traditional methods are about 62% and 64%. We conclude that designed deep networks working with wavelet scalogram images of ECG signals significantly outperform KNN and random forest classifiers which are trained on HRV-based features.

Inattentive Driving Detection Using Body-Worn Sensors: Feasibility Study

This study aims to build a system for detecting a driver’s internal state using body-worn sensors. Our system is intended to detect inattentive driving that occurs during long-term driving on a monotonous road, such as a high-way road. The inattentive state of a driver in this study is an absent-minded state caused by a decrease in driver vigilance levels due to fatigue or drowsiness. However, it is difficult to clearly define these inattentive states because it is difficult for the driver to recognize when they fall into an absent-minded state. To address this problem and achieve our goal, we have proposed a detection algorithm for inattentive driving that not only uses a heart rate sensor, but also uses body-worn inertial sensors, which have the potential to detect driver behavior more accurately and at a much lower cost. The proposed method combines three detection models: body movement, drowsiness, and inattention detection, based on an anomaly detection algorithm. Furthermore, we have verified the accuracy of the algorithm with the experimental data for five participants that were measured in long-term and monotonous driving scenarios by using a driving simulator. The results indicate that our approach can detect both the inattentive and drowsiness states of drivers using signals from both the heart rate sensor and accelerometers placed on wrists.

Design of a rapid diagnostic model for bladder compliance based on real-time intravesical pressure monitoring system

OBJECTIVE

The diagnosis of bladder dysfunction for children depends on the confirmation of abnormal bladder shape and bladder compliance. The existing gold standard needs to conduct voiding cystourethrogram (VCUG) examination and urodynamic studies (UDS) examination on patients separately. To reduce the time and injury of children's inspection, we propose a novel method to judge the bladder compliance by measuring the intravesical pressure during the VCUG examination without extra UDS.

METHODS

Our method consisted of four steps. We firstly developed a single-tube device that can measure, display, store, and transmit real-time pressure data. Secondly, we conducted clinical trials with the equipment on a cohort of 52 patients (including 32 negative and 20 positive cases). Thirdly, we preprocessed the data to eliminate noise and extracted features, then we used the least absolute shrinkage and selection operator (LASSO) to screen out important features. Finally, several machine learning methods were applied to classify and predict the bladder compliance level, including support vector machine (SVM), Random Forest, XGBoost, perceptron, logistic regression, and Naive Bayes, and the classification performance was evaluated.

RESULTS

73 features were extracted, including first-order and second-order time-domain features, wavelet features, and frequency domain features. 15 key features were selected and the model showed promising classification performance. The highest AUC value was 0.873 by the SVM algorithm, and the corresponding accuracy was 84%.

CONCLUSION

We designed a system to quickly obtain the intravesical pressure during the VCUG test, and our classification model is competitive in judging patients' bladder compliance.

SIGNIFICANCE

This could facilitate rapid auxiliary diagnosis of bladder disease based on real-time data. The promising result of classification is expected to provide doctors with a reliable basis in the auxiliary diagnosis of some bladder diseases prior to UDS.

Increase in rear-end collision risk by acute stress-induced fatigue in on-road truck driving

Increasing road crashes related to occupational drivers’ deteriorating health has become a social problem. To prevent road crashes, warnings and predictions of increased crash risk based on drivers’ conditions are important. However, in on-road driving, the relationship between drivers’ physiological condition and crash risk remains unclear due to difficulties in the simultaneous measurement of both. This study aimed to elucidate the relationship between drivers’ physiological condition assessed by autonomic nerve function (ANF) and an indicator of rear-end collision risk in on-road driving. Data from 20 male truck drivers (mean ± SD, 49.0±8.2 years; range, 35–63 years) were analyzed. Over a period of approximately three months, drivers’ working behavior data, such as automotive sensor data, and their ANF data were collected during their working shift. Using the gradient boosting decision tree method, a rear-end collision risk index was developed based on the working behavior data, which enabled continuous risk quantification. Using the developed risk index and drivers’ ANF data, effects of their physiological condition on risk were analyzed employing a logistic quantile regression method, which provides wider information on the effects of the explanatory variables, after hierarchical model selection. Our results revealed that in on-road driving, activation of sympathetic nerve activity and inhibition of parasympathetic nerve activity increased each quantile of the rear-end collision risk index. The findings suggest that acute stress-induced drivers’ fatigue increases rear-end collision risk. Hence, in on-road driving, drivers’ physiological condition monitoring and ANF-based stress warning and relief system can contribute to promoting the prevention of rear-end truck collisions.

Driving-Induced Neurological Biomarkers in an Advanced Driver-Assistance System

Physiological signals are immediate and sensitive to neurological changes resulting from the mental workload induced by various driving environments and are considered a quantifying tool for understanding the association between neurological outcomes and driving cognitive workloads. Neurological assessment, outside of a highly-equipped clinical setting, requires an ambulatory electroencephalography (EEG) headset. This study aimed to quantify neurological biomarkers during a resting state and two different scenarios of driving states in a virtual driving environment. We investigated the neurological responses of seventeen healthy male drivers. EEG data were measured in an initial resting state, city-roadways driving state, and expressway driving state using a portable EEG headset in a driving simulator. During the experiment, the participants drove while experiencing cognitive workloads due to various driving environments, such as road traffic conditions, lane changes of surrounding vehicles, the speed limit, etc. The power of the beta and gamma bands decreased, and the power of the delta waves, theta, and frontal theta asymmetry increased in the driving state relative to the resting state. Delta-alpha ratio (DAR) and delta-theta ratio (DTR) showed a strong correlation with a resting state, city-roadways driving state, and expressway driving state. Binary machine-learning (ML) classification models showed a near-perfect accuracy between the resting state and driving state. Moderate classification performances were observed between the resting state, city-roadways state, and expressway state in multi-class classification. An EEG-based neurological state prediction approach may be utilized in an advanced driver-assistance system (ADAS).

Prediction of GABA receptor antagonist-induced convulsion in cynomolgus monkeys by combining machine learning and heart rate variability analysis

Drug-induced convulsion is a severe adverse event; however, no useful biomarkers for it have been discovered. We propose a new method for predicting drug-induced convulsions in monkeys based on heart rate variability (HRV) and a machine learning technique. Because autonomic nervous activities are altered around the time of a convulsion and such alterations affect HRV, they may be predicted by monitoring HRV. In the proposed method, anomalous changes in multiple HRV parameters are monitored by means of a convulsion prediction model, and convulsion alarms are issued when abnormal changes in HRV are detected. The convulsion prediction model is constructed based on multivariate statistical process control (MSPC), a well-known anomaly detection algorithm in machine learning. In this study, HRV data were collected from four cynomolgus monkeys administered with multiple doses of pentylenetetrazol (PTZ) and picrotoxin (PTX), which are GABA receptor antagonists, as convulsant agents. In addition, low doses of pilocarpine (PILO) were administered as a negative control. Twelve HRV parameters in three hours after drug administration were monitored by means of the prediction model. The number and duration of convulsion alarms from HRV increased at medium and high doses of PTZ and PTX (1/3 or 1/4 of convulsion dose). On the other hand, the frequency of convulsion alarms did not increase with PILO. Although vomiting was observed in all drugs, convulsion alarms were not associated with the vomiting. Thus, convulsion alarms can be used as a biomarker for convulsions induced by GABA receptor antagonists.

Autoencoder-Based Extrasystole Detection and Modification of RRI Data for Precise Heart Rate Variability Analysis

Heart rate variability, which is the fluctuation of the R-R interval (RRI) in electrocardiograms (ECG), has been widely adopted for autonomous evaluation. Since the HRV features that are extracted from RRI data easily fluctuate when arrhythmia occurs, RRI data with arrhythmia need to be modified appropriately before HRV analysis. In this study, we consider two types of extrasystoles-premature ventricular contraction (PVC) and premature atrial contraction (PAC)-which are types of extrasystoles that occur every day, even in healthy persons who have no cardiovascular diseases. A unified framework for ectopic RRI detection and a modification algorithm that utilizes an autoencoder (AE) type of neural network is proposed. The proposed framework consists of extrasystole occurrence detection from the RRI data and modification, whose targets are PVC and PAC. The RRI data are monitored by means of the AE in real time in the detection phase, and a denoising autoencoder (DAE) modifies the ectopic RRI caused by the detected extrasystole. These are referred to as AE-based extrasystole detection (AED) and DAE-based extrasystole modification (DAEM), respectively. The proposed framework was applied to real RRI data with PVC and PAC. The result showed that AED achieved a sensitivity of 93% and a false positive rate of 0.08 times per hour. The root mean squared error of the modified RRI decreased to 31% in PVC and 73% in PAC from the original RRI data by DAEM. In addition, the proposed framework was validated through application to a clinical epileptic seizure problem, which showed that it correctly suppressed the false positives caused by PVC. Thus, the proposed framework can contribute to realizing accurate HRV-based health monitoring and medical sensing systems.

Identification of Pilots' Fatigue Status Based on Electrocardiogram Signals

Fatigue is an important factor affecting modern flight safety. It can easily lead to a decline in pilots’ operational ability, misjudgments, and flight illusions. Moreover, it can even trigger serious flight accidents. In this paper, a wearable wireless physiological device was used to obtain pilots’ electrocardiogram (ECG) data in a simulated flight experiment, and 1440 effective samples were determined. The Friedman test was adopted to select the characteristic indexes that reflect the fatigue state of the pilot from the time domain, frequency domain, and non-linear characteristics of the effective samples. Furthermore, the variation rules of the characteristic indexes were analyzed. Principal component analysis (PCA) was utilized to extract the features of the selected feature indexes, and the feature parameter set representing the fatigue state of the pilot was established. For the study on pilots’ fatigue state identification, the feature parameter set was used as the input of the learning vector quantization (LVQ) algorithm to train the pilots’ fatigue state identification model. Results show that the recognition accuracy of the LVQ model reached 81.94%, which is 12.84% and 9.02% higher than that of traditional back propagation neural network (BPNN) and support vector machine (SVM) model, respectively. The identification model based on the LVQ established in this paper is suitable for identifying pilots’ fatigue states. This is of great practical significance to reduce flight accidents caused by pilot fatigue, thus providing a theoretical foundation for pilot fatigue risk management and the development of intelligent aircraft autopilot systems.

Non-Contact Heart-Rate Measurement Method Using Both Transmitted Wave Extraction and Wavelet Transform

Continuous monitoring of heart-rate is expected to lead to early detection of physical discomfort. In this study, we propose a non-contact heart-rate measurement method which can be used in an environment such as driver heart-rate monitoring with body movement. The method is based on the electric field strength transmitted through the human body that changes with the diastole and systole of the heart. Unlike conventional displacement detection of the skin surface, we attempted to capture changes in the internal structure of the human body by irradiating the human body with microwaves and acquiring microwaves that pass through the heart. We first estimated the electric field strength transmitted through the heart using three receiving sensors to reduce the body movement effect. Then we decomposed the estimated transmitted electric field using stationary wavelet transform to eliminate significant distortion due to body movement. As a result, we achieved an estimation accuracy of heart-rate as high as 98% in a verification experiment with normal body movement.

The Design of CNN Architectures for Optimal Six Basic Emotion Classification Using Multiple Physiological Signals

This study aimed to design an optimal emotion recognition method using multiple physiological signal parameters acquired by bio-signal sensors for improving the accuracy of classifying individual emotional responses. Multiple physiological signals such as respiration (RSP) and heart rate variability (HRV) were acquired in an experiment from 53 participants when six basic emotion states were induced. Two RSP parameters were acquired from a chest-band respiration sensor, and five HRV parameters were acquired from a finger-clip blood volume pulse (BVP) sensor. A newly designed deep-learning model based on a convolutional neural network (CNN) was adopted for detecting the identification accuracy of individual emotions. Additionally, the signal combination of the acquired parameters was proposed to obtain high classification accuracy. Furthermore, a dominant factor influencing the accuracy was found by comparing the relativeness of the parameters, providing a basis for supporting the results of emotion classification. The users of this proposed model will soon be able to improve the emotion recognition model further based on CNN using multimodal physiological signals and their sensors.

Obstructive sleep apnea screening by heart rate variability-based apnea/normal respiration discriminant model

OBJECTIVE

Obstructive sleep apnea (OSA) is a common sleep disorder; however, most patients are undiagnosed and untreated because it is difficult for patients themselves to notice OSA in daily living. Polysomnography (PSG), which is the gold standard test for sleep disorder diagnosis, cannot be performed in many hospitals. This fact motivates us to develop a simple system for screening OSA at home.

APPROACH

The autonomic nervous system changes during apnea, and such changes affect heart rate variability (HRV). This work develops a new apnea screening method based on HRV analysis and machine learning technologies. An apnea/normal respiration (A/N) discriminant model is built for respiration condition estimation for every heart rate measurement, and an apnea/sleep ratio is introduced for final diagnosis. A random forest is adopted for the A/N discriminant model construction, which is trained with the PhysioNet apnea-ECG database.

MAIN RESULTS

The screening performance of the proposed method was evaluated by applying it to clinical PSG data. Sensitivity and specificity achieved 76% and 92%, respectively, which are comparable to existing portable sleep monitoring devices used in sleep laboratories.

SIGNIFICANCE

Since the proposed OSA screening method can be used more easily than existing devices, it will contribute to OSA treatment.

Low-Noise Photoplethysmography Sensor Using Correlated Double Sampling for Heartbeat Interval Acquisition

This study designs a low-power photoplethysmography (PPG) sensor based on the error compensation method for heartbeat interval acquisition. To perform heartbeat monitoring in daily life, it is necessary to obtain long-term and accurate heartbeat interval data with low power consumption, because of the limited size and battery capacity of the PPG sensor. Effective reduction in the power consumption of the sensor requires the duty-cycled LEDs and lowering pulse repetition frequency (PRF), i.e., decreasing the sampling rate. However, these methods reduce the accuracy of the heartbeat interval measurement because of signal-to-noise ratio (SNR) degradation and sampling errors. We propose an algorithm for heartbeat interval error compensation and incorporate a low-noise readout circuit to improve SNR. The readout circuit uses current integration to achieve low duty-cycle LED driving. A correlated double sampling (CDS) is introduced to minimize the random noise arising from the switching operation of the integration circuit. An error compensation method based on the PPG waveform similarity is also introduced using the autocorrelation and linear interpolation. The measurement results obtained from nine subjects show that a total current consumption of 28.2 μA is achieved with a 20-Hz PRF and 0.3% LED duty cycle. The proposed design effectively reduces the mean absolute error (MAE) of the heartbeat interval to an average of 6.2 ms.

Classification of Drowsiness Levels Based on a Deep Spatio-Temporal Convolutional Bidirectional LSTM Network Using Electroencephalography Signals

Non-invasive brain-computer interfaces (BCI) have been developed for recognizing human mental states with high accuracy and for decoding various types of mental conditions. In particular, accurately decoding a pilot's mental state is a critical issue as more than 70% of aviation accidents are caused by human factors, such as fatigue or drowsiness. In this study, we report the classification of not only two mental states (i.e., alert and drowsy states) but also five drowsiness levels from electroencephalogram (EEG) signals. To the best of our knowledge, this approach is the first to classify drowsiness levels in detail using only EEG signals. We acquired EEG data from ten pilots in a simulated night flight environment. For accurate detection, we proposed a deep spatio-temporal convolutional bidirectional long short-term memory network (DSTCLN) model. We evaluated the classification performance using Karolinska sleepiness scale (KSS) values for two mental states and five drowsiness levels. The grand-averaged classification accuracies were 0.87 (±0.01) and 0.69 (±0.02), respectively. Hence, we demonstrated the feasibility of classifying five drowsiness levels with high accuracy using deep learning.

Feature Extraction and Evaluation for Driver Drowsiness Detection Based on Thermoregulation

Numerous reports state that drowsiness is one of the major factors affecting driving performance and resulting in traffic accidents. In the past, methods to detect driver drowsiness have been developed based on physiological, behavioral, and vehicular features. In this pilot study, we test the use of a new set of features for detecting driver drowsiness based on physiological changes related to thermoregulation. Nineteen participants successfully performed a driving simulation, while the temperature of the nose (Tnose) and wrist (Twrist) as well as the heart rate (HR) were monitored. On average, an initial increase in temperature followed by a gradual decrease was observed in drivers who experienced drowsiness. For non-drowsy drivers, no such trends were observed. In addition, HR decreased on average in both groups, yet the decrease in the drowsy group was more distinct. Next, a classification based on each of these variables resulted in an accuracy of 68.4%, 88.9%, and 70.6% for Tnose, Twrist, and HR, respectively. Combining the information of all variables resulted in an accuracy of 89.5%, meaning that ultimately the state of 17 out of 19 drivers was detected correctly. Hence, we conclude that the use of physiological features related to thermoregulation shows potential for future research in this field.

Ad-Hoc Shallow Neural Network to Learn Hyper Filtered PhotoPlethysmoGraphic (PPG) Signal for Efficient Car-Driver Drowsiness Monitoring

In next-generation cars, safety equipment related to assisted driving systems commonly known as ADAS (advanced driver-assistance systems) are of particular interest for the major car-makers. When we talk about the “ADAS system”, we mean the devices and sensors having the precise objective of improving and making car driving safer, and among which it is worth mentioning rain sensors, the twilight sensor, adaptive cruise control, automatic emergency braking, parking sensors, automatic signal recognition, and so on. All these devices and sensors are installed on the new homologated cars to minimize the risk of an accident and make life on board of the car easier. Some sensors evaluate the movement and the opening of the eyes, the position of the head and its angle, or some physiological signals of the driver obtainable from the palm of the hands placed in the steering. In the present contribution, the authors will present an innovative recognition and monitoring system of the driver’s attention level through the study of the photoplethysmographic (PPG) signal detectable from the palm of the driver’s hands through special devices housed in the steering of the car. Through a particular and innovative post-processing algorithm of the PPG signal through a hyper-filtering framework, then processed by a machine learning framework, the entire pipeline proposed will be able to recognize and monitor the attention level of the driver with high accuracy and acceptable timing.

Missing RRI Interpolation Algorithm based on Locally Weighted Partial Least Squares for Precise Heart Rate Variability Analysis

The R-R interval (RRI) fluctuation in electrocardiogram (ECG) is called heart rate variability (HRV), which reflects activities of the autonomic nervous system (ANS) and has been used for various health monitoring services. Accurate R wave detection is crucial for success in HRV-based health monitoring services; however, ECG artifacts often cause missing R waves and deteriorate the accuracy of HRV analysis. The present work proposes a new missing RRI interpolation technique based on Just-In-Time (JIT) modeling. In the JIT modeling framework, a local regression model is built by weighing samples stored in the database according to the distance from a query and output is estimated only when an estimate is requested. The proposed method builds a local model and estimates missing RRI only when an RRI detection error is detected. Locally weighted partial least squares (LWPLS) is adopted for local model construction. The proposed method is referred to as LWPLS-based RRI interpolation (LWPLS-RI). The performance of the proposed LWPLS-RI was evaluated through its application to RRI data with artificial missing RRIs. We used the MIT-BIH Normal Sinus Rhythm Database for nominal RRI dataset construction. Missing RRIs were artificially introduced and they were interpolated by the proposed LWPLS-RI. In addition, MEAN that replaces the missing RRI by a mean of the past RRI data was compared as a conventional method. The result showed that the proposed LWPLS-RI improved root mean squared error (RMSE) of RRI by about 70% in comparison with MEAN. In addition, the proposed method realized precise HRV analysis. The proposed method will contribute to the realization of precise HRV-based health monitoring services.