Detecting driver fatigue using heart rate variability: A systematic review

Exploring driving anger-caused impairment of takeover performance among professional taxi drivers during partially automated driving

Partially automated systems are expected to reduce road crashes related to human error, even amongst professional drivers. Consequently, the applications of these systems into the taxi industry would potentially improve transportation safety. However, taxi drivers are prone to experiencing driving anger, which may subsequently affect their takeover performance. In this research, we explored how driving anger emotion affects taxi drivers' driving performance in various takeover scenarios, namely Mandatory Automation-Initiated transition (MAIT), Mandatory Driver-Initiated transition (MDIT), and Optional Driver-Initiated transition (ODIT). Forty-seven taxi drivers participated in this 2·3 mixed design simulator experiment (between-subjects: anger vs. calmness; within-subjects: MAIT vs. MDIT vs. ODIT). Compared to calmness, driving anger emotion led to a narrower field of attention (e.g., smaller standard deviations of horizontal fixation points position) and worse hazard perception (e.g., longer saccade latency, smaller amplitude of skin conductance responses), which resulted in longer takeover time and inferior vehicle control stability (e.g., higher standard deviations of lateral position) in MAIT and MDIT scenarios. Angry taxi drivers were more likely to deactivate vehicle automation and take over the vehicle in a more aggressive manner (e.g., higher maximal resulting acceleration, refusing to yield to other road users) in ODIT scenarios. The findings will contribute to addressing the safety concerns related to driving anger among professional taxi drivers and promote the widespread acceptance and integration of partially automated systems within the taxi industry.

Autonomic and cognitive control in memory: Investigating the psychophysiological link using heart rate variability biofeedback

Vagally mediated heart rate variability (vmHRV) at resting state has been associated to cognitive functions dependent on cognitive control, such as memory. However, little is known about the phasic interaction between cognitive and autonomic control. In a pre-registered within-between-subject designed experiment, the potential of vmHRV biofeedback to simultaneously stimulate vmHRV during memory processing and cognitive control over long-term memory was tested, along with investigating psychophysiological association. 71 young healthy adults completed (twice) a false memory task in virtual reality. Immediately before memory encoding and retrieval, participants practiced either vmHRV biofeedback or a control breathing exercise. Cognitive control over long-term memory was assessed as the confidence toward false memories and the capability to discriminate them from true memories. Resting-state vmHRV before each test and phasic vmHRV during memory encoding and retrieval were measured as the root mean square differences (RMSSD) in the heart period. vmHRV biofeedback had neither an immediate effect on measures of cognitive control over long-term memory nor on phasic RMSSD. Moreover, neither resting-state nor phasic vmHRV correlated to the cognitive scores. Consequently, the utility of HRV biofeedback as a psychophysiological stimulation tool and a link between vmHRV and cognitive control over long-term memory could not be verified. Exploratory analyses revealed that baseline shift in parasympathetic activity confounded the psychophysiological association. Future directions are provided that could shed light on the relationship between cognition and vmHRV.

Driver Fatigue Detection Using Heart Rate Variability Features from 2-Minute Electrocardiogram Signals While Accounting for Sex Differences

Traffic accidents due to fatigue account for a large proportion of road fatalities. Based on simulated driving experiments with drivers recruited from college students, this paper investigates the use of heart rate variability (HRV) features to detect driver fatigue while considering sex differences. Sex-independent and sex-specific differences in HRV features between alert and fatigued states derived from 2 min electrocardiogram (ECG) signals were determined. Then, decision trees were used for driver fatigue detection using the HRV features of either all subjects or those of only males or females. Nineteen, eighteen, and thirteen HRV features were significantly different (Mann-Whitney U test, p < 0.01) between the two mental states for all subjects, males, and females, respectively. The fatigue detection models for all subjects, males, and females achieved classification accuracies of 86.3%, 94.8%, and 92.0%, respectively. In conclusion, sex differences in HRV features between drivers' mental states were found according to both the statistical analysis and classification results. By considering sex differences, precise HRV feature-based driver fatigue detection systems can be developed. Moreover, in contrast to conventional methods using HRV features from 5 min ECG signals, our method uses HRV features from 2 min ECG signals, thus enabling more rapid driver fatigue detection.

Detecting and Predicting Pilot Mental Workload Using Heart Rate Variability: A Systematic Review

Measuring pilot mental workload (MWL) is crucial for enhancing aviation safety. However, MWL is a multi-dimensional construct that could be affected by multiple factors. Particularly, in the context of a more automated cockpit setting, the traditional methods of assessing pilot MWL may face challenges. Heart rate variability (HRV) has emerged as a potential tool for detecting pilot MWL during real-flight operations. This review aims to investigate the relationship between HRV and pilot MWL and to assess the performance of machine-learning-based MWL detection systems using HRV parameters. A total of 29 relevant papers were extracted from three databases for review based on rigorous eligibility criteria. We observed significant variability across the reviewed studies, including study designs and measurement methods, as well as machine-learning techniques. Inconsistent results were observed regarding the differences in HRV measures between pilots under varying levels of MWL. Furthermore, for studies that developed HRV-based MWL detection systems, we examined the diverse model settings and discovered that several advanced techniques could be used to address specific challenges. This review serves as a practical guide for researchers and practitioners who are interested in employing HRV indicators for evaluating MWL and wish to incorporate cutting-edge techniques into their MWL measurement approaches.

Development of sleepiness in professional truck drivers: Real-road testing for driver drowsiness and attention warning (DDAW) system evaluation

All new vehicle types within the European Union must now be equipped with a driver drowsiness and attention warning system starting from 2022. The specific requirements for the test procedure necessary for type approval are defined in the Annex of EU Regulation C/2021/2639. The objectives of this study were to: (i) investigate how sleepiness develops in professional truck drivers under real-road driving conditions; and (ii) assess the feasibility of a test procedure for validating driver drowsiness and attention warning systems according to the EU regulation. Twenty-four professional truck drivers participated in the test. They drove for 180 km on a dual-lane motorway, first during daytime after a normal night's sleep and then at nighttime after being awake since early morning. The results showed higher sleepiness levels during nighttime driving compared with daytime, with a faster increase in sleepiness with distance driven, especially during the night. Psychomotor vigilance task results corroborated these findings. From a driver drowsiness and attention warning testing perspective, the study design with sleep-deprived drivers at night was successful in inducing the targeted sleepiness level of a Karolinska Sleepiness Scale score of ≥ 8. Many drivers who reported a Karolinska Sleepiness Scale ≥ 8 during the drives also acknowledged feeling sleepy in the post-drive questionnaire. Reaching high levels of sleepiness on real roads during daytime is more problematic, not the least from legal and ethical perspectives as higher traffic densities during the daytime lead to increased risks.

Predicting Fatigue-Associated Aberrant Driving Behaviors Using a Dynamic Weighted Moving Average Model With a Long Short-Term Memory Network Based on Heart Rate Variability

OBJECTIVE

This study proposed a moving average (MA) approach to dynamically process heart rate variability (HRV) and developed aberrant driving behavior (ADB) prediction models by using long short-term memory (LSTM) networks.

BACKGROUND

Fatigue-associated ADBs have traffic safety implications. Numerous models to predict such acts based on physiological responses have been developed but are still in embryonic stages.

METHOD

This study recorded the data of 20 commercial bus drivers during their routine tasks on four consecutive days and subsequently asked them to complete questionnaires, including subjective sleep quality, driver behavior questionnaire and the Karolinska Sleepiness Scale. Driving behaviors and corresponding HRV were determined using a navigational mobile application and a wristwatch. The dynamic-weighted MA (DWMA) and exponential-weighted MA were used to process HRV in 5-min intervals. The data were independently separated for training and testing. Models were trained with 10-fold cross-validation strategy, their accuracies were evaluated, and Shapley additive explanation (SHAP) values were used to determine feature importance.

RESULTS

Significant increases in the standard deviation of NN intervals (SDNN), root mean square of successive heartbeat interval differences (RMSSD), and normalized spectrum of high frequency (nHF) were observed in the pre-event stage. The DWMA-based model exhibited the highest accuracy for both driver types (urban: 84.41%; highway: 80.56%). The SDNN, RMSSD, and nHF demonstrated relatively high SHAP values.

CONCLUSION

HRV metrics can serve as indicators of mental fatigue. DWMA-based LSTM could predict the occurrence of the level of fatigue associated with ADBs.

APPLICATION

The established models can be used in realistic driving scenarios.

Drinking and driving: A systematic review of the impacts of alcohol consumption on manual and automated driving performance

INTRODUCTION

Almost a third of car accidents involve driving after alcohol consumption. Autonomous vehicles (AVs) may offer accident-prevention benefits, but at current automation levels, drivers must still perform manual driving tasks when automated systems fail. Therefore, understanding how alcohol affects driving in both manual and automated contexts offers insight into the role of future vehicle design in mediating crash risks for alcohol-impaired driving.

METHOD

This study conducted a systematic review on alcohol effects on manual and automated (takeover) driving performance. Fifty-three articles from eight databases were analyzed, with findings structured based on the information processing model, which can be extended to the AV takeover model.

RESULTS

The literature indicates that different Blood Alcohol Concentration (BAC) levels affect driving skills essential for traffic safety at various information processing stages, such as delayed reacting time, impaired cognitive abilities, and hindered execution of driving tasks. Additionally, the driver's driving experience, drinking habits, and external driving environment play important roles in influencing driving performance.

CONCLUSIONS

Future work is needed to examine the effects of alcohol on driving performance, particularly in AVs and takeover situations, and to develop driver monitoring systems.

PRACTICAL APPLICATIONS

Findings from this review can inform future experiments, AV technology design, and the development of driver state monitoring systems.

Understanding patterns of fatigue in health and disease: protocol for an ecological momentary assessment study using digital technologies

INTRODUCTION

Fatigue is prevalent across a wide range of medical conditions and can be debilitating and distressing. It is likely that fatigue is experienced differently according to the underlying aetiology, but this is poorly understood. Digital health technologies present a promising approach to give new insights into fatigue.The aim of this study is to use digital health technologies, real-time self-reports and qualitative interview data to investigate how fatigue is experienced over time in participants with myeloma, long COVID, heart failure and in controls without problematic fatigue. Objectives are to understand which sensed parameters add value to the characterisation of fatigue and to determine whether study processes are feasible, acceptable and scalable.

METHODS AND ANALYSIS

An ecological momentary assessment study will be carried out over 2 or 4 weeks (participant defined). Individuals with fatigue relating to myeloma (n=10), heart failure (n=10), long COVID (n=10) and controls without problematic fatigue or a study condition (n=10) will be recruited. ECG patches will measure heart rate variability, respiratory rate, body temperature, activity and posture. A wearable bracelet accompanied by environment beacons will measure physical activity, sleep and room location within the home. Self-reports of mental and physical fatigue will be collected via smartphone app four times daily and on-demand. Validated fatigue and affect questionnaires will be completed at baseline and at 2 weeks. End-of-study interviews will investigate experiences of fatigue and study participation. A feedback session will be offered to participants to discuss their data.Data will be analysed using multilevel modelling and machine learning. Interviews and feedback sessions will be analysed using content or thematic analyses.

ETHICS AND DISSEMINATION

This study was approved by the East of England-Cambridge East Research Ethics Committee (22/EE/0261). The results will be disseminated in peer-reviewed journals and at international conferences.

TRIAL REGISTRATION NUMBER

NCT05622669.

Cardiac indices of driver fatigue across in-lab and on-road studies

Driver fatigue is a major contributor to road accidents. Therefore, driver assistance systems (DAS) that would monitor drivers' states may contribute to road safety. Such monitoring can potentially be achieved with input from ECG indices (e.g., heart rate). We reviewed the empirical literature on responses of cardiac measures to driver fatigue and on detecting fatigue with cardiac indices and classification algorithms. We used meta-analytical methods to explore the pooled effect sizes of different cardiac indices of fatigue, their heterogeneity, and the consistency of their responses across studies. Our large pool of studies (N = 39) allowed us to stratify the results across on-road and simulator studies. We found that despite the large heterogeneity of the effect sizes between the studies, many indices had significant pooled effect sizes across the studies, and more frequently across the on-road studies. We also found that most indices showed consistent responses across both on-road and simulator studies. Regarding the detection accuracy, we found that even on-road classification could have been as accurate as 70% with only 2-min of data. However, we could only find two on-road studies that employed fatigue classification algorithms. Overall, our findings are encouraging with respect to the prospect of using cardiac measures for detecting driver fatigue. Yet, to fully explore this possibility, there is a need for additional on-road studies that would employ a similar set of cardiac indices and detection algorithms, a unified definition of fatigue, and additional levels of fatigue than the two fatigue vs alert states.

Fatigue at the wheel: A non-visual approach to truck driver fatigue detection by multi-feature fusion

BACKGROUND

Monitoring of long-haul truck driver fatigue state has attracted considerable interest. Conventional fatigue driving detection methods based on the physiological and visual features are scarcely applicable, due to the intrusiveness, reliability, and cost-effectiveness concerns.

METHODS

We elaborately developed a fatigue driving detection method by fusion of non-visual features derived from the customized wristbands, vehicle-mounted equipment, and trip logs. To capture the spatiotemporal information within the sequential data, the bidirectional long short-term memory network with attention mechanism was proposed to determine whether the truck driver was fatigued within a fine-grained episode of one minute. The model was validated using a natural driving dataset with nine truck drivers on real-world roads in Guiyang, China during June and July 2021.

RESULTS

Our approach yielded 99.21 %, 84.44 %, 82.01 %, 99.63 %, and 83.21 % in accuracy, precision, recall, specificity, and F1-score, respectively. Compared with the mainstream visual-based methods, our approach outperformed particularly in terms of precision and recall. Photoplethysmogram stood out as the most important feature for truck driver fatigue state detection. Vehicle load, driving forward angle, cumulative driving time, midnight, and recent working hours were found to be positively associated with the probability of fatigue driving, while the galvanic skin response, vehicle acceleration, current time, and recent rest hours had a negative relationship. Specifically, truck drivers were more likely to fatigue when driving at 20-40 km/h, braking abruptly at 5-10 m/s2, with vehicle loads over 70 tons, and driving more than 100 min consecutively.

CONCLUSIONS

Our study is among the first to harness the natural driving dataset to delve into the real-life fatigue pattern of long-haul truck drivers without disruptions on routine driving tasks. The proposed method holds pragmatic prospects by providing a privacy-preserving, robust, real-time, and non-intrusive technical pathway for truck driver fatigue monitoring.

Remote Photoplethysmography and Motion Tracking Convolutional Neural Network with Bidirectional Long Short-Term Memory: Non-Invasive Fatigue Detection Method Based on Multi-Modal Fusion

Existing vision-based fatigue detection methods commonly utilize RGB cameras to extract facial and physiological features for monitoring driver fatigue. These features often include single indicators such as eyelid movement, yawning frequency, and heart rate. However, the accuracy of RGB cameras can be affected by factors like varying lighting conditions and motion. To address these challenges, we propose a non-invasive method for multi-modal fusion fatigue detection called RPPMT-CNN-BiLSTM. This method incorporates a feature extraction enhancement module based on the improved Pan-Tompkins algorithm and 1D-MTCNN. This enhances the accuracy of heart rate signal extraction and eyelid features. Furthermore, we use one-dimensional neural networks to construct two models based on heart rate and PERCLOS values, forming a fatigue detection model. To enhance the robustness and accuracy of fatigue detection, the trained model data results are input into the BiLSTM network. This generates a time-fitting relationship between the data extracted from the CNN, allowing for effective dynamic modeling and achieving multi-modal fusion fatigue detection. Numerous experiments validate the effectiveness of the proposed method, achieving an accuracy of 98.2% on the self-made MDAD (Multi-Modal Driver Alertness Dataset). This underscores the feasibility of the algorithm. In comparison with traditional methods, our approach demonstrates higher accuracy and positively contributes to maintaining traffic safety, thereby advancing the field of smart transportation.

Change in heart rate variability with increasing time-on-task as a marker for mental fatigue: A systematic review

Fatigue-specific changes in the autonomic nervous system are often assumed to underlie the development of mental fatigue caused by prolonged cognitive tasks (i.e. Time-on-Task). Therefore, several previous studies have chosen to investigate the Time-on-Task related changes in heart rate variability (HRV). However, previous studies have used many different HRV indices, and their results often show inconsistencies. The present study, therefore, systematically reviewed previous empirical HRV studies with healthy individuals and in which mental fatigue is induced by prolonged cognitive tasks. Articles relevant to the objectives were systematically searched and selected by applying the PRISMA guidelines. We screened 360 records found on 4 databases and found that 19 studies were eligible for full review in accordance with the inclusion criteria. In general, all studies reviewed (with the exception of two studies) found significant changes in HRV with increasing Time-on-Task, suggesting that HRV is a reliable autonomic marker for Time-on-Task induced fatigue. The most conclusive HRV indices that showed a consistent Time-on-Task effect were the low frequency component of HRV and the time domain indices, particularly the root mean square of successive differences. Time-on-Task typically induced an increasing trend in both type of measures.

Drowsiness Mitigation Through Driver State Monitoring Systems: A Scoping Review

OBJECTIVE

To explore the scope of available research and to identify research gaps on in-vehicle interventions for drowsiness that utilize driver monitoring systems (DMS).

BACKGROUND

DMS are gaining popularity as a countermeasure against drowsiness. However, how these systems can be best utilized to guide driver attention is unclear.

METHODS

A scoping review was conducted in adherence to PRISMA guidelines. Five electronic databases (ACM Digital Library, Scopus, IEEE Xplore, TRID, and SAE Mobilus) were systematically searched in April 2022. Original studies examining in-vehicle drowsiness interventions that use DMS in a driving context (e.g., driving simulator and driver interviews) passed the screening. Data on study details, state detection methods, and interventions were extracted.

RESULTS

Twenty studies qualified for inclusion. Majority of interventions involved warnings (n = 16) with an auditory component (n = 14). Feedback displays (n = 4) and automation takeover (n = 4) were also investigated. Multistage interventions (n = 12) first cautioned the driver, then urged them to take an action, or initiated an automation takeover. Overall, interventions had a positive impact on sleepiness levels, driving performance, and user evaluations. Whether interventions effective for one type of sleepiness (e.g., passive vs. active fatigue) will perform well for another type is unclear.

CONCLUSION

Literature mainly focused on developing sensors and improving the accuracy of DMS, but not on the driver interactions with these technologies. More intervention studies are needed in general and for investigating their long-term effects.

APPLICATION

We list gaps and limitations in the DMS literature to guide researchers and practitioners in designing and evaluating effective safety systems for drowsy driving.

Assessing the physiological effect of non-driving-related task performance and task modality in conditionally automated driving systems: A systematic review and meta-analysis

In conditionally automated driving, the driver is free to disengage from controlling the vehicle, but they are expected to resume driving in response to certain situations or events that the system is not equipped to respond to. As the level of vehicle automation increases, drivers often engage in non-driving-related tasks (NDRTs), defined as any secondary task unrelated to the primary task of driving. This engagement can have a detrimental effect on the driver's situation awareness and attentional resources. NDRTs with resource demands that overlap with the driving task, such as visual or manual tasks, may be particularly deleterious. Therefore, monitoring the driver's state is an important safety feature for conditionally automated vehicles, and physiological measures constitute a promising means of doing this. The present systematic review and meta-analysis synthesises findings from 32 studies concerning the effect of NDRTs on drivers' physiological responses, in addition to the effect of NDRTs with a visual or a manual modality. Evidence was found that NDRT engagement led to higher physiological arousal, indicated by increased heart rate, electrodermal activity and a decrease in heart rate variability. There was mixed evidence for an effect of both visual and manual NDRT modalities on all physiological measures. Understanding the relationship between task performance and arousal during automated driving is of critical importance to the development of driver monitoring systems and improving the safety of this technology.

EEG and ECG-Based Multi-Sensor Fusion Computing for Real-Time Fatigue Driving Recognition Based on Feedback Mechanism

A variety of technologies that could enhance driving safety are being actively explored, with the aim of reducing traffic accidents by accurately recognizing the driver's state. In this field, three mainstream detection methods have been widely applied, namely visual monitoring, physiological indicator monitoring and vehicle behavior analysis. In order to achieve more accurate driver state recognition, we adopted a multi-sensor fusion approach. We monitored driver physiological signals, electroencephalogram (EEG) signals and electrocardiogram (ECG) signals to determine fatigue state, while an in-vehicle camera observed driver behavior and provided more information for driver state assessment. In addition, an outside camera was used to monitor vehicle position to determine whether there were any driving deviations due to distraction or fatigue. After a series of experimental validations, our research results showed that our multi-sensor approach exhibited good performance for driver state recognition. This study could provide a solid foundation and development direction for future in-depth driver state recognition research, which is expected to further improve road safety.

ECG-Free Heartbeat Detection in Seismocardiography and Gyrocardiography Signals Provides Acceptable Heart Rate Variability Indices in Healthy and Pathological Subjects

Cardio-mechanical monitoring techniques, such as Seismocardiography (SCG) and Gyrocardiography (GCG), have received an ever-growing interest in recent years as potential alternatives to Electrocardiography (ECG) for heart rate monitoring. Wearable SCG and GCG devices based on lightweight accelerometers and gyroscopes are particularly appealing for continuous, long-term monitoring of heart rate and its variability (HRV). Heartbeat detection in cardio-mechanical signals is usually performed with the support of a concurrent ECG lead, which, however, limits their applicability in standalone cardio-mechanical monitoring applications. The complex and variable morphology of SCG and GCG signals makes the ECG-free heartbeat detection task quite challenging; therefore, only a few methods have been proposed. Very recently, a template matching method based on normalized cross-correlation (NCC) has been demonstrated to provide very accurate detection of heartbeats and estimation of inter-beat intervals in SCG and GCG signals of pathological subjects. In this study, the accuracy of HRV indices obtained with this template matching method is evaluated by comparison with ECG. Tests were performed on two public datasets of SCG and GCG signals from healthy and pathological subjects. Linear regression, correlation, and Bland-Altman analyses were carried out to evaluate the agreement of 24 HRV indices obtained from SCG and GCG signals with those obtained from ECG signals, simultaneously acquired from the same subjects. The results of this study show that the NCC-based template matching method allowed estimating HRV indices from SCG and GCG signals of healthy subjects with acceptable accuracy. On healthy subjects, the relative errors on time-domain indices ranged from 0.25% to 15%, on frequency-domain indices ranged from 10% to 20%, and on non-linear indices were within 8%. The estimates obtained on signals from pathological subjects were affected by larger errors. Overall, GCG provided slightly better performances as compared to SCG, both on healthy and pathological subjects. These findings provide, for the first time, clear evidence that monitoring HRV via SCG and GCG sensors without concurrent ECG is feasible with the NCC-based template matching method for heartbeat detection.

ECG-Free Heartbeat Detection in Seismocardiography Signals via Template Matching

Cardiac monitoring can be performed by means of an accelerometer attached to a subject's chest, which produces the Seismocardiography (SCG) signal. Detection of SCG heartbeats is commonly carried out by taking advantage of a simultaneous electrocardiogram (ECG). SCG-based long-term monitoring would certainly be less obtrusive and easier to implement without an ECG. Few studies have addressed this issue using a variety of complex approaches. This study proposes a novel approach to ECG-free heartbeat detection in SCG signals via template matching, based on normalized cross-correlation as heartbeats similarity measure. The algorithm was tested on the SCG signals acquired from 77 patients with valvular heart diseases, available from a public database. The performance of the proposed approach was assessed in terms of sensitivity and positive predictive value (PPV) of the heartbeat detection and accuracy of inter-beat intervals measurement. Sensitivity and PPV of 96% and 97%, respectively, were obtained by considering templates that included both systolic and diastolic complexes. Regression, correlation, and Bland-Altman analyses carried out on inter-beat intervals reported slope and intercept of 0.997 and 2.8 ms (R² > 0.999), as well as non-significant bias and limits of agreement of ±7.8 ms. The results are comparable or superior to those achieved by far more complex algorithms, also based on artificial intelligence. The low computational burden of the proposed approach makes it suitable for direct implementation in wearable devices.