A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals

Cognitive workload classification of law enforcement officers using physiological responses

Motor vehicle crashes (MVCs) are a leading cause of death for law enforcement officers (LEOs) in the U.S. LEOs and more specifically novice LEOs (nLEOs) are susceptible to high cognitive workload while driving which can lead to fatal MVCs. The objective of this study was to develop a machine learning algorithm (MLA) that can estimate cognitive workload of LEOs while performing secondary tasks in a patrol vehicle. A ride-along study was conducted with 24 nLEOs. Participants performed their normal patrol operations while their physiological responses such as heartrate, eye movement, and galvanic skin response were recorded using unobtrusive devices. Findings suggested that the random forest algorithm could predict cognitive workload with relatively high accuracy (>70%) given that it was entirely reliant on physiological signals. The developed MLA can be used to develop adaptive in-vehicle technology based on real-time estimation of cognitive workload, which can reduce the risk of MVCs in police operations.

Detection of driver drowsiness level using a hybrid learning model based on ECG signals

OBJECTIVES

Fatigue has a considerable impact on the driver's vehicle and even the driver's own operating ability.

METHODS

An intelligent algorithm is proposed for the problem that it is difficult to classify the degree of drowsiness generated by the driver during the driving process. By studying the driver's electrocardiogram (ECG) during driving, two models were established to jointly classify the ECG signals as awake, stress, and fatigue or drowsiness states for drowsiness levels. Firstly, the deep learning method was used to establish the model_1 to predict the drowsiness of the original ECG, and model_2 was developed using the combination of principal component analysis (PCA) and weighted K-nearest neighbor (WKNN) algorithm to classify the heart rate variability characteristics. Then, the drowsiness prediction results of the two models were weighted according to certain rules, and the hybrid learning model combining dilated convolution and bidirectional long short-term memory network with PCA and WKNN algorithm was established, and the mixed model was denoted as DiCNN-BiLSTM and PCA-WKNN (DBPW). Finally, the validity of the DBPW model was verified by simulation of the public database.

RESULTS

The experimental results show that the average accuracy, sensitivity and F1 score of the test model in the dataset containing multiple drivers are 98.79, 98.81, and 98.79 % respectively, and the recognition accuracy for drowsiness or drowsiness state is 99.33 %.

CONCLUSIONS

Using the proposed algorithm, it is possible to identify driver anomalies and provide new ideas for the development of intelligent vehicles.

Contactless Electrostatic Piloerection for Haptic Sensations

In this project, we create artificial piloerection using contactless electrostatics to induce tactile sensations in a contactless way. Firstly, we design various high-voltage generators and evaluate them in terms of their static charge, safety and frequency response with different electrodes as well as grounding strategies. Secondly, a psychophysics user study revealed which parts of the upper body are more sensitive to electrostatic piloerection and what adjectives are associated with them. Finally, we combine an electrostatic generator to produce artificial piloerection on the nape with a head-mounted display, this device provides an augmented virtual experience related to fear. We hope that work encourages designers to explore contactless piloerection for enhancing experiences such as music, short movies, video games, or exhibitions.

Assessment of Drivers' Mental Workload by Multimodal Measures during Auditory-Based Dual-Task Driving Scenarios

Assessing drivers' mental workload is crucial for reducing road accidents. This study examined drivers' mental workload in a simulated auditory-based dual-task driving scenario, with driving tasks as the main task, and auditory-based N-back tasks as the secondary task. A total of three levels of mental workload (i.e., low, medium, high) were manipulated by varying the difficulty levels of the secondary task (i.e., no presence of secondary task, 1-back, 2-back). Multimodal measures, including a set of subjective measures, physiological measures, and behavioral performance measures, were collected during the experiment. The results showed that an increase in task difficulty led to increased subjective ratings of mental workload and a decrease in task performance for the secondary N-back tasks. Significant differences were observed across the different levels of mental workload in multimodal physiological measures, such as delta waves in EEG signals, fixation distance in eye movement signals, time- and frequency-domain measures in ECG signals, and skin conductance in EDA signals. In addition, four driving performance measures related to vehicle velocity and the deviation of pedal input and vehicle position also showed sensitivity to the changes in drivers' mental workload. The findings from this study can contribute to a comprehensive understanding of effective measures for mental workload assessment in driving scenarios and to the development of smart driving systems for the accurate recognition of drivers' mental states.

An analysis of physiological responses as indicators of driver takeover readiness in conditionally automated driving

By the year 2045, it is projected that Autonomous Vehicles (AVs) will make up half of the new vehicle market. Successful adoption of AVs can reduce drivers' stress and fatigue, curb traffic congestion, and improve safety, mobility, and economic efficiency. Due to the limited intelligence in relevant technologies, human-in-the-loop modalities are still necessary to ensure the safety of AVs at current or near future stages, because the vehicles may not be able to handle all emergencies. Therefore, it is important to know the takeover readiness of the drivers to ensure the takeover quality and avoid any potential accidents. To achieve this, a comprehensive understanding of the drivers' physiological states is crucial. However, there is a lack of systematic analysis of the correlation between different human physiological responses and takeover behaviors which could serve as important references for future studies to determine the types of data to use. This paper provides a comprehensive analysis of the effects of takeover behaviors on the common physiological indicators. A program for conditional automation was developed based on a game engine and applied to a driving simulator. The experiment incorporated three types of secondary tasks, three takeover events, and two traffic densities. Brain signals, Skin Conductance Level (SCL), and Heart Rate (HR) of the participants were collected while they were performing the driving simulations. The Frontal Asymmetry Index (FAI) (as an indicator of engagement) and Mental Workload (MWL) were calculated from the brain signals to indicate the mental states of the participants. The results revealed that the FAI of the drivers would slightly decrease after the takeover alerts were issued when they were doing secondary tasks prior to the takeover activities, and the higher difficulty of the secondary tasks could lead to lower overall FAI during the takeover periods. In contrast, The MWL and SCL increased during the takeover periods. The HR also increased rapidly at the beginning of the takeover period but dropped back to a normal level quickly. It was found that a fake takeover alert would lead to lower overall HR, slower increase, and lower peak of SCL during the takeover periods. Moreover, the higher traffic density scenarios were associated with higher MWL, and a more difficult secondary task would lead to higher MWL and HR during the takeover activities. A preliminary discussion of the correlation between the physiological data, takeover scenario, and vehicle data (that relevant to takeover readiness) was then conducted, revealing that although takeover event, SCL, and HR had slightly higher correlations with the maximum acceleration and reaction time, none of them dominated the takeover readiness. In addition, the analysis of the data across different participants was conducted, which emphasized the importance of considering standardization or normalization of the data when they were further used as input features for estimating takeover readiness. Overall, the results presented in this paper offer profound insights into the patterns of physiological data changes during takeover periods. These findings can be used as benchmarks for utilizing these variables as indicators of takeover preparedness and performance in future research endeavors.

A Review of Recurrent Neural Network-Based Methods in Computational Physiology

Artificial intelligence and machine learning techniques have progressed dramatically and become powerful tools required to solve complicated tasks, such as computer vision, speech recognition, and natural language processing. Since these techniques have provided promising and evident results in these fields, they emerged as valuable methods for applications in human physiology and healthcare. General physiological recordings are time-related expressions of bodily processes associated with health or morbidity. Sequence classification, anomaly detection, decision making, and future status prediction drive the learning algorithms to focus on the temporal pattern and model the nonstationary dynamics of the human body. These practical requirements give birth to the use of recurrent neural networks (RNNs), which offer a tractable solution in dealing with physiological time series and provide a way to understand complex time variations and dependencies. The primary objective of this article is to provide an overview of current applications of RNNs in the area of human physiology for automated prediction and diagnosis within different fields. Finally, we highlight some pathways of future RNN developments for human physiology.

Predator-prey systems as models for integrative research in biology: the value of a non-consumptive effects framework

Predator-prey interactions are a cornerstone of many ecological and evolutionary processes that influence various levels of biological organization, from individuals to ecosystems. Predators play a crucial role in shaping ecosystems through the consumption of prey species and non-consumptive effects. Non-consumptive effects (NCEs) can induce changes in prey behavior, including altered foraging strategies, habitat selection, life history and anti-predator responses. These defensive strategies have physiological consequences for prey, affecting their growth, reproduction and immune function to name a few. Numerous experimental studies have incorporated NCEs in investigating predator-prey dynamics in the past decade. Interestingly, predator-prey systems can also be used as experimental models to answer physiology, cognition and adaptability questions. In this Commentary, I highlight research that uses NCEs in predator-prey systems to provide novel insights into cognition, adaptation, epigenetic inheritance and aging. I discuss the evolution of instinct, anxiety and other cognitive disorders, the shaping of brain connectomes, stress-induced aging and the development of behavioral coping styles. I outline how studies can integrate the investigation of NCEs with advanced behavioral, genomic and neurological tools to provide novel insights into physiological and cognitive health.

Analysis of Driving Fatigue Characteristics in Cold and Hypoxia Environment of High-Altitude Areas

The cold and hypoxic environment at high altitudes can easily lead to driving fatigue. For improving highway safety in high-altitude areas, a driver fatigue test is conducted using the Kangtai PM-60A car heart rate and oxygen tester to collect drivers' heart rate oximetry in National Highway 214 in Qinghai Province. Standard deviation (SDNN), mean (M), coefficient of RR (two R heart rate waves), RR interval coefficient of variation (RRVC), and cumulative rate of driving fatigue based on the driver's heart rate RR interval are calculated using SPSS. This study aims to derive degree of driving fatigue (DFD) in high-altitude areas when driving from lower to higher altitude. The analysis shows that the DFD growth trend of different altitude ranges presents an S-shaped curve. The driving fatigue thresholds in the altitude range of 3000-3500, 3500-4000, 4000-4500, and 4500-5000 m are 2.86, 3.82, 4.54, and 10.2, which are significantly higher than that of ordinary roads in plain areas. The start times of severe fatigue in the four altitude ranges are 35, 34, 32, and 25 minutes. The start time of driving fatigue continued to advance with the increase of age, and the DFD continued to increase with the increase of age. Results provide an empirical basis for the design of the horizontal alignment index system and antifatigue strategies to improve highway safety in high-altitude areas.

Assessment of Vigilance Level during Work: Fitting a Hidden Markov Model to Heart Rate Variability

This study aimed to enhance the real-time performance and accuracy of vigilance assessment by developing a hidden Markov model (HMM). Electrocardiogram (ECG) signals were collected and processed to remove noise and baseline drift. A group of 20 volunteers participated in the study. Their heart rate variability (HRV) was measured to train parameters of the modified hidden Markov model for a vigilance assessment. The data were collected to train the model using the Baum-Welch algorithm and to obtain the state transition probability matrix A^ and the observation probability matrix B^. Finally, the data of three volunteers with different transition patterns of mental state were selected randomly and the Viterbi algorithm was used to find the optimal state, which was compared with the actual state. The constructed vigilance assessment model had a high accuracy rate, and the accuracy rate of data prediction for these three volunteers exceeded 80%. Our approach can be used in wearable products to improve their vigilance level assessment functionality or in other fields that have key positions with high concentration requirements and monotonous repetitive work.

Machine Learning Techniques for Arousal Classification from Electrodermal Activity: A Systematic Review

This article introduces a systematic review on arousal classification based on electrodermal activity (EDA) and machine learning (ML). From a first set of 284 articles searched for in six scientific databases, fifty-nine were finally selected according to various criteria established. The systematic review has made it possible to analyse all the steps to which the EDA signals are subjected: acquisition, pre-processing, processing and feature extraction. Finally, all ML techniques applied to the features of these signals for arousal classification have been studied. It has been found that support vector machines and artificial neural networks stand out within the supervised learning methods given their high-performance values. In contrast, it has been shown that unsupervised learning is not present in the detection of arousal through EDA. This systematic review concludes that the use of EDA for the detection of arousal is widely spread, with particularly good results in classification with the ML methods found.

Galvanic Skin Response Features in Psychiatry and Mental Disorders: A Narrative Review

This narrative review is aimed at presenting the galvanic skin response (GSR) Biofeedback method and possibilities for its application in persons with mental disorders as a modern form of neurorehabilitation. In the treatment of mental disorders of various backgrounds and courses, attention is focused on methods that would combine pharmacological treatment with therapies improving functioning. Currently, the focus is on neuronal mechanisms which, being physiological markers, offer opportunities for correction of existing deficits. One such indicator is electrodermal activity (EDA), providing information about emotions, cognitive processes, and behavior, and thus, about the function of various brain regions. Measurement of the galvanic skin response (GSR), both skin conductance level (SCL) and skin conductance responses (SCR), is used in diagnostics and treatment of mental disorders, and the training method itself, based on GSR Biofeedback, allows for modulation of the emotional state depending on needs occurring. Summary: It is relatively probable that neurorehabilitation based on GSR-BF is a method worth noticing, which-in the future-can represent an interesting area of rehabilitation supplementing a comprehensive treatment for people with mental disorders.

A Review of AI Cloud and Edge Sensors, Methods, and Applications for the Recognition of Emotional, Affective and Physiological States

Affective, emotional, and physiological states (AFFECT) detection and recognition by capturing human signals is a fast-growing area, which has been applied across numerous domains. The research aim is to review publications on how techniques that use brain and biometric sensors can be used for AFFECT recognition, consolidate the findings, provide a rationale for the current methods, compare the effectiveness of existing methods, and quantify how likely they are to address the issues/challenges in the field. In efforts to achieve the key goals of Society 5.0, Industry 5.0, and human-centered design better, the recognition of emotional, affective, and physiological states is progressively becoming an important matter and offers tremendous growth of knowledge and progress in these and other related fields. In this research, a review of AFFECT recognition brain and biometric sensors, methods, and applications was performed, based on Plutchik's wheel of emotions. Due to the immense variety of existing sensors and sensing systems, this study aimed to provide an analysis of the available sensors that can be used to define human AFFECT, and to classify them based on the type of sensing area and their efficiency in real implementations. Based on statistical and multiple criteria analysis across 169 nations, our outcomes introduce a connection between a nation's success, its number of Web of Science articles published, and its frequency of citation on AFFECT recognition. The principal conclusions present how this research contributes to the big picture in the field under analysis and explore forthcoming study trends.

Stress Classification Using Brain Signals Based on LSTM Network

The early diagnosis of stress symptoms is essential for preventing various mental disorder such as depression. Electroencephalography (EEG) signals are frequently employed in stress detection research and are both inexpensive and noninvasive modality. This paper proposes a stress classification system by utilizing an EEG signal. EEG signals from thirty-five volunteers were analysed which were acquired using four EEG sensors using a commercially available 4-electrode Muse EEG headband. Four movie clips were chosen as stress elicitation material. Two clips were selected to induce stress as it contains emotionally inductive scenes. The other two clips were chosen that do not induce stress as it has many comedy scenes. The recorded signals were then used to build the stress classification model. We compared the Multilayer Perceptron (MLP) and Long Short-Term Memory (LSTM) for classifying stress and nonstress group. The maximum classification accuracy of 93.17% was achieved using two-layer LSTM architecture.

Happy work: Improving enterprise human resource management by predicting workers' stress using deep learning

Recently, workers in most enterprises suffer from excessive occupational stress in the workplace, which negatively affects workers’ productivity, safety, and health. To deal with stress in workers, it is vital for the human resource management (HRM) department to manage stress effectively, bridging the gap between management and stressed employees. To manage stress effectively, the first step is to predict workers’ stress and detect the factors causing stress among workers. Existing methods often rely on the stress assessment questionnaire, which may not be effective to predict workers’ stress, due to 1) the difficulty of collecting the questionnaire data, and 2) the bias brought by workers’ subjectivity when completing the questionnaires. In this paper, we aim to address this issue and accurately predict workers’ stress status based on Deep Learning (DL) approach. We develop two stress prediction models (i.e., stress classification model and stress regression model) and correspondingly design two neural network architectures. We train these two stress prediction models based on workers’ data (e.g., salary, working time, KPI). By conducting experiments over two real-world datasets: ESI and HAJP, we validate that our proposed deep learning-based approach can effectively predict workers’ stress status with 71.2% accuracy in the classification model and 11.1 prediction loss in the regression model. By accurately predicting workers’ stress status with our method, the HRM of enterprises can be improved.

Manipulating Stress Responses during Spaceflight Training with Virtual Stressors

Virtual reality (VR) provides the ability to simulate stressors to replicated real-world situations. It allows for the creation and validation of training, therapy, and stress countermeasures in a safe and controlled setting. However, there is still much unknown about the cognitive appraisal of stressors and underlying elements. More research is needed on the creation of stressors and to verify that stress levels can be effectively manipulated by the virtual environment. The objective of this study was to investigate and validate different VR stressor levels from existing emergency spaceflight procedures. Experts in spaceflight procedures and the human stress response helped design a VR spaceflight environment and emergency fire task procedure. A within-subject experiment evaluated three stressor levels. Forty healthy participants each completed three trials (low, medium, high stressor levels) in VR to locate and extinguish a fire on the International Space Station (VR-ISS). Since stress is a complex construct, physiological data (heart rate, heart rate variability, blood pressure, electrodermal activity) and self-assessment (workload, stress, anxiety) were collected for each stressor level. The results suggest that the environmental-based stressors can induce significantly different, distinguishable levels of stress in individuals.

Objective Scoring of Physiologically Induced Dyspnea by Non-Invasive RF Sensors

OBJECTIVE

Dyspnea, also known as the patient's feeling of difficult or labored breathing, is one of the most common symptoms for respiratory disorders. Dyspnea is usually self-reported by patients using, for example, the Borg scale from 0 - 10, which is however subjective and problematic for those who refuse to cooperate or cannot communicate. The objective of this paper was to develop a learning-based model that can evaluate the correlation between the self-report Borg score and the respiratory metrics for dyspnea induced by exertion and increased airway resistance.

METHODS

A non-invasive wearable radio-frequency sensor by near-field coherent sensing was employed to retrieve continuous respiratory data with user comfort and convenience. Self-report dyspnea scores and respiratory features were collected on 32 healthy participants going through various physical and breathing exercises. A machine learning model based on the decision tree and random forest then produced an objective dyspnea score.

RESULTS

For unseen data as well as unseen participants, the objective dyspnea score can be in reasonable agreement with the self-report score, and the importance factor of each respiratory metrics can be assessed.

CONCLUSION

An objective dyspnea score can potentially complement or substitute the self-report for physiologically induced dyspnea.

SIGNIFICANCE

The method can potentially formulate a baseline for clinical dyspnea assessment and help caregivers track dyspnea continuously, especially for patients who cannot report themselves.

Evaluation of tractor driving vibration fatigue based on multiple physiological parameters

The vibration generated by tractor field operations will seriously affect the comfort and health of the driver. The low frequency vibration generated by the engine and ground excitation is similar to the natural frequency of human organs. Long term operation in this environment will resonate with the organs and affect drivers' health. To investigate this possibility, in this paper we carried out a collection experiment of human physiological indicators relevant to vibration fatigue. Four physiological signals of surface electromyography, skin electricity, skin temperature, and photoplethysmography signal were collected while the subjects experienced vibration. Several features of physiological signals as well as the law of signal features changing with fatigue are studied. The test results show that with the increase of human fatigue, the overall physiological parameters show the following trends: The median frequency of the human body surface electromyography and the slope of skin surface temperature decreases, the value of skin conductivity and the mean value of the photoplethysmography signal increases. Furthermore, this paper proposes a vibration comfort evaluation method based on multiple physiological parameters of the human body. An artificial neural network model is trained with test samples, and the prediction accuracy rate reaches 88.9%. Finally, the vibration conditions are changed by the shock-absorbing suspension of a tractor, verifying the effectiveness of the physiological signal changing with the vibration of the human body. The established prediction model can also be used to objectively reflect the discomfort of the human body under different working conditions and provide a basis for structural design optimization.

Initial Results of Tests Using GSR Biofeedback as a New Neurorehabilitation Technology Complementing Pharmacological Treatment of Patients with Schizophrenia

Galvanic skin response (GSR) Biofeedback uses training to reduce tension and anxiety and improve concentration and self-regulation. The study was aimed to evaluate this method as a form of rehabilitation and quantify the outcomes achieved by patients undergoing training using this technique. Six schizophrenic patients were enrolled in the study and underwent training based on the relaxation training module (CENTER), concentration training module (BALANCE), and self-regulation training module (INSECTS). Training sessions were held twice a week for 6 weeks. From the total group of subjects involved in the study, two patients had a statistically significant increase in measured values after the CENTER exercise, indicating that relaxation was achieved. Four patients showed a statistically significant decrease in measured values after the BALANCE exercise, which was reflective of an improvement in concentration. Three patients had a statistically significant decrease in measured values after the INSECTS exercise, which indicated an improvement in self-regulation. GSR Biofeedback may be used to complement the pharmacological treatment of patients diagnosed with schizophrenia.

Review of Studies on Older Drivers' Behavior and Stress-Methods, Results, and Outlook

This paper presents a review on relevant studies and reports related to older drivers’ behavior and stress. Questionnaires, simulators, and on-road/in-vehicle systems are used to collect driving data in most studies. In addition, research either directly compares older drivers and the other drivers or considers participants according to various age groups. Nevertheless, the definition of ‘older driver’ varies not only across studies but also across different government reports. Although questionnaire surveys are widely used to affordably obtain massive data in a short time, they lack objectivity. In contrast, biomedical information can increase the reliability of a driving stress assessment when collected in environments such as driving simulators and on-road experiments. Various studies determined that driving behavior and stress remain stable regardless of age, whereas others reported degradation of driving abilities and increased driving stress among older drivers. Instead of age, many researchers recommended considering other influencing factors, such as gender, living area, and driving experience. To mitigate bias in findings, this literature review suggests a hybrid method by applying surveys and collecting on-road/in-vehicle data.

A comprehensive review and analysis of supervised-learning and soft computing techniques for stress diagnosis in humans

Stress is the most prevailing and global psychological condition that inevitably disrupts the mood and behavior of individuals. Chronic stress may gravely affect the physical, mental, and social behavior of victims and consequently induce myriad critical human disorders. Herein, a review has been presented where supervised learning (SL) and soft computing (SC) techniques used in stress diagnosis have been meticulously investigated to highlight the contributions, strengths, and challenges faced in the implementation of these methods in stress diagnostic models. A three-tier review strategy comprising of manuscript selection, data synthesis, and data analysis was adopted. The issues in SL strategies and the potential possibility of using hybrid techniques in stress diagnosis have been intensively investigated. The strengths and weaknesses of different SL (Bayesian classifier, random forest, support vector machine, and nearest neighbours) and SC (fuzzy logic, nature-inspired, and deep learning) techniques have been presented to obtain clear insights into these optimization strategies. The effects of social, behavioral, and biological stresses have been highlighted. The psychological, biological, and behavioral responses to stress have also been briefly elucidated. The findings of the study confirmed that different types of data/signals (related to skin temperature, electro-dermal activity, blood circulation, heart rate, facial expressions, etc.) have been used in stress diagnosis. Moreover, there is a potential scope for using distinct nature-inspired computing techniques (Genetic Algorithm, Particle Swarm Optimization, Ant Colony Optimization, Whale Optimization Algorithm, Butterfly Optimization, Harris Hawks Optimizer, and Crow Search Algorithm) and deep learning techniques (Deep-Belief Network, Convolutional-Neural Network, and Recurrent-Neural Network) on multimodal data compiled using behavioral testing, electroencephalogram signals, finger temperature, respiration rate, pupil diameter, galvanic-skin-response, and blood pressure. Likewise, there is a wider scope to investigate the use of SL and SC techniques in stress diagnosis using distinct dimensions such as sentiment analysis, speech recognition, handwriting recognition, and facial expressions. Finally, a hybrid model based on distinct computational methods influenced by both SL and SC techniques, adaption, parameter tuning, and the use of chaos, levy, and Gaussian distribution may address exploration and exploitation issues. However, factors such as real-time data collection, bias, integrity, multi-dimensional data, and data privacy make it challenging to design precise and innovative stress diagnostic systems based on artificial intelligence.

Driving Stress Detection Using Multimodal Convolutional Neural Networks with Nonlinear Representation of Short-Term Physiological Signals

Mental stress can lead to traffic accidents by reducing a driver’s concentration or increasing fatigue while driving. In recent years, demand for methods to detect drivers’ stress in advance to prevent dangerous situations increased. Thus, we propose a novel method for detecting driving stress using nonlinear representations of short-term (30 s or less) physiological signals for multimodal convolutional neural networks (CNNs). Specifically, from hand/foot galvanic skin response (HGSR, FGSR) and heart rate (HR) short-term input signals, first, we generate corresponding two-dimensional nonlinear representations called continuous recurrence plots (Cont-RPs). Second, from the Cont-RPs, we use multimodal CNNs to automatically extract FGSR, HGSR, and HR signal representative features that can effectively differentiate between stressed and relaxed states. Lastly, we concatenate the three extracted features into one integrated representation vector, which we feed to a fully connected layer to perform classification. For the evaluation, we use a public stress dataset collected from actual driving environments. Experimental results show that the proposed method demonstrates superior performance for 30-s signals, with an overall accuracy of 95.67%, an approximately 2.5–3% improvement compared with that of previous works. Additionally, for 10-s signals, the proposed method achieves 92.33% classification accuracy, which is similar to or better than the performance of other methods using long-term signals (over 100 s).

Driver Fatigue Detection Systems Using Multi-Sensors, Smartphone, and Cloud-Based Computing Platforms: A Comparative Analysis

Internet of things (IoT) cloud-based applications deliver advanced solutions for smart cities to decrease traffic accidents caused by driver fatigue while driving on the road. Environmental conditions or driver behavior can ultimately lead to serious roadside accidents. In recent years, the authors have developed many low-cost, computerized, driver fatigue detection systems (DFDs) to help drivers, by using multi-sensors, and mobile and cloud-based computing architecture. To promote safe driving, these are the most current emerging platforms that were introduced in the past. In this paper, we reviewed state-of-the-art approaches for predicting unsafe driving styles using three common IoT-based architectures. The novelty of this article is to show major differences among multi-sensors, smartphone-based, and cloud-based architectures in multimodal feature processing. We discussed all of the problems that machine learning techniques faced in recent years, particularly the deep learning (DL) model, to predict driver hypovigilance, especially in terms of these three IoT-based architectures. Moreover, we performed state-of-the-art comparisons by using driving simulators to incorporate multimodal features of the driver. We also mention online data sources in this article to test and train network architecture in the field of DFDs on public available multimodal datasets. These comparisons assist other authors to continue future research in this domain. To evaluate the performance, we mention the major problems in these three architectures to help researchers use the best IoT-based architecture for detecting DFDs in a real-time environment. Moreover, the important factors of Multi-Access Edge Computing (MEC) and 5th generation (5G) networks are analyzed in the context of deep learning architecture to improve the response time of DFD systems. Lastly, it is concluded that there is a research gap when it comes to implementing the DFD systems on MEC and 5G technologies by using multimodal features and DL architecture.

Distracted and Drowsy Driving Modeling Using Deep Physiological Representations and Multitask Learning

In this paper, we investigated various physiological indicators on their ability to identify distracted and drowsy driving. In particular, four physiological signals are being tested: blood volume pulse (BVP), respiration, skin conductance and skin temperature. Data were collected from 45 participants, under a simulated driving scenario, through different times of the day and during their engagement on a variety of physical and cognitive distractors. We explore several statistical features extracted from those signals and their efficiency to discriminate between the presence or not of each of the two conditions. To that end, we evaluate three traditional classifiers (Random Forests, KNN and SVM), which have been extensively applied by the related literature and we compare their performance against a deep CNN-LSTM network that learns spatio-temporal physiological representations. In addition, we explore the potential of learning multiple conditions in parallel using a single machine learning model, and we discuss how such a problem could be formulated and what are the benefits and disadvantages of the different approaches. Overall, our findings indicate that information related to the BVP data, especially features that describe patterns with respect to the inter-beat-intervals (IBI), are highly associates with both targeted conditions. In addition, features related to the respiratory behavior of the driver can be indicative of drowsiness, while being less associated with distractions. Moreover, spatio-temporal deep methods seem to have a clear advantage against traditional classifiers on detecting both driver conditions. Our experiments show, that even though learning both conditions jointly can not compete directly to individual, task-specific CNN-LSTM models, deep multitask learning approaches have a great potential towards that end as they offer the second best performance on both tasks against all other evaluated alternatives in terms of sensitivity, specificity and the area under the receiver operating characteristic curve (AUC).

CNN and LSTM-Based Emotion Charting Using Physiological Signals

Novel trends in affective computing are based on reliable sources of physiological signals such as Electroencephalogram (EEG), Electrocardiogram (ECG), and Galvanic Skin Response (GSR). The use of these signals provides challenges of performance improvement within a broader set of emotion classes in a less constrained real-world environment. To overcome these challenges, we propose a computational framework of 2D Convolutional Neural Network (CNN) architecture for the arrangement of 14 channels of EEG, and a combination of Long Short-Term Memory (LSTM) and 1D-CNN architecture for ECG and GSR. Our approach is subject-independent and incorporates two publicly available datasets of DREAMER and AMIGOS with low-cost, wearable sensors to extract physiological signals suitable for real-world environments. The results outperform state-of-the-art approaches for classification into four classes, namely High Valence-High Arousal, High Valence-Low Arousal, Low Valence-High Arousal, and Low Valence-Low Arousal. Emotion elicitation average accuracy of 98.73% is achieved with ECG right-channel modality, 76.65% with EEG modality, and 63.67% with GSR modality for AMIGOS. The overall highest accuracy of 99.0% for the AMIGOS dataset and 90.8% for the DREAMER dataset is achieved with multi-modal fusion. A strong correlation between spectral- and hidden-layer feature analysis with classification performance suggests the efficacy of the proposed method for significant feature extraction and higher emotion elicitation performance to a broader context for less constrained environments.

A multimodal and signals fusion approach for assessing the impact of stressful events on Air Traffic Controllers

Stress is a word used to describe human reactions to emotionally, cognitively and physically challenging experiences. A hallmark of the stress response is the activation of the autonomic nervous system, resulting in the "fight-freeze-flight" response to a threat from a dangerous situation. Consequently, the capability to objectively assess and track a controller's stress level while dealing with air traffic control (ATC) activities would make it possible to better tailor the work shift and maintain high safety levels, as well as to preserve the operator's health. In this regard, sixteen controllers were asked to perform a realistic air traffic management (ATM) simulation during which subjective data (i.e. stress perception) and neurophysiological data (i.e. brain activity, heart rate, and galvanic skin response) were collected with the aim of accurately characterising the controller's stress level experienced in the various experimental conditions. In addition, external supervisors regularly evaluated the controllers in terms of manifested stress, safety, and efficiency throughout the ATM scenario. The results demonstrated 1) how the stressful events caused both supervisors and controllers to underestimate the experienced stress level, 2) the advantage of taking into account both cognitive and hormonal processes in order to define a reliable stress index, and 3) the importance of the points in time at which stress is measured owing to the potential transient effect once the stressful events have ceased.

Nonintrusive Monitoring of Mental Fatigue Status Using Epidermal Electronic Systems and Machine-Learning Algorithms

Mental fatigue, characterized by subjective feelings of "tiredness" and "lack of energy", can degrade individual performance in a variety of situations, for example, in motor vehicle driving or while performing surgery. Thus, a method for nonintrusive monitoring of mental fatigue status is urgently needed. Recent research shows that physiological signal-based fatigue-classification methods using wearable electronics can be sufficiently accurate; by contrast, rigid, bulky devices constrain the behavior of those wearing them, potentially interfering with test signals. Recently, wearable electronics, such as epidermal electronics systems (EES) and electronic tattoos (E-tattoos), have been developed to meet the requirements for the comfortable measurement of various physiological signals. However, comfortable, effective, and nonintrusive monitoring of mental fatigue levels remains to be fulfilled. In this work, an EES is established to simultaneously detect multiple physiological signals in a comfortable and nonintrusive way. Machine-learning algorithms are employed to determine the mental fatigue levels and a predictive accuracy of up to 89% is achieved based on six different kinds of physiological features using decision tree algorithms. Furthermore, EES with the trained predictive model are applied to monitor in situ human mental fatigue levels when doing several routine research jobs, as well as the effect of relaxation methods in relieving fatigue.

A Novel Classification Method for a Driver's Cognitive Stress Level by Transferring Interbeat Intervals of the ECG Signal to Pictures

In this study, a novel classification method for a driver's cognitive stress level was proposed, whereby the interbeat intervals extracted from an electrocardiogram (ECG) signal were transferred to pictures, and a convolution neural network (CNN) was used to train the pictures to classify a driver's cognitive stress level. First, we defined three levels of tasks and collected the ECG signal of the driver at different cognitive stress levels by designing and performing a driving simulation experiment. We extracted the interbeat intervals and converted them to pictures according to the number of consecutive interbeat intervals in each picture. Second, the CNN model was used to train the data set to recognize the cognitive stress levels. Classification accuracies of 100%, 91.6% and 92.8% were obtained for the training set, validation set and test set, respectively, and were compared with those the BP neural network. Last, we discussed the influence of the number of interbeat intervals in each picture on the performance of the proposed classification method. The results showed that the performance initially improved with an increase in the number of interbeat intervals. A downward trend was observed when the number exceeded 40, and when the number was 40, the model performed best with the highest accuracy (98.79%) and a relatively low relative standard deviation (0.019).

A Sensor-enabled Digital Trier Social Stress Test in an Enterprise Context

The Trier Social Stress Test (TSST) protocol is a widely accepted method of inducing social and/or cognitive stress in participants and studying its effects. Traditionally, this protocol is administered in laboratory or university settings, which are less formal than in offices. In this paper, we report the results of the analysis of multi-modal sensor data collected from employees of an enterprise who underwent the test. We briefly discuss the adaptations that enabled administering it digitally in a semi-automatic mode with minimal researcher/test-administrator intervention. In our setup, noninvasive sensor-signals, including the Galvanic Skin Response and Photoplethysmogram, were collected during and outside the stress-inducing tasks. We analyze the data collected from twenty participants and show that the State Trait Anxiety Inventory (STAI) score is needed in assessing the effect of the digital version of the TSST. A support vector machine classifier yielded an F₁ score of 0.723 with the STAI score taken as ground truth. We also introduce the idea of ground truth based on the change in the STAI scores to reduce variation due to subjective interpretation, for which an F₁ score of 0.847 was obtained.

Reckoning respiratory signals to affectively decipher mental state

Recognizing mental states from physiological signal is a concern not only for medical diagnostics, but also for cognitive science, behavioral studies as well as brain machine interfaces. This study employs an unique approach of solely utilizing the respiration signals in order to decipher mental states. A public dataset, Affective Pacman, is considered for this study, where the various physiological signals are acquired during normal and frustrated mental states. An efficient way to remove the non-linear baseline drifts in the signal is implemented to extract the respiratory features in most effective way. Another major adversity is the presence of class imbalance, which is effectively rectified using Synthetic Minority Oversampling TEchnique (SMOTE). Application of SMOTE algorithm to resolve class imbalance problem, not only increased the classification accuracy, but also reduced the classifier bias towards the majority class, which in turn exceedingly enhanced the classifier sensitivity. The multilayer perceptron classifier performed best with SMOTE generated feature set, with classification accuracy (CA), true positive rate (TPR) and true negative rate (TNR) of 97.9%, 92.6% and 99.3% respectively. The current approach is found to perform better compared to relevant literature.

Associating Vehicles Automation With Drivers Functional State Assessment Systems: A Challenge for Road Safety in the Future

In the near future, vehicles will gradually gain more autonomous functionalities. Drivers' activity will be less about driving than about monitoring intelligent systems to which driving action will be delegated. Road safety, therefore, remains dependent on the human factor and we should identify the limits beyond which driver's functional state (DFS) may no longer be able to ensure safety. Depending on the level of automation, estimating the DFS may have different targets, e.g., assessing driver's situation awareness in lower levels of automation and his ability to respond to emerging hazard or assessing driver's ability to monitor the vehicle performing operational tasks in higher levels of automation. Unfitted DFS (e.g., drowsiness) may impact the driver ability respond to taking over abilities. This paper reviews the most appropriate psychophysiological indices in naturalistic driving while considering the DFS through exogenous sensors, providing the more efficient trade-off between reliability and intrusiveness. The DFS also originates from kinematic data of the vehicle, thus providing information that indirectly relates to drivers behavior. The whole data should be synchronously processed, providing a diagnosis on the DFS, and bringing it to the attention of the decision maker in real time. Next, making the information available can be permanent or intermittent (or even undelivered), and may also depend on the automation level. Such interface can include recommendations for decision support or simply give neutral instruction. Mapping of relevant psychophysiological and behavioral indicators for DFS will enable practitioners and researchers provide reliable estimates, fitted to the level of automation.

Large-scale wearable data reveal digital phenotypes for daily-life stress detection

Physiological signals have shown to be reliable indicators of stress in laboratory studies, yet large-scale ambulatory validation is lacking. We present a large-scale cross-sectional study for ambulatory stress detection, consisting of 1002 subjects, containing subjects' demographics, baseline psychological information, and five consecutive days of free-living physiological and contextual measurements, collected through wearable devices and smartphones. This dataset represents a healthy population, showing associations between wearable physiological signals and self-reported daily-life stress. Using a data-driven approach, we identified digital phenotypes characterized by self-reported poor health indicators and high depression, anxiety and stress scores that are associated with blunted physiological responses to stress. These results emphasize the need for large-scale collections of multi-sensor data, to build personalized stress models for precision medicine.

Eliminating Individual Bias to Improve Stress Detection from Multimodal Physiological Data

Stress monitoring is important for mental wellbeing and early detection of related disorders. The current work is focused on stress detection from multiple non-invasive physiological signals like Electroencephalogram (EEG), Photoplethysmogram (PPG) and Galvanic Skin Response (GSR). We show that, compared to using only the well known EEG band powers in different frequencies for stress detection, an early fusion with GSR and PPG features shows a significant improvement. Maximum Relevance Minimum Redundancy (mRMR) based feature selection is used to identify the most suitable physiological features correlating with stress. A major contribution of this work lies in eliminating subject-specific bias to improve the classification accuracy. We use self-reported values of Valence, Arousal and Dominance to cluster subjects and build separate classification models specific to clusters. The proposed approach is validated on a publicly available dataset comprising 146 data instances from 10 subjects. The performances of Leave-One- Subject-Out cross validation (LOSOCV) in terms of mean Fscores are 0.61 using EEG features only, 0.64 using early fusion of EEG, GSR and PPG features and 0.69 by applying our clustering technique before fusion and classification.

A quantitative performance study of two automatic methods for the diagnosis of ovarian cancer

We present a quantitative study of the performance of two automatic methods for the early detection of ovarian cancer that can exploit longitudinal measurements of multiple biomarkers. The study is carried out for a subset of the data collected in the UK Collaborative Trial of Ovarian Cancer Screening (UKCTOCS). We use statistical analysis techniques, such as the area under the Receiver Operating Characteristic (ROC) curve, for evaluating the performance of two techniques that aim at the classification of subjects as either healthy or suffering from the disease using time-series of multiple biomarkers as inputs. The first method relies on a Bayesian hierarchical model that establishes connections within a set of clinically interpretable parameters. The second technique is a purely discriminative method that employs a recurrent neural network (RNN) for the binary classification of the inputs. For the available dataset, the performance of the two detection schemes is similar (the area under ROC curve is 0.98 for the combination of three biomarkers) and the Bayesian approach has the advantage that its outputs (parameters estimates and their uncertainty) can be further analysed by a clinical expert.

Evaluation of an Integrated System of Wearable Physiological Sensors for Stress Monitoring in Working Environments by Using Biological Markers

OBJECTIVE

The objectives of this paper are to develop and test the ability of a wearable physiological sensors system, based on ECG, EDA, and EEG, to capture human stress and to assess whether the detected changes in physiological signals correlate with changes in salivary cortisol level, which is a reliable, objective biomarker of stress.

METHODS

15 healthy participants, eight males and seven females, mean age 40.8 ± 9.5 years, wore a set of three commercial sensors to record physiological signals during the Maastricht Acute Stress Test, an experimental protocol known to elicit robust physical and mental stress in humans. Salivary samples were collected throughout the different phases of the test. Statistical analysis was performed using a support vector machine (SVM) classification algorithm. A correlation analysis between extracted physiological features and salivary cortisol levels was also performed.

RESULTS

15 features extracted from heart rate variability, electrodermal, and electroencephalography signals showed a high degree of significance in disentangling stress from a relaxed state. The classification algorithm, based on significant features, provided satisfactory outcomes with 86% accuracy. Furthermore, correlation analysis showed that the observed changes in physiological features were consistent with the trend of salivary cortisol levels (R² = 0.714).

CONCLUSION

The tested set of wearable sensors was able to successfully capture human stress and quantify stress level.

SIGNIFICANCE

The results of this pilot study may be useful in designing portable and remote control systems, such as medical devices used to turn on interventions and prevent stress consequences.

Computer tool to evaluate the cue reactivity of chemically dependent individuals

BACKGROUND AND OBJECTIVE

Anxiety is one of the major influences on the dropout of relapse and treatment of substance abuse treatment. Chemically dependent individuals need (CDI) to be aware of their emotional state in situations of risk during their treatment. Many patients do not agree with the diagnosis of the therapist when considering them vulnerable to environmental stimuli related to drugs. This research presents a cue reactivity detection tool based on a device acquiring physiological signals connected to personal computer. Depending on the variations of the emotional state of the drug addict, alteration of the physiological signals will be detected by the computer tool (CT) which will modify the displayed virtual sets without intervention of the therapist.

METHODS

Developed in 3ds Max® software, the CT is composed of scenarios and objects that are in the habit of marijuana and cocaine dependent individual's daily life. The interaction with the environment is accomplished using a Human-Computer Interface (HCI) that converts incoming physiological signals indicating anxiety state into commands that change the scenes. Anxiety was characterized by the average variability from cardiac and respiratory rate of 30 volunteers submitted stress environment situations. To evaluate the effectiveness of cue reactivity a total of 50 volunteers who were marijuana, cocaine or both dependent were accompanied.

RESULTS

Prior to CT, the results demonstrated a poor correlation between the therapists' predictions and those of the chemically dependent individuals. After exposure to the CT, there was a significant increase of 73% in awareness of the risks of relapse.

CONCLUSION

We confirmed the hypothesis that the CT, controlled only by physiological signals, increases the perception of vulnerability to risk situations of individuals with dependence on marijuana, cocaine or both.

An efficient automatic workload estimation method based on electrodermal activity using pattern classifier combinations

Automatic workload estimation has received much attention because of its application in error prevention, diagnosis, and treatment of neural system impairment. The development of a simple but reliable method using minimum number of psychophysiological signals is a challenge in automatic workload estimation. To address this challenge, this paper presented three different decomposition techniques (Fourier, cepstrum, and wavelet transforms) to analyze electrodermal activity (EDA). The efficiency of various statistical and entropic features was investigated and compared. To recognize different levels of an arithmetic task, the features were processed by principal component analysis and machine-learning techniques. These methods have been incorporated into a workload estimation system based on two types: feature-level and decision-level combinations. The results indicated the reliability of the method for automatic and real-time inference of psychological states. This method provided a quantitative estimation of the workload levels and a bias-free evaluation approach. The high-average accuracy of 90% and cost effective requirement were the two important attributes of the proposed workload estimation system. New entropic features were proved to be more sensitive measures for quantifying time and frequency changes in EDA. The effectiveness of these measures was also compared with conventional tonic EDA measures, demonstrating the superiority of the proposed method in achieving accurate estimation of workload levels.

Mental stress assessment using simultaneous measurement of EEG and fNIRS

Previous studies reported mental stress as one of the major contributing factors leading to various diseases such as heart attack, depression and stroke. An accurate stress assessment method may thus be of importance to clinical intervention and disease prevention. We propose a joint independent component analysis (jICA) based approach to fuse simultaneous measurement of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) on the prefrontal cortex (PFC) as a means of stress assessment. For the purpose of this study, stress was induced by using an established mental arithmetic task under time pressure with negative feedback. The induction of mental stress was confirmed by salivary alpha amylase test. Experiment results showed that the proposed fusion of EEG and fNIRS measurements improves the classification accuracy of mental stress by +3.4% compared to EEG alone and +11% compared to fNIRS alone. Similar improvements were also observed in sensitivity and specificity of proposed approach over unimodal EEG/fNIRS. Our study suggests that combination of EEG (frontal alpha rhythm) and fNIRS (concentration change of oxygenated hemoglobin) could be a potential means to assess mental stress objectively.

Integration of Body Sensor Networks and Vehicular Ad-hoc Networks for Traffic Safety

The emergence of Body Sensor Networks (BSNs) constitutes a new and fast growing trend for the development of daily routine applications. However, in the case of heterogeneous BSNs integration with Vehicular ad hoc Networks (VANETs) a large number of difficulties remain, that must be solved, especially when talking about the detection of human state factors that impair the driving of motor vehicles. The main contributions of this investigation are principally three: (1) an exhaustive review of the current mechanisms to detect four basic physiological behavior states (drowsy, drunk, driving under emotional state disorders and distracted driving) that may cause traffic accidents is presented; (2) A middleware architecture is proposed. This architecture can communicate with the car dashboard, emergency services, vehicles belonging to the VANET and road or street facilities. This architecture seeks on the one hand to improve the car driving experience of the driver and on the other hand to extend security mechanisms for the surrounding individuals; and (3) as a proof of concept, an Android real-time attention low level detection application that runs in a next-generation smartphone is developed. The application features mechanisms that allow one to measure the degree of attention of a driver on the base of her/his EEG signals, establish wireless communication links via various standard wireless means, GPRS, Bluetooth and WiFi and issue alarms of critical low driver attention levels.