Classification of a Driver's cognitive workload levels using artificial neural network on ECG signals

Cardiac autonomic responses in relation to cognitive workload during simulated military flight

Understanding the operator's cognitive workload is crucial for efficiency and safety in human-machine systems. This study investigated how cognitive workload modulates cardiac autonomic regulation during a standardized military simulator flight. Military student pilots completed simulated flight tasks in a Hawk flight simulator. Continuous electrocardiography was recorded to analyze time and frequency domain heart rate variability (HRV). After the simulation, a flight instructor used a standardized method to evaluate student pilot's individual cognitive workload from video-recorded flight simulator data. Results indicated that HRV was able to differentiate flight phases that induced varying levels of cognitive workload; an increasing level of cognitive workload caused significant decreases in many HRV variables, mainly reflecting parasympathetic deactivation of cardiac autonomic regulation. In conclusion, autonomic physiological responses can be used to examine reactions to increased cognitive workload during simulated military flights. HRV could be beneficial in assessing individual responses to cognitive workload and pilot performance during simulator training.

The effects of different interruption conditions on mental workload: an experimental study based on multimodal measurements

Interruptions in the working environment cause extra mental workload for the operators, and this phenomenon has garnered significant research attention. This study designed four interruption conditions based on the perceptual and cognitive perspectives of human information processing, using a 2(perceptual primary task and cognitive primary task)*2(perceptual interruption task and cognitive interruption task) factorial design. Multimodal measurement methods were used to evaluate mental workload in different interruption conditions. The results show that when the primary task and the interruption task are different load types, they generate a higher mental workload than the same load type. It can be attributed to the fact that perceptual tasks and cognitive tasks increase mental workload during switching. In addition, based on the multimodal index data, the prediction model of interruption recovery delay time and the classification model of interruption conditions are established, which provides a basis for rational scheduling of work and preventing mental overload.

Cognitive workload estimation using physiological measures: a review

Estimating cognitive workload levels is an emerging research topic in the cognitive neuroscience domain, as participants' performance is highly influenced by cognitive overload or underload results. Different physiological measures such as Electroencephalography (EEG), Functional Magnetic Resonance Imaging, Functional near-infrared spectroscopy, respiratory activity, and eye activity are efficiently used to estimate workload levels with the help of machine learning or deep learning techniques. Some reviews focus only on EEG-based workload estimation using machine learning classifiers or multimodal fusion of different physiological measures for workload estimation. However, a detailed analysis of all physiological measures for estimating cognitive workload levels still needs to be discovered. Thus, this survey highlights the in-depth analysis of all the physiological measures for assessing cognitive workload. This survey emphasizes the basics of cognitive workload, open-access datasets, the experimental paradigm of cognitive tasks, and different measures for estimating workload levels. Lastly, we emphasize the significant findings from this review and identify the open challenges. In addition, we also specify future scopes for researchers to overcome those challenges.

Evaluation of physiological responses to mental workload in n-back and arithmetic tasks

Working memory tasks, such as n-back and arithmetic tasks, are frequently used in studying mental workload. The present study investigated and compared the sensitivity of several physiological measures at three levels of difficulty of n-back and arithmetic tasks. The results showed significant differences in fixation duration and pupil diameter among three task difficulty levels for both n-back and arithmetic tasks. Pupil diameters increase with increasing mental workload, whereas fixation duration decreases. Blink duration and heart rate (HR) were significantly increased as task difficulty increased in the n-back task, while root mean square of successive differences (RMSSD) and standard deviation of R-R intervals (SDNN) were significantly decreased in the arithmetic task. On the other hand, blink rate and Galvanic Skin Response (GSR) were not sensitive enough to assess the differences in task difficulty for both tasks. All significant physiological measures yielded significant differences between low and high task difficulty except for SDNN.Practitioner summary: This study aimed to assess the sensitivity levels of several physiological measures of mental workload in n-back and arithmetic tasks. It showed that pupil diameter was the most sensitive in both tasks. This study also found that most physiological indices are sensitive to an extreme change in task difficulty levels.

Predicting Driver's mental workload using physiological signals: A functional data analysis approach

This study investigates the impact of advanced driver-assistance systems on drivers' mental workload. Using a combination of physiological signals including ECG, EMG, EDA, EEG (af4 and fc6 channels from the theta band), and eye diameter data, this study aims to predict and categorize drivers' mental workload into low, adequate, and high levels. Data were collected from five different driving situations with varying cognitive demands. A functional linear regression model was employed for prediction, and the accuracy rate was calculated. Among the 31 tested combinations of physiological variables, 9 combinations achieved the highest accuracy result of 90%. These results highlight the potential benefits of utilizing raw physiological signal data and employing functional data analysis methods to understand and assess driver mental workload. The findings of this study have implications for the design and improvement of driver-assistance systems to optimize safety and performance.

Predicting Fatigue-Associated Aberrant Driving Behaviors Using a Dynamic Weighted Moving Average Model With a Long Short-Term Memory Network Based on Heart Rate Variability

OBJECTIVE

This study proposed a moving average (MA) approach to dynamically process heart rate variability (HRV) and developed aberrant driving behavior (ADB) prediction models by using long short-term memory (LSTM) networks.

BACKGROUND

Fatigue-associated ADBs have traffic safety implications. Numerous models to predict such acts based on physiological responses have been developed but are still in embryonic stages.

METHOD

This study recorded the data of 20 commercial bus drivers during their routine tasks on four consecutive days and subsequently asked them to complete questionnaires, including subjective sleep quality, driver behavior questionnaire and the Karolinska Sleepiness Scale. Driving behaviors and corresponding HRV were determined using a navigational mobile application and a wristwatch. The dynamic-weighted MA (DWMA) and exponential-weighted MA were used to process HRV in 5-min intervals. The data were independently separated for training and testing. Models were trained with 10-fold cross-validation strategy, their accuracies were evaluated, and Shapley additive explanation (SHAP) values were used to determine feature importance.

RESULTS

Significant increases in the standard deviation of NN intervals (SDNN), root mean square of successive heartbeat interval differences (RMSSD), and normalized spectrum of high frequency (nHF) were observed in the pre-event stage. The DWMA-based model exhibited the highest accuracy for both driver types (urban: 84.41%; highway: 80.56%). The SDNN, RMSSD, and nHF demonstrated relatively high SHAP values.

CONCLUSION

HRV metrics can serve as indicators of mental fatigue. DWMA-based LSTM could predict the occurrence of the level of fatigue associated with ADBs.

APPLICATION

The established models can be used in realistic driving scenarios.

Risk assessment of driver performance in the oil and gas transportation industry: Analyzing the relationship between driver vigilance, attention, reaction time, and safe driving practices

The increasing use of road traffic for land transportation has resulted in numerous road accidents and casualties, including those involving oil and gas tanker vehicles. Despite this, little empirical research has been conducted on the factors influencing tanker drivers' performance. This study aims to address this knowledge gap, particularly in the energy transportation industry, by examining the driving performance factors that affect tanker drivers and incorporating risk assessment measures. The model variables were identified from the literature and used to develop a survey questionnaire for the study. A total of 307 surveys were collected from Malaysian oil and gas tanker drivers, and the driving performance factors were contextually adjusted using the Exploratory Factor Analysis (EFA) approach. The driving performance model was developed using partial least squares structural equation modeling (PLS-SEM). The EFA results categorized driving performance into two constructs: 1) drivers' reaction time with β = 0.320 and 2) attention and vigilance with β value = 0.749. The proposed model provided full insight into how drivers' reaction time, attention, and vigilance impact drivers' performance in this sector, which can help identify potential risks and prevent accidents. The findings are significant in understanding the factors that affect oil and gas drivers' performance and can aid in enhancing oil and gas transportation management by including effective risk assessment measures to prevent fatal crashes.

Recognition and evaluation of mental workload in different stages of perceptual and cognitive information processing using a multimodal approach

This study explores the effects of different perceptual and cognitive information processing stages on mental workload by assessing multimodal indicators of mental workload such as the NASA-TLX, task performance, ERPs and eye movements. Repeated measures ANOVA of the data showed that among ERP indicators, P1, N1 and N2 amplitudes were sensitive to perceptual load (P-load), P3 amplitude was sensitive to P-load only in the prefrontal region during high cognitive load (C-load) states, and P3 amplitude in the occipital and parietal regions was sensitive to C-load. Among the eye movement indicators, blink frequency was sensitive to P-load in all C-load states, but to C-load in only low P-load states; pupil diameter and blink duration were sensitive to both P-load and C-load. Based on the above indicators, the k-nearest neighbours (KNN) algorithm was used to propose a classification method for the four different mental workload states with an accuracy of 97.89%.Practitioner summary: Based on the results of this study, it is possible to implement the monitoring of mental workload states and optimise brain task allocation in operations involving high mental workload, such as human-computer interaction.

A Systematic Review of In-Vehicle Physiological Indices and Sensor Technology for Driver Mental Workload Monitoring

The concept of vehicle automation ceases to seem futuristic with the current advancement of the automotive industry. With the introduction of conditional automated vehicles, drivers are no longer expected to focus only on driving activities but are still required to stay alert to resume control. However, fluctuations in driving demands are known to alter the driver's mental workload (MWL), which might affect the driver's vehicle take-over capabilities. Driver mental workload can be specified as the driver's capacity for information processing for task performance. This paper summarizes the literature that relates to analysing driver mental workload through various in-vehicle physiological sensors focusing on cardiovascular and respiratory measures. The review highlights the type of study, hardware, method of analysis, test variable, and results of studies that have used physiological indices for MWL analysis in the automotive context.

Vehicle and Driver Monitoring System Using On-Board and Remote Sensors

This paper presents an integrated monitoring system for the driver and the vehicle in a single case of study easy to configure and replicate. On-board vehicle sensors and remote sensors are combined to model algorithms for estimating polluting emissions, fuel consumption, driving style and driver's health. The main contribution of this paper is the analysis of interactions among the above monitored features highlighting the influence of the driver in the vehicle performance and vice versa. This analysis was carried out experimentally using one vehicle with different drivers and routes and implemented on a mobile application. Compared to commercial driver and vehicle monitoring systems, this approach is not customized, uses classical sensor measurements, and is based on simple algorithms that have been already proven but not in an interactive environment with other algorithms. In the procedure design of this global vehicle and driver monitoring system, a principal component analysis was carried out to reduce the variables used in the training/testing algorithms with objective to decrease the transfer data via Bluetooth between the used devices: a biometric wristband, a smartphone and the vehicle's central computer. Experimental results show that the proposed vehicle and driver monitoring system predicts correctly the fuel consumption index in 84%, the polluting emissions 89%, and the driving style 89%. Indeed, interesting correlation results between the driver's heart condition and vehicular traffic have been found in this analysis.

Quantifying Cognitive Workload Using a Non-Contact Magnetocardiography (MCG) Wearable Sensor

Quantifying cognitive workload, i.e., the level of mental effort put forth by an individual in response to a cognitive task, is relevant for healthcare, training and gaming applications. However, there is currently no technology available that can readily and reliably quantify the cognitive workload of an individual in a real-world environment at a seamless way and affordable price. In this work, we overcome these limitations and demonstrate the feasibility of a magnetocardiography (MCG) sensor to reliably classify high vs. low cognitive workload while being non-contact, fully passive and low-cost, with the potential to have a wearable form factor. The operating principle relies on measuring the naturally emanated magnetic fields from the heart and subsequently analyzing the heart rate variability (HRV) matrix in three time-domain parameters: standard deviation of RR intervals (SDRR); root mean square of successive differences between heartbeats (RMSSD); and mean values of adjacent R-peaks in the cardiac signals (MeanRR). A total of 13 participants were recruited, two of whom were excluded due to low signal quality. The results show that SDRR and RMSSD achieve a 100% success rate in classifying high vs. low cognitive workload, while MeanRR achieves a 91% success rate. Tests for the same individual yield an intra-subject classification accuracy of 100% for all three HRV parameters. Future studies should leverage machine learning and advanced digital signal processing to achieve automated classification of cognitive workload and reliable operation in a natural environment.

Classification of Drivers' Mental Workload Levels: Comparison of Machine Learning Methods Based on ECG and Infrared Thermal Signals

Mental workload (MW) represents the amount of brain resources required to perform concurrent tasks. The evaluation of MW is of paramount importance for Advanced Driver-Assistance Systems, given its correlation with traffic accidents risk. In the present research, two cognitive tests (Digit Span Test-DST and Ray Auditory Verbal Learning Test-RAVLT) were administered to participants while driving in a simulated environment. The tests were chosen to investigate the drivers' response to predefined levels of cognitive load to categorize the classes of MW. Infrared (IR) thermal imaging concurrently with heart rate variability (HRV) were used to obtain features related to the psychophysiology of the subjects, in order to feed machine learning (ML) classifiers. Six categories of models have been compared basing on unimodal IR/unimodal HRV/multimodal IR + HRV features. The best classifier performances were reached by the multimodal IR + HRV features-based classifiers (DST: accuracy = 73.1%, sensitivity = 0.71, specificity = 0.69; RAVLT: accuracy = 75.0%, average sensitivity = 0.75, average specificity = 0.87). The unimodal IR features based classifiers revealed high performances as well (DST: accuracy = 73.1%, sensitivity = 0.73, specificity = 0.73; RAVLT: accuracy = 71.1%, average sensitivity = 0.71, average specificity = 0.85). These results demonstrated the possibility to assess drivers' MW levels with high accuracy, also using a completely non-contact and non-invasive technique alone, representing a key advancement with respect to the state of the art in traffic accident prevention.

Machine-Learning Based Monitoring of Cognitive Workload in Rescue Missions with Drones

In search and rescue missions, drone operations are challenging and cognitively demanding. High levels of cognitive workload can affect rescuers' performance, leading to failure with catastrophic outcomes. To face this problem, we propose a machine learning algorithm for real-time cognitive workload monitoring to understand if a search and rescue operator has to be replaced or if more resources are required. Our multimodal cognitive workload monitoring model combines the information of 25 features extracted from physiological signals, such as respiration, electrocardiogram, photoplethysmogram, and skin temperature, acquired in a noninvasive way. To reduce both subject and day inter-variability of the signals, we explore different feature normalization techniques, and introduce a novel weighted-learning method based on support vector machines suitable for subject-specific optimizations. On an unseen test set acquired from 34 volunteers, our proposed subject-specific model is able to distinguish between low and high cognitive workloads with an average accuracy of 87.3% and 91.2% while controlling a drone simulator using both a traditional controller and a new-generation controller, respectively.

Effects of whole-body vibration on driver drowsiness: A review

INTRODUCTION

Whole-body vibration has direct impacts on driver vigilance by increasing physical and cognitive stress on the driver, which leads to drowsiness, fatigue and road traffic accidents. Although sleep deprivation, sleep apnoea and alcohol consumption can also lead to driver drowsiness, exposure to steady vibration is the factor most readily controlled by changes to vehicle design, yet it has received comparatively less attention.

METHODS

This review investigated interrelationships between the various components of whole-body vibration and the physiological and cognitive parameters that lead to driver drowsiness, as well as the effects of vibration parameters (frequency, amplitude, waveform and duration). Vibrations transmitted to the driver body from the vehicle floor and/or seat have been considered for this review, whereas hand-arm vibration, shocks, acute or transient vibration were excluded from consideration.

RESULTS

Drowsiness is affected by interactions between the frequency, amplitude, waveform and duration of the vibration. Under optimal conditions, whole-body vibration can induce significant drowsiness within 30 min. Low frequency whole-body vibrations, particularly vibrations of 4-10 Hz, are most effective at inducing drowsiness. This review notes some limitations of current studies and suggests directions for future research.

CONCLUSIONS

This review demonstrated a strong causal link exists between whole-body vibration and driver drowsiness. Since driver drowsiness has been established to be a significant contributor to motor vehicle accidents, research is needed to identify ways to minimise the components of whole-body vibration that contribute to drowsiness, as well as devising more effective ways to counteract drowsiness.

PRACTICAL APPLICATIONS

By raising awareness of the vibrational factors that contribute to drowsiness, manufacturers will be prompted to design vehicles that reduce the influence of these factors.

High expectancy influences the role of cognitive load in inattentional deafness during landing decision-making

Neglecting a critical auditory alarm is a major obstacle to maintaining a safe environment, especially in aviation. Earlier studies have indicated that tasks with a higher perceptual or cognitive load in the visual modality influence the processing of auditory stimuli. It is unclear, however, whether other factors, such as memory failure, active neglect, or expectancy influence the effect of cognitive load on auditory alarm detection sensitivity during aeronautical decision-making. In this study, we investigated this issue in three laboratory experiments using the technique of signal detection analysis, in which participants were asked to make a landing decision based on indicators of the instrument landing system while also trying to detect an audible alarm. We found that the sensitivity of auditory alarm detection was reduced under conditions of high cognitive load and that this effect persisted even when the auditory detection response occurred first (before the landing decision response) and when the probability of an auditory alarm was 40%. However, the sensitivity of auditory detection was not influenced by cognitive load under high expectancy conditions (60% probability of alarm presentation). Furthermore, the value of the response bias was reduced under high cognitive load conditions when the probability of an auditory alarm was low (20%). With an increase in the level of expectancy (40% and 60% probability of alarm presentation), it was found that cognitive load did not influence the response bias. These findings indicate that visual cognitive load affects the sensitivity to an auditory alarm only at a low expectancy level (20% and 40% probability of alarm presentation). The effect of cognitive load on the sensitivity to an auditory alarm was not due to memory failure or active neglect and the response bias was more sensitive to the expectancy factor.

Increase in rear-end collision risk by acute stress-induced fatigue in on-road truck driving

Increasing road crashes related to occupational drivers’ deteriorating health has become a social problem. To prevent road crashes, warnings and predictions of increased crash risk based on drivers’ conditions are important. However, in on-road driving, the relationship between drivers’ physiological condition and crash risk remains unclear due to difficulties in the simultaneous measurement of both. This study aimed to elucidate the relationship between drivers’ physiological condition assessed by autonomic nerve function (ANF) and an indicator of rear-end collision risk in on-road driving. Data from 20 male truck drivers (mean ± SD, 49.0±8.2 years; range, 35–63 years) were analyzed. Over a period of approximately three months, drivers’ working behavior data, such as automotive sensor data, and their ANF data were collected during their working shift. Using the gradient boosting decision tree method, a rear-end collision risk index was developed based on the working behavior data, which enabled continuous risk quantification. Using the developed risk index and drivers’ ANF data, effects of their physiological condition on risk were analyzed employing a logistic quantile regression method, which provides wider information on the effects of the explanatory variables, after hierarchical model selection. Our results revealed that in on-road driving, activation of sympathetic nerve activity and inhibition of parasympathetic nerve activity increased each quantile of the rear-end collision risk index. The findings suggest that acute stress-induced drivers’ fatigue increases rear-end collision risk. Hence, in on-road driving, drivers’ physiological condition monitoring and ANF-based stress warning and relief system can contribute to promoting the prevention of rear-end truck collisions.

The Impacts of Temporal Variation and Individual Differences in Driver Cognitive Workload on ECG-Based Detection

OBJECTIVE

This paper aimed to investigate the robustness of driver cognitive workload detection based on electrocardiogram (ECG) when considering temporal variation and individual differences in cognitive workload.

BACKGROUND

Cognitive workload is a critical component to be monitored for error prevention in human-machine systems. It may fluctuate instantaneously over time even in the same tasks and differ across individuals.

METHOD

A driving simulation study was conducted to classify driver cognitive workload underlying four experimental conditions (baseline, N-back, texting, and N-back + texting distraction) in two repeated 1-hr blocks. Heart rate (HR) and heart rate variability (HRV) were compared among the experimental conditions and between the blocks. Random forests were built on HR and HRV to classify cognitive workload in different blocks and for different individuals.

RESULTS

HR and HRV were significantly different between repeated blocks in the study, demonstrating the time-induced variation in cognitive workload. The performance of cognitive workload classification across blocks and across individuals was significantly improved after normalizing HR and HRV in each block by the corresponding baseline.

CONCLUSION

The temporal variation and individual differences in cognitive workload affects ECG-based cognitive workload detection. But normalization approaches relying on the choice of appropriate baselines help compensate for the effects of temporal variation and individual differences.

APPLICATION

The findings provide insight into the value and limitations of ECG-based driver cognitive workload monitoring during prolonged driving for individual drivers.

Recognition of the Impulse of Love at First Sight Based on Electrocardiograph Signal

The impulse of love at first sight (ILFS) is a well known but rarely studied phenomenon. Despite the privacy of these emotions, knowing how attractive one finds a partner may be beneficial for building a future relationship in an open society, where partners are accepting each other. Therefore, this study adopted the electrocardiograph (ECG) signal collection method, which has been widely used in wearable devices, to collect signals and conduct corresponding recognition analysis. First, we used photos to induce ILFS and obtained ECG signals from 46 healthy students (24 women and 22 men) in a laboratory. Second, we extracted the time- and frequency-domain features of the ECG signals and performed a nonlinear analysis. We subsequently used a feature selection algorithm and a set of classifiers to classify the features. Combined with the sequence floating forward selection and random forest algorithms, the identification accuracy of the ILFS was 69.07%. The sensitivity, specificity, F1, and area under the curve of the other parameters were all greater than 0.6. The classification of ECG signals according to their characteristics demonstrated that the signals could be recognized. Through the information provided by the ECG signals, it can be determined whether the participant possesses the desire to fall in love, helping to determine the right partner in the fastest time; this is conducive to establishing a romantic relationship.

Cardiovascular Biomarkers' Inherent Timescales in Mental Workload Assessment During Simulated Air Traffic Control Tasks

One central topic in ergonomics and human-factors research is the assessment of mental workload. Heart rate and heart rate variability are common for registering mental workload. However, a major problem of workload assessment is the dissociation among different workload measures. One potential reason could be the disregard of their inherent timescales and the interrelation between participants' individual differences and timescales. The aim of our study was to determine if different cardiovascular biomarkers exhibit different timescales. We focused on air traffic controller and investigated biomarkers' ability to distinguish between conditions with different load levels connected to prior work experience and different time slots. During an interactive real-time simulation, we varied the load situations with two independent variables: the traffic volume and the occurrence of a priority-flight request. Dependent variables for registering mental workload were the heart rate and heart rate variability from two time slots. Our results show that all cardiovascular biomarkers were sensitive to workload differences with different inherent timescales. The heart rate responded sooner than the heart rate variability features from the frequency domain and it was most indicative during the time slot immediately after the priority-flight request. The heart rate variability parameters from the frequency domain responded with latency and were most indicative during the subsequent time slot. Furthermore, by consideration of biomarkers' inherent timescales, we were able to assess a significant effect of work experience on heart rate and mid/high frequency-band ratio of the heart rate variability. Results indicated that different cardiovascular biomarkers reveal different inherent timescales.

Car drivers coping with hazardous events in real versus simulated situations: Declarative, behavioral and physiological data used to assess drivers' feeling of presence

More than 1.3 million people lose their lives every year in traffic accidents. Improving road safety requires designing better vehicles and investigating drivers’ abilities more closely. Driving simulators are constantly being used for this purpose, but the question which often arises as to their validity tends to be a barrier to developments in this field. Here we studied the validity of a simulator, defined as how closely users’ behavior under simulated conditions resembles their behavior on the road, based on the concept of drivers’ feeling of presence. For this purpose, the driving behavior, physiological state and declarative data of 41 drivers were tested in the Sherpa2 simulator and in a real vehicle on a track while driving at a constant speed. During each trial, drivers had to cope with an unexpected hazardous event (a one-meter diameter gym ball crossing the road right in front of the vehicle), which occurred twice. During the speed-maintenance task, the simulator showed absolute validity, in terms of the driving and physiological parameters recorded. During the first hazardous event, the physiological parameters showed that the level of arousal (Low Heart Rate/High Heart Rate ratio x10) increased up to the end of the drive. On the other hand, the drivers’ behavioral (braking) responses were 20% more frequent in the simulator than in the real vehicle, and the physiological state parameters showed that stress reactions occurred only in the real vehicle (+5 beats per minute, +2 breaths per minute and the phasic skin conductance increased by 2). In the subjects’ declarative data, several feeling of presence sub-scales were lower under simulated conditions. These results suggest that the validity of motion based simulators for testing drivers coping with hazards needs to be questioned.

Arrhythmias Prediction Using an Hybrid Model Based on Convolutional Neural Network and Nonlinear Regression

In biomedical signal processing, artificial intelligence techniques are used for identifying and extracting relevant information. However, it lacks effective solutions based on machine learning for the prediction of cardiac arrhythmias. The heart diseases diagnosis rests essentially on the analysis of various properties of ECG signal. The arrhythmia is one of the most common heart diseases. A cardiac arrhythmia is a disturbance of the heart rhythm. It occurs when the heart beats too slowly, too fast or anarchically, with no apparent cause. The diagnosis of cardiac arrhythmias is based on the analysis of the ECG properties, especially, the durations (P, QRS, T), the amplitudes (P, Q, R, S, T), the intervals (PQ, QT, RR), the cardiac frequency and the rhythm. In this paper we propose a system of arrhythmias diagnosis assistance based on the analysis of the temporal and frequential properties of the ECG signal. After the features extraction step, the ECG properties are then used as input for a convolutional neural network to detect and classify the arrhythmias. Finally, the classification results are used to perform a prediction of arrhythmias with nonlinear regression model. The method is illustrated using the MIT-BIH database.

Stress detection using ECG and EMG signals: A comprehensive study

BACKGROUND AND OBJECTIVE

In recent years, stress and mental health have been considered as important worldwide concerns. Stress detection using physiological signals such as electrocardiogram (ECG), skin conductance (SC), electromyogram (EMG) and electroencephalogram (EEG) is a traditional approach. However, the effect of stress on the EMG signal of different muscles and the efficacy of combination of the EMG and other biological signals for stress detection have not been taken into account yet. This paper presents a comprehensive review of the EMG signal of the right and left trapezius and right and left erector spinae muscles for multi-level stress recognition. Also, the ECG signal was employed to evaluate the efficacy of EMG signals for stress detection.

METHODS

Both EMG and ECG signals were acquired simultaneously from 34 healthy students (23 females and 11 males, aged 20-37 years). Mental arithmetic, Stroop color-word test, time pressure, and stressful environment were employed to induce stress in the laboratory.

RESULTS

The accuracies of stress recognition in two, three and four levels were 100%, 97.6%, and 96.2%, respectively, obtained from the distinct combination of feature selection and machine learning algorithms.

CONCLUSIONS

The comparison of stress detection accuracies resulted from EMG and ECG indicators demonstrated the strong ability and the effectiveness of EMG signal for multi-level stress detection.

Towards Intelligent Data Analytics: A Case Study in Driver Cognitive Load Classification

One debatable issue in traffic safety research is that the cognitive load by secondary tasks reduces primary task performance, i.e., driving. In this paper, the study adopted a version of the n-back task as a cognitively loading secondary task on the primary task, i.e., driving; where drivers drove in three different simulated driving scenarios. This paper has taken a multimodal approach to perform 'intelligent multivariate data analytics' based on machine learning (ML). Here, the k-nearest neighbour (k-NN), support vector machine (SVM), and random forest (RF) are used for driver cognitive load classification. Moreover, physiological measures have proven to be sophisticated in cognitive load identification, yet it suffers from confounding factors and noise. Therefore, this work uses multi-component signals, i.e., physiological measures and vehicular features to overcome that problem. Both multiclass and binary classifications have been performed to distinguish normal driving from cognitive load tasks. To identify the optimal feature set, two feature selection algorithms, i.e., sequential forward floating selection (SFFS) and random forest have been applied where out of 323 features, a subset of 42 features has been selected as the best feature subset. For the classification, RF has shown better performance with F1-score of 0.75 and 0.80 than two other algorithms. Moreover, the result shows that using multicomponent features classifiers could classify better than using features from a single source.

Measurement and identification of mental workload during simulated computer tasks with multimodal methods and machine learning

This study attempted to multimodally measure mental workload and validate indicators for estimating mental workload. A simulated computer work composed of mental arithmetic tasks with different levels of difficulty was designed and used in the experiment to measure physiological signals (heart rate, heart rate variability, electromyography, electrodermal activity, and respiration), subjective ratings of mental workload (the NASA Task Load Index), and task performance. The indices from electrodermal activity and respiration had a significant increment as task difficulty increased. There were no significant differences between the average heart rate and the low-frequency/high-frequency ratio among tasks. The classification of mental workload using combined indices as inputs showed that classification models combining physiological signals and task performance can reach satisfying accuracy at 96.4% and an accuracy of 78.3% when only using physiological indices as inputs. The present study also showed that ECG and EDA signals have good discriminating power for mental workload detection. Practitioner summary: The methods used in this study could be applied to office workers, and the findings provide preliminary support and theoretical exploration for follow-up early mental workload detection systems, whose implementation in the real world could beneficially impact worker health and company efficiency. Abbreviations: NASA-TLX: the national aeronautics and space administration-task load index; ECG: electrocardiographic; EDA: electrodermal activity; EEG: electroencephalogram; LDA: linear discriminant analysis; SVM: support vector machine; KNN: k-nearest neighbor; ANNs: artificial neural networks; EMG: electromyography; PPG: photoplethysmography; SD: standard deviation; BMI: body mass index; DSSQ: dundee stress state questionnaire; ANOVA: analysis of variance; SC: skin conductance; RMS: root mean square; AVHR: the average heart rate; HR: heart rate; LF/HF: the ratio between the low frequencies band and the high frequency band; PSD: power spectral density; MF: median frequency; HRV: heart rate variability; BPNN: backpropagation neural network.

Fuzzy support vector machine-based personalizing method to address the inter-subject variance problem of physiological signals in a driver monitoring system

Physiological signals can be utilized to monitor conditions of a driver, but the inter-subject variance of physiological signals can degrade the classification accuracy of the monitoring system. Personalization of the system using the data of a tested subject, called local data, can be a solution, but the acquisition of sufficient local data may not be possible in real situations. Therefore, this paper proposes an effective personalizing method using small-sized local data. The proposed method utilizes a fuzzy support vector machine to allocate higher weight to the local data than to others, and a fuzzy membership is assigned to the training data by analyzing the importance of each datum. Three classification problems for a physiological signal-based driver monitoring system are introduced and utilized to validate the proposed method. The classification accuracy is compared with that of other personalizing methods, and the results show that the proposed method achieves a better accuracy on average, which is 3.46% higher than that of the simple approach using a basic support vector machine, thereby proving its effectiveness. The proposed method can train a personalized classifier with improved accuracy for a tested subject. The advantages of the proposed method can be utilized to develop a practical driver monitoring system.

A Novel Classification Method for a Driver's Cognitive Stress Level by Transferring Interbeat Intervals of the ECG Signal to Pictures

In this study, a novel classification method for a driver's cognitive stress level was proposed, whereby the interbeat intervals extracted from an electrocardiogram (ECG) signal were transferred to pictures, and a convolution neural network (CNN) was used to train the pictures to classify a driver's cognitive stress level. First, we defined three levels of tasks and collected the ECG signal of the driver at different cognitive stress levels by designing and performing a driving simulation experiment. We extracted the interbeat intervals and converted them to pictures according to the number of consecutive interbeat intervals in each picture. Second, the CNN model was used to train the data set to recognize the cognitive stress levels. Classification accuracies of 100%, 91.6% and 92.8% were obtained for the training set, validation set and test set, respectively, and were compared with those the BP neural network. Last, we discussed the influence of the number of interbeat intervals in each picture on the performance of the proposed classification method. The results showed that the performance initially improved with an increase in the number of interbeat intervals. A downward trend was observed when the number exceeded 40, and when the number was 40, the model performed best with the highest accuracy (98.79%) and a relatively low relative standard deviation (0.019).

Towards Mixed-Initiative Human-Robot Interaction: Assessment of Discriminative Physiological and Behavioral Features for Performance Prediction

The design of human–robot interactions is a key challenge to optimize operational performance. A promising approach is to consider mixed-initiative interactions in which the tasks and authority of each human and artificial agents are dynamically defined according to their current abilities. An important issue for the implementation of mixed-initiative systems is to monitor human performance to dynamically drive task allocation between human and artificial agents (i.e., robots). We, therefore, designed an experimental scenario involving missions whereby participants had to cooperate with a robot to fight fires while facing hazards. Two levels of robot automation (manual vs. autonomous) were randomly manipulated to assess their impact on the participants’ performance across missions. Cardiac activity, eye-tracking, and participants’ actions on the user interface were collected. The participants performed differently to an extent that we could identify high and low score mission groups that also exhibited different behavioral, cardiac and ocular patterns. More specifically, our findings indicated that the higher level of automation could be beneficial to low-scoring participants but detrimental to high-scoring ones, and vice versa. In addition, inter-subject single-trial classification results showed that the studied behavioral and physiological features were relevant to predict mission performance. The highest average balanced accuracy (74%) was reached using the features extracted from all input devices. These results suggest that an adaptive HRI driving system, that would aim at maximizing performance, would be capable of analyzing such physiological and behavior markers online to further change the level of automation when it is relevant for the mission purpose.

Driving Fatigue Detection from EEG Using a Modified PCANet Method

The rapid development of the automotive industry has brought great convenience to our life, which also leads to a dramatic increase in the amount of traffic accidents. A large proportion of traffic accidents were caused by driving fatigue. EEG is considered as a direct, effective, and promising modality to detect driving fatigue. In this study, we presented a novel feature extraction strategy based on a deep learning model to achieve high classification accuracy and efficiency in using EEG for driving fatigue detection. EEG signals were recorded from six healthy volunteers in a simulated driving experiment. The feature extraction strategy was developed by integrating the principal component analysis (PCA) and a deep learning model called PCA network (PCANet). In particular, the principal component analysis (PCA) was used to preprocess EEG data to reduce its dimension in order to overcome the limitation of dimension explosion caused by PCANet, making this approach feasible for EEG-based driving fatigue detection. Results demonstrated high and robust performance of the proposed modified PCANet method with classification accuracy up to 95%, which outperformed the conventional feature extraction strategies in the field. We also identified that the parietal and occipital lobes of the brain were strongly associated with driving fatigue. This is the first study, to the best of our knowledge, to practically apply the modified PCANet technique for EEG-based driving fatigue detection.

Exploring the fatigue affecting electroencephalography based functional brain networks during real driving in young males

In recent years, a large proportion of traffic accidents are caused by driver fatigue. The brain has been conceived as a complex network, whose function can be assessed with EEG. Hence, in this research, fourteen subjects participated in the real driving experiments, and a comprehensive EEG-based expert system was designed for detecting driver fatigue. Collected EEG signals were first decomposed into delta-range, theta-range, alpha-range and beta-range by wavelet packet transform (WPT). Unlike other approaches, a multi-channel network construction method based on Phase Lag Index (PLI) was then proposed in this paper. Finally, the functional connectivity between alert state (at the beginning of the drive) and fatigue state (at the end of the drive) in multiple frequency bands were analyzed. The results indicate that functional connectivity of the brain area was significantly different between alert and fatigue states, especially in alpha-range and beta-range. Particularly, the frontal-to-parietal functional connectivity was weakened. Meanwhile, lower clustering coefficient (C) values and higher characteristic path length (L) values were observed in fatigue state in comparison with alert state. Based on this, two new EEG feature selection approaches, C and L in the corresponding sub-frequency range were applied to feature recognition and classification system. Using a support vector machine (SVM) machine learning algorithm, these features were combined to distinguish between alert and fatigue states, achieving an accuracy of 94.4%, precision of 94.3%, sensitivity of 94.6% and false alarm rate of 5.7%. The results suggest that brain network analysis approaches combined with SVM are helpful to alert drivers while being sleepy or even fatigue.

The effect of motor task precision on pupil diameter

It is well established that an increase in cognitive task demands is associated with increased pupil diameter. However, the effect of increased motor task demands on pupil diameter is less clear. Previous research indicates that higher motor task complexity increases pupil diameter but suggests that higher motor task precision demands may decrease pupil diameter during task movement. The current study investigated the effect of increased motor task precision on pupil diameter using a Fitts' Law movement task to manipulate motor response precision. Increased precision demands were associated with reduced pupil diameter during the response preparation and response execution phases of the movement trials. This result has implications for the interpretation of pupil diameter as an index of workload during tasks which involve precise motor movements.