CorrNet: Fine-Grained Emotion Recognition for Video Watching Using Wearable Physiological Sensors

Optimal Electrodermal Activity Segment for Enhanced Emotion Recognition Using Spectrogram-Based Feature Extraction and Machine Learning

In clinical and scientific research on emotion recognition using physiological signals, selecting the appropriate segment is of utmost importance for enhanced results. In our study, we optimized the electrodermal activity (EDA) segment for an emotion recognition system. Initially, we obtained EDA signals from two publicly available datasets: the Continuously annotated signals of emotion (CASE) and Wearable stress and affect detection (WESAD) for 4-class dimensional and three-class categorical emotional classification, respectively. These signals were pre-processed, and decomposed into phasic signals using the 'convex optimization to EDA' method. Further, the phasic signals were segmented into two equal parts, each subsequently segmented into five nonoverlapping windows. Spectrograms were then generated using short-time Fourier transform and Mel-frequency cepstrum for each window, from which we extracted 85 features. We built four machine learning models for the first part, second part, and whole phasic signals to investigate their performance in emotion recognition. In the CASE dataset, we achieved the highest multi-class accuracy of 62.54% using the whole phasic and 61.75% with the second part phasic signals. Conversely, the WESAD dataset demonstrated superior performance in three-class emotions classification, attaining an accuracy of 96.44% for both whole phasic and second part phasic segments. As a result, the second part of EDA is strongly recommended for optimal outcomes.

Generalizable machine learning for stress monitoring from wearable devices: A systematic literature review

INTRODUCTION

Wearable sensors have shown promise as a non-intrusive method for collecting biomarkers that may correlate with levels of elevated stress. Stressors cause a variety of biological responses, and these physiological reactions can be measured using biomarkers including Heart Rate Variability (HRV), Electrodermal Activity (EDA) and Heart Rate (HR) that represent the stress response from the Hypothalamic-Pituitary-Adrenal (HPA) axis, the Autonomic Nervous System (ANS), and the immune system. While Cortisol response magnitude remains the gold standard indicator for stress assessment [1], recent advances in wearable technologies have resulted in the availability of a number of consumer devices capable of recording HRV, EDA and HR sensor biomarkers, amongst other signals. At the same time, researchers have been applying machine learning techniques to the recorded biomarkers in order to build models that may be able to predict elevated levels of stress.

OBJECTIVE

The aim of this review is to provide an overview of machine learning techniques utilized in prior research with a specific focus on model generalization when using these public datasets as training data. We also shed light on the challenges and opportunities that machine learning-enabled stress monitoring and detection face.

METHODS

This study reviewed published works contributing and/or using public datasets designed for detecting stress and their associated machine learning methods. The electronic databases of Google Scholar, Crossref, DOAJ and PubMed were searched for relevant articles and a total of 33 articles were identified and included in the final analysis. The reviewed works were synthesized into three categories of publicly available stress datasets, machine learning techniques applied using those, and future research directions. For the machine learning studies reviewed, we provide an analysis of their approach to results validation and model generalization. The quality assessment of the included studies was conducted in accordance with the IJMEDI checklist [2].

RESULTS

A number of public datasets were identified that are labeled for stress detection. These datasets were most commonly produced from sensor biomarker data recorded using the Empatica E4 device, a well-studied, medical-grade wrist-worn wearable that provides sensor biomarkers most notable to correlate with elevated levels of stress. Most of the reviewed datasets contain less than twenty-four hours of data, and the varied experimental conditions and labeling methodologies potentially limit their ability to generalize for unseen data. In addition, we discuss that previous works show shortcomings in areas such as their labeling protocols, lack of statistical power, validity of stress biomarkers, and model generalization ability.

CONCLUSION

Health tracking and monitoring using wearable devices is growing in popularity, while the generalization of existing machine learning models still requires further study, and research in this area will continue to provide improvements as newer and more substantial datasets become available.

Predicting Emotion with Biosignals: A Comparison of Classification and Regression Models for Estimating Valence and Arousal Level Using Wearable Sensors

This study aims to predict emotions using biosignals collected via wrist-worn sensor and evaluate the performance of different prediction models. Two dimensions of emotions were considered: valence and arousal. The data collected by the sensor were used in conjunction with target values obtained from questionnaires. A variety of classification and regression models were compared, including Long Short-Term Memory (LSTM) models. Additionally, the effects of different normalization methods and the impact of using different sensors were studied, and the way in which the results differed between the study subjects was analyzed. The results revealed that regression models generally performed better than classification models, with LSTM regression models achieving the best results. The normalization method called baseline reduction was found to be the most effective, and when used with an LSTM-based regression model it achieved high accuracy in detecting valence (mean square error = 0.43 and R2-score = 0.71) and arousal (mean square error = 0.59 and R2-score = 0.81). Moreover, it was found that even if all biosignals were not used in the training phase, reliable models could be obtained; in fact, for certain study subjects the best results were obtained using only a few of the sensors.

An AI-Inspired Spatio-Temporal Neural Network for EEG-Based Emotional Status

The accurate identification of the human emotional status is crucial for an efficient human-robot interaction (HRI). As such, we have witnessed extensive research efforts made in developing robust and accurate brain-computer interfacing models based on diverse biosignals. In particular, previous research has shown that an Electroencephalogram (EEG) can provide deep insight into the state of emotion. Recently, various handcrafted and deep neural network (DNN) models were proposed by researchers for extracting emotion-relevant features, which offer limited robustness to noise that leads to reduced precision and increased computational complexity. The DNN models developed to date were shown to be efficient in extracting robust features relevant to emotion classification; however, their massive feature dimensionality problem leads to a high computational load. In this paper, we propose a bag-of-hybrid-deep-features (BoHDF) extraction model for classifying EEG signals into their respective emotion class. The invariance and robustness of the BoHDF is further enhanced by transforming EEG signals into 2D spectrograms before the feature extraction stage. Such a time-frequency representation fits well with the time-varying behavior of EEG patterns. Here, we propose to combine the deep features from the GoogLeNet fully connected layer (one of the simplest DNN models) together with the OMTLBP_SMC texture-based features, which we recently developed, followed by a K-nearest neighbor (KNN) clustering algorithm. The proposed model, when evaluated on the DEAP and SEED databases, achieves a 93.83 and 96.95% recognition accuracy, respectively. The experimental results using the proposed BoHDF-based algorithm show an improved performance in comparison to previously reported works with similar setups.

Machine Learning Techniques for Arousal Classification from Electrodermal Activity: A Systematic Review

This article introduces a systematic review on arousal classification based on electrodermal activity (EDA) and machine learning (ML). From a first set of 284 articles searched for in six scientific databases, fifty-nine were finally selected according to various criteria established. The systematic review has made it possible to analyse all the steps to which the EDA signals are subjected: acquisition, pre-processing, processing and feature extraction. Finally, all ML techniques applied to the features of these signals for arousal classification have been studied. It has been found that support vector machines and artificial neural networks stand out within the supervised learning methods given their high-performance values. In contrast, it has been shown that unsupervised learning is not present in the detection of arousal through EDA. This systematic review concludes that the use of EDA for the detection of arousal is widely spread, with particularly good results in classification with the ML methods found.

Smart Consumer Wearables as Digital Diagnostic Tools: A Review

The increasing usage of smart wearable devices has made an impact not only on the lifestyle of the users, but also on biological research and personalized healthcare services. These devices, which carry different types of sensors, have emerged as personalized digital diagnostic tools. Data from such devices have enabled the prediction and detection of various physiological as well as psychological conditions and diseases. In this review, we have focused on the diagnostic applications of wrist-worn wearables to detect multiple diseases such as cardiovascular diseases, neurological disorders, fatty liver diseases, and metabolic disorders, including diabetes, sleep quality, and psychological illnesses. The fruitful usage of wearables requires fast and insightful data analysis, which is feasible through machine learning. In this review, we have also discussed various machine-learning applications and outcomes for wearable data analyses. Finally, we have discussed the current challenges with wearable usage and data, and the future perspectives of wearable devices as diagnostic tools for research and personalized healthcare domains.

Design of Interactive Vocal Guidance and Artistic Psychological Intervention System Based on Emotion Recognition

The research on artistic psychological intervention to judge emotional fluctuations by extracting emotional features from interactive vocal signals has become a research topic with great potential for development. Based on the interactive vocal music instruction theory of emotion recognition, this paper studies the design of artistic psychological intervention system. This paper uses the vocal music emotion recognition algorithm to first train the interactive recognition network, in which the input is a row vector composed of different vocal music characteristics, and finally recognizes the vocal music of different emotional categories, which solves the problem of low data coupling in the artistic psychological intervention system. Among them, the vocal music emotion recognition experiment based on the interactive recognition network is mainly carried out from six aspects: the number of iterative training, the vocal music instruction rate, the number of emotion recognition signal nodes in the artistic psychological intervention layer, the number of sample sets, different feature combinations, and the number of emotion types. The input data of the system is a training class learning video, and actions and expressions need to be recognized before scoring. In the simulation process, before the completion of the sample indicators is unbalanced, the R language statistical analysis tool is used to balance the existing unbalanced data based on the artificial data synthesis method, and 279 uniformly classified samples are obtained. The 279∗7 dataset was used for statistical identification of the participants. The experimental results show that under the guidance of four different interactive vocal music, the vocal emotion recognition rate is between 65.85%-91.00%, which promotes the intervention of music therapy on artistic psychological intervention.

Personalized PPG Normalization Based on Subject Heartbeat in Resting State Condition

Physiological responses are currently widely used to recognize the affective state of subjects in real-life scenarios. However, these data are intrinsically subject-dependent, making machine learning techniques for data classification not easily applicable due to inter-subject variability. In this work, the reduction of inter-subject heterogeneity was considered in the case of Photoplethysmography (PPG), which was successfully used to detect stress and evaluate experienced cognitive load. To face the inter-subject heterogeneity, a novel personalized PPG normalization is herein proposed. A subject-normalized discrete domain where the PPG signals are properly re-scaled is introduced, considering the subject’s heartbeat frequency in resting state conditions. The effectiveness of the proposed normalization was evaluated in comparison to other normalization procedures in a binary classification task, where cognitive load and relaxed state were considered. The results obtained on two different datasets available in the literature confirmed that applying the proposed normalization strategy permitted increasing the classification performance.

The Concept of Advanced Multi-Sensor Monitoring of Human Stress

Many people live under stressful conditions which has an adverse effect on their health. Human stress, especially long-term one, can lead to a serious illness. Therefore, monitoring of human stress influence can be very useful. We can monitor stress in strictly controlled laboratory conditions, but it is time-consuming and does not capture reactions, on everyday stressors or in natural environment using wearable sensors, but with limited accuracy. Therefore, we began to analyze the current state of promising wearable stress-meters and the latest advances in the record of related physiological variables. Based on these results, we present the concept of an accurate, reliable and easier to use telemedicine device for long-term monitoring of people in a real life. In our concept, we ratify with two synchronized devices, one on the finger and the second on the chest. The results will be obtained from several physiological variables including electrodermal activity, heart rate and respiration, body temperature, blood pressure and others. All these variables will be measured using a coherent multi-sensors device. Our goal is to show possibilities and trends towards the production of new telemedicine equipment and thus, opening the door to a widespread application of human stress-meters.

Facial emotion mimicry in older adults with and without cognitive impairments due to Alzheimer's disease

Facial expression of humans is one of the main channels of everyday communication. The reported research work investigated communication regarding the pattern of emotional expression of healthy older adults and with mild cognitive impairments (MCI) or Alzheimer's disease (AD). It focuses on mimicking of displayed emotional facial expression on a sample of 25 older adults (healthy, MCI and AD patients). The adequacy of the patients' individual facial expressions in six basic emotions was measured with the Kinect 3D recording of the participants' facial expressions and compared to their own typical emotional facial expressions. The reactions were triggered by mimicking 49 still pictures of emotional facial expressions. No statistically significant differences in terms of frequency nor adequacy of emotional facial expression were reported in healthy and MCI groups. Unique patterns of emotional expressions have been observed in the AD group. Further investigating the pattern of older adults' facial expression may decrease the misunderstandings and increase the quality of life of the patients.