An Ensemble Learning Approach for Electrocardiogram Sensor Based Human Emotion Recognition

ECG Multi-Emotion Recognition Based on Heart Rate Variability Signal Features Mining

Heart rate variability (HRV) serves as a significant physiological measure that mirrors the regulatory capacity of the cardiac autonomic nervous system. It not only indicates the extent of the autonomic nervous system's influence on heart function but also unveils the connection between emotions and psychological disorders. Currently, in the field of emotion recognition using HRV, most methods focus on feature extraction through the comprehensive analysis of signal characteristics; however, these methods lack in-depth analysis of the local features in the HRV signal and cannot fully utilize the information of the HRV signal. Therefore, we propose the HRV Emotion Recognition (HER) method, utilizing the amplitude level quantization (ALQ) technique for feature extraction. First, we employ the emotion quantification analysis (EQA) technique to impartially assess the semantic resemblance of emotions within the domain of emotional arousal. Then, we use the ALQ method to extract rich local information features by analyzing the local information in each frequency range of the HRV signal. Finally, the extracted features are classified using a logistic regression (LR) classification algorithm, which can achieve efficient and accurate emotion recognition. According to the experiment findings, the approach surpasses existing techniques in emotion recognition accuracy, achieving an average accuracy rate of 84.3%. Therefore, the HER method proposed in this paper can effectively utilize the local features in HRV signals to achieve efficient and accurate emotion recognition. This will provide strong support for emotion research in psychology, medicine, and other fields.

Using Wearable Passive Sensing to Predict Binge Eating in Response to Negative Affect Among Individuals With Transdiagnostic Binge Eating: Protocol for an Observational Study

BACKGROUND

Binge eating (BE), characterized by eating a large amount of food accompanied by a sense of loss of control over eating, is a public health crisis. Negative affect is a well-established antecedent for BE. The affect regulation model of BE posits that elevated negative affect increases momentary risk for BE, as engaging in BE alleviates negative affect and reinforces the behavior. The eating disorder field's capacity to identify moments of elevated negative affect, and thus BE risk, has exclusively relied on ecological momentary assessment (EMA). EMA involves the completion of surveys in real time on one's smartphone to report behavioral, cognitive, and emotional symptoms throughout the day. Although EMA provides ecologically valid information, EMA surveys are often delivered only 5-6 times per day, involve self-report of affect intensity only, and are unable to assess affect-related physiological arousal. Wearable, psychophysiological sensors that measure markers of affect arousal including heart rate, heart rate variability, and electrodermal activity may augment EMA surveys to improve accurate real-time prediction of BE. These sensors can objectively and continuously measure biomarkers of nervous system arousal that coincide with affect, thus allowing them to measure affective trajectories on a continuous timescale, detect changes in negative affect before the individual is consciously aware of them, and reduce user burden to improve data completeness. However, it is unknown whether sensor features can distinguish between positive and negative affect states, given that physiological arousal may occur during both negative and positive affect states.

OBJECTIVE

The aims of this study are (1) to test the hypothesis that sensor features will distinguish positive and negative affect states in individuals with BE with >60% accuracy and (2) test the hypothesis that a machine learning algorithm using sensor data and EMA-reported negative affect to predict the occurrence of BE will predict BE with greater accuracy than an algorithm using EMA-reported negative affect alone.

METHODS

This study will recruit 30 individuals with BE who will wear Fitbit Sense 2 wristbands to passively measure heart rate and electrodermal activity and report affect and BE on EMA surveys for 4 weeks. Machine learning algorithms will be developed using sensor data to distinguish instances of high positive and high negative affect (aim 1) and to predict engagement in BE (aim 2).

RESULTS

This project will be funded from November 2022 to October 2024. Recruitment efforts will be conducted from January 2023 through March 2024. Data collection is anticipated to be completed in May 2024.

CONCLUSIONS

This study is anticipated to provide new insight into the relationship between negative affect and BE by integrating wearable sensor data to measure affective arousal. The findings from this study may set the stage for future development of more effective digital ecological momentary interventions for BE.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/47098.

A new deep convolutional neural network incorporating attentional mechanisms for ECG emotion recognition

Using ECG signals captured by wearable devices for emotion recognition is a feasible solution. We propose a deep convolutional neural network incorporating attentional mechanisms for ECG emotion recognition. In order to address the problem of individuality differences in emotion recognition tasks, we incorporate an improved Convolutional Block Attention Module (CBAM) into the proposed deep convolutional neural network. The deep convolutional neural network is responsible for capturing ECG features. Channel attention in CBAM is responsible for adding weight information to ECG features of different channels and spatial attention is responsible for the weighted representation of ECG features of different regions inside the channel. We used three publicly available datasets, WESAD, DREAMER, and ASCERTAIN, for the ECG emotion recognition task. The new state-of-the-art results are set in three datasets for multi-class classification results, WESAD for tri-class results, and ASCERTAIN for two-category results, respectively. A large number of experiments are performed, providing an interesting analysis of the design of the convolutional structure parameters and the role of the attention mechanism used. We propose to use large convolutional kernels to improve the effective perceptual field of the model and thus fully capture the ECG signal features, which achieves better performance compared to the commonly used small kernels. In addition, channel attention and spatial attention were added to the deep convolutional model separately to explore their contribution levels. We found that in most cases, channel attention contributed to the model at a higher level than spatial attention.

Multi-Input Speech Emotion Recognition Model Using Mel Spectrogram and GeMAPS

The existing research on emotion recognition commonly uses mel spectrogram (MelSpec) and Geneva minimalistic acoustic parameter set (GeMAPS) as acoustic parameters to learn the audio features. MelSpec can represent the time-series variations of each frequency but cannot manage multiple types of audio features. On the other hand, GeMAPS can handle multiple audio features but fails to provide information on their time-series variations. Thus, this study proposes a speech emotion recognition model based on a multi-input deep neural network that simultaneously learns these two audio features. The proposed model comprises three parts, specifically, for learning MelSpec in image format, learning GeMAPS in vector format, and integrating them to predict the emotion. Additionally, a focal loss function is introduced to address the imbalanced data problem among the emotion classes. The results of the recognition experiments demonstrate weighted and unweighted accuracies of 0.6657 and 0.6149, respectively, which are higher than or comparable to those of the existing state-of-the-art methods. Overall, the proposed model significantly improves the recognition accuracy of the emotion "happiness", which has been difficult to identify in previous studies owing to limited data. Therefore, the proposed model can effectively recognize emotions from speech and can be applied for practical purposes with future development.

A Review on the Applications of Time-Frequency Methods in ECG Analysis

The joint time-frequency analysis method represents a signal in both time and frequency. Thus, it provides more information compared to other one-dimensional methods. Several researchers recently used time-frequency methods such as the wavelet transform, short-time Fourier transform, empirical mode decomposition and reported impressive results in various electrophysiological studies. The current review provides comprehensive knowledge about different time-frequency methods and their applications in various ECG-based analyses. Typical applications include ECG signal denoising, arrhythmia detection, sleep apnea detection, biometric identification, emotion detection, and driver drowsiness detection. The paper also discusses the limitations of these methods. The review will form a reference for future researchers willing to conduct research in the same field.

Cognitive physiological data analysis based on the XGBoost algorithm to realize positive perceptual sample classification

Designers refer to existing product cases and innovate products to develop new products. However, when designers screen product cases, there is no user participation, which leads to the lack of user-side knowledge and emotional drive that is very important for design. Therefore, it is necessary to play the role of user emotional knowledge in promoting the whole design process. This paper proposes the concept of the positive perceptual sample, which applies the knowledge emotion integration of designers and users to the screening sample case stage at the beginning of the design process. This study is based on the lack of user-side knowledge and emotional drive of reference cases and integrates user emotion into the reference case screening process. Then, in the emotion measurement process, users’ cognitive data in the screening process are obtained through the eye-brain fusion cognitive experiment. Finally, the XGBoost algorithm is used to process feature index data to realize the classification and recognition of cognitive data and applied to the positive perceptual classification of products. The results show that the classification accuracy of physiological cognitive data with user emotional representation by the XGBoost algorithm is 90.87% . The results of cognitive data classification are applied to the screening of positive perceptual samples, and the satisfaction rate is 98.35% . The results show that the method proposed in this paper provides a new source of ideas for obtaining positive perceptual samples and can be applied to new product development.

A Machine Learning Approach for Walking Classification in Elderly People with Gait Disorders

Walking ability of elderly individuals, who suffer from walking difficulties, is limited, which restricts their mobility independence. The physical health and well-being of the elderly population are affected by their level of physical activity. Therefore, monitoring daily activities can help improve the quality of life. This becomes especially a huge challenge for those, who suffer from dementia and Alzheimer's disease. Thus, it is of great importance for personnel in care homes/rehabilitation centers to monitor their daily activities and progress. Unlike normal subjects, it is required to place the sensor on the back of this group of patients, which makes it even more challenging to detect walking from other activities. With the latest advancements in the field of health sensing and sensor technology, a huge amount of accelerometer data can be easily collected. In this study, a Machine Learning (ML) based algorithm was developed to analyze the accelerometer data collected from patients with walking difficulties, who live in one of the municipalities in Denmark. The ML algorithm is capable of accurately classifying the walking activity of these individuals with different walking abnormalities. Various statistical, temporal, and spectral features were extracted from the time series data collected using an accelerometer sensor placed on the back of the participants. The back sensor placement is desirable in patients with dementia and Alzheimer's disease since they may remove visible sensors to them due to the nature of their diseases. Then, an evolutionary optimization algorithm called Particle Swarm Optimization (PSO) was used to select a subset of features to be used in the classification step. Four different ML classifiers such as k-Nearest Neighbors (kNN), Random Forest (RF), Stacking Classifier (Stack), and Extreme Gradient Boosting (XGB) were trained and compared on an accelerometry dataset consisting of 20 participants. These models were evaluated using the leave-one-group-out cross-validation (LOGO-CV) technique. The Stack model achieved the best performance with average sensitivity, positive predictive values (precision), F₁-score, and accuracy of 86.85%, 93.25%, 88.81%, and 93.32%, respectively, to classify walking episodes. In general, the empirical results confirmed that the proposed models are capable of classifying the walking episodes despite the challenging sensor placement on the back of the patients, who suffer from walking disabilities.

M1M2: Deep-Learning-Based Real-Time Emotion Recognition from Neural Activity

Emotion recognition, or the ability of computers to interpret people's emotional states, is a very active research area with vast applications to improve people's lives. However, most image-based emotion recognition techniques are flawed, as humans can intentionally hide their emotions by changing facial expressions. Consequently, brain signals are being used to detect human emotions with improved accuracy, but most proposed systems demonstrate poor performance as EEG signals are difficult to classify using standard machine learning and deep learning techniques. This paper proposes two convolutional neural network (CNN) models (M1: heavily parameterized CNN model and M2: lightly parameterized CNN model) coupled with elegant feature extraction methods for effective recognition. In this study, the most popular EEG benchmark dataset, the DEAP, is utilized with two of its labels, valence, and arousal, for binary classification. We use Fast Fourier Transformation to extract the frequency domain features, convolutional layers for deep features, and complementary features to represent the dataset. The M1 and M2 CNN models achieve nearly perfect accuracy of 99.89% and 99.22%, respectively, which outperform every previous state-of-the-art model. We empirically demonstrate that the M2 model requires only 2 seconds of EEG signal for 99.22% accuracy, and it can achieve over 96% accuracy with only 125 milliseconds of EEG data for valence classification. Moreover, the proposed M2 model achieves 96.8% accuracy on valence using only 10% of the training dataset, demonstrating our proposed system's effectiveness. Documented implementation codes for every experiment are published for reproducibility.

Emotion Recognition: Photoplethysmography and Electrocardiography in Comparison

Automatically recognizing negative emotions, such as anger or stress, and also positive ones, such as euphoria, can contribute to improving well-being. In real-life, emotion recognition is a difficult task since many of the technologies used for this purpose in both laboratory and clinic environments, such as electroencephalography (EEG) and electrocardiography (ECG), cannot realistically be used. Photoplethysmography (PPG) is a non-invasive technology that can be easily integrated into wearable sensors. This paper focuses on the comparison between PPG and ECG concerning their efficacy in detecting the psychophysical and affective states of the subjects. It has been confirmed that the levels of accuracy in the recognition of affective variables obtained by PPG technology are comparable to those achievable with the more traditional ECG technology. Moreover, the affective psychological condition of the participants (anxiety and mood levels) may influence the psychophysiological responses recorded during the experimental tests.

Automated Emotion Identification Using Fourier-Bessel Domain-Based Entropies

Human dependence on computers is increasing day by day; thus, human interaction with computers must be more dynamic and contextual rather than static or generalized. The development of such devices requires knowledge of the emotional state of the user interacting with it; for this purpose, an emotion recognition system is required. Physiological signals, specifically, electrocardiogram (ECG) and electroencephalogram (EEG), were studied here for the purpose of emotion recognition. This paper proposes novel entropy-based features in the Fourier-Bessel domain instead of the Fourier domain, where frequency resolution is twice that of the latter. Further, to represent such non-stationary signals, the Fourier-Bessel series expansion (FBSE) is used, which has non-stationary basis functions, making it more suitable than the Fourier representation. EEG and ECG signals are decomposed into narrow-band modes using FBSE-based empirical wavelet transform (FBSE-EWT). The proposed entropies of each mode are computed to form the feature vector, which are further used to develop machine learning models. The proposed emotion detection algorithm is evaluated using publicly available DREAMER dataset. K-nearest neighbors (KNN) classifier provides accuracies of 97.84%, 97.91%, and 97.86% for arousal, valence, and dominance classes, respectively. Finally, this paper concludes that the obtained entropy features are suitable for emotion recognition from given physiological signals.

Measuring emotional variables in occupational performance: A scoping review

BACKGROUND: As interest in job-related psychology increased, the need to focus on understanding workplace stress was emphasized. Negative emotional states such as anxiety and stress permeate the organization and, if uncontrolled, can negatively impact the health and work performance of workers. Therefore, attempts to analyze various signals to understand human emotional states or attitudes may be important for future technological development. OBJECTIVE: The purpose of this study was to identify what biological variables can discriminate emotions that can significantly affect work results. METHODS: Databases (Embase, PsychINFO, PubMed, and CINAHL) were searched for all relevant literature published as of December 31, 2019. RESULTS: Brain activity (BA) and heart rate (HR) or heart rate variability (HRV) are adequate for assessing negative emotions, while BA, galvanic skin response (GSR), and salivary samples (SS) can confirm positive and negative emotions. CONCLUSION: In the future, researchers should study measurement tools and bio-related variables while workers perform tasks and develop intervention strategies to address emotions associated with work. This may enable workers to perform tasks more efficiently, prevent accidents, and satisfy clients.

Visualization deep learning model for automatic arrhythmias classification

With the improvement of living standards, heart disease has become one of the common diseases that threaten human health. As a widely used reliable, non-invasive technology, the Electrocardiography (ECG) data has been increasingly used for diagnosing cardiovascular diseases. With the rapid growth of ECG examinations and the shortage of cardiologists, accurate and automatic recognition of ECG patterns has become a research hotspot. In order to improve the accuracy in detecting abnormal ECG patterns, this paper proposes a hybrid 1D Resnet-GRU consisting of the Resnet and gated recurrent unit (GRU) modules to implement classification of arrhythmias from 12-lead ECG recordings. In addition, the focal Loss function is used to solve the problem of unbalanced datasets. Based on the proposed 1D Resnet-GRU model, the Grad-CAM++ mechanism, one of class-discriminative visualization methods, has been employed to the trained network model and generate thermal images superimposed on raw signals, so as to improve interpretability and transparency for arrhythmia classification. Experimental results show the proposed Resnet-GRU method can improve the F_1 score and accuracy effectively, and Grad-CAM++ can provide interpretability and clinical information for arrhythmia classification.

Machine Learning Algorithms for Detection and Classifications of Emotions in Contact Center Applications

Over the past few years, virtual assistant solutions used in Contact Center systems are gaining popularity. One of the main tasks of the virtual assistant is to recognize the intentions of the customer. It is important to note that quite often the actual intention expressed in a conversation is also directly influenced by the emotions that accompany that conversation. Unfortunately, scientific literature has not identified what specific types of emotions in Contact Center applications are relevant to the activities they perform. Therefore, the main objective of this work was to develop an Emotion Classification for Machine Detection of Affect-Tinged Conversational Contents dedicated directly to the Contact Center industry. In the conducted study, Contact Center voice and text channels were considered, taking into account the following families of emotions: anger, fear, happiness, sadness vs. affective neutrality of the statements. The obtained results confirmed the usefulness of the proposed classification—for the voice channel, the highest efficiency was obtained using the Convolutional Neural Network (accuracy, 67.5%; precision, 80.3; F1-Score, 74.5%), while for the text channel, the Support Vector Machine algorithm proved to be the most efficient (accuracy, 65.9%; precision, 58.5; F1-Score, 61.7%).

Relationship between electrocardiogram-based features and personality traits: Machine learning approach

Background

Based on the known relationship between the human emotion and standard surface electrocardiogram (ECG), we explored the relationship between features extracted from standard ECG recorded during relaxation and seven personality traits (Honesty/humility, Emotionality, eXtraversion, Agreeableness, Conscientiousness, Openness, and Disintegration) by using the machine learning (ML) approach which learns from the ECG‐based features and predicts the appropriate personality trait by adopting an automated software algorithm.

Methods

A total of 71 healthy university students participated in the study. For quantification of 62 ECG‐based parameters (heart rate variability, as well as temporal and amplitude‐based parameters) for each ECG record, we used computation procedures together with publicly available data and code. Among 62 parameters, 34 were segregated into separate features according to their diagnostic relevance in clinical practice. To examine the feature influence on personality trait classification and to perform classification, we used random forest ML algorithm.

Results

Classification accuracy when clinically relevant ECG features were employed was high for Disintegration (81.3%) and Honesty/humility (75.0%) and moderate to high for Openness (73.3%) and Conscientiousness (70%), while it was low for Agreeableness (56.3%), eXtraversion (47.1%), and Emotionality (43.8%). When all calculated features were used, the classification accuracies were the same or lower, except for the eXtraversion (52.9%). Correlation analysis for selected features is presented.

Conclusions

Results indicate that clinically relevant features might be applicable for personality traits prediction, although no remarkable differences were found among selected groups of parameters. Physiological associations of established relationships should be further explored.

Experience with an Affective Robot Assistant for Children with Hearing Disabilities

This study presents an assistive robotic system enhanced with emotion recognition capabilities for children with hearing disabilities. The system is designed and developed for the audiometry tests and rehabilitation of children in a clinical setting and includes a social humanoid robot (Pepper), an interactive interface, gamified audiometry tests, sensory setup and a machine/deep learning based emotion recognition module. Three scenarios involving conventional setup, tablet setup and setup with the robot+tablet are evaluated with 16 children having cochlear implant or hearing aid. Several machine learning techniques and deep learning models are used for the classification of the three test setups and for the classification of the emotions (pleasant, neutral, unpleasant) of children using the recorded physiological signals by E4 wristband. The results show that the collected signals during the tests can be separated successfully and the positive and negative emotions of children can be better distinguished when they interact with the robot than in the other two setups. In addition, the children’s objective and subjective evaluations as well as their impressions about the robot and its emotional behaviors are analyzed and discussed extensively.

Artificial intelligence in the diagnosis and management of arrhythmias

The field of cardiac electrophysiology (EP) had adopted simple artificial intelligence (AI) methodologies for decades. Recent renewed interest in deep learning techniques has opened new frontiers in electrocardiography analysis including signature identification of diseased states. Artificial intelligence advances coupled with simultaneous rapid growth in computational power, sensor technology, and availability of web-based platforms have seen the rapid growth of AI-aided applications and big data research. Changing lifestyles with an expansion of the concept of internet of things and advancements in telecommunication technology have opened doors to population-based detection of atrial fibrillation in ways, which were previously unimaginable. Artificial intelligence-aided advances in 3D cardiac imaging heralded the concept of virtual hearts and the simulation of cardiac arrhythmias. Robotics, completely non-invasive ablation therapy, and the concept of extended realities show promise to revolutionize the future of EP. In this review, we discuss the impact of AI and recent technological advances in all aspects of arrhythmia care.

Electrocardiogram-Based Emotion Recognition Systems and Their Applications in Healthcare-A Review

Affective computing is a field of study that integrates human affects and emotions with artificial intelligence into systems or devices. A system or device with affective computing is beneficial for the mental health and wellbeing of individuals that are stressed, anguished, or depressed. Emotion recognition systems are an important technology that enables affective computing. Currently, there are a lot of ways to build an emotion recognition system using various techniques and algorithms. This review paper focuses on emotion recognition research that adopted electrocardiograms (ECGs) as a unimodal approach as well as part of a multimodal approach for emotion recognition systems. Critical observations of data collection, pre-processing, feature extraction, feature selection and dimensionality reduction, classification, and validation are conducted. This paper also highlights the architectures with accuracy of above 90%. The available ECG-inclusive affective databases are also reviewed, and a popularity analysis is presented. Additionally, the benefit of emotion recognition systems towards healthcare systems is also reviewed here. Based on the literature reviewed, a thorough discussion on the subject matter and future works is suggested and concluded. The findings presented here are beneficial for prospective researchers to look into the summary of previous works conducted in the field of ECG-based emotion recognition systems, and for identifying gaps in the area, as well as in developing and designing future applications of emotion recognition systems, especially in improving healthcare.

Research on the Prediction of Green Plum Acidity Based on Improved XGBoost

The acidity of green plum has an important influence on the fruit’s deep processing. Traditional physical and chemical analysis methods for green plum acidity detection are destructive, time-consuming, and unable to achieve online detection. In response, a rapid and non-destructive detection method based on hyperspectral imaging technology was studied in this paper. Research on prediction performance comparisons between supervised learning methods and unsupervised learning methods is currently popular. To further improve the accuracy of component prediction, a new hyperspectral imaging system was developed, and the kernel principle component analysis—linear discriminant analysis—extreme gradient boosting algorithm (KPCA-LDA-XGB) model was proposed to predict the acidity of green plum. The KPCA-LDA-XGB model is a supervised learning model combined with the extreme gradient boosting algorithm (XGBoost), kernel principal component analysis (KPCA), and linear discriminant analysis (LDA). The experimental results proved that the KPCA-LDA-XGB model offers good acidity predictions for green plum, with a correlation coefficient (R) of 0.829 and a root mean squared error (RMSE) of 0.107 for the prediction set. Compared with the basic XGBoost model, the KPCA-LDA-XGB model showed a 79.4% increase in R and a 31.2% decrease in RMSE. The use of linear, radial basis function (RBF), and polynomial (Poly) kernel functions were also compared and analyzed in this paper to further optimize the KPCA-LDA-XGB model.

Emotion Assessment Using Feature Fusion and Decision Fusion Classification Based on Physiological Data: Are We There Yet?

Emotion recognition based on physiological data classification has been a topic of increasingly growing interest for more than a decade. However, there is a lack of systematic analysis in literature regarding the selection of classifiers to use, sensor modalities, features and range of expected accuracy, just to name a few limitations. In this work, we evaluate emotion in terms of low/high arousal and valence classification through Supervised Learning (SL), Decision Fusion (DF) and Feature Fusion (FF) techniques using multimodal physiological data, namely, Electrocardiography (ECG), Electrodermal Activity (EDA), Respiration (RESP), or Blood Volume Pulse (BVP). The main contribution of our work is a systematic study across five public datasets commonly used in the Emotion Recognition (ER) state-of-the-art, namely: (1) Classification performance analysis of ER benchmarking datasets in the arousal/valence space; (2) Summarising the ranges of the classification accuracy reported across the existing literature; (3) Characterising the results for diverse classifiers, sensor modalities and feature set combinations for ER using accuracy and F1-score; (4) Exploration of an extended feature set for each modality; (5) Systematic analysis of multimodal classification in DF and FF approaches. The experimental results showed that FF is the most competitive technique in terms of classification accuracy and computational complexity. We obtain superior or comparable results to those reported in the state-of-the-art for the selected datasets.

Physiological Sensors Based Emotion Recognition While Experiencing Tactile Enhanced Multimedia

Emotion recognition has increased the potential of affective computing by getting an instant feedback from users and thereby, have a better understanding of their behavior. Physiological sensors have been used to recognize human emotions in response to audio and video content that engages single (auditory) and multiple (two: auditory and vision) human senses, respectively. In this study, human emotions were recognized using physiological signals observed in response to tactile enhanced multimedia content that engages three (tactile, vision, and auditory) human senses. The aim was to give users an enhanced real-world sensation while engaging with multimedia content. To this end, four videos were selected and synchronized with an electric fan and a heater, based on timestamps within the scenes, to generate tactile enhanced content with cold and hot air effect respectively. Physiological signals, i.e., electroencephalography (EEG), photoplethysmography (PPG), and galvanic skin response (GSR) were recorded using commercially available sensors, while experiencing these tactile enhanced videos. The precision of the acquired physiological signals (including EEG, PPG, and GSR) is enhanced using pre-processing with a Savitzky-Golay smoothing filter. Frequency domain features (rational asymmetry, differential asymmetry, and correlation) from EEG, time domain features (variance, entropy, kurtosis, and skewness) from GSR, heart rate and heart rate variability from PPG data are extracted. The K nearest neighbor classifier is applied to the extracted features to classify four (happy, relaxed, angry, and sad) emotions. Our experimental results show that among individual modalities, PPG-based features gives the highest accuracy of 78.57% as compared to EEG- and GSR-based features. The fusion of EEG, GSR, and PPG features further improved the classification accuracy to 79.76% (for four emotions) when interacting with tactile enhanced multimedia.

Emotional State Recognition from Peripheral Physiological Signals Using Fused Nonlinear Features and Team-Collaboration Identification Strategy

Emotion recognition realizing human inner perception has a very important application prospect in human-computer interaction. In order to improve the accuracy of emotion recognition, a novel method combining fused nonlinear features and team-collaboration identification strategy was proposed for emotion recognition using physiological signals. Four nonlinear features, namely approximate entropy (ApEn), sample entropy (SaEn), fuzzy entropy (FuEn) and wavelet packet entropy (WpEn) are employed to reflect emotional states deeply with each type of physiological signal. Then the features of different physiological signals are fused to represent the emotional states from multiple perspectives. Each classifier has its own advantages and disadvantages. In order to make full use of the advantages of other classifiers and avoid the limitation of single classifier, the team-collaboration model is built and the team-collaboration decision-making mechanism is designed according to the proposed team-collaboration identification strategy which is based on the fusion of support vector machine (SVM), decision tree (DT) and extreme learning machine (ELM). Through analysis, SVM is selected as the main classifier with DT and ELM as auxiliary classifiers. According to the designed decision-making mechanism, the proposed team-collaboration identification strategy can effectively employ different classification methods to make decision based on the characteristics of the samples through SVM classification. For samples which are easy to be identified by SVM, SVM directly determines the identification results, whereas SVM-DT-ELM collaboratively determines the identification results, which can effectively utilize the characteristics of each classifier and improve the classification accuracy. The effectiveness and universality of the proposed method are verified by Augsburg database and database for emotion analysis using physiological (DEAP) signals. The experimental results uniformly indicated that the proposed method combining fused nonlinear features and team-collaboration identification strategy presents better performance than the existing methods.

Continuous Vital Monitoring During Sleep and Light Activity Using Carbon-Black Elastomer Sensors

The comfortable, continuous monitoring of vital parameters is still a challenge. The long-term measurement of respiration and cardiovascular signals is required to diagnose cardiovascular and respiratory diseases. Similarly, sleep quality assessment and the recovery period following acute treatments require long-term vital parameter datalogging. To address these requirements, we have developed “VitalCore”, a wearable continuous vital parameter monitoring device in the form of a T-shirt targeting the uninterrupted monitoring of respiration, pulse, and actigraphy. VitalCore uses polymer-based stretchable resistive bands as the primary sensor to capture breathing and pulse patterns from chest expansion. The carbon black-impregnated polymer is implemented in a U-shaped configuration and attached to the T-shirt with “interfacing” material along with the accompanying electronics. In this paper, VitalCore is bench tested and compared to gold standard respiration and pulse measurements to verify its functionality and further to assess the quality of data captured during sleep and during light exercise (walking). We show that these polymer-based sensors could identify respiratory peaks with a sensitivity of 99.44%, precision of 96.23%, and false-negative rate of 0.557% during sleep. We also show that this T-shirt configuration allows the wearer to sleep in all sleeping positions with a negligible difference of data quality. The device was also able to capture breathing during gait with 88.9–100% accuracy in respiratory peak detection.

The Design of CNN Architectures for Optimal Six Basic Emotion Classification Using Multiple Physiological Signals

This study aimed to design an optimal emotion recognition method using multiple physiological signal parameters acquired by bio-signal sensors for improving the accuracy of classifying individual emotional responses. Multiple physiological signals such as respiration (RSP) and heart rate variability (HRV) were acquired in an experiment from 53 participants when six basic emotion states were induced. Two RSP parameters were acquired from a chest-band respiration sensor, and five HRV parameters were acquired from a finger-clip blood volume pulse (BVP) sensor. A newly designed deep-learning model based on a convolutional neural network (CNN) was adopted for detecting the identification accuracy of individual emotions. Additionally, the signal combination of the acquired parameters was proposed to obtain high classification accuracy. Furthermore, a dominant factor influencing the accuracy was found by comparing the relativeness of the parameters, providing a basis for supporting the results of emotion classification. The users of this proposed model will soon be able to improve the emotion recognition model further based on CNN using multimodal physiological signals and their sensors.

Human Emotion Recognition: Review of Sensors and Methods

Automated emotion recognition (AEE) is an important issue in various fields of activities which use human emotional reactions as a signal for marketing, technical equipment, or human-robot interaction. This paper analyzes scientific research and technical papers for sensor use analysis, among various methods implemented or researched. This paper covers a few classes of sensors, using contactless methods as well as contact and skin-penetrating electrodes for human emotion detection and the measurement of their intensity. The results of the analysis performed in this paper present applicable methods for each type of emotion and their intensity and propose their classification. The classification of emotion sensors is presented to reveal area of application and expected outcomes from each method, as well as their limitations. This paper should be relevant for researchers using human emotion evaluation and analysis, when there is a need to choose a proper method for their purposes or to find alternative decisions. Based on the analyzed human emotion recognition sensors and methods, we developed some practical applications for humanizing the Internet of Things (IoT) and affective computing systems.