Electrocardiogram-Based Emotion Recognition Systems and Their Applications in Healthcare-A Review

Affective computing is a field of study that integrates human affects and emotions with artificial intelligence into systems or devices. A system or device with affective computing is beneficial for the mental health and wellbeing of individuals that are stressed, anguished, or depressed. Emotion recognition systems are an important technology that enables affective computing. Currently, there are a lot of ways to build an emotion recognition system using various techniques and algorithms. This review paper focuses on emotion recognition research that adopted electrocardiograms (ECGs) as a unimodal approach as well as part of a multimodal approach for emotion recognition systems. Critical observations of data collection, pre-processing, feature extraction, feature selection and dimensionality reduction, classification, and validation are conducted. This paper also highlights the architectures with accuracy of above 90%. The available ECG-inclusive affective databases are also reviewed, and a popularity analysis is presented. Additionally, the benefit of emotion recognition systems towards healthcare systems is also reviewed here. Based on the literature reviewed, a thorough discussion on the subject matter and future works is suggested and concluded. The findings presented here are beneficial for prospective researchers to look into the summary of previous works conducted in the field of ECG-based emotion recognition systems, and for identifying gaps in the area, as well as in developing and designing future applications of emotion recognition systems, especially in improving healthcare.

An Ensemble Learning Approach for Electrocardiogram Sensor Based Human Emotion Recognition

Recently, researchers in the area of biosensor based human emotion recognition have used different types of machine learning models for recognizing human emotions. However, most of them still lack the ability to recognize human emotions with higher classification accuracy incorporating a limited number of bio-sensors. In the domain of machine learning, ensemble learning methods have been successfully applied to solve different types of real-world machine learning problems which require improved classification accuracies. Emphasising on that, this research suggests an ensemble learning approach for developing a machine learning model that can recognize four major human emotions namely: anger; sadness; joy; and pleasure incorporating electrocardiogram (ECG) signals. As feature extraction methods, this analysis combines four ECG signal based techniques, namely: heart rate variability; empirical mode decomposition; with-in beat analysis; and frequency spectrum analysis. The first three feature extraction methods are well-known ECG based feature extraction techniques mentioned in the literature, and the fourth technique is a novel method proposed in this study. The machine learning procedure of this investigation evaluates the performance of a set of well-known ensemble learners for emotion classification and further improves the classification results using feature selection as a prior step to ensemble model training. Compared to the best performing single biosensor based model in the literature, the developed ensemble learner has the accuracy gain of 10.77%. Furthermore, the developed model outperforms most of the multiple biosensor based emotion recognition models with a significantly higher classification accuracy gain.

Frequency Band Analysis of Electrocardiogram (ECG) Signals for Human Emotional State Classification Using Discrete Wavelet Transform (DWT)

[Purpose] Intelligent emotion assessment systems have been highly successful in a variety of applications, such as e-learning, psychology, and psycho-physiology. This study aimed to assess five different human emotions (happiness, disgust, fear, sadness, and neutral) using heart rate variability (HRV) signals derived from an electrocardiogram (ECG). [Subjects] Twenty healthy university students (10 males and 10 females) with a mean age of 23 years participated in this experiment. [Methods] All five emotions were induced by audio-visual stimuli (video clips). ECG signals were acquired using 3 electrodes and were preprocessed using a Butterworth 3rd order filter to remove noise and baseline wander. The Pan-Tompkins algorithm was used to derive the HRV signals from ECG. Discrete wavelet transform (DWT) was used to extract statistical features from the HRV signals using four wavelet functions: Daubechies6 (db6), Daubechies7 (db7), Symmlet8 (sym8), and Coiflet5 (coif5). The k-nearest neighbor (KNN) and linear discriminant analysis (LDA) were used to map the statistical features into corresponding emotions. [Results] KNN provided the maximum average emotion classification rate compared to LDA for five emotions (sadness − 50.28%; happiness − 79.03%; fear − 77.78%; disgust − 88.69%; and neutral − 78.34%). [Conclusion] The results of this study indicate that HRV may be a reliable indicator of changes in the emotional state of subjects and provides an approach to the development of a real-time emotion assessment system with a higher reliability than other systems.

PHYSIOLOGICAL SIGNAL ANALYSIS FOR COGNITIVE STATE ESTIMATION

The purpose of this paper is to identify inconsistency in human physiological signals based on cognitive states by measuring and analyzing bio-signals. In this paper, the cognitive states are estimated using physiological signal analysis. The parameters are electrocardiogram (ECG), electromyogram (EMG), electroencephalogram (EEG) and blood pressure (BP). The signals have been collected using BIOPAC system in which the subjects were induced to undergo the specific sequence of the cognitive state. For getting physiological signals during different conditions, we utilized power point slide show, video clips and question answer method which elicits mental reactions from the subjects. Data is taken before and after four tasks that encompassed the motor action (MA), thought (TH), memory related (MR) and emotion (EM). These measured values are analyzed using BIOPAC Acknowledge software. It was found that the motor action and thought states have effects on BP while MR and EM state mainly affect the ECG measurement. The decibel value and frequency found for EM state in ECG are minimum compared to relaxed state (RS) condition. Similarly, the maximum frequency and dB value is found for MR state. No significant variation was seen for MA and TH states. Thus it was decided that the MR and EM states mainly affect the ECG measurement. For BP the value increases in MA state and decreases in TH state. The MA state mainly affects the EMG signal while other states have no significant changes. The EEG mainly detects the signal of task performed by the specific brain region where the electrodes are placed. In EEG analysis, the electrodes are placed in occipital lobe region which gives mainly the variation in alpha amplitude of EEG with eyes closed and eyes opened. Alpha wave amplitudes vary with the subjects attention to mental tasks performed with eyes closed.