Multimodal Emotion Evaluation: A Physiological Model for Cost-Effective Emotion Classification

Remote Emotion Recognition Using Continuous-Wave Bio-Radar System

The Bio-Radar is herein presented as a non-contact radar system able to capture vital signs remotely without requiring any physical contact with the subject. In this work, the ability to use the proposed system for emotion recognition is verified by comparing its performance on identifying fear, happiness and a neutral condition, with certified measuring equipment. For this purpose, machine learning algorithms were applied to the respiratory and cardiac signals captured simultaneously by the radar and the referenced contact-based system. Following a multiclass identification strategy, one could conclude that both systems present a comparable performance, where the radar might even outperform under specific conditions. Emotion recognition is possible using a radar system, with an accuracy equal to 99.7% and an F1-score of 99.9%. Thus, we demonstrated that it is perfectly possible to use the Bio-Radar system for this purpose, which is able to be operated remotely, avoiding the subject awareness of being monitored and thus providing more authentic reactions.

Understanding the role of emotion in decision making process: using machine learning to analyze physiological responses to visual, auditory, and combined stimulation

Emotions significantly shape decision-making, and targeted emotional elicitations represent an important factor in neuromarketing, where they impact advertising effectiveness by capturing potential customers' attention intricately associated with emotional triggers. Analyzing biometric parameters after stimulus exposure may help in understanding emotional states. This study investigates autonomic and central nervous system responses to emotional stimuli, including images, auditory cues, and their combination while recording physiological signals, namely the electrocardiogram, blood volume pulse, galvanic skin response, pupillometry, respiration, and the electroencephalogram. The primary goal of the proposed analysis is to compare emotional stimulation methods and to identify the most effective approach for distinct physiological patterns. A novel feature selection technique is applied to further optimize the separation of four emotional states. Basic machine learning approaches are used in order to discern emotions as elicited by different kinds of stimulation. Electroencephalographic signals, Galvanic skin response and cardio-respiratory coupling-derived features provided the most significant features in distinguishing the four emotional states. Further findings highlight how auditory stimuli play a crucial role in creating distinct physiological patterns that enhance classification within a four-class problem. When combining all three types of stimulation, a validation accuracy of 49% was achieved. The sound-only and the image-only phases resulted in 52% and 44% accuracy respectively, whereas the combined stimulation of images and sounds led to 51% accuracy. Isolated visual stimuli yield less distinct patterns, necessitating more signals for relatively inferior performance compared to other types of stimuli. This surprising significance arises from limited auditory exploration in emotional recognition literature, particularly contrasted with the pleathora of studies performed using visual stimulation. In marketing, auditory components might hold a more relevant potential to significantly influence consumer choices.

Automatic Classification of Emotions Based on Cardiac Signals: A Systematic Literature Review

Emotions play a pivotal role in human cognition, exerting influence across diverse domains of individuals' lives. The widespread adoption of artificial intelligence and machine learning has spurred interest in systems capable of automatically recognizing and classifying emotions and affective states. However, the accurate identification of human emotions remains a formidable challenge, as they are influenced by various factors and accompanied by physiological changes. Numerous solutions have emerged to enable emotion recognition, leveraging the characterization of biological signals, including the utilization of cardiac signals acquired from low-cost and wearable sensors. The objective of this work was to comprehensively investigate the current trends in the field by conducting a Systematic Literature Review (SLR) that focuses specifically on the detection, recognition, and classification of emotions based on cardiac signals, to gain insights into the prevailing techniques employed for signal acquisition, the extracted features, the elicitation process, and the classification methods employed in these studies. A SLR was conducted using four research databases, and articles were assessed concerning the proposed research questions. Twenty seven articles met the selection criteria and were assessed for the feasibility of using cardiac signals, acquired from low-cost and wearable devices, for emotion recognition. Several emotional elicitation methods were found in the literature, including the algorithms applied for automatic classification, as well as the key challenges associated with emotion recognition relying solely on cardiac signals. This study extends the current body of knowledge and enables future research by providing insights into suitable techniques for designing automatic emotion recognition applications. It emphasizes the importance of utilizing low-cost, wearable, and unobtrusive devices to acquire cardiac signals for accurate and accessible emotion recognition.

Multimodal Sentiment Analysis of Online Product Information Based on Text Mining Under the Influence of Social Media

Currently, with the dramatic increase in social media users and the greater variety of online product information, manual processing of this information is time-consuming and labour-intensive. Therefore, based on the text mining of online information, this paper analyzes the text representation method of online information, discusses the long short-term memory network, and constructs an interactive attention graph convolutional network (IAGCN) model based on graph convolutional neural network (GCNN) and attention mechanism to study the multimodal sentiment analysis (MSA) of online product information. The results show that the IAGCN model improves the accuracy by 4.78% and the F1 value by 29.25% compared with the pure interactive attention network. Meanwhile, it is found that the performance of the model is optimal when the GCNN is two layers and uses syntactic position attention. This research has important practical significance for MSA of online product information in social media.

Multidimensional assessment of anxiety through the State-Trait Inventory for Cognitive and Somatic Anxiety (STICSA): From dimensionality to response prediction across emotional contexts

The assessment of mal-adaptive anxiety is crucial, considering the associated personal, economic, and societal burden. The State-Trait Inventory for Cognitive and Somatic Anxiety (STICSA) is a self-report instrument developed to provide multidimensional anxiety assessment in four dimensions: trait-cognitive, trait-somatic, state-cognitive and state-somatic. This research aimed to extend STICSA’s psychometric studies through the assessment of its dimensionality, reliability, measurement invariance and nomological validity in the Portuguese population. Additionally, the predictive validity of STICSA-Trait was also evaluated, through the analysis of the relationship between self-reported trait anxiety and both the subjective and the psychophysiological response across distinct emotional situations. Similarly to previous studies, results supported both a four-factor and two separated bi-factor structures. Measurement invariance across sex groups was also supported, and good nomological validity was observed. Moreover, STICSA trait-cognitive dimension was associated with differences in self-reported arousal between groups of high/low anxiety, whereas STICSA trait-somatic dimension was related to differences in both the subjective and psychophysiological response. Together, these results support STICSA as a useful instrument for a broader anxiety assessment, crucial for an informed diagnosis and practice.

The Multimodal Sentiment Analysis of Online Product Marketing Information Using Text Mining and Big Data

Currently, the internet is increasingly popular. More people are used to sharing their feelings about various things on the internet. Online product marketing information is also growing. How to mine the required information from the massive information with the support of big data technology has become a big problem. Thereby, based on the text mining of online product marketing information, this work discusses the text preprocessing methods and the temporal convolution network (TCN) based on a convolutional neural network (CNN). Moreover, on this basis, multimodal attention mechanism (AM) and cross-modal transformer structure are added to build a TCN based on AM (AM-TCN) model to analyze the multimodal emotion of online product marketing information. The results show that the accuracy of the AM-TCN model is 2.88% higher than that of the TCN model alone, and F1 is 3.47% higher. Moreover, the accuracy rate of the AM-TCN is 1.22% higher than that of the next highest recurrent multistage fusion network, and the F1 value is 0.95% higher.

Neuroorganoleptics: Organoleptic Testing Based on Psychophysiological Sensing

There is an increasing interest, in consumer behaviour research related to food and beverage, in taking a step further from the traditional self-report questionnaires and organoleptic properties assessment. With the growing availability of psychophysiological data acquisition devices, and advancements in the study of the underlying signal sources seeking affective state assessment, the use of psychophysiological data analysis is a natural evolution in organoleptic testing. In this paper we propose a protocol for what can be defined as neuroorganoleptic analysis, a method that combines traditional approaches with psychophysiological data acquired during sensory testing. Our protocol was applied to a case study project named MobFood, where four samples of food were tested by a total of 83 participants, using preference and acceptance tasks, across three different sessions. Best practices and lessons learned regarding the laboratory setting and the acquisition of psychophysiological data were derived from this case study, which are herein described. Preliminary results show that certain Heart Rate Variability (HRV) features have a strong correlation with the preferences self-reported by the participants.

Affective State during Physiotherapy and Its Analysis Using Machine Learning Methods

Invasive or uncomfortable procedures especially during healthcare trigger emotions. Technological development of the equipment and systems for monitoring and recording psychophysiological functions enables continuous observation of changes to a situation responding to a situation. The presented study aimed to focus on the analysis of the individual’s affective state. The results reflect the excitation expressed by the subjects’ statements collected with psychological questionnaires. The research group consisted of 49 participants (22 women and 25 men). The measurement protocol included acquiring the electrodermal activity signal, cardiac signals, and accelerometric signals in three axes. Subjective measurements were acquired for affective state using the JAWS questionnaires, for cognitive skills the DST, and for verbal fluency the VFT. The physiological and psychological data were subjected to statistical analysis and then to a machine learning process using different features selection methods (JMI or PCA). The highest accuracy of the kNN classifier was achieved in combination with the JMI method (81.63%) concerning the division complying with the JAWS test results. The classification sensitivity and specificity were 85.71% and 71.43%.

The Concept of Advanced Multi-Sensor Monitoring of Human Stress

Many people live under stressful conditions which has an adverse effect on their health. Human stress, especially long-term one, can lead to a serious illness. Therefore, monitoring of human stress influence can be very useful. We can monitor stress in strictly controlled laboratory conditions, but it is time-consuming and does not capture reactions, on everyday stressors or in natural environment using wearable sensors, but with limited accuracy. Therefore, we began to analyze the current state of promising wearable stress-meters and the latest advances in the record of related physiological variables. Based on these results, we present the concept of an accurate, reliable and easier to use telemedicine device for long-term monitoring of people in a real life. In our concept, we ratify with two synchronized devices, one on the finger and the second on the chest. The results will be obtained from several physiological variables including electrodermal activity, heart rate and respiration, body temperature, blood pressure and others. All these variables will be measured using a coherent multi-sensors device. Our goal is to show possibilities and trends towards the production of new telemedicine equipment and thus, opening the door to a widespread application of human stress-meters.

Hybrid System of Emotion Evaluation in Physiotherapeutic Procedures

Nowadays, the dynamic development of technology allows for the design of systems based on various information sources and their integration into hybrid expert systems. One of the areas of research where such systems are especially helpful is emotion analysis. The sympathetic nervous system controls emotions, while its function is directly reflected by the electrodermal activity (EDA) signal. The presented study aimed to develop a tool and propose a physiological data set to complement the psychological data. The study group consisted of 41 students aged from 19 to 26 years. The presented research protocol was based on the acquisition of the electrodermal activity signal using the Empatica E4 device during three exercises performed in a prototype Disc4Spine system and using the psychological research methods. Different methods (hierarchical and non-hierarchical) of subsequent data clustering and optimisation in the context of emotions experienced were analysed. The best results were obtained for the k-means classifier during Exercise 3 (80.49%) and for the combination of the EDA signal with negative emotions (80.48%). A comparison of accuracy of the k-means classification with the independent division made by a psychologist revealed again the best results for negative emotions (78.05%).