Analysis and Use of the Emotional Context with Wearable Devices for Games and Intelligent Assistants

Investigating Receptivity and Affect Using Machine Learning: Ecological Momentary Assessment and Wearable Sensing Study

BACKGROUND

As mobile health (mHealth) studies become increasingly productive owing to the advancements in wearable and mobile sensor technology, our ability to monitor and model human behavior will be constrained by participant receptivity. Many health constructs are dependent on subjective responses, and without such responses, researchers are left with little to no ground truth to accompany our ever-growing biobehavioral data. This issue can significantly impact the quality of a study, particularly for populations known to exhibit lower compliance rates. To address this challenge, researchers have proposed innovative approaches that use machine learning (ML) and sensor data to modify the timing and delivery of surveys. However, an overarching concern is the potential introduction of biases or unintended influences on participants' responses when implementing new survey delivery methods.

OBJECTIVE

This study aims to demonstrate the potential impact of an ML-based ecological momentary assessment (EMA) delivery system (using receptivity as the predictor variable) on the participants' reported emotional state. We examine the factors that affect participants' receptivity to EMAs in a 10-day wearable and EMA-based emotional state-sensing mHealth study. We study the physiological relationships indicative of receptivity and affect while also analyzing the interaction between the 2 constructs.

METHODS

We collected data from 45 healthy participants wearing 2 devices measuring electrodermal activity, accelerometer, electrocardiography, and skin temperature while answering 10 EMAs daily, containing questions about perceived mood. Owing to the nature of our constructs, we can only obtain ground truth measures for both affect and receptivity during responses. Therefore, we used unsupervised and supervised ML methods to infer affect when a participant did not respond. Our unsupervised method used k-means clustering to determine the relationship between physiology and receptivity and then inferred the emotional state during nonresponses. For the supervised learning method, we primarily used random forest and neural networks to predict the affect of unlabeled data points as well as receptivity.

RESULTS

Our findings showed that using a receptivity model to trigger EMAs decreased the reported negative affect by >3 points or 0.29 SDs in our self-reported affect measure, scored between 13 and 91. The findings also showed a bimodal distribution of our predicted affect during nonresponses. This indicates that this system initiates EMAs more commonly during states of higher positive emotions.

CONCLUSIONS

Our results showed a clear relationship between affect and receptivity. This relationship can affect the efficacy of an mHealth study, particularly those that use an ML algorithm to trigger EMAs. Therefore, we propose that future work should focus on a smart trigger that promotes EMA receptivity without influencing affect during sampled time points.

Does Emotional State Improve Following Temporomandibular Joint Total Joint Replacement?

BACKGROUND

Temporomandibular joint total joint replacement (TMJTJR) offers patients the opportunity for improved function and reduced pain. TMJTJR also has the potential to affect a patient's emotions in a positive or negative manner.

PURPOSE

The purpose of this study was to evaluate changes in emotional state for subjects undergoing TMJTJR.

STUDY DESIGN, SETTING, SAMPLE

The authors implemented a retrospective cohort study. Subjects who received TMJTJR were identified from the TMJ Inter Network, which is a study group comprising more than 130 temporomandibular joint surgeons. Subjects between the ages of 18 and 65 years with complete medical records and pre/post TMJTJR video/audio recordings were enrolled in the study.

PREDICTOR VARIABLE

The predictor variable was time (preoperative and postoperative).

MAIN OUTCOME VARIABLES

The primary outcome variable is change in the emotional state. All subjects had preoperative (T0) recorded interview as well as a postoperative (T1) interview at 3 to 6 months. The eight-category emotional state was classified as neutral, happy, sad, angry, fearful, disgusted, surprised, and bored. The three-category emotional state was classified as neutral, positive, and negative. The emotional state was measured using artificial intelligence at T0 and T1. The secondary outcome variable was pain score and maximal interincisal opening.

COVARIATES

The covariates are gender, age, diagnosis, prosthetic side, TMJTJR design, and TMJTJR type.

ANALYSES

The relationship between emotional state change and covariates was examined using both the χ2 test and the Kruskal-Wallis H test. The significance of the change in categorical data after surgery was examined using the McNemar-Bowker test. P values < .05 were considered statistically significant.

RESULTS

Thirty-three subjects were included in the study. The mean age was 30.09 ± 8.69 with 15 males (45%) and 18 females (55%). The percentage of subjects with preoperative neutral, happy, sad, angry, and fearful emotional states was 24, 15, 24, 9, and 27%, respectively. The percentage of subjects with postoperative neutral, happy, sad, angry, and fearful emotional states was 21, 39, 21, 12, and 6%, respectively. The change in emotional state was statistically significant (P = .037). There was no statistically significant relationship between covariates and emotional state changes (P > .05).

CONCLUSION

According to the assessment of artificial intelligence, TMJTJR improves the emotional state of patients.

Continued usage of smart wearable devices (SWDs): cross-level analysis of gamification and network externality

With the increasing maturity of mobile networks and big data technology, smart wearable devices (SWDs) are regarded as a new technology trend following smartphones. Especially during the COVID-19 pandemic, the increase in telework and the growing interest in self-health monitoring have greatly promoted the market growth of SWDs. This study aimed to investigate the factors affecting the continued use of SWDs. A cross-level analysis model that integrates technical characteristics, gamification theory, perceived value theory, and network externality was constructed. A hierarchical linear model was employed to evaluate the data and test it against the hypotheses. The empirical results showed that, at the individual level, gamification enhances users' value perceptions. Users pay more attention to rewards in gamification than to competition. Rewards were also found to effectively promote the users' value perception and increase the intention to continue using the device. At the group level, the effect of network externality significantly influences the intention to continue using SWDs. Moreover, SWDs are associated with the phenomenon by which consumers conspicuously display and highlight their own characteristics, and this attribute is also a crucial factor enticing consumers to continue using SWDs. Developers should therefore establish clear product positioning and strengthen interactivity as early as possible to build a loyal customer base.

BIRAFFE2, a multimodal dataset for emotion-based personalization in rich affective game environments

Generic emotion prediction models based on physiological data developed in the field of affective computing apparently are not robust enough. To improve their effectiveness, one needs to personalize them to specific individuals and incorporate broader contextual information. To address the lack of relevant datasets, we propose the 2nd Study in Bio-Reactions and Faces for Emotion-based Personalization for AI Systems (BIRAFFE2) dataset. In addition to the classical procedure in the stimulus-appraisal paradigm, it also contains data from an affective gaming session in which a range of contextual data was collected from the game environment. This is complemented by accelerometer, ECG and EDA signals, participants’ facial expression data, together with personality and game engagement questionnaires. The dataset was collected on 102 participants. Its potential usefulness is presented by validating the correctness of the contextual data and indicating the relationships between personality and participants’ emotions and between personality and physiological signals.

Measurement(s)	personality trait • gaming experience • game experience • Electrocardiography • Galvanic Skin Response • Accelerometer • Gyroscope • In-Game Activity • Facial Expression • Emotion • Screen recording • Age
Technology Type(s)	personality trait measurement • own questionnaire • game experience questionnaire • Electrocardiography • Galvanic Skin Response • Gamepad • Own Logging Mechanism • Camera Device • Microsoft Face API • Own Widget • Screencast tool
Sample Characteristic - Organism	Homo sapiens
Sample Characteristic - Environment	Game environment
Sample Characteristic - Location	Krakow

Emognition dataset: emotion recognition with self-reports, facial expressions, and physiology using wearables

The Emognition dataset is dedicated to testing methods for emotion recognition (ER) from physiological responses and facial expressions. We collected data from 43 participants who watched short film clips eliciting nine discrete emotions: amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, and sadness. Three wearables were used to record physiological data: EEG, BVP (2x), HR, EDA, SKT, ACC (3x), and GYRO (2x); in parallel with the upper-body videos. After each film clip, participants completed two types of self-reports: (1) related to nine discrete emotions and (2) three affective dimensions: valence, arousal, and motivation. The obtained data facilitates various ER approaches, e.g., multimodal ER, EEG- vs. cardiovascular-based ER, discrete to dimensional representation transitions. The technical validation indicated that watching film clips elicited the targeted emotions. It also supported signals’ high quality.

Measurement(s)	cardiac output measurement • Electroencephalography • Galvanic Skin Response • Temperature • acceleration • facial expressions
Technology Type(s)	photoplethysmogram • electroencephalogram (5 electrodes) • electrodermal activity measurement • Sensor • Accelerometer • Video Recording
Sample Characteristic - Organism	Homo sapiens
Sample Characteristic - Environment	laboratory environment

Datasets for Automated Affect and Emotion Recognition from Cardiovascular Signals Using Artificial Intelligence- A Systematic Review

Our review aimed to assess the current state and quality of publicly available datasets used for automated affect and emotion recognition (AAER) with artificial intelligence (AI), and emphasising cardiovascular (CV) signals. The quality of such datasets is essential to create replicable systems for future work to grow. We investigated nine sources up to 31 August 2020, using a developed search strategy, including studies considering the use of AI in AAER based on CV signals. Two independent reviewers performed the screening of identified records, full-text assessment, data extraction, and credibility. All discrepancies were resolved by discussion. We descriptively synthesised the results and assessed their credibility. The protocol was registered on the Open Science Framework (OSF) platform. Eighteen records out of 195 were selected from 4649 records, focusing on datasets containing CV signals for AAER. Included papers analysed and shared data of 812 participants aged 17 to 47. Electrocardiography was the most explored signal (83.33% of datasets). Authors utilised video stimulation most frequently (52.38% of experiments). Despite these results, much information was not reported by researchers. The quality of the analysed papers was mainly low. Researchers in the field should concentrate more on methodology.

Bringing Emotion Recognition Out of the Lab into Real Life: Recent Advances in Sensors and Machine Learning

Bringing emotion recognition (ER) out of the controlled laboratory setup into everyday life can enable applications targeted at a broader population, e.g., helping people with psychological disorders, assisting kids with autism, monitoring the elderly, and general improvement of well-being. This work reviews progress in sensors and machine learning methods and techniques that have made it possible to move ER from the lab to the field in recent years. In particular, the commercially available sensors collecting physiological data, signal processing techniques, and deep learning architectures used to predict emotions are discussed. A survey on existing systems for recognizing emotions in real-life scenarios—their possibilities, limitations, and identified problems—is also provided. The review is concluded with a debate on what challenges need to be overcome in the domain in the near future.

Tourist Experiences Recommender System Based on Emotion Recognition with Wearable Data

The collection of physiological data from people has been facilitated due to the mass use of cheap wearable devices. Although the accuracy is low compared to specialized healthcare devices, these can be widely applied in other contexts. This study proposes the architecture for a tourist experiences recommender system (TERS) based on the user's emotional states who wear these devices. The issue lies in detecting emotion from Heart Rate (HR) measurements obtained from these wearables. Unlike most state-of-the-art studies, which have elicited emotions in controlled experiments and with high-accuracy sensors, this research's challenge consisted of emotion recognition (ER) in the daily life context of users based on the gathering of HR data. Furthermore, an objective was to generate the tourist recommendation considering the emotional state of the device wearer. The method used comprises three main phases: The first was the collection of HR measurements and labeling emotions through mobile applications. The second was emotional detection using deep learning algorithms. The final phase was the design and validation of the TERS-ER. In this way, a dataset of HR measurements labeled with emotions was obtained as results. Among the different algorithms tested for ER, the hybrid model of Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks had promising results. Moreover, concerning TERS, Collaborative Filtering (CF) using CNN showed better performance.

Resource Allocation Schemes for 5G Network: A Systematic Review

Fifth-generation (5G) communication technology is intended to offer higher data rates, outstanding user exposure, lower power consumption, and extremely short latency. Such cellular networks will implement a diverse multi-layer model comprising device-to-device networks, macro-cells, and different categories of small cells to assist customers with desired quality-of-service (QoS). This multi-layer model affects several studies that confront utilizing interference management and resource allocation in 5G networks. With the growing need for cellular service and the limited resources to provide it, capably handling network traffic and operation has become a problem of resource distribution. One of the utmost serious problems is to alleviate the jamming in the network in support of having a better QoS. However, although a limited number of review papers have been written on resource distribution, no review papers have been written specifically on 5G resource allocation. Hence, this article analyzes the issue of resource allocation by classifying the various resource allocation schemes in 5G that have been reported in the literature and assessing their ability to enhance service quality. This survey bases its discussion on the metrics that are used to evaluate network performance. After consideration of the current evidence on resource allocation methods in 5G, the review hopes to empower scholars by suggesting future research areas on which to focus.

Personality-Based Affective Adaptation Methods for Intelligent Systems

In this article, we propose using personality assessment as a way to adapt affective intelligent systems. This psychologically-grounded mechanism will divide users into groups that differ in their reactions to affective stimuli for which the behaviour of the system can be adjusted. In order to verify the hypotheses, we conducted an experiment on 206 people, which consisted of two proof-of-concept demonstrations: a “classical” stimuli presentation part, and affective games that provide a rich and controllable environment for complex emotional stimuli. Several significant links between personality traits and the psychophysiological signals (electrocardiogram (ECG), galvanic skin response (GSR)), which were gathered while using the BITalino (r)evolution kit platform, as well as between personality traits and reactions to complex stimulus environment, are promising results that indicate the potential of the proposed adaptation mechanism.

Can We Ditch Feature Engineering? End-to-End Deep Learning for Affect Recognition from Physiological Sensor Data

To further extend the applicability of wearable sensors in various domains such as mobile health systems and the automotive industry, new methods for accurately extracting subtle physiological information from these wearable sensors are required. However, the extraction of valuable information from physiological signals is still challenging—smartphones can count steps and compute heart rate, but they cannot recognize emotions and related affective states. This study analyzes the possibility of using end-to-end multimodal deep learning (DL) methods for affect recognition. Ten end-to-end DL architectures are compared on four different datasets with diverse raw physiological signals used for affect recognition, including emotional and stress states. The DL architectures specialized for time-series classification were enhanced to simultaneously facilitate learning from multiple sensors, each having their own sampling frequency. To enable fair comparison among the different DL architectures, Bayesian optimization was used for hyperparameter tuning. The experimental results showed that the performance of the models depends on the intensity of the physiological response induced by the affective stimuli, i.e., the DL models recognize stress induced by the Trier Social Stress Test more successfully than they recognize emotional changes induced by watching affective content, e.g., funny videos. Additionally, the results showed that the CNN-based architectures might be more suitable than LSTM-based architectures for affect recognition from physiological sensors.

ADAPTS: An Intelligent Sustainable Conceptual Framework for Engineering Projects

This paper presents a conceptual framework for the optimization of environmental sustainability in engineering projects, both for products and industrial facilities or processes. The main objective of this work is to propose a conceptual framework to help researchers to approach optimization under the criteria of sustainability of engineering projects, making use of current Machine Learning techniques. For the development of this conceptual framework, a bibliographic search has been carried out on the Web of Science. From the selected documents and through a hermeneutic procedure the texts have been analyzed and the conceptual framework has been carried out. A graphic representation pyramid shape is shown to clearly define the variables of the proposed conceptual framework and their relationships. The conceptual framework consists of 5 dimensions; its acronym is ADAPTS. In the base are: (1) the Application to which it is intended, (2) the available DAta, (3) the APproach under which it is operated, and (4) the machine learning Tool used. At the top of the pyramid, (5) the necessary Sensing. A study case is proposed to show its applicability. This work is part of a broader line of research, in terms of optimization under sustainability criteria.