Reliability of crowdsourcing as a method for collecting emotions labels on pictures

Association between perception of harm and valence of shoulder movement images with disability levels related to chronic shoulder pain

ABSTRACT

Recent studies highlight an interplay between pain perception and emotional responses. This necessitates a thorough investigation into how beliefs and motivational influences respond to visual stimuli of movements. Such an analysis is crucial for understanding the extent to which these factors contribute to disability levels associated with shoulder pain. We aimed to investigate the relationship between the perception of harm and the valence in images depicting shoulder movements and determine how these perceptions are linked to disability levels associated with shoulder pain. This cross-sectional study recruited 42 individuals with chronic shoulder pain. Participants were presented with 58 shoulder movements images. Each participant evaluated these images for emotional valence and arousal using the self-assessment manikin. For every image, they provided their level of avoidance, fear, and perception of harm in a numerical scale. We measured disability levels and pain catastrophizing using the Shoulder Pain and Disability Index and the Pain Catastrophizing Scale. A direct acyclic graph was used. Multiple linear regression analysis was conducted with shoulder disability as the dependent variable and perception of harm and valence as independent variables, adjusted for the confounders catastrophizing and arousal. This analysis resulted in a significant model (F4,37 = 11.44; adjusted R2 = 0.547; P < 0.01). The perception of harm to shoulder movement (β = 0.11; P < 0.001; 95% confidence interval = 5.6-11.8) was significantly associated with the level of shoulder disability, whereas valence did not show a significant association (β = 0.26; P = 0.15; 95% confidence interval = 1.7-10.8). The perception of harm associated with shoulder movements images during daily activities was associated with disability. Individuals who believe that shoulder movements are harmful have greater disability.

As Strong as We Are United: Effects of Intrapersonal and Interpersonal Emotion Regulation on Quality of Life in Women With Breast Cancer

Women diagnosed with breast cancer often experience unpleasant emotions, resulting in higher levels of emotional burden and decreased levels of wellbeing and quality of life. The present correlational and cross-sectional study aims to compare the implementation of two regulatory levels, intrapersonal and interpersonal (as social sharing of emotions), and two types of strategies, antecedent-focused and response-focused, and explore their impact on breast cancer patients’ perception of quality of life. Sixty-eight women previously diagnosed with the disease participated in this study, with a mean age of 63years old (SD=11.58). Data were collected through a self-report questionnaire to assess emotional experience, intrapersonal regulation, social sharing of emotions, and breast cancer-related wellbeing and quality of life. Data yielded that most of the participants regulated their negative emotions within social interactions and made more use of antecedent-focused strategies to cognitively reformulate the emotional episode. Social and family wellbeing were positively associated with antecedent-focused strategies, as well as intrapersonal and interpersonal regulatory levels. Moreover, the occurrence of sharing episodes and social interactions played an important and beneficial role on patients’ perceived quality of life. These findings reinforce the importance of promoting an adaptive intrapersonal regulation among breast cancer patients. Results also suggest that social sharing of emotions is an efficient process to help them to better cope with the psychological and emotional burden of the disease, thus positively influencing the way they perceive their social and family wellbeing, as well as their quality of life.

Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels

Background/Introduction

Emotion detection classifiers traditionally predict discrete emotions. However, emotion expressions are often subjective, thus requiring a method to handle compound and ambiguous labels. We explore the feasibility of using crowdsourcing to acquire reliable soft-target labels and evaluate an emotion detection classifier trained with these labels. We hypothesize that training with labels that are representative of the diversity of human interpretation of an image will result in predictions that are similarly representative on a disjoint test set. We also hypothesize that crowdsourcing can generate distributions which mirror those generated in a lab setting.

Methods

We center our study on the Child Affective Facial Expression (CAFE) dataset, a gold standard collection of images depicting pediatric facial expressions along with 100 human labels per image. To test the feasibility of crowdsourcing to generate these labels, we used Microworkers to acquire labels for 207 CAFE images. We evaluate both unfiltered workers as well as workers selected through a short crowd filtration process. We then train two versions of a ResNet-152 neural network on soft-target CAFE labels using the original 100 annotations provided with the dataset: (1) a classifier trained with traditional one-hot encoded labels, and (2) a classifier trained with vector labels representing the distribution of CAFE annotator responses. We compare the resulting softmax output distributions of the two classifiers with a 2-sample independent t-test of L1 distances between the classifier's output probability distribution and the distribution of human labels.

Results

While agreement with CAFE is weak for unfiltered crowd workers, the filtered crowd agree with the CAFE labels 100% of the time for happy, neutral, sad and "fear + surprise", and 88.8% for "anger + disgust". While the F1-score for a one-hot encoded classifier is much higher (94.33% vs. 78.68%) with respect to the ground truth CAFE labels, the output probability vector of the crowd-trained classifier more closely resembles the distribution of human labels (t=3.2827, p=0.0014).

Conclusions

For many applications of affective computing, reporting an emotion probability distribution that accounts for the subjectivity of human interpretation can be more useful than an absolute label. Crowdsourcing, including a sufficient filtering mechanism for selecting reliable crowd workers, is a feasible solution for acquiring soft-target labels.

K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

Recognizing emotions during social interactions has many potential applications with the popularization of low-cost mobile sensors, but a challenge remains with the lack of naturalistic affective interaction data. Most existing emotion datasets do not support studying idiosyncratic emotions arising in the wild as they were collected in constrained environments. Therefore, studying emotions in the context of social interactions requires a novel dataset, and K-EmoCon is such a multimodal dataset with comprehensive annotations of continuous emotions during naturalistic conversations. The dataset contains multimodal measurements, including audiovisual recordings, EEG, and peripheral physiological signals, acquired with off-the-shelf devices from 16 sessions of approximately 10-minute long paired debates on a social issue. Distinct from previous datasets, it includes emotion annotations from all three available perspectives: self, debate partner, and external observers. Raters annotated emotional displays at intervals of every 5 seconds while viewing the debate footage, in terms of arousal-valence and 18 additional categorical emotions. The resulting K-EmoCon is the first publicly available emotion dataset accommodating the multiperspective assessment of emotions during social interactions.