EmojiGrid: A 2D pictorial scale for cross-cultural emotion assessment of negatively and positively valenced food

How high-intensity sensory consumption fills up resource scarcity: The boundary condition of self-acceptance

OBJECTIVE

Everyone in life will experience resource scarcity, which causes self-discrepancy. It is widely known that individuals participate in reactive consumption to solve the problems of self-discrepancy and resources scarcity. This kind of consumption may be symbolically related to the essence of the resource scarcity or may occur in an unrelated domain. This study proposes a theory for "filling up" one's resource scarcity through high-intensity sensory consumption (HISC).

METHODS

We used different methods, including one-way analysis of variance (ANOVA), linear regression, mediating effect, and moderating effect, to test the four hypotheses. Four experiments in the study were conducted from May 2022 and August 2022 and involved undergraduates from a university and volunteers recruited online. All participants are adults and verbally agree to participate voluntarily. Study 1a (N = 96 (male 47, female 49), participants from a business school in China) measured resource scarcity in the laboratory experiments and verified the effect of resource scarcity on consumer HISC preference by using linear regression (H1). Study 1b (N = 191 (male 98, female 93), students and teachers from a university in China) measured resource scarcity in the laboratory experiments and manipulated positively and negatively valenced experiences. Using the PROCESS SPSS Mode l, we verified that negatively valenced stimuli also lead to higher levels of arousal, which in turn restores the self-discrepancy caused by resource scarcity (H2). Study 2 (an online experiment, N = 182 (male 91, female 91), participants from China) manipulated the resource scarcity in a color sensory stimulant context, replicating the preliminary effect and examined the mediating effect of the self-worth by using the PROCESS SPSS Mode 4 (H3). Study 3 (an online experiment, N = 251 (male 125, female 126), participants from China) manipulated resource scarcity and self-acceptance in the tactile sensory experience, and tested the moderating effect of self-acceptance by using the PROCESS SPSS Mode 8 (H4).

RESULTS

Four studies suggest that not only do individuals facing resources scarcity prefer HISC but also that this consumption is mediated and moderated by self-worth and self-acceptance, respectively. This preference for HISC is negated when individuals have high self-acceptance traits. The findings are tested in the auditory domain (as evidenced by a propensity for louder volume), the visual domain (as evidenced by a propensity for more intense colors), and the tactile domain (as evidenced by a propensity for more intense need for touch). The findings also demonstrate that individual preferences for HISC is shown to operate regardless of the valence (positive valence vs. negative valence) of the sensory consumption.

CONCLUSIONS

Across four experiments, we find that individuals who are subjected to resource scarcity show a preference for high-intensity sensory consumption in the auditory, visual, and tactile domains. We also find that both negatively and positively valenced sensory stimuli have the same impact on resource-scarce individuals' preference for HISC. Furthermore, we demonstrate that the sense of self-worth significantly mediates the effect of resource scarcity on HISC. Finally, we reveal that self-acceptance moderates the effect of resource scarcity on HISC preference.

Linking Categorical and Dimensional Approaches to Assess Food-Related Emotions

Reflecting the two main prevailing and opposing views on the nature of emotions, emotional responses to food and beverages are typically measured using either (a) a categorical (lexicon-based) approach where users select or rate the terms that best express their food-related feelings or (b) a dimensional approach where they rate perceived food items along the dimensions of valence and arousal. Relating these two approaches is problematic since a response in terms of valence and arousal is not easily expressed in terms of emotions (like happy or disgusted). In this study, we linked the dimensional approach to a categorical approach by establishing mapping between a set of 25 emotion terms (EsSense25) and the valence–arousal space (via the EmojiGrid graphical response tool), using a set of 20 food images. In two ‘matching’ tasks, the participants first imagined how the food shown in a given image would make them feel and then reported either the emotional terms or the combination of valence and arousal that best described their feelings. In two labeling tasks, the participants first imagined experiencing a given emotion term and then they selected either the foods (images) that appeared capable to elicit that feeling or reported the combination of valence and arousal that best reflected that feeling. By combining (1) the mapping between the emotion terms and the food images with (2) the mapping of the food images to the valence–arousal space, we established (3) an indirect (via the images) mapping of the emotion terms to the valence–arousal space. The results show that the mapping between terms and images was reliable and that the linkages have straightforward and meaningful interpretations. The valence and arousal values that were assigned to the emotion terms through indirect mapping to the valence–arousal space were typically less extreme than those that were assigned through direct mapping.

Serial Dependence of Emotion Within and Between Stimulus Sensory Modalities

How we perceive the world is not solely determined by what we sense at a given moment in time, but also by what we processed recently. Here we investigated whether such serial dependencies for emotional stimuli transfer from one modality to another. Participants were presented a random sequence of emotional sounds and images and instructed to rate the valence and arousal of each stimulus (Experiment 1). For both ratings, we conducted an intertrial analysis, based on whether the rating on the previous trial was low or high. We found a positive serial dependence for valence and arousal regardless of the stimulus modality on two consecutive trials. In Experiment 2, we examined whether passively perceiving a stimulus is sufficient to induce a serial dependence. In Experiment 2, participants were instructed to rate the stimuli only on active trials and not on passive trials. The participants were informed that the active and passive trials were presented in alternating order, so that they were able to prepare for the task. We conducted an intertrial analysis on active trials, based on whether the rating on the previous passive trial (determined in Experiment 1) was low or high. For both ratings, we again observed positive serial dependencies regardless of the stimulus modality. We conclude that the emotional experience triggered by one stimulus affects the emotional experience for a subsequent stimulus regardless of their sensory modalities, that this occurs in a bottom-up fashion, and that this can be explained by residual activation in the emotional network in the brain.

Measuring Implicit Approach-Avoidance Tendencies towards Food Using a Mobile Phone outside the Lab

Implicit ('unconscious') approach-avoidance tendencies towards stimuli can be measured using the Approach Avoidance Task (AAT). We recently expanded a toolbox for analyzing the raw data of a novel, mobile version of the AAT (mAAT), that asks participants to move their phone towards their face (pull) or away (push) in response to images presented on the phone. We here tested the mAAT reaction time and the mAAT distance in a study with 71 Dutch participants that were recruited online and performed an experiment without coming to the laboratory. The participants used both the mAAT and (explicit) rating scales to respond to photographic images of food. As hypothesized, the rated wanting, rated valence and mAAT reaction time indicated a preference for palatable over unpalatable food, and for Dutch over Asian food. Additionally, as expected, arousal was rated higher for unpalatable than for palatable food, and higher for Dutch than for Asian food. The mAAT distance indicated that the unpalatable food images were moved across larger distances, regardless of the movement direction (pull or push), compared to the palatable food images; and the Dutch food images were moved across larger distances than the Asian food images. We conclude that the mAAT can be used to implicitly probe approach-avoidance motivation for complex images in the food domain. The new measure of mAAT distance may be used as an implicit measure of arousal. The ratings and the mAAT measures do not reflect the exact same information and may complement each other. Implicit measures, such as mAAT variables, are particularly valuable when response biases that can occur when using explicit ratings are expected.

Reading Food Experiences from the Face: Effects of Familiarity and Branding of Soy Sauce on Facial Expressions and Video-Based RPPG Heart Rate

Food experiences are not only driven by the food's intrinsic properties, such as its taste, texture, and aroma, but also by extrinsic properties such as visual brand information and the consumers' previous experiences with the foods. Recent developments in automated facial expression analysis and heart rate detection based on skin color changes (remote photoplethysmography or RPPG) allow for the monitoring of food experiences based on video images of the face. RPPG offers the possibility of large-scale non-laboratory and web-based testing of food products. In this study, results from the video-based analysis were compared to the more conventional tests (scores of valence and arousal using Emojis and photoplethysmography heart rate (PPG)). Forty participants with varying degrees of familiarity with soy sauce were presented with samples of rice and three commercial soy sauces with and without brand information. The results showed that (1) liking and arousal were affected primarily by the specific tastes, but not by branding and familiarity. In contrast, facial expressions were affected by branding and familiarity, and to a lesser degree by specific tastes. (2) RPPG heart rate and PPG both showed effects of branding and familiarity. However, RPPG heart rate needs further development because it underestimated the heart rate compared to PPG and was less sensitive to changes over time and with activity (viewing of brand information and tasting). In conclusion, this study suggests that recording of facial expressions and heart rates may no longer be limited to laboratories but can be done remotely using video images, which offers opportunities for large-scale testing in consumer science.

Effect of Branding and Familiarity of Soy Sauces on Valence and Arousal as Determined by Facial Expressions, Physiological Measures, Emojis, and Ratings

Food experiences can be summarized along two main dimensions: valence and arousal, which can be measured explicitly with subjective ratings or implicitly with physiological and behavioral measures. Food experiences are not only driven by the food's intrinsic properties, such as its taste, texture, and aroma, but also by extrinsic properties such as brand information and the consumers' previous experiences with the foods. In this study, valence and arousal to intrinsic and extrinsic properties of soy sauce were measured in consumers that varied in their previous experience with soy sauce, using a combination of explicit (scores and emojis), implicit (heart rate and skin conductance), and behavioral measures (facial expressions). Forty participants, high- and low-frequency users, were presented with samples of rice and three commercial soy sauces without and with brand information that either matched or non-matched the taste of the soy sauce. In general, skin conductance and facial expressions showed relatively low arousal during exposure to the brand name and again lowest arousal during tasting. Heart rate was lowest during exposure to the brand name and increased during tasting probably resulting from the motor activity during chewing. Furthermore, the results showed that explicit liking and arousal scores were primarily affected by the taste of the specific soy sauce and by the participants' previous experience with soy sauces. These scores were not affected by branding information. In contrast, facial expressions, skin conductance, and heart rate were primarily affected by (1) the participants' level of experience with soy sauce, (2) whether or not branding information was provided, and (3) whether or not the branding information matched with the taste. In conclusion, this study suggests that liking scores may be most sensitive to the food's intrinsic taste properties, whereas implicit measures and facial expressions may be most sensitive to extrinsic properties such as brand information. All measures were affected by the consumers' previous food experiences.

Comparing Explicit and Implicit Measures for Assessing Cross-Cultural Food Experience

The present study investigated the potential of implicit physiological measures to provide objective measures of affective food experience in contrast to explicit self-report ratings in a cross-cultural context. Dutch and Thai participants viewed 120 food images portraying universal food image categories (regular and molded food) and cultural food image categories (typically Dutch and Thai food). The universal food images were taken as ground truth high and low valence stimuli, where we assumed no genuine difference in affective experience between nationalities. In contrast, for the cultural food images, we did expect a genuine difference between nationalities. Participants were asked to rate valence, arousal and liking of each food image. In addition, heart rate (HR) and phasic electrodermal activity (EDA) responses to the images were recorded. Typically Asian and Western response biases were found for explicit ratings of regular and molded food with an extreme response style for Dutch, and a middle response style for Thai participants. However, such bias was not observed in HR. For cultural food image categories, HR showed the hypothesized interaction between participant nationality and food image category, reflecting the expected genuine difference between nationalities in affective food experience. Besides presenting participants with images, we also asked participants to taste typically Thai and Dutch drinks. Similar to images, a significant interaction between participant nationality and cultural food category was found for HR. An interaction was also found for sip size, while this was not seen in explicit measures. We attribute this to differences in the moment that these measures were taken. In this study, phasic EDA did not appear to be a sensitive measure of affective food experience, possibly since stimuli mostly differed in valence rather than arousal. To conclude, our study constitutes an example where cultural bias negatively affected the accuracy of self-reports, and only the implicit physiological measures followed the prior expectations of genuine food experience, indicating the potential of these measures to study cross-cultural food experience.

The EmojiGrid as a rating tool for the affective appraisal of touch

In this study we evaluate the convergent validity of a new graphical self-report tool (the EmojiGrid) for the affective appraisal of perceived touch events. The EmojiGrid is a square grid labeled with facial icons (emoji) showing different levels of valence and arousal. The EmojiGrid is language independent and efficient (a single click suffices to report both valence and arousal), making it a practical instrument for studies on affective appraisal. We previously showed that participants can intuitively and reliably report their affective appraisal (valence and arousal) of visual, auditory and olfactory stimuli using the EmojiGrid, even without additional (verbal) instructions. However, because touch events can be bidirectional and dynamic, these previous results cannot be generalized to the touch domain. In this study, participants reported their affective appraisal of video clips showing different interpersonal (social) and object-based touch events, using either the validated 9-point SAM (Self-Assessment Mannikin) scale or the EmojiGrid. The valence ratings obtained with the EmojiGrid and the SAM are in excellent agreement. The arousal ratings show good agreement for object-based touch and moderate agreement for social touch. For social touch and at more extreme levels of valence, the EmojiGrid appears more sensitive to arousal than the SAM. We conclude that the EmojiGrid can also serve as a valid and efficient graphical self-report instrument to measure human affective response to a wide range of tactile signals.

The effect of an image of watchful eyes on the evaluation of the appearance of food

It is known that an eye-like image promotes generosity. It is also known that the evaluation of the visual deliciousness of food is improved in the presence of an emotionally positive stimulus. The purpose of the present study was to examine whether the presence of open eyes (OPEN) causes generous behavior altering the evaluation of the visual deliciousness of food, and how the images of open and closed eyes (CLOSED) affect human emotions. Seventeen women participated in the present study. A picture of food was presented on a computer screen, and the participants predicted and evaluated its visual deliciousness. An image of OPEN or that of CLOSED was presented simultaneously with a picture of food. There was a significant difference between the OPEN and CLOSED conditions, as demonstrated by the scores on a nine-point Likert scale for visual deliciousness; the ratings in the OPEN condition were significantly higher than those in the CLOSED condition (p = 0.004). There was no significant difference in the image of watchful eyes for the perceived relaxation state; the ratings in the OPEN condition were not significantly higher than those in the CLOSED condition (p = 0.716). The results of the present study revealed that the evaluation of the visual deliciousness of food based on its appearance was likely due to the presence of an image of open watchful eyes, increasing the perceived visual deliciousness of the food without any changes in the participants’ emotions.

Affective rating of audio and video clips using the EmojiGrid

Background: In this study we measured the affective appraisal of sounds and video clips using a newly developed graphical self-report tool: the EmojiGrid. The EmojiGrid is a square grid, labeled with emoji that express different degrees of valence and arousal. Users rate the valence and arousal of a given stimulus by simply clicking on the grid. Methods: In Experiment I, observers (N=150, 74 males, mean age=25.2±3.5) used the EmojiGrid to rate their affective appraisal of 77 validated sound clips from nine different semantic categories, covering a large area of the affective space. In Experiment II, observers (N=60, 32 males, mean age=24.5±3.3) used the EmojiGrid to rate their affective appraisal of 50 validated film fragments varying in positive and negative affect (20 positive, 20 negative, 10 neutral). Results: The results of this study show that for both sound and video, the agreement between the mean ratings obtained with the EmojiGrid and those obtained with an alternative and validated affective rating tool in previous studies in the literature, is excellent for valence and good for arousal. Our results also show the typical universal U-shaped relation between mean valence and arousal that is commonly observed for affective sensory stimuli, both for sound and video. Conclusions: We conclude that the EmojiGrid can be used as an affective self-report tool for the assessment of sound and video-evoked emotions.

Affective rating of audio and video clips using the EmojiGrid

Background: In this study we measured the affective appraisal of sounds and video clips using a newly developed graphical self-report tool: the EmojiGrid. The EmojiGrid is a square grid, labeled with emoji that express different degrees of valence and arousal. Users rate the valence and arousal of a given stimulus by simply clicking on the grid. Methods: In Experiment I, observers (N=150, 74 males, mean age=25.2±3.5) used the EmojiGrid to rate their affective appraisal of 77 validated sound clips from nine different semantic categories, covering a large area of the affective space. In Experiment II, observers (N=60, 32 males, mean age=24.5±3.3) used the EmojiGrid to rate their affective appraisal of 50 validated film fragments varying in positive and negative affect (20 positive, 20 negative, 10 neutral). Results: The results of this study show that for both sound and video, the agreement between the mean ratings obtained with the EmojiGrid and those obtained with an alternative and validated affective rating tool in previous studies in the literature, is excellent for valence and good for arousal. Our results also show the typical universal U-shaped relation between mean valence and arousal that is commonly observed for affective sensory stimuli, both for sound and video. Conclusions: We conclude that the EmojiGrid can be used as an affective self-report tool for the assessment of sound and video-evoked emotions.

A Systematic Review of Emoji: Current Research and Future Perspectives

A growing body of research explores emoji, which are visual symbols in computer mediated communication (CMC). In the 20 years since the first set of emoji was released, research on it has been on the increase, albeit in a variety of directions. We reviewed the extant body of research on emoji and noted the development, usage, function, and application of emoji. In this review article, we provide a systematic review of the extant body of work on emoji, reviewing how they have developed, how they are used differently, what functions they have and what research has been conducted on them in different domains. Furthermore, we summarize directions for future research on this topic.

The EmojiGrid as a Tool to Assess Experienced and Perceived Emotions

In a recent study on food-evoked emotions, we observed that people often misunderstood the currently available affective self-report tools. We, therefore, developed a new intuitive and language-independent self-report instrument called the EmojiGrid: a rectangular response grid labeled with facial icons (emoji) that express different degrees of valence and arousal. We found that participants intuitively and reliably reported their affective appraisal of food by clicking on the EmojiGrid, even without verbal instructions. In this study, we investigated whether the EmojiGrid can also serve as a tool to assess one’s own (experienced) emotions and perceived emotions of others. In the first experiment, participants (N = 90) used the EmojiGrid to report their own emotions, evoked by affective images from a database with corresponding normative ratings (obtained with a 9-point self-assessment mannikin scale). In the second experiment, participants (N = 61) used the EmojiGrid to report the perceived emotional state of persons shown in different affective situations, in pictures from a database with corresponding normative ratings (obtained with a 7-point Likert scale). For both experiments, the affective (valence and arousal) ratings obtained with the EmojiGrid show excellent agreement with the data provided in the literature (intraclass correlations of at least 0.90). Also, the relation between valence and arousal shows the classic U-shape at the group level. Thus, the EmojiGrid appears to be a useful graphical self-report instrument for the assessment of evoked and perceived emotions.

CROCUFID: A Cross-Cultural Food Image Database for Research on Food Elicited Affective Responses

We present CROCUFID: a CROss-CUltural Food Image Database that currently contains 840 images, including 479 food images with detailed metadata and 165 images of non-food items. The database includes images of sweet, savory, natural, and processed food from Western and Asian cuisines. To create sufficient variability in valence and arousal we included images of food with different degrees of appetitiveness (fresh, unfamiliar, molded or rotten, spoiled, and partly consumed). We used a standardized photographing protocol, resulting in high resolution images depicting all food items on a standard background (a white plate), seen from a fixed viewing (45°) angle. CROCUFID is freely available under the CC-By Attribution 4.0 International license and hosted on the OSF repository. The advantages of the CROCUFID database over other databases are its (1) free availability, (2) full coverage of the valence - arousal space, (3) use of standardized recording methods, (4) inclusion of multiple cuisines and unfamiliar foods, (5) availability of normative and demographic data, (6) high image quality and (7) capability to support future (e.g., virtual and augmented reality) applications. Individuals from the United Kingdom (N = 266), North-America (N = 275), and Japan (N = 264) provided normative ratings of valence, arousal, perceived healthiness, and desire-to-eat using visual analog scales (VAS). In addition, for each image we computed 17 characteristics that are known to influence affective observer responses (e.g., texture, regularity, complexity, and colorfulness). Significant differences between groups and significant correlations between image characteristics and normative ratings were in accordance with previous research, indicating the validity of CROCUFID. We expect that CROCUFID will facilitate comparability across studies and advance experimental research on the determinants of food-elicited emotions. We plan to extend CROCUFID in the future with images of food from a wide range of different cuisines and with non-food images (for applications in for instance neuro-physiological studies). We invite researchers from all parts of the world to contribute to this effort by creating similar image sets that can be linked to this collection, so that CROCUFID will grow into a truly multicultural food database.