Tsinghua facial expression database - A database of facial expressions in Chinese young and older women and men: Development and validation

Face age modulates face ensemble coding

Research has demonstrated that humans possess the remarkable ability to swiftly extract ensemble statistics, specifically the average identity, from sets of stimuli, such as facial crowds. This phenomenon is known as ensemble perception. Although previous studies have investigated how physiognomic features like gender and race influence face ensemble perception, the impact of face age on face ensemble coding performance remains a relatively unexplored area. Here, we demonstrated ensemble coding of multiple faces in terms of an average face was impacted by face age. In both Experiment 1 and 2, adult participants viewed sets of four faces that were of either own-age or other-age and then judged whether the subsequently presented probe face was present or not in the preceding set. The other-age faces were manipulated as older faces in Experiment 1 and baby faces in Experiment 2. The results suggested participants incorrectly endorsed a morphed set average to be the member of the set, pointing to face ensemble coding ability. Furthermore, the results of Experiment 1 revealed adult participants displayed an own-age superiority when other-age faces were manipulated as older faces, however, the results of Experiment 2 found when other-age faces were manipulated as baby faces, participants displayed stronger visual averaging tendency towards other-age faces rather own-age faces, showing a babyface effect. Together, the present research provided initial evidence that face ensemble coding performance was modulated by face age.

Social Category Modulation of the Happy Face Advantage

The size of the happy face advantage-faster categorization of happy faces-is modulated by interactions between perceiver and target social categories, with reliable happy face advantages for ingroups but not necessarily outgroups. The current understanding of this phenomenon is constrained by the limited social categories typically used in experiments. To better understand the mechanism(s) underpinning social category modulation of the happy face advantage, we used racially more diverse samples of perceivers and target faces and manipulated the intergroup context in which they appeared. We found evidence of ingroup bias, with perceivers often showing a larger happy face advantage for ingroups than outgroups (Experiments 1-2). We also found evidence of majority/minority group bias, with perceivers showing a larger happy face advantage for majority outgroups than minority outgroups (Experiments 2-3c). These findings suggest social category modulation of the happy face advantage is a dynamic context-dependent process.

Faces of different socio-cultural identities impact emotional meaning learning for L2 words

This study investigated how exposure to Caucasian and Chinese faces influences native Mandarin-Chinese speakers' learning of emotional meanings for English L2 words. Participants were presented with English pseudowords repeatedly paired with either Caucasian faces or Chinese faces showing emotions of disgust, sadness, or neutrality as a control baseline. Participants' learning was evaluated through both within-modality (i.e., testing participants with new sets of faces) and cross-modality (i.e., testing participants with sentences expressing the learned emotions) generalization tests. When matching newly learned L2 words with new faces, participants from both groups were more accurate under the neutral condition compared to sad condition. The advantage of neutrality extended to sentences as participants matched newly learned L2 words with neutral sentences more accurately than with both disgusting and sad ones. Differences between the two groups were also found in the cross-modality generalization test in which the Caucasian-face Group outperformed the Chinese-face Group in terms of accuracy in sad trials. However, the Chinese-face Group was more accurate in neutral trials in the same test. We thus conclude that faces of diverse socio-cultural identities exert different impacts on the emotional meaning learning for L2 words.

Limitation of switching sensory information flow in flexible perceptual decision making

Humans can flexibly change rules to categorize sensory stimuli, but their performance degrades immediately after a task switch. This switch cost is believed to reflect a limitation in cognitive control, although the bottlenecks remain controversial. Here, we show that humans exhibit a brief reduction in the efficiency of using sensory inputs to form a decision after a rule change. Participants classified face stimuli based on one of two rules, switching every few trials. Psychophysical reverse correlation and computational modeling reveal a reduction in sensory weighting, which recovers within a few hundred milliseconds after stimulus presentation. This reduction depends on the sensory features being switched, suggesting a constraint in routing the sensory information flow. We propose that decision-making circuits cannot fully adjust their sensory readout based on a context cue alone, but require the presence of an actual stimulus to tune it, leading to a limitation in flexible perceptual decision making.

Hemispheric dominance in reading system alters contribution to face processing lateralization across development

Face processing dominates the right hemisphere. This lateralization can be affected by co-lateralization within the same system and influence between different systems, such as neural competition from reading acquisition. Yet, how the relationship pattern changes through development remains unknown. This study examined the lateralization of core face processing and word processing in different age groups. By comparing fMRI data from 36 school-aged children and 40 young adults, we investigated whether there are age and regional effects on lateralization, and how relationships between lateralization within and between systems change across development. Our results showed significant right hemispheric lateralization in the core face system and left hemispheric lateralization in reading-related areas for both age groups when viewing faces and texts passively. While all participants showed stronger lateralization in brain regions of higher functional hierarchy when viewing faces, only adults exhibited this lateralization when viewing texts. In both age cohorts, there was intra-system co-lateralization for face processing, whereas an inter-system relationship was only found in adults. Specifically, functional lateralization of Broca's area during reading negatively predicted functional asymmetry in the FFA during face perception. This study initially provides neuroimaging evidence for the reading-induced neural competition theory from a maturational perspective in Chinese cohorts.

Automatic processing of unattended mean emotion: Evidence from visual mismatch responses

The mean emotion from multiple facial expressions can be extracted rapidly and precisely. However, it remains debated whether mean emotion processing is automatic which can occur under no attention. To address this question, we used a passive oddball paradigm and recorded event-related brain potentials when participants discriminated the changes in the central fixation while a set of four faces was presented in the periphery. The face set consisted of one happy and three angry expressions (mean negative) or one angry and three happy expressions (mean positive), and the mean negative and mean positive face sets were shown with a probability of 20% (deviant) and 80% (standard) respectively in the sequence, or the vice versa. The cluster-based permutation analyses showed that the visual mismatch negativity started early at around 92 ms and was also observed in later time windows when the mean emotion was negative, while a mismatch positivity was observed at around 168-266 ms when the mean emotion was positive. The results suggest that there might be different mechanisms underlying the processing of mean negative and mean positive emotions. More importantly, the brain can detect the changes in the mean emotion automatically, and ensemble coding for multiple facial expressions can occur in an automatic fashion without attention.

Effects of age on the interactions of attentional and emotional processes: a prefrontal fNIRS study

The aging of attentional and emotional functions has been extensively studied but relatively independently. Therefore, the relationships between aging and the interactions of attentional and emotional processes remain elusive. This study aimed to determine how age affected the interactions between attentional and emotional processes during adulthood. One-hundred forty adults aged 18-79 performed the emotional variant of the Attention Network Test, which probed alerting, orienting, and executive control in the presence and absence of threatening faces. During this task, contexts with varying levels of task preparatory processes were created to modulate the effect of threatening faces on attention, and functional near-infrared spectroscopy (fNIRS) was used to examine the neural underpinnings of the behavioural effects. The behavioural results showed that aging was associated with a significant decline in alerting efficiency, and there was a statistical trend for age-related deficits in executive control. Despite these age differences, age did not significantly moderate the interactions among attentional networks or between attention and emotion. Additionally, the fNIRS results showed that decreased frontal cortex functioning might underlie the age-related decline in executive control. Therefore, while aging has varying effects on different attentional networks, the interactions of attentional and emotional processes remain relatively unaffected by age.

Auto Diagnosis of Parkinson's Disease Via a Deep Learning Model Based on Mixed Emotional Facial Expressions

Parkinson's disease (PD) is a common degenerative disease of the nervous system in the elderly. The early diagnosis of PD is very important for potential patients to receive prompt treatment and avoid the aggravation of the disease. Recent studies have found that PD patients always suffer from emotional expression disorder, thus forming the characteristics of "masked faces". Based on this, we thus propose an auto PD diagnosis method based on mixed emotional facial expressions in the paper. Specifically, the proposed method is cast into four steps: Firstly, we synthesize virtual face images containing six basic expressions (i.e., anger, disgust, fear, happiness, sadness, and surprise) via generative adversarial learning, in order to approximate the premorbid expressions of PD patients; Secondly, we design an effective screening scheme to assess the quality of the above synthesized facial expression images and then shortlist the high-quality ones; Thirdly, we train a deep feature extractor accompanied with a facial expression classifier based on the mixture of the original facial expression images of the PD patients, the high-quality synthesized facial expression images of PD patients, and the normal facial expression images from other public face datasets; Finally, with the well-trained deep feature extractor, we thus adopt it to extract the latent expression features for six facial expression images of a potential PD patient to conduct PD/non-PD prediction. To show real-world impacts, we also collected a new facial expression dataset of PD patients in collaboration with a hospital. Extensive experiments are conducted to validate the effectiveness of the proposed method for PD diagnosis and facial expression recognition.

Motivation and emotional distraction interact and affect executive functions

Previous research on cool-hot executive function (EF) interactions has examined the effects of motivation and emotional distraction on cool EF separately, focusing on one EF component at a time. Although both incentives and emotional distractors have been shown to modulate attention, how they interact and affect cool EF processes is still unclear. Here, we used an experimental paradigm that manipulated updating, inhibition, and shifting demands to determine the interactions of motivation and emotional distraction in the context of cool EF. Forty-five young adults (16 males, 29 females) completed the go/no-go (inhibition), two-back (updating), and task-switching (shifting) tasks. Monetary incentives were implemented to manipulate motivation, and task-irrelevant threatening or neutral faces were presented before the target stimulus to manipulate emotional distraction. We found that incentives significantly improved no-go accuracy, two-back accuracy, and reaction time (RT) switch cost. While emotional distractors had no significant effects on overall task performance, they abolished the incentive effects on no-go accuracy and RT switch cost. Altogether, these findings suggest that motivation and emotional distraction interact in the context of cool EF. Specifically, transient emotional distraction disrupts the upregulation of control activated by incentives. The present investigation has advanced knowledge about the relationship between cool and hot EF and highlights the importance of considering motivation-emotion interactions for a fuller understanding of control.

Grip strength, working memory, and emotion perception in middle-aged males

This study examined the association between grip strength and emotional working memory in middle-aged adults. Seventy-six males aged 40-60years (mean=48.5years, SD=5.4) participated in this cross-sectional study. They completed a muscular fitness assessment using a maximum grip strength test and emotional n-back tasks under two emotion conditions (fearful and neutral facial pictures) and two working memory loads (1-back and 2-back tasks). Hierarchical regression analyses indicated that greater muscular fitness was associated with superior working memory performance in the fearful condition in both the 1-back and 2-back tasks, after controlling for confounders. Greater muscular fitness was also associated with superior working memory performance in the neutral condition when the working memory load was high (2-back task) but not low (1-back task). These findings suggest a positive association between muscular fitness and emotional working memory and highlight the importance of maintaining muscular fitness for physical and cognitive-emotional well-being in middle-aged adults.

The pictorial set of Emotional Social Interactive Scenarios between Chinese Adults (ESISCA): Development and validation

Affective picture databases with a single facial expression or body posture in one image have been widely applied to investigate emotion. However, to date, there was no standardized database containing the stimuli which involve multiple emotional signals in social interactive scenarios. The current study thus developed a pictorial set comprising 274 images depicting two Chinese adults' interactive scenarios conveying emotions of happiness, anger, sadness, fear, disgust, and neutral. The data of the valence and arousal ratings of the scenes and the emotional categories of the scenes and the faces in the images were provided in the present study. Analyses of the data collected from 70 undergraduate students suggested high reliabilities of the valence and arousal ratings of the scenes and high judgmental agreements in categorizing the scene and facial emotions. The findings suggested that the present dataset is well constructed and could be useful for future studies to investigate the emotion recognition or empathy in social interactions in both healthy and clinical (e.g., ASD) populations.

Ensemble coding of multiple facial expressions is not affected by attentional load

Human observers can extract the mean emotion from multiple faces rapidly and precisely. However, whether attention is required in the ensemble coding of facial expressions remains debated. In this study, we examined the effect of attentional load on mean emotion processing with the dual-task paradigm. Individual emotion processing was also investigated as the control task. In the experiment, the letter string and a set of four happy or angry faces of various emotional intensities were shown. Participants had to complete the string task first, judging either the string color (low attention load) or the presence of the target letter (high attention load). Then a cue appeared indicating whether the secondary task was to evaluate the mean emotion of the faces or the emotion of the cued single face, and participants made their judgments on the visual analog scale. The results showed that compared with the color task, the letter task had a longer response time and lower accuracy, which verified the valid manipulation of the attention loads. More importantly, there was no significant difference in averaging performance between the low and high attention loads. By contrast, the individual face processing was impaired under the high attention load relative to the low attentional load. In addition, the advantage of extracting mean emotion over individual emotion was larger under the high attentional load. These results support the power of averaging and provide new evidence that a rather small amount of attention is needed in the ensemble coding of multiple facial expressions.

A Chinese Face Dataset with Dynamic Expressions and Diverse Ages Synthesized by Deep Learning

Facial stimuli have gained increasing popularity in research. However, the existing Chinese facial datasets primarily consist of static facial expressions and lack variations in terms of facial aging. Additionally, these datasets are limited to stimuli from a small number of individuals, in that it is difficult and time-consuming to recruit a diverse range of volunteers across different age groups to capture their facial expressions. In this paper, a deep-learning based face editing approach, StyleGAN, is used to synthesize a Chinese face dataset, namely SZU-EmoDage, where faces with different expressions and ages are synthesized. Leverage on the interpolations of latent vectors, continuously dynamic expressions with different intensities, are also available. Participants assessed emotional categories and dimensions (valence, arousal and dominance) of the synthesized faces. The results show that the face database has good reliability and validity, and can be used in relevant psychological experiments. The availability of SZU-EmoDage opens up avenues for further research in psychology and related fields, allowing for a deeper understanding of facial perception.

Negative emotion amplifies retrieval practice effect for both task-relevant and task-irrelevant information

Selective retrieval of task-relevant information often facilitates memory retention of that information. However, it is still unclear if selective retrieval of task-relevant information can alter memory for task-irrelevant information, and the role of emotional arousal in it. In two experiments, we used emotional and neutral faces as stimuli, and participants were asked to memorise the name (who is this person?) and location (where does he/she come from?) associated with each face in initial study. Then, half of the studied faces were presented as cues, and participants were asked to retrieve the corresponding names (Experiment 1) or locations (Experiment 2). Finally, all the faces were presented and participants were asked to retrieve both the corresponding names and locations. The results of the final test showed that retrieval practice not only enhanced memory of task-relevant information but also enhanced memory of task-irrelevant information. More importantly, negative emotion amplified the retrieval practice effect overall, with a larger retrieval-induced benefit for the negative than neutral condition. These findings demonstrated an emotional arousal amplification effect on retrieval-induced enhancement effects, suggesting that the advantage of the retrieved memory representations can be amplified by emotional arousal even without explicit goals in a task setting.

The Sabancı University Dynamic Face Database (SUDFace): Development and validation of an audiovisual stimulus set of recited and free speeches with neutral facial expressions

Faces convey a wide range of information, including one's identity, and emotional and mental states. Face perception is a major research topic in many research fields, such as cognitive science, social psychology, and neuroscience. Frequently, stimuli are selected from a range of available face databases. However, even though faces are highly dynamic, most databases consist of static face stimuli. Here, we introduce the Sabancı University Dynamic Face (SUDFace) database. The SUDFace database consists of 150 high-resolution audiovisual videos acquired in a controlled lab environment and stored with a resolution of 1920 × 1080 pixels at a frame rate of 60 Hz. The multimodal database consists of three videos of each human model in frontal view in three different conditions: vocalizing two scripted texts (conditions 1 and 2) and one Free Speech (condition 3). The main focus of the SUDFace database is to provide a large set of dynamic faces with neutral facial expressions and natural speech articulation. Variables such as face orientation, illumination, and accessories (piercings, earrings, facial hair, etc.) were kept constant across all stimuli. We provide detailed stimulus information, including facial features (pixel-wise calculations of face length, eye width, etc.) and speeches (e.g., duration of speech and repetitions). In two validation experiments, a total number of 227 participants rated each video on several psychological dimensions (e.g., neutralness and naturalness of expressions, valence, and the perceived mental states of the models) using Likert scales. The database is freely accessible for research purposes.

Development and Validation of the AIIMS Facial Toolbox for Emotion Recognition

Background

Emotional facial expression database, used in emotion regulation studies, is a special set of pictures with high social and biological relevance. We present the AIIMS Facial Toolbox for Emotion Recognition (AFTER) database. It consists of pictures of 15 adult professional artists displaying seven facial expressions-neutral, happiness, anger, sadness, disgust, fear, and surprise.

Methods

This cross-sectional study enrolled 15 volunteer students from a professional drama college in India (six males and nine females; mean age = 26.2 ± 1.93 years). They were instructed to pose with different emotional expressions in high and low intensity. A total of 240 pictures were captured in a brightly lit room against a common, light background. Each picture was validated independently by 19 mental health professionals and two professional teachers of dramatic art. Apart from recognition of emotional quality, ratings were done for each emotion on a 5-point Likert scale with respect to three dimensions-intensity, clarity, and genuineness. Results are discussed in terms of mean scores on all four parameters.

Results

The percentage hit rate for all the emotions, after exclusion of contempt, was 84.3%, with the mean kappa for emotional expression being 0.68. Mean scores on intensity, clarity, and genuineness of the emotions depicted in the pictures were high.

Conclusions

The database would be useful in the Indian context for researching facial emotion recognition. It has been validated among a group of experts and was found to have high inter-rater reliability.

Effects of Robot Animacy and Emotional Expressions on Perspective-Taking Abilities: A Comparative Study across Age Groups

The global population is inevitably aging due to increased life expectancy and declining birth rates, leading to an amplified demand for innovative social and healthcare services. One promising avenue is the introduction of companion robots. These robots are designed to provide physical assistance as well as emotional support and companionship, necessitating effective human-robot interaction (HRI). This study explores the role of cognitive empathy within HRI, focusing on the influence of robot facial animacy and emotional expressions on perspective-taking abilities-a key aspect of cognitive empathy-across different age groups. To this end, a director task involving 60 participants (30 young and 30 older adults) with varying degrees of robot facial animacy (0%, 50%, 100%) and emotional expressions (happy, neutral) was conducted. The results revealed that older adults displayed enhanced perspective-taking with higher animacy faces. Interestingly, while happiness on high-animacy faces improved perspective-taking, the same expression on low-animacy faces reduced it. These findings highlight the importance of considering facial animacy and emotional expressions in designing companion robots for older adults to optimize user engagement and acceptance. The study's implications are pertinent to the design and development of socially effective service robots, particularly for the aging population.

The prefrontal cortex is differentially involved in implicit and explicit facial emotion processing: An fNIRS study

Despite extensive research, the differential roles of the prefrontal cortex (PFC) in implicit and explicit facial emotion processing remain elusive. Functional near-infrared spectroscopy (fNIRS) is a neuroimaging technique that can measure changes in both oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) concentrations. Currently, how HbO and HbR change during facial emotion processing remains unclear. Here, fNIRS was used to examine and compare PFC activation during implicit and explicit facial emotion processing. Forty young adults performed a facial-matching task that required either emotion discrimination (explicit task) or age discrimination (implicit task), and the activation of their PFCs was measured by fNIRS. Participants attempted the task on two occasions to determine whether their activation patterns were maintained over time. The PFC displayed increases in HbO and/or decreases in HbR during the implicit and explicit facial emotion tasks. Importantly, there were significantly greater changes in PFC HbO during the explicit task, whereas no significant difference in HbR changes between conditions was found. Between sessions, HbO changes were reduced across tasks, but the difference in HbO changes between the implicit and explicit tasks remained unchanged. The test-retest reliability of the behavioral measures was excellent, whereas that of fNIRS measures was mostly poor to fair. Thus, the PFC plays a specific role in recognizing facial expressions, and its differential involvement in implicit and explicit facial emotion processing can be consistently captured at the group level by changes in HbO. This study demonstrates the potential of fNIRS for elucidating the neural mechanisms underlying facial emotion recognition.

Micro-architecture design exploration template for AutoML case study on SqueezeSEMAuto

Convolutional Neural Network (CNN) models have been commonly used primarily in image recognition tasks in the deep learning area. Finding the right architecture needs a lot of hand-tune experiments which are time-consuming. In this paper, we exploit an AutoML framework that adds to the exploration of the micro-architecture block and the multi-input option. The proposed adaption has been applied to SqueezeNet with SE blocks combined with the residual block combinations. The experiments assume three search strategies: Random, Hyperband, and Bayesian algorithms. Such combinations can lead to solutions with superior accuracy while the model size can be monitored. We demonstrate the application of the approach against benchmarks: CIFAR-10 and Tsinghua Facial Expression datasets. The searches allow the designer to find the architectures with better accuracy than the traditional architectures without hand-tune efforts. For example, CIFAR-10, leads to the SqueezeNet architecture using only 4 fire modules with 59% accuracy. When exploring SE block insertion, the model with good insertion points can lead to an accuracy of 78% while the traditional SqueezeNet can achieve an accuracy of around 50%. For other tasks, such as facial expression recognition, the proposed approach can lead up to an accuracy of 71% with the proper insertion of SE blocks, the appropriate number of fire modules, and adequate input merging, while the traditional model can achieve the accuracy under 20%.

Kick Cat Effect: Social Context Shapes the Form and Extent of Emotional Contagion

Emotional contagion refers to the transmission and interaction of emotions among people. Researchers have mainly focused on its process and mechanism, often simplifying its social background due to its complexity. Therefore, in this study, we attempt to explore whether the presence and clarity of social context affect emotional contagion and the related neural mechanisms. In Experiment 1, participants were asked to report their subjective experiences after being exposed to the facial expressions of emotional expressers, with or without the corresponding social context being presented. The results revealed that positive or negative expressions from the expressers elicited corresponding emotional experiences in the receivers, regardless of the presence of social context. However, when the social context was absent, the degree of emotional contagion was greater. In Experiment 2, we further investigated the effect of the clarity of social contexts on emotional contagion and its neural mechanisms. The results showed an effect consistent with those in Experiment 1 and highlighted the special role of N1, N2, P3, and LPP components in this process. According to the emotions as social information theory, individuals may rely more on social appraisal when they lack sufficient contextual information. By referencing the expressions of others and maintaining emotional convergence with them, individuals can adapt more appropriately to their current environment.

Study on emotion recognition bias in different regional groups

Human-machine communication can be substantially enhanced by the inclusion of high-quality real-time recognition of spontaneous human emotional expressions. However, successful recognition of such expressions can be negatively impacted by factors such as sudden variations of lighting, or intentional obfuscation. Reliable recognition can be more substantively impeded due to the observation that the presentation and meaning of emotional expressions can vary significantly based on the culture of the expressor and the environment within which the emotions are expressed. As an example, an emotion recognition model trained on a regionally-specific database collected from North America might fail to recognize standard emotional expressions from another region, such as East Asia. To address the problem of regional and cultural bias in emotion recognition from facial expressions, we propose a meta-model that fuses multiple emotional cues and features. The proposed approach integrates image features, action level units, micro-expressions and macro-expressions into a multi-cues emotion model (MCAM). Each of the facial attributes incorporated into the model represents a specific category: fine-grained content-independent features, facial muscle movements, short-term facial expressions and high-level facial expressions. The results of the proposed meta-classifier (MCAM) approach show that a) the successful classification of regional facial expressions is based on non-sympathetic features b) learning the emotional facial expressions of some regional groups can confound the successful recognition of emotional expressions of other regional groups unless it is done from scratch and c) the identification of certain facial cues and features of the data-sets that serve to preclude the design of the perfect unbiased classifier. As a result of these observations we posit that to learn certain regional emotional expressions, other regional expressions first have to be "forgotten".

Context-specific effects of threatening faces on alerting, orienting, and executive control: A fNIRS study

Real-world threatening faces possess both useful and irrelevant attributes with respect to the current goal. How these attributes interact and affect attention, which comprises at least three processes hypothesized to engage the frontal lobes (alerting, orienting, and executive control), remains poorly understood. Here, the neurocognitive effects of threatening facial expressions on the three processes of attention were examined through the emotional Attention Network Test (ANT) and functional near-infrared spectroscopy (fNIRS). Forty-seven (20M, 27F) young adults performed a blocked version of the arrow flanker task with neutral and angry facial cues applied in three cue conditions (no, center, and spatial). Hemodynamic changes occurring in participants' frontal cortices during task performance were recorded by multichannel fNIRS. Behavioral results indicated that alerting, orienting, and executive control processes existed in both the neutral and angry conditions. However, depending on the context, angry facial cues affected these processes differently compared with neutral facial cues. Specifically, the angry face disrupted the classical decrease in reaction time from the no-cue to center-cue condition specifically during the congruent condition. Additionally, fNIRS results revealed significant frontal cortical activation during the incongruent vs. congruent task; neither cue nor emotion significantly affected frontal activation. Thus, the findings suggest that the angry face affects all three attentional processes while exerting context-specific effects on attention. They also imply that during the ANT, the frontal cortex is most involved in executive control. The present study offers essential insights into how various attributes of threatening faces interact and alter attention.

A database of heterogeneous faces for studying naturalistic expressions

Facial expressions are thought to be complex visual signals, critical for communication between social agents. Most prior work aimed at understanding how facial expressions are recognized has relied on stimulus databases featuring posed facial expressions, designed to represent putative emotional categories (such as 'happy' and 'angry'). Here we use an alternative selection strategy to develop the Wild Faces Database (WFD); a set of one thousand images capturing a diverse range of ambient facial behaviors from outside of the laboratory. We characterized the perceived emotional content in these images using a standard categorization task in which participants were asked to classify the apparent facial expression in each image. In addition, participants were asked to indicate the intensity and genuineness of each expression. While modal scores indicate that the WFD captures a range of different emotional expressions, in comparing the WFD to images taken from other, more conventional databases, we found that participants responded more variably and less specifically to the wild-type faces, perhaps indicating that natural expressions are more multiplexed than a categorical model would predict. We argue that this variability can be employed to explore latent dimensions in our mental representation of facial expressions. Further, images in the WFD were rated as less intense and more genuine than images taken from other databases, suggesting a greater degree of authenticity among WFD images. The strong positive correlation between intensity and genuineness scores demonstrating that even the high arousal states captured in the WFD were perceived as authentic. Collectively, these findings highlight the potential utility of the WFD as a new resource for bridging the gap between the laboratory and real world in studies of expression recognition.

Training and Profiling a Pediatric Facial Expression Classifier for Children on Mobile Devices: Machine Learning Study

BACKGROUND

Implementing automated facial expression recognition on mobile devices could provide an accessible diagnostic and therapeutic tool for those who struggle to recognize facial expressions, including children with developmental behavioral conditions such as autism. Despite recent advances in facial expression classifiers for children, existing models are too computationally expensive for smartphone use.

OBJECTIVE

We explored several state-of-the-art facial expression classifiers designed for mobile devices, used posttraining optimization techniques for both classification performance and efficiency on a Motorola Moto G6 phone, evaluated the importance of training our classifiers on children versus adults, and evaluated the models' performance against different ethnic groups.

METHODS

We collected images from 12 public data sets and used video frames crowdsourced from the GuessWhat app to train our classifiers. All images were annotated for 7 expressions: neutral, fear, happiness, sadness, surprise, anger, and disgust. We tested 3 copies for each of 5 different convolutional neural network architectures: MobileNetV3-Small 1.0x, MobileNetV2 1.0x, EfficientNetB0, MobileNetV3-Large 1.0x, and NASNetMobile. We trained the first copy on images of children, second copy on images of adults, and third copy on all data sets. We evaluated each model against the entire Child Affective Facial Expression (CAFE) set and by ethnicity. We performed weight pruning, weight clustering, and quantize-aware training when possible and profiled each model's performance on the Moto G6.

RESULTS

Our best model, a MobileNetV3-Large network pretrained on ImageNet, achieved 65.78% accuracy and 65.31% F1-score on the CAFE and a 90-millisecond inference latency on a Moto G6 phone when trained on all data. This accuracy is only 1.12% lower than the current state of the art for CAFE, a model with 13.91x more parameters that was unable to run on the Moto G6 due to its size, even when fully optimized. When trained solely on children, this model achieved 60.57% accuracy and 60.29% F1-score. When trained only on adults, the model received 53.36% accuracy and 53.10% F1-score. Although the MobileNetV3-Large trained on all data sets achieved nearly a 60% F1-score across all ethnicities, the data sets for South Asian and African American children achieved lower accuracy (as much as 11.56%) and F1-score (as much as 11.25%) than other groups.

CONCLUSIONS

With specialized design and optimization techniques, facial expression classifiers can become lightweight enough to run on mobile devices and achieve state-of-the-art performance. There is potentially a "data shift" phenomenon between facial expressions of children compared with adults; our classifiers performed much better when trained on children. Certain underrepresented ethnic groups (e.g., South Asian and African American) also perform significantly worse than groups such as European Caucasian despite similar data quality. Our models can be integrated into mobile health therapies to help diagnose autism spectrum disorder and provide targeted therapeutic treatment to children.

Rich Intercultural Music Engagement Enhances Cultural Understanding: The Impact of Learning a Musical Instrument Outside of One's Lived Experience

Rich intercultural music engagement (RIME) is an embodied form of engagement whereby individuals immerse themselves in foreign musical practice, for example, by learning a traditional instrument from that culture. The present investigation evaluated whether RIME with Chinese or Middle Eastern music can nurture intercultural understanding. White Australian participants were randomly assigned to one of two plucked-string groups: Chinese pipa (n = 29) or Middle Eastern oud (n = 29). Before and after the RIME intervention, participants completed measures of ethnocultural empathy, tolerance, social connectedness, explicit and implicit attitudes towards ethnocultural groups, and open-ended questions about their experience. Following RIME, White Australian participants reported a significant increase in ethnocultural empathy, tolerance, feelings of social connection, and improved explicit and implicit attitudes towards Chinese and Middle Eastern people. However, these benefits differed between groups. Participants who learned Chinese pipa reported reduced bias and increased social connectedness towards Chinese people, but not towards Middle Eastern people. Conversely, participants who learned Middle Eastern oud reported a significant increase in social connectedness towards Middle Eastern people, but not towards Chinese people. This is the first experimental evidence that participatory RIME is an effective tool for understanding a culture other than one's own, with the added potential to reduce cultural bias.

Iranian emotional face database: Acquisition and validation of a stimulus set of basic facial expressions

Facial expressions play an essential role in social interactions. Databases of face images have furnished theories of emotion perception, as well as having applications in other disciplines such as facial recognition technology. However, the faces of many ethnicities remain largely underrepresented in the existing face databases, which can impact the generalizability of the theories and technologies developed based on them. Here, we present the first survey-validated database of Iranian faces. It consists of 248 images from 40 Iranian individuals portraying six emotional expressions-anger, sadness, fear, disgust, happiness, and surprise-as well as the neutral state. The photos were taken in a studio setting, following the common scenarios of emotion induction, and controlling for conditions of lighting, camera setup, and the model's head posture. An evaluation survey confirmed high agreement between the models' intended expressions and the raters' perception of them. The database is freely available online for academic research purposes.

Construction of face databases for tasks to recognize facial expressions of basic emotions: a systematic review

ABSTRACT. Recognizing the other's emotions is an important skill for the social context that can be modulated by variables such as gender, age, and race. A number of studies seek to elaborate specific face databases to assess the recognition of basic emotions in different contexts. Objectives: This systematic review sought to gather these studies, describing and comparing the methodologies used in their elaboration. Methods: The databases used to select the articles were the following: PubMed, Web of Science, PsycInfo, and Scopus. The following word crossing was used: “Facial expression database OR Stimulus set AND development OR Validation.” Results: A total of 36 articles showed that most of the studies used actors to express the emotions that were elicited from specific situations to generate the most spontaneous emotion possible. The databases were mainly composed of colorful and static stimuli. In addition, most of the studies sought to establish and describe patterns to record the stimuli, such as color of the garments used and background. The psychometric properties of the databases are also described. Conclusions: The data presented in this review point to the methodological heterogeneity among the studies. Nevertheless, we describe their patterns, contributing to the planning of new research studies that seek to create databases for new contexts.

Towards East Asian Facial Expression Recognition in the Real World: A New Database and Deep Recognition Baseline

In recent years, the focus of facial expression recognition (FER) has gradually shifted from laboratory settings to challenging natural scenes. This requires a great deal of real-world facial expression data. However, most existing real-world databases are based on European-American cultures, and only one is for Asian cultures. This is mainly because the data on European-American expressions are more readily accessed and publicly available online. Owing to the diversity of huge data, FER in European-American cultures has recently developed rapidly. In contrast, the development of FER in Asian cultures is limited by the data. To narrow this gap, we construct a challenging real-world East Asian facial expression (EAFE) database, which contains 10,000 images collected from 113 Chinese, Japanese, and Korean movies and five search engines. We apply three neural network baselines including VGG-16, ResNet-50, and Inception-V3 to classify the images in EAFE. Then, we conduct two sets of experiments to find the optimal learning rate schedule and loss function. Finally, by training with the cosine learning rate schedule and island loss, ResNet-50 can achieve the best accuracy of 80.53% on the testing set, proving that the database is challenging. In addition, we used the Microsoft Cognitive Face API to extract facial attributes in EAFE, so that the database can also be used for facial recognition and attribute analysis. The release of the EAFE can encourage more research on Asian FER in natural scenes and can also promote the development of FER in cross-cultural domains.

Face masks inhibit facial cues for approachability and trustworthiness: an eyetracking study

Wearing face masks during the Covid-19 pandemic has undeniable benefits from our health perspective. However, the interpersonal costs on social interactions may have been underappreciated. Because masks obscure critical facial regions signaling approach/avoidance intent and social trust, this implies that facial inference of approachability and trustworthiness may be severely discounted. Here, in our eyetracking experiment, we show that people judged masked faces as less approachable and trustworthy. Further analyses showed that the attention directed towards the eye region relative to the mouth region mediated the effect on approachability, but not on trustworthiness. This is because for masked faces, with the mouth region obscured, visual attention is then automatically diverted away from the mouth and towards the eye region, which is an undiagnostic cue for judging a target's approachability. Together, these findings support that mask-wearing inhibits the critical facial cues needed for social judgements.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12144-022-03705-8.

Effects of wearing a transparent face mask on perception of facial expressions

Wearing face masks in public has become the norm in many countries post-2020. Although mask-wearing is effective in controlling infection, it has the negative side effect of occluding the mask wearer's facial expressions. The purpose of this study was to investigate the effects of wearing transparent masks on the perception of facial expressions. Participants were required to categorize the perceived facial emotion of female (Experiment 1) and male (Experiment 2) faces with different facial expressions and to rate the perceived emotion intensity of the faces. Based on the group, the participants were assigned to, the faces were presented with a surgical mask, a transparent mask, or without a mask. The results showed that wearing a surgical mask impaired the performance of reading facial expressions, both with respect to recognition and perceived intensity of facial emotions. Specifically, the impairments were robustly observed in fear and happy faces for emotion recognition, and in happy faces for perceived intensity of emotion in Experiments 1 and 2. However, the impairments were moderated by wearing a transparent mask instead of a surgical mask. During the coronavirus disease 2019 (COVID-19) pandemic, the transparent mask can be used in a range of situations where face-to-face communication is important.

The Attitude towards a Stranger and Assessment of his Age based on a Photo Image of a Face Transformed in the FaceApp Application

The hypothesis of the study was the assumption that significant differences can be found in the attitude of the subject of perception to the object of perception (“model”) depending on the conditional age stage associated with age-related changes in appearance. Methods: 1) The procedure of “Photovideopresentation of appearance” by T.A. Vorontsova (a set of 36 photos transformed in the FaceApp application); 2) “Methodology for the study of conscious personal relationships to each member of the group and to oneself” by T.A. Vorontsova. Selection: 178 women and 156 men from 21 to 60 years old (M=37.24; SD=10.46). Results: 1) the attitude of subjects of perception to objects of perception significantly changes depending on the conditional age stage associated with changes in appearance: antipathy increases (in 64% of observations); antipathy decreases (in 36% of observations); disrespect increases (in 25% of observations); disrespect decreases (in 75% of observations); distance increases / decreases (50%); 2) gender differences in the dynamics of attitudes towards objects of perception were found: an increase in respect for men, in contrast to the multidirectional dynamics of respect for women. The recorded dynamics of relations reveals benevolent (an increase in respect) and hostile ageism (an increase in antipathy) towards older people who have obvious age-related changes in appearance. Also, the data obtained on the Russian sample confirm the existence of the age stereotype “a woman is getting old, a man is getting mature”. The data are discussed in connection with age stigma, the influence of additional factors, and the possibilities of using FaceApp in scientific research.

The Influence of Key Facial Features on Recognition of Emotion in Cartoon Faces

Cartoon faces are widely used in social media, animation production, and social robots because of their attractive ability to convey different emotional information. Despite their popular applications, the mechanisms of recognizing emotional expressions in cartoon faces are still unclear. Therefore, three experiments were conducted in this study to systematically explore a recognition process for emotional cartoon expressions (happy, sad, and neutral) and to examine the influence of key facial features (mouth, eyes, and eyebrows) on emotion recognition. Across the experiments, three presentation conditions were employed: (1) a full face; (2) individual feature only (with two other features concealed); and (3) one feature concealed with two other features presented. The cartoon face images used in this study were converted from a set of real faces acted by Chinese posers, and the observers were Chinese. The results show that happy cartoon expressions were recognized more accurately than neutral and sad expressions, which was consistent with the happiness recognition advantage revealed in real face studies. Compared with real facial expressions, sad cartoon expressions were perceived as sadder, and happy cartoon expressions were perceived as less happy, regardless of whether full-face or single facial features were viewed. For cartoon faces, the mouth was demonstrated to be a feature that is sufficient and necessary for the recognition of happiness, and the eyebrows were sufficient and necessary for the recognition of sadness. This study helps to clarify the perception mechanism underlying emotion recognition in cartoon faces and sheds some light on directions for future research on intelligent human-computer interactions.

An Exploratory Study on Cross-Cultural Differences in Facial Emotion Recognition Between Adults From Malaysia and Australia

While culture and depression influence the way in which humans process emotion, these two areas of investigation are rarely combined. Therefore, the aim of this study was to investigate the difference in facial emotion recognition among Malaysian Malays and Australians with a European heritage with and without depression. A total of 88 participants took part in this study (Malays n = 47, Australians n = 41). All participants were screened using The Structured Clinical Interview for DSM-5 Clinician Version (SCID-5-CV) to assess the Major Depressive Disorder (MDD) diagnosis and they also completed the Beck Depression Inventory (BDI). This study consisted of the facial emotion recognition (FER) task whereby the participants were asked to look at facial images and determine the emotion depicted by each of the facial expressions. It was found that depression status and cultural group did not significantly influence overall FER accuracy. Malaysian participants without MDD and Australian participants with MDD performed quicker as compared to Australian participants without MDD on the FER task. Also, Malaysian participants more accurately recognized fear as compared to Australian participants. Future studies can focus on the extent of the influence and other aspects of culture and participant condition on facial emotion recognition.