Accuracy of Inferring Self- and Other-Preferences from Spontaneous Facial Expressions

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Human facial expressions are regarded as a vital indicator of one's emotion and intention, and even reveal the state of health and wellbeing. Emotional states have been associated with information processing within and between subcortical and cortical areas of the brain, including the amygdala and prefrontal cortex. In this study, we evaluated the relationship between spontaneous human facial affective expressions and multi-modal brain activity measured via non-invasive and wearable sensors: functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) signals. The affective states of twelve male participants detected via fNIRS, EEG, and spontaneous facial expressions were investigated in response to both image-content stimuli and video-content stimuli. We propose a method to jointly evaluate fNIRS and EEG signals for affective state detection (emotional valence as positive or negative). Experimental results reveal a strong correlation between spontaneous facial affective expressions and the perceived emotional valence. Moreover, the affective states were estimated by the fNIRS, EEG, and fNIRS + EEG brain activity measurements. We show that the proposed EEG + fNIRS hybrid method outperforms fNIRS-only and EEG-only approaches. Our findings indicate that the dynamic (video-content based) stimuli triggers a larger affective response than the static (image-content based) stimuli. These findings also suggest joint utilization of facial expression and wearable neuroimaging, fNIRS, and EEG, for improved emotional analysis and affective brain-computer interface applications.

Resolving uncertainty in a social world

Consider the range of social behaviours we engage in every day. In each case, there are a multitude of unknowns, reflecting the many sources of uncertainty inherent to social inference. We describe how uncertainty manifests in social environments (the thoughts and intentions of others are largely hidden, making it difficult to predict a person's behaviour) and why people are motivated to reduce the aversive feelings generated by uncertainty. We propose a three-part model whereby social uncertainty is initially reduced through automatic modes of inference (such as impression formation) before more control-demanding modes of inference (such as perspective-taking) are deployed to narrow one's predictions even more. Finally, social uncertainty is attenuated further through learning processes that update these predictions based on new information. Our framework integrates research across fields to offer an account of the mechanisms motivating social cognition and action, laying the groundwork for future experiments that can illuminate the impact of uncertainty on social cognition.

"You Should Have Seen the Look on Your Face…": Self-awareness of Facial Expressions

The awareness of facial expressions allows one to better understand, predict, and regulate his/her states to adapt to different social situations. The present research investigated individuals' awareness of their own facial expressions and the influence of the duration and intensity of expressions in two self-reference modalities, a real-time condition and a video-review condition. The participants were instructed to respond as soon as they became aware of any facial movements. The results revealed that awareness rates were 57.79% in the real-time condition and 75.92% in the video-review condition. The awareness rate was influenced by the intensity and (or) the duration. The intensity thresholds for individuals to become aware of their own facial expressions were calculated using logistic regression models. The results of Generalized Estimating Equations (GEE) revealed that video-review awareness was a significant predictor of real-time awareness. These findings extend understandings of human facial expression self-awareness in two modalities.

Bias and accuracy in judging sexism in mixed-gender social interactions

This research examined bias and accuracy in judging hostile and benevolent sexism during mixed-gender interactions. Bias is defined as underestimation or overestimation of a partner’s sexism. Accuracy is defined as covariation in two different ways, as (a) the strength of the association between a dyad member’s judgment and their partner’s sexism, across dyads, and (b) the ability to differentiate sexism between multiple targets. In Studies 1 and 2, members of mixed-gender dyads rated their own and their partners’ benevolent and hostile sexism. Overall, there was no covariation, across dyads, between perceptions and the partner’s self-reported sexism. However, women overestimated men’s hostile sexism; there was no evidence of biases for women judging men’s benevolent sexism. Men underestimated women’s hostile sexism and overestimated benevolent sexism. In Study 3, participants watched brief videos of male or female students (targets) from Study 1 and completed benevolent or hostile sexism items for each target as they thought the target would fill them out. Accuracy for detecting sexism across multiple targets (using sensitivity correlations) was significantly above chance for both forms of sexism. Male and female participants were more accurate at detecting hostile sexism in male targets than female targets. Participants were more accurate at detecting benevolent sexism of same-gender targets than opposite-gender targets. When judging targets of opposite gender, women’s judgments were significantly above chance for both forms of sexism, but men were not accurate for either forms of sexism. These studies suggest that there is bias and accuracy in first impression judgments of sexism.

Automated prediction of preferences using facial expressions

We introduce a computer vision problem from social cognition, namely, the automated detection of attitudes from a person's spontaneous facial expressions. To illustrate the challenges, we introduce two simple algorithms designed to predict observers’ preferences between images (e.g., of celebrities) based on covert videos of the observers’ faces. The two algorithms are almost as accurate as human judges performing the same task but nonetheless far from perfect. Our approach is to locate facial landmarks, then predict preference on the basis of their temporal dynamics. The database contains 768 videos involving four different kinds of preferences. We make it publically available.