Integration or separation in the processing of facial properties--a computational view

Varying sex and identity of faces affects face categorization differently in humans and computational models

Our faces display socially important sex and identity information. How perceptually independent are these facial characteristics? Here, we used a sex categorization task to investigate how changing faces in terms of either their sex or identity affects sex categorization of those faces, whether these manipulations affect sex categorization similarly when the original faces were personally familiar or unknown, and, whether computational models trained for sex classification respond similarly to human observers. Our results show that varying faces along either sex or identity dimension affects their sex categorization. When the sex was swapped (e.g., female faces became male looking, Experiment 1), sex categorization performance was different from that with the original unchanged faces, and significantly more so for people who were familiar with the original faces than those who were not. When the identity of the faces was manipulated by caricaturing or anti-caricaturing them (these manipulations either augment or diminish idiosyncratic facial information, Experiment 2), sex categorization performance to caricatured, original, and anti-caricatured faces increased in that order, independently of face familiarity. Moreover, our face manipulations showed different effects upon computational models trained for sex classification and elicited different patterns of responses in humans and computational models. These results not only support the notion that the sex and identity of faces are processed integratively by human observers but also demonstrate that computational models of face categorization may not capture key characteristics of human face categorization.

Memory for facial expressions on the autism spectrum: The influence of gaze direction and type of expression

Face memory research in autism has largely neglected memory for facial expressions, in favor of memory for identity. This study in three experiments examined the role of gaze direction and type of expression on memory for facial expressions in relation to the autism spectrum. In the learning phase, four combinations of facial expressions (joy/anger) and gaze direction (toward/away), displayed by 16 different identities, were presented. In a subsequent surprise test the same identities were presented displaying neutral expressions, and the expression of each identity had to be recalled. In Experiment 1, typically-developed (TD) individuals with low and high Autism Quotient (AQ) scores were tested with three repetitions of each emotion/gaze combination, which did not produce any modulations. In Experiment 2, another group of TD individuals with low and high AQ scores were tested with eight repetitions, resulting in a "happy advantage" and a "direct gaze advantage", but no interactions. In Experiment 3, individuals with high-functioning autism (HFA) and a matched TD group were tested using eight repetitions. The HFA group revealed no emotion or gaze effects, while the matched TD group showed both a happy and a direct gaze advantage, and again no interaction. The results suggest that in autistic individuals the memory for facial expressions is intact, but is not modulated by the person's expression type and gaze direction. We discuss whether anomalous implicit learning of facial cues could have contributed to these findings, its relevance for social intuition, and its possible contribution to social deficits in autism. LAY SUMMARY: It has often been found that memory for someone's face (facial identity) is less good in autism. However, it is not yet known whether memory for someone's facial expression is also less good in autism. In this study, the memory for expressions of joy and anger was investigated in typically-developed (TD) individuals who possessed either few or many autistic-like traits (Experiments 1 and 2), and in individuals with high-functioning autism (Experiment 3). The gaze direction was also varied (directed either toward, or away from, the observer). We found that TD individuals best remembered expressions of joy, and remembered expressions of both joy and anger better when the gaze was directed at them. These effects did not depend on the extent to which they possessed autistic-like traits. Autistic participants remembered the facial expression of a previously encountered person as good as TD participants did. However, in contrast to the TD participants, the memory of autistic participants was not influenced by the expression type and gaze direction of the previously encountered persons. We discuss whether this may lead to difficulties in the development of social intuition, which in turn could give rise to difficulties in social interaction that are characteristic for autism.

A data-driven characterisation of natural facial expressions when giving good and bad news

Facial expressions carry key information about an individual's emotional state. Research into the perception of facial emotions typically employs static images of a small number of artificially posed expressions taken under tightly controlled experimental conditions. However, such approaches risk missing potentially important facial signals and within-person variability in expressions. The extent to which patterns of emotional variance in such images resemble more natural ambient facial expressions remains unclear. Here we advance a novel protocol for eliciting natural expressions from dynamic faces, using a dimension of emotional valence as a test case. Subjects were video recorded while delivering either positive or negative news to camera, but were not instructed to deliberately or artificially pose any specific expressions or actions. A PCA-based active appearance model was used to capture the key dimensions of facial variance across frames. Linear discriminant analysis distinguished facial change determined by the emotional valence of the message, and this also generalised across subjects. By sampling along the discriminant dimension, and back-projecting into the image space, we extracted a behaviourally interpretable dimension of emotional valence. This dimension highlighted changes commonly represented in traditional face stimuli such as variation in the internal features of the face, but also key postural changes that would typically be controlled away such as a dipping versus raising of the head posture from negative to positive valences. These results highlight the importance of natural patterns of facial behaviour in emotional expressions, and demonstrate the efficacy of using data-driven approaches to study the representation of these cues by the perceptual system. The protocol and model described here could be readily extended to other emotional and non-emotional dimensions of facial variance.

Convolutional neural net face recognition works in non-human-like ways

Convolutional neural networks (CNNs) give the state-of-the-art performance in many pattern recognition problems but can be fooled by carefully crafted patterns of noise. We report that CNN face recognition systems also make surprising 'errors'. We tested six commercial face recognition CNNs and found that they outperform typical human participants on standard face-matching tasks. However, they also declare matches that humans would not, where one image from the pair has been transformed to appear a different sex or race. This is not due to poor performance; the best CNNs perform almost perfectly on the human face-matching tasks, but also declare the most matches for faces of a different apparent race or sex. Although differing on the salience of sex and race, humans and computer systems are not working in completely different ways. They tend to find the same pairs of images difficult, suggesting some agreement about the underlying similarity space.

Evaluating the independence of age, sex, and race in judgment of faces

Extracting the dimensions of age, sex, race from faces is fundamental for many aspects of social cognition such as person construal, impression formation, and social interaction. While cognitive researchers consider these dimensions to be independent in processing, social psychology researchers have recently demonstrated the emergence of strong interactive patterns between these categories, especially, when social biases are involved. The current study harnessed the classic Garner's speeded classification task (Garner, 1974) and Stroop task (1935) to evaluate the level of independence between age, sex, and race in a systematic and exhaustive fashion, with an eye on the potential influence of social biases. The degree of separability was evaluated in a pairwise fashion, with each experiment testing one pair. In Experiment 1a and 1b, sex and race were tested with strong (Experiment 1b) or weak (Experiment 1a) social bias. Experiment 2 was set to assess the separability of sex and race. And Experiment 3 was aimed at evaluating the separability of age and sex. The results revealed that neither of the pairs of dimensions produced Garner interferences (and are therefore separable dimensions). However, when strong social bias was present, the dimensions did produce redundancy gains and Stroop-like effects, which reflected the presence of abiding social biases. A holistic-to-analytic model is presented to account for these results, according to which, the dimensions are initially processed as integral dimensions, but then become gradually separable.

Social information in equine movement gestalts

One model of signal evolution is based on the notion that behaviours become increasingly detached from their original biological functions to obtain a communicative value. Selection may not always favour the evolution of such transitions, for instance, if signalling is costly due to predators usurping signal production. Here, we collected inertial movement sensing data recorded from multiple locations in free-ranging horses (Equus caballus), which we subjected to a machine learning algorithm to extract kinematic gestalt profiles. This yielded surprisingly rich and multi-layered sets of information. In particular, we were able to discriminate identity, breed, sex and some personality traits from the overall movement patterns of freely moving subjects. Our study suggests that, by attending to movement gestalts, domestic horses, and probably many other group-living animals, have access to rich social information passively but reliably made available by conspecifics, a finding that we discuss in relation with current signal evolution theories.

Structural connections support emotional connections: Uncinate Fasciculus microstructure is related to the ability to decode facial emotion expressions

The Uncinate Fasciculus (UF) is an association fibre tract connecting regions in the frontal and anterior temporal lobes. UF disruption is seen in several disorders associated with impaired social behaviour, but its functional role is unclear. Here we set out to test the hypothesis that the UF is important for facial expression processing, an ability fundamental to adaptive social behaviour. In two separate experiments in healthy adults, we used high-angular resolution diffusion-weighted imaging (HARDI) and constrained spherical deconvolution (CSD) tractography to virtually dissect the UF, plus a control tract (the corticospinal tract (CST)), and quantify, via fractional anisotropy (FA), individual differences in tract microstructure. In Experiment 1, participants completed the Reading the Mind in the Eyes Task (RMET), a well-validated assay of facial expression decoding. In Experiment 2, a different set of participants completed the RMET, plus an odd-emotion-out task of facial emotion discrimination. In both experiments, participants also completed a control odd-identity-out facial identity discrimination task. In Experiment 1, FA of the right-, but not the left-hemisphere, UF was significantly correlated with performance on the RMET task, specifically for emotional, but not neutral expressions. UF FA was not significantly correlated with facial identity discrimination performance. In Experiment 2, FA of the right-, but not left-hemisphere, UF was again significantly correlated with performance on emotional items from the RMET, together with performance on the facial emotion discrimination task. Again, no significant association was found between UF FA and facial identity discrimination performance. Our findings highlight the contribution of right-hemisphere UF microstructure to inter-individual variability in the ability to decode facial emotion expressions, and may explain why disruption of this pathway affects social behaviour.

•

We studied white matter microstructure correlates of facial emotion decoding skills.

•

Focused on the role of a key limbic tract, the Uncinate Fasciculus (UF).

•

Right UF microstructure linked to facial expression decoding skills.

•

UF microstructure not related to facial identity discrimination skills.

•

Right UF has a distinct role in the processing of facial expressions of emotion.