A Cautionary Note on Predicting Social Judgments from Faces with Deep Neural Networks

A first impression of the future

We offer a response to six commentaries on our target article 'Understanding trait impressions from faces'. A broad consensus emerged with authors emphasizing the importance of increasing the diversity of faces and participants, integrating research on impressions beyond the face, and continuing to develop methods needed for data-driven approaches. We propose future directions for the field based on these themes.

Modeling Biological Face Recognition with Deep Convolutional Neural Networks

Deep convolutional neural networks (DCNNs) have become the state-of-the-art computational models of biological object recognition. Their remarkable success has helped vision science break new ground, and recent efforts have started to transfer this achievement to research on biological face recognition. In this regard, face detection can be investigated by comparing face-selective biological neurons and brain areas to artificial neurons and model layers. Similarly, face identification can be examined by comparing in vivo and in silico multidimensional "face spaces." In this review, we summarize the first studies that use DCNNs to model biological face recognition. On the basis of a broad spectrum of behavioral and computational evidence, we conclude that DCNNs are useful models that closely resemble the general hierarchical organization of face recognition in the ventral visual pathway and the core face network. In two exemplary spotlights, we emphasize the unique scientific contributions of these models. First, studies on face detection in DCNNs indicate that elementary face selectivity emerges automatically through feedforward processing even in the absence of visual experience. Second, studies on face identification in DCNNs suggest that identity-specific experience and generative mechanisms facilitate this particular challenge. Taken together, as this novel modeling approach enables close control of predisposition (i.e., architecture) and experience (i.e., training data), it may be suited to inform long-standing debates on the substrates of biological face recognition.

Three's a crowd: Fast ensemble perception of first impressions of trustworthiness

Trustworthiness impressions are fundamental social judgements with far-reaching consequences in many aspects of society, including criminal justice, leadership selection and partner preferences. Thus far, most research has focused on facial characteristics that make a face individually appear more or less trustworthy. However, in everyday life, faces are not always perceived in isolation but are often encountered in crowds. It has been proposed that we deal with the large amount of facial information in a group by extracting summary statistics of the crowd, a phenomenon called ensemble perception. Prior research showed that ensemble perception occurs for various facial features, such as emotional expression, facial identity, and attractiveness. Here, we investigated whether observers can integrate the level of trustworthiness from multiple faces to extract an average impression of the crowd. Across four studies, participants were presented with crowds of faces and were asked to report their average level of trustworthiness with an adjustment (Experiment 1) and a rating task (Experiments 2 and 3). Participants were able to extract an ensemble perception of trustworthiness impressions from multiple faces. Moreover, observers were able to form a summary statistic of trustworthiness impressions from a group of faces as quickly as 250 ms (Experiment 4). Taken together, these results demonstrate that ensemble perception can occur at the level of impressions of trustworthiness. Thus, these critical social judgements not only occur for individual faces but are also integrated into a unique ensemble impression of crowds. Our findings contribute to the development of a more ecological approach to the study of trust impressions, since they provide an understanding of trustworthiness judgements not only on an individual level, but on a much broader social group level. Furthermore, our results drive forward new theory because they demonstrate for the first time that ensemble representations cover a broad range of phenomena than previously recognized, including complex high-level facial trait judgements such as trustworthiness impressions.

Comprehensive Social Trait Judgments From Faces in Autism Spectrum Disorder

Processing social information from faces is difficult for individuals with autism spectrum disorder (ASD). However, it remains unclear whether individuals with ASD make high-level social trait judgments from faces in the same way as neurotypical individuals. Here, we comprehensively addressed this question using naturalistic face images and representatively sampled traits. Despite similar underlying dimensional structures across traits, online adult participants with self-reported ASD showed different judgments and reduced specificity within each trait compared with neurotypical individuals. Deep neural networks revealed that these group differences were driven by specific types of faces and differential utilization of features within a face. Our results were replicated in well-characterized in-lab participants and partially generalized to more controlled face images (a preregistered study). By investigating social trait judgments in a broader population, including individuals with neurodevelopmental variations, we found important theoretical implications for the fundamental dimensions, variations, and potential behavioral consequences of social cognition.