Vienna Talking Faces (ViTaFa): A multimodal person database with synchronized videos, images, and voices

Social perception relies on different sensory channels, including vision and audition, which are specifically important for judgements of appearance. Therefore, to understand multimodal integration in person perception, it is important to study both face and voice in a synchronized form. We introduce the Vienna Talking Faces (ViTaFa) database, a high-quality audiovisual database focused on multimodal research of social perception. ViTaFa includes different stimulus modalities: audiovisual dynamic, visual dynamic, visual static, and auditory dynamic. Stimuli were recorded and edited under highly standardized conditions and were collected from 40 real individuals, and the sample matches typical student samples in psychological research (young individuals aged 18 to 45). Stimuli include sequences of various types of spoken content from each person, including German sentences, words, reading passages, vowels, and language-unrelated pseudo-words. Recordings were made with different emotional expressions (neutral, happy, angry, sad, and flirtatious). ViTaFa is freely accessible for academic non-profit research after signing a confidentiality agreement form via https://osf.io/9jtzx/ and stands out from other databases due to its multimodal format, high quality, and comprehensive quantification of stimulus features and human judgements related to attractiveness. Additionally, over 200 human raters validated emotion expression of the stimuli. In summary, ViTaFa provides a valuable resource for investigating audiovisual signals of social perception.

Perception of threat and intent to harm from vocal and facial cues

What constitutes a "threatening tone of voice"? There is currently little research exploring how listeners infer threat, or the intention to cause harm, from speakers' voices. Here, we investigated the influence of key linguistic variables on these evaluations (Study 1). Results showed a trend for voices perceived to be lower in pitch, particularly those of male speakers, to be evaluated as sounding more threatening and conveying greater intent to harm. We next investigated the evaluation of multimodal stimuli comprising voices and faces varying in perceived dominance (Study 2). Visual information about the speaker's face had a significant effect on threat and intent ratings. In both experiments, we observed a relatively low level of agreement among individual listeners' evaluations, emphasising idiosyncrasy in the ways in which threat and intent-to-harm are perceived. This research provides a basis for the perceptual experience of a "threatening tone of voice," along with an exploration of vocal and facial cue integration in social evaluation.

Dominance of face over voice in human attractiveness judgments: ERP evidence

The attractiveness of a person, a complex, and socially relevant type of information, is transmitted in many ways, not least through face and voice. However, it is unclear how the stimulus domains carrying attractiveness information interact. The present study explored the audiovisual perception of attractiveness in a Stroop-like paradigm using event-related potentials (ERPs). Participants were presented with face-voice pairs carrying congruent or incongruent attractiveness information and, in turn, judged the attractiveness level of each domain while ignoring the other. Voice attractiveness judgments were influenced by unattended face attractiveness, in terms of both, early perceptual encoding (N170, P200) as well as later evaluative stages (N400, LPC). In contrast, effects of unattended voice attractiveness on face attractiveness judgments were confined to early perceptual encoding (N170). These results demonstrate not only the interaction of multiple domains in human attractiveness perception at different processing stages but also a relative dominance of face over voice attractiveness.

The Effects of Pitch Manipulation on Male Ratings of Female Speakers and Their Voices

Vocal and facial cues typically co-occur in natural settings, and multisensory processing of voice and face relies on their synchronous presentation. Psychological research has examined various facial and vocal cues to attractiveness as well as to judgements of sexual dimorphism, health, and age. However, few studies have investigated the interaction of vocal and facial cues in attractiveness judgments under naturalistic conditions using dynamic, ecologically valid stimuli. Here, we used short videos or audio tracks of females speaking full sentences and used a manipulation of voice pitch to investigate cross-modal interactions of voice pitch on facial attractiveness and related ratings. Male participants had to rate attractiveness, femininity, age, and health of synchronized audio-video recordings or voices only, with either original or modified voice pitch. We expected audio stimuli with increased voice pitch to be rated as more attractive, more feminine, healthier, and younger. If auditory judgements cross-modally influence judgements of facial attributes, we additionally expected the voice pitch manipulation to affect ratings of audiovisual stimulus material. We tested 106 male participants in a within-subject design in two sessions. Analyses revealed that voice recordings with increased voice pitch were perceived to be more feminine and younger, but not more attractive or healthier. When coupled with video recordings, increased pitch lowered perceived age of faces, but did not significantly influence perceived attractiveness, femininity, or health. Our results suggest that our manipulation of voice pitch has a measurable impact on judgements of femininity and age, but does not measurably influence vocal and facial attractiveness in naturalistic conditions.

A cross-cultural study of sex-typicality and averageness: Correlation between frontal and lateral measures of human faces

OBJECTIVES

Facial averageness and sexual dimorphism are extensively studied attractiveness markers, which are viewed as possible indicators of biological quality. Both are complex morphological traits, and both can be easily assessed from frontal and lateral projection of a human face. Interestingly, examination of mutual relations between the frontal and lateral dimensions of these markers has so far received little attention in published research.

METHODS

In our cross-cultural study, we used geometric morphometric data from male and female faces from Brazil, Cameroon, Colombia, and the Czech Republic, and analyzed correlations between frontal and lateral measurements of averageness and degree of maleness/femaleness, that is, the individual variation in features that characterize sexual dimorphism. We also analyzed whether the association between frontal and lateral measurements differed in men and women.

RESULTS

In general, our results showed a moderate correlation in sexually dimorphic features between lateral and frontal facial configuration in both sexes, while frontal and lateral facial averageness was moderately correlated only in women. This pattern was less consistent when individual populations were analyzed separately.

CONCLUSIONS

We suggest that, in general, the weak association between lateral and frontal facial configurations may be the result of selection pressures in favor of individual identity signals. Moreover, especially in women, the frontal and lateral dimension of a given facial attractiveness marker may provide similar information about the qualities of the individual. The absence of a significant correlation in male facial averageness suggests that frontal and lateral averageness convey different information about an individual.

The Role of Face and Voice Cues in Predicting the Outcome of Student Representative Elections

First impressions formed after seeing someone's face or hearing their voice can affect many social decisions, including voting in political elections. Despite the many studies investigating the independent contribution of face and voice cues to electoral success, their integration is still not well understood. Here, we examine a novel electoral context, student representative ballots, allowing us to test the generalizability of previous studies. We also examine the independent contributions of visual, auditory, and audiovisual information to social judgments of the candidates, and their relationship to election outcomes. Results showed that perceived trustworthiness was the only trait significantly related to election success. These findings contrast with previous reports on the importance of perceived competence using audio or visual cues only in the context of national political elections. The present study highlights the role of real-world context and emphasizes the importance of using ecologically valid stimulus presentation in understanding real-life social judgment.

Positive association between facial and vocal femininity/masculinity in women but not in men

Multicomponent stimuli improve information reception. In women, perceived facial and vocal femininity-masculinity (FM) are concordant; however, mixed results are found for men. Some feminine and masculine traits are related to sex hormone action and can indicate reproductive qualities. However, most of the current research about human mate choice focuses on isolated indicators, especially visual assessment of faces. We therefore examined the cross-modal concordance hypothesis by testing correlations between perceptions of FM based on facial, vocal, and behavioral stimuli. Standardized facial pictures, vocal recordings and dance videos of 38 men and 41 women, aged 18-35 years, were rated by 21 male and 43 female students, aged 18-35 years, on 100-point scale (0 = very feminine; 100 = very masculine). All participants were Brazilian students from University of Sao Paulo. In women, facial and vocal FM correlated positively, suggesting concordant information about mate quality. Such results were not found in men, indicating multiple messages, which agree with women's multifaceted preference for male FM. In both sexes, FM of dance did not correlate with voices or faces, indicating different information and distinct process of development. We thus partially supported the cross-modal concordance hypothesis.

Positive association between vocal and facial attractiveness in women but not in men: A cross-cultural study

Various species use multiple sensory modalities in the communication processes. In humans, female facial appearance and vocal display are correlated and it has been suggested that they serve as redundant markers indicating the bearer's reproductive potential and/or residual fertility. In men, evidence for redundancy of facial and vocal attractiveness is ambiguous. We tested the redundancy/multiple signals hypothesis by correlating perceived facial and vocal attractiveness in men and women from two different populations, Brazil and the Czech Republic. We also investigated whether facial and vocal attractiveness are linked to facial morphology. Standardized facial pictures and vocal samples of 86 women (47 from Brazil) and 81 men (41 from Brazil), aged 18-35, were rated for attractiveness by opposite-sex raters. Facial and vocal attractiveness were found to positively correlate in women but not in men. We further applied geometric morphometrics and regressed facial shape coordinates on facial and vocal attractiveness ratings. In women, facial shape was linked to their facial attractiveness but there was no association between facial shape and vocal attractiveness. In men, none of these associations was significant. Having shown that women with more attractive faces possess also more attractive voices, we thus only partly supported the redundant signal hypothesis.

Concordant Cues in Faces and Voices

Information from faces and voices combines to provide multimodal signals about a person. Faces and voices may offer redundant, overlapping (backup signals), or complementary information (multiple messages). This article reports two experiments which investigated the extent to which faces and voices deliver concordant information about dimensions of fitness and quality. In Experiment 1, participants rated faces and voices on scales for masculinity/femininity, age, health, height, and weight. The results showed that people make similar judgments from faces and voices, with particularly strong correlations for masculinity/femininity, health, and height. If, as these results suggest, faces and voices constitute backup signals for various dimensions, it is hypothetically possible that people would be able to accurately match novel faces and voices for identity. However, previous investigations into novel face–voice matching offer contradictory results. In Experiment 2, participants saw a face and heard a voice and were required to decide whether the face and voice belonged to the same person. Matching accuracy was significantly above chance level, suggesting that judgments made independently from faces and voices are sufficiently similar that people can match the two. Both sets of results were analyzed using multilevel modeling and are interpreted as being consistent with the backup signal hypothesis.

Variation and signatures of selection on the human face

There has been much debate about why humans throughout the world differ in facial form. Previous studies of human skull morphology found levels of among-population differentiation that were comparable to those of neutral genetic markers, suggesting that genetic drift (neutral processes) played an important role in influencing facial differentiation. However, variation in soft-tissue morphology has not been studied in detail. In this study, we analyzed high-resolution 3D images of soft-tissue facial form in four Eurasian populations: Han Chinese, Tibetans, Uyghur and Europeans. A novel method was used to establish a high-density alignment across all of the faces, allowing facial diversity to be examined at an unprecedented resolution. These data exhibit signatures of population structure and history. However, among-population differentiation was higher for soft-tissue facial form than for genome-wide genetic loci, and high-resolution analyses reveal that the nose, brow area and cheekbones exhibit particularly strong signals of differentiation (Qst estimates: 0.3-0.8) between Europeans and Han Chinese. Our results suggest that local adaptation and/or sexual selection have been important in shaping human soft-tissue facial morphology.

Perceptions of human attractiveness comprising face and voice cues

In human mate choice, sexually dimorphic faces and voices comprise hormone-mediated cues that purportedly develop as an indicator of mate quality or the ability to compete with same-sex rivals. If preferences for faces communicate the same biologically relevant information as do voices, then ratings of these cues should correlate. Sixty participants (30 male and 30 female) rated a series of opposite-sex faces, voices, and faces together with voices for attractiveness in a repeated measures computer-based experiment. The effects of face and voice attractiveness on face-voice compound stimuli were analyzed using a multilevel model. Faces contributed proportionally more than voices to ratings of face-voice compound attractiveness. Faces and voices positively and independently contributed to the attractiveness of male compound stimuli although there was no significant correlation between their rated attractiveness. A positive interaction and correlation between attractiveness was shown for faces and voices in relation to the attractiveness of female compound stimuli. Rather than providing a better estimate of a single characteristic, male faces and voices may instead communicate independent information that, in turn, provides a female with a better assessment of overall mate quality. Conversely, female faces and voices together provide males with a more accurate assessment of a single dimension of mate quality.

Preferences for facial and vocal masculinity in homosexual men: the role of relationship status, sexual restrictiveness, and self-perceived masculinity

Studies on mate preferences usually examine heterosexual attraction; comparatively little is known about preferences of individuals whose sexuality is aimed at the same sex. We examined preferences of two groups of androphilic individuals--homosexual men and heterosexual women--for male facial and vocal level of masculinity. Facial images of 58 men and vocal recordings of 30 men were rated by 51 heterosexual women and 33 homosexual men for their attractiveness and masculinity--femininity. In both groups of raters, ratings of vocal attractiveness and masculinity were positively correlated, but there was no overall preference for facial masculinity. After splitting raters according to their relationship status, sexual restrictiveness, and self-rated masculinity, we found significant preferences for masculine voices only in single homosexual men and coupled heterosexual women, while a preference for feminine male faces was found in coupled homosexual men. Furthermore, homosexual men describing themselves as relatively masculine significantly preferred masculine voices but also more feminine male faces. Our results demonstrate that conditional mate preferences are not restricted to heterosexual interactions, and homosexual men prefer a mixture of masculine and feminine traits in their potential male partners.

Determinants of Hand Attractiveness—A Study Involving Digitally Manipulated Stimuli

Although attractiveness of the human hand is of significance in the social and mating context, thus far it has attracted little scientific interest. In this study, young women and men were presented with pairs of digitally manipulated images of opposite-sex hands and asked to indicate the hand perceived to be the more attractive in each pair. The hands within a pair differed from one another by a single feature: shape averageness, femininity, finger length, second-to-fourth-digit ratio, or skin smoothness. All these features, with the exception of the digit ratio, were shown to increase hand attractiveness in each sex in both dorsal and ventral views. Skin smoothness was preferred more strongly in female than in male hands. Women also tended to prefer medium degrees of shape femininity and skin smoothness in male hands compared to both high and low levels. Adaptive and non-adaptive (related to perceptual bias) mechanisms underlying these preferences are discussed.