Globally, songs and instrumental melodies are slower and higher and use more stable pitches than speech: A Registered Report

Both music and language are found in all known human societies, yet no studies have compared similarities and differences between song, speech, and instrumental music on a global scale. In this Registered Report, we analyzed two global datasets: (i) 300 annotated audio recordings representing matched sets of traditional songs, recited lyrics, conversational speech, and instrumental melodies from our 75 coauthors speaking 55 languages; and (ii) 418 previously published adult-directed song and speech recordings from 209 individuals speaking 16 languages. Of our six preregistered predictions, five were strongly supported: Relative to speech, songs use (i) higher pitch, (ii) slower temporal rate, and (iii) more stable pitches, while both songs and speech used similar (iv) pitch interval size and (v) timbral brightness. Exploratory analyses suggest that features vary along a "musi-linguistic" continuum when including instrumental melodies and recited lyrics. Our study provides strong empirical evidence of cross-cultural regularities in music and speech.

Sounds like a fight: listeners can infer behavioural contexts from spontaneous nonverbal vocalisations

When we hear another person laugh or scream, can we tell the kind of situation they are in - for example, whether they are playing or fighting? Nonverbal expressions are theorised to vary systematically across behavioural contexts. Perceivers might be sensitive to these putative systematic mappings and thereby correctly infer contexts from others' vocalisations. Here, in two pre-registered experiments, we test the prediction that listeners can accurately deduce production contexts (e.g. being tickled, discovering threat) from spontaneous nonverbal vocalisations, like sighs and grunts. In Experiment 1, listeners (total n = 3120) matched 200 nonverbal vocalisations to one of 10 contexts using yes/no response options. Using signal detection analysis, we show that listeners were accurate at matching vocalisations to nine of the contexts. In Experiment 2, listeners (n = 337) categorised the production contexts by selecting from 10 response options in a forced-choice task. By analysing unbiased hit rates, we show that participants categorised all 10 contexts at better-than-chance levels. Together, these results demonstrate that perceivers can infer contexts from nonverbal vocalisations at rates that exceed that of random selection, suggesting that listeners are sensitive to systematic mappings between acoustic structures in vocalisations and behavioural contexts.

Loosening the leash: The unique emotional canvas of human screams

We use screams to explore ideas presented in the target article. Evolving first in animals as a response to predation, screams reveal more complex social use in nonhuman primates and, in humans, uniquely, are associated with a much greater variety of emotional contexts including fear, anger, surprise, and happiness. This expansion, and the potential for manipulation, promotes listener social vigilance.

Cerebral Activity in Female Baboons (Papio anubis) During the Perception of Conspecific and Heterospecific Agonistic Vocalizations: a Functional Near Infrared Spectroscopy Study

The "voice areas" in the superior temporal cortex have been identified in both humans and non-human primates as selective to conspecific vocalizations only (i.e., expressed by members of our own species), suggesting its old evolutionary roots across the primate lineage. With respect to non-human primate species, it remains unclear whether the listening of vocal emotions from conspecifics leads to similar or different cerebral activations when compared to heterospecific calls (i.e., expressed by another primate species) triggered by the same emotion. Using a neuroimaging technique rarely employed in monkeys so far, functional Near Infrared Spectroscopy, the present study investigated in three lightly anesthetized female baboons (Papio anubis), temporal cortex activities during exposure to agonistic vocalizations from conspecifics and from other primates (chimpanzees-Pan troglodytes), and energy matched white noises in order to control for this low-level acoustic feature. Permutation test analyses on the extracted OxyHemoglobin signal revealed great inter-individual differences on how conspecific and heterospecific vocal stimuli were processed in baboon brains with a cortical response recorded either in the right or the left temporal cortex. No difference was found between emotional vocalizations and their energy-matched white noises. Despite the phylogenetic gap between Homo sapiens and African monkeys, modern humans and baboons both showed a highly heterogeneous brain process for the perception of vocal and emotional stimuli. The results of this study do not exclude that old evolutionary mechanisms for vocal emotional processing may be shared and inherited from our common ancestor.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s42761-022-00164-z.

Laughter and culture

Like most human non-verbal vocalizations, laughter is produced by speakers of all languages, across all known societies. But despite this obvious fact (or perhaps because of it), there is little comparative research examining the structural and functional similarity of laughter across speakers from different cultures. Here, we describe existing research examining (i) the perception of laughter across disparate cultures, (ii) conversation analysis examining how laughter manifests itself during discourse across different languages, and (iii) computational methods developed for automatically detecting laughter in spoken language databases. Together, these three areas of investigation provide clues regarding universals and cultural variations in laughter production and perception, and offer methodological tools that can be useful for future large-scale cross-cultural studies. We conclude by providing suggestions for areas of research and predictions of what we should expect to discover. Overall, we highlight how important questions regarding human vocal communication across cultures can be addressed through the examination of spontaneous and volitional laughter.

This article is part of the theme issue ‘Cracking the laugh code: laughter through the lens of biology, psychology and neuroscience’.

The structure underlying core affect and perceived affective qualities of human vocal bursts

Vocal bursts are non-linguistic affectively-laden sounds with a crucial function in human communication, yet their affective structure is still debated. Studies showed that ratings of valence and arousal follow a V-shaped relationship in several kinds of stimuli: high arousal ratings are more likely to go on a par with very negative or very positive valence. Across two studies, we asked participants to listen to 1,008 vocal bursts and judge both how they felt when listening to the sound (i.e. core affect condition), and how the speaker felt when producing it (i.e. perception of affective quality condition). We show that a V-shaped fit outperforms a linear model in explaining the valence-arousal relationship across conditions and studies, even after equating the number of exemplars across emotion categories. Also, although subjective experience can be significantly predicted using affective quality ratings, core affect scores are significantly lower in arousal, less extreme in valence, more variable between individuals, and less reproducible between studies. Nonetheless, stimuli rated with opposite valence between conditions range from 11% (study 1) to 17% (study 2). Lastly, we demonstrate that ambiguity in valence (i.e. high between-participants variability) explains violations of the V-shape and relates to higher arousal.

How Tone, Intonation and Emotion Shape the Development of Infants’ Fundamental Frequency Perception

Fundamental frequency (ƒ₀), perceived as pitch, is the first and arguably most salient auditory component humans are exposed to since the beginning of life. It carries multiple linguistic (e.g., word meaning) and paralinguistic (e.g., speakers’ emotion) functions in speech and communication. The mappings between these functions and ƒ₀ features vary within a language and differ cross-linguistically. For instance, a rising pitch can be perceived as a question in English but a lexical tone in Mandarin. Such variations mean that infants must learn the specific mappings based on their respective linguistic and social environments. To date, canonical theoretical frameworks and most empirical studies do not view or consider the multi-functionality of ƒ₀, but typically focus on individual functions. More importantly, despite the eventual mastery of ƒ₀ in communication, it is unclear how infants learn to decompose and recognize these overlapping functions carried by ƒ₀. In this paper, we review the symbioses and synergies of the lexical, intonational, and emotional functions that can be carried by ƒ₀ and are being acquired throughout infancy. On the basis of our review, we put forward the Learnability Hypothesis that infants decompose and acquire multiple ƒ₀ functions through native/environmental experiences. Under this hypothesis, we propose representative cases such as the synergy scenario, where infants use visual cues to disambiguate and decompose the different ƒ₀ functions. Further, viable ways to test the scenarios derived from this hypothesis are suggested across auditory and visual modalities. Discovering how infants learn to master the diverse functions carried by ƒ₀ can increase our understanding of linguistic systems, auditory processing and communication functions.

In Search of State and Trait Emotion Markers in Mobile-Sensed Language: Field Study

BACKGROUND

Emotions and mood are important for overall well-being. Therefore, the search for continuous, effortless emotion prediction methods is an important field of study. Mobile sensing provides a promising tool and can capture one of the most telling signs of emotion: language.

OBJECTIVE

The aim of this study is to examine the separate and combined predictive value of mobile-sensed language data sources for detecting both momentary emotional experience as well as global individual differences in emotional traits and depression.

METHODS

In a 2-week experience sampling method study, we collected self-reported emotion ratings and voice recordings 10 times a day, continuous keyboard activity, and trait depression severity. We correlated state and trait emotions and depression and language, distinguishing between speech content (spoken words), speech form (voice acoustics), writing content (written words), and writing form (typing dynamics). We also investigated how well these features predicted state and trait emotions using cross-validation to select features and a hold-out set for validation.

RESULTS

Overall, the reported emotions and mobile-sensed language demonstrated weak correlations. The most significant correlations were found between speech content and state emotions and between speech form and state emotions, ranging up to 0.25. Speech content provided the best predictions for state emotions. None of the trait emotion-language correlations remained significant after correction. Among the emotions studied, valence and happiness displayed the most significant correlations and the highest predictive performance.

CONCLUSIONS

Although using mobile-sensed language as an emotion marker shows some promise, correlations and predictive R² values are low.

Voice emotion recognition by Mandarin-speaking pediatric cochlear implant users in Taiwan

OBJECTIVES

To explore the effects of obligatory lexical tone learning on speech emotion recognition and the cross-culture differences between United States and Taiwan for speech emotion understanding in children with cochlear implant.

METHODS

This cohort study enrolled 60 cochlear-implanted (cCI) Mandarin-speaking, school-aged children who underwent cochlear implantation before 5 years of age and 53 normal-hearing children (cNH) in Taiwan. The emotion recognition and the sensitivity of fundamental frequency (F0) changes for those school-aged cNH and cCI (6-17 years old) were examined in a tertiary referred center.

RESULTS

The mean emotion recognition score of the cNH group was significantly better than the cCI. Female speakers' vocal emotions are more easily to be recognized than male speakers' emotion. There was a significant effect of age at test on voice recognition performance. The average score of cCI with full-spectrum speech was close to the average score of cNH with eight-channel narrowband vocoder speech. The average performance of voice emotion recognition across speakers for cCI could be predicted by their sensitivity to changes in F0.

CONCLUSIONS

Better pitch discrimination ability comes with better voice emotion recognition for Mandarin-speaking cCI. Besides the F0 cues, cCI are likely to adapt their voice emotion recognition by relying more on secondary cues such as intensity and duration. Although cross-culture differences exist for the acoustic features of voice emotion, Mandarin-speaking cCI and their English-speaking cCI peer expressed a positive effect for age at test on emotion recognition, suggesting the learning effect and brain plasticity. Therefore, further device/processor development to improve presentation of pitch information and more rehabilitative efforts are needed to improve the transmission and perception of voice emotion in Mandarin.

LEVEL OF EVIDENCE

3.

Vocal communication across cultures: theoretical and methodological issues

The study of human vocal communication has been conducted primarily in Western, educated, industrialized, rich, democratic (WEIRD) societies. Recently, cross-cultural investigations in several domains of voice research have been expanding into more diverse populations. Theoretically, it is important to understand how universals and cultural variations interact in vocal production and perception, but cross-cultural voice research presents many methodological challenges. Experimental methods typically used in WEIRD societies are often not possible to implement in many populations such as rural, small-scale societies. Moreover, theoretical and methodological issues are often unnecessarily intertwined. Here, I focus on three areas of cross-cultural voice modulation research: (i) vocal signalling of formidability and dominance, (ii) vocal emotions, and (iii) production and perception of infant-directed speech. Research in these specific areas illustrates challenges that apply more generally across the human behavioural sciences but also reveals promise as we develop our understanding of the evolution of human communication. This article is part of the theme issue 'Voice modulation: from origin and mechanism to social impact (Part II)'.

Vocal modulation in human mating and competition

The human voice is dynamic, and people modulate their voices across different social interactions. This article presents a review of the literature examining natural vocal modulation in social contexts relevant to human mating and intrasexual competition. Altering acoustic parameters during speech, particularly pitch, in response to mating and competitive contexts can influence social perception and indicate certain qualities of the speaker. For instance, a lowered voice pitch is often used to exert dominance, display status and compete with rivals. Changes in voice can also serve as a salient medium for signalling a person's attraction to another, and there is evidence to support the notion that attraction and/or romantic interest can be distinguished through vocal tones alone. Individuals can purposely change their vocal behaviour in attempt to sound more attractive and to facilitate courtship success. Several findings also point to the effectiveness of vocal change as a mechanism for communicating relationship status. As future studies continue to explore vocal modulation in the arena of human mating, we will gain a better understanding of how and why vocal modulation varies across social contexts and its impact on receiver psychology. This article is part of the theme issue 'Voice modulation: from origin and mechanism to social impact (Part I)'.

Emotional authenticity modulates affective and social trait inferences from voices

The human voice is a primary tool for verbal and nonverbal communication. Studies on laughter emphasize a distinction between spontaneous laughter, which reflects a genuinely felt emotion, and volitional laughter, associated with more intentional communicative acts. Listeners can reliably differentiate the two. It remains unclear, however, if they can detect authenticity in other vocalizations, and whether authenticity determines the affective and social impressions that we form about others. Here, 137 participants listened to laughs and cries that could be spontaneous or volitional and rated them on authenticity, valence, arousal, trustworthiness and dominance. Bayesian mixed models indicated that listeners detect authenticity similarly well in laughter and crying. Speakers were also perceived to be more trustworthy, and in a higher arousal state, when their laughs and cries were spontaneous. Moreover, spontaneous laughs were evaluated as more positive than volitional ones, and we found that the same acoustic features predicted perceived authenticity and trustworthiness in laughter: high pitch, spectral variability and less voicing. For crying, associations between acoustic features and ratings were less reliable. These findings indicate that emotional authenticity shapes affective and social trait inferences from voices, and that the ability to detect authenticity in vocalizations is not limited to laughter. This article is part of the theme issue 'Voice modulation: from origin and mechanism to social impact (Part I)'.

Comment: The Next Frontier: Prosody Research Gets Interpersonal

Neurocognitive models (e.g., Schirmer & Kotz, 2006) have helped to characterize how listeners incrementally derive meaning from vocal expressions of emotion in spoken language, what neural mechanisms are involved at different processing stages, and their relative time course. But how can these insights be applied to communicative situations in which prosody serves a predominantly interpersonal function? This comment examines recent data highlighting the dynamic interplay of prosody and language, when vocal attributes serve the sociopragmatic goals of the speaker or reveal interpersonal information that listeners use to construct a mental representation of what is being communicated. Our comment serves as a beacon to researchers interested in how the neurocognitive system “makes sense” of socioemotive aspects of prosody.

Comment: Advances in Studying the Vocal Expression of Emotion: Current Contributions and Further Options

I consider the five contributions in this special section as evidence that the research area dealing with the vocal expression of emotion is advancing rapidly, both in terms of the number of pertinent empirical studies and with respect to an ever increasing sophistication of methodology. I provide some suggestions on promising areas for future interdisciplinary research, including work on emotion expression in singing and the potential of vocal symptoms of emotional disorder. As to the popular discussion of the respective role of universality versus language/culture differences, I suggest to move on from exclusively studying the accuracy of recognition in judgment studies to a more differentiated approach adding production aspects, taking into account the multiple vocal and acoustic features that interact to communicate emotion.