Attitudinal prosody: what we know and directions for future study

Fast and friendly: The role of vocal cues in adolescents' responses to and perceptions of peer provocation

Adolescents self-report using different strategies to respond to peer provocation. However, we have a limited understanding of how these responses are behaviorally enacted and perceived by peers. This study examined the extent to which adolescents' self-reported responses to peer provocation (i.e., aggressive, assertive, and withdrawn) predicted how their vocal enactments of standardized responses to peer provocation were perceived by other adolescents. Three vocal cues relevant to the communication of emotional intent-average pitch, average intensity, and speech rate-were explored as moderators of these associations. Adolescent speakers (n = 39; Mage = 12.67; 66.7% girls) completed a self-report measure of how they would choose to respond to scenarios involving peer provocation; they also enacted standardized vocal responses to hypothetical peer provocation scenarios. Recordings of speakers' vocal responses were presented to a separate sample of adolescent listeners (n = 129; Mage = 12.12; 52.7% girls) in an online listening task. Speakers who self-reported greater use of assertive response strategies enacted standardized vocal responses that were rated as significantly friendlier by listeners. Vocal responses enacted with faster speech rates were also rated as significantly friendlier by listeners. Speakers' self-reported use of aggression and withdrawal was not significantly related to listeners' ratings of their standardized vocal responses. These findings suggest that adolescents may be perceived differently by their peers depending on the way in which their response is enacted; specifically, faster speech rate may be perceived as friendlier and thus de-escalate peer conflict. Future studies should consider not only what youth say and/or do when responding to peer provocation but also how they say it.

Affective Prosody and Its Impact on the Neurology of Language, Depression, Memory and Emotions

Based on the seminal publications of Paul Broca and Carl Wernicke who established that aphasic syndromes (disorders of the verbal-linguistic aspects of communication) were predominantly the result of focal left-hemisphere lesions, "language" is traditionally viewed as a lateralized function of the left hemisphere. This, in turn, has diminished and delayed the acceptance that the right hemisphere also has a vital role in language, specifically in modulating affective prosody, which is essential for communication competency and psychosocial well-being. Focal lesions of the right hemisphere may result in disorders of affective prosody (aprosodic syndromes) that are functionally and anatomically analogous to the aphasic syndromes that occur following focal left-hemisphere lesions. This paper will review the deductive research published over the last four decades that has elucidated the neurology of affective prosody which, in turn, has led to a more complete and nuanced understanding of the neurology of language, depression, emotions and memory. In addition, the paper will also present the serendipitous clinical observations (inductive research) and fortuitous inter-disciplinary collaborations that were crucial in guiding and developing the deductive research processes that culminated in the concept that primary emotions and related display behaviors are a lateralized function of the right hemisphere and social emotions, and related display behaviors are a lateralized function of the left hemisphere.

The impacts of aging on the comprehension of affective prosody: A systematic review

Recent clinical reports have suggested a possible decline in the ability to understand emotions in speech (affective prosody comprehension) with aging. The present study aims to further examine the differences in performance between older and younger adults in terms of affective prosody comprehension. Following a recent cognitive model dividing affective prosody comprehension into perceptual and lexico-semantic components, a cognitive approach targeting these components was adopted. The influence of emotions' valence and category on aging performance was also investigated. A systematic review of the literature was carried out using six databases. Twenty-one articles, presenting 25 experiments, were included. All experiments analyzed affective prosody comprehension performance of older versus younger adults. The results confirmed that older adults' performance in identifying emotions in speech was reduced compared to younger adults. The results also brought out the fact that affective prosody comprehension abilities could be modulated by the emotion category but not by the emotional valence. Various theories account for this difference in performance, namely auditory perception, brain aging, and socioemotional selectivity theory suggesting that older people tend to neglect negative emotions. However, the explanation of the underlying deficits of the affective prosody decline is still limited.

The role of prosody in interpreting causality in English discourse

Previous studies have well established that certain causal connectives encode information about the semantic-pragmatic distinction between different types of causal relations such as CAUSE-CONSEQUENCE versus CLAIM-ARGUMENT relations. These "specialized" causal connectives assist listeners in discerning different types of causality. Additionally, research has demonstrated that utterances expressing CLAIM-ARGUMENT relations exhibit distinct prosodic characteristics compared to utterances expressing CAUSE-CONSEQUENCE relations. However, it remains unknown whether the prosodic characteristics of utterances expressing causality can aid listeners in determining the specific type of causality being conveyed. To address this knowledge gap, this study investigates the impact of the prosody, specifically the prosody of the causal connective so in English, on listeners' interpretation of the type of causality expressed. We conducted a perception experiment employing a forced-choice discourse completion task, where the participants were required to select a continuation for each sound clip they heard. The sound clip consisted of factual events followed by the causal connective so. We found that the odds of listeners choosing subjective continuations over objective continuations increased when the connective so at the end of the sound clip was pronounced with subjective causality prosodic features, such as prolonged duration and a concave f0 contour. This finding suggests that the prosody of the connective so plays a role in conveying subjectivity in causality, guiding listeners in interpreting causal relations. In addition, it is important to note that our data revealed individual variation among listeners in their interpretations of prosodic information related to subjective-objective causality contrast.

Aprosodia Following Focal Brain Damage: What's Right and What's Left?

PURPOSE

Hemispheric specialization for the comprehension and expression of linguistic and emotional prosody is typically attributed to the right hemisphere. This study used techniques adapted from meta-analysis to critically examine the strength of existing evidence for hemispheric lateralization of prosody following brain damage.

METHOD

Twenty-one databases were searched for articles published from 1970 to 2020 addressing differences in prosody performance between groups defined by right hemisphere damage and left hemisphere damage. Hedges's g effect sizes were calculated for all possible prosody comparisons. Primary analyses summarize effects for four types: linguistic production, linguistic comprehension, emotion comprehension, and emotion production. Within each primary analysis, Hedges's g values were averaged across comparisons (usually from a single article) based on the same sample of individuals. Secondary analyses explore more specific classifications of comparisons.

RESULTS

Out of the 113 articles investigating comprehension and production of emotional and linguistic prosody, 62 were deemed appropriate for data extraction, but only 21 met inclusion criteria, passed quality reviews, and provided sufficient information for analysis. Evidence from this review illustrates the heterogeneity of research methods and results from studies that have investigated aprosodia. This review provides inconsistent support for selective contribution of the two cerebral hemispheres to prosody comprehension and production; however, the strongest finding suggests that right hemisphere lesions disrupt emotional prosody comprehension more than left hemisphere lesions.

CONCLUSION

This review highlights the impoverished nature of the existing literature; offers suggestions for future research; and highlights relevant clinical implications for the prognostication, evaluation, and treatment of aprosodia.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.20334987.

Functional patterns of neural activation during vocal emotion recognition in youth with and without refractory epilepsy

Epilepsy has been associated with deficits in the social cognitive ability to decode others' nonverbal cues to infer their emotional intent (emotion recognition). Studies have begun to identify potential neural correlates of these deficits, but have focused primarily on one type of nonverbal cue (facial expressions) to the detriment of other crucial social signals that inform the tenor of social interactions (e.g., tone of voice). Less is known about how individuals with epilepsy process these forms of social stimuli, with a particular gap in knowledge about representation of vocal cues in the developing brain. The current study compared vocal emotion recognition skills and functional patterns of neural activation to emotional voices in youth with and without refractory focal epilepsy. We made novel use of inter-subject pattern analysis to determine brain areas in which activation to emotional voices was predictive of epilepsy status. Results indicated that youth with epilepsy were comparatively less able to infer emotional intent in vocal expressions than their typically developing peers. Activation to vocal emotional expressions in regions of the mentalizing and/or default mode network (e.g., right temporo-parietal junction, right hippocampus, right medial prefrontal cortex, among others) differentiated youth with and without epilepsy. These results are consistent with emerging evidence that pediatric epilepsy is associated with altered function in neural networks subserving social cognitive abilities. Our results contribute to ongoing efforts to understand the neural markers of social cognitive deficits in pediatric epilepsy, in order to better tailor and funnel interventions to this group of youth at risk for poor social outcomes.

Longitudinal Change in Neural Response to Vocal Emotion in Adolescence

Adolescence is associated with maturation of function within neural networks supporting the processing of social information. Previous longitudinal studies have established developmental influences on youth’s neural response to facial displays of emotion. Given the increasing recognition of the importance of non-facial cues to social communication, we build on existing work by examining longitudinal change in neural response to vocal expressions of emotion in 8- to 19-year-old youth. Participants completed a vocal emotion recognition task at two timepoints (1 year apart) while undergoing functional magnetic resonance imaging. The right inferior frontal gyrus, right dorsal striatum and right precentral gyrus showed decreases in activation to emotional voices across timepoints, which may reflect focalization of response in these areas. Activation in the dorsomedial prefrontal cortex was positively associated with age but was stable across timepoints. In addition, the slope of change across visits varied as a function of participants’ age in the right temporo-parietal junction (TPJ): this pattern of activation across timepoints and age may reflect ongoing specialization of function across childhood and adolescence. Decreased activation in the striatum and TPJ across timepoints was associated with better emotion recognition accuracy. Findings suggest that specialization of function in social cognitive networks may support the growth of vocal emotion recognition skills across adolescence.

Cortical processing of speaker politeness: Tracking the dynamic effects of voice tone and politeness markers

Information in the tone of voice alters social impressions and underlying brain activity as listeners evaluate the interpersonal relevance of utterances. Here, we presented requests that expressed politeness distinctions through the voice (polite/rude) and explicit linguistic markers (half of the requests began with Please). Thirty participants performed a social perception task (rating friendliness) while their electroencephalogram was recorded. Behaviorally, vocal politeness strategies had a much stronger influence on the perceived friendliness than the linguistic marker. Event-related potentials revealed rapid effects of (im)polite voices on cortical activity prior to ~300 ms; P200 amplitudes increased for polite versus rude voices, suggesting that the speaker's polite stance was registered as more salient in our task. At later stages, politeness distinctions encoded by the speaker's voice and their use of Please interacted, modulating activity in the N400 (300-500 ms) and late positivity (600-800 ms) time windows. Patterns of results suggest that initial attention deployment to politeness cues is rapidly influenced by the motivational significance of a speaker's voice. At later stages, processes for integrating vocal and lexical information resulted in increased cognitive effort to reevaluate utterances with ambiguous/contradictory cues. The potential influence of social anxiety on the P200 effect is also discussed.

A bottom-up model of functional outcome in schizophrenia

Schizophrenia results in poor functional outcomes owing to numerous factors. This study provides the first test of a bottom-up causal model of functional outcome in schizophrenia, using neurocognition, vocal emotional cognition, alexithymia, and negative symptoms as predictors of functional outcome. We investigated a cross-sectional sample of 135 individuals with schizophrenia and 78 controls. Using a series of structural equation modelling analyses, a single pathway was generated among scores from the MATRICS Consensus Cognitive Battery (MCCB), vocal emotion recognition test, Toronto Alexithymia Scale (TAS), Brief Negative Symptom Scale, and the Personal and Social Performance Scale. The scores for each dimension of the MCCB in the schizophrenia group were significantly lower than that in the control group. The recognition accuracy for different emotions (anger, disgust, fear, sadness, surprise, and satire, but not calm was significantly lower in the schizophrenia group than in the control group. Moreover, the scores on the three dimensions of TAS were significantly higher in the schizophrenia group than in the control group. On path analysis modelling, the proposed bottom-up causal model showed a strong fit with the data and formed a single pathway, from neurocognition to vocal emotional cognition, to alexithymia, to negative symptoms, and to poor functional outcomes. The study results strongly support the proposed bottom-up causal model of functional outcome in schizophrenia. The model could be used to better understand the causal factors related to the functional outcome, as well as for the development of intervention strategies to improve functional outcomes in schizophrenia.

Perceived Closeness and Autistic Traits Modulate Interpersonal Vocal Communication

Vocal modulation is a critical component of interpersonal communication. It not only serves as a dynamic and flexible tool for self-expression and linguistic information but also plays a key role in social behavior. Variation in vocal modulation can be driven by individual traits of interlocutors as well as factors relating to the dyad, such as the perceived closeness between interlocutors. In this study we examine both of these sources of variation. At an individual level, we examine the impact of autistic traits, since lack of appropriate vocal modulation has often been associated with Autism Spectrum Disorders. At a dyadic level, we examine the role of perceived closeness between interlocutors on vocal modulation. The study was conducted in three separate samples from India, Italy, and the UK. Articulatory features were extracted from recorded conversations between a total of 85 same-sex pairs of participants, and the articulation space calculated. A larger articulation space corresponds to greater number of spectro-temporal modulations (articulatory variations) sampled by the speaker. Articulation space showed a positive association with interpersonal closeness and a weak negative association with autistic traits. This study thus provides novel insights into individual and dyadic variation that can influence interpersonal vocal communication.

Age-related differences in neural activation and functional connectivity during the processing of vocal prosody in adolescence

The ability to recognize others' emotions based on vocal emotional prosody follows a protracted developmental trajectory during adolescence. However, little is known about the neural mechanisms supporting this maturation. The current study investigated age-related differences in neural activation during a vocal emotion recognition (ER) task. Listeners aged 8 to 19 years old completed the vocal ER task while undergoing functional magnetic resonance imaging. The task of categorizing vocal emotional prosody elicited activation primarily in temporal and frontal areas. Age was associated with a) greater activation in regions in the superior, middle, and inferior frontal gyri, b) greater functional connectivity between the left precentral and inferior frontal gyri and regions in the bilateral insula and temporo-parietal junction, and c) greater fractional anisotropy in the superior longitudinal fasciculus, which connects frontal areas to posterior temporo-parietal regions. Many of these age-related differences in brain activation and connectivity were associated with better performance on the ER task. Increased activation in, and connectivity between, areas typically involved in language processing and social cognition may facilitate the development of vocal ER skills in adolescence.

Tracking Training-Related Plasticity by Combining fMRI and DTI: The Right Hemisphere Ventral Stream Mediates Musical Syntax Processing

As a functional homolog for left-hemispheric syntax processing in language, neuroimaging studies evidenced involvement of right prefrontal regions in musical syntax processing, of which underlying white matter connectivity remains unexplored so far. In the current experiment, we investigated the underlying pathway architecture in subjects with 3 levels of musical expertise. Employing diffusion tensor imaging tractography, departing from seeds from our previous functional magnetic resonance imaging study on music syntax processing in the same participants, we identified a pathway in the right ventral stream that connects the middle temporal lobe with the inferior frontal cortex via the extreme capsule, and corresponds to the left hemisphere ventral stream, classically attributed to syntax processing in language comprehension. Additional morphometric consistency analyses allowed dissociating tract core from more dispersed fiber portions. Musical expertise related to higher tract consistency of the right ventral stream pathway. Specifically, tract consistency in this pathway predicted the sensitivity for musical syntax violations. We conclude that enduring musical practice sculpts ventral stream architecture. Our results suggest that training-related pathway plasticity facilitates the right hemisphere ventral stream information transfer, supporting an improved sound-to-meaning mapping in music.

CLEESE: An open-source audio-transformation toolbox for data-driven experiments in speech and music cognition

Over the past few years, the field of visual social cognition and face processing has been dramatically impacted by a series of data-driven studies employing computer-graphics tools to synthesize arbitrary meaningful facial expressions. In the auditory modality, reverse correlation is traditionally used to characterize sensory processing at the level of spectral or spectro-temporal stimulus properties, but not higher-level cognitive processing of e.g. words, sentences or music, by lack of tools able to manipulate the stimulus dimensions that are relevant for these processes. Here, we present an open-source audio-transformation toolbox, called CLEESE, able to systematically randomize the prosody/melody of existing speech and music recordings. CLEESE works by cutting recordings in small successive time segments (e.g. every successive 100 milliseconds in a spoken utterance), and applying a random parametric transformation of each segment’s pitch, duration or amplitude, using a new Python-language implementation of the phase-vocoder digital audio technique. We present here two applications of the tool to generate stimuli for studying intonation processing of interrogative vs declarative speech, and rhythm processing of sung melodies.

The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures

Central to emotion science is the degree to which categories, such as Awe, or broader affective features, such as Valence, underlie the recognition of emotional expression. To explore the processes by which people recognize emotion from prosody, US and Indian participants were asked to judge the emotion categories or affective features communicated by 2,519 speech samples produced by 100 actors from 5 cultures. With large-scale statistical inference methods, we find that prosody can communicate at least 12 distinct kinds of emotion that are preserved across the 2 cultures. Analyses of the semantic and acoustic structure of the recognition of emotions reveal that emotion categories drive the recognition of emotions more so than affective features, including Valence. In contrast to discrete emotion theories, however, emotion categories are bridged by gradients representing blends of emotions. Our findings, visualized within an interactive map, reveal a complex, high-dimensional space of emotional states recognized cross-culturally in speech prosody.

Mapping 24 emotions conveyed by brief human vocalization

Emotional vocalizations are central to human social life. Recent studies have documented that people recognize at least 13 emotions in brief vocalizations. This capacity emerges early in development, is preserved in some form across cultures, and informs how people respond emotionally to music. What is poorly understood is how emotion recognition from vocalization is structured within what we call a semantic space, the study of which addresses questions critical to the field: How many distinct kinds of emotions can be expressed? Do expressions convey emotion categories or affective appraisals (e.g., valence, arousal)? Is the recognition of emotion expressions discrete or continuous? Guided by a new theoretical approach to emotion taxonomies, we apply large-scale data collection and analysis techniques to judgments of 2,032 emotional vocal bursts produced in laboratory settings (Study 1) and 48 found in the real world (Study 2) by U.S. English speakers (N = 1,105). We find that vocal bursts convey at least 24 distinct kinds of emotion. Emotion categories (sympathy, awe), more so than affective appraisals (including valence and arousal), organize emotion recognition. In contrast to discrete emotion theories, the emotion categories conveyed by vocal bursts are bridged by smooth gradients with continuously varying meaning. We visualize the complex, high-dimensional space of emotion conveyed by brief human vocalization within an online interactive map. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

ERP correlates of motivating voices: quality of motivation and time-course matters

Here, we conducted the first study to explore how motivations expressed through speech are processed in real-time. Participants listened to sentences spoken in two types of well-studied motivational tones (autonomy-supportive and controlling), or a neutral tone of voice. To examine this, listeners were presented with sentences that either signaled motivations through prosody (tone of voice) and words simultaneously (e.g. ‘You absolutely have to do it my way’ spoken in a controlling tone of voice), or lacked motivationally biasing words (e.g. ‘Why don’t we meet again tomorrow’ spoken in a motivational tone of voice). Event-related brain potentials (ERPs) in response to motivations conveyed through words and prosody showed that listeners rapidly distinguished between motivations and neutral forms of communication as shown in enhanced P2 amplitudes in response to motivational when compared with neutral speech. This early detection mechanism is argued to help determine the importance of incoming information. Once assessed, motivational language is continuously monitored and thoroughly evaluated. When compared with neutral speech, listening to controlling (but not autonomy-supportive) speech led to enhanced late potential ERP mean amplitudes, suggesting that listeners are particularly attuned to controlling messages. The importance of controlling motivation for listeners is mirrored in effects observed for motivations expressed through prosody only. Here, an early rapid appraisal, as reflected in enhanced P2 amplitudes, is only found for sentences spoken in controlling (but not autonomy-supportive) prosody. Once identified as sounding pressuring, the message seems to be preferentially processed, as shown by enhanced late potential amplitudes in response to controlling prosody. Taken together, results suggest that motivational and neutral language are differentially processed; further, the data suggest that listening to cues signaling pressure and control cannot be ignored and lead to preferential, and more in-depth processing mechanisms.

Cracking the social code of speech prosody using reverse correlation

In speech, social evaluations of a speaker’s dominance or trustworthiness are conveyed by distinguishing, but little-understood, pitch variations. This work describes how to combine state-of-the-art vocal pitch transformations with the psychophysical technique of reverse correlation and uses this methodology to uncover the prosodic prototypes that govern such social judgments in speech. This finding is of great significance, because the exact shape of these prototypes, and how they vary with sex, age, and culture, is virtually unknown, and because prototypes derived with the method can then be reapplied to arbitrary spoken utterances, thus providing a principled way to modulate personality impressions in speech.

Human listeners excel at forming high-level social representations about each other, even from the briefest of utterances. In particular, pitch is widely recognized as the auditory dimension that conveys most of the information about a speaker’s traits, emotional states, and attitudes. While past research has primarily looked at the influence of mean pitch, almost nothing is known about how intonation patterns, i.e., finely tuned pitch trajectories around the mean, may determine social judgments in speech. Here, we introduce an experimental paradigm that combines state-of-the-art voice transformation algorithms with psychophysical reverse correlation and show that two of the most important dimensions of social judgments, a speaker’s perceived dominance and trustworthiness, are driven by robust and distinguishing pitch trajectories in short utterances like the word “Hello,” which remained remarkably stable whether male or female listeners judged male or female speakers. These findings reveal a unique communicative adaptation that enables listeners to infer social traits regardless of speakers’ physical characteristics, such as sex and mean pitch. By characterizing how any given individual’s mental representations may differ from this generic code, the method introduced here opens avenues to explore dysprosody and social-cognitive deficits in disorders like autism spectrum and schizophrenia. In addition, once derived experimentally, these prototypes can be applied to novel utterances, thus providing a principled way to modulate personality impressions in arbitrary speech signals.

Neural systems for evaluating speaker (Un)believability

Our voice provides salient cues about how confident we sound, which promotes inferences about how believable we are. However, the neural mechanisms involved in these social inferences are largely unknown. Employing functional magnetic resonance imaging, we examined the brain networks and individual differences underlying the evaluation of speaker believability from vocal expressions. Participants (n = 26) listened to statements produced in a confident, unconfident, or "prosodically unmarked" (neutral) voice, and judged how believable the speaker was on a 4-point scale. We found frontal-temporal networks were activated for different levels of confidence, with the left superior and inferior frontal gyrus more activated for confident statements, the right superior temporal gyrus for unconfident expressions, and bilateral cerebellum for statements in a neutral voice. Based on listener's believability judgment, we observed increased activation in the right superior parietal lobule (SPL) associated with higher believability, while increased left posterior central gyrus (PoCG) was associated with less believability. A psychophysiological interaction analysis found that the anterior cingulate cortex and bilateral caudate were connected to the right SPL when higher believability judgments were made, while supplementary motor area was connected with the left PoCG when lower believability judgments were made. Personal characteristics, such as interpersonal reactivity and the individual tendency to trust others, modulated the brain activations and the functional connectivity when making believability judgments. In sum, our data pinpoint neural mechanisms that are involved when inferring one's believability from a speaker's voice and establish ways that these mechanisms are modulated by individual characteristics of a listener. Hum Brain Mapp 38:3732-3749, 2017. © 2017 Wiley Periodicals, Inc.

White matter tracts critical for recognition of sarcasm

Failure to recognize sarcasm can lead to important miscommunications. Few previous studies have identified brain lesions associated with impaired recognition of sarcasm. We tested the hypothesis that percent damage to specific white matter tracts, age, and education together predict accuracy in sarcasm recognition. Using multivariable linear regression, with age, education, and percent damage to each of eight white matter tracts as independent variables, and percent accuracy on sarcasm recognition as the dependent variable, we developed a model for predicting sarcasm recognition. Percent damage to the sagittal stratum had the greatest weight and was the only independent predictor of sarcasm recognition.

What Do You Mean by That?! An Electrophysiological Study of Emotional and Attitudinal Prosody

The use of prosody during verbal communication is pervasive in everyday language and whilst there is a wealth of research examining the prosodic processing of emotional information, much less is known about the prosodic processing of attitudinal information. The current study investigated the online neural processes underlying the prosodic processing of non-verbal emotional and attitudinal components of speech via the analysis of event-related brain potentials related to the processing of anger and sarcasm. To examine these, sentences with prosodic expectancy violations created by cross-splicing a prosodically neutral head (‘he has’) and a prosodically neutral, angry, or sarcastic ending (e.g., ‘a serious face’) were used. Task demands were also manipulated, with participants in one experiment performing prosodic classification and participants in another performing probe-verification. Overall, whilst minor differences were found across the tasks, the results suggest that angry and sarcastic prosodic expectancy violations follow a similar processing time-course underpinned by similar neural resources.