Paralinguistic correlates of conceptual structure

Falling Vocal Intonation Signals Speaker Confidence and Conditionally Boosts Persuasion

People are often advised to project confidence with their bodies and voices to convince others. Prior research has focused on the high and low thinking processes through which vocal confidence signals (e.g., fast speed, falling intonation, low pitch) can influence attitude change. In contrast, this research examines how the vocal confidence of speakers operates under more moderate elaboration levels, revealing that falling intonation only benefits persuasion under certain circumstances. In three experiments, we show that falling (vs. rising) vocal intonation at the ends of sentences can signal speaker confidence. Under moderate elaboration conditions, falling (vs. rising) vocal intonation increased message processing, bolstering the benefit of strong over weak messages, increasing the proportion of message-relevant thoughts, and increasing thought-attitude correspondence. In sum, the present work examined an unstudied role of vocal confidence in guiding persuasion, revealing new processes by which vocal signals increase or fail to increase persuasion.

Listener- Versus Speaker-Oriented Disfluencies in Autistic Adults: Insights From Wearable Eye-Tracking and Skin Conductance Within a Live Face-to-Face Paradigm

PURPOSE

Our study addresses three main questions: (a) Do autistics and neurotypicals produce different patterns of disfluencies, depending on the experimenter's direct versus averted gaze? (b) Are these patterns correlated to gender, skin conductance responses, fixations on the experimenter's face, alexithymia, or social anxiety scores? Lastly, (c) can eye-tracking and electrodermal activity data be used in distinguishing listener- versus speaker-oriented disfluencies?

METHOD

Within a live face-to-face paradigm combining a wearable eye-tracker with electrodermal activity sensors, 80 adults (40 autistics, 40 neurotypicals) defined words in front of an experimenter who was either staring at their eyes (direct gaze condition) or looking elsewhere (averted gaze condition).

RESULTS

Autistics produce less listener-oriented (uh, um) and more speaker-oriented (prolongations, breath) disfluencies than neurotypicals. In both groups, men produce less um than women. Both autistics' and neurotypicals' speech are influenced by whether their interlocutor systematically looks at them in the eyes or not, but their reactions go in opposite directions. Disfluencies seem to primarily be linguistic phenomena as experienced stress, social attention, alexithymia, and social anxiety scores do not influence any of the reported results. Finally, eye-tracking and electrodermal activity data suggest that laughter could be a listener-oriented disfluency.

CONCLUSIONS

This article studies disfluencies in a fine-grained way in autistic and neurotypical adults while controlling for social attention, experienced stress, and experimental condition (direct vs. averted gaze). It adds to current literature by (a) enlightening our knowledge of speech in autism, (b) opening new perspectives on disfluency patterns as important signals in social interaction, (c) addressing theoretical issues on the dichotomy between listener- and speaker-oriented disfluencies, and (d) considering understudied phenomena as potential disfluencies (e.g., laughter, breath).

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.23549550.

Cracking the social code of speech prosody using reverse correlation

In speech, social evaluations of a speaker’s dominance or trustworthiness are conveyed by distinguishing, but little-understood, pitch variations. This work describes how to combine state-of-the-art vocal pitch transformations with the psychophysical technique of reverse correlation and uses this methodology to uncover the prosodic prototypes that govern such social judgments in speech. This finding is of great significance, because the exact shape of these prototypes, and how they vary with sex, age, and culture, is virtually unknown, and because prototypes derived with the method can then be reapplied to arbitrary spoken utterances, thus providing a principled way to modulate personality impressions in speech.

Human listeners excel at forming high-level social representations about each other, even from the briefest of utterances. In particular, pitch is widely recognized as the auditory dimension that conveys most of the information about a speaker’s traits, emotional states, and attitudes. While past research has primarily looked at the influence of mean pitch, almost nothing is known about how intonation patterns, i.e., finely tuned pitch trajectories around the mean, may determine social judgments in speech. Here, we introduce an experimental paradigm that combines state-of-the-art voice transformation algorithms with psychophysical reverse correlation and show that two of the most important dimensions of social judgments, a speaker’s perceived dominance and trustworthiness, are driven by robust and distinguishing pitch trajectories in short utterances like the word “Hello,” which remained remarkably stable whether male or female listeners judged male or female speakers. These findings reveal a unique communicative adaptation that enables listeners to infer social traits regardless of speakers’ physical characteristics, such as sex and mean pitch. By characterizing how any given individual’s mental representations may differ from this generic code, the method introduced here opens avenues to explore dysprosody and social-cognitive deficits in disorders like autism spectrum and schizophrenia. In addition, once derived experimentally, these prototypes can be applied to novel utterances, thus providing a principled way to modulate personality impressions in arbitrary speech signals.

Conflict and metacognitive control: the mismatch-monitoring hypothesis of how others' knowledge states affect recall

Information about others' success in remembering is frequently available. For example, students taking an exam may assess its difficulty by monitoring when others turn in their exams. In two experiments, we investigated how rememberers use this information to guide recall. Participants studied paired associates, some semantically related (and thus easier to retrieve) and some unrelated (and thus harder). During a subsequent cued recall test, participants viewed fictive information about an opponent's accuracy on each item. In Experiment 1, participants responded to each cue once before seeing the opponent's performance and once afterwards. Participants reconsidered their responses least often when the opponent's accuracy matched the item difficulty (easy items the opponent recalled, hard items the opponent forgot) and most often when the opponent's accuracy and the item difficulty mismatched. When participants responded only after seeing the opponent's performance (Experiment 2), the same mismatch conditions that led to reconsideration even produced superior recall. These results suggest that rememberers monitor whether others' knowledge states accord or conflict with their own experience, and that this information shifts how they interrogate their memory and what they recall.

Neural responses towards a speaker's feeling of (un)knowing

During interpersonal communication, listeners must rapidly evaluate verbal and vocal cues to arrive at an integrated meaning about the utterance and about the speaker, including a representation of the speaker's 'feeling of knowing' (i.e., how confident they are in relation to the utterance). In this study, we investigated the time course and neural responses underlying a listener's ability to evaluate speaker confidence from combined verbal and vocal cues. We recorded real-time brain responses as listeners judged statements conveying three levels of confidence with the speaker's voice (confident, close-to-confident, unconfident), which were preceded by meaning-congruent lexical phrases (e.g. I am positive, Most likely, Perhaps). Event-related potentials to utterances with combined lexical and vocal cues about speaker confidence were compared to responses elicited by utterances without the verbal phrase in a previous study (Jiang and Pell, 2015). Utterances with combined cues about speaker confidence elicited reduced, N1, P2 and N400 responses when compared to corresponding utterances without the phrase. When compared to confident statements, close-to-confident and unconfident expressions elicited reduced N1 and P2 responses and a late positivity from 900 to 1250 ms; unconfident and close-to-confident expressions were differentiated later in the 1250-1600 ms time window. The effect of lexical phrases on confidence processing differed for male and female participants, with evidence that female listeners incorporated information from the verbal and vocal channels in a distinct manner. Individual differences in trait empathy and trait anxiety also moderated neural responses during confidence processing. Our findings showcase the cognitive processing mechanisms and individual factors governing how we infer a speaker's mental (knowledge) state from the speech signal.

Moving to the speed of sound: context modulation of the effect of acoustic properties of speech

Suprasegmental acoustic patterns in speech can convey meaningful information and affect listeners' interpretation in various ways, including through systematic analog mapping of message-relevant information onto prosody. We examined whether the effect of analog acoustic variation is governed by the acoustic properties themselves. For example, fast speech may always prime the concept of speed or a faster response. Alternatively, the effect may be modulated by the context-dependent interpretation of those properties; the effect of rate may depend on how listeners construe its meaning in the immediate linguistic or communicative context. In two experiments, participants read short scenarios that implied, or did not imply, urgency. Scenarios were followed by recorded instructions, spoken at varying rates. The results show that speech rate had an effect on listeners' response speed; however, this effect was modulated by discourse context. Speech rate affected response speed following contexts that emphasized speed, but not without such contextual information.

Analysis of speech fluency in Williams syndrome

Williams syndrome (WS) is a neurodevelopmental genetic disorder, often referred as being characterized by dissociation between verbal and non-verbal abilities, although the number of studies disputing this proposal is emerging. Indeed, although they have been traditionally reported as displaying increased speech fluency, this topic has not been fully addressed in research. In previous studies carried out with a small group of individuals with WS, we reported speech breakdowns during conversational and autobiographical narratives suggestive of language difficulties. In the current study, we characterized the speech fluency profile using an ecologically based measure--a narrative task (story generation) was collected from a group of individuals with WS (n = 30) and typically developing group (n = 39) matched in mental age. Oral narratives were elicited using a picture stimulus--the cookie theft picture from Boston Diagnosis Aphasia Test. All narratives were analyzed according to typology and frequency of fluency breakdowns (non-stuttered and stuttered disfluencies). Oral narratives in WS group differed from typically developing group, mainly due to a significant increase in the frequency of disfluencies, particularly in terms of hesitations, repetitions and pauses. This is the first evidence of disfluencies in WS using an ecologically based task (oral narrative task), suggesting that these speech disfluencies may represent a significant marker of language problems in WS.

The disfluent discourse: Effects of filled pauses on recall

We investigated the mechanisms by which fillers, such as uh and um, affect memory for discourse. Participants listened to and attempted to recall recorded passages adapted from Alice's Adventures in Wonderland. The type and location of interruptions were manipulated through digital splicing. In Experiment 1, we tested a processing time account of fillers' effects. While fillers facilitated recall, coughs matched in duration to the fillers impaired recall, suggesting that fillers' benefits cannot be attributed to adding processing time. In Experiment 2, fillers' locations were manipulated based on norming data to be either predictive or non-predictive of upcoming material. Fillers facilitated recall in both cases, inconsistent with an account in which listeners predict upcoming material using past experience with the distribution of fillers. Instead, these results suggest an attentional orienting account in which fillers direct attention to the speech stream but do not always result in specific predictions about upcoming material.

Listeners' comprehension of uptalk in spontaneous speech

Listeners' comprehension of phrase final rising pitch on declarative utterances, or uptalk, was examined to test the hypothesis that prolongations might differentiate conflicting functions of rising pitch. In Experiment 1 we found that listeners rated prolongations as indicating more speaker uncertainty, but that rising pitch was unrelated to ratings. In Experiment 2 we found that prolongations interacted with rising pitch when listeners monitored for words in the subsequent utterance. Words preceded by prolonged uptalk were monitored faster than words preceded by non-prolonged uptalk. In Experiment 3 we found that the interaction between rising pitch and prolongations depended on listeners' beliefs about speakers' mental states. Results support the theory that temporal and situational context are important in determining intonational meaning.

Impaired social cognition 30 years after hemispherectomy for intractable epilepsy: the importance of the right hemisphere in complex social functioning

Clinical research with individuals following hemispherectomy typically quantifies the success of surgical outcomes by focusing primarily on the achievement of seizure control and the preservation of general brain functions, such as movement, sensation, language, and memory. In addition to these outcomes, careful study of individuals following hemispherectomy also has the potential to contribute to our understanding of functional brain asymmetries involving other complex cognitive behaviors. In this study, we report preliminary evidence for the lateralization of social perception. We administered a series of neuropsychological tests that were developed to assess emotional recognition and the formation of social inferences and advanced social cognitive judgments, as they occur in everyday situations, to two adult participants who underwent complete anatomic left- or right-sided hemispherectomy. Our results show that despite a 30-year postsurgical period of recovery and consistent and high levels of family support and social engagement, distinct cognitive profiles are still evident between our right- and left-sided participants. In particular, participant S.M., who underwent an anatomic right hemispherectomy, showed the most severe impairments in identifying negative emotional expressions and conversational exchanges involving lies and sarcasm and in "mentalizing" the intent of others. In contrast, participant J.H., who underwent an anatomic left hemispherectomy was highly skilled interpersonally, despite evident language-related limitations, and showed only mild difficulties when asked to identify emotional expressions involving disgust and anger. These results suggest that the right hemisphere plays a particularly important role in social cognitive functioning and reasoning. Further examination of the extent of social perceptual difficulties prior to and following surgical intervention for epilepsy may guide the development of effective social skills training programs that can improve quality of life beyond seizure control.

Reduced sensitivity to prosodic attitudes in adults with focal right hemisphere brain damage

Although there is a strong link between the right hemisphere and understanding emotional prosody in speech, there are few data on how the right hemisphere is implicated for understanding the emotive "attitudes" of a speaker from prosody. This report describes two experiments which compared how listeners with and without focal right hemisphere damage (RHD) rate speaker attitudes of "confidence" and "politeness" which are signalled in large part by prosodic features of an utterance. The RHD listeners displayed abnormal sensitivity to both the expressed confidence and politeness of speakers, underscoring a major role for the right hemisphere in the processing of emotions and speaker attitudes from prosody, although the source of these deficits may sometimes vary.

Surface features of utterances, credibility judgments, and memory

Information acquired in conversation is often not credible, which makes monitoring its credibility critical. Listeners of conversations often use surface features of utterances, such as pause and intonation, to guide their credibility judgments. In this research, we explore whether listeners' delayed credibility judgments about remembered information are affected by the surface features of the speakers' utterances. In addition, we examine some of the specific factors involved in this issue: (1) how listeners' listening strategies influence their subsequent credibility judgments and (2) how the type of surface features of the utterances influences listeners' ability to make delayed credibility judgments. The results indicate that intonation of the utterances continues to influence listeners' assessment of the credibility of remembered information, that the influences of intonation depend on listening strategies, and that people have difficulty using/remembering pause length when making a delayed credibility judgment. The implications of these findings are discussed.

Judging emotion and attitudes from prosody following brain damage

Research has long indicated a role for the right hemisphere in the decoding of basic emotions from speech prosody, although there are few data on how the right hemisphere is implicated in processes for understanding the emotive "attitudes" of a speaker from prosody. We describe recent clinical studies that compared how well listeners with and without focal right hemisphere damage (RHD) understand speaker attitudes such as "confidence" or "politeness," which are signaled in large part by prosodic features of an utterance. We found that RHD listeners as a group were abnormally sensitive to both the expressed confidence and expressed politeness of speakers, and that these difficulties often correlated with impairments for understanding basic emotions from prosody in many RHD individuals. Our data emphasize a central role for the right hemisphere in the ability to appreciate emotions and speaker attitudes from prosody, although the precise source of these social-pragmatic deficits may arise in different ways in the context of right hemisphere compromise.

Paralinguistic correlates of conceptual structure

How is conceptual knowledge transmitted during conversation? When a speaker refers to an object, the name that the speaker chooses conveys information about category identity. In addition, I propose that a speaker's confidence in a classification can convey information about category structure. Because atypical instances of a category are more difficult to classify than typical instances, when speakers refer to these instances their lack of confidence will manifest itself "paralinguistically"--that is, in the form of hesitations, filled pauses, or rising prosody. These features can help listeners learn by enabling them to differentiate good from bad examples of a category. So that this hypothesis could be evaluated, in a category learning experiment participants learned a set of novel colors from a speaker. When the speaker's paralinguistically expressed confidence was consistent with the underlying category structure, learners acquired the categories more rapidly and showed better category differentiation from the earliest moments of learning. These findings have important implications for theories of conversational coordination and language learning.