On how the brain decodes vocal cues about speaker confidence

The time course of person perception from voices in the brain

When listeners hear a voice, they rapidly form a complex first impression of who the person behind that voice might be. We characterize how these multivariate first impressions from voices emerge over time across different levels of abstraction using electroencephalography and representational similarity analysis. We find that for eight perceived physical (gender, age, and health), trait (attractiveness, dominance, and trustworthiness), and social characteristics (educatedness and professionalism), representations emerge early (~80 ms after stimulus onset), with voice acoustics contributing to those representations between ~100 ms and 400 ms. While impressions of person characteristics are highly correlated, we can find evidence for highly abstracted, independent representations of individual person characteristics. These abstracted representationse merge gradually over time. That is, representations of physical characteristics (age, gender) arise early (from ~120 ms), while representations of some trait and social characteristics emerge later (~360 ms onward). The findings align with recent theoretical models and shed light on the computations underpinning person perception from voices.

Gender stereotypes and social perception of vocal confidence is mitigated by salience of socio-indexical cues to gender

Introduction

Socio-indexical cues to gender and vocal affect often interact and sometimes lead listeners to make differential judgements of affective intent based on the gender of the speaker. Previous research suggests that rising intonation is a common cue that both women and men produce to communicate lack of confidence, but listeners are more sensitive to this cue when it is produced by women. Some speech perception theories assume that listeners will track conditional statistics of speech and language cues (e.g., frequency of the socio-indexical cues to gender and affect) in their listening and communication environments during speech perception. It is currently less clear if these conditional statistics will impact listener ratings when context varies (e.g., number of talkers).

Methods

To test this, we presented listeners with vocal utterances from one female and one male-pitched voice (single talker condition) or many female/male-pitched voices (4 female voices; 4 female voices pitch-shifted to a male range) to examine how they impacted perceptions of talker confidence.

Results

Results indicated that when one voice was evaluated, listeners defaulted to the gender stereotype that the female voice using rising intonation (a cue to lack of confidence) was less confident than the male-pitched voice (using the same cue). However, in the multi-talker condition, this effect went away and listeners equally rated the confidence of the female and male-pitched voices.

Discussion

Findings support dual process theories of information processing, such that listeners may rely on heuristics when speech perception is devoid of context, but when there are no differentiating qualities across talkers (regardless of gender), listeners may be ideal adapters who focus on only the relevant cues.

Complain like you mean it! How prosody conveys suffering even about innocuous events

When complaining, speakers can use their voice to convey a feeling of pain, even when describing innocuous events. Rapid detection of emotive and identity features of the voice may constrain how the semantic content of complaints is processed, as indexed by N400 and P600 effects evoked by the final, pain-related word. Twenty-six participants listened to statements describing painful and innocuous events expressed in a neutral or complaining voice, produced by ingroup and outgroup accented speakers. Participants evaluated how hurt the speaker felt under EEG monitoring. Principal Component Analysis of Event-Related Potentials from the final word onset demonstrated N400 and P600 increases when complainers described innocuous vs. painful events in a neutral voice, but these effects were altered when utterances were expressed in a complaining voice. Independent of prosody, N400 amplitudes increased for complaints spoken in outgroup vs. ingroup accents. Results demonstrate that prosody and accent constrain the processing of spoken complaints as proposed in a parallel-constraint-satisfaction model.

The Time Course of Person Perception From Voices: A Behavioral Study

Listeners spontaneously form impressions of a person from their voice: Is someone old or young? Trustworthy or untrustworthy? Some studies suggest that these impressions emerge rapidly (e.g., < 400 ms for traits), but it is unclear just how rapidly different impressions can emerge and whether the time courses differ across characteristics. I presented 618 adult listeners with voice recordings ranging from 25 ms to 800 ms in duration and asked them to rate physical (age, sex, health), trait (trustworthiness, dominance, attractiveness), and social (educatedness, poshness, professionalism) characteristics. I then used interrater agreement as an index for impression formation. Impressions of physical characteristics and dominance emerged fastest, showing high agreement after only 25 ms of exposure. In contrast, agreement for trait and social characteristics was initially low to moderate and gradually increased. Such a staggered time course suggests that there could be a temporo-perceptual hierarchy for person perception in which faster impressions could influence later ones.

A Semi-Supervised Speech Deception Detection Algorithm Combining Acoustic Statistical Features and Time-Frequency Two-Dimensional Features

Human lying is influenced by cognitive neural mechanisms in the brain, and conducting research on lie detection in speech can help to reveal the cognitive mechanisms of the human brain. Inappropriate deception detection features can easily lead to dimension disaster and make the generalization ability of the widely used semi-supervised speech deception detection model worse. Because of this, this paper proposes a semi-supervised speech deception detection algorithm combining acoustic statistical features and time-frequency two-dimensional features. Firstly, a hybrid semi-supervised neural network based on a semi-supervised autoencoder network (AE) and a mean-teacher network is established. Secondly, the static artificial statistical features are input into the semi-supervised AE to extract more robust advanced features, and the three-dimensional (3D) mel-spectrum features are input into the mean-teacher network to obtain features rich in time-frequency two-dimensional information. Finally, a consistency regularization method is introduced after feature fusion, effectively reducing the occurrence of over-fitting and improving the generalization ability of the model. This paper carries out experiments on the self-built corpus for deception detection. The experimental results show that the highest recognition accuracy of the algorithm proposed in this paper is 68.62% which is 1.2% higher than the baseline system and effectively improves the detection accuracy.

Segmental and suprasegmental encoding of speaker confidence in Wuxi dialect vowels

Introduction

Wuxi dialect is a variation of Wu dialect spoken in eastern China and is characterized by a rich tonal system. Compared with standard Mandarin speakers, those of Wuxi dialect as their mother tongue can be more efficient in varying vocal cues to encode communicative meanings in speech communication. While literature has demonstrated that speakers encode high vs. low confidence in global prosodic cues at the sentence level, it is unknown how speakers' intended confidence is encoded at a more local, phonetic level. This study aimed to explore the effects of speakers' intended confidence on both prosodic and formant features of vowels in two lexical tones (the flat tone and the contour tone) of Wuxi dialect.

Methods

Words of a single vowel were spoken in confident, unconfident, or neutral tone of voice by native Wuxi dialect speakers using a standard elicitation procedure. Linear-mixed effects modeling and parametric bootstrapping testing were performed.

Results

The results showed that (1) the speakers raised both F1 and F2 in the confident level (compared with the neutral-intending expression). Additionally, F1 can distinguish between the confident and unconfident expressions; (2) Compared with the neutral-intending expression, the speakers raised mean f0, had a greater variation of f0 and prolonged pronunciation time in the unconfident level while they raised mean intensity, had a greater variation of intensity and prolonged pronunciation time in the confident level. (3) The speakers modulated mean f0 and mean intensity to a larger extent on the flat tone than the contour tone to differentiate between levels of confidence in the voice, while they modulated f0 and intensity range more only on the contour tone.

Discussion

These findings shed new light on the mechanisms of segmental and suprasegmental encoding of speaker confidence and lack of confidence at the vowel level, highlighting the interplay of lexical tone and vocal expression in speech communication.

Listen to my feelings! How prosody and accent drive the empathic relevance of complaining speech

Interpersonal communication often involves sharing our feelings with others; complaining, for example, aims to elicit empathy in listeners by vocally expressing a speaker's suffering. Despite the growing neuroscientific interest in the phenomenon of empathy, few have investigated how it is elicited in real time by vocal signals (prosody), and how this might be affected by interpersonal factors, such as a speaker's cultural background (based on their accent). To investigate the neural processes at play when hearing spoken complaints, twenty-six French participants listened to complaining and neutral utterances produced by in-group French and out-group Québécois (i.e., French-Canadian) speakers. Participants rated how hurt the speaker felt while their cerebral activity was monitored with electroencephalography (EEG). Principal Component Analysis of Event-Related Potentials (ERPs) taken at utterance onset showed culture-dependent time courses of emotive prosody processing. The high motivational relevance of ingroup complaints increased the P200 response compared to all other utterance types; in contrast, outgroup complaints selectively elicited an early posterior negativity in the same time window, followed by an increased N400 (due to ongoing effort to derive affective meaning from outgroup voices). Ingroup neutral utterances evoked a late negativity which may reflect re-analysis of emotively less salient, but culturally relevant ingroup speech. Results highlight the time-course of neurocognitive responses that contribute to emotive speech processing for complaints, establishing the critical role of prosody as well as social-relational factors (i.e., cultural identity) on how listeners are likely to "empathize" with a speaker.

Epigenetic mechanisms regulate cue memory underlying discriminative behavior

The burgeoning field of neuroepigenetics has introduced chromatin modification as an important interface between experience and brain function. For example, epigenetic mechanisms like histone acetylation and DNA methylation operate throughout a lifetime to powerfully regulate gene expression in the brain that is required for experiences to be transformed into long-term memories. This review highlights emerging evidence from sensory models of memory that converge on the premise that epigenetic regulation of activity-dependent transcription in the sensory brain facilitates highly precise memory recall. Chromatin modifications may be key for neurophysiological responses to transient sensory cue features experienced in the "here and now" to be recapitulated over the long term. We conclude that the function of epigenetic control of sensory system neuroplasticity is to regulate the amount and type of sensory information retained in long-term memories by regulating neural representations of behaviorally relevant cues that guide behavior. This is of broad importance in the neuroscience field because there are few circumstances in which behavioral acts are devoid of an initiating sensory experience.

Gender stereotypes drive perceptual differences of vocal confidence

One's ability to express confidence is critical to achieve one's goals in a social context-such as commanding respect from others, establishing higher social status, and persuading others. How individuals perceive confidence may be shaped by the socio-indexical cues produced by the speaker. In the current production/perception study, we asked four speakers (two cisgender women/men) to answer trivia questions under three speaking contexts: natural, overconfident, and underconfident (i.e., lack of confidence). An evaluation of the speakers' acoustics indicated that the speakers significantly varied their acoustic cues as a function of speaking context and that the women and men had significantly different acoustic cues. The speakers' answers to the trivia questions in the three contexts (natural, overconfident, underconfident) were then presented to listeners (N = 26) in a social judgment task using a computer mouse-tracking paradigm. Listeners were sensitive to the speakers' acoustic modulations of confidence and differentially interpreted these cues based on the perceived gender of the speaker, thereby impacting listeners' cognition and social decision making. We consider, then, how listeners' social judgments about confidence were impacted by gender stereotypes about women and men from social, heuristic-based processes.

Gentle Stroking Elicits Somatosensory ERP that Differentiates Between Hairy and Glabrous Skin

Here we asked whether, similar to visual and auditory event-related potentials (ERPs), somatosensory ERPs reflect affect. Participants were stroked on hairy or glabrous skin at five stroking velocities (0.5, 1, 3, 10 and 20 cm/s). For stroking of hairy skin, pleasantness ratings related to velocity in an inverted u-shaped manner. ERPs showed a negativity at 400 ms following touch onset over somatosensory cortex contra-lateral to the stimulation site. This negativity, referred to as sN400, was larger for intermediate than for faster and slower velocities and positively predicted pleasantness ratings. For stroking of glabrous skin, pleasantness showed again an inverted u-shaped relation with velocity and, additionally, increased linearly with faster stroking. The sN400 revealed no quadratic effect and instead was larger for faster velocities. Its amplitude failed to significantly predict pleasantness. In sum, as was reported for other senses, a touch’s affective value modulates the somatosensory ERP. Notably, however, this ERP and associated subjective pleasantness dissociate between hairy and glabrous skin underscoring functional differences between the skin with which we typically receive touch and the skin with which we typically reach out to touch.

The neural basis of authenticity recognition in laughter and crying

Deciding whether others' emotions are genuine is essential for successful communication and social relationships. While previous fMRI studies suggested that differentiation between authentic and acted emotional expressions involves higher-order brain areas, the time course of authenticity discrimination is still unknown. To address this gap, we tested the impact of authenticity discrimination on event-related potentials (ERPs) related to emotion, motivational salience, and higher-order cognitive processing (N100, P200 and late positive complex, the LPC), using vocalised non-verbal expressions of sadness (crying) and happiness (laughter) in a 32-participant, within-subject study. Using a repeated measures 2-factor (authenticity, emotion) ANOVA, we show that N100's amplitude was larger in response to authentic than acted vocalisations, particularly in cries, while P200's was larger in response to acted vocalisations, particularly in laughs. We suggest these results point to two different mechanisms: (1) a larger N100 in response to authentic vocalisations is consistent with its link to emotional content and arousal (putatively larger amplitude for genuine emotional expressions); (2) a larger P200 in response to acted ones is in line with evidence relating it to motivational salience (putatively larger for ambiguous emotional expressions). Complementarily, a significant main effect of emotion was found on P200 and LPC amplitudes, in that the two were larger for laughs than cries, regardless of authenticity. Overall, we provide the first electroencephalographic examination of authenticity discrimination and propose that authenticity processing of others' vocalisations is initiated early, along that of their emotional content or category, attesting for its evolutionary relevance for trust and bond formation.

Side- and similarity-biases during confidence conformity

Memory conformity may develop when people are confronted with distinct memories reported by others in social situations and knowingly/unknowingly adhere to these exogenous memories. Earlier research on memory conformity suggests that (1) subjects were more likely to conform to confederate with high confidence; (2) subjects with low confidence on their memory accuracy were more likely to conform, and; (3) this subjective confidence could be adjusted by social manipulations. Nonetheless, it remains unclear how the confidence levels of ours and others may interact and produce a combined effect on our degree of conformity. More importantly, is memory conformity, defined by a complete adoption of the opposite side, the result of a gradual accumulation of subtler changes at the confidence level, i.e., a buildup of confidence conformity? Here, we followed participant’s confidence transformation quantitatively over three confederate sessions in a memory test. After studying a set of human motion videos, participants had to answer simultaneously whether a target or lure video had appeared before by indicating their side (i.e., Yes/No) and their associated confidence rating. Participants were allowed to adjust their responses as they were being shown randomly-generated confederates’ answers and confidence values. Results show that participants indeed demonstrated confidence conformity. Interestingly, they tended to become committed to their side early on and gain confidence gradually over subsequent sessions. This polarizing behaviour may be explained by two kinds of preferences: (1) Participant’s confidence enhancement towards same-sided confederates was greater in magnitude compared to the decrement towards an opposite-sided confederate; and (2) Participants had the most effective confidence boost when the same-sided confederates shared similar, but not considerably different, confidence level to theirs. In other words, humans exhibit side- and similarity-biases during confidence conformity.

Paralinguistic Features Communicated through Voice can Affect Appraisals of Confidence and Evaluative Judgments

This article unpacks the basic mechanisms by which paralinguistic features communicated through the voice can affect evaluative judgments and persuasion. Special emphasis is placed on exploring the rapidly emerging literature on vocal features linked to appraisals of confidence (e.g., vocal pitch, intonation, speech rate, loudness, etc.), and their subsequent impact on information processing and meta-cognitive processes of attitude change. The main goal of this review is to advance understanding of the different psychological processes by which paralinguistic markers of confidence can affect attitude change, specifying the conditions under which they are more likely to operate. In sum, we highlight the importance of considering basic mechanisms of attitude change to predict when and why appraisals of paralinguistic markers of confidence can lead to more or less persuasion.

Cortical processing of speaker politeness: Tracking the dynamic effects of voice tone and politeness markers

Information in the tone of voice alters social impressions and underlying brain activity as listeners evaluate the interpersonal relevance of utterances. Here, we presented requests that expressed politeness distinctions through the voice (polite/rude) and explicit linguistic markers (half of the requests began with Please). Thirty participants performed a social perception task (rating friendliness) while their electroencephalogram was recorded. Behaviorally, vocal politeness strategies had a much stronger influence on the perceived friendliness than the linguistic marker. Event-related potentials revealed rapid effects of (im)polite voices on cortical activity prior to ~300 ms; P200 amplitudes increased for polite versus rude voices, suggesting that the speaker's polite stance was registered as more salient in our task. At later stages, politeness distinctions encoded by the speaker's voice and their use of Please interacted, modulating activity in the N400 (300-500 ms) and late positivity (600-800 ms) time windows. Patterns of results suggest that initial attention deployment to politeness cues is rapidly influenced by the motivational significance of a speaker's voice. At later stages, processes for integrating vocal and lexical information resulted in increased cognitive effort to reevaluate utterances with ambiguous/contradictory cues. The potential influence of social anxiety on the P200 effect is also discussed.

Are foreign-accented speakers that 'incredible'? The impact of the speaker's indexical properties on sentence processing

This study investigated the impact of the speaker's identity generated by the voice on sentence processing. We examined the relation between ERP components associated with the processing of the voice (N100 and P200) from voice onset and those associated with sentence processing (N400 and late positivity) from critical word onset. We presented Dutch native speakers with sentences containing true (and known) information, unknown (but true) information or information violating world knowledge and had them perform a truth evaluation task. Sentences were spoken either in a native or a foreign accent. Truth evaluation judgments were not different for statements spoken by the native-accented and the foreign-accented speakers. Reduced N100 and P200 were observed in response to the foreign speaker's voice compared to the native speaker's. While statements containing unknown information or world knowledge violations generated a larger N400 than true statements in the native condition, they were not significantly different in the foreign condition, suggesting shallower processing of foreign-accented speech. The N100 was a significant predictor for the N400 in that the reduced N100 observed for the foreign speaker compared to the native speaker was related to a smaller N400 effect. These finding suggest that the impression of the speaker that listeners rapidly form from the voice affects semantic processing, which confirms that speaker's identity and language comprehension cannot be dissociated.

Comment: The Next Frontier: Prosody Research Gets Interpersonal

Neurocognitive models (e.g., Schirmer & Kotz, 2006) have helped to characterize how listeners incrementally derive meaning from vocal expressions of emotion in spoken language, what neural mechanisms are involved at different processing stages, and their relative time course. But how can these insights be applied to communicative situations in which prosody serves a predominantly interpersonal function? This comment examines recent data highlighting the dynamic interplay of prosody and language, when vocal attributes serve the sociopragmatic goals of the speaker or reveal interpersonal information that listeners use to construct a mental representation of what is being communicated. Our comment serves as a beacon to researchers interested in how the neurocognitive system “makes sense” of socioemotive aspects of prosody.

Nonverbal auditory communication - Evidence for integrated neural systems for voice signal production and perception

While humans have developed a sophisticated and unique system of verbal auditory communication, they also share a more common and evolutionarily important nonverbal channel of voice signaling with many other mammalian and vertebrate species. This nonverbal communication is mediated and modulated by the acoustic properties of a voice signal, and is a powerful - yet often neglected - means of sending and perceiving socially relevant information. From the viewpoint of dyadic (involving a sender and a signal receiver) voice signal communication, we discuss the integrated neural dynamics in primate nonverbal voice signal production and perception. Most previous neurobiological models of voice communication modelled these neural dynamics from the limited perspective of either voice production or perception, largely disregarding the neural and cognitive commonalities of both functions. Taking a dyadic perspective on nonverbal communication, however, it turns out that the neural systems for voice production and perception are surprisingly similar. Based on the interdependence of both production and perception functions in communication, we first propose a re-grouping of the neural mechanisms of communication into auditory, limbic, and paramotor systems, with special consideration for a subsidiary basal-ganglia-centered system. Second, we propose that the similarity in the neural systems involved in voice signal production and perception is the result of the co-evolution of nonverbal voice production and perception systems promoted by their strong interdependence in dyadic interactions.

To believe or not to believe? How voice and accent information in speech alter listener impressions of trust

Our decision to believe what another person says can be influenced by vocally expressed confidence in speech and by whether the speaker–listener are members of the same social group. The dynamic effects of these two information sources on neurocognitive processes that promote believability impressions from vocal cues are unclear. Here, English Canadian listeners were presented personal statements ( She has access to the building) produced in a confident or doubtful voice by speakers of their own dialect (in-group) or speakers from two different “out-groups” (regional or foreign-accented English). Participants rated how believable the speaker is for each statement and event-related potentials (ERPs) were analysed from utterance onset. Believability decisions were modulated by both the speaker’s vocal confidence level and their perceived in-group status. For in-group speakers, ERP effects revealed an early differentiation of vocally expressed confidence (i.e., N100, P200), highlighting the motivational significance of doubtful voices for drawing believability inferences. These early effects on vocal confidence perception were qualitatively different or absent when speakers had an accent; evaluating out-group voices was associated with increased demands on contextual integration and re-analysis of a non-native representation of believability (i.e., increased N400, late negativity response). Accent intelligibility and experience with particular out-group accents each influenced how vocal confidence was processed for out-group speakers. The N100 amplitude was sensitive to out-group attitudes and predicted actual believability decisions for certain out-group speakers. We propose a neurocognitive model in which vocal identity information (social categorization) dynamically influences how vocal expressions are decoded and used to derive social inferences during person perception.

Acoustic correlates of female confidence: A production and comprehension study

In the current study, an interactive approach is used to explore possible contributors to the misattributions listeners make about female talker expression of confidence. To do this, the expression and identification of confidence was evaluated through the evaluation of talker- (e.g., talker knowledge and affective acoustic modulation) and listener-specific factors (e.g., interaction between talker acoustic cues and listener knowledge). Talker and listener contexts were manipulated by implementing a social constraint for talkers and withholding information from listeners. Results indicated that listeners were sensitive to acoustic information produced by the female talkers in this study. However, when world knowledge and acoustics competed, judgments of talker confidence by listeners were less accurate. In fact, acoustic cues to female talker confidence were more accurately used by listeners as a cue to perceived confidence when relevant world knowledge was missing. By targeting speech dynamics between female talkers and both female and male listeners, the current study provides a better understanding of how confidence is realized acoustically and, perhaps more importantly, how those cues may be interpreted/misinterpreted by listeners.

Neural bases of social communicative intentions in speech

Our ability to understand others’ communicative intentions in speech is key to successful social interaction. Indeed, misunderstanding an ‘excuse me’ as apology, while meant as criticism, may have important consequences. Recent behavioural studies have provided evidence that prosody, that is, vocal tone, is an important indicator for speakers’ intentions. Using a novel audio-morphing paradigm, the present functional magnetic resonance imaging study examined the neurocognitive mechanisms that allow listeners to ‘read’ speakers’ intents from vocal prosodic patterns. Participants categorized prosodic expressions that gradually varied in their acoustics between criticism, doubt, and suggestion. Categorizing typical exemplars of the three intentions induced activations along the ventral auditory stream, complemented by amygdala and mentalizing system. These findings likely depict the stepwise conversion of external perceptual information into abstract prosodic categories and internal social semantic concepts, including the speaker’s mental state. Ambiguous tokens, in turn, involved cingulo-opercular areas known to assist decision-making in case of conflicting cues. Auditory and decision-making processes were flexibly coupled with the amygdala, depending on prosodic typicality, indicating enhanced categorization efficiency of overtly relevant, meaningful prosodic signals. Altogether, the results point to a model in which auditory prosodic categorization and socio-inferential conceptualization cooperate to translate perceived vocal tone into a coherent representation of the speaker’s intent.

Judgements of a speaker's personality are correlated across differing content and stimulus type

It has previously been shown that first impressions of a speaker's personality, whether accurate or not, can be judged from short utterances of vowels and greetings, as well as from prolonged sentences and readings of complex paragraphs. From these studies, it is established that listeners' judgements are highly consistent with one another, suggesting that different people judge personality traits in a similar fashion, with three key personality traits being related to measures of valence (associated with trustworthiness), dominance, and attractiveness. Yet, particularly in voice perception, limited research has established the reliability of such personality judgements across stimulus types of varying lengths. Here we investigate whether first impressions of trustworthiness, dominance, and attractiveness of novel speakers are related when a judgement is made on hearing both one word and one sentence from the same speaker. Secondly, we test whether what is said, thus adjusting content, influences the stability of personality ratings. 60 Scottish voices (30 females) were recorded reading two texts: one of ambiguous content and one with socially-relevant content. One word (~500 ms) and one sentence (~3000 ms) were extracted from each recording for each speaker. 181 participants (138 females) rated either male or female voices across both content conditions (ambiguous, socially-relevant) and both stimulus types (word, sentence) for one of the three personality traits (trustworthiness, dominance, attractiveness). Pearson correlations showed personality ratings between words and sentences were strongly correlated, with no significant influence of content. In short, when establishing an impression of a novel speaker, judgments of three key personality traits are highly related whether you hear one word or one sentence, irrespective of what they are saying. This finding is consistent with initial personality judgments serving as elucidators of approach or avoidance behaviour, without modulation by time or content. All data and sounds are available on OSF (osf.io/s3cxy).

The right touch: Stroking of CT-innervated skin promotes vocal emotion processing

Research has revealed a special mechanoreceptor, called C-tactile (CT) afferent, that is situated in hairy skin and that seems relevant for the processing of social touch. We pursued a possible role of this receptor in the perception of other social signals such as a person’s voice. Participants completed three sessions in which they heard surprised and neutral vocal and nonvocal sounds and detected rare sound repetitions. In a given session, participants received no touch or soft brushstrokes to the arm (CT innervated) or palm (CT free). Event-related potentials elicited to sounds revealed that stroking to the arm facilitated the integration of vocal and emotional information. The late positive potential was greater for surprised vocal relative to neutral vocal and nonvocal sounds, and this effect was greater for arm touch relative to both palm touch and no touch. Together, these results indicate that stroking to the arm facilitates the allocation of processing resources to emotional voices, thus supporting the possibility that CT stimulation benefits social perception cross-modally.

ERP correlates of motivating voices: quality of motivation and time-course matters

Here, we conducted the first study to explore how motivations expressed through speech are processed in real-time. Participants listened to sentences spoken in two types of well-studied motivational tones (autonomy-supportive and controlling), or a neutral tone of voice. To examine this, listeners were presented with sentences that either signaled motivations through prosody (tone of voice) and words simultaneously (e.g. ‘You absolutely have to do it my way’ spoken in a controlling tone of voice), or lacked motivationally biasing words (e.g. ‘Why don’t we meet again tomorrow’ spoken in a motivational tone of voice). Event-related brain potentials (ERPs) in response to motivations conveyed through words and prosody showed that listeners rapidly distinguished between motivations and neutral forms of communication as shown in enhanced P2 amplitudes in response to motivational when compared with neutral speech. This early detection mechanism is argued to help determine the importance of incoming information. Once assessed, motivational language is continuously monitored and thoroughly evaluated. When compared with neutral speech, listening to controlling (but not autonomy-supportive) speech led to enhanced late potential ERP mean amplitudes, suggesting that listeners are particularly attuned to controlling messages. The importance of controlling motivation for listeners is mirrored in effects observed for motivations expressed through prosody only. Here, an early rapid appraisal, as reflected in enhanced P2 amplitudes, is only found for sentences spoken in controlling (but not autonomy-supportive) prosody. Once identified as sounding pressuring, the message seems to be preferentially processed, as shown by enhanced late potential amplitudes in response to controlling prosody. Taken together, results suggest that motivational and neutral language are differentially processed; further, the data suggest that listening to cues signaling pressure and control cannot be ignored and lead to preferential, and more in-depth processing mechanisms.

Neural systems for evaluating speaker (Un)believability

Our voice provides salient cues about how confident we sound, which promotes inferences about how believable we are. However, the neural mechanisms involved in these social inferences are largely unknown. Employing functional magnetic resonance imaging, we examined the brain networks and individual differences underlying the evaluation of speaker believability from vocal expressions. Participants (n = 26) listened to statements produced in a confident, unconfident, or "prosodically unmarked" (neutral) voice, and judged how believable the speaker was on a 4-point scale. We found frontal-temporal networks were activated for different levels of confidence, with the left superior and inferior frontal gyrus more activated for confident statements, the right superior temporal gyrus for unconfident expressions, and bilateral cerebellum for statements in a neutral voice. Based on listener's believability judgment, we observed increased activation in the right superior parietal lobule (SPL) associated with higher believability, while increased left posterior central gyrus (PoCG) was associated with less believability. A psychophysiological interaction analysis found that the anterior cingulate cortex and bilateral caudate were connected to the right SPL when higher believability judgments were made, while supplementary motor area was connected with the left PoCG when lower believability judgments were made. Personal characteristics, such as interpersonal reactivity and the individual tendency to trust others, modulated the brain activations and the functional connectivity when making believability judgments. In sum, our data pinpoint neural mechanisms that are involved when inferring one's believability from a speaker's voice and establish ways that these mechanisms are modulated by individual characteristics of a listener. Hum Brain Mapp 38:3732-3749, 2017. © 2017 Wiley Periodicals, Inc.

Emotion Perception from Face, Voice, and Touch: Comparisons and Convergence

Historically, research on emotion perception has focused on facial expressions, and findings from this modality have come to dominate our thinking about other modalities. Here we examine emotion perception through a wider lens by comparing facial with vocal and tactile processing. We review stimulus characteristics and ensuing behavioral and brain responses and show that audition and touch do not simply duplicate visual mechanisms. Each modality provides a distinct input channel and engages partly nonoverlapping neuroanatomical systems with different processing specializations (e.g., specific emotions versus affect). Moreover, processing of signals across the different modalities converges, first into multi- and later into amodal representations that enable holistic emotion judgments.

Unacquainted callers can predict which citizens will vote over and above citizens' stated self-predictions

People are regularly asked to report on their likelihoods of carrying out consequential future behaviors, including complying with medical advice, completing educational assignments, and voting in upcoming elections. Despite these stated self-predictions being notoriously unreliable, they are used to inform many strategic decisions. We report two studies examining stated self-prediction about whether citizens will vote. We find that most self-predicted voters do not actually vote despite saying they will, and that campaign callers can discern which self-predicted voters will not actually vote. In study 1 (n = 4,463), self-predicted voters rated by callers as "100% likely to vote" were 2 times more likely to actually vote than those rated unlikely to vote. Study 2 (n = 3,064) replicated this finding and further demonstrated that callers' prediction accuracy was mediated by citizens' nonverbal signals of uncertainty and deception. Strangers can use nonverbal signals to improve predictions of follow through on self-reported intentions-an insight of potential value for politics, medicine, and education.

Neural responses towards a speaker's feeling of (un)knowing

During interpersonal communication, listeners must rapidly evaluate verbal and vocal cues to arrive at an integrated meaning about the utterance and about the speaker, including a representation of the speaker's 'feeling of knowing' (i.e., how confident they are in relation to the utterance). In this study, we investigated the time course and neural responses underlying a listener's ability to evaluate speaker confidence from combined verbal and vocal cues. We recorded real-time brain responses as listeners judged statements conveying three levels of confidence with the speaker's voice (confident, close-to-confident, unconfident), which were preceded by meaning-congruent lexical phrases (e.g. I am positive, Most likely, Perhaps). Event-related potentials to utterances with combined lexical and vocal cues about speaker confidence were compared to responses elicited by utterances without the verbal phrase in a previous study (Jiang and Pell, 2015). Utterances with combined cues about speaker confidence elicited reduced, N1, P2 and N400 responses when compared to corresponding utterances without the phrase. When compared to confident statements, close-to-confident and unconfident expressions elicited reduced N1 and P2 responses and a late positivity from 900 to 1250 ms; unconfident and close-to-confident expressions were differentiated later in the 1250-1600 ms time window. The effect of lexical phrases on confidence processing differed for male and female participants, with evidence that female listeners incorporated information from the verbal and vocal channels in a distinct manner. Individual differences in trait empathy and trait anxiety also moderated neural responses during confidence processing. Our findings showcase the cognitive processing mechanisms and individual factors governing how we infer a speaker's mental (knowledge) state from the speech signal.

Who is respectful? Effects of social context and individual empathic ability on ambiguity resolution during utterance comprehension

Verbal communication is often ambiguous. By employing the event-related potential (ERP) technique, this study investigated how a comprehender resolves referential ambiguity by using information concerning the social status of communicators. Participants read a conversational scenario which included a minimal conversational context describing a speaker and two other persons of the same or different social status and a directly quoted utterance. A singular, second-person pronoun in the respectful form (nin/nin-de in Chinese) in the utterance could be ambiguous with respect to which of the two persons was the addressee (the “Ambiguous condition”). Alternatively, the pronoun was not ambiguous either because one of the two persons was of higher social status and hence should be the addressee according to social convention (the “Status condition”) or because a word referring to the status of a person was additionally inserted before the pronoun to help indicate the referent of the pronoun (the “Referent condition”). Results showed that the perceived ambiguity decreased over the Ambiguous, Status, and Referent conditions. Electrophysiologically, the pronoun elicited an increased N400 in the Referent than in the Status and the Ambiguous conditions, reflecting an increased integration demand due to the necessity of linking the pronoun to both its antecedent and the status word. Relative to the Referent condition, a late, sustained positivity was elicited for the Status condition starting from 600 ms, while a more delayed, anterior negativity was elicited for the Ambiguous condition. Moreover, the N400 effect was modulated by individuals' sensitivity to the social status information, while the late positivity effect was modulated by individuals' empathic ability. These findings highlight the neurocognitive flexibility of contextual bias in referential processing during utterance comprehension.

Preferential decoding of emotion from human non-linguistic vocalizations versus speech prosody

This study used event-related brain potentials (ERPs) to compare the time course of emotion processing from non-linguistic vocalizations versus speech prosody, to test whether vocalizations are treated preferentially by the neurocognitive system. Participants passively listened to vocalizations or pseudo-utterances conveying anger, sadness, or happiness as the EEG was recorded. Simultaneous effects of vocal expression type and emotion were analyzed for three ERP components (N100, P200, late positive component). Emotional vocalizations and speech were differentiated very early (N100) and vocalizations elicited stronger, earlier, and more differentiated P200 responses than speech. At later stages (450-700ms), anger vocalizations evoked a stronger late positivity (LPC) than other vocal expressions, which was similar but delayed for angry speech. Individuals with high trait anxiety exhibited early, heightened sensitivity to vocal emotions (particularly vocalizations). These data provide new neurophysiological evidence that vocalizations, as evolutionarily primitive signals, are accorded precedence over speech-embedded emotions in the human voice.