Early sensory encoding of affective prosody: Neuromagnetic tomography of emotional category changes

Chunk boundaries disrupt dependency processing in an AG: Reconciling incremental processing and discrete sampling

Language is rooted in our ability to compose: We link words together, fusing their meanings. Links are not limited to neighboring words but often span intervening words. The ability to process these non-adjacent dependencies (NADs) conflicts with the brain's sampling of speech: We consume speech in chunks that are limited in time, containing only a limited number of words. It is unknown how we link words together that belong to separate chunks. Here, we report that we cannot-at least not so well. In our electroencephalography (EEG) study, 37 human listeners learned chunks and dependencies from an artificial grammar (AG) composed of syllables. Multi-syllable chunks to be learned were equal-sized, allowing us to employ a frequency-tagging approach. On top of chunks, syllable streams contained NADs that were either confined to a single chunk or crossed a chunk boundary. Frequency analyses of the EEG revealed a spectral peak at the chunk rate, showing that participants learned the chunks. NADs that cross boundaries were associated with smaller electrophysiological responses than within-chunk NADs. This shows that NADs are processed readily when they are confined to the same chunk, but not as well when crossing a chunk boundary. Our findings help to reconcile the classical notion that language is processed incrementally with recent evidence for discrete perceptual sampling of speech. This has implications for language acquisition and processing as well as for the general view of syntax in human language.

Detecting Emotional Prosody in Real Words: Electrophysiological Evidence From a Modified Multifeature Oddball Paradigm

PURPOSE

Emotional voice conveys important social cues that demand listeners' attention and timely processing. This event-related potential study investigated the feasibility of a multifeature oddball paradigm to examine adult listeners' neural responses to detecting emotional prosody changes in nonrepeating naturally spoken words.

METHOD

Thirty-three adult listeners completed the experiment by passively listening to the words in neutral and three alternating emotions while watching a silent movie. Previous research documented preattentive change-detection electrophysiological responses (e.g., mismatch negativity [MMN], P3a) to emotions carried by fixed syllables or words. Given that the MMN and P3a have also been shown to reflect extraction of abstract regularities over repetitive acoustic patterns, this study employed a multifeature oddball paradigm to compare listeners' MMN and P3a to emotional prosody change from neutral to angry, happy, and sad emotions delivered with hundreds of nonrepeating words in a single recording session.

RESULTS

Both MMN and P3a were successfully elicited by the emotional prosodic change over the varying linguistic context. Angry prosody elicited the strongest MMN compared with happy and sad prosodies. Happy prosody elicited the strongest P3a in the centro-frontal electrodes, and angry prosody elicited the smallest P3a.

CONCLUSIONS

The results demonstrated that listeners were able to extract the acoustic patterns for each emotional prosody category over constantly changing spoken words. The findings confirm the feasibility of the multifeature oddball paradigm in investigating emotional speech processing beyond simple acoustic change detection, which may potentially be applied to pediatric and clinical populations.

Listen to my feelings! How prosody and accent drive the empathic relevance of complaining speech

Interpersonal communication often involves sharing our feelings with others; complaining, for example, aims to elicit empathy in listeners by vocally expressing a speaker's suffering. Despite the growing neuroscientific interest in the phenomenon of empathy, few have investigated how it is elicited in real time by vocal signals (prosody), and how this might be affected by interpersonal factors, such as a speaker's cultural background (based on their accent). To investigate the neural processes at play when hearing spoken complaints, twenty-six French participants listened to complaining and neutral utterances produced by in-group French and out-group Québécois (i.e., French-Canadian) speakers. Participants rated how hurt the speaker felt while their cerebral activity was monitored with electroencephalography (EEG). Principal Component Analysis of Event-Related Potentials (ERPs) taken at utterance onset showed culture-dependent time courses of emotive prosody processing. The high motivational relevance of ingroup complaints increased the P200 response compared to all other utterance types; in contrast, outgroup complaints selectively elicited an early posterior negativity in the same time window, followed by an increased N400 (due to ongoing effort to derive affective meaning from outgroup voices). Ingroup neutral utterances evoked a late negativity which may reflect re-analysis of emotively less salient, but culturally relevant ingroup speech. Results highlight the time-course of neurocognitive responses that contribute to emotive speech processing for complaints, establishing the critical role of prosody as well as social-relational factors (i.e., cultural identity) on how listeners are likely to "empathize" with a speaker.

Lesion loci of impaired affective prosody: A systematic review of evidence from stroke

Affective prosody, or the changes in rate, rhythm, pitch, and loudness that convey emotion, has long been implicated as a function of the right hemisphere (RH), yet there is a dearth of literature identifying the specific neural regions associated with its processing. The current systematic review aimed to evaluate the evidence on affective prosody localization in the RH. One hundred and ninety articles from 1970 to February 2020 investigating affective prosody comprehension and production in patients with focal brain damage were identified via database searches. Eleven articles met inclusion criteria, passed quality reviews, and were analyzed for affective prosody localization. Acute, subacute, and chronic lesions demonstrated similar profile characteristics. Localized right antero-superior (i.e., dorsal stream) regions contributed to affective prosody production impairments, whereas damage to more postero-lateral (i.e., ventral stream) regions resulted in affective prosody comprehension deficits. This review provides support that distinct RH regions are vital for affective prosody comprehension and production, aligning with literature reporting RH activation for affective prosody processing in healthy adults as well. The impact of study design on resulting interpretations is discussed.

A novel beamformer-based imaging of phase-amplitude coupling (BIPAC) unveiling the inter-regional connectivity of emotional prosody processing in women with primary dysmenorrhea

Objective. Neural communication or the interactions of brain regions play a key role in the formation of functional neural networks. A type of neural communication can be measured in the form of phase-amplitude coupling (PAC), which is the coupling between the phase of low-frequency oscillations and the amplitude of high-frequency oscillations. This paper presents a beamformer-based imaging method, beamformer-based imaging of PAC (BIPAC), to quantify the strength of PAC between a seed region and other brain regions.Approach. A dipole is used to model the ensemble of neural activity within a group of nearby neurons and represents a mixture of multiple source components of cortical activity. From ensemble activity at each brain location, the source component with the strongest coupling to the seed activity is extracted, while unrelated components are suppressed to enhance the sensitivity of coupled-source estimation.Main results. In evaluations using simulation data sets, BIPAC proved advantageous with regard to estimation accuracy in source localization, orientation, and coupling strength. BIPAC was also applied to the analysis of magnetoencephalographic signals recorded from women with primary dysmenorrhea in an implicit emotional prosody experiment. In response to negative emotional prosody, auditory areas revealed strong PAC with the ventral auditory stream and occipitoparietal areas in the theta-gamma and alpha-gamma bands, which may respectively indicate the recruitment of auditory sensory memory and attention reorientation. Moreover, patients with more severe pain experience appeared to have stronger coupling between auditory areas and temporoparietal regions.Significance. Our findings indicate that the implicit processing of emotional prosody is altered by menstrual pain experience. The proposed BIPAC is feasible and applicable to imaging inter-regional connectivity based on cross-frequency coupling estimates. The experimental results also demonstrate that BIPAC is capable of revealing autonomous brain processing and neurodynamics, which are more subtle than active and attended task-driven processing.

Supramodal neural networks support top-down processing of social signals

The perception of facial and vocal stimuli is driven by sensory input and cognitive top‐down influences. Important top‐down influences are attentional focus and supramodal social memory representations. The present study investigated the neural networks underlying these top‐down processes and their role in social stimulus classification. In a neuroimaging study with 45 healthy participants, we employed a social adaptation of the Implicit Association Test. Attentional focus was modified via the classification task, which compared two domains of social perception (emotion and gender), using the exactly same stimulus set. Supramodal memory representations were addressed via congruency of the target categories for the classification of auditory and visual social stimuli (voices and faces). Functional magnetic resonance imaging identified attention‐specific and supramodal networks. Emotion classification networks included bilateral anterior insula, pre‐supplementary motor area, and right inferior frontal gyrus. They were pure attention‐driven and independent from stimulus modality or congruency of the target concepts. No neural contribution of supramodal memory representations could be revealed for emotion classification. In contrast, gender classification relied on supramodal memory representations in rostral anterior cingulate and ventromedial prefrontal cortices. In summary, different domains of social perception involve different top‐down processes which take place in clearly distinguishable neural networks.

Hyper-Activated Brain Resting-State Network and Mismatch Negativity Deficit in Schizophrenia With Auditory Verbal Hallucination Revealed by an Event-Related Potential Evidence

Schizophrenia is a holergasia with unclear mechanism and high heterogeneity. Auditory verbal hallucination (AVH) study might help in understanding schizophrenia from the perspective of individual symptoms. This study aimed to investigate the activities of the resting-state networks (RSN) in the electroencephalogram (EEG) and mismatch negativity (MMN) in task-related state of schizophrenia patients with AVH. We recruited 30 schizophrenia patients without any medication for more than 4 weeks (15 AVH patients and 15 Non-AVH patients) and 15 healthy controls. We recorded the EEG data of the participants in the resting-state for 7 min and the event-related potential (ERP) data under an auditory oddball paradigm. In the resting-state EEG network, AVH patients exhibited a higher clustering coefficient than Non-AVH patients and healthy controls on delta and beta bands and a shorter characteristic path length than Non-AVH patients and healthy controls on all frequency bands. For ERP data, AVH patients showed a lower MMN amplitude than healthy controls (p = 0.017) and Non-AVH patients (p = 0.033). What's more, MMN amplitude was positively correlated with clustering coefficient, and negatively correlated with characteristic path length on delta, theta, beta and gamma band in AVH patients. Our results indicate that AVH patients showed a hyper-activity in resting-state and may have impaired higher-order auditory expectations in the task-related state than healthy controls and Non-AVH patients. And it seems reasonable to conclude that the formation of AVH may occupy certain brain resources and compete for brain resources with external auditory stimuli.

Test-Retest Reliability of Mismatch Negativity (MMN) to Emotional Voices

A voice from kin species conveys indispensable social and affective signals with uniquely phylogenetic and ontogenetic standpoints. However, the neural underpinning of emotional voices, beyond low-level acoustic features, activates a processing chain that proceeds from the auditory pathway to the brain structures implicated in cognition and emotion. By using a passive auditory oddball paradigm, which employs emotional voices, this study investigates the test–retest reliability of emotional mismatch negativity (MMN), indicating that the deviants of positively (happily)- and negatively (angrily)-spoken syllables, as compared to neutral standards, can trigger MMN as a response to an automatic discrimination of emotional salience. The neurophysiological estimates of MMN to positive and negative deviants appear to be highly reproducible, irrespective of the subject’s attentional disposition: whether the subjects are set to a condition that involves watching a silent movie or do a working memory task. Specifically, negativity bias is evinced as threatening, relative to positive vocalizations, consistently inducing larger MMN amplitudes, regardless of the day and the time of a day. The present findings provide evidence to support the fact that emotional MMN offers a stable platform to detect subtle changes in current emotional shifts.

Emotional prosodic change detection in autism Spectrum disorder: an electrophysiological investigation in children and adults

Background

Autism spectrum disorder (ASD) is characterized by atypical behaviors in social environments and in reaction to changing events. While this dyad of symptoms is at the core of the pathology along with atypical sensory behaviors, most studies have investigated only one dimension. A focus on the sameness dimension has shown that intolerance to change is related to an atypical pre-attentional detection of irregularity. In the present study, we addressed the same process in response to emotional change in order to evaluate the interplay between alterations of change detection and socio-emotional processing in children and adults with autism.

Methods

Brain responses to neutral and emotional prosodic deviancies (mismatch negativity (MMN) and P3a, reflecting change detection and orientation of attention toward change, respectively) were recorded in children and adults with autism and in controls. Comparison of neutral and emotional conditions allowed distinguishing between general deviancy and emotional deviancy effects. Moreover, brain responses to the same neutral and emotional stimuli were recorded when they were not deviants to evaluate the sensory processing of these vocal stimuli.

Results

In controls, change detection was modulated by prosody: in children, this was characterized by a lateralization of emotional MMN to the right hemisphere, and in adults, by an earlier MMN for emotional deviancy than for neutral deviancy. In ASD, an overall atypical change detection was observed with an earlier MMN and a larger P3a compared to controls suggesting an unusual pre-attentional orientation toward any changes in the auditory environment. Moreover, in children with autism, deviancy detection depicted reduced MMN amplitude. In addition in children with autism, contrary to adults with autism, no modulation of the MMN by prosody was present and sensory processing of both neutral and emotional vocal stimuli appeared atypical.

Conclusions

Overall, change detection remains altered in people with autism. However, differences between children and adults with ASD evidence a trend toward normalization of vocal processing and of the automatic detection of emotion deviancy with age.

Oxytocin facilitates reciprocity in social communication

Synchrony in social groups may confer significant evolutionary advantages by improving group cohesion and social interaction. However, the neurobiological mechanisms translating social synchrony into refined social information transmission between interacting individuals are still elusive. In two successively conducted experiments involving a total of 306 healthy volunteers, we explored the involvement of the neuropeptide oxytocin (OXT) in reciprocal social interaction. First, we show that synchronous social interactions evoke heightened endogenous OXT release in dyadic partners. In a second step, we examined the consequences of elevated OXT concentrations on emotion transmission by intranasally administering synthetic OXT before recording emotional expressions. Intriguingly, our data demonstrate that the subjects’ facial and vocal expressiveness of fear and happiness is enhanced after OXT compared with placebo administration. Collectively, our findings point to a central role of social synchrony in facilitating reciprocal communication between individuals via heightened OXT signaling. Elevated OXT concentrations among synchronized individuals seem to augment the partners’ emotional expressiveness, thereby contributing to improved transmission of emotional information in social communication.

Prosody production networks are modulated by sensory cues and social context

The neurobiology of emotional prosody production is not well investigated. In particular, the effects of cues and social context are not known. The present study sought to differentiate cued from free emotion generation and the effect of social feedback from a human listener. Online speech filtering enabled functional magnetic resonance imaging during prosodic communication in 30 participants. Emotional vocalizations were (i) free, (ii) auditorily cued, (iii) visually cued or (iv) with interactive feedback. In addition to distributed language networks, cued emotions increased activity in auditory and—in case of visual stimuli—visual cortex. Responses were larger in posterior superior temporal gyrus at the right hemisphere and the ventral striatum when participants were listened to and received feedback from the experimenter. Sensory, language and reward networks contributed to prosody production and were modulated by cues and social context. The right posterior superior temporal gyrus is a central hub for communication in social interactions—in particular for interpersonal evaluation of vocal emotions.

Neurophysiological differentiation between preattentive and attentive processing of emotional expressions on French vowels

The present electrophysiological study investigated the processing of emotional prosody by minimizing as much as possible the effect of emotional information conveyed by the lexical-semantic context. Emotionally colored French vowels (i.e., happiness, sadness, fear, and neutral) were presented in a mismatch negativity (MMN) oddball paradigm. Both the MMN, i.e., an event-related potential (ERP) component thought to reflect preattentive change detection, and the P3a, i.e., an ERP marker of involuntary orientation of attention toward deviant stimuli, were significantly modulated by the emotional deviants compared to the neutral ones. Critically, the largest amplitude (MMN, P3a) and the shortest peak latency (MMN) were observed for fear deviants, all other things being equal. Taken together, the present findings lend support to a sequential neurocognitive model of emotion processing (Scherer, 2001) which postulates, among other checks, a first stage of automatic emotion detection (MMN) followed by a second stage of subjective evaluation of the stimulus or event (P3a). Consistently with previous studies, our data suggest that among the six universal emotions, fear could have a special status probably because of its adaptive role in the evolution of the human species.

Emotional Oddball: A Review on Variants, Results, and Mechanisms

It is of the utmost importance for an organism to rapidly detect and react to changes in its environment. The oddball paradigm has repeatedly been used to explore the underlying cognitive and neurophysiological components of change detection. It is also used to investigate the special role of emotional content in perception and attention (emotional oddball paradigm; EOP). In this article, the EOP is systematically reviewed. The EOP is, for instance, used as a tool to address questions as to what degree emotional deviant stimuli trigger orientation reactions, which role the emotional context plays in the processing of deviant information, and how the processing of emotional deviant information differs interindividually (including clinical populations). Two main variants with regard to the emotionality of standards and deviants are defined. Most of the identified EOP studies report EEG data but an overview of behavioral data is also provided in this review. We integrate evidence from 99 EOP experiments and shape the EOP's theoretical background under the consideration of other paradigms’ mechanisms and theories.

Influence of attention on bimodal integration during emotional change decoding: ERP evidence

Recent findings on audiovisual emotional interactions suggest that selective attention affects cross-sensory interaction from an early processing stage. However, the influence of attention manipulation on facial-vocal integration during emotional change perception is still elusive at this point. To address this issue, we asked participants to detect emotional changes conveyed by prosodies (vocal task) or facial expressions (facial task) while facial, vocal, and facial-vocal expressions were presented. At the same time, behavioral responses and electroencephalogram (EEG) were recorded. Behavioral results showed that bimodal emotional changes were detected with shorter response latencies compared to each unimodal condition, suggesting that bimodal emotional cues facilitated emotional change detection. Moreover, while the P3 amplitudes were larger for the bimodal change condition than for the sum of the two unimodal conditions regardless of attention direction, the N1 amplitudes were larger for the bimodal emotional change condition than for the sum of the two unimodal conditions under the attend-voice condition, but not under the attend-face condition. These findings suggest that selective attention modulates facial-vocal integration during emotional change perception in early sensory processing, but not in late cognitive processing stages.

Neural Substrates of Auditory Emotion Recognition Deficits in Schizophrenia

Deficits in auditory emotion recognition (AER) are a core feature of schizophrenia and a key component of social cognitive impairment. AER deficits are tied behaviorally to impaired ability to interpret tonal ("prosodic") features of speech that normally convey emotion, such as modulations in base pitch (F0M) and pitch variability (F0SD). These modulations can be recreated using synthetic frequency modulated (FM) tones that mimic the prosodic contours of specific emotional stimuli. The present study investigates neural mechanisms underlying impaired AER using a combined event-related potential/resting-state functional connectivity (rsfMRI) approach in 84 schizophrenia/schizoaffective disorder patients and 66 healthy comparison subjects. Mismatch negativity (MMN) to FM tones was assessed in 43 patients/36 controls. rsfMRI between auditory cortex and medial temporal (insula) regions was assessed in 55 patients/51 controls. The relationship between AER, MMN to FM tones, and rsfMRI was assessed in the subset who performed all assessments (14 patients, 21 controls). As predicted, patients showed robust reductions in MMN across FM stimulus type (p = 0.005), particularly to modulations in F0M, along with impairments in AER and FM tone discrimination. MMN source analysis indicated dipoles in both auditory cortex and anterior insula, whereas rsfMRI analyses showed reduced auditory-insula connectivity. MMN to FM tones and functional connectivity together accounted for ∼50% of the variance in AER performance across individuals. These findings demonstrate that impaired preattentive processing of tonal information and reduced auditory-insula connectivity are critical determinants of social cognitive dysfunction in schizophrenia, and thus represent key targets for future research and clinical intervention.

SIGNIFICANCE STATEMENT

Schizophrenia patients show deficits in the ability to infer emotion based upon tone of voice [auditory emotion recognition (AER)] that drive impairments in social cognition and global functional outcome. This study evaluated neural substrates of impaired AER in schizophrenia using a combined event-related potential/resting-state fMRI approach. Patients showed impaired mismatch negativity response to emotionally relevant frequency modulated tones along with impaired functional connectivity between auditory and medial temporal (anterior insula) cortex. These deficits contributed in parallel to impaired AER and accounted for ∼50% of variance in AER performance. Overall, these findings demonstrate the importance of both auditory-level dysfunction and impaired auditory/insula connectivity in the pathophysiology of social cognitive dysfunction in schizophrenia.

Detecting Temporal Change in Dynamic Sounds: On the Role of Stimulus Duration, Speed, and Emotion

For dynamic sounds, such as vocal expressions, duration often varies alongside speed. Compared to longer sounds, shorter sounds unfold more quickly. Here, we asked whether listeners implicitly use this confound when representing temporal regularities in their environment. In addition, we explored the role of emotions in this process. Using a mismatch negativity (MMN) paradigm, we asked participants to watch a silent movie while passively listening to a stream of task-irrelevant sounds. In Experiment 1, one surprised and one neutral vocalization were compressed and stretched to create stimuli of 378 and 600 ms duration. Stimuli were presented in four blocks, two of which used surprised and two of which used neutral expressions. In one surprised and one neutral block, short and long stimuli served as standards and deviants, respectively. In the other two blocks, the assignment of standards and deviants was reversed. We observed a climbing MMN-like negativity shortly after deviant onset, which suggests that listeners implicitly track sound speed and detect speed changes. Additionally, this MMN-like effect emerged earlier and was larger for long than short deviants, suggesting greater sensitivity to duration increments or slowing down than to decrements or speeding up. Last, deviance detection was facilitated in surprised relative to neutral blocks, indicating that emotion enhances temporal processing. Experiment 2 was comparable to Experiment 1 with the exception that sounds were spectrally rotated to remove vocal emotional content. This abolished the emotional processing benefit, but preserved the other effects. Together, these results provide insights into listener sensitivity to sound speed and raise the possibility that speed biases duration judgements implicitly in a feed-forward manner. Moreover, this bias may be amplified for duration increments relative to decrements and within an emotional relative to a neutral stimulus context.

Attentional enhancement for positive and negative tones at an early stage of auditory processing

We report an event-related potential (ERP) study based on the hypothesis that valenced (i.e., positive and/or negative) tones are prioritized over neutral ones at an early, perceptual stage of auditory processing. In order to avoid perceptual confounds, we induced valence experimentally during a learning phase by assigning positive, negative, and neutral valences to tone-frequencies in a balanced design. In a subsequent test phase, EEG was recorded while these tones were entirely task-irrelevant. The amplitude of the auditory N1 was increased for valenced compared with neutral tones, indicating enhanced attention. While behavioral results of the learning phase, and both implicit and explicit measures of tone evaluation indicated differentiation between positive and negative valence, there was no such differentiation on the N1 amplitude. Our results suggest that it is the general relevance of the valenced tones that governs early attentional processes.

Neural Processing of Emotional Prosody across the Adult Lifespan

Emotion recognition deficits emerge with the increasing age, in particular, a decline in the identification of sadness. However, little is known about the age-related changes of emotion processing in sensory, affective, and executive brain areas. This functional magnetic resonance imaging (fMRI) study investigated neural correlates of auditory processing of prosody across adult lifespan. Unattended detection of emotional prosody changes was assessed in 21 young (age range: 18-35 years), 19 middle-aged (age range: 36-55 years), and 15 older (age range: 56-75 years) adults. Pseudowords uttered with neutral prosody were standards in an oddball paradigm with angry, sad, happy, and gender deviants (total 20% deviants). Changes in emotional prosody and voice gender elicited bilateral superior temporal gyri (STG) responses reflecting automatic encoding of prosody. At the right STG, responses to sad deviants decreased linearly with age, whereas happy events exhibited a nonlinear relationship. In contrast to behavioral data, no age by sex interaction emerged on the neural networks. The aging decline of emotion processing of prosodic cues emerges already at an early automatic stage of information processing at the level of the auditory cortex. However, top-down modulation may lead to an additional perceptional bias, for example, towards positive stimuli, and may depend on context factors such as the listener's sex.

What Do You Mean by That?! An Electrophysiological Study of Emotional and Attitudinal Prosody

The use of prosody during verbal communication is pervasive in everyday language and whilst there is a wealth of research examining the prosodic processing of emotional information, much less is known about the prosodic processing of attitudinal information. The current study investigated the online neural processes underlying the prosodic processing of non-verbal emotional and attitudinal components of speech via the analysis of event-related brain potentials related to the processing of anger and sarcasm. To examine these, sentences with prosodic expectancy violations created by cross-splicing a prosodically neutral head (‘he has’) and a prosodically neutral, angry, or sarcastic ending (e.g., ‘a serious face’) were used. Task demands were also manipulated, with participants in one experiment performing prosodic classification and participants in another performing probe-verification. Overall, whilst minor differences were found across the tasks, the results suggest that angry and sarcastic prosodic expectancy violations follow a similar processing time-course underpinned by similar neural resources.

The integration of facial and vocal cues during emotional change perception: EEG markers

The ability to detect emotional changes is of primary importance for social living. Though emotional signals are often conveyed by multiple modalities, how emotional changes in vocal and facial modalities integrate into a unified percept has yet to be directly investigated. To address this issue, we asked participants to detect emotional changes delivered by facial, vocal and facial-vocal expressions while behavioral responses and electroencephalogram were recorded. Behavioral results showed that bimodal emotional changes were detected with higher accuracy and shorter response latencies compared with each unimodal condition. Moreover, the detection of emotional change, regardless of modalities, was associated with enhanced amplitudes in the N2 and P3 component, as well as greater theta synchronization. More importantly, the P3 amplitudes and theta synchronization were larger for the bimodal emotional change condition than for the sum of the two unimodal conditions. The superadditive responses in P3 amplitudes and theta synchronization were both positively correlated with the magnitude of the bimodal superadditivity in accuracy. These behavioral and electrophysiological data consistently illustrated an effect of audiovisual integration during the detection of emotional changes, which is most likely mediated by the P3 activity and theta oscillations in brain responses.

The music of language: an ERP investigation of the effects of musical training on emotional prosody processing

Recent studies have demonstrated the positive effects of musical training on the perception of vocally expressed emotion. This study investigated the effects of musical training on event-related potential (ERP) correlates of emotional prosody processing. Fourteen musicians and fourteen control subjects listened to 228 sentences with neutral semantic content, differing in prosody (one third with neutral, one third with happy and one third with angry intonation), with intelligible semantic content (semantic content condition--SCC) and unintelligible semantic content (pure prosody condition--PPC). Reduced P50 amplitude was found in musicians. A difference between SCC and PPC conditions was found in P50 and N100 amplitude in non-musicians only, and in P200 amplitude in musicians only. Furthermore, musicians were more accurate in recognizing angry prosody in PPC sentences. These findings suggest that auditory expertise characterizing extensive musical training may impact different stages of vocal emotional processing.

EEG oscillations reflect task effects for the change detection in vocal emotion

How task focus affects recognition of change in vocal emotion remains in debate. In this study, we investigated the role of task focus for change detection in emotional prosody by measuring changes in event-related electroencephalogram (EEG) power. EEG was recorded for prosodies with and without emotion change while subjects performed emotion change detection task (explicit) and visual probe detection task (implicit). We found that vocal emotion change induced theta event-related synchronization during 100-600 ms regardless of task focus. More importantly, vocal emotion change induced significant beta event-related desynchronization during 400-750 ms under explicit instead of implicit task condition. These findings suggest that the detection of emotional changes is independent of task focus, while the task focus effect in neural processing of vocal emotion change is specific to the integration of emotional deviations.

Anterior insular cortex activity to emotional salience of voices in a passive oddball paradigm

The human voice, which has a pivotal role in communication, is processed in specialized brain regions. Although a general consensus holds that the anterior insular cortex (AIC) plays a critical role in negative emotional experience, previous studies have not observed AIC activation in response to hearing disgust in voices. We used magnetoencephalography to measure the magnetic counterparts of mismatch negativity (MMNm) and P3a (P3am) in healthy adults while the emotionally meaningless syllables dada, spoken as neutral, happy, or disgusted prosodies, along with acoustically matched simple and complex tones, were presented in a passive oddball paradigm. The results revealed that disgusted relative to happy syllables elicited stronger MMNm-related cortical activities in the right AIC and precentral gyrus along with the left posterior insular cortex, supramarginal cortex, transverse temporal cortex, and upper bank of superior temporal cortex. The AIC activity specific to disgusted syllables (corrected p < 0.05) was associated with the hit rate of the emotional categorization task. These findings may clarify the neural correlates of emotional MMNm and lend support to the role of AIC in the processing of emotional salience already at the preattentive level.

Upregulation of the rostral anterior cingulate cortex can alter the perception of emotions: fMRI-based neurofeedback at 3 and 7 T

Recent advances in real-time functional magnetic resonance imaging (rt-fMRI) techniques enable online feedback about momentary brain activity from a localized region of interest. The anterior cingulate cortex (ACC) as a central hub for cognitive and emotional networks and its modulation has been suggested to elicit mood changes. In the presented real-time fMRI neurofeedback experiment at a 3 and a 7 T scanner we enabled participants to regulate ACC activity within one training session. The session consisted of three training runs of 8.5 min where subjects received online feedback about their current ACC activity. Before and after each run we presented emotional prosody. Subjects rated these stimuli according to their emotional valence and arousal, which served as an implicit mood measure. We found increases in ACC activation at 3 T (n = 15) and at 7 T (n = 9) with a higher activation success for the 3 T group. FMRI signal control of the rostral ACC depended on signal quality and predicted a valence bias in the rating of emotional prosody. Real-time fMRI neurofeedback of the ACC is feasible at different magnetic field strengths and can modulate localized ACC activity and emotion perception. It promises non-invasive therapeutic approaches for different psychiatric disorders characterized by impaired self-regulation.

MMN responses during implicit processing of changes in emotional prosody: an ERP study using Chinese pseudo-syllables

In this study, we tested the underlying mechanisms of early emotional prosody perception, especially examined whether change detection in oddball paradigm was caused by emotional category and physical properties. Using implicit oddball paradigms, the current study manipulated the cues for detecting deviant stimuli from standards in three conditions: the simultaneous changes in emotional category and physical properties (EP condition), change in emotional category alone (E condition), and change in physical properties alone (P condition). ERP results revealed that physical property change increased brain responses to deviant stimuli in the EP than in the E condition at early stage 90-160 ms, suggesting that physical property change of emotional sounds can also be detected at the early stage. At the later stage 160-260 ms, the simultaneous and respective changes in emotional category and physical properties were reliably detected, and the sum of the brain responses to the corresponding changes in E and P conditions was equal to the brain responses to the simultaneous changes in EP condition. Source analysis further revealed that stimuli-driven regions (inferior parietal lobule), temporal and frontal cortices were activated at early stage, while only frontal cortices for higher cognitive processing were activated at later stage. These findings suggest that emotional prosody changes in physical properties and emotion category are perceived as domain-general change information in emotional prosody perception.

Mismatch negativity of sad syllables is absent in patients with major depressive disorder

BACKGROUND

Major depressive disorder (MDD) is an important and highly prevalent mental disorder characterized by anhedonia and a lack of interest in everyday activities. Additionally, patients with MDD appear to have deficits in various cognitive abilities. Although a number of studies investigating the central auditory processing of low-level sound features in patients with MDD have demonstrated that this population exhibits impairments in automatic processing, the influence of emotional voice processing has yet to be addressed. To explore the automatic processing of emotional prosodies in patients with MDD, we analyzed the ability to detect automatic changes using event-related potentials (ERPs).

METHOD

This study included 18 patients with MDD and 22 age- and sex-matched healthy controls. Subjects were instructed to watch a silent movie but to ignore the afferent acoustic emotional prosodies presented to both ears while continuous electroencephalographic activity was synchronously recorded. Prosodies included meaningless syllables, such as "dada" spoken with happy, angry, sad, or neutral tones. The mean amplitudes of the ERPs elicited by emotional stimuli and the peak latency of the emotional differential waveforms were analyzed.

RESULTS

The sad MMN was absent in patients with MDD, whereas the happy and angry MMN components were similar across groups. The abnormal sad emotional MMN component was not significantly correlated with the HRSD-17 and HAMA scores, respectively.

CONCLUSION

The data indicate that patients with MDD are impaired in their ability to automatically process sad prosody, whereas their ability to process happy and angry prosodies remains normal. The dysfunctional sad emotion-related MMN in patients with MDD were not correlated with depression symptoms. The blunted MMN of sad prosodies could be considered a trait of MDD.

Activation of auditory cortex by anticipating and hearing emotional sounds: an MEG study

To study how auditory cortical processing is affected by anticipating and hearing of long emotional sounds, we recorded auditory evoked magnetic fields with a whole-scalp MEG device from 15 healthy adults who were listening to emotional or neutral sounds. Pleasant, unpleasant, or neutral sounds, each lasting for 6 s, were played in a random order, preceded by 100-ms cue tones (0.5, 1, or 2 kHz) 2 s before the onset of the sound. The cue tones, indicating the valence of the upcoming emotional sounds, evoked typical transient N100m responses in the auditory cortex. During the rest of the anticipation period (until the beginning of the emotional sound), auditory cortices of both hemispheres generated slow shifts of the same polarity as N100m. During anticipation, the relative strengths of the auditory-cortex signals depended on the upcoming sound: towards the end of the anticipation period the activity became stronger when the subject was anticipating emotional rather than neutral sounds. During the actual emotional and neutral sounds, sustained fields were predominant in the left hemisphere for all sounds. The measured DC MEG signals during both anticipation and hearing of emotional sounds implied that following the cue that indicates the valence of the upcoming sound, the auditory-cortex activity is modulated by the upcoming sound category during the anticipation period.

The sound of feelings: electrophysiological responses to emotional speech in alexithymia

BACKGROUND

Alexithymia is a personality trait characterized by difficulties in the cognitive processing of emotions (cognitive dimension) and in the experience of emotions (affective dimension). Previous research focused mainly on visual emotional processing in the cognitive alexithymia dimension. We investigated the impact of both alexithymia dimensions on electrophysiological responses to emotional speech in 60 female subjects.

METHODOLOGY

During unattended processing, subjects watched a movie while an emotional prosody oddball paradigm was presented in the background. During attended processing, subjects detected deviants in emotional prosody. The cognitive alexithymia dimension was associated with a left-hemisphere bias during early stages of unattended emotional speech processing, and with generally reduced amplitudes of the late P3 component during attended processing. In contrast, the affective dimension did not modulate unattended emotional prosody perception, but was associated with reduced P3 amplitudes during attended processing particularly to emotional prosody spoken in high intensity.

CONCLUSIONS

Our results provide evidence for a dissociable impact of the two alexithymia dimensions on electrophysiological responses during the attended and unattended processing of emotional prosody. The observed electrophysiological modulations are indicative of a reduced sensitivity to the emotional qualities of speech, which may be a contributing factor to problems in interpersonal communication associated with alexithymia.

It's special the way you say it: an ERP investigation on the temporal dynamics of two types of prosody

Sentence prosody is long known to serve both linguistic functions (e.g. to differentiate between questions and statements) and emotional functions (e.g. to detect the emotional state of a speaker). These different functions of prosodic information need to be encoded rapidly during sentence comprehension to ensure successful speech communication. However, systematic investigations of the comparative nature of these two functions, i.e. are the two functions of prosody independent or interdependent, are sparse. The question at hand is whether the two prosodic functions engage a similar neural network and run a similar time-course or not. To this aim we investigated whether emotional and linguistic prosody are processed independently or dependently in an event-related brain potential (ERP) experiment. We merged a prosodically neutral head of a sentence to a second half of a sentence that differed in emotional and/or linguistic prosody. In a within-subjects design, two tasks were administered: in the "emotion task", participants judged whether the sentence that they had just heard was spoken in a neutral tone of voice or not (emotional task); in the "linguistic task", participants decided whether the sentence was a declarative sentence or not. As predicted, the previously reported prosodic expectancy positivity (PEP) was elicited by linguistic and emotional prosodic expectancy violations. However, the latency and distribution of the ERP component differed: whilst responses to emotional prosodic expectancy violations were elicited shortly after an expectancy violation (∼470 ms post splicing-point) and most prominently at posterior electrode-sites, the positivity in response to linguistic prosody had a later onset (∼620 ms post splicing-point) with a more frontal distribution. Interestingly, responses to combined (linguistic and emotional) expectancy violations resulted in a broadly distributed positivity with an onset of ∼170 ms post expectancy violation. These effects were found irrespective of the task setting. Given the differences in latency and distribution, we conclude that the processing of emotional and linguistic prosody relies at least partly on differing neural mechanisms and that emotional prosodic aspects of language are processed in a prioritized processing stream.

Seeing emotion with your ears: emotional prosody implicitly guides visual attention to faces

Interpersonal communication involves the processing of multimodal emotional cues, particularly facial expressions (visual modality) and emotional speech prosody (auditory modality) which can interact during information processing. Here, we investigated whether the implicit processing of emotional prosody systematically influences gaze behavior to facial expressions of emotion. We analyzed the eye movements of 31 participants as they scanned a visual array of four emotional faces portraying fear, anger, happiness, and neutrality, while listening to an emotionally-inflected pseudo-utterance (Someone migged the pazing) uttered in a congruent or incongruent tone. Participants heard the emotional utterance during the first 1250 milliseconds of a five-second visual array and then performed an immediate recall decision about the face they had just seen. The frequency and duration of first saccades and of total looks in three temporal windows ([0–1250 ms], [1250–2500 ms], [2500–5000 ms]) were analyzed according to the emotional content of faces and voices. Results showed that participants looked longer and more frequently at faces that matched the prosody in all three time windows (emotion congruency effect), although this effect was often emotion-specific (with greatest effects for fear). Effects of prosody on visual attention to faces persisted over time and could be detected long after the auditory information was no longer present. These data imply that emotional prosody is processed automatically during communication and that these cues play a critical role in how humans respond to related visual cues in the environment, such as facial expressions.

Supramodal representation of emotions

Supramodal representation of emotion and its neural substrates have recently attracted attention as a marker of social cognition. However, the question whether perceptual integration of facial and vocal emotions takes place in primary sensory areas, multimodal cortices, or in affective structures remains unanswered yet. Using novel computer-generated stimuli, we combined emotional faces and voices in congruent and incongruent ways and assessed functional brain data (fMRI) during an emotional classification task. Both congruent and incongruent audiovisual stimuli evoked larger responses in thalamus and superior temporal regions compared with unimodal conditions. Congruent emotions were characterized by activation in amygdala, insula, ventral posterior cingulate (vPCC), temporo-occipital, and auditory cortices; incongruent emotions activated a frontoparietal network and bilateral caudate nucleus, indicating a greater processing load in working memory and emotion-encoding areas. The vPCC alone exhibited differential reactions to congruency and incongruency for all emotion categories and can thus be considered a central structure for supramodal representation of complex emotional information. Moreover, the left amygdala reflected supramodal representation of happy stimuli. These findings document that emotional information does not merge at the perceptual audiovisual integration level in unimodal or multimodal areas, but in vPCC and amygdala.

Preliminary Evidence of Pre-Attentive Distinctions of Frequency-Modulated Tones that Convey Affect

Recognizing emotion is an evolutionary imperative. An early stage of auditory scene analysis involves the perceptual grouping of acoustic features, which can be based on both temporal coincidence and spectral features such as perceived pitch. Perceived pitch, or fundamental frequency (F₀), is an especially salient cue for differentiating affective intent through speech intonation (prosody). We hypothesized that: (1) simple frequency-modulated tone abstractions, based on the parameters of actual prosodic stimuli, would be reliably classified as representing differing emotional categories; and (2) that such differences would yield significant mismatch negativities (MMNs) – an index of pre-attentive deviance detection within the auditory environment. We constructed a set of FM tones that approximated the F₀ mean and variation of reliably recognized happy and neutral prosodic stimuli. These stimuli were presented to 13 subjects using a passive listening oddball paradigm. We additionally included stimuli with no frequency modulation (FM) and FM tones with identical carrier frequencies but differing modulation depths as control conditions. Following electrophysiological recording, subjects were asked to identify the sounds they heard as happy, sad, angry, or neutral. We observed that FM tones abstracted from happy and no-expression speech stimuli elicited MMNs. Post hoc behavioral testing revealed that subjects reliably identified the FM tones in a consistent manner. Finally, we also observed that FM tones and no-FM tones elicited equivalent MMNs. MMNs to FM tones that differentiate affect suggests that these abstractions may be sufficient to characterize prosodic distinctions, and that these distinctions can be represented in pre-attentive auditory sensory memory.

Preattentive processing of audio-visual emotional signals

Previous research has shown that redundant information in faces and voices leads to faster emotional categorization compared to incongruent emotional information even when attending to only one modality. The aim of the present study was to test whether these crossmodal effects are predominantly due to a response conflict rather than interference at earlier, e.g. perceptual processing stages. In Experiment 1, participants had to categorize the valence and rate the intensity of happy, sad, angry and neutral unimodal or bimodal face-voice stimuli. They were asked to rate either the facial or vocal expression and ignore the emotion expressed in the other modality. Participants responded faster and more precisely to emotionally congruent compared to incongruent face-voice pairs in both the Attend Face and in the Attend Voice condition. Moreover, when attending to faces, emotionally congruent bimodal stimuli were more efficiently processed than unimodal visual stimuli. To study the role of a possible response conflict, Experiment 2 used a modified paradigm in which emotional and response conflicts were disentangled. Incongruency effects were significant even in the absence of response conflicts. The results suggest that emotional signals available through different sensory channels are automatically combined prior to response selection.

Event-related potential correlates of the expectancy violation effect during emotional prosody processing

The present study investigated the expectancy violation effects evoked by deviation in sentential emotional prosody (EP), and their association with the deviation patterns. Event-related potentials (ERPs) were recorded for mismatching EPs with different patterns of deviation and for matching control EPs while subjects performed emotional congruousness judgment in Experiment 1 and visual probe detection tasks in Experiment 2. In the control experiment, EPs and acoustically matched non-emotional materials were presented and ERPs were recorded while participants judged the sound intensity congruousness. It was found that an early negativity, whose peak latency varied with deviation pattern, was elicited by mismatching EPs relative to matching ones, irrespective of task-relevance. A late positivity was specifically induced by mismatching EPs, and was modulated by both deviation pattern and task-relevance. Moreover, these effects cannot be simply attributed to the change in non-emotional acoustic properties. These findings suggest that the brain detects the EP deviation rapidly, and then integrates it with context for comprehension, during which the emotionality plays a role of speeding up the perception and enhancing vigilance.

[Magnetoencephalography in psychiatry]

Neuropsychiatric disorders usually come with only sublime structural changes. Functional imaging can point at specific disturbances in information processing in neural networks. Besides imaging of receptor and metabolic functions with PET and fMRI, electromagnetic methods such as electroencephalography (EEG) and magnetoencephalography (MEG) offer the possibility for imaging of dynamic dysfunctions. As compared to EEG, MEG has a shorter history and is less common despite offering considerable advantages in temporospatial resolution and sensitivity to detect impaired signal processing and network functioning which renders it particularly interesting for psychiatric applications. Disturbed processing in the auditory and visual domain emerging in schizophrenic, affective and anxiety disorders can be detected with high sensitivity. Moreover, the neuromagnetic baseline activity allows conclusions to be drawn regarding neural network functions. Due to its high sensitivity to single deficits in information processing and to pharmacological effects, MEG will achieve clinical significance in specific areas.