The Language, Tone and Prosody of Emotions: Neural Substrates and Dynamics of Spoken-Word Emotion Perception

Social-Emotional and Behavioural Difficulties in Children with Neurodevelopmental Disorders: Emotion Perception in Daily Life and in a Formal Assessment Context

Children with neurodevelopmental disorders often have social-emotional and behavioural difficulties. The present study explored these difficulties in children (n = 50, aged 6-10 years) with autism spectrum disorder, attention-deficit/hyperactivity disorder and developmental language disorder. Parents, teachers and therapists evaluated children's social-emotional and behavioural difficulties through a self-devised questionnaire and the Strengths and Difficulties Questionnaire. Additionally, the children, along with their typically developing age peers (n = 106), completed six emotion discrimination tasks. Analysis revealed some impaired emotion discrimination skills that were predictive for behavioural challenges in daily life and associated with the parent-reported existence of friends. Timely intervention in these children is needed, and it should also include emotion perception training.

Associations of resting-state perfusion and auditory verbal hallucinations with and without emotional content in schizophrenia

Auditory Verbal Hallucinations (AVH) are highly prevalent in patients with schizophrenia. AVH with high emotional content lead to particularly poor functional outcome. Increasing evidence shows that AVH are associated with alterations in structure and function in language and memory related brain regions. However, neural correlates of AVH with emotional content remain unclear. In our study (n = 91), we related resting-state cerebral perfusion to AVH and emotional content, comparing four groups: patients with AVH with emotional content (n = 13), without emotional content (n = 14), without hallucinations (n = 20) and healthy controls (n = 44). Patients with AVH and emotional content presented with increased perfusion within the amygdala and the ventromedial and dorsomedial prefrontal cortex (vmPFC/ dmPFC) compared to patients with AVH without emotional content. In addition, patients with any AVH showed hyperperfusion within the anterior cingulate gyrus, the vmPFC/dmPFC, the right hippocampus, and the left pre- and postcentral gyrus compared to patients without AVH. Our results indicate metabolic alterations in brain areas critical for the processing of emotions as key for the pathophysiology of AVH with emotional content. Particularly, hyperperfusion of the amygdala may reflect and even trigger emotional content of AVH, while hyperperfusion of the vmPFC/dmPFC cluster may indicate insufficient top-down amygdala regulation in patients with schizophrenia.

Intracellular recordings reveal integrative function of the basolateral amygdala in acoustic communication

The amygdala, a brain center of emotional expression, contributes to appropriate behavior responses during acoustic communication. In support of that role, the basolateral amygdala (BLA) analyzes the meaning of vocalizations through the integration of multiple acoustic inputs with information from other senses and an animal's internal state. The mechanisms underlying this integration are poorly understood. This study focuses on the integration of vocalization-related inputs to the BLA from auditory centers during this processing. We used intracellular recordings of BLA neurons in unanesthetized big brown bats that rely heavily on a complex vocal repertoire during social interactions. Postsynaptic and spiking responses of BLA neurons were recorded to three vocal sequences that are closely related to distinct behaviors (appeasement, low-level aggression, and high-level aggression) and have different emotional valence. Our novel findings are that most BLA neurons showed postsynaptic responses to one or more vocalizations (31 of 46) but that many fewer neurons showed spiking responses (8 of 46). The spiking responses were more selective than postsynaptic potential (PSP) responses. Furthermore, vocal stimuli associated with either positive or negative valence were similarly effective in eliciting excitatory postsynaptic potentials (EPSPs), inhibitory postsynaptic potentials (IPSPs), and spiking responses. This indicates that BLA neurons process both positive- and negative-valence vocal stimuli. The greater selectivity of spiking responses than PSP responses suggests an integrative role for processing within the BLA to enhance response specificity in acoustic communication.NEW & NOTEWORTHY The amygdala plays an important role in social communication by sound, but little is known about how it integrates diverse auditory inputs to form selective responses to social vocalizations. We show that BLA neurons receive inputs that are responsive to both negative- and positive-affect vocalizations but their spiking outputs are fewer and highly selective for vocalization type. Our work demonstrates that BLA neurons perform an integrative function in shaping appropriate behavioral responses to social vocalizations.

Exploring the Emotional Functions of Co-Speech Hand Gesture in Language and Communication

Research over the past four decades has built a convincing case that co-speech hand gestures play a powerful role in human cognition . However, this recent focus on the cognitive function of gesture has, to a large extent, overlooked its emotional role-a role that was once central to research on bodily expression. In the present review, we first give a brief summary of the wealth of research demonstrating the cognitive function of co-speech gestures in language acquisition, learning, and thinking. Building on this foundation, we revisit the emotional function of gesture across a wide range of communicative contexts, from clinical to artistic to educational, and spanning diverse fields, from cognitive neuroscience to linguistics to affective science. Bridging the cognitive and emotional functions of gesture highlights promising avenues of research that have varied practical and theoretical implications for human-machine interactions, therapeutic interventions, language evolution, embodied cognition, and more.

Acoustic Features Distinguishing Emotions in Swedish Speech

Few studies have examined which acoustic features of speech can be used to distinguish between different emotions, and how combinations of acoustic parameters contribute to identification of emotions. The aim of the present study was to investigate which acoustic parameters in Swedish speech are most important for differentiation between, and identification of, the emotions anger, fear, happiness, sadness, and surprise in Swedish sentences. One-way ANOVAs were used to compare acoustic parameters between the emotions and both simple and multiple logistic regression models were used to examine the contribution of different acoustic parameters to differentiation between emotions. Results showed differences between emotions for several acoustic parameters in Swedish speech: surprise was the most distinct emotion, with significant differences compared to the other emotions across a range of acoustic parameters, while anger and happiness did not differ from each other on any parameter. The logistic regression models showed that fear was the best-predicted emotion while happiness was most difficult to predict. Frequency- and spectral-balance-related parameters were best at predicting fear. Amplitude- and temporal-related parameters were most important for surprise, while a combination of frequency-, amplitude- and spectral balance-related parameters are important for sadness. Assuming that there are similarities between acoustic models and how listeners infer emotions in speech, results suggest that individuals with hearing loss, who lack abilities of frequency detection, may compared to normal hearing individuals have difficulties in identifying fear in Swedish speech. Since happiness and fear relied primarily on amplitude- and spectral-balance-related parameters, detection of them are probably facilitated more by hearing aid use.

Disentangling emotional signals in the brain: an ALE meta-analysis of vocal affect perception

Recent advances in neuroimaging research on vocal emotion perception have revealed voice-sensitive areas specialized in processing affect. Experimental data on this subject is varied, investigating a wide range of emotions through different vocal signals and task demands. The present meta-analysis was designed to disentangle this diversity of results by summarizing neuroimaging data in the vocal emotion perception literature. Data from 44 experiments contrasting emotional and neutral voices was analyzed to assess brain areas involved in vocal affect perception in general, as well as depending on the type of voice signal (speech prosody or vocalizations), the task demands (implicit or explicit attention to emotions), and the specific emotion perceived. Results reassessed a consistent bilateral network of Emotional Voices Areas consisting of the superior temporal cortex and primary auditory regions. Specific activations and lateralization of these regions, as well as additional areas (insula, middle temporal gyrus) were further modulated by signal type and task demands. Exploring the sparser data on single emotions also suggested the recruitment of other regions (insula, inferior frontal gyrus, frontal operculum) for specific aspects of each emotion. These novel meta-analytic results suggest that while the bulk of vocal affect processing is localized in the STC, the complexity and variety of such vocal signals entails functional specificities in complex and varied cortical (and potentially subcortical) response pathways.

A systematic review on production and comprehension of linguistic prosody in people with acquired language and communication disorders resulting from unilateral brain lesions

BACKGROUND

Prosody serves central functions in language processing including linguistic functions (linguistic prosody), like structuring the speech signal. Impairments in production and comprehension of linguistic prosody have been described for persons with unilateral right (RHDP) or left hemisphere damage (LHDP). However, reported results differ with respect to the characteristics and severities of these impairments AIMS: We conducted a systematic literature review focusing on production and comprehension of linguistic prosody at the prosody-syntax interface (i.e., phrase or sentence level) in LHDP and RHDP.

METHODS & PROCEDURES

In a systematic literature search we included: (i) empirical studies with (ii) adult RHDP and/or LHDP (iii) investigating production and/or comprehension of linguistic prosody at the (iv) phrase or sentence level (v) reporting quantitative data on prosodic measures. We excluded overview papers; studies involving participants with dysarthria, apraxia of speech, foreign accent syndrome, psychiatric diseases, and/or neurodegenerative diseases; studies focusing primarily on emotional prosody; and on lexical stress / word level; studies of which no full text was available and/or that were published in a language other than English. We searched the databases BIOSIS, MEDLINE, EMBASE, PubMed, Web of Science, CINAHL, Cochrane Library, PSYNDEX, PsycINFO and speechBITE, last searched on January 13^th 2022.We found 2,631 studies without duplicates. We identified 43 studies which were included into our systematic review. For data extraction and synthesis of results, we grouped studies by (i) modality (production vs. comprehension), (ii) function (syntactic structure vs. information structure), and (iii) by experiment task. For production studies, outcome measures were defined as the productive use of the different prosodic cues (lengthening, pause, f0, amplitude). For comprehension studies, performance measures (accuracy and reaction times) were defined as outcome measures. In accordance with the PRISMA 2020 statement (Page et al., 2021), we conducted a quality check to assess study risk of bias. Our review was pre-registered with PROSPERO (CRD42019120308).

OUTCOMES & RESULTS

Of the 43 studies reviewed, 30 studies involved RHDP (n = 309), assessing production in 15 studies and focusing on comprehension of prosody in 16 studies (one study investigated production and comprehension). LHDP (n = 438) were included in 35 studies of which 15 studied production and 21 evaluated comprehension of prosody (one study investigated production and comprehension). Despite the heterogeneity of results in the studies reviewed, our synthesis of results suggests that both LHDP and RHDP show limitations, but no complete impairment, in their production and/or comprehension of linguistic prosody. Prosodic limitations are evident in different areas of processing linguistic prosody, like syntactic disambiguation or the distinction between sentence types. There is a tendency towards more severe limitations in LHDP as compared to RHDP.

CONCLUSIONS

We only included published studies into our review and did not perform an assessment of risk of reporting bias as well as systematic certainty assessments of the outcomes. Despite these limitations, we conclude that both groups show deficits in production and comprehension of linguistic prosody, but neither LHDP nor RHDP are completely impaired in their prosodic processing. This suggests that prosody is a relevant communicative resource for LHDP and RHDP worth being addressed in speech-language-therapy.

Neurocognitive Dynamics of Prosodic Salience over Semantics during Explicit and Implicit Processing of Basic Emotions in Spoken Words

How language mediates emotional perception and experience is poorly understood. The present event-related potential (ERP) study examined the explicit and implicit processing of emotional speech to differentiate the relative influences of communication channel, emotion category and task type in the prosodic salience effect. Thirty participants (15 women) were presented with spoken words denoting happiness, sadness and neutrality in either the prosodic or semantic channel. They were asked to judge the emotional content (explicit task) and speakers' gender (implicit task) of the stimuli. Results indicated that emotional prosody (relative to semantics) triggered larger N100, P200 and N400 amplitudes with greater delta, theta and alpha inter-trial phase coherence (ITPC) and event-related spectral perturbation (ERSP) values in the corresponding early time windows, and continued to produce larger LPC amplitudes and faster responses during late stages of higher-order cognitive processing. The relative salience of prosodic and semantics was modulated by emotion and task, though such modulatory effects varied across different processing stages. The prosodic salience effect was reduced for sadness processing and in the implicit task during early auditory processing and decision-making but reduced for happiness processing in the explicit task during conscious emotion processing. Additionally, across-trial synchronization of delta, theta and alpha bands predicted the ERP components with higher ITPC and ERSP values significantly associated with stronger N100, P200, N400 and LPC enhancement. These findings reveal the neurocognitive dynamics of emotional speech processing with prosodic salience tied to stage-dependent emotion- and task-specific effects, which can reveal insights into understanding language and emotion processing from cross-linguistic/cultural and clinical perspectives.

How is emotional resonance achieved in storytellings of sadness/distress?

Storytelling pivots around stance seen as a window unto emotion: storytellers project a stance expressing their emotion toward the events and recipients preferably mirror that stance by affiliating with the storyteller’s stance. Whether the recipient’s affiliative stance is at the same time expressive of his/her emotional resonance with the storyteller and of emotional contagion is a question that has recently attracted intriguing research in Physiological Interaction Research. Connecting to this line of inquiry, this paper concerns itself with storytellings of sadness/distress. Its aim is to identify factors that facilitate emotion contagion in storytellings of sadness/distress and factors that impede it. Given the complexity and novelty of this question, this study is designed as a pilot study to scour the terrain and sketch out an interim roadmap before a larger study is undertaken. The data base is small, comprising two storytellings of sadness/distress. The methodology used to address the above research question is expansive: it includes CA methods to transcribe and analyze interactionally relevant aspects of the storytelling interaction; it draws on psychophysiological measures to establish whether and to what degree emotional resonance between co-participants is achieved. In discussing possible reasons why resonance is (not or not fully) achieved, the paper embarks on an extended analysis of the storytellers’ multimodal storytelling performance (reenactments, prosody, gaze, gesture) and considers factors lying beyond the storyteller’s control, including relevance, participation framework, personality, and susceptibility to emotion contagion.

TrackUSF, a novel tool for automated ultrasonic vocalization analysis, reveals modified calls in a rat model of autism

Background

Various mammalian species emit ultrasonic vocalizations (USVs), which reflect their emotional state and mediate social interactions. USVs are usually analyzed by manual or semi-automated methodologies that categorize discrete USVs according to their structure in the frequency-time domains. This laborious analysis hinders the effective use of USVs as a readout for high-throughput analysis of behavioral changes in animals.

Results

Here we present a novel automated open-source tool that utilizes a different approach towards USV analysis, termed TrackUSF. To validate TrackUSF, we analyzed calls from different animal species, namely mice, rats, and bats, recorded in various settings and compared the results with a manual analysis by a trained observer. We found that TrackUSF detected the majority of USVs, with less than 1% of false-positive detections. We then employed TrackUSF to analyze social vocalizations in Shank3-deficient rats, a rat model of autism, and revealed that these vocalizations exhibit a spectrum of deviations from appetitive calls towards aversive calls.

Conclusions

TrackUSF is a simple and easy-to-use system that may be used for a high-throughput comparison of ultrasonic vocalizations between groups of animals of any kind in any setting, with no prior assumptions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12915-022-01299-y.

ALS/FTD: Evolution, Aging, and Cellular Metabolic Exhaustion

Amyotrophic lateral sclerosis and frontotemporal dementia (ALS/FTD) are neurodegenerations with evolutionary underpinnings, expansive clinical presentations, and multiple genetic risk factors involving a complex network of pathways. This perspective considers the complex cellular pathology of aging motoneuronal and frontal/prefrontal cortical networks in the context of evolutionary, clinical, and biochemical features of the disease. We emphasize the importance of evolution in the development of the higher cortical function, within the influence of increasing lifespan. Particularly, the role of aging on the metabolic competence of delicately optimized neurons, age-related increased proteostatic costs, and specific genetic risk factors that gradually reduce the energy available for neuronal function leading to neuronal failure and disease.

Localized task-invariant emotional valence encoding revealed by intracranial recordings

The ability to distinguish between negative, positive and neutral valence is a key part of emotion perception. Emotional valence has conceptual meaning that supersedes any particular type of stimulus, although it is typically captured experimentally in association with particular tasks. We sought to identify neural encoding for task-invariant emotional valence. We evaluated whether high-gamma responses (HGRs) to visually displayed words conveying emotions could be used to decode emotional valence from HGRs to facial expressions. Intracranial electroencephalography was recorded from 14 individuals while they participated in two tasks, one involving reading words with positive, negative, and neutral valence, and the other involving viewing faces with positive, negative, and neutral facial expressions. Quadratic discriminant analysis was used to identify information in the HGR that differentiates the three emotion conditions. A classifier was trained on the emotional valence labels from one task and was cross-validated on data from the same task (within-task classifier) as well as the other task (between-task classifier). Emotional valence could be decoded in the left medial orbitofrontal cortex and middle temporal gyrus, both using within-task classifiers and between-task classifiers. These observations suggest the presence of task-independent emotional valence information in the signals from these regions.

How Tone, Intonation and Emotion Shape the Development of Infants’ Fundamental Frequency Perception

Fundamental frequency (ƒ₀), perceived as pitch, is the first and arguably most salient auditory component humans are exposed to since the beginning of life. It carries multiple linguistic (e.g., word meaning) and paralinguistic (e.g., speakers’ emotion) functions in speech and communication. The mappings between these functions and ƒ₀ features vary within a language and differ cross-linguistically. For instance, a rising pitch can be perceived as a question in English but a lexical tone in Mandarin. Such variations mean that infants must learn the specific mappings based on their respective linguistic and social environments. To date, canonical theoretical frameworks and most empirical studies do not view or consider the multi-functionality of ƒ₀, but typically focus on individual functions. More importantly, despite the eventual mastery of ƒ₀ in communication, it is unclear how infants learn to decompose and recognize these overlapping functions carried by ƒ₀. In this paper, we review the symbioses and synergies of the lexical, intonational, and emotional functions that can be carried by ƒ₀ and are being acquired throughout infancy. On the basis of our review, we put forward the Learnability Hypothesis that infants decompose and acquire multiple ƒ₀ functions through native/environmental experiences. Under this hypothesis, we propose representative cases such as the synergy scenario, where infants use visual cues to disambiguate and decompose the different ƒ₀ functions. Further, viable ways to test the scenarios derived from this hypothesis are suggested across auditory and visual modalities. Discovering how infants learn to master the diverse functions carried by ƒ₀ can increase our understanding of linguistic systems, auditory processing and communication functions.

Longitudinal Change in Neural Response to Vocal Emotion in Adolescence

Adolescence is associated with maturation of function within neural networks supporting the processing of social information. Previous longitudinal studies have established developmental influences on youth’s neural response to facial displays of emotion. Given the increasing recognition of the importance of non-facial cues to social communication, we build on existing work by examining longitudinal change in neural response to vocal expressions of emotion in 8- to 19-year-old youth. Participants completed a vocal emotion recognition task at two timepoints (1 year apart) while undergoing functional magnetic resonance imaging. The right inferior frontal gyrus, right dorsal striatum and right precentral gyrus showed decreases in activation to emotional voices across timepoints, which may reflect focalization of response in these areas. Activation in the dorsomedial prefrontal cortex was positively associated with age but was stable across timepoints. In addition, the slope of change across visits varied as a function of participants’ age in the right temporo-parietal junction (TPJ): this pattern of activation across timepoints and age may reflect ongoing specialization of function across childhood and adolescence. Decreased activation in the striatum and TPJ across timepoints was associated with better emotion recognition accuracy. Findings suggest that specialization of function in social cognitive networks may support the growth of vocal emotion recognition skills across adolescence.

Effects of mild-to-moderate sensorineural hearing loss and signal amplification on vocal emotion recognition in middle-aged–older individuals

Previous research has shown deficits in vocal emotion recognition in sub-populations of individuals with hearing loss, making this a high priority research topic. However, previous research has only examined vocal emotion recognition using verbal material, in which emotions are expressed through emotional prosody. There is evidence that older individuals with hearing loss suffer from deficits in general prosody recognition, not specific to emotional prosody. No study has examined the recognition of non-verbal vocalization, which constitutes another important source for the vocal communication of emotions. It might be the case that individuals with hearing loss have specific difficulties in recognizing emotions expressed through prosody in speech, but not non-verbal vocalizations. We aim to examine whether vocal emotion recognition difficulties in middle- aged-to older individuals with sensorineural mild-moderate hearing loss are better explained by deficits in vocal emotion recognition specifically, or deficits in prosody recognition generally by including both sentences and non-verbal expressions. Furthermore a, some of the studies which have concluded that individuals with mild-moderate hearing loss have deficits in vocal emotion recognition ability have also found that the use of hearing aids does not improve recognition accuracy in this group. We aim to examine the effects of linear amplification and audibility on the recognition of different emotions expressed both verbally and non-verbally. Besides examining accuracy for different emotions we will also look at patterns of confusion (which specific emotions are mistaken for other specific emotion and at which rates) during both amplified and non-amplified listening, and we will analyze all material acoustically and relate the acoustic content to performance. Together these analyses will provide clues to effects of amplification on the perception of different emotions. For these purposes, a total of 70 middle-aged-older individuals, half with mild-moderate hearing loss and half with normal hearing will perform a computerized forced-choice vocal emotion recognition task with and without amplification.

Simulated proximity enhances perceptual and physiological responses to emotional facial expressions

Physical proximity is important in social interactions. Here, we assessed whether simulated physical proximity modulates the perceived intensity of facial emotional expressions and their associated physiological signatures during observation or imitation of these expressions. Forty-four healthy volunteers rated intensities of dynamic angry or happy facial expressions, presented at two simulated locations, proximal (0.5 m) and distant (3 m) from the participants. We tested whether simulated physical proximity affected the spontaneous (in the observation task) and voluntary (in the imitation task) physiological responses (activity of the corrugator supercilii face muscle and pupil diameter) as well as subsequent ratings of emotional intensity. Angry expressions provoked relative activation of the corrugator supercilii muscle and pupil dilation, whereas happy expressions induced a decrease in corrugator supercilii muscle activity. In proximal condition, these responses were enhanced during both observation and imitation of the facial expressions, and were accompanied by an increase in subsequent affective ratings. In addition, individual variations in condition related EMG activation during imitation of angry expressions predicted increase in subsequent emotional ratings. In sum, our results reveal novel insights about the impact of physical proximity in the perception of emotional expressions, with early proximity-induced enhancements of physiological responses followed by an increased intensity rating of facial emotional expressions.

Discretisation and continuity: The emergence of symbols in communication

Vocal signalling systems, as used by humans and various non-human animals, exhibit discrete and continuous properties that can naturally be used to express discrete and continuous information, such as distinct words to denote objects in the world and prosodic features to convey the emotions of the speaker. However, continuous aspects are not always expressed with the continuous properties of an utterance but are frequently categorised into discrete symbols. While the existence of symbols in communication is self-evident, the emergence of discretisation from a continuous space is not well understood. In this paper, we investigate the emergence of discrete symbols in regions with a continuous semantics by simulating the learning process of two agents that acquire a shared signalling system. The task is formalised as a reinforcement learning problem with a continuous form and meaning space. We identify two causes for the emergence of discretisation that do not originate in discrete semantics: 1) premature convergence to sub-optimal signalling conventions and 2) topological mismatch between the continuous form space and the continuous semantic space. The insights presented in this paper shed light on the origins of discrete symbols, whose existence is assumed by a large body of research concerned with the emergence of syntactic structures and meaning in language.

Cognitive Neuroscience Methods in Enhancing Health Literacy

The aim of the article is to identify the usefulness of cognitive neuroscience methods in assessing the effectiveness of social advertising and constructing messages referring to the generally understood health promotion, which is to contribute to the development of health awareness, and hence to health literacy. The presented research has also proven useful in the field of managing the processes that improve the communication between the organization and its environment. The researchers experimentally applied cognitive neuroscience methods, mainly EEG measurements, including a metric which is one of the most frequently used to measure the reception of advertising messages, i.e., frontal asymmetry. The purpose of the study was to test cognitive responses as expressed by neural indices (memorization, interest) to the reception of an advertisement for the construction of a hospice for adults. For comparative purposes, a questionnaire survey was also conducted. The research findings have confirmed that there are significant differences in remembering the advertisement in question by different groups of recipients (women/men). They also indicate a different level of interest in the advertisement, which may result from different preferences of the recipients concerning the nature of ads. The obtained results contribute to a better understanding of how to design advertising messages concerning health, so that they increase the awareness of the recipients’ responsibility for their own health and induce specific behavior patterns aimed at supporting health-related initiatives, e.g., donating funds for building hospices or performing preventive tests. In this respect, the study findings help improve the organizations’ communication with their environment, thus enhancing their performance. The study has also confirmed the potential and innovativeness of cognitive neuroscience methods as well as their considerable possibilities for application in this field.

The correlation between white matter integrity and pragmatic language processing in first episode schizophrenia

Objective: Higher-order language disturbances could be the result of white matter tract abnormalities. The study explores the relationship between white matter and pragmatic skills in first-episode schizophrenia. Methods: Thirty-four first-episode patients with schizophrenia and 32 healthy subjects participated in a pragmatic language and Diffusion Tensor Imaging study, where fractional anisotropy of the arcuate fasciculus, corpus callosum and cingulum was correlated with the Polish version of the Right Hemisphere Language Battery. Results: The patients showed reduced fractional anisotropy in the right arcuate fasciculus, left anterior cingulum bundle and left forceps minor. Among the first episode patients, reduced understanding of written metaphors correlated with reduced fractional anisotropy of left forceps minor, and greater explanation of written and picture metaphors correlated with reduced fractional anisotropy of the left anterior cingulum. Conclusions: The white matter dysfunctions may underlie the pragmatic language impairment in schizophrenia. Our results shed further light on the functional neuroanatomical basis of pragmatic language use by patients with schizophrenia.

Facial expressions can be categorized along the upper-lower facial axis, from a perceptual perspective

A critical question, fundamental for building models of emotion, is how to categorize emotions. Previous studies have typically taken one of two approaches: (a) they focused on the pre-perceptual visual cues, how salient facial features or configurations were displayed; or (b) they focused on the post-perceptual affective experiences, how emotions affected behavior. In this study, we attempted to group emotions at a peri-perceptual processing level: it is well known that humans perceive different facial expressions differently, therefore, can we classify facial expressions into distinct categories in terms of their perceptual similarities? Here, using a novel non-lexical paradigm, we assessed the perceptual dissimilarities between 20 facial expressions using reaction times. Multidimensional-scaling analysis revealed that facial expressions were organized predominantly along the upper-lower face axis. Cluster analysis of behavioral data delineated three superordinate categories, and eye-tracking measurements validated these clustering results. Interestingly, these superordinate categories can be conceptualized according to how facial displays interact with acoustic communications: One group comprises expressions that have salient mouth features. They likely link to species-specific vocalization, for example, crying, laughing. The second group comprises visual displays with diagnosing features in both the mouth and the eye regions. They are not directly articulable but can be expressed prosodically, for example, sad, angry. Expressions in the third group are also whole-face expressions but are completely independent of vocalization, and likely being blends of two or more elementary expressions. We propose a theoretical framework to interpret the tripartite division in which distinct expression subsets are interpreted as successive phases in an evolutionary chain.

The relationship between vocal affect recognition and psychosocial functioning for people with moderate to severe traumatic brain injury: a systematic review

The purpose of this review was to explore how vocal affect recognition deficits impact the psychosocial functioning of people with moderate to severe traumatic brain injury (TBI). A systematic review following the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) guidelines was conducted, whereby six databases were searched, with additional hand searching of key journals also completed. The search identified 1847 records after duplicates were removed, and 1749 were excluded through title and abstract screening. After full text screening of 65 peer-reviewed articles published between January 1999 and August 2019, only five met inclusion criteria. The methodological quality of selected studies was assessed using the Mixed Methods Appraisal Tool (MMAT) Version 2018 with a fair level of agreement reached. A narrative synthesis of the results was completed, exploring vocal affect recognition and psychosocial functioning of people with moderate to severe TBI, including aspects of social cognition (i.e., empathy; Theory of Mind) and social behaviour. Results of the review were limited by a paucity of research in this area, a lack of high-level evidence, and wide variation in the outcome measures used. More rigorous study designs are required to establish more conclusive evidence regarding the degree and direction of the association between vocal affect recognition and aspects of psychosocial functioning. This review is registered with Prospero.

Lack of neural evidence for implicit language learning in 9-month-old infants at high risk for autism

Word segmentation is a fundamental aspect of language learning, since identification of word boundaries in continuous speech must occur before the acquisition of word meanings can take place. We previously used functional magnetic resonance imaging (fMRI) to show that youth with autism spectrum disorder (ASD) are less sensitive to statistical and speech cues that guide implicit word segmentation. However, little is known about the neural mechanisms underlying this process during infancy and how this may be associated with ASD risk. Here, we examined early neural signatures of language-related learning in 9-month-old infants at high (HR) and low familial risk (LR) for ASD. During natural sleep, infants underwent fMRI while passively listening to three speech streams containing strong statistical and prosodic cues, strong statistical cues only, or minimal statistical cues to word boundaries. Compared to HR infants, LR infants showed greater activity in the left amygdala for the speech stream containing statistical and prosodic cues. While listening to this same speech stream, LR infants also showed more learning-related signal increases in left temporal regions as well as increasing functional connectivity between bilateral primary auditory cortex and right anterior insula. Importantly, learning-related signal increases at 9 months positively correlated with expressive language outcome at 36 months in both groups. In the HR group, greater signal increases were additionally associated with less severe ASD symptomatology at 36 months. These findings suggest that early differences in the neural networks underlying language learning may predict subsequent language development and altered trajectories associated with ASD risk.

Brain correlates of emotional prosodic change detection in autism spectrum disorder

•

We used an oddball paradigm with vocal stimuli to record hemodynamic responses.

•

Brain processing of vocal change relies on STG, insula and lingual area.

•

Activity of the change processing network can be modulated by saliency and emotion.

•

Brain processing of vocal deviancy/novelty appears typical in adults with autism.

Autism Spectrum Disorder (ASD) is currently diagnosed by the joint presence of social impairments and restrictive, repetitive patterns of behaviors. While the co-occurrence of these two categories of symptoms is at the core of the pathology, most studies investigated only one dimension to understand underlying physiopathology. In this study, we analyzed brain hemodynamic responses in neurotypical adults (CTRL) and adults with autism spectrum disorder during an oddball paradigm allowing to explore brain responses to vocal changes with different levels of saliency (deviancy or novelty) and different emotional content (neutral, angry).

Change detection relies on activation of the supratemporal gyrus and insula and on deactivation of the lingual area. The activity of these brain areas involved in the processing of deviancy with vocal stimuli was modulated by saliency and emotion. No group difference between CTRL and ASD was reported for vocal stimuli processing or for deviancy/novelty processing, regardless of emotional content.

Findings highlight that brain processing of voices and of neutral/ emotional vocal changes is typical in adults with ASD. Yet, at the behavioral level, persons with ASD still experience difficulties with those cues. This might indicate impairments at latter processing stages or simply show that alterations present in childhood might have repercussions at adult age.

Children and adolescents' neural response to emotional faces and voices: Age-related changes in common regions of activation

The perception of facial and vocal emotional expressions engages overlapping regions of the brain. However, at a behavioral level, the ability to recognize the intended emotion in both types of nonverbal cues follows a divergent developmental trajectory throughout childhood and adolescence. The current study a) identified regions of common neural activation to facial and vocal stimuli in 8- to 19-year-old typically-developing adolescents, and b) examined age-related changes in blood-oxygen-level dependent (BOLD) response within these areas. Both modalities elicited activation in an overlapping network of subcortical regions (insula, thalamus, dorsal striatum), visual-motor association areas, prefrontal regions (inferior frontal cortex, dorsomedial prefrontal cortex), and the right superior temporal gyrus. Within these regions, increased age was associated with greater frontal activation to voices, but not faces. Results suggest that processing facial and vocal stimuli elicits activation in common areas of the brain in adolescents, but that age-related changes in response within these regions may vary by modality.

Coregulation of therapist and client emotion during psychotherapy

OBJECTIVE

Close interpersonal relationships are fundamental to emotion regulation. Clinical theory suggests that one role of therapists in psychotherapy is to help clients regulate emotions, however, if and how clients and therapists serve to regulate each other's emotions has not been empirically tested. Emotion coregulation - the bidirectional emotional linkage of two people that promotes emotional stability - is a specific, temporal process that provides a framework for testing the way in which therapists' and clients' emotions may be related on a moment to moment basis in clinically relevant ways.

METHOD

Utilizing 227 audio recordings from a relationally oriented treatment (Motivational Interviewing), we estimated continuous values of vocally encoded emotional arousal via mean fundamental frequency. We used dynamic systems models to examine emotional coregulation, and tested the hypothesis that each individual's emotional arousal would be significantly associated with fluctuations in the other's emotional state over the course of a psychotherapy session.

RESULTS

Results indicated that when clients became more emotionally labile over the course of the session, therapists became less so. When changes in therapist arousal increased, the client's tendency to become more aroused during session slowed. Alternatively, when changes in client arousal increased, the therapist's tendency to become less aroused slowed.

ERP Markers of Valence Coding in Emotional Speech Processing

How is auditory emotional information processed? The study's aim was to compare cerebral responses to emotionally positive or negative spoken phrases matched for structure and content. Twenty participants listened to 198 vocal stimuli while detecting filler phrases containing first names. EEG was recorded from 128 sites. Three event-related potential (ERP) components were quantified and found to be sensitive to emotional valence since 350 ms of latency. P450 and late positivity were enhanced by positive content, whereas anterior negativity was larger to negative content. A similar set of markers (P300, N400, LP) was found previously for the processing of positive versus negative affective vocalizations, prosody, and music, which suggests a common neural mechanism for extracting the emotional content of auditory information. SwLORETA applied to potentials recorded between 350 and 550 ms showed that negative speech activated the right temporo/parietal areas (BA40, BA20/21), whereas positive speech activated the left homologous and inferior frontal areas.

Investigation of functional brain network reconfiguration during vocal emotional processing using graph-theoretical analysis

Vocal expression is essential for conveying the emotion during social interaction. Although vocal emotion has been explored in previous studies, little is known about how perception of different vocal emotional expressions modulates the functional brain network topology. In this study, we aimed to investigate the functional brain networks under different attributes of vocal emotion by graph-theoretical network analysis. Functional magnetic resonance imaging (fMRI) experiments were performed on 36 healthy participants. We utilized the Power-264 functional brain atlas to calculate the interregional functional connectivity (FC) from fMRI data under resting state and vocal stimuli at different arousal and valence levels. The orthogonal minimal spanning trees method was used for topological filtering. The paired-sample t-test with Bonferroni correction across all regions and arousal-valence levels were used for statistical comparisons. Our results show that brain network exhibits significantly altered network attributes at FC, nodal and global levels, especially under high-arousal or negative-valence vocal emotional stimuli. The alterations within/between well-known large-scale functional networks were also investigated. Through the present study, we have gained more insights into how comprehending emotional speech modulates brain networks. These findings may shed light on how the human brain processes emotional speech and how it distinguishes different emotional conditions.

Age-related differences in neural activation and functional connectivity during the processing of vocal prosody in adolescence

The ability to recognize others' emotions based on vocal emotional prosody follows a protracted developmental trajectory during adolescence. However, little is known about the neural mechanisms supporting this maturation. The current study investigated age-related differences in neural activation during a vocal emotion recognition (ER) task. Listeners aged 8 to 19 years old completed the vocal ER task while undergoing functional magnetic resonance imaging. The task of categorizing vocal emotional prosody elicited activation primarily in temporal and frontal areas. Age was associated with a) greater activation in regions in the superior, middle, and inferior frontal gyri, b) greater functional connectivity between the left precentral and inferior frontal gyri and regions in the bilateral insula and temporo-parietal junction, and c) greater fractional anisotropy in the superior longitudinal fasciculus, which connects frontal areas to posterior temporo-parietal regions. Many of these age-related differences in brain activation and connectivity were associated with better performance on the ER task. Increased activation in, and connectivity between, areas typically involved in language processing and social cognition may facilitate the development of vocal ER skills in adolescence.

Large-scale plurimodal networks common to listening to, producing and reading word lists: an fMRI study combining task-induced activation and intrinsic connectivity in 144 right-handers

We aimed at identifying plurimodal large-scale networks for producing, listening to and reading word lists based on the combined analyses of task-induced activation and resting-state intrinsic connectivity in 144 healthy right-handers. In the first step, we identified the regions in each hemisphere showing joint activation and joint asymmetry during the three tasks. In the left hemisphere, 14 homotopic regions of interest (hROIs) located in the left Rolandic sulcus, precentral gyrus, cingulate gyrus, cuneus and inferior supramarginal gyrus (SMG) met this criterion, and 7 hROIs located in the right hemisphere were located in the preSMA, medial superior frontal gyrus, precuneus and superior temporal sulcus (STS). In a second step, we calculated the BOLD temporal correlations across these 21 hROIs at rest and conducted a hierarchical clustering analysis to unravel their network organization. Two networks were identified, including the WORD-LIST_CORE network that aggregated 14 motor, premotor and phonemic areas in the left hemisphere plus the right STS that corresponded to the posterior human voice area (pHVA). The present results revealed that word-list processing is based on left articulatory and storage areas supporting the action-perception cycle common not only to production and listening but also to reading. The inclusion of the right pHVA acting as a prosodic integrative area highlights the importance of prosody in the three modalities and reveals an intertwining across hemispheres between prosodic (pHVA) and phonemic (left SMG) processing. These results are consistent with the motor theory of speech postulating that articulatory gestures are the central motor units on which word perception, production, and reading develop and act together.

Feedforward Neural Network-Based Architecture for Predicting Emotions from Speech

We propose a novel feedforward neural network (FFNN)-based speech emotion recognition system built on three layers: A base layer where a set of speech features are evaluated and classified; a middle layer where a speech matrix is built based on the classification scores computed in the base layer; a top layer where an FFNN- and a rule-based classifier are used to analyze the speech matrix and output the predicted emotion. The system offers 80.75% accuracy for predicting the six basic emotions and surpasses other state-of-the-art methods when tested on emotion-stimulated utterances. The method is robust and the fastest in the literature, computing a stable prediction in less than 78 s and proving attractive for replacing questionnaire-based methods and for real-time use. A set of correlations between several speech features (intensity contour, speech rate, pause rate, and short-time energy) and the evaluated emotions is determined, which enhances previous similar studies that have not analyzed these speech features. Using these correlations to improve the system leads to a 6% increase in accuracy. The proposed system can be used to improve human–computer interfaces, in computer-mediated education systems, for accident prevention, and for predicting mental disorders and physical diseases.

Congenital facial palsy and emotion processing: The case of Moebius syndrome

According to the Darwinian perspective, facial expressions of emotions evolved to quickly communicate emotional states and would serve adaptive functions that promote social interactions. Embodied cognition theories suggest that we understand others' emotions by reproducing the perceived expression in our own facial musculature (facial mimicry) and the mere observation of a facial expression can evoke the corresponding emotion in the perceivers. Consequently, the inability to form facial expressions would affect the experience of emotional understanding. In this review, we aimed at providing account on the link between the lack of emotion production and the mechanisms of emotion processing. We address this issue by taking into account Moebius syndrome, a rare neurological disorder that primarily affects the muscles controlling facial expressions. Individuals with Moebius syndrome are born with facial paralysis and inability to form facial expressions. This makes them the ideal population to study whether facial mimicry is necessary for emotion understanding. Here, we discuss behavioral ambiguous/mixed results on emotion recognition deficits in Moebius syndrome suggesting the need to investigate further aspects of emotional processing such as the physiological responses associated with the emotional experience during developmental age.

CACNA1C risk variant affects microstructural connectivity of the amygdala

Deficits in perception of emotional prosody have been described in patients with affective disorders at behavioral and neural level. In the current study, we use an imaging genetics approach to examine the impact of CACNA1C, one of the most promising genetic risk factors for psychiatric disorders, on prosody processing on a behavioral, functional and microstructural level. Using functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI) we examined key areas involved in prosody processing, i.e. the amygdala and voice areas, in a healthy population. We found stronger activation to emotional than neutral prosody in the voice areas and the amygdala, but CACNA1C rs1006737 genotype had no influence on fMRI activity. However, significant microstructural differences (i.e. mean diffusivity) between CACNA1C rs1006737 risk allele carriers and non carriers were found in the amygdala, but not the voice areas. These modifications in brain architecture associated with CACNA1C might reflect a neurobiological marker predisposing to affective disorders and concomitant alterations in emotion perception.

Emotional prosodic change detection in autism Spectrum disorder: an electrophysiological investigation in children and adults

Background

Autism spectrum disorder (ASD) is characterized by atypical behaviors in social environments and in reaction to changing events. While this dyad of symptoms is at the core of the pathology along with atypical sensory behaviors, most studies have investigated only one dimension. A focus on the sameness dimension has shown that intolerance to change is related to an atypical pre-attentional detection of irregularity. In the present study, we addressed the same process in response to emotional change in order to evaluate the interplay between alterations of change detection and socio-emotional processing in children and adults with autism.

Methods

Brain responses to neutral and emotional prosodic deviancies (mismatch negativity (MMN) and P3a, reflecting change detection and orientation of attention toward change, respectively) were recorded in children and adults with autism and in controls. Comparison of neutral and emotional conditions allowed distinguishing between general deviancy and emotional deviancy effects. Moreover, brain responses to the same neutral and emotional stimuli were recorded when they were not deviants to evaluate the sensory processing of these vocal stimuli.

Results

In controls, change detection was modulated by prosody: in children, this was characterized by a lateralization of emotional MMN to the right hemisphere, and in adults, by an earlier MMN for emotional deviancy than for neutral deviancy. In ASD, an overall atypical change detection was observed with an earlier MMN and a larger P3a compared to controls suggesting an unusual pre-attentional orientation toward any changes in the auditory environment. Moreover, in children with autism, deviancy detection depicted reduced MMN amplitude. In addition in children with autism, contrary to adults with autism, no modulation of the MMN by prosody was present and sensory processing of both neutral and emotional vocal stimuli appeared atypical.

Conclusions

Overall, change detection remains altered in people with autism. However, differences between children and adults with ASD evidence a trend toward normalization of vocal processing and of the automatic detection of emotion deviancy with age.

Neural correlates of processing emotional prosody in unipolar depression

Major depressive disorder (MDD) is characterized by a biased emotion perception. In the auditory domain, MDD patients have been shown to exhibit attenuated processing of positive emotions expressed by speech melody (prosody). So far, no neuroimaging studies examining the neural basis of altered processing of emotional prosody in MDD are available. In this study, we addressed this issue by examining the emotion bias in MDD during evaluation of happy, neutral, and angry prosodic stimuli on a five-point Likert scale during functional magnetic resonance imaging (fMRI). As expected, MDD patients rated happy prosody less intense than healthy controls (HC). At neural level, stronger activation in the middle superior temporal gyrus (STG) and the amygdala was found in all participants when processing emotional as compared to neutral prosody. MDD patients exhibited an increased activation of the amygdala during processing prosody irrespective of valence while no significant differences between groups were found for the STG, indicating that altered processing of prosodic emotions in MDD occurs rather within the amygdala than in auditory areas. Concurring with the valence-specific behavioral effect of attenuated evaluation of positive prosodic stimuli, activation within the left amygdala of MDD patients correlated with ratings of happy, but not neutral or angry prosody. Our study provides first insights in the neural basis of reduced experience of positive information and an abnormally increased amygdala activity during prosody processing.

fMRI Reveals a Novel Region for Evaluating Acoustic Information for Mate Choice in a Female Songbird

Selection of sexual partners is among the most critical decisions that individuals make and is therefore strongly shaped by evolution. In social species, where communication signals can convey substantial information about the identity, state, or quality of the signaler, accurate interpretation of communication signals for mate choice is crucial. Despite the importance of social information processing, to date, relatively little is known about the neurobiological mechanisms that contribute to sexual decision making and preferences. In this study, we used a combination of whole-brain functional magnetic resonance imaging (fMRI), immediate early gene expression, and behavior tests to identify the circuits that are important for the perception and evaluation of courtship songs in a female songbird, the zebra finch (Taeniopygia guttata). Female zebra finches are sensitive to subtle differences in male song performance and strongly prefer the longer, faster, and more stereotyped courtship songs to non-courtship renditions. Using BOLD fMRI and EGR1 expression assays, we uncovered a novel region involved in auditory perceptual decision making located in a sensory integrative region of the avian central nidopallium outside the traditionally studied auditory forebrain pathways. Changes in activity in this region in response to acoustically similar but categorically divergent stimuli showed stronger parallels to behavioral responses than an auditory sensory region. These data highlight a potential role for the caudocentral nidopallium (NCC) as a novel node in the avian circuitry underlying the evaluation of acoustic signals and their use in mate choice.

Speech Prosodies of Different Emotional Categories Activate Different Brain Regions in Adult Cortex: an fNIRS Study

Emotional expressions of others embedded in speech prosodies are important for social interactions. This study used functional near-infrared spectroscopy to investigate how speech prosodies of different emotional categories are processed in the cortex. The results demonstrated several cerebral areas critical for emotional prosody processing. We confirmed that the superior temporal cortex, especially the right middle and posterior parts of superior temporal gyrus (BA 22/42), primarily works to discriminate between emotional and neutral prosodies. Furthermore, the results suggested that categorization of emotions occurs within a high-level brain region–the frontal cortex, since the brain activation patterns were distinct when positive (happy) were contrasted to negative (fearful and angry) prosody in the left middle part of inferior frontal gyrus (BA 45) and the frontal eye field (BA8), and when angry were contrasted to neutral prosody in bilateral orbital frontal regions (BA 10/11). These findings verified and extended previous fMRI findings in adult brain and also provided a “developed version” of brain activation for our following neonatal study.

Altered emotional recognition and expression in patients with Parkinson's disease

BACKGROUND

Parkinson's disease (PD) patients exhibit deficits in emotional recognition and expression abilities, including emotional faces and voices. The aim of this study was to explore emotional processing in pre-deep brain stimulation (pre-DBS) PD patients using two sensory modalities (visual and auditory).

METHODS

Fifteen PD patients who needed DBS surgery and 15 healthy, age- and gender-matched controls were recruited as participants. All participants were assessed by the Karolinska Directed Emotional Faces database 50 Faces Recognition test. Vocal recognition was evaluated by the Montreal Affective Voices database 50 Voices Recognition test. For emotional facial expression, the participants were asked to imitate five basic emotions (neutral, happiness, anger, fear, and sadness). The subjects were required to express nonverbal vocalizations of the five basic emotions. Fifteen Chinese native speakers were recruited as decoders. We recorded the accuracy of the responses, reaction time, and confidence level.

RESULTS

For emotional recognition and expression, the PD group scored lower on both facial and vocal emotional processing than did the healthy control group. There were significant differences between the two groups in both reaction time and confidence level. A significant relationship was also found between emotional recognition and emotional expression when considering all participants between the two groups together.

CONCLUSION

The PD group exhibited poorer performance on both the recognition and expression tasks. Facial emotion deficits and vocal emotion abnormalities were associated with each other. In addition, our data allow us to speculate that emotional recognition and expression may share a common system.

Altered emotional prosody processing in patients with Parkinson's disease after subthalamic nucleus stimulation

BACKGROUND

Patients with Parkinson's disease (PD) exhibit deficits in recognizing and expressing vocal emotional prosody. The aim of this study was to explore emotional prosody processing in patients with PD shortly after subthalamic nucleus (STN) deep brain stimulation (DBS).

METHODS

Two groups of patients with PD (pre-DBS and post-DBS) and one healthy control (HC) group were recruited as participants. All participants (PD and HC) were assessed using the Montreal Affective Voices database 50 Voices Recognition test. All participants were asked to nonverbally express five basic emotions (happiness, anger, fear, sadness, and neutral) to test emotional prosody expression. Fifteen native Chinese speakers were recruited as raters. We recorded the accuracy rate, reaction time, confidence level, and two acoustic parameters (mean pitch and mean intensity).

RESULTS

The PD groups scored lower than the HC group in recognizing and expressing emotional prosody. STN DBS had no significant effect on the recognition of emotional prosody but had a significant effect on fear prosody expression. Pearson's correlation analysis revealed significant correlations between performance on emotional prosody recognition tests and performance on emotional prosody expression tests in both the pre-DBS PD and post-DBS PD groups.

CONCLUSION

Shortly after STN DBS, the ability to recognize emotional prosody was not altered, but fear expression was impaired. We identified associations between abnormalities in emotional prosody recognition and expression deficits both before and after STN DBS, indicating that the processes involved in recognizing and expressing emotional prosody may share a common system.