Nonverbal auditory communication - Evidence for integrated neural systems for voice signal production and perception

Evaluating speech latencies during structured psychiatric interviews as an automated objective measure of psychomotor slowing

We sought to derive an objective measure of psychomotor slowing from speech analytics during a psychiatric interview to avoid potential burden of dedicated neurophysiological testing. Speech latency, which reflects response time between speakers, shows promise from the literature. Speech data was obtained from 274 subjects with a diagnosis of bipolar I depression enrolled in a randomized, doubleblind, 6-week phase 2 clinical trial. Audio recordings of structured Montgomery-Åsberg Depression Rating Scale (MADRS) interviews at 6 time points were examined (k = 1,352). We evaluated speech latencies, and other aspects of speech, for temporal stability, convergent validity, sensitivity/responsivity to clinical change, and generalization across seven socio-linguistically diverse countries. Speech latency was minimally associated with demographic features, and explained nearly a third of the variance in depression (categorically defined). Speech latency significantly decreased as depression symptoms improved over time, explaining nearly 20 % of variance in depression remission. Classification for differentiating people with versus without concurrent depression was high (AUCs > 0.85) both cross-sectionally and longitudinally. Results replicated across countries. Other speech features offered modest incremental contribution. Neurophysiological speech parameters with face validity can be derived from psychiatric interviews without the added patient burden of additional testing.

Organic Optoelectronic Synapses for Sound Perception

The neuromorphic systems for sound perception is under highly demanding for the future bioinspired electronics and humanoid robots. However, the sound perception based on volume, tone and timbre remains unknown. Herein, organic optoelectronic synapses (OOSs) are constructed for unprecedented sound recognition. The volume, tone and timbre of sound can be regulated appropriately by the input signal of voltages, frequencies and light intensities of OOSs, according to the amplitude, frequency, and waveform of the sound. The quantitative relation between recognition factor (ζ) and postsynaptic current (I = Ilight - Idark) is established to achieve sound perception. Interestingly, the bell sound for University of Chinese Academy of Sciences is recognized with an accuracy of 99.8%. The mechanism studies reveal that the impedance of the interfacial layers play a critical role in the synaptic performances. This contribution presents unprecedented artificial synapses for sound perception at hardware levels.

Psychopathic and autistic traits differentially influence the neural mechanisms of social cognition from communication signals

Psychopathy is associated with severe deviations in social behavior and cognition. While previous research described such cognitive and neural alterations in the processing of rather specific social information from human expressions, some open questions remain concerning central and differential neurocognitive deficits underlying psychopathic behavior. Here we investigated three rather unexplored factors to explain these deficits, first, by assessing psychopathy subtypes in social cognition, second, by investigating the discrimination of social communication sounds (speech, non-speech) from other non-social sounds, and third, by determining the neural overlap in social cognition impairments with autistic traits, given potential common deficits in the processing of communicative voice signals. The study was exploratory with a focus on how psychopathic and autistic traits differentially influence the function of social cognitive and affective brain networks in response to social voice stimuli. We used a parametric data analysis approach from a sample of 113 participants (47 male, 66 female) with ages ranging between 18 and 40 years (mean 25.59, SD 4.79). Our data revealed four important findings. First, we found a phenotypical overlap between secondary but not primary psychopathy with autistic traits. Second, primary psychopathy showed various neural deficits in neural voice processing nodes (speech, non-speech voices) and in brain systems for social cognition (mirroring, mentalizing, empathy, emotional contagion). Primary psychopathy also showed deficits in the basal ganglia (BG) system that seems specific to the social decoding of communicative voice signals. Third, neural deviations in secondary psychopathy were restricted to social mirroring and mentalizing impairments, but with additional and so far undescribed deficits at the level of auditory sensory processing, potentially concerning deficits in ventral auditory stream mechanisms (auditory object identification). Fourth, high autistic traits also revealed neural deviations in sensory cortices, but rather in the dorsal auditory processing streams (communicative context encoding). Taken together, social cognition of voice signals shows considerable deviations in psychopathy, with differential and newly described deficits in the BG system in primary psychopathy and at the neural level of sensory processing in secondary psychopathy. These deficits seem especially triggered during the social cognition from vocal communication signals.

A Comparison of Machine Learning Algorithms and Feature Sets for Automatic Vocal Emotion Recognition in Speech

Vocal emotion recognition (VER) in natural speech, often referred to as speech emotion recognition (SER), remains challenging for both humans and computers. Applied fields including clinical diagnosis and intervention, social interaction research or Human Computer Interaction (HCI) increasingly benefit from efficient VER algorithms. Several feature sets were used with machine-learning (ML) algorithms for discrete emotion classification. However, there is no consensus for which low-level-descriptors and classifiers are optimal. Therefore, we aimed to compare the performance of machine-learning algorithms with several different feature sets. Concretely, seven ML algorithms were compared on the Berlin Database of Emotional Speech: Multilayer Perceptron Neural Network (MLP), J48 Decision Tree (DT), Support Vector Machine with Sequential Minimal Optimization (SMO), Random Forest (RF), k-Nearest Neighbor (KNN), Simple Logistic Regression (LOG) and Multinomial Logistic Regression (MLR) with 10-fold cross validation using four openSMILE feature sets (i.e., IS-09, emobase, GeMAPS and eGeMAPS). Results indicated that SMO, MLP and LOG show better performance (reaching to 87.85%, 84.00% and 83.74% accuracies, respectively) compared to RF, DT, MLR and KNN (with minimum 73.46%, 53.08%, 70.65% and 58.69% accuracies, respectively). Overall, the emobase feature set performed best. We discuss the implications of these findings for applications in diagnosis, intervention or HCI.

Enhancing socio-emotional communication and quality of life in young cochlear implant recipients: Perspectives from parameter-specific morphing and caricaturing

The use of digitally modified stimuli with enhanced diagnostic information to improve verbal communication in children with sensory or central handicaps was pioneered by Tallal and colleagues in 1996, who targeted speech comprehension in language-learning impaired children. Today, researchers are aware that successful communication cannot be reduced to linguistic information—it depends strongly on the quality of communication, including non-verbal socio-emotional communication. In children with cochlear implants (CIs), quality of life (QoL) is affected, but this can be related to the ability to recognize emotions in a voice rather than speech comprehension alone. In this manuscript, we describe a family of new methods, termed parameter-specific facial and vocal morphing. We propose that these provide novel perspectives for assessing sensory determinants of human communication, but also for enhancing socio-emotional communication and QoL in the context of sensory handicaps, via training with digitally enhanced, caricatured stimuli. Based on promising initial results with various target groups including people with age-related macular degeneration, people with low abilities to recognize faces, older people, and adult CI users, we discuss chances and challenges for perceptual training interventions for young CI users based on enhanced auditory stimuli, as well as perspectives for CI sound processing technology.

Parameter-Specific Morphing Reveals Contributions of Timbre to the Perception of Vocal Emotions in Cochlear Implant Users

Objectives:

Research on cochlear implants (CIs) has focused on speech comprehension, with little research on perception of vocal emotions. We compared emotion perception in CI users and normal-hearing (NH) individuals, using parameter-specific voice morphing.

Design:

Twenty-five CI users and 25 NH individuals (matched for age and gender) performed fearful-angry discriminations on bisyllabic pseudoword stimuli from morph continua across all acoustic parameters (Full), or across selected parameters (F0, Timbre, or Time information), with other parameters set to a noninformative intermediate level.

Results:

Unsurprisingly, CI users as a group showed lower performance in vocal emotion perception overall. Importantly, while NH individuals used timbre and fundamental frequency (F0) information to equivalent degrees, CI users were far more efficient in using timbre (compared to F0) information for this task. Thus, under the conditions of this task, CIs were inefficient in conveying emotion based on F0 alone. There was enormous variability between CI users, with low performers responding close to guessing level. Echoing previous research, we found that better vocal emotion perception was associated with better quality of life ratings.

Conclusions:

Some CI users can utilize timbre cues remarkably well when perceiving vocal emotions.

Contributions of fundamental frequency and timbre to vocal emotion perception and their electrophysiological correlates

Our ability to infer a speaker's emotional state depends on the processing of acoustic parameters such as fundamental frequency (F0) and timbre. Yet, how these parameters are processed and integrated to inform emotion perception remains largely unknown. Here we pursued this issue using a novel parameter-specific voice morphing technique to create stimuli with emotion modulations in only F0 or only timbre. We used these stimuli together with fully modulated vocal stimuli in an event-related potential (ERP) study in which participants listened to and identified stimulus emotion. ERPs (P200 and N400) and behavioral data converged in showing that both F0 and timbre support emotion processing but do so differently for different emotions: Whereas F0 was most relevant for responses to happy, fearful and sad voices, timbre was most relevant for responses to voices expressing pleasure. Together, these findings offer original insights into the relative significance of different acoustic parameters for early neuronal representations of speaker emotion and show that such representations are predictive of subsequent evaluative judgments.

Vocal emotion adaptation aftereffects within and across speaker genders: Roles of timbre and fundamental frequency

While the human perceptual system constantly adapts to the environment, some of the underlying mechanisms are still poorly understood. For instance, although previous research demonstrated perceptual aftereffects in emotional voice adaptation, the contribution of different vocal cues to these effects is unclear. In two experiments, we used parameter-specific morphing of adaptor voices to investigate the relative roles of fundamental frequency (F0) and timbre in vocal emotion adaptation, using angry and fearful utterances. Participants adapted to voices containing emotion-specific information in either F0 or timbre, with all other parameters kept constant at an intermediate 50% morph level. Full emotional voices and ambiguous voices were used as reference conditions. All adaptor stimuli were either of the same (Experiment 1) or opposite speaker gender (Experiment 2) of subsequently presented target voices. In Experiment 1, we found consistent aftereffects in all adaptation conditions. Crucially, aftereffects following timbre adaptation were much larger than following F0 adaptation and were only marginally smaller than those following full adaptation. In Experiment 2, adaptation aftereffects appeared massively and proportionally reduced, with differences between morph types being no longer significant. These results suggest that timbre plays a larger role than F0 in vocal emotion adaptation, and that vocal emotion adaptation is compromised by eliminating gender-correspondence between adaptor and target stimuli. Our findings also add to mounting evidence suggesting a major role of timbre in auditory adaptation.

The evolutionary benefit of less-credible affective musical signals for emotion induction during storytelling

The credible signaling theory underexplains the evolutionary added value of less-credible affective musical signals compared to vocal signals. The theory might be extended to account for the motivation for, and consequences of, culturally decontextualizing a biologically contextualized signal. Musical signals are twofold, communicating "emotional fiction" alongside biological meaning, and could have filled an adaptive need for affect induction during storytelling.

Auditory cortical micro-networks show differential connectivity during voice and speech processing in humans

The temporal voice areas (TVAs) in bilateral auditory cortex (AC) appear specialized for voice processing. Previous research assumed a uniform functional profile for the TVAs which are broadly spread along the bilateral AC. Alternatively, the TVAs might comprise separate AC nodes controlling differential neural functions for voice and speech decoding, organized as local micro-circuits. To investigate micro-circuits, we modeled the directional connectivity between TVA nodes during voice processing in humans while acquiring brain activity using neuroimaging. Results show several bilateral AC nodes for general voice decoding (speech and non-speech voices) and for speech decoding in particular. Furthermore, non-hierarchical and differential bilateral AC networks manifest distinct excitatory and inhibitory pathways for voice and speech processing. Finally, while voice and speech processing seem to have distinctive but integrated neural circuits in the left AC, the right AC reveals disintegrated neural circuits for both sounds. Altogether, we demonstrate a functional heterogeneity in the TVAs for voice decoding based on local micro-circuits.

Neurocognitive processing efficiency for discriminating human non-alarm rather than alarm scream calls

Across many species, scream calls signal the affective significance of events to other agents. Scream calls were often thought to be of generic alarming and fearful nature, to signal potential threats, with instantaneous, involuntary, and accurate recognition by perceivers. However, scream calls are more diverse in their affective signaling nature than being limited to fearfully alarming a threat, and thus the broader sociobiological relevance of various scream types is unclear. Here we used 4 different psychoacoustic, perceptual decision-making, and neuroimaging experiments in humans to demonstrate the existence of at least 6 psychoacoustically distinctive types of scream calls of both alarming and non-alarming nature, rather than there being only screams caused by fear or aggression. Second, based on perceptual and processing sensitivity measures for decision-making during scream recognition, we found that alarm screams (with some exceptions) were overall discriminated the worst, were responded to the slowest, and were associated with a lower perceptual sensitivity for their recognition compared with non-alarm screams. Third, the neural processing of alarm compared with non-alarm screams during an implicit processing task elicited only minimal neural signal and connectivity in perceivers, contrary to the frequent assumption of a threat processing bias of the primate neural system. These findings show that scream calls are more diverse in their signaling and communicative nature in humans than previously assumed, and, in contrast to a commonly observed threat processing bias in perceptual discriminations and neural processes, we found that especially non-alarm screams, and positive screams in particular, seem to have higher efficiency in speeded discriminations and the implicit neural processing of various scream types in humans.