The Sound of Emotional Prosody: Nearly 3 Decades of Research and Future Directions

Emotional voices attract considerable attention. A search on any browser using "emotional prosody" as a key phrase leads to more than a million entries. Such interest is evident in the scientific literature as well; readers are reminded in the introductory paragraphs of countless articles of the great importance of prosody and that listeners easily infer the emotional state of speakers through acoustic information. However, despite decades of research on this topic and important achievements, the mapping between acoustics and emotional states is still unclear. In this article, we chart the rich literature on emotional prosody for both newcomers to the field and researchers seeking updates. We also summarize problems revealed by a sample of the literature of the last decades and propose concrete research directions for addressing them, ultimately to satisfy the need for more mechanistic knowledge of emotional prosody.

Stressed Speech Emotion Recognition Using Teager Energy and Spectral Feature Fusion with Feature Optimization

The objective of speech emotion recognition (SER) is to enhance man-machine interface. It can also be used to cover the physiological state of a person in critical situations. In recent time, speech emotion recognition also finds its operations in medicine and forensics. A new feature extraction technique using Teager energy operator (TEO) is proposed for the detection of stressed emotions as Teager energy-autocorrelation envelope (TEO-Auto-Env). TEO is basically designed for increasing the energies of the stressed speech signals whose energies are reduced during the speech production process and hence used in this analysis. A stressed speech emotion recognition (SSER) system is developed using TEO-Auto-Env and spectral feature combination for detecting the emotions. The spectral features considered are Mel-frequency cepstral coefficients (MFCC), linear prediction cepstral coefficients (LPCC), and relative spectra-perceptual linear prediction (RASTA-PLP). EMO-DB (German), EMOVO (Italian), IITKGP (Telugu), and EMA (English) databases are used in this analysis. The classification of the emotions is carried out using the k-nearest neighborhood (k-NN) classifier for gender-dependent (GD) and speaker-independent (SI) cases. The proposed SSER system provides improved accuracy compared to the existing ones. Average recall is used for performance evaluation. The highest classification accuracy is achieved using the feature combination of TEO-Auto-Env, MFCC, and LPCC features with 91.4% (SI), 91.4% (GD-male), and 93.1%(GD-female) for EMO-DB; 68.5% (SI), 68.5% (GD-male), and 74.6% (GD-female) for EMOVO; 90.6%(SI), 91% (GD-male), and 92.3% (GD-female) for EMA; and 95.1% (GD-female) for IITKGP female database.

PVR-AFM: A Pathological Voice Repair System based on Non-linear Structure

OBJECTIVE

Speech signal processing has become an important technique to ensure that the voice interaction system communicates accurately with the user by improving the clarity or intelligibility of speech signals. However, most existing works only focus on whether to process the voice of average human but ignore the communication needs of individuals suffering from voice disorder, including voice-related professionals, older people, and smokers. To solve this demand, it is essential to design a non-invasive repair system that processes pathological voices.

METHODS

In this paper, we propose a repair system for multiple polyp vowels, such as /a/, /i/ and /u/. We utilize a non-linear model based on amplitude-modulation (AM) and a frequency-modulation (FM) structure to extract the pitch and formant of pathological voice. To solve the fracture and instability of pitch, we provide a pitch extraction algorithm, which ensures that pitch's stability and avoids the errors of double pitch caused by the instability of low-frequency signal. Furthermore, we design a formant reconstruction mechanism, which can effectively determine the frequency and bandwidth to accomplish formant repair.

RESULTS

Finally, spectrum observation and objective indicators show that the system has better performance in improving the intelligibility of pathological speech.

Emotional Climate Recognition in Conversations using Peers' Speech-based Bispectral Features and Affect Dynamics. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023;2023:1-5. [PMID: 38083331 DOI: 10.1109/embc40787.2023.10340028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Emotion recognition in conversations using artificial intelligence (AI) has recently gained a lot of attention, as it can provide additional emotion cues that can be correlated with human social behavior. An extension towards an AI-based emotional climate (EC) recognition, i.e., the recognition of the joint emotional atmosphere dynamically created and perceived by the peers throughout a conversation, is proposed here. In our approach, namely MLBispeC (Machine Learning Based Bispectral Classification), the peers' speech signals during their conversation are subjected to time-windowed bispectral analysis, allowing for feature extraction related to dynamic harmonics nonlinear interactions. In addition, peers' affect dynamics, derived from their same time-windowed emotion labeling, are combined to form an extended feature vector, inputted into two well-known machine learning classifiers (Support Vector Machine, K-Nearest Neighbor). MLBispeC was evaluated on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) open access dataset, which contains 2D emotions, i.e., Arousal (A) and valence (V) that are divided into (low/high) classes. The experimental results have shown that MLBispeC outperforms previous state-of-the-art techniques, achieving an accuracy of 0.826A/0.754V, sensitivity of 0.864A/0.774V, and area under the curve (AUC) of 0.821A/0.799V. This demonstrates the effectiveness of MLBispeC to objectively recognize peers' EC during their conversation, allowing for insights into their emotional and social interactions.Clinical relevance-Unobtrusive, objective and dynamic recognition of the EC built during peers' conversation can scaffold effective assessment of patients with physiological, psychological, and mental diseases, at various age ranges (children, adults, and older adults).

A speech corpus of Quechua Collao for automatic dimensional emotion recognition

Automatic speech emotion recognition is an important research topic for human-computer interaction and affective computing. Over ten million people speak the Quechua language throughout South America, and one of the most known variants is the Quechua Collao one. However, this language can be considered a low resource for machine emotion recognition, creating a barrier for Quechua speakers who want to use this technology. Therefore, the contribution of this work is a 15 hours speech corpus in Quechua Collao, which is made publicly available to the research community. The corpus was created from a set of words and sentences explicitly collected for this task, divided into nine categorical emotions: happy, sad, bored, fear, sleepy, calm, excited, angry, and neutral. The annotation was performed on a 5-value discrete scale according to 3 dimensions: valence, arousal, and dominance. To demonstrate the usefulness of the corpus, we have performed speech emotion recognition using machine learning methods and neural networks.

Deep Multimodal Emotion Recognition on Human Speech: A Review

This work reviews the state of the art in multimodal speech emotion recognition methodologies, focusing on audio, text and visual information. We provide a new, descriptive categorization of methods, based on the way they handle the inter-modality and intra-modality dynamics in the temporal dimension: (i) non-temporal architectures (NTA), which do not significantly model the temporal dimension in both unimodal and multimodal interaction; (ii) pseudo-temporal architectures (PTA), which also assume an oversimplification of the temporal dimension, although in one of the unimodal or multimodal interactions; and (iii) temporal architectures (TA), which try to capture both unimodal and cross-modal temporal dependencies. In addition, we review the basic feature representation methods for each modality, and we present aggregated evaluation results on the reported methodologies. Finally, we conclude this work with an in-depth analysis of the future challenges related to validation procedures, representation learning and method robustness.

Acoustic and prosodic information for home monitoring of bipolar disorder

Epidemiological studies suggest that bipolar disorder has a prevalence of about 1% in European countries, becoming one of the most disabling illnesses in working age adults, and often long-term and persistent with complex management and treatment. Therefore, the capacity of home monitoring for patients with this disorder is crucial for their quality of life. The current paper introduces the use of speech-based information as an easy-to-record, ubiquitous and non-intrusive health sensor suitable for home monitoring, and its application in the framework on the NYMPHA-MD project. Some preliminary results also show the potential of acoustic and prosodic features to detect and classify bipolar disorder, by predicting the values of the Hamilton Depression Rating Scale (HDRS) and the Young Mania Rating Scale (YMRS) from speech.

Analyzing the vocal tract characteristics for out-of-breath speech

In this work, vocal tract characteristic changes under the out-of-breath condition are explored. Speaking under the influence of physical exercise is called out-of-breath speech. The change in breathing pattern results in perceptual changes in the produced sound. For vocal tract, the first four formants show a lowering in their average frequency. The bandwidths B_F1 and B_F2 widen, whereas the other two get narrowed. The change in bandwidth is small for the last three. For a speaker, the change in frequency and bandwidth may not be uniform across formants. Subband analysis is carried out around formants for comparing the variation of the vocal tract with the source. A vocal tract adaptive empirical wavelet transform is used for extracting formant specific subbands from speech and source. The support vector machine performs the subband-based binary classification between the normal and out-of-breath speech. For all speakers, it shows an F1-score improvement of 4% over speech subbands. Similarly, a performance improvement of 5% can be seen for both male and female speakers. Furthermore, the misclassification amount is less for source compared to speech. These results suggest that physical exercise influences the source more than the vocal tract.

Cascaded Convolutional Neural Network Architecture for Speech Emotion Recognition in Noisy Conditions

Convolutional neural networks (CNNs) are a state-of-the-art technique for speech emotion recognition. However, CNNs have mostly been applied to noise-free emotional speech data, and limited evidence is available for their applicability in emotional speech denoising. In this study, a cascaded denoising CNN (DnCNN)–CNN architecture is proposed to classify emotions from Korean and German speech in noisy conditions. The proposed architecture consists of two stages. In the first stage, the DnCNN exploits the concept of residual learning to perform denoising; in the second stage, the CNN performs the classification. The classification results for real datasets show that the DnCNN–CNN outperforms the baseline CNN in overall accuracy for both languages. For Korean speech, the DnCNN–CNN achieves an accuracy of 95.8%, whereas the accuracy of the CNN is marginally lower (93.6%). For German speech, the DnCNN–CNN has an overall accuracy of 59.3–76.6%, whereas the CNN has an overall accuracy of 39.4–58.1%. These results demonstrate the feasibility of applying the DnCNN with residual learning to speech denoising and the effectiveness of the CNN-based approach in speech emotion recognition. Our findings provide new insights into speech emotion recognition in adverse conditions and have implications for language-universal speech emotion recognition.

Discrimination of simultaneous psychological and physical stressors using wristband biosignals

BACKGROUND AND OBJECTIVE

In this work, we address the problem of detecting and discriminating acute psychological stress (APS) in the presence of concurrent physical activity (PA) using wristband biosignals. We focused on signals available from wearable devices that can be worn in daily life because the ultimate objective of this work is to provide APS and PA information in real-time management of chronic conditions such as diabetes by automated personalized insulin delivery. Monitoring APS noninvasively throughout free-living conditions remains challenging because the responses to APS and PA of many physiological variables measured by wearable devices are similar.

METHODS

Various classification algorithms are compared to simultaneously detect and discriminate the PA (sedentary state, treadmill running, and stationary bike) and the type of APS (non-stress state, mental stress, and emotional anxiety). The impact of APS inducements is verified with commonly used self-reported questionnaires (The State-Trait Anxiety Inventory (STAI)). To aid the classification algorithms, novel features are generated from the physiological variables reported by a wristband device during 117 hours of experiments involving simultaneous APS inducement and PA. We also translate the APS assessment into a quantitative metric for use in predicting the adverse outcomes.

RESULTS

An accurate classification of the concurrent PA and APS states is achieved with an overall classification accuracy of 99% for PA and 92% for APS. The average accuracy of APS detection during sedentary state, treadmill running, and stationary bike is 97.3, 94.1, and 84.5%, respectively.

CONCLUSIONS

The simultaneous assessment of APS and PA throughout free-living conditions from a convenient wristband device is useful for monitoring the factors contributing to an elevated risk of acute events in people with chronic diseases like cardiovascular complications and diabetes.

A Computerized Method for Automatic Detection of Schizophrenia Using EEG Signals

Diagnosis of schizophrenia (SZ) is traditionally performed through patient's interviews by a skilled psychiatrist. This process is time-consuming, burdensome, subject to error and bias. Hence the aim of this study is to develop an automatic SZ identification scheme using electroencephalogram (EEG) signals that can eradicate the aforementioned problems and support clinicians and researchers. This study introduces a methodology design involving empirical mode decomposition (EMD) technique for diagnosis of SZ from EEG signals to perfectly handle the behavior of non-stationary and nonlinear EEG signals. In this study, each EEG signal is decomposed into intrinsic mode functions (IMFs) by the EMD algorithm and then twenty-two statistical characteristics/features are calculated from these IMFs. Among them, five features are selected as significant feature applying Kruskal Wallis test. The performance of the obtained feature set is tested through several renowned classifierson a SZ EEG dataset. Among the considered classifiers, theensemble bagged tree performed as the best classifier producing 93.21% correct classification rate for SZ, with an overall accuracy of 89.59% for IMF 2. These results indicate that EEG signals discriminate SZ patients from healthy control (HC) subjects efficiently and have the potential to become a tool for the psychiatrist to support the positive diagnosis of SZ.

Determining Physical Activity Characteristics from Wristband Data for Use in Automated Insulin Delivery Systems

Algorithms that can determine the type of physical activity (PA) and quantify the intensity can allow precision medicine approaches, such as automated insulin delivery systems that modulate insulin administration in response to PA. In this work, data from a multi-sensor wristband is used to design classifiers to distinguish among five different physical states (PS) (resting, activities of daily living, running, biking, and resistance training), and to develop models to estimate the energy expenditure (EE) of the PA for diabetes therapy. The data collected are filtered, features are extracted from the reconciled signals, and the extracted features are used by machine learning algorithms, including deep-learning techniques, to obtain accurate PS classification and EE estimation. The various machine learning techniques have different success rates ranging from 75.7% to 94.8% in classifying the five different PS. The deep neural network model with long short-term memory has 94.8% classification accuracy. We achieved 0.5 MET (Metabolic Equivalent of Task) root-mean-square error for EE estimation accuracy, relative to indirect calorimetry with randomly selected testing data (10% of collected data). We also demonstrate a 5% improvement in PS classification accuracy and a 0.34 MET decrease in the mean absolute error when using multi-sensor approach relative to using only accelerometer data.

Speech variability: A cross-language study on acoustic variations of speaking versus untrained singing

Speech production variability introduces significant challenges for existing speech technologies such as speaker identification (SID), speaker diarization, speech recognition, and language identification (ID). There has been limited research analyzing changes in acoustic characteristics for speech produced by untrained singing versus speaking. To better understand changes in speech production of the untrained singing voice, this study presents the first cross-language comparison between normal speaking and untrained karaoke singing of the same text content. Previous studies comparing professional singing versus speaking have shown deviations in both prosodic and spectral features. Some investigations also considered assigning the intrinsic activity of the singing. Motivated by these studies, a series of experiments to investigate both prosodic and spectral variations of untrained karaoke singers for three languages, American English, Hindi, and Farsi, are considered. A comprehensive comparison on common prosodic features, including phoneme duration, mean fundamental frequency (F0), and formant center frequencies of vowels was performed. Collective changes in the corresponding overall acoustic spaces based on the Kullback-Leibler distance using Gaussian probability distribution models trained on spectral features were analyzed. Finally, these models were used in a Gausian mixture model with universal background model SID evaluation to quantify speaker changes between speaking and singing when the audio text content is the same. The experiments showed that many acoustic characteristics of untrained singing are considerably different from speaking when the text content is the same. It is suggested that these results would help advance automatic speech production normalization/compensation to improve performance of speech processing applications (e.g., speaker ID, speech recognition, and language ID).

A speech perturbation strategy based on "Lombard effect" for enhanced intelligibility for cochlear implant listeners

The goal of this study is to determine potential intelligibility benefits from Lombard speech for cochlear implant (CI) listeners in speech-in-noise conditions. "Lombard effect" (LE) is the natural response of adjusting speech production via auditory feedback due to noise exposure within acoustic environments. To evaluate intelligibility performance of natural and artificially induced Lombard speech, a corpus was generated to create natural LE from large crowd noise (LCN) exposure at 70, 80, and 90 dB sound pressure level (SPL). Clean speech was mixed with 15 and 10 dB SNR LCN and presented to five CI users. First, speech intelligibility was analyzed as a function of increasing LE and decreasing SNR. Results indicate significant improvements (p < 0.05) with Lombard speech intelligibility in noise conditions for 80 and 90 dB SPL. Next, an offline perturbation strategy was formulated to modify/perturb neutral speech so as to mimic LE through amplification of highly intelligible segments, uniform time stretching, and spectral mismatch filtering. This process effectively introduces aspects of LE into the neutral speech, with the hypothesis that this would benefit intelligibility for CI users. Significant (p < 0.01) intelligibility improvements of 13% and 16% percentage points were observed for 15 and 10 dB SNR conditions respectively for CI users. The results indicate how LE and LE-inspired acoustic and frequency-based modifications can be leveraged within signal processing to improve intelligibility of speech for CI users.

Stress measurement using speech: Recent advancements, validation issues, and ethical and privacy considerations

Life stress is a well-established risk factor for a variety of mental and physical health problems, including anxiety disorders, depression, chronic pain, heart disease, asthma, autoimmune diseases, and neurodegenerative disorders. The purpose of this article is to describe emerging approaches for assessing stress using speech, which we do by reviewing the methodological advantages of these digital health tools, and the validation, ethical, and privacy issues raised by these technologies. As we describe, it is now possible to assess stress via the speech signal using smartphones and smart speakers that employ software programs and artificial intelligence to analyze several features of speech and speech acoustics, including pitch, jitter, energy, rate, and length and number of pauses. Because these digital devices are ubiquitous, we can now assess individuals' stress levels in real time in almost any natural environment in which people speak. These technologies thus have great potential for advancing digital health initiatives that involve continuously monitoring changes in psychosocial functioning and disease risk over time. However, speech-based indices of stress have yet to be well-validated against stress biomarkers (e.g., cortisol, cytokines) that predict disease risk. In addition, acquiring speech samples raises the possibility that conversations intended to be private could one day be made public; moreover, obtaining real-time psychosocial risk information prompts ethical questions regarding how these data should be used for medical, commercial, and personal purposes. Although assessing stress using speech thus has enormous potential, there are critical validation, privacy, and ethical issues that must be addressed.

Speech Technology Progress Based on New Machine Learning Paradigm

Speech technologies have been developed for decades as a typical signal processing area, while the last decade has brought a huge progress based on new machine learning paradigms. Owing not only to their intrinsic complexity but also to their relation with cognitive sciences, speech technologies are now viewed as a prime example of interdisciplinary knowledge area. This review article on speech signal analysis and processing, corresponding machine learning algorithms, and applied computational intelligence aims to give an insight into several fields, covering speech production and auditory perception, cognitive aspects of speech communication and language understanding, both speech recognition and text-to-speech synthesis in more details, and consequently the main directions in development of spoken dialogue systems. Additionally, the article discusses the concepts and recent advances in speech signal compression, coding, and transmission, including cognitive speech coding. To conclude, the main intention of this article is to highlight recent achievements and challenges based on new machine learning paradigms that, over the last decade, had an immense impact in the field of speech signal processing.

Voice Stress Analysis: A New Framework for Voice and Effort in Human Performance

People rely on speech for communication, both in a personal and professional context, and often under different conditions of physical, cognitive and/or emotional load. Since vocalization is entirely integrated within both our central (CNS) and autonomic nervous system (ANS), a mounting number of studies have examined the relationship between voice output and the impact of stress. In the current paper, we will outline the different stages of voice output, i.e., breathing, phonation and resonance in relation to a neurovisceral integrated perspective on stress and human performance. In reviewing the function of these three stages of voice output, we will give an overview of the voice parameters encountered in studies on voice stress analysis (VSA) and review the impact of the different types of physiological, cognitive and/or emotional load. In the section "Discussion," with regard to physical load, a competition for ventilation processes required to speak and those to meet metabolic demand of exercised muscles is described. With regard to cognitive and emotional load, we will present the "Model for Voice and Effort" (MoVE) that comprises the integration of ongoing top-down and bottom-up activity under different types of load and combined patterns of voice output. In the MoVE, it is proposed that the fundamental frequency (F0) values as well as jitter give insight in bottom-up/arousal activity and the effort a subject is capable to generate but that its range and variance are related to ongoing top-down processes and the amount of control a subject can maintain. Within the MoVE, a key-role is given to the anterior cingulate cortex (ACC) which is known to be involved in both the equilibration between bottom-up arousal and top-down regulation and vocal activity. Moreover, the connectivity between the ACC and the nervus vagus (NV) is underlined as an indication of the importance of respiration. Since respiration is the driving force of both stress and voice production, it is hypothesized to be the missing-link in our understanding of the underlying mechanisms of the dynamic between speech and stress.

Distinguish self- and hetero-perceived stress through behavioral imaging and physiological features

Stress reactivity is a complex phenomenon associated to multiple and multimodal expressions. Response to stressors has an obvious survival function and may be seen as an internal regulation to adapt to threat or danger. The intensity of this internal response can be assessed as the self-perception of the stress response. In species with social organization, this response also serves a communicative function, so-called hetero-perception. Our study presents multimodal stress detection assessment - a new methodology combining behavioral imaging and physiological monitoring for analyzing stress from these two perspectives. The system is based on automatic extraction of 39 behavioral (2D+3D video recording) and 62 physiological (Nexus-10 recording) features during a socially evaluated mental arithmetic test. The analysis with machine learning techniques for automatic classification using Support Vector Machine (SVM) show that self-perception and hetero-perception of social stress are both close but different phenomena: self-perception was significantly correlated with hetero-perception but significantly differed from it. Also, assessing stress with SVM through multimodality gave excellent classification results (F1 score values: 0.9±0.012 for hetero-perception and 0.87±0.021 for self-perception). In the best selected feature subsets, we found some common behavioral and physiological features that allow classification of both self- and hetero-perceived stress. However, we also found the contributing features for automatic classifications had opposite distributions: self-perception classification was mainly based on physiological features and hetero-perception was mainly based on behavioral features.

Multiscale Amplitude Feature and Significance of Enhanced Vocal Tract Information for Emotion Classification

In this paper, a novel multiscale amplitude feature is proposed using multiresolution analysis (MRA) and the significance of the vocal tract is investigated for emotion classification from the speech signal. MRA decomposes the speech signal into number of sub-band signals. The proposed feature is computed by using sinusoidal model on each sub-band signal. Different emotions have different impacts on the vocal tract. As a result, vocal tract responds in a unique way for each emotion. The vocal tract information is enhanced using pre-emphasis. Therefore, emotion information manifested in the vocal tract can be well exploited. This may help in improving the performance of emotion classification. Emotion recognition is performed using German emotional EMODB database, interactive emotional dyadic motion capture database, simulated stressed speech database, and FAU AIBO database with speech signal and speech with enhanced vocal tract information (SEVTI). The performance of the proposed multiscale amplitude feature is compared with three different types of features: 1) the mel frequency cepstral coefficients; 2) the Teager energy operator (TEO)-based feature (TEO-CB-Auto-Env); and 3) the breathinesss feature. The proposed feature outperforms the other features. In terms of recognition rates, the features derived from the SEVTI signal, give better performance compared to the features derived from the speech signal. Combination of the features with SEVTI signal shows average recognition rate of 86.7% using EMODB database.

Employing Emotion Cues to Verify Speakers in Emotional Talking Environments

Usually, people talk neutrally in environments where there are no abnormal talking conditions such as stress and emotion. Other emotional conditions that might affect people’s talking tone include happiness, anger, and sadness. Such emotions are directly affected by the patient’s health status. In neutral talking environments, speakers can be easily verified; however, in emotional talking environments, speakers cannot be easily verified as in neutral talking ones. Consequently, speaker verification systems do not perform well in emotional talking environments as they do in neutral talking environments. In this work, a two-stage approach has been employed and evaluated to improve speaker verification performance in emotional talking environments. This approach employs speaker’s emotion cues (text-independent and emotion-dependent speaker verification problem) based on both hidden Markov models (HMMs) and suprasegmental HMMs as classifiers. The approach is composed of two cascaded stages that combine and integrate an emotion recognizer and a speaker recognizer into one recognizer. The architecture has been tested on two different and separate emotional speech databases: our collected database and the Emotional Prosody Speech and Transcripts database. The results of this work show that the proposed approach gives promising results with a significant improvement over previous studies and other approaches such as emotion-independent speaker verification approach and emotion-dependent speaker verification approach based completely on HMMs.

Expanding the functionality of speech recognition in radiology: creating a real-time methodology for measurement and analysis of occupational stress and fatigue

While occupational stress and fatigue have been well described throughout medicine, the radiology community is particularly susceptible due to declining reimbursements, heightened demands for service deliverables, and increasing exam volume and complexity. The resulting occupational stress can be variable in nature and dependent upon a number of intrinsic and extrinsic stressors. Intrinsic stressors largely account for inter-radiologist stress variability and relate to unique attributes of the radiologist such as personality, emotional state, education/training, and experience. Extrinsic stressors may account for intra-radiologist stress variability and include cumulative workload and task complexity. The creation of personalized stress profiles creates a mechanism for accounting for both inter- and intra-radiologist stress variability, which is essential in creating customizable stress intervention strategies. One viable option for real-time occupational stress measurement is voice stress analysis, which can be directly implemented through existing speech recognition technology and has been proven to be effective in stress measurement and analysis outside of medicine. This technology operates by detecting stress in the acoustic properties of speech through a number of different variables including duration, glottis source factors, pitch distribution, spectral structure, and intensity. The correlation of these speech derived stress measures with outcomes data can be used to determine the user-specific inflection point at which stress becomes detrimental to clinical performance.

Objective measures, sensors and computational techniques for stress recognition and classification: a survey

Stress is a major growing concern in our day and age adversely impacting both individuals and society. Stress research has a wide range of benefits from improving personal operations, learning, and increasing work productivity to benefiting society - making it an interesting and socially beneficial area of research. This survey reviews sensors that have been used to measure stress and investigates techniques for modelling stress. It discusses non-invasive and unobtrusive sensors for measuring computed stress, a term we coin in the paper. Sensors that do not impede everyday activities that could be used by those who would like to monitor stress levels on a regular basis (e.g. vehicle drivers, patients with illnesses linked to stress) is the focus of the discussion. Computational techniques have the capacity to determine optimal sensor fusion and automate data analysis for stress recognition and classification. Several computational techniques have been developed to model stress based on techniques such as Bayesian networks, artificial neural networks, and support vector machines, which this survey investigates. The survey concludes with a summary and provides possible directions for further computational stress research.

Multichannel weighted speech classification system for prediction of major depression in adolescents

Early identification of adolescents at high imminent risk for clinical depression could significantly reduce the burden of the disease. This study demonstrated that acoustic speech analysis and classification can be used to determine early signs of major depression in adolescents, up to two years before they meet clinical diagnostic criteria for the full-blown disorder. Individual contributions of four different types of acoustic parameters [prosodic, glottal, Teager's energy operator (TEO), and spectral] to depression-related changes of speech characteristics were examined. A new computational methodology for the early prediction of depression in adolescents was developed and tested. The novel aspect of this methodology is in the introduction of multichannel classification with a weighted decision procedure. It was observed that single-channel classification was effective in predicting depression with a desirable specificity-to-sensitivity ratio and accuracy higher than chance level only when using glottal or prosodic features. The best prediction performance was achieved with the new multichannel method, which used four features (prosodic, glottal, TEO, and spectral). In the case of the person-based approach with two sets of weights, the new multichannel method provided a high accuracy level of 73% and the sensitivity-to-specificity ratio of 79%/67% for predicting future depression.

A new discriminant NMF algorithm and its application to the extraction of subtle emotional differences in speech

In this study we propose a new feature extraction algorithm, dNMF (discriminant non-negative matrix factorization), to learn subtle class-related differences while maintaining an accurate generative capability. In addition to the minimum representation error for the standard NMF (non-negative matrix factorization) algorithm, the dNMF algorithm also results in higher between-class variance for discriminant power. The multiplicative NMF learning algorithm has been modified to cope with this additional constraint. The cost function was carefully designed so that the extraction of feature coefficients from a single testing pattern with pre-trained feature vectors resulted in a quadratic convex optimization problem in non-negative space for uniqueness. It also resolves issues related to the previous discriminant NMF algorithms. The developed dNMF algorithm has been applied to the emotion recognition task for speech, where it needs to emphasize the emotional differences while de-emphasizing the dominant phonetic components. The dNMF algorithm successfully extracted subtle emotional differences, demonstrated much better recognition performance and showed a smaller representation error from an emotional speech database.

Detection of clinical depression in adolescents' speech during family interactions

The properties of acoustic speech have previously been investigated as possible cues for depression in adults. However, these studies were restricted to small populations of patients and the speech recordings were made during patients' clinical interviews or fixed-text reading sessions. Symptoms of depression often first appear during adolescence at a time when the voice is changing, in both males and females, suggesting that specific studies of these phenomena in adolescent populations are warranted. This study investigated acoustic correlates of depression in a large sample of 139 adolescents (68 clinically depressed and 71 controls). Speech recordings were made during naturalistic interactions between adolescents and their parents. Prosodic, cepstral, spectral, and glottal features, as well as features derived from the Teager energy operator (TEO), were tested within a binary classification framework. Strong gender differences in classification accuracy were observed. The TEO-based features clearly outperformed all other features and feature combinations, providing classification accuracy ranging between 81%-87% for males and 72%-79% for females. Close, but slightly less accurate, results were obtained by combining glottal features with prosodic and spectral features (67%-69% for males and 70%-75% for females). These findings indicate the importance of nonlinear mechanisms associated with the glottal flow formation as cues for clinical depression.

Application of poincare-mapping of voiced-speech segments for emotion sensing

The following paper introduces a group of novel speech-signal descriptors that reflect phoneme-pronunciation variability and that can be considered as potentially useful features for emotion sensing. The proposed group includes a set of statistical parameters of Poincare maps, derived for formant-frequency evolution and energy evolution of voiced-speech segments. Two groups of Poincare-map characteristics were considered in the research: descriptors of sample-scatter, which reflect magnitudes of phone-uttering variations and descriptors of cross-correlations that exist among samples and that evaluate consistency of variations. It has been shown that inclusion of the proposed characteristics into the pool of commonly used speech descriptors, results in a noticeable increase-at the level of 10%-in emotion sensing performance. Standard pattern recognition methodology has been adopted for evaluation of the proposed descriptors, with the assumption that three- or four-dimensional feature spaces can provide sufficient emotion sensing. Binary decision trees have been selected for data classification, as they provide with detailed information on emotion-specific discriminative power of various speech descriptors.