1
|
Larrouy-Maestri P, Poeppel D, Pell MD. The Sound of Emotional Prosody: Nearly 3 Decades of Research and Future Directions. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2024:17456916231217722. [PMID: 38232303 DOI: 10.1177/17456916231217722] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/19/2024]
Abstract
Emotional voices attract considerable attention. A search on any browser using "emotional prosody" as a key phrase leads to more than a million entries. Such interest is evident in the scientific literature as well; readers are reminded in the introductory paragraphs of countless articles of the great importance of prosody and that listeners easily infer the emotional state of speakers through acoustic information. However, despite decades of research on this topic and important achievements, the mapping between acoustics and emotional states is still unclear. In this article, we chart the rich literature on emotional prosody for both newcomers to the field and researchers seeking updates. We also summarize problems revealed by a sample of the literature of the last decades and propose concrete research directions for addressing them, ultimately to satisfy the need for more mechanistic knowledge of emotional prosody.
Collapse
Affiliation(s)
- Pauline Larrouy-Maestri
- Max Planck Institute for Empirical Aesthetics, Frankfurt, Germany
- School of Communication Sciences and Disorders, McGill University
- Max Planck-NYU Center for Language, Music, and Emotion, New York, New York
| | - David Poeppel
- Max Planck-NYU Center for Language, Music, and Emotion, New York, New York
- Department of Psychology and Center for Neural Science, New York University
- Ernst Strüngmann Institute for Neuroscience, Frankfurt, Germany
| | - Marc D Pell
- School of Communication Sciences and Disorders, McGill University
- Centre for Research on Brain, Language, and Music, Montreal, Quebec, Canada
| |
Collapse
|
2
|
Bandela SR, Siva Priyanka S, Sunil Kumar K, Vijay Bhaskar Reddy Y, Berhanu AA. Stressed Speech Emotion Recognition Using Teager Energy and Spectral Feature Fusion with Feature Optimization. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2023; 2023:5765760. [PMID: 37868755 PMCID: PMC10586421 DOI: 10.1155/2023/5765760] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/04/2022] [Accepted: 07/04/2022] [Indexed: 10/24/2023]
Abstract
The objective of speech emotion recognition (SER) is to enhance man-machine interface. It can also be used to cover the physiological state of a person in critical situations. In recent time, speech emotion recognition also finds its operations in medicine and forensics. A new feature extraction technique using Teager energy operator (TEO) is proposed for the detection of stressed emotions as Teager energy-autocorrelation envelope (TEO-Auto-Env). TEO is basically designed for increasing the energies of the stressed speech signals whose energies are reduced during the speech production process and hence used in this analysis. A stressed speech emotion recognition (SSER) system is developed using TEO-Auto-Env and spectral feature combination for detecting the emotions. The spectral features considered are Mel-frequency cepstral coefficients (MFCC), linear prediction cepstral coefficients (LPCC), and relative spectra-perceptual linear prediction (RASTA-PLP). EMO-DB (German), EMOVO (Italian), IITKGP (Telugu), and EMA (English) databases are used in this analysis. The classification of the emotions is carried out using the k-nearest neighborhood (k-NN) classifier for gender-dependent (GD) and speaker-independent (SI) cases. The proposed SSER system provides improved accuracy compared to the existing ones. Average recall is used for performance evaluation. The highest classification accuracy is achieved using the feature combination of TEO-Auto-Env, MFCC, and LPCC features with 91.4% (SI), 91.4% (GD-male), and 93.1%(GD-female) for EMO-DB; 68.5% (SI), 68.5% (GD-male), and 74.6% (GD-female) for EMOVO; 90.6%(SI), 91% (GD-male), and 92.3% (GD-female) for EMA; and 95.1% (GD-female) for IITKGP female database.
Collapse
Affiliation(s)
| | | | | | | | - Afework Aemro Berhanu
- Department of Environmental Engineering, College of Biological and Chemical Engineering Addis Ababa Science and Technology University, Addis Ababa, Ethiopia
| |
Collapse
|
3
|
Zhang T, Liu X, Liu G, Shao Y. PVR-AFM: A Pathological Voice Repair System based on Non-linear Structure. J Voice 2023; 37:648-662. [PMID: 37717981 DOI: 10.1016/j.jvoice.2021.05.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2021] [Revised: 05/05/2021] [Accepted: 05/06/2021] [Indexed: 11/21/2022]
Abstract
OBJECTIVE Speech signal processing has become an important technique to ensure that the voice interaction system communicates accurately with the user by improving the clarity or intelligibility of speech signals. However, most existing works only focus on whether to process the voice of average human but ignore the communication needs of individuals suffering from voice disorder, including voice-related professionals, older people, and smokers. To solve this demand, it is essential to design a non-invasive repair system that processes pathological voices. METHODS In this paper, we propose a repair system for multiple polyp vowels, such as /a/, /i/ and /u/. We utilize a non-linear model based on amplitude-modulation (AM) and a frequency-modulation (FM) structure to extract the pitch and formant of pathological voice. To solve the fracture and instability of pitch, we provide a pitch extraction algorithm, which ensures that pitch's stability and avoids the errors of double pitch caused by the instability of low-frequency signal. Furthermore, we design a formant reconstruction mechanism, which can effectively determine the frequency and bandwidth to accomplish formant repair. RESULTS Finally, spectrum observation and objective indicators show that the system has better performance in improving the intelligibility of pathological speech.
Collapse
Affiliation(s)
- Tao Zhang
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China, 300072
| | - Xiaonan Liu
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China, 300072
| | - Ganjun Liu
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China, 300072.
| | - Yangyang Shao
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China, 300072
| |
Collapse
|
4
|
Alhussein G, Alkhodari M, Lamprou C, Ziogas I, Ganiti-Roumeliotou E, Khandoker A, Hadjileontiadis LJ. Emotional Climate Recognition in Conversations using Peers' Speech-based Bispectral Features and Affect Dynamics . ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-5. [PMID: 38083331 DOI: 10.1109/embc40787.2023.10340028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Emotion recognition in conversations using artificial intelligence (AI) has recently gained a lot of attention, as it can provide additional emotion cues that can be correlated with human social behavior. An extension towards an AI-based emotional climate (EC) recognition, i.e., the recognition of the joint emotional atmosphere dynamically created and perceived by the peers throughout a conversation, is proposed here. In our approach, namely MLBispeC (Machine Learning Based Bispectral Classification), the peers' speech signals during their conversation are subjected to time-windowed bispectral analysis, allowing for feature extraction related to dynamic harmonics nonlinear interactions. In addition, peers' affect dynamics, derived from their same time-windowed emotion labeling, are combined to form an extended feature vector, inputted into two well-known machine learning classifiers (Support Vector Machine, K-Nearest Neighbor). MLBispeC was evaluated on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) open access dataset, which contains 2D emotions, i.e., Arousal (A) and valence (V) that are divided into (low/high) classes. The experimental results have shown that MLBispeC outperforms previous state-of-the-art techniques, achieving an accuracy of 0.826A/0.754V, sensitivity of 0.864A/0.774V, and area under the curve (AUC) of 0.821A/0.799V. This demonstrates the effectiveness of MLBispeC to objectively recognize peers' EC during their conversation, allowing for insights into their emotional and social interactions.Clinical relevance-Unobtrusive, objective and dynamic recognition of the EC built during peers' conversation can scaffold effective assessment of patients with physiological, psychological, and mental diseases, at various age ranges (children, adults, and older adults).
Collapse
|
5
|
Natarajan I, Shanmugam M, Dhanalakshmi S, Easwaramoorthy S, Kuppusamy S, Balu S. Longitudinal Investigation of Work Stressors Using Human Voice Features. ACTA INFORMATICA PRAGENSIA 2023. [DOI: 10.18267/j.aip.208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/09/2023] Open
|
6
|
Paccotacya-Yanque RYG, Huanca-Anquise CA, Escalante-Calcina J, Ramos-Lovón WR, Cuno-Parari ÁE. A speech corpus of Quechua Collao for automatic dimensional emotion recognition. Sci Data 2022; 9:778. [PMID: 36566260 PMCID: PMC9789950 DOI: 10.1038/s41597-022-01855-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Accepted: 11/21/2022] [Indexed: 12/25/2022] Open
Abstract
Automatic speech emotion recognition is an important research topic for human-computer interaction and affective computing. Over ten million people speak the Quechua language throughout South America, and one of the most known variants is the Quechua Collao one. However, this language can be considered a low resource for machine emotion recognition, creating a barrier for Quechua speakers who want to use this technology. Therefore, the contribution of this work is a 15 hours speech corpus in Quechua Collao, which is made publicly available to the research community. The corpus was created from a set of words and sentences explicitly collected for this task, divided into nine categorical emotions: happy, sad, bored, fear, sleepy, calm, excited, angry, and neutral. The annotation was performed on a 5-value discrete scale according to 3 dimensions: valence, arousal, and dominance. To demonstrate the usefulness of the corpus, we have performed speech emotion recognition using machine learning methods and neural networks.
Collapse
Affiliation(s)
- Rosa Y. G. Paccotacya-Yanque
- grid.441685.a0000 0004 0385 0297Universidad Nacional de San Agustín de Arequipa, School of Computer Science, Arequipa, Peru
| | - Candy A. Huanca-Anquise
- grid.441685.a0000 0004 0385 0297Universidad Nacional de San Agustín de Arequipa, School of Computer Science, Arequipa, Peru
| | - Judith Escalante-Calcina
- grid.441685.a0000 0004 0385 0297Universidad Nacional de San Agustín de Arequipa, School of Computer Science, Arequipa, Peru
| | - Wilber R. Ramos-Lovón
- grid.441685.a0000 0004 0385 0297Universidad Nacional de San Agustín de Arequipa, School of Computer Science, Arequipa, Peru
| | - Álvaro E. Cuno-Parari
- grid.441685.a0000 0004 0385 0297Universidad Nacional de San Agustín de Arequipa, School of Computer Science, Arequipa, Peru
| |
Collapse
|
7
|
Deb S, Dandapat S. Analysis of out-of-breath speech for assessment of person’s physical fitness. COMPUT SPEECH LANG 2022. [DOI: 10.1016/j.csl.2022.101391] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/01/2022]
|
8
|
Depression detection based on nonlinear and linear speech features in I-vector/SVDA framework. Comput Biol Med 2022; 149:105926. [DOI: 10.1016/j.compbiomed.2022.105926] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2022] [Revised: 07/07/2022] [Accepted: 07/30/2022] [Indexed: 11/18/2022]
|
9
|
Improving the potential of Enhanced Teager Energy Cepstral Coefficients (ETECC) for replay attack detection. COMPUT SPEECH LANG 2022. [DOI: 10.1016/j.csl.2021.101281] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
|
10
|
Novel hybrid DNN approaches for speaker verification in emotional and stressful talking environments. Neural Comput Appl 2021. [DOI: 10.1007/s00521-021-06226-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
11
|
Abstract
This work reviews the state of the art in multimodal speech emotion recognition methodologies, focusing on audio, text and visual information. We provide a new, descriptive categorization of methods, based on the way they handle the inter-modality and intra-modality dynamics in the temporal dimension: (i) non-temporal architectures (NTA), which do not significantly model the temporal dimension in both unimodal and multimodal interaction; (ii) pseudo-temporal architectures (PTA), which also assume an oversimplification of the temporal dimension, although in one of the unimodal or multimodal interactions; and (iii) temporal architectures (TA), which try to capture both unimodal and cross-modal temporal dependencies. In addition, we review the basic feature representation methods for each modality, and we present aggregated evaluation results on the reported methodologies. Finally, we conclude this work with an in-depth analysis of the future challenges related to validation procedures, representation learning and method robustness.
Collapse
|
12
|
Farrús M, Codina-Filbà J, Escudero J. Acoustic and prosodic information for home monitoring of bipolar disorder. Health Informatics J 2021; 27:1460458220972755. [PMID: 33438502 DOI: 10.1177/1460458220972755] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Epidemiological studies suggest that bipolar disorder has a prevalence of about 1% in European countries, becoming one of the most disabling illnesses in working age adults, and often long-term and persistent with complex management and treatment. Therefore, the capacity of home monitoring for patients with this disorder is crucial for their quality of life. The current paper introduces the use of speech-based information as an easy-to-record, ubiquitous and non-intrusive health sensor suitable for home monitoring, and its application in the framework on the NYMPHA-MD project. Some preliminary results also show the potential of acoustic and prosodic features to detect and classify bipolar disorder, by predicting the values of the Hamilton Depression Rating Scale (HDRS) and the Young Mania Rating Scale (YMRS) from speech.
Collapse
|
13
|
Sahoo S, Dandapat S. Analyzing the vocal tract characteristics for out-of-breath speech. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA 2021; 150:1524. [PMID: 34470262 DOI: 10.1121/10.0005945] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/26/2021] [Accepted: 08/03/2021] [Indexed: 06/13/2023]
Abstract
In this work, vocal tract characteristic changes under the out-of-breath condition are explored. Speaking under the influence of physical exercise is called out-of-breath speech. The change in breathing pattern results in perceptual changes in the produced sound. For vocal tract, the first four formants show a lowering in their average frequency. The bandwidths BF1 and BF2 widen, whereas the other two get narrowed. The change in bandwidth is small for the last three. For a speaker, the change in frequency and bandwidth may not be uniform across formants. Subband analysis is carried out around formants for comparing the variation of the vocal tract with the source. A vocal tract adaptive empirical wavelet transform is used for extracting formant specific subbands from speech and source. The support vector machine performs the subband-based binary classification between the normal and out-of-breath speech. For all speakers, it shows an F1-score improvement of 4% over speech subbands. Similarly, a performance improvement of 5% can be seen for both male and female speakers. Furthermore, the misclassification amount is less for source compared to speech. These results suggest that physical exercise influences the source more than the vocal tract.
Collapse
Affiliation(s)
- Sibasis Sahoo
- Department of Electronics and Electrical Engineering, Indian Institute of Technology Guwahati, Guwahati 781039, India
| | - Samarendra Dandapat
- Department of Electronics and Electrical Engineering, Indian Institute of Technology Guwahati, Guwahati 781039, India
| |
Collapse
|
14
|
Cascaded Convolutional Neural Network Architecture for Speech Emotion Recognition in Noisy Conditions. SENSORS 2021; 21:s21134399. [PMID: 34199027 PMCID: PMC8271804 DOI: 10.3390/s21134399] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/25/2021] [Revised: 06/20/2021] [Accepted: 06/24/2021] [Indexed: 01/09/2023]
Abstract
Convolutional neural networks (CNNs) are a state-of-the-art technique for speech emotion recognition. However, CNNs have mostly been applied to noise-free emotional speech data, and limited evidence is available for their applicability in emotional speech denoising. In this study, a cascaded denoising CNN (DnCNN)–CNN architecture is proposed to classify emotions from Korean and German speech in noisy conditions. The proposed architecture consists of two stages. In the first stage, the DnCNN exploits the concept of residual learning to perform denoising; in the second stage, the CNN performs the classification. The classification results for real datasets show that the DnCNN–CNN outperforms the baseline CNN in overall accuracy for both languages. For Korean speech, the DnCNN–CNN achieves an accuracy of 95.8%, whereas the accuracy of the CNN is marginally lower (93.6%). For German speech, the DnCNN–CNN has an overall accuracy of 59.3–76.6%, whereas the CNN has an overall accuracy of 39.4–58.1%. These results demonstrate the feasibility of applying the DnCNN with residual learning to speech denoising and the effectiveness of the CNN-based approach in speech emotion recognition. Our findings provide new insights into speech emotion recognition in adverse conditions and have implications for language-universal speech emotion recognition.
Collapse
|
15
|
Sevil M, Rashid M, Hajizadeh I, Askari MR, Hobbs N, Brandt R, Park M, Quinn L, Cinar A. Discrimination of simultaneous psychological and physical stressors using wristband biosignals. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 199:105898. [PMID: 33360529 PMCID: PMC7878428 DOI: 10.1016/j.cmpb.2020.105898] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/14/2020] [Accepted: 11/30/2020] [Indexed: 05/07/2023]
Abstract
BACKGROUND AND OBJECTIVE In this work, we address the problem of detecting and discriminating acute psychological stress (APS) in the presence of concurrent physical activity (PA) using wristband biosignals. We focused on signals available from wearable devices that can be worn in daily life because the ultimate objective of this work is to provide APS and PA information in real-time management of chronic conditions such as diabetes by automated personalized insulin delivery. Monitoring APS noninvasively throughout free-living conditions remains challenging because the responses to APS and PA of many physiological variables measured by wearable devices are similar. METHODS Various classification algorithms are compared to simultaneously detect and discriminate the PA (sedentary state, treadmill running, and stationary bike) and the type of APS (non-stress state, mental stress, and emotional anxiety). The impact of APS inducements is verified with commonly used self-reported questionnaires (The State-Trait Anxiety Inventory (STAI)). To aid the classification algorithms, novel features are generated from the physiological variables reported by a wristband device during 117 hours of experiments involving simultaneous APS inducement and PA. We also translate the APS assessment into a quantitative metric for use in predicting the adverse outcomes. RESULTS An accurate classification of the concurrent PA and APS states is achieved with an overall classification accuracy of 99% for PA and 92% for APS. The average accuracy of APS detection during sedentary state, treadmill running, and stationary bike is 97.3, 94.1, and 84.5%, respectively. CONCLUSIONS The simultaneous assessment of APS and PA throughout free-living conditions from a convenient wristband device is useful for monitoring the factors contributing to an elevated risk of acute events in people with chronic diseases like cardiovascular complications and diabetes.
Collapse
Affiliation(s)
- Mert Sevil
- Department of Biomedical Engineering, Illinois Institute of Technology, Chicago, IL, 60616, USA
| | - Mudassir Rashid
- Department of Chemical and Biological Engineering, Illinois Institute of Technology, Chicago, IL, 60616, USA
| | - Iman Hajizadeh
- Department of Chemical and Biological Engineering, Illinois Institute of Technology, Chicago, IL, 60616, USA
| | - Mohammad Reza Askari
- Department of Chemical and Biological Engineering, Illinois Institute of Technology, Chicago, IL, 60616, USA
| | - Nicole Hobbs
- Department of Biomedical Engineering, Illinois Institute of Technology, Chicago, IL, 60616, USA
| | - Rachel Brandt
- Department of Biomedical Engineering, Illinois Institute of Technology, Chicago, IL, 60616, USA
| | - Minsun Park
- College of Nursing, University of Illinois at Chicago, IL, 60616, USA
| | - Laurie Quinn
- College of Nursing, University of Illinois at Chicago, IL, 60616, USA
| | - Ali Cinar
- Department of Biomedical Engineering, Illinois Institute of Technology, Chicago, IL, 60616, USA; Department of Chemical and Biological Engineering, Illinois Institute of Technology, Chicago, IL, 60616, USA.
| |
Collapse
|
16
|
Siuly S, Khare SK, Bajaj V, Wang H, Zhang Y. A Computerized Method for Automatic Detection of Schizophrenia Using EEG Signals. IEEE Trans Neural Syst Rehabil Eng 2020; 28:2390-2400. [PMID: 32897863 DOI: 10.1109/tnsre.2020.3022715] [Citation(s) in RCA: 47] [Impact Index Per Article: 11.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Diagnosis of schizophrenia (SZ) is traditionally performed through patient's interviews by a skilled psychiatrist. This process is time-consuming, burdensome, subject to error and bias. Hence the aim of this study is to develop an automatic SZ identification scheme using electroencephalogram (EEG) signals that can eradicate the aforementioned problems and support clinicians and researchers. This study introduces a methodology design involving empirical mode decomposition (EMD) technique for diagnosis of SZ from EEG signals to perfectly handle the behavior of non-stationary and nonlinear EEG signals. In this study, each EEG signal is decomposed into intrinsic mode functions (IMFs) by the EMD algorithm and then twenty-two statistical characteristics/features are calculated from these IMFs. Among them, five features are selected as significant feature applying Kruskal Wallis test. The performance of the obtained feature set is tested through several renowned classifierson a SZ EEG dataset. Among the considered classifiers, theensemble bagged tree performed as the best classifier producing 93.21% correct classification rate for SZ, with an overall accuracy of 89.59% for IMF 2. These results indicate that EEG signals discriminate SZ patients from healthy control (HC) subjects efficiently and have the potential to become a tool for the psychiatrist to support the positive diagnosis of SZ.
Collapse
|
17
|
Sevil M, Rashid M, Maloney Z, Hajizadeh I, Samadi S, Askari MR, Hobbs N, Brandt R, Park M, Quinn L, Cinar A. Determining Physical Activity Characteristics from Wristband Data for Use in Automated Insulin Delivery Systems. IEEE SENSORS JOURNAL 2020; 20:12859-12870. [PMID: 33100923 PMCID: PMC7584145 DOI: 10.1109/jsen.2020.3000772] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2023]
Abstract
Algorithms that can determine the type of physical activity (PA) and quantify the intensity can allow precision medicine approaches, such as automated insulin delivery systems that modulate insulin administration in response to PA. In this work, data from a multi-sensor wristband is used to design classifiers to distinguish among five different physical states (PS) (resting, activities of daily living, running, biking, and resistance training), and to develop models to estimate the energy expenditure (EE) of the PA for diabetes therapy. The data collected are filtered, features are extracted from the reconciled signals, and the extracted features are used by machine learning algorithms, including deep-learning techniques, to obtain accurate PS classification and EE estimation. The various machine learning techniques have different success rates ranging from 75.7% to 94.8% in classifying the five different PS. The deep neural network model with long short-term memory has 94.8% classification accuracy. We achieved 0.5 MET (Metabolic Equivalent of Task) root-mean-square error for EE estimation accuracy, relative to indirect calorimetry with randomly selected testing data (10% of collected data). We also demonstrate a 5% improvement in PS classification accuracy and a 0.34 MET decrease in the mean absolute error when using multi-sensor approach relative to using only accelerometer data.
Collapse
Affiliation(s)
- Mert Sevil
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| | - Mudassir Rashid
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| | - Zacharie Maloney
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| | - Iman Hajizadeh
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| | - Sediqeh Samadi
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| | - Mohammad Reza Askari
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| | - Nicole Hobbs
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| | - Rachel Brandt
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| | - Minsun Park
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| | - Laurie Quinn
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| | - Ali Cinar
- Mert Sevil, Rachel Brandt, Nicole Hobbs and Zacharie Maloney are with the Department of Biomedical Engineering (BME); Mudassir Rashid, Mohammad Reza Askari, Iman Hajizadeh and Sedigeh Samadi are with the Department of Chemical and Biological Engineering (ChBE); Ali Cinar is with the Departments of ChBE and BME, Illinois Institute of Technology, Chicago, IL 60616; Minsun Park and Laurie Quinn are with the College of Nursing, University of Illinois at Chicago, IL, 60616
| |
Collapse
|
18
|
Pandit V, Schmitt M, Cummins N, Schuller B. I see it in your eyes: Training the shallowest-possible CNN to recognise emotions and pain from muted web-assisted in-the-wild video-chats in real-time. Inf Process Manag 2020. [DOI: 10.1016/j.ipm.2020.102347] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
19
|
Exploration of glottal characteristics and the vocal folds behavior for the speech under emotion. Neurocomputing 2020. [DOI: 10.1016/j.neucom.2020.06.010] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
20
|
Hansen JHL, Bokshi M, Khorram S. Speech variability: A cross-language study on acoustic variations of speaking versus untrained singing. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA 2020; 148:829. [PMID: 32873043 PMCID: PMC7438159 DOI: 10.1121/10.0001526] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/25/2019] [Revised: 06/17/2020] [Accepted: 06/19/2020] [Indexed: 06/11/2023]
Abstract
Speech production variability introduces significant challenges for existing speech technologies such as speaker identification (SID), speaker diarization, speech recognition, and language identification (ID). There has been limited research analyzing changes in acoustic characteristics for speech produced by untrained singing versus speaking. To better understand changes in speech production of the untrained singing voice, this study presents the first cross-language comparison between normal speaking and untrained karaoke singing of the same text content. Previous studies comparing professional singing versus speaking have shown deviations in both prosodic and spectral features. Some investigations also considered assigning the intrinsic activity of the singing. Motivated by these studies, a series of experiments to investigate both prosodic and spectral variations of untrained karaoke singers for three languages, American English, Hindi, and Farsi, are considered. A comprehensive comparison on common prosodic features, including phoneme duration, mean fundamental frequency (F0), and formant center frequencies of vowels was performed. Collective changes in the corresponding overall acoustic spaces based on the Kullback-Leibler distance using Gaussian probability distribution models trained on spectral features were analyzed. Finally, these models were used in a Gausian mixture model with universal background model SID evaluation to quantify speaker changes between speaking and singing when the audio text content is the same. The experiments showed that many acoustic characteristics of untrained singing are considerably different from speaking when the text content is the same. It is suggested that these results would help advance automatic speech production normalization/compensation to improve performance of speech processing applications (e.g., speaker ID, speech recognition, and language ID).
Collapse
Affiliation(s)
- John H L Hansen
- Robust Speech Technologies Laboratory (RSTL), Center for Robust Speech Systems (CRSS), University of Texas at Dallas, Richardson, Texas 75080, USA
| | - Marigona Bokshi
- Robust Speech Technologies Laboratory (RSTL), Center for Robust Speech Systems (CRSS), University of Texas at Dallas, Richardson, Texas 75080, USA
| | - Soheil Khorram
- Robust Speech Technologies Laboratory (RSTL), Center for Robust Speech Systems (CRSS), University of Texas at Dallas, Richardson, Texas 75080, USA
| |
Collapse
|
21
|
Hansen JHL, Lee J, Ali H, Saba JN. A speech perturbation strategy based on "Lombard effect" for enhanced intelligibility for cochlear implant listeners. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA 2020; 147:1418. [PMID: 32237802 PMCID: PMC7054124 DOI: 10.1121/10.0000690] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/02/2019] [Revised: 12/09/2019] [Accepted: 01/21/2020] [Indexed: 06/02/2023]
Abstract
The goal of this study is to determine potential intelligibility benefits from Lombard speech for cochlear implant (CI) listeners in speech-in-noise conditions. "Lombard effect" (LE) is the natural response of adjusting speech production via auditory feedback due to noise exposure within acoustic environments. To evaluate intelligibility performance of natural and artificially induced Lombard speech, a corpus was generated to create natural LE from large crowd noise (LCN) exposure at 70, 80, and 90 dB sound pressure level (SPL). Clean speech was mixed with 15 and 10 dB SNR LCN and presented to five CI users. First, speech intelligibility was analyzed as a function of increasing LE and decreasing SNR. Results indicate significant improvements (p < 0.05) with Lombard speech intelligibility in noise conditions for 80 and 90 dB SPL. Next, an offline perturbation strategy was formulated to modify/perturb neutral speech so as to mimic LE through amplification of highly intelligible segments, uniform time stretching, and spectral mismatch filtering. This process effectively introduces aspects of LE into the neutral speech, with the hypothesis that this would benefit intelligibility for CI users. Significant (p < 0.01) intelligibility improvements of 13% and 16% percentage points were observed for 15 and 10 dB SNR conditions respectively for CI users. The results indicate how LE and LE-inspired acoustic and frequency-based modifications can be leveraged within signal processing to improve intelligibility of speech for CI users.
Collapse
Affiliation(s)
- John H L Hansen
- Cochlear Implant Processing Laboratory, Center for Robust Speech Systems (CRSS-CILab), Department of Electrical Engineering and Computer Science, University of Texas at Dallas, Richardson, Texas 75080, USA
| | - Jaewook Lee
- Cochlear Implant Processing Laboratory, Center for Robust Speech Systems (CRSS-CILab), Department of Electrical Engineering and Computer Science, University of Texas at Dallas, Richardson, Texas 75080, USA
| | - Hussnain Ali
- Cochlear Implant Processing Laboratory, Center for Robust Speech Systems (CRSS-CILab), Department of Electrical Engineering and Computer Science, University of Texas at Dallas, Richardson, Texas 75080, USA
| | - Juliana N Saba
- Cochlear Implant Processing Laboratory, Center for Robust Speech Systems (CRSS-CILab), Department of Electrical Engineering and Computer Science, University of Texas at Dallas, Richardson, Texas 75080, USA
| |
Collapse
|
22
|
Slavich GM, Taylor S, Picard RW. Stress measurement using speech: Recent advancements, validation issues, and ethical and privacy considerations. Stress 2019; 22:408-413. [PMID: 30945584 PMCID: PMC7081839 DOI: 10.1080/10253890.2019.1584180] [Citation(s) in RCA: 25] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/01/2023] Open
Abstract
Life stress is a well-established risk factor for a variety of mental and physical health problems, including anxiety disorders, depression, chronic pain, heart disease, asthma, autoimmune diseases, and neurodegenerative disorders. The purpose of this article is to describe emerging approaches for assessing stress using speech, which we do by reviewing the methodological advantages of these digital health tools, and the validation, ethical, and privacy issues raised by these technologies. As we describe, it is now possible to assess stress via the speech signal using smartphones and smart speakers that employ software programs and artificial intelligence to analyze several features of speech and speech acoustics, including pitch, jitter, energy, rate, and length and number of pauses. Because these digital devices are ubiquitous, we can now assess individuals' stress levels in real time in almost any natural environment in which people speak. These technologies thus have great potential for advancing digital health initiatives that involve continuously monitoring changes in psychosocial functioning and disease risk over time. However, speech-based indices of stress have yet to be well-validated against stress biomarkers (e.g., cortisol, cytokines) that predict disease risk. In addition, acquiring speech samples raises the possibility that conversations intended to be private could one day be made public; moreover, obtaining real-time psychosocial risk information prompts ethical questions regarding how these data should be used for medical, commercial, and personal purposes. Although assessing stress using speech thus has enormous potential, there are critical validation, privacy, and ethical issues that must be addressed.
Collapse
Affiliation(s)
- George M. Slavich
- Cousins Center for Psychoneuroimmunology and Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, CA 90095-7076, USA
| | - Sara Taylor
- Affective Computing Group, MIT Media Lab, Massachusetts Institute of Technology,Cambridge, MA02139, USA
| | - Rosalind W. Picard
- Affective Computing Group, MIT Media Lab, Massachusetts Institute of Technology,Cambridge, MA02139, USA
| |
Collapse
|
23
|
Delić V, Perić Z, Sečujski M, Jakovljević N, Nikolić J, Mišković D, Simić N, Suzić S, Delić T. Speech Technology Progress Based on New Machine Learning Paradigm. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2019; 2019:4368036. [PMID: 31341467 PMCID: PMC6614991 DOI: 10.1155/2019/4368036] [Citation(s) in RCA: 24] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/22/2019] [Revised: 04/29/2019] [Accepted: 05/09/2019] [Indexed: 11/17/2022]
Abstract
Speech technologies have been developed for decades as a typical signal processing area, while the last decade has brought a huge progress based on new machine learning paradigms. Owing not only to their intrinsic complexity but also to their relation with cognitive sciences, speech technologies are now viewed as a prime example of interdisciplinary knowledge area. This review article on speech signal analysis and processing, corresponding machine learning algorithms, and applied computational intelligence aims to give an insight into several fields, covering speech production and auditory perception, cognitive aspects of speech communication and language understanding, both speech recognition and text-to-speech synthesis in more details, and consequently the main directions in development of spoken dialogue systems. Additionally, the article discusses the concepts and recent advances in speech signal compression, coding, and transmission, including cognitive speech coding. To conclude, the main intention of this article is to highlight recent achievements and challenges based on new machine learning paradigms that, over the last decade, had an immense impact in the field of speech signal processing.
Collapse
Affiliation(s)
- Vlado Delić
- University of Novi Sad, Faculty of Technical Sciences, 21000 Novi Sad, Serbia
| | - Zoran Perić
- University of Niš, Faculty of Electronic Engineering, 18000 Niš, Serbia
| | - Milan Sečujski
- University of Novi Sad, Faculty of Technical Sciences, 21000 Novi Sad, Serbia
| | - Nikša Jakovljević
- University of Novi Sad, Faculty of Technical Sciences, 21000 Novi Sad, Serbia
| | - Jelena Nikolić
- University of Niš, Faculty of Electronic Engineering, 18000 Niš, Serbia
| | - Dragiša Mišković
- University of Novi Sad, Faculty of Technical Sciences, 21000 Novi Sad, Serbia
| | - Nikola Simić
- University of Niš, Faculty of Electronic Engineering, 18000 Niš, Serbia
| | - Siniša Suzić
- University of Novi Sad, Faculty of Technical Sciences, 21000 Novi Sad, Serbia
| | - Tijana Delić
- University of Novi Sad, Faculty of Technical Sciences, 21000 Novi Sad, Serbia
| |
Collapse
|
24
|
Van Puyvelde M, Neyt X, McGlone F, Pattyn N. Voice Stress Analysis: A New Framework for Voice and Effort in Human Performance. Front Psychol 2018; 9:1994. [PMID: 30515113 PMCID: PMC6255927 DOI: 10.3389/fpsyg.2018.01994] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2018] [Accepted: 09/28/2018] [Indexed: 11/24/2022] Open
Abstract
People rely on speech for communication, both in a personal and professional context, and often under different conditions of physical, cognitive and/or emotional load. Since vocalization is entirely integrated within both our central (CNS) and autonomic nervous system (ANS), a mounting number of studies have examined the relationship between voice output and the impact of stress. In the current paper, we will outline the different stages of voice output, i.e., breathing, phonation and resonance in relation to a neurovisceral integrated perspective on stress and human performance. In reviewing the function of these three stages of voice output, we will give an overview of the voice parameters encountered in studies on voice stress analysis (VSA) and review the impact of the different types of physiological, cognitive and/or emotional load. In the section "Discussion," with regard to physical load, a competition for ventilation processes required to speak and those to meet metabolic demand of exercised muscles is described. With regard to cognitive and emotional load, we will present the "Model for Voice and Effort" (MoVE) that comprises the integration of ongoing top-down and bottom-up activity under different types of load and combined patterns of voice output. In the MoVE, it is proposed that the fundamental frequency (F0) values as well as jitter give insight in bottom-up/arousal activity and the effort a subject is capable to generate but that its range and variance are related to ongoing top-down processes and the amount of control a subject can maintain. Within the MoVE, a key-role is given to the anterior cingulate cortex (ACC) which is known to be involved in both the equilibration between bottom-up arousal and top-down regulation and vocal activity. Moreover, the connectivity between the ACC and the nervus vagus (NV) is underlined as an indication of the importance of respiration. Since respiration is the driving force of both stress and voice production, it is hypothesized to be the missing-link in our understanding of the underlying mechanisms of the dynamic between speech and stress.
Collapse
Affiliation(s)
- Martine Van Puyvelde
- VIPER Research Unit, LIFE Department, Royal Military Academy, Brussels, Belgium
- Brain, Body and Cognition, Experimental and Applied Psychology, Department of Psychological and Educational Sciences, Vrije Universiteit Brussel, Brussels, Belgium
- Clinical and Lifespan Psychology, Department of Psychological and Educational Sciences, Vrije Universiteit Brussel, Brussels, Belgium
| | - Xavier Neyt
- VIPER Research Unit, LIFE Department, Royal Military Academy, Brussels, Belgium
| | - Francis McGlone
- School of Natural Sciences and Psychology, Faculty of Science, Liverpool John Moores University, Liverpool, United Kingdom
| | - Nathalie Pattyn
- VIPER Research Unit, LIFE Department, Royal Military Academy, Brussels, Belgium
- Brain, Body and Cognition, Experimental and Applied Psychology, Department of Psychological and Educational Sciences, Vrije Universiteit Brussel, Brussels, Belgium
- MFYS-BLITS, Department of Human Physiology, Vrije Universiteit Brussel, Brussels, Belgium
| |
Collapse
|
25
|
Blanco JA, Johnson MK, Jaquess KJ, Oh H, Lo LC, Gentili RJ, Hatfield BD. Quantifying Cognitive Workload in Simulated Flight Using Passive, Dry EEG Measurements. IEEE Trans Cogn Dev Syst 2018. [DOI: 10.1109/tcds.2016.2628702] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
26
|
Spodenkiewicz M, Aigrain J, Bourvis N, Dubuisson S, Chetouani M, Cohen D. Distinguish self- and hetero-perceived stress through behavioral imaging and physiological features. Prog Neuropsychopharmacol Biol Psychiatry 2018; 82:107-114. [PMID: 29191571 DOI: 10.1016/j.pnpbp.2017.11.023] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/05/2017] [Revised: 11/20/2017] [Accepted: 11/27/2017] [Indexed: 11/25/2022]
Abstract
Stress reactivity is a complex phenomenon associated to multiple and multimodal expressions. Response to stressors has an obvious survival function and may be seen as an internal regulation to adapt to threat or danger. The intensity of this internal response can be assessed as the self-perception of the stress response. In species with social organization, this response also serves a communicative function, so-called hetero-perception. Our study presents multimodal stress detection assessment - a new methodology combining behavioral imaging and physiological monitoring for analyzing stress from these two perspectives. The system is based on automatic extraction of 39 behavioral (2D+3D video recording) and 62 physiological (Nexus-10 recording) features during a socially evaluated mental arithmetic test. The analysis with machine learning techniques for automatic classification using Support Vector Machine (SVM) show that self-perception and hetero-perception of social stress are both close but different phenomena: self-perception was significantly correlated with hetero-perception but significantly differed from it. Also, assessing stress with SVM through multimodality gave excellent classification results (F1 score values: 0.9±0.012 for hetero-perception and 0.87±0.021 for self-perception). In the best selected feature subsets, we found some common behavioral and physiological features that allow classification of both self- and hetero-perceived stress. However, we also found the contributing features for automatic classifications had opposite distributions: self-perception classification was mainly based on physiological features and hetero-perception was mainly based on behavioral features.
Collapse
Affiliation(s)
- Michel Spodenkiewicz
- Sorbonne Universités, UPMC Univ Paris 06, CNRS UMR 7222, Institut des Systèmes Intelligents et de Robotique, Paris, France; CEPOI EA 7388, Unité de Pédopsychiatrie de Liaison, Pôle de Santé Mentale, CHU Sud Réunion, Université de la Réunion, Saint-Pierre, France; CESP Inserm U1178, Groupe IPSOM, Paris, France.
| | - Jonathan Aigrain
- Sorbonne Universités, UPMC Univ Paris 06, CNRS UMR 7222, Institut des Systèmes Intelligents et de Robotique, Paris, France
| | - Nadège Bourvis
- Sorbonne Universités, UPMC Univ Paris 06, CNRS UMR 7222, Institut des Systèmes Intelligents et de Robotique, Paris, France; Département de Psychiatrie de l'Enfant et de l'Adolescent, AP-HP, GHU Pitié-Salpêtrière, Paris, France
| | - Séverine Dubuisson
- Sorbonne Universités, UPMC Univ Paris 06, CNRS UMR 7222, Institut des Systèmes Intelligents et de Robotique, Paris, France
| | - Mohamed Chetouani
- Sorbonne Universités, UPMC Univ Paris 06, CNRS UMR 7222, Institut des Systèmes Intelligents et de Robotique, Paris, France
| | - David Cohen
- Sorbonne Universités, UPMC Univ Paris 06, CNRS UMR 7222, Institut des Systèmes Intelligents et de Robotique, Paris, France; Département de Psychiatrie de l'Enfant et de l'Adolescent, AP-HP, GHU Pitié-Salpêtrière, Paris, France.
| |
Collapse
|
27
|
Deb S, Dandapat S. Multiscale Amplitude Feature and Significance of Enhanced Vocal Tract Information for Emotion Classification. IEEE TRANSACTIONS ON CYBERNETICS 2018; 49:802-815. [PMID: 29993975 DOI: 10.1109/tcyb.2017.2787717] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
In this paper, a novel multiscale amplitude feature is proposed using multiresolution analysis (MRA) and the significance of the vocal tract is investigated for emotion classification from the speech signal. MRA decomposes the speech signal into number of sub-band signals. The proposed feature is computed by using sinusoidal model on each sub-band signal. Different emotions have different impacts on the vocal tract. As a result, vocal tract responds in a unique way for each emotion. The vocal tract information is enhanced using pre-emphasis. Therefore, emotion information manifested in the vocal tract can be well exploited. This may help in improving the performance of emotion classification. Emotion recognition is performed using German emotional EMODB database, interactive emotional dyadic motion capture database, simulated stressed speech database, and FAU AIBO database with speech signal and speech with enhanced vocal tract information (SEVTI). The performance of the proposed multiscale amplitude feature is compared with three different types of features: 1) the mel frequency cepstral coefficients; 2) the Teager energy operator (TEO)-based feature (TEO-CB-Auto-Env); and 3) the breathinesss feature. The proposed feature outperforms the other features. In terms of recognition rates, the features derived from the SEVTI signal, give better performance compared to the features derived from the speech signal. Combination of the features with SEVTI signal shows average recognition rate of 86.7% using EMODB database.
Collapse
|
28
|
Automatic sleep stages classification based on iterative filtering of electroencephalogram signals. Neural Comput Appl 2017. [DOI: 10.1007/s00521-017-2919-6] [Citation(s) in RCA: 48] [Impact Index Per Article: 6.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
|
29
|
Shahin I. Employing Emotion Cues to Verify Speakers in Emotional Talking Environments. JOURNAL OF INTELLIGENT SYSTEMS 2016. [DOI: 10.1515/jisys-2014-0118] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022] Open
Abstract
AbstractUsually, people talk neutrally in environments where there are no abnormal talking conditions such as stress and emotion. Other emotional conditions that might affect people’s talking tone include happiness, anger, and sadness. Such emotions are directly affected by the patient’s health status. In neutral talking environments, speakers can be easily verified; however, in emotional talking environments, speakers cannot be easily verified as in neutral talking ones. Consequently, speaker verification systems do not perform well in emotional talking environments as they do in neutral talking environments. In this work, a two-stage approach has been employed and evaluated to improve speaker verification performance in emotional talking environments. This approach employs speaker’s emotion cues (text-independent and emotion-dependent speaker verification problem) based on both hidden Markov models (HMMs) and suprasegmental HMMs as classifiers. The approach is composed of two cascaded stages that combine and integrate an emotion recognizer and a speaker recognizer into one recognizer. The architecture has been tested on two different and separate emotional speech databases: our collected database and the Emotional Prosody Speech and Transcripts database. The results of this work show that the proposed approach gives promising results with a significant improvement over previous studies and other approaches such as emotion-independent speaker verification approach and emotion-dependent speaker verification approach based completely on HMMs.
Collapse
Affiliation(s)
- Ismail Shahin
- 1Department of Electrical and Computer Engineering, University of Sharjah, P.O. Box 27272, Sharjah, United Arab Emirates
| |
Collapse
|
30
|
Prediction of major depression in adolescents using an optimized multi-channel weighted speech classification system. Biomed Signal Process Control 2014. [DOI: 10.1016/j.bspc.2014.08.006] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
31
|
Lech M, He L. Stress and Emotion Recognition Using Acoustic Speech Analysis. MENTAL HEALTH INFORMATICS 2014. [DOI: 10.1007/978-3-642-38550-6_9] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/25/2023]
|
32
|
Reiner BI. Expanding the functionality of speech recognition in radiology: creating a real-time methodology for measurement and analysis of occupational stress and fatigue. J Digit Imaging 2013; 26:5-9. [PMID: 23053910 PMCID: PMC3553361 DOI: 10.1007/s10278-012-9540-0] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022] Open
Abstract
While occupational stress and fatigue have been well described throughout medicine, the radiology community is particularly susceptible due to declining reimbursements, heightened demands for service deliverables, and increasing exam volume and complexity. The resulting occupational stress can be variable in nature and dependent upon a number of intrinsic and extrinsic stressors. Intrinsic stressors largely account for inter-radiologist stress variability and relate to unique attributes of the radiologist such as personality, emotional state, education/training, and experience. Extrinsic stressors may account for intra-radiologist stress variability and include cumulative workload and task complexity. The creation of personalized stress profiles creates a mechanism for accounting for both inter- and intra-radiologist stress variability, which is essential in creating customizable stress intervention strategies. One viable option for real-time occupational stress measurement is voice stress analysis, which can be directly implemented through existing speech recognition technology and has been proven to be effective in stress measurement and analysis outside of medicine. This technology operates by detecting stress in the acoustic properties of speech through a number of different variables including duration, glottis source factors, pitch distribution, spectral structure, and intensity. The correlation of these speech derived stress measures with outcomes data can be used to determine the user-specific inflection point at which stress becomes detrimental to clinical performance.
Collapse
Affiliation(s)
- Bruce I Reiner
- Department of Radiology, Veterans Affairs Maryland Healthcare System, Baltimore, MD 21201, USA.
| |
Collapse
|
33
|
Sharma N, Gedeon T. Objective measures, sensors and computational techniques for stress recognition and classification: a survey. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2012; 108:1287-301. [PMID: 22921417 DOI: 10.1016/j.cmpb.2012.07.003] [Citation(s) in RCA: 87] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/02/2012] [Revised: 07/02/2012] [Accepted: 07/25/2012] [Indexed: 05/15/2023]
Abstract
Stress is a major growing concern in our day and age adversely impacting both individuals and society. Stress research has a wide range of benefits from improving personal operations, learning, and increasing work productivity to benefiting society - making it an interesting and socially beneficial area of research. This survey reviews sensors that have been used to measure stress and investigates techniques for modelling stress. It discusses non-invasive and unobtrusive sensors for measuring computed stress, a term we coin in the paper. Sensors that do not impede everyday activities that could be used by those who would like to monitor stress levels on a regular basis (e.g. vehicle drivers, patients with illnesses linked to stress) is the focus of the discussion. Computational techniques have the capacity to determine optimal sensor fusion and automate data analysis for stress recognition and classification. Several computational techniques have been developed to model stress based on techniques such as Bayesian networks, artificial neural networks, and support vector machines, which this survey investigates. The survey concludes with a summary and provides possible directions for further computational stress research.
Collapse
Affiliation(s)
- Nandita Sharma
- Information and Human Centred Computing Research Group, Research School of Computer Science, Australian National University, Canberra, ACT 0200, Australia.
| | | |
Collapse
|
34
|
Ooi KEB, Lech M, Allen NB. Multichannel weighted speech classification system for prediction of major depression in adolescents. IEEE Trans Biomed Eng 2012. [PMID: 23192475 DOI: 10.1109/tbme.2012.2228646] [Citation(s) in RCA: 46] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
Early identification of adolescents at high imminent risk for clinical depression could significantly reduce the burden of the disease. This study demonstrated that acoustic speech analysis and classification can be used to determine early signs of major depression in adolescents, up to two years before they meet clinical diagnostic criteria for the full-blown disorder. Individual contributions of four different types of acoustic parameters [prosodic, glottal, Teager's energy operator (TEO), and spectral] to depression-related changes of speech characteristics were examined. A new computational methodology for the early prediction of depression in adolescents was developed and tested. The novel aspect of this methodology is in the introduction of multichannel classification with a weighted decision procedure. It was observed that single-channel classification was effective in predicting depression with a desirable specificity-to-sensitivity ratio and accuracy higher than chance level only when using glottal or prosodic features. The best prediction performance was achieved with the new multichannel method, which used four features (prosodic, glottal, TEO, and spectral). In the case of the person-based approach with two sets of weights, the new multichannel method provided a high accuracy level of 73% and the sensitivity-to-specificity ratio of 79%/67% for predicting future depression.
Collapse
Affiliation(s)
- Kuan Ee Brian Ooi
- School of Electrical and Computer Engineering, Royal Melbourne Institute of Technology, Melbourne, Victoria, Australia.
| | | | | |
Collapse
|
35
|
Lee SY, Song HA, Amari SI. A new discriminant NMF algorithm and its application to the extraction of subtle emotional differences in speech. Cogn Neurodyn 2012; 6:525-35. [PMID: 24294336 DOI: 10.1007/s11571-012-9213-1] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2011] [Revised: 06/20/2012] [Accepted: 07/09/2012] [Indexed: 10/28/2022] Open
Abstract
In this study we propose a new feature extraction algorithm, dNMF (discriminant non-negative matrix factorization), to learn subtle class-related differences while maintaining an accurate generative capability. In addition to the minimum representation error for the standard NMF (non-negative matrix factorization) algorithm, the dNMF algorithm also results in higher between-class variance for discriminant power. The multiplicative NMF learning algorithm has been modified to cope with this additional constraint. The cost function was carefully designed so that the extraction of feature coefficients from a single testing pattern with pre-trained feature vectors resulted in a quadratic convex optimization problem in non-negative space for uniqueness. It also resolves issues related to the previous discriminant NMF algorithms. The developed dNMF algorithm has been applied to the emotion recognition task for speech, where it needs to emphasize the emotional differences while de-emphasizing the dominant phonetic components. The dNMF algorithm successfully extracted subtle emotional differences, demonstrated much better recognition performance and showed a smaller representation error from an emotional speech database.
Collapse
Affiliation(s)
- Soo-Young Lee
- Department of Electrical Engineering and Brain Science Research Center, KAIST, 373-1 Guseong-dong, Yuseong-gu, Daejeon, 305-701 South Korea
| | | | | |
Collapse
|
36
|
Study of empirical mode decomposition and spectral analysis for stress and emotion classification in natural speech. Biomed Signal Process Control 2011. [DOI: 10.1016/j.bspc.2010.11.001] [Citation(s) in RCA: 43] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
37
|
Batliner A, Schuller B, Seppi D, Steidl S, Devillers L, Vidrascu L, Vogt T, Aharonson V, Amir N. The Automatic Recognition of Emotions in Speech. COGNITIVE TECHNOLOGIES 2011. [DOI: 10.1007/978-3-642-15184-2_6] [Citation(s) in RCA: 33] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/24/2023]
|
38
|
Low LSA, Maddage NC, Lech M, Sheeber LB, Allen NB. Detection of clinical depression in adolescents' speech during family interactions. IEEE Trans Biomed Eng 2010; 58:574-86. [PMID: 21075715 DOI: 10.1109/tbme.2010.2091640] [Citation(s) in RCA: 128] [Impact Index Per Article: 9.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
The properties of acoustic speech have previously been investigated as possible cues for depression in adults. However, these studies were restricted to small populations of patients and the speech recordings were made during patients' clinical interviews or fixed-text reading sessions. Symptoms of depression often first appear during adolescence at a time when the voice is changing, in both males and females, suggesting that specific studies of these phenomena in adolescent populations are warranted. This study investigated acoustic correlates of depression in a large sample of 139 adolescents (68 clinically depressed and 71 controls). Speech recordings were made during naturalistic interactions between adolescents and their parents. Prosodic, cepstral, spectral, and glottal features, as well as features derived from the Teager energy operator (TEO), were tested within a binary classification framework. Strong gender differences in classification accuracy were observed. The TEO-based features clearly outperformed all other features and feature combinations, providing classification accuracy ranging between 81%-87% for males and 72%-79% for females. Close, but slightly less accurate, results were obtained by combining glottal features with prosodic and spectral features (67%-69% for males and 70%-75% for females). These findings indicate the importance of nonlinear mechanisms associated with the glottal flow formation as cues for clinical depression.
Collapse
Affiliation(s)
- Lu-Shih Alex Low
- School of Electrical and Computer Engineering, Royal Melbourne Institute of Technology, Vic. 3001, Australia.
| | | | | | | | | |
Collapse
|
39
|
Spoken emotion recognition through optimum-path forest classification using glottal features. COMPUT SPEECH LANG 2010. [DOI: 10.1016/j.csl.2009.02.005] [Citation(s) in RCA: 64] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
|
40
|
|
41
|
Ślot K, Bronakowski Ł, Cichosz J, Kim H. Application of poincare-mapping of voiced-speech segments for emotion sensing. SENSORS (BASEL, SWITZERLAND) 2009; 9:9858-9872. [PMID: 22303151 PMCID: PMC3267199 DOI: 10.3390/s91209858] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/30/2009] [Revised: 11/14/2009] [Accepted: 11/17/2009] [Indexed: 05/31/2023]
Abstract
The following paper introduces a group of novel speech-signal descriptors that reflect phoneme-pronunciation variability and that can be considered as potentially useful features for emotion sensing. The proposed group includes a set of statistical parameters of Poincare maps, derived for formant-frequency evolution and energy evolution of voiced-speech segments. Two groups of Poincare-map characteristics were considered in the research: descriptors of sample-scatter, which reflect magnitudes of phone-uttering variations and descriptors of cross-correlations that exist among samples and that evaluate consistency of variations. It has been shown that inclusion of the proposed characteristics into the pool of commonly used speech descriptors, results in a noticeable increase-at the level of 10%-in emotion sensing performance. Standard pattern recognition methodology has been adopted for evaluation of the proposed descriptors, with the assumption that three- or four-dimensional feature spaces can provide sufficient emotion sensing. Binary decision trees have been selected for data classification, as they provide with detailed information on emotion-specific discriminative power of various speech descriptors.
Collapse
Affiliation(s)
- Krzysztof Ślot
- Institute of Electronics, Technical University of Lodz, Poland, Wolczanska 213/215, 90-924 Lodz, Poland; E-Mails: (K.S.); (L.B.); (J.C.)
| | - Łukasz Bronakowski
- Institute of Electronics, Technical University of Lodz, Poland, Wolczanska 213/215, 90-924 Lodz, Poland; E-Mails: (K.S.); (L.B.); (J.C.)
| | - Jaroslaw Cichosz
- Institute of Electronics, Technical University of Lodz, Poland, Wolczanska 213/215, 90-924 Lodz, Poland; E-Mails: (K.S.); (L.B.); (J.C.)
| | - Hyongsuk Kim
- Division of Electrical, Electronics and Computer Engineering, Chonbuk National University, 561-756 Chonju, Korea
| |
Collapse
|
42
|
Hansen J, Varadarajan V. Analysis and Compensation of Lombard Speech Across Noise Type and Levels With Application to In-Set/Out-of-Set Speaker Recognition. ACTA ACUST UNITED AC 2009. [DOI: 10.1109/tasl.2008.2009019] [Citation(s) in RCA: 55] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
43
|
|
44
|
Ramamohan S, Dandapat S. Sinusoidal model-based analysis and classification of stressed speech. ACTA ACUST UNITED AC 2006. [DOI: 10.1109/tsa.2005.858071] [Citation(s) in RCA: 64] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
45
|
Shimizu T, Furuse N, Yamazaki T, Ueta Y, Sato T, Nagata S. Chaos of Vowel /a/ in Japanese Patients with Depression: A Preliminary Study. J Occup Health 2005; 47:267-9. [PMID: 15953849 DOI: 10.1539/joh.47.267] [Citation(s) in RCA: 13] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Affiliation(s)
- Takashi Shimizu
- Department of Mental Health, Institute of Industrial and Ecological Sciences, University of Occupational and Environmental Health, Japan.
| | | | | | | | | | | |
Collapse
|
46
|
Xianxian Zhang, Hansen J. CSA-BF: a constrained switched adaptive beamformer for speech enhancement and recognition in real car environments. ACTA ACUST UNITED AC 2003. [DOI: 10.1109/tsa.2003.818034] [Citation(s) in RCA: 17] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|