1
|
Yang M, El-Attar AA, Chaspari T. Deconstructing demographic bias in speech-based machine learning models for digital health. Front Digit Health 2024; 6:1351637. [PMID: 39119589 PMCID: PMC11306200 DOI: 10.3389/fdgth.2024.1351637] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2023] [Accepted: 07/15/2024] [Indexed: 08/10/2024] Open
Abstract
Introduction Machine learning (ML) algorithms have been heralded as promising solutions to the realization of assistive systems in digital healthcare, due to their ability to detect fine-grain patterns that are not easily perceived by humans. Yet, ML algorithms have also been critiqued for treating individuals differently based on their demography, thus propagating existing disparities. This paper explores gender and race bias in speech-based ML algorithms that detect behavioral and mental health outcomes. Methods This paper examines potential sources of bias in the data used to train the ML, encompassing acoustic features extracted from speech signals and associated labels, as well as in the ML decisions. The paper further examines approaches to reduce existing bias via using the features that are the least informative of one's demographic information as the ML input, and transforming the feature space in an adversarial manner to diminish the evidence of the demographic information while retaining information about the focal behavioral and mental health state. Results Results are presented in two domains, the first pertaining to gender and race bias when estimating levels of anxiety, and the second pertaining to gender bias in detecting depression. Findings indicate the presence of statistically significant differences in both acoustic features and labels among demographic groups, as well as differential ML performance among groups. The statistically significant differences present in the label space are partially preserved in the ML decisions. Although variations in ML performance across demographic groups were noted, results are mixed regarding the models' ability to accurately estimate healthcare outcomes for the sensitive groups. Discussion These findings underscore the necessity for careful and thoughtful design in developing ML models that are capable of maintaining crucial aspects of the data and perform effectively across all populations in digital healthcare applications.
Collapse
Affiliation(s)
- Michael Yang
- Computer Science & Engineering, Texas A&M University, College Station, TX, United States
| | - Abd-Allah El-Attar
- Computer Science & Engineering, Texas A&M University Qatar, Al Rayyan, Qatar
| | - Theodora Chaspari
- Institute of Cognitive Science & Computer Science, University of Colorado Boulder, Boulder, CO, United States
| |
Collapse
|
2
|
Maciejewska B, Maciejewska-Szaniec Z, Małaczyńska B, Rajewska-Rager A, Michalak M, Iwanowski P. Effects of Age-Dependent Hormonal Changes and Estrogen Supplementation on Voice in Girls with Anorexia Nervosa-Preliminary Report. J Voice 2024:S0892-1997(24)00028-6. [PMID: 38493018 DOI: 10.1016/j.jvoice.2024.02.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/12/2023] [Revised: 02/07/2024] [Accepted: 02/08/2024] [Indexed: 03/18/2024]
Abstract
INTRODUCTION Human development includes lots of physical and emotional changes. The human voice depends on age. Voice production is a complex physiological and acoustic phenomenon that depends on many factors such as structure, hormone level, degree of fatigue or nutrition and hydration of the body, systemic diseases, and emotional state. All these factors can be present in anorexia nervosa (AN), such as excessive weight loss, generated hydro-electrolytic changes, nutritional deficiencies, hormonal disturbances in the function of the hypothalamic-pituitary-adrenal axis, the hypothalamic-pituitary-thyroid axis, the hypothalamic-pituitary-ovarian axis, and emotional distress. The prevalence of AN ranges between 0.3% and 3%, and it is the third most common chronic disease affecting adolescent girls. However, voice changes related to AN have not been fully investigated. OBJECTIVE The purpose of this study was to evaluate the impact of AN on age-related changes in the voice of adolescent women-before and after puberty, particularly through acoustic analysis. An additional objective was to evaluate estrogen substitution in female patients with AN in order to investigate their effect on voice condition. MATERIALS AND METHODS 126 girls diagnosed with AN (15.32 ± 2.12 years, range 12-19, BMI = 14.38 ± 1.67), were assessed for the condition of the voice such as perceptual voice evaluation on the GRBAS scale, maximal phonation time (MPT), laryngoscopy, with special attention to voice acoustic analysis-Multi-Dimensional Voice Program (MDVP). The control group (B) included 93 girls without eating disturbances (aged 12-19, mean age 15.52 ± 2.40, BMI = 21.50 ± 1.54). Perceptual voice assessment, aerodynamic test MPT, and acoustic parameters were analyzed in age groups (≤16 years and >16 years). The human vocal tract is sensitive to sex hormones, so the analysis was carried out in the group up to the age of 16 and above 16 to check possible effects. RESULTS GRBAS scale was higher in girls with AN compared to the control group for breathiness (B) (P = 0.0002) and asthenia (A) (P < 0.05). The median GRBAS scale for the older group of anorexic women was the highest (2.0). The mean MPT for group A was significantly lower (15.40 ± 3.51 seconds). Comparing age subgroups there was a prolongation of MPT in the healthy group (in groups ≤16 years and >16 years respectively 21.13 seconds versus 25.40 seconds) and a shortening in the anorectic group (≤16 years versus >16 years: 17.06 seconds versus 14.17 seconds). There was no difference between groups A and C up to 16 years of age, but above 16 years of age appeared (14.17 seconds versus 25.40 seconds). Acoustic analysis revealed lower F0 values in group A and C in older subgroups (215,85 Hz versus 236,01 Hz-statistically significant), as well as between subgroups both groups (A: 251,38 Hz versus 215,85 Hz; C: 248,20 Hz versus 236,01 Hz). A narrowing of the vocal range in girls over 16 years in group A was observed. There were no statistically significant differences in F0 between subgroups ≤16 years in groups A and C (251.38 Hz versus 248.20 Hz). The ENT study found that more than half of the girls (54.55%) over the age of 16 who took hormone supplementation manifested laryngeal structure that was normal for their age, there was no effect of hormone supplementation on any of the MDVP parameters between the drug-taking and non-drug-taking groups. CONCLUSIONS The acoustic results of the voice in MDVP measurements in adolescent women with AN are not within the normal range and do not mimic the normal developmental changes of the voice. The most important acoustic characteristics of the voice are changes in the fundamental frequency F0 and the range of the voice tended to be more severe in anorectic women >16 years of age and to increase with age, indicating a possible cumulative effect of malnutrition-related disorders as well as hormonal dysfunctions. MDVP can be considered a simple, non-invasive method of assessing the voice organ in AN. MPT differentiated the study groups well: statistically significant differences were noted both between the groups, as well as between age groups. There was no significant effect of oral hormone supplementation on any parameters of the voice. In conclusion, body mass and fat volume in AN may be related to voice production/physiology, affecting voice quality, voice acoustic parameters, voice aerodynamics, and phonatory range in an age-dependent manner. Future studies are needed to assess the long-term efficacy of estrogen treatment in AN.
Collapse
Affiliation(s)
- Barbara Maciejewska
- Department and Clinic of Phoniatrics and Audiology, Poznan University of Medical Sciences
| | | | - Bogna Małaczyńska
- Department and Clinic of Phoniatrics and Audiology, Poznan University of Medical Sciences
| | | | - Michał Michalak
- Department of Computer Science and Statistics, Poznan University of Medical Sciences
| | - Piotr Iwanowski
- Department of Neurology, Poznan University of Medical Sciences.
| |
Collapse
|
3
|
Bandela SR, Siva Priyanka S, Sunil Kumar K, Vijay Bhaskar Reddy Y, Berhanu AA. Stressed Speech Emotion Recognition Using Teager Energy and Spectral Feature Fusion with Feature Optimization. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2023; 2023:5765760. [PMID: 37868755 PMCID: PMC10586421 DOI: 10.1155/2023/5765760] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/04/2022] [Accepted: 07/04/2022] [Indexed: 10/24/2023]
Abstract
The objective of speech emotion recognition (SER) is to enhance man-machine interface. It can also be used to cover the physiological state of a person in critical situations. In recent time, speech emotion recognition also finds its operations in medicine and forensics. A new feature extraction technique using Teager energy operator (TEO) is proposed for the detection of stressed emotions as Teager energy-autocorrelation envelope (TEO-Auto-Env). TEO is basically designed for increasing the energies of the stressed speech signals whose energies are reduced during the speech production process and hence used in this analysis. A stressed speech emotion recognition (SSER) system is developed using TEO-Auto-Env and spectral feature combination for detecting the emotions. The spectral features considered are Mel-frequency cepstral coefficients (MFCC), linear prediction cepstral coefficients (LPCC), and relative spectra-perceptual linear prediction (RASTA-PLP). EMO-DB (German), EMOVO (Italian), IITKGP (Telugu), and EMA (English) databases are used in this analysis. The classification of the emotions is carried out using the k-nearest neighborhood (k-NN) classifier for gender-dependent (GD) and speaker-independent (SI) cases. The proposed SSER system provides improved accuracy compared to the existing ones. Average recall is used for performance evaluation. The highest classification accuracy is achieved using the feature combination of TEO-Auto-Env, MFCC, and LPCC features with 91.4% (SI), 91.4% (GD-male), and 93.1%(GD-female) for EMO-DB; 68.5% (SI), 68.5% (GD-male), and 74.6% (GD-female) for EMOVO; 90.6%(SI), 91% (GD-male), and 92.3% (GD-female) for EMA; and 95.1% (GD-female) for IITKGP female database.
Collapse
Affiliation(s)
| | | | | | | | - Afework Aemro Berhanu
- Department of Environmental Engineering, College of Biological and Chemical Engineering Addis Ababa Science and Technology University, Addis Ababa, Ethiopia
| |
Collapse
|
4
|
Wang Y, Liang L, Zhang Z, Xu X, Liu R, Fang H, Zhang R, Wei Y, Liu Z, Zhu R, Zhang X, Wang F. Fast and accurate assessment of depression based on voice acoustic features: a cross-sectional and longitudinal study. Front Psychiatry 2023; 14:1195276. [PMID: 37415683 PMCID: PMC10320390 DOI: 10.3389/fpsyt.2023.1195276] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/28/2023] [Accepted: 06/02/2023] [Indexed: 07/08/2023] Open
Abstract
Background Depression is a widespread mental disorder that affects a significant portion of the population. However, the assessment of depression is often subjective, relying on standard questions or interviews. Acoustic features have been suggested as a reliable and objective alternative for depression assessment. Therefore, in this study, we aim to identify and explore voice acoustic features that can effectively and rapidly predict the severity of depression, as well as investigate the potential correlation between specific treatment options and voice acoustic features. Methods We utilized voice acoustic features correlated with depression scores to train a prediction model based on artificial neural network. Leave-one-out cross-validation was performed to evaluate the performance of the model. We also conducted a longitudinal study to analyze the correlation between the improvement of depression and changes in voice acoustic features after an Internet-based cognitive-behavioral therapy (ICBT) program consisting of 12 sessions. Results Our study showed that the neural network model trained based on the 30 voice acoustic features significantly correlated with HAMD scores can accurately predict the severity of depression with an absolute mean error of 3.137 and a correlation coefficient of 0.684. Furthermore, four out of the 30 features significantly decreased after ICBT, indicating their potential correlation with specific treatment options and significant improvement in depression (p < 0.05). Conclusion Voice acoustic features can effectively and rapidly predict the severity of depression, providing a low-cost and efficient method for screening patients with depression on a large scale. Our study also identified potential acoustic features that may be significantly related to specific treatment options for depression.
Collapse
Affiliation(s)
- Yang Wang
- Psychology Institute, Inner Mongolia Normal University, Hohhot, Inner Mongolia, China
- Early Intervention Unit, Department of Psychiatry, The Affiliated Brain Hospital of Nanjing Medical University, Nanjing, China
- Functional Brain Imaging Institute, Nanjing Medical University, Nanjing, China
| | - Lijuan Liang
- Early Intervention Unit, Department of Psychiatry, The Affiliated Brain Hospital of Nanjing Medical University, Nanjing, China
- Functional Brain Imaging Institute, Nanjing Medical University, Nanjing, China
- Laboratory of Psychology, The First Affiliated Hospital of Hainan Medical University, Haikou, Hainan, China
| | - Zhongguo Zhang
- Early Intervention Unit, Department of Psychiatry, The Affiliated Brain Hospital of Nanjing Medical University, Nanjing, China
- Functional Brain Imaging Institute, Nanjing Medical University, Nanjing, China
- The Fourth People’s Hospital of Yancheng, Yancheng, Jiangsu, China
| | - Xiao Xu
- School of Biomedical Engineering and Informatics, Nanjing Medical University, Nanjing, China
| | - Rongxun Liu
- Early Intervention Unit, Department of Psychiatry, The Affiliated Brain Hospital of Nanjing Medical University, Nanjing, China
- Functional Brain Imaging Institute, Nanjing Medical University, Nanjing, China
- College of Medical Engineering, Xinxiang Medical University, Xinxiang, Henan, China
| | - Hanzheng Fang
- School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China
| | - Ran Zhang
- Early Intervention Unit, Department of Psychiatry, The Affiliated Brain Hospital of Nanjing Medical University, Nanjing, China
- Functional Brain Imaging Institute, Nanjing Medical University, Nanjing, China
| | - Yange Wei
- Early Intervention Unit, Department of Psychiatry, The Affiliated Brain Hospital of Nanjing Medical University, Nanjing, China
- Functional Brain Imaging Institute, Nanjing Medical University, Nanjing, China
| | - Zhongchun Liu
- Department of Psychiatry, Renmin Hospital of Wuhan University, Wuhan, Hubei, China
| | - Rongxin Zhu
- Early Intervention Unit, Department of Psychiatry, The Affiliated Brain Hospital of Nanjing Medical University, Nanjing, China
- Functional Brain Imaging Institute, Nanjing Medical University, Nanjing, China
| | - Xizhe Zhang
- School of Biomedical Engineering and Informatics, Nanjing Medical University, Nanjing, China
| | - Fei Wang
- Early Intervention Unit, Department of Psychiatry, The Affiliated Brain Hospital of Nanjing Medical University, Nanjing, China
- Functional Brain Imaging Institute, Nanjing Medical University, Nanjing, China
| |
Collapse
|
5
|
Luo J, Liu M, Feng L, Li Z, Wu Y, Lu J, He F. Multidimensional voiceprint feature assessment system for identifying the depression in children and adolescents: a diagnostic test. Front Psychiatry 2023; 14:1105534. [PMID: 37234211 PMCID: PMC10205976 DOI: 10.3389/fpsyt.2023.1105534] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/22/2022] [Accepted: 04/14/2023] [Indexed: 05/27/2023] Open
Abstract
Objective We designed a diagnostic test to evaluate the effectiveness and accuracy of a multidimensional voiceprint feature diagnostic assessment (MVFDA) system vs. the 24-item Hamilton Rating Scale for Depression (HAMD-24) for adjunctive diagnosis of children and adolescents with major depressive disorder (MDD). Methods This study included 55 children aged 6-16 years who were clinically diagnosed with MDD according to the DSM-5 and analyzed by professional physicians, and 55 healthy children (typically developing). Each subject completed a voice recording and was scored on the HAMD-24 scale by a trained rater. We calculated the validity indices, including sensitivity, specificity, Youden's index, likelihood ratio, and other indices including predictive value, diagnostic odds ratio, diagnostic accuracy, and area under the curve (AUC), to assess the effectiveness of the MVFDA system in addition to the HAMD-24. Results The sensitivity (92.73 vs. 76.36%) and the specificity (90.91 vs. 85.45%) of the MVFDA system are significantly higher than those of the HAMD-24. The AUC of the MVFDA system is also higher than that of the HAMD-24. There is a statistically significant difference between the groups (p < 0.05), and both of them have high diagnostic accuracy. In addition, the diagnostic efficacy of the MVFDA system is higher than that of HAMD-24 in terms of the Youden index, diagnostic accuracy, likelihood ratio, diagnostic odds ratio, and predictive value. Conclusion The MVFDA has performed well in clinical diagnostic trials for the identification of MDD in children and adolescents by capturing objective sound features. Compared with the scale assessment method, the MVFDA system could be further promoted in clinical practice due to its advantages of simple operation, objective rating, and high diagnostic efficiency.
Collapse
Affiliation(s)
- Jie Luo
- National Clinical Research Center for Mental Disorders, Beijing Key Laboratory of Mental Disorders, Beijing Anding Hospital, Beijing Institute for Brain Disorders, Capital Medical University, Beijing, China
| | - Mengqi Liu
- National Clinical Research Center for Mental Disorders, Beijing Key Laboratory of Mental Disorders, Beijing Anding Hospital, Beijing Institute for Brain Disorders, Capital Medical University, Beijing, China
| | - Lihui Feng
- Beijing Institute of Technology, School of Optics and Photonics, Beijing, China
| | - Zhaojun Li
- Beijing Institute of Technology, School of Integrated Circuits and Electronics, Beijing, China
| | - Yuanzhen Wu
- National Clinical Research Center for Mental Disorders, Beijing Key Laboratory of Mental Disorders, Beijing Anding Hospital, Beijing Institute for Brain Disorders, Capital Medical University, Beijing, China
| | - Jihua Lu
- Beijing Institute of Technology, School of Integrated Circuits and Electronics, Beijing, China
| | - Fan He
- National Clinical Research Center for Mental Disorders, Beijing Key Laboratory of Mental Disorders, Beijing Anding Hospital, Beijing Institute for Brain Disorders, Capital Medical University, Beijing, China
| |
Collapse
|
6
|
Takano T, Mizuguchi D, Omiya Y, Higuchi M, Nakamura M, Shinohara S, Mitsuyoshi S, Saito T, Yoshino A, Toda H, Tokuno S. Estimating Depressive Symptom Class from Voice. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2023; 20:3965. [PMID: 36900976 PMCID: PMC10002315 DOI: 10.3390/ijerph20053965] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 02/12/2023] [Accepted: 02/20/2023] [Indexed: 06/18/2023]
Abstract
Voice-based depression detection methods have been studied worldwide as an objective and easy method to detect depression. Conventional studies estimate the presence or severity of depression. However, an estimation of symptoms is a necessary technique not only to treat depression, but also to relieve patients' distress. Hence, we studied a method for clustering symptoms from HAM-D scores of depressed patients and by estimating patients in different symptom groups based on acoustic features of their speech. We could separate different symptom groups with an accuracy of 79%. The results suggest that voice from speech can estimate the symptoms associated with depression.
Collapse
Affiliation(s)
- Takeshi Takano
- Department of Bioengineering, Graduate School of Engineering, The University of Tokyo, Tokyo 113-8656, Japan
| | | | - Yasuhiro Omiya
- Department of Bioengineering, Graduate School of Engineering, The University of Tokyo, Tokyo 113-8656, Japan
- PST Inc., Yokohama 231-0023, Japan
| | - Masakazu Higuchi
- Department of Bioengineering, Graduate School of Engineering, The University of Tokyo, Tokyo 113-8656, Japan
| | - Mitsuteru Nakamura
- Department of Bioengineering, Graduate School of Engineering, The University of Tokyo, Tokyo 113-8656, Japan
| | - Shuji Shinohara
- School of Science and Engineering, Tokyo Denki University, Saitama 350-0394, Japan
| | - Shunji Mitsuyoshi
- Department of Bioengineering, Graduate School of Engineering, The University of Tokyo, Tokyo 113-8656, Japan
| | - Taku Saito
- Department of Psychiatry, National Defense Medical College, Tokorozawa 359-8513, Japan
| | - Aihide Yoshino
- Department of Psychiatry, National Defense Medical College, Tokorozawa 359-8513, Japan
| | - Hiroyuki Toda
- Department of Psychiatry, National Defense Medical College, Tokorozawa 359-8513, Japan
| | - Shinichi Tokuno
- Department of Bioengineering, Graduate School of Engineering, The University of Tokyo, Tokyo 113-8656, Japan
- Graduate School of Health Innovation, Kanagawa University of Human Services, Kawasaki 210-0821, Japan
| |
Collapse
|
7
|
Paccotacya-Yanque RYG, Huanca-Anquise CA, Escalante-Calcina J, Ramos-Lovón WR, Cuno-Parari ÁE. A speech corpus of Quechua Collao for automatic dimensional emotion recognition. Sci Data 2022; 9:778. [PMID: 36566260 PMCID: PMC9789950 DOI: 10.1038/s41597-022-01855-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Accepted: 11/21/2022] [Indexed: 12/25/2022] Open
Abstract
Automatic speech emotion recognition is an important research topic for human-computer interaction and affective computing. Over ten million people speak the Quechua language throughout South America, and one of the most known variants is the Quechua Collao one. However, this language can be considered a low resource for machine emotion recognition, creating a barrier for Quechua speakers who want to use this technology. Therefore, the contribution of this work is a 15 hours speech corpus in Quechua Collao, which is made publicly available to the research community. The corpus was created from a set of words and sentences explicitly collected for this task, divided into nine categorical emotions: happy, sad, bored, fear, sleepy, calm, excited, angry, and neutral. The annotation was performed on a 5-value discrete scale according to 3 dimensions: valence, arousal, and dominance. To demonstrate the usefulness of the corpus, we have performed speech emotion recognition using machine learning methods and neural networks.
Collapse
Affiliation(s)
- Rosa Y. G. Paccotacya-Yanque
- grid.441685.a0000 0004 0385 0297Universidad Nacional de San Agustín de Arequipa, School of Computer Science, Arequipa, Peru
| | - Candy A. Huanca-Anquise
- grid.441685.a0000 0004 0385 0297Universidad Nacional de San Agustín de Arequipa, School of Computer Science, Arequipa, Peru
| | - Judith Escalante-Calcina
- grid.441685.a0000 0004 0385 0297Universidad Nacional de San Agustín de Arequipa, School of Computer Science, Arequipa, Peru
| | - Wilber R. Ramos-Lovón
- grid.441685.a0000 0004 0385 0297Universidad Nacional de San Agustín de Arequipa, School of Computer Science, Arequipa, Peru
| | - Álvaro E. Cuno-Parari
- grid.441685.a0000 0004 0385 0297Universidad Nacional de San Agustín de Arequipa, School of Computer Science, Arequipa, Peru
| |
Collapse
|
8
|
Exploration of Despair Eccentricities Based on Scale Metrics with Feature Sampling Using a Deep Learning Algorithm. Diagnostics (Basel) 2022; 12:diagnostics12112844. [PMID: 36428903 PMCID: PMC9689169 DOI: 10.3390/diagnostics12112844] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2022] [Revised: 11/11/2022] [Accepted: 11/15/2022] [Indexed: 11/19/2022] Open
Abstract
The majority of people in the modern biosphere struggle with depression as a result of the coronavirus pandemic's impact, which has adversely impacted mental health without warning. Even though the majority of individuals are still protected, it is crucial to check for post-corona virus symptoms if someone is feeling a little lethargic. In order to identify the post-coronavirus symptoms and attacks that are present in the human body, the recommended approach is included. When a harmful virus spreads inside a human body, the post-diagnosis symptoms are considerably more dangerous, and if they are not recognised at an early stage, the risks will be increased. Additionally, if the post-symptoms are severe and go untreated, it might harm one's mental health. In order to prevent someone from succumbing to depression, the technology of audio prediction is employed to recognise all the symptoms and potentially dangerous signs. Different choral characters are used to combine machine-learning algorithms to determine each person's mental state. Design considerations are made for a separate device that detects audio attribute outputs in order to evaluate the effectiveness of the suggested technique; compared to the previous method, the performance metric is substantially better by roughly 67%.
Collapse
|
9
|
Kaur B, Rathi S, Agrawal RK. Enhanced depression detection from speech using Quantum Whale Optimization Algorithm for feature selection. Comput Biol Med 2022; 150:106122. [PMID: 36182759 DOI: 10.1016/j.compbiomed.2022.106122] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2022] [Revised: 08/27/2022] [Accepted: 09/17/2022] [Indexed: 11/03/2022]
Abstract
There is an urgent need to detect depression using a non-intrusive approach that is reliable and accurate. In this paper, a simple and efficient unimodal depression detection approach based on speech is proposed, which is non-invasive, cost-effective and computationally inexpensive. A set of spectral, temporal and spectro-temporal features is derived from the speech signal of healthy and depressed subjects. To select a minimal subset of the relevant and non-redundant speech features to detect depression, a two-phase approach based on the nature-inspired wrapper-based feature selection Quantum-based Whale Optimization Algorithm (QWOA) is proposed. Experiments are performed on the publicly available Distress Analysis Interview Corpus Wizard-of-Oz (DAICWOZ) dataset and compared with three established univariate filtering techniques for feature selection and four well-known evolutionary algorithms. The proposed model outperforms all the univariate filter feature selection techniques and the evolutionary algorithms. It has low computational complexity in comparison to traditional wrapper-based evolutionary methods. The performance of the proposed approach is superior in comparison to existing unimodal and multimodal automated depression detection models. The combination of spectral, temporal and spectro-temporal speech features gave the best result with the LDA classifier. The performance achieved with the proposed approach, in terms of F1-score for the depressed class and the non-depressed class and error is 0.846, 0.932 and 0.094 respectively. Statistical tests demonstrate that the acoustic features selected using the proposed approach are non-redundant and discriminatory. Statistical tests also establish that the performance of the proposed approach is significantly better than that of the traditional wrapper-based evolutionary methods.
Collapse
Affiliation(s)
| | - Swati Rathi
- School of Computer and Systems Sciences, Jawaharlal Nehru University, Delhi, India.
| | - R K Agrawal
- School of Computer and Systems Sciences, Jawaharlal Nehru University, Delhi, India.
| |
Collapse
|
10
|
Othmani A, Zeghina AO, Muzammel M. A Model of Normality Inspired Deep Learning Framework for Depression Relapse Prediction Using Audiovisual Data. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 226:107132. [PMID: 36183638 DOI: 10.1016/j.cmpb.2022.107132] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Revised: 09/04/2022] [Accepted: 09/13/2022] [Indexed: 06/16/2023]
Abstract
BACKGROUND Depression (Major Depressive Disorder) is one of the most common mental illnesses. According to the World Health Organization, more than 300 million people in the world are affected. A first depressive episode can be solved by a spontaneous remission within 6 to 12 months. It has been shown that depression affects speech production and facial expressions. Although numerous studies are proposed in the literature for depression recognition using audiovisual cues, depression relapse using audiovisual cues has not been studied in the literature. METHOD In this paper, we propose a deep learning-based approach for depression recognition and depression relapse prediction using audiovisual data. For more versatility and reusability, the proposed approach is based on a Model of Normality inspired framework where we define depression relapse by the closeness of the audiovisual patterns of a subject after a symptom-free period to the audiovisual patterns of depressed subjects. A model of Normality is an anomaly detection distance-based approach that computes a distance of normality between the deep audiovisual encoding of a test sample and a learned representation from audiovisual encodings of anomaly-free data. RESULTS The proposed approach shows a very promising results with an accuracy of 87.4% and a F1-score of 82.3% for relapse/depression prediction using a Leave-One-Subject-Out training strategy on the DAIC-Woz dataset. CONCLUSION The proposed model of normality-based framework is accurate in detecting depression and in predicting depression relapse. A prospective monitoring system is proposed for assisting depressed patients. The proposed framework is easily extensible and others modalities will be integrated in future works.
Collapse
Affiliation(s)
- Alice Othmani
- Université Paris-Est Créteil (UPEC), LISSI, Vitry sur Seine 94400, France.
| | | | - Muhammad Muzammel
- Université Paris-Est Créteil (UPEC), LISSI, Vitry sur Seine 94400, France
| |
Collapse
|
11
|
Wu P, Wang R, Lin H, Zhang F, Tu J, Sun M. Automatic depression recognition by intelligent speech signal processing: A systematic survey. CAAI TRANSACTIONS ON INTELLIGENCE TECHNOLOGY 2022. [DOI: 10.1049/cit2.12113] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022] Open
Affiliation(s)
- Pingping Wu
- Jiangsu Key Laboratory of Public Project Audit, School of Engineering Audit Nanjing Audit University Nanjing China
| | - Ruihao Wang
- School of Information Engineering Nanjing Audit University Nanjing China
| | - Han Lin
- Jiangsu Key Laboratory of Public Project Audit, School of Engineering Audit Nanjing Audit University Nanjing China
| | - Fanlong Zhang
- School of Information Engineering Nanjing Audit University Nanjing China
| | - Juan Tu
- Key Laboratory of Modern Acoustics (MOE), School of Physics Nanjing University Nanjing China
| | - Miao Sun
- Faculty of Electrical Engineering, Mathematics & Computer Science Delft University of Technology Delft The Netherlands
| |
Collapse
|
12
|
Chen Z, Li M, Wang R, Sun W, Liu J, Li H, Wang T, Lian Y, Zhang J, Wang X. Diagnosis of COVID-19 via acoustic analysis and artificial intelligence by monitoring breath sounds on smartphones. J Biomed Inform 2022; 130:104078. [PMID: 35489595 PMCID: PMC9044719 DOI: 10.1016/j.jbi.2022.104078] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2021] [Revised: 04/09/2022] [Accepted: 04/16/2022] [Indexed: 01/25/2023]
Abstract
Scientific evidence shows that acoustic analysis could be an indicator for diagnosing COVID-19. From analyzing recorded breath sounds on smartphones, it is discovered that patients with COVID-19 have different patterns in both the time domain and frequency domain. These patterns are used in this paper to diagnose the infection of COVID-19. Statistics of the sound signals, analysis in the frequency domain, and Mel-Frequency Cepstral Coefficients (MFCCs) are then calculated and applied in two classifiers, k-Nearest Neighbors (kNN) and Convolutional Neural Network (CNN), to diagnose whether a user is contracted with COVID-19 or not. Test results show that, amazingly, an accuracy of over 97% could be achieved with a CNN classifier and more than 85% on kNN with optimized features. Optimization methods for selecting the best features and using various metrics to evaluate the performance are also demonstrated in this paper. Owing to the high accuracy of the CNN model, the CNN model was implemented in an Android app to diagnose COVID-19 with a probability to indicate the confidence level. The initial medical test shows a similar test result between the method proposed in this paper and the lateral flow method, which indicates that the proposed method is feasible and effective. Because of the use of breath sound and tested on the smartphone, this method could be used by everybody regardless of the availability of other medical resources, which could be a powerful tool for society to diagnose COVID-19.
Collapse
|
13
|
Calić G, Petrović-Lazić M, Mentus T, Babac S. Acoustic features of voice in adults suffering from depression. PSIHOLOSKA ISTRAZIVANJA 2022. [DOI: 10.5937/psistra25-39224] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/08/2023]
Abstract
In order to examine the differences in people suffering from depression (EG, N=18) compared to the healthy controls (CG1, N=24) and people with the diagnosed psychogenic voice disorder (CG2, N=9), nine acoustic features of voice were assessed among the total of 51 participants using the MDVP software programme ("Kay Elemetrics" Corp., model 4300). Nine acoustic parameters were analysed on the basis of the sustained phonation of the vowel /a/. The results revealed that the mean values of all acoustic parameters differed in the EG compared to both the CG1 and CG2 as follows: the parameters which indicate frequency variability (Jitt, PPQ), amplitude variability (Shim, vAm, APQ) and noise and tremor parameters (NHR, VTI) were higher; only the parameters of fundamental frequency (F0) and soft index phonation (SPI) were lower (F0 compared to CG1, and SPI compared to CG1 and CG2). Only the PPQ parameter was not significant. vAm and APQ had the highest discriminant value for depression. The acoustic features of voice, analysed in this study with regard to the sustained phonation of a vowel, were different and discriminant in the EG compared to CG1 and CG2. In voice analysis, the parameters vAm and APQ could potentially be the markers indicative of depression. The results of this research point to the importance of the voice, that is, its acoustic indicators, in recognizing depression. Important parameters that could help create a programme for the automatic recognition of depression are those from the domain of voice intensity variation.
Collapse
|
14
|
Hajduska-Dér B, Kiss G, Sztahó D, Vicsi K, Simon L. The applicability of the Beck Depression Inventory and Hamilton Depression Scale in the automatic recognition of depression based on speech signal processing. Front Psychiatry 2022; 13:879896. [PMID: 35990073 PMCID: PMC9385975 DOI: 10.3389/fpsyt.2022.879896] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/20/2022] [Accepted: 07/18/2022] [Indexed: 11/25/2022] Open
Abstract
Depression is a growing problem worldwide, impacting on an increasing number of patients, and also affecting health systems and the global economy. The most common diagnostical rating scales of depression are self-reported or clinician-administered, which differ in the symptoms that they are sampling. Speech is a promising biomarker in the diagnostical assessment of depression, due to non-invasiveness and cost and time efficiency. In our study, we try to achieve a more accurate, sensitive model for determining depression based on speech processing. Regression and classification models were also developed using a machine learning method. During the research, we had access to a large speech database that includes speech samples from depressed and healthy subjects. The database contains the Beck Depression Inventory (BDI) score of each subject and the Hamilton Rating Scale for Depression (HAMD) score of 20% of the subjects. This fact provided an opportunity to compare the usefulness of BDI and HAMD for training models of automatic recognition of depression based on speech signal processing. We found that the estimated values of the acoustic model trained on BDI scores are closer to HAMD assessment than to the BDI scores, and the partial application of HAMD scores instead of BDI scores in training improves the accuracy of automatic recognition of depression.
Collapse
Affiliation(s)
- Bálint Hajduska-Dér
- Department of Psychiatry and Psychotherapy, Semmelweis University, Budapest, Hungary
| | - Gábor Kiss
- Department of Telecommunications and Media Informatics, Budapest University of Technology and Economics, Budapest, Hungary
| | - Dávid Sztahó
- Department of Telecommunications and Media Informatics, Budapest University of Technology and Economics, Budapest, Hungary
| | - Klára Vicsi
- Department of Telecommunications and Media Informatics, Budapest University of Technology and Economics, Budapest, Hungary
| | - Lajos Simon
- Department of Psychiatry and Psychotherapy, Semmelweis University, Budapest, Hungary
| |
Collapse
|
15
|
Rejaibi E, Komaty A, Meriaudeau F, Agrebi S, Othmani A. MFCC-based Recurrent Neural Network for automatic clinical depression recognition and assessment from speech. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103107] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
16
|
Yamada Y, Shinkawa K, Nemoto M, Arai T. Automatic Assessment of Loneliness in Older Adults Using Speech Analysis on Responses to Daily Life Questions. Front Psychiatry 2021; 12:712251. [PMID: 34966297 PMCID: PMC8710612 DOI: 10.3389/fpsyt.2021.712251] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/20/2021] [Accepted: 11/19/2021] [Indexed: 11/13/2022] Open
Abstract
Loneliness is a perceived state of social and emotional isolation that has been associated with a wide range of adverse health effects in older adults. Automatically assessing loneliness by passively monitoring daily behaviors could potentially contribute to early detection and intervention for mitigating loneliness. Speech data has been successfully used for inferring changes in emotional states and mental health conditions, but its association with loneliness in older adults remains unexplored. In this study, we developed a tablet-based application and collected speech responses of 57 older adults to daily life questions regarding, for example, one's feelings and future travel plans. From audio data of these speech responses, we automatically extracted speech features characterizing acoustic, prosodic, and linguistic aspects, and investigated their associations with self-rated scores of the UCLA Loneliness Scale. Consequently, we found that with increasing loneliness scores, speech responses tended to have less inflections, longer pauses, reduced second formant frequencies, reduced variances of the speech spectrum, more filler words, and fewer positive words. The cross-validation results showed that regression and binary-classification models using speech features could estimate loneliness scores with an R 2 of 0.57 and detect individuals with high loneliness scores with 95.6% accuracy, respectively. Our study provides the first empirical results suggesting the possibility of using speech data that can be collected in everyday life for the automatic assessments of loneliness in older adults, which could help develop monitoring technologies for early detection and intervention for mitigating loneliness.
Collapse
Affiliation(s)
| | | | - Miyuki Nemoto
- Dementia Medical Center, University of Tsukuba Hospital, Tsukuba, Japan
| | - Tetsuaki Arai
- Division of Clinical Medicine, Department of Psychiatry, Faculty of Medicine, University of Tsukuba, Tsukuba, Japan
| |
Collapse
|
17
|
Wang J, Lv K, Liu C, Nie X, Gowda D, Luan S. Automatic Assessment for Severe Self-Reported Depressive Symptoms Using Speech Cues. IEEE Trans Cogn Dev Syst 2021. [DOI: 10.1109/tcds.2020.3002512] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
18
|
Klangpornkun N, Ruangritchai M, Munthuli A, Onsuwan C, Jaisin K, Pattanaseri K, Lortrakul J, Thanakulakkarachai P, Anansiripinyo T, Amornlaksananon A, Laohawee S, Tantibundhit C. Classification of Depression and Other Psychiatric Conditions Using Speech Features Extracted from a Thai Psychiatric and Verbal Screening Test. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:651-656. [PMID: 34891377 DOI: 10.1109/embc46164.2021.9629571] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Depression is a common and serious mental illness which negatively affects daily functioning. To prevent the progression of the illness into severe or long-term consequences, early diagnosis is crucial. We developed an automated speech feature analysis application for depression and other psychiatric disorders derived from a developed Thai psychiatric and verbal screening test. The screening test includes Thai's version of Patient Health Questionnaire-9 (PHQ-9) and Hamilton Depression Rating Scale (HAM-D), and 32 additional emotion-induced questions. Case-control study was conducted on speech features from 66 participants. Twenty seven of those had depression (DP), 12 had other psychiatric disorders (OP), and 27 were normal controls (NC). The five-fold cross-validation from 6 settings of 5 classifiers with the combination of PHQ-9 and HAM-D scores, and speech features were examined. Results showed highest performance from the multilayer perceptron (MLP) classifier which yielded 83.33% sensitivity, 91.67% specificity, and 83.33% accuracy, where negative-emotional questions were most effective in classification. The automated speech feature analysis showed promising results for screening patients with depression or other psychiatric disorders. The current application is accessible through smartphone, making it a feasible and intuitive setup for low-resource countries such as Thailand.
Collapse
|
19
|
Muzammel M, Salam H, Othmani A. End-to-end multimodal clinical depression recognition using deep neural networks: A comparative analysis. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 211:106433. [PMID: 34614452 DOI: 10.1016/j.cmpb.2021.106433] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/02/2020] [Accepted: 09/15/2021] [Indexed: 06/13/2023]
Abstract
BACKGROUND AND OBJECTIVE Major Depressive Disorder is a highly prevalent and disabling mental health condition. Numerous studies explored multimodal fusion systems combining visual, audio, and textual features via deep learning architectures for clinical depression recognition. Yet, no comparative analysis for multimodal depression analysis has been proposed in the literature. METHODS In this paper, an up-to-date literature overview of multimodal depression recognition is presented and an extensive comparative analysis of different deep learning architectures for depression recognition is performed. First, audio features based Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) are studied. Then, early-level and model-level fusion of deep audio features with visual and textual features through LSTM and CNN architectures are investigated. RESULTS The performance of the proposed architectures using an hold-out strategy on the DAIC-WOZ dataset (80% training, 10% validation, 10% test split) for binary and severity levels of depression recognition is tested. Using this strategy, a set of experiments have been performed and they have demonstrated: (1) LSTM-based audio features perform slightly better than CNN ones with an accuracy of 66.25% versus 65.60% for binary depression classes. (2) the model level fusion of deep audio and visual features using LSTM network performed the best with an accuracy of 77.16%, a precision of 53% for the depressed class, and a precision of 83% for the non-depressed class. The given network obtained a normalized Root Mean Square Error (RMSE) of 0.15 for depression severity level prediction. Using a Leave-One-Subject-Out strategy, this network achieved an accuracy of 95.38% for binary depression detection, and a normalized RMSE of 0.1476 for depression severity level prediction. Our best-performing architecture outperforms all state-of-the-art approaches on DAIC-WOZ dataset. CONCLUSIONS The obtained results show that the proposed LSTM-based surpass the proposed CNN-based architectures allowing to learn temporal dynamics representations of multimodal features. Furthermore, model-level fusion of audio and visual features using an LSTM network leads to the best performance. Our best-performing architecture successfully detects depression using a speech segment of less than 8 seconds, and an average prediction computation time of less than 6ms; making it suitable for real-world clinical applications.
Collapse
Affiliation(s)
- Muhammad Muzammel
- Université Paris-Est Créteil (UPEC), LISSI, Vitry sur Seine 94400, France
| | - Hanan Salam
- New York University, SMART Lab, Saadiyat Island, Abu Dhabi
| | - Alice Othmani
- Université Paris-Est Créteil (UPEC), LISSI, Vitry sur Seine 94400, France.
| |
Collapse
|
20
|
MacLeod L, Suruliraj B, Gall D, Bessenyei K, Hamm S, Romkey I, Bagnell A, Mattheisen M, Muthukumaraswamy V, Orji R, Meier S. A Mobile Sensing App to Monitor Youth Mental Health: Observational Pilot Study. JMIR Mhealth Uhealth 2021; 9:e20638. [PMID: 34698650 PMCID: PMC8579216 DOI: 10.2196/20638] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2020] [Revised: 02/02/2021] [Accepted: 07/27/2021] [Indexed: 01/19/2023] Open
Abstract
Background Internalizing disorders are the most common psychiatric problems observed among youth in Canada. Sadly, youth with internalizing disorders often avoid seeking clinical help and rarely receive adequate treatment. Current methods of assessing internalizing disorders usually rely on subjective symptom ratings, but internalizing symptoms are frequently underreported, which creates a barrier to the accurate assessment of these symptoms in youth. Therefore, novel assessment tools that use objective data need to be developed to meet the highest standards of reliability, feasibility, scalability, and affordability. Mobile sensing technologies, which unobtrusively record aspects of youth behaviors in their daily lives with the potential to make inferences about their mental health states, offer a possible method of addressing this assessment barrier. Objective This study aims to explore whether passively collected smartphone sensor data can be used to predict internalizing symptoms among youth in Canada. Methods In this study, the youth participants (N=122) completed self-report assessments of symptoms of anxiety, depression, and attention-deficit hyperactivity disorder. Next, the participants installed an app, which passively collected data about their mobility, screen time, sleep, and social interactions over 2 weeks. Then, we tested whether these passive sensor data could be used to predict internalizing symptoms among these youth participants. Results More severe depressive symptoms correlated with more time spent stationary (r=0.293; P=.003), less mobility (r=0.271; P=.006), higher light intensity during the night (r=0.227; P=.02), and fewer outgoing calls (r=−0.244; P=.03). In contrast, more severe anxiety symptoms correlated with less time spent stationary (r=−0.249; P=.01) and greater mobility (r=0.234; P=.02). In addition, youths with higher anxiety scores spent more time on the screen (r=0.203; P=.049). Finally, adding passively collected smartphone sensor data to the prediction models of internalizing symptoms significantly improved their fit. Conclusions Passively collected smartphone sensor data provide a useful way to monitor internalizing symptoms among youth. Although the results replicated findings from adult populations, to ensure clinical utility, they still need to be replicated in larger samples of youth. The work also highlights intervention opportunities via mobile technology to reduce the burden of internalizing symptoms early on.
Collapse
Affiliation(s)
- Lucy MacLeod
- Department of Psychiatry, Dalhousie University, Halifax, NS, Canada
| | | | - Dominik Gall
- Department of Psychology, University of Würzburg, Würzburg, Germany
| | - Kitti Bessenyei
- Department of Psychiatry, Dalhousie University, Halifax, NS, Canada
| | - Sara Hamm
- Department of Psychiatry, Dalhousie University, Halifax, NS, Canada
| | - Isaac Romkey
- Department of Psychiatry, Dalhousie University, Halifax, NS, Canada
| | - Alexa Bagnell
- Department of Psychiatry, Dalhousie University, Halifax, NS, Canada
| | | | | | - Rita Orji
- Faculty of Computer Science, Dalhousie University, Halifax, NS, Canada
| | - Sandra Meier
- Department of Psychiatry, Dalhousie University, Halifax, NS, Canada
| |
Collapse
|
21
|
König A, Mallick E, Tröger J, Linz N, Zeghari R, Manera V, Robert P. Measuring neuropsychiatric symptoms in patients with early cognitive decline using speech analysis. Eur Psychiatry 2021; 64:e64. [PMID: 34641989 PMCID: PMC8581700 DOI: 10.1192/j.eurpsy.2021.2236] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/14/2022] Open
Abstract
BACKGROUND Certain neuropsychiatric symptoms (NPS), namely apathy, depression, and anxiety demonstrated great value in predicting dementia progression, representing eventually an opportunity window for timely diagnosis and treatment. However, sensitive and objective markers of these symptoms are still missing. Therefore, the present study aims to investigate the association between automatically extracted speech features and NPS in patients with mild neurocognitive disorders. METHODS Speech of 141 patients aged 65 or older with neurocognitive disorder was recorded while performing two short narrative speech tasks. NPS were assessed by the neuropsychiatric inventory. Paralinguistic markers relating to prosodic, formant, source, and temporal qualities of speech were automatically extracted, correlated with NPS. Machine learning experiments were carried out to validate the diagnostic power of extracted markers. RESULTS Different speech variables are associated with specific NPS; apathy correlates with temporal aspects, and anxiety with voice quality-and this was mostly consistent between male and female after correction for cognitive impairment. Machine learning regressors are able to extract information from speech features and perform above baseline in predicting anxiety, apathy, and depression scores. CONCLUSIONS Different NPS seem to be characterized by distinct speech features, which are easily extractable automatically from short vocal tasks. These findings support the use of speech analysis for detecting subtypes of NPS in patients with cognitive impairment. This could have great implications for the design of future clinical trials as this cost-effective method could allow more continuous and even remote monitoring of symptoms.
Collapse
Affiliation(s)
- Alexandra König
- Stars Team, Sophia Antipolis, Institut National de Recherche en Informatique et en Automatique (INRIA), Valbonne, France.,Clinical Research, ki:elements, Saarbrücken, Germany.,CoBTeK (Cognition-Behaviour-Technology) Lab, FRIS-University Côte d'Azur, Nice, France
| | - Elisa Mallick
- Stars Team, Sophia Antipolis, Institut National de Recherche en Informatique et en Automatique (INRIA), Valbonne, France.,Clinical Research, ki:elements, Saarbrücken, Germany.,CoBTeK (Cognition-Behaviour-Technology) Lab, FRIS-University Côte d'Azur, Nice, France
| | - Johannes Tröger
- Stars Team, Sophia Antipolis, Institut National de Recherche en Informatique et en Automatique (INRIA), Valbonne, France.,Clinical Research, ki:elements, Saarbrücken, Germany.,CoBTeK (Cognition-Behaviour-Technology) Lab, FRIS-University Côte d'Azur, Nice, France
| | - Nicklas Linz
- Stars Team, Sophia Antipolis, Institut National de Recherche en Informatique et en Automatique (INRIA), Valbonne, France.,Clinical Research, ki:elements, Saarbrücken, Germany.,CoBTeK (Cognition-Behaviour-Technology) Lab, FRIS-University Côte d'Azur, Nice, France
| | - Radia Zeghari
- Stars Team, Sophia Antipolis, Institut National de Recherche en Informatique et en Automatique (INRIA), Valbonne, France.,Clinical Research, ki:elements, Saarbrücken, Germany.,CoBTeK (Cognition-Behaviour-Technology) Lab, FRIS-University Côte d'Azur, Nice, France
| | - Valeria Manera
- Stars Team, Sophia Antipolis, Institut National de Recherche en Informatique et en Automatique (INRIA), Valbonne, France.,Clinical Research, ki:elements, Saarbrücken, Germany.,CoBTeK (Cognition-Behaviour-Technology) Lab, FRIS-University Côte d'Azur, Nice, France
| | - Philippe Robert
- Stars Team, Sophia Antipolis, Institut National de Recherche en Informatique et en Automatique (INRIA), Valbonne, France.,Clinical Research, ki:elements, Saarbrücken, Germany.,CoBTeK (Cognition-Behaviour-Technology) Lab, FRIS-University Côte d'Azur, Nice, France
| |
Collapse
|
22
|
Aydemir E, Tuncer T, Dogan S, Gururajan R, Acharya UR. Automated major depressive disorder detection using melamine pattern with EEG signals. APPL INTELL 2021. [DOI: 10.1007/s10489-021-02426-y] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
23
|
Lee S, Suh SW, Kim T, Kim K, Lee KH, Lee JR, Han G, Hong JW, Han JW, Lee K, Kim KW. Screening major depressive disorder using vocal acoustic features in the elderly by sex. J Affect Disord 2021; 291:15-23. [PMID: 34022551 DOI: 10.1016/j.jad.2021.04.098] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/09/2020] [Revised: 01/12/2021] [Accepted: 04/25/2021] [Indexed: 10/21/2022]
Abstract
BACKGROUND Vocal acoustic features are potential biomarkers of elderly depression. Previous automated diagnostic tests for depression have employed unstandardized speech samples, and few studies have considered differences in voice reactivity. We aimed to develop a voice-based screening test for depression measuring vocal acoustic features of elderly Koreans while they read a series of mood-inducing sentences (MIS). METHODS In this case-control study, we recruited 61 individuals with major depressive disorder and 143 healthy controls (mean age [SD]: 72 [6]; female, 70%) from the community-dwelling elderly population. Participants were asked to read MIS and their variation pattern of acoustic features represented by the correlation distance between two MIS were analyzed as input features using the univariate feature selection technique and subsequently classified by AdaBoost. RESULTS Acoustic features showing significant discriminatory performances were spectral and energy-related features for males (sensitivity 0.95, specificity 0.88, and accuracy 0.86) and prosody-related features for females (sensitivity 0.73, specificity 0.86, and accuracy 0.77). The correlation distance between negative and positive MIS was significantly shorter in the depressed group than in the healthy control (F = 18.574, P < 0.001). LIMITATIONS Small sample size and relatively homogenous clinical profile of depression could limit the generalizability. CONCLUSIONS While reading MIS, spectral and energy-related acoustic features for males and prosody-related features for females are good discriminators for major depressive disorder. These features may be used as biomarkers of depression in the elderly.
Collapse
Affiliation(s)
- Subin Lee
- Music and Audio Research Group, Seoul National University, Seoul, Korea
| | - Seung Wan Suh
- Department of Psychiatry, Kangdong Sacred Heart Hospital, Hallym University College of Medicine, Seoul, Korea
| | - Taehyun Kim
- Department of Neuropsychiatry, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Kayoung Kim
- Department of Neuropsychiatry, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Kyoung Hwan Lee
- Department of Neuropsychiatry, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Ju Ri Lee
- Department of Neuropsychiatry, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Guehee Han
- Department of Neuropsychiatry, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Jong Woo Hong
- Department of Neuropsychiatry, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Ji Won Han
- Department of Neuropsychiatry, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Kyogu Lee
- Music and Audio Research Group, Seoul National University, Seoul, Korea; Graduate School of Convergence Science and Technology, Seoul National University, Seoul, Korea.
| | - Ki Woong Kim
- Department of Neuropsychiatry, Seoul National University Bundang Hospital, Seongnam, Korea; Department of Psychiatry, Seoul National University, College of Medicine, Seoul, Korea; Department of Brain and Cognitive Sciences, Seoul National University, College of Natural Sciences, Seoul, Korea.
| |
Collapse
|
24
|
König A, Riviere K, Linz N, Lindsay H, Elbaum J, Fabre R, Derreumaux A, Robert P. Measuring Stress in Health Professionals Over the Phone Using Automatic Speech Analysis During the COVID-19 Pandemic: Observational Pilot Study. J Med Internet Res 2021; 23:e24191. [PMID: 33739930 PMCID: PMC8057197 DOI: 10.2196/24191] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2020] [Revised: 11/13/2020] [Accepted: 03/17/2021] [Indexed: 01/14/2023] Open
Abstract
Background During the COVID-19 pandemic, health professionals have been directly confronted with the suffering of patients and their families. By making them main actors in the management of this health crisis, they have been exposed to various psychosocial risks (stress, trauma, fatigue, etc). Paradoxically, stress-related symptoms are often underreported in this vulnerable population but are potentially detectable through passive monitoring of changes in speech behavior. Objective This study aims to investigate the use of rapid and remote measures of stress levels in health professionals working during the COVID-19 outbreak. This was done through the analysis of participants’ speech behavior during a short phone call conversation and, in particular, via positive, negative, and neutral storytelling tasks. Methods Speech samples from 89 health care professionals were collected over the phone during positive, negative, and neutral storytelling tasks; various voice features were extracted and compared with classical stress measures via standard questionnaires. Additionally, a regression analysis was performed. Results Certain speech characteristics correlated with stress levels in both genders; mainly, spectral (ie, formant) features, such as the mel-frequency cepstral coefficient, and prosodic characteristics, such as the fundamental frequency, appeared to be sensitive to stress. Overall, for both male and female participants, using vocal features from the positive tasks for regression yielded the most accurate prediction results of stress scores (mean absolute error 5.31). Conclusions Automatic speech analysis could help with early detection of subtle signs of stress in vulnerable populations over the phone. By combining the use of this technology with timely intervention strategies, it could contribute to the prevention of burnout and the development of comorbidities, such as depression or anxiety.
Collapse
Affiliation(s)
- Alexandra König
- Stars Team, Institut national de recherche en informatique et en automatique, Valbonne, France
| | - Kevin Riviere
- Département de Santé Publique, Centre Hospitalier Universitaire de Nice, Université Côte d'Azur, Nice, France
| | | | - Hali Lindsay
- German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany
| | - Julia Elbaum
- Département de Santé Publique, Centre Hospitalier Universitaire de Nice, Université Côte d'Azur, Nice, France
| | - Roxane Fabre
- Département de Santé Publique, Centre Hospitalier Universitaire de Nice, Université Côte d'Azur, Nice, France
| | - Alexandre Derreumaux
- CoBteK (Cognition-Behaviour-Technology) Lab, La Fédération de Recherche Interventions en Santé, Université Côte d'Azur, Nice, France
| | - Philippe Robert
- CoBteK (Cognition-Behaviour-Technology) Lab, La Fédération de Recherche Interventions en Santé, Université Côte d'Azur, Nice, France
| |
Collapse
|
25
|
What You Say or How You Say It? Depression Detection Through Joint Modeling of Linguistic and Acoustic Aspects of Speech. Cognit Comput 2021. [DOI: 10.1007/s12559-020-09808-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
Abstract
AbstractDepression is one of the most common mental health issues. (It affects more than 4% of the world’s population, according to recent estimates.) This article shows that the joint analysis of linguistic and acoustic aspects of speech allows one to discriminate between depressed and nondepressed speakers with an accuracy above 80%. The approach used in the work is based on networks designed for sequence modeling (bidirectional Long-Short Term Memory networks) and multimodal analysis methodologies (late fusion, joint representation and gated multimodal units). The experiments were performed over a corpus of 59 interviews (roughly 4 hours of material) involving 29 individuals diagnosed with depression and 30 control participants. In addition to an accuracy of 80%, the results show that multimodal approaches perform better than unimodal ones owing to people’s tendency to manifest their condition through one modality only, a source of diversity across unimodal approaches. In addition, the experiments show that it is possible to measure the “confidence” of the approach and automatically identify a subset of the test data in which the performance is above a predefined threshold. It is possible to effectively detect depression by using unobtrusive and inexpensive technologies based on the automatic analysis of speech and language.
Collapse
|
26
|
Latif S, Qadir J, Qayyum A, Usama M, Younis S. Speech Technology for Healthcare: Opportunities, Challenges, and State of the Art. IEEE Rev Biomed Eng 2021; 14:342-356. [PMID: 32746367 DOI: 10.1109/rbme.2020.3006860] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Speech technology is not appropriately explored even though modern advances in speech technology-especially those driven by deep learning (DL) technology-offer unprecedented opportunities for transforming the healthcare industry. In this paper, we have focused on the enormous potential of speech technology for revolutionising the healthcare domain. More specifically, we review the state-of-the-art approaches in automatic speech recognition (ASR), speech synthesis or text to speech (TTS), and health detection and monitoring using speech signals. We also present a comprehensive overview of various challenges hindering the growth of speech-based services in healthcare. To make speech-based healthcare solutions more prevalent, we discuss open issues and suggest some possible research directions aimed at fully leveraging the advantages of other technologies for making speech-based healthcare solutions more effective.
Collapse
|
27
|
Muzammel M, Salam H, Hoffmann Y, Chetouani M, Othmani A. AudVowelConsNet: A phoneme-level based deep CNN architecture for clinical depression diagnosis. MACHINE LEARNING WITH APPLICATIONS 2020. [DOI: 10.1016/j.mlwa.2020.100005] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022] Open
|
28
|
What reveals about depression level? The role of multimodal features at the level of interview questions. INFORMATION & MANAGEMENT 2020. [DOI: 10.1016/j.im.2020.103349] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
|
29
|
Robin J, Harrison JE, Kaufman LD, Rudzicz F, Simpson W, Yancheva M. Evaluation of Speech-Based Digital Biomarkers: Review and Recommendations. Digit Biomark 2020; 4:99-108. [PMID: 33251474 DOI: 10.1159/000510820] [Citation(s) in RCA: 57] [Impact Index Per Article: 14.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2020] [Accepted: 08/11/2020] [Indexed: 12/23/2022] Open
Abstract
Speech represents a promising novel biomarker by providing a window into brain health, as shown by its disruption in various neurological and psychiatric diseases. As with many novel digital biomarkers, however, rigorous evaluation is currently lacking and is required for these measures to be used effectively and safely. This paper outlines and provides examples from the literature of evaluation steps for speech-based digital biomarkers, based on the recent V3 framework (Goldsack et al., 2020). The V3 framework describes 3 components of evaluation for digital biomarkers: verification, analytical validation, and clinical validation. Verification includes assessing the quality of speech recordings and comparing the effects of hardware and recording conditions on the integrity of the recordings. Analytical validation includes checking the accuracy and reliability of data processing and computed measures, including understanding test-retest reliability, demographic variability, and comparing measures to reference standards. Clinical validity involves verifying the correspondence of a measure to clinical outcomes which can include diagnosis, disease progression, or response to treatment. For each of these sections, we provide recommendations for the types of evaluation necessary for speech-based biomarkers and review published examples. The examples in this paper focus on speech-based biomarkers, but they can be used as a template for digital biomarker development more generally.
Collapse
Affiliation(s)
| | - John E Harrison
- Metis Cognition Ltd., Park House, Kilmington Common, Warminster, United Kingdom.,Alzheimer Center, AUmc, Amsterdam, The Netherlands.,Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
| | | | - Frank Rudzicz
- Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, Canada.,Department of Computer Science, University of Toronto, Toronto, Ontario, Canada.,Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada
| | - William Simpson
- Winterlight Labs, Toronto, Ontario, Canada.,Department of Psychiatry and Behavioural Neuroscience, McMaster University, Hamilton, Ontario, Canada
| | | |
Collapse
|
30
|
Sequeira L, Perrotta S, LaGrassa J, Merikangas K, Kreindler D, Kundur D, Courtney D, Szatmari P, Battaglia M, Strauss J. Mobile and wearable technology for monitoring depressive symptoms in children and adolescents: A scoping review. J Affect Disord 2020; 265:314-324. [PMID: 32090755 DOI: 10.1016/j.jad.2019.11.156] [Citation(s) in RCA: 34] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/20/2019] [Revised: 10/29/2019] [Accepted: 11/30/2019] [Indexed: 02/08/2023]
Abstract
BACKGROUND There has been rapid growth of mobile and wearable tools that may help to overcome challenges in the diagnosis and prediction of Major Depressive Disorder in children and adolescents, tasks that rely on clinical reporting that is inherently based on retrospective recall of symptoms and associated features. This article reviews more objective ways of measuring and monitoring mood within this population. METHODS A scoping review of peer-reviewed studies examined published research that employs mobile and wearable tools to characterize depression in children and/or adolescents. Our search strategy included the following terms: (1) monitoring or prediction (2) depression (3) mobile apps or wearables and (4) children and youth (including adolescents), and was applied to five databases. RESULTS Our search produced 829 citations (2008- Feb 2019), of which 30 (journal articles, conference papers and abstracts) were included in the analysis, and 2 reviews included in our discussion. The majority of the evidence involved smartphone apps, with very few studies using actigraphy. Mobile and wearables captured a variety of data including unobtrusive passive analytics, movement and light data, plus physical and mental health data, including depressive symptom monitoring. Most studies also examined feasibility. LIMITATIONS This review was limited to published research in the English language. The review criteria excluded any apps that were mainly treatment focused, therefore there was not much of a focus on clinical outcomes. CONCLUSIONS This scoping review yielded a variety of studies with heterogeneous populations, research methods and study objectives, which limited our ability to address our research objectives cohesively. Certain mobile technologies, however, have demonstrated feasibility for tracking depression that could inform models for predicting relapse.
Collapse
Affiliation(s)
- Lydia Sequeira
- Cundill Centre for Child and Youth Depression, Centre for Addiction and Mental Health, Toronto, ON, Canada; Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada
| | - Steve Perrotta
- Cundill Centre for Child and Youth Depression, Centre for Addiction and Mental Health, Toronto, ON, Canada
| | - Jennifer LaGrassa
- Cundill Centre for Child and Youth Depression, Centre for Addiction and Mental Health, Toronto, ON, Canada
| | | | - David Kreindler
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada; Centre for Mobile Computing in Mental Health, Sunnybrook Health Sciences Centre, Toronto, ON, Canada
| | - Deepa Kundur
- Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON, Canada
| | - Darren Courtney
- Cundill Centre for Child and Youth Depression, Centre for Addiction and Mental Health, Toronto, ON, Canada; Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Peter Szatmari
- Cundill Centre for Child and Youth Depression, Centre for Addiction and Mental Health, Toronto, ON, Canada; Department of Psychiatry, University of Toronto, Toronto, ON, Canada; Hospital for Sick Children, Toronto, ON, Canada
| | - Marco Battaglia
- Cundill Centre for Child and Youth Depression, Centre for Addiction and Mental Health, Toronto, ON, Canada; Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - John Strauss
- Cundill Centre for Child and Youth Depression, Centre for Addiction and Mental Health, Toronto, ON, Canada; Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada; Department of Psychiatry, University of Toronto, Toronto, ON, Canada.
| |
Collapse
|
31
|
Su MH, Wu CH, Huang KY, Yang TH. Cell-Coupled Long Short-Term Memory With L -Skip Fusion Mechanism for Mood Disorder Detection Through Elicited Audiovisual Features. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2020; 31:124-135. [PMID: 30892247 DOI: 10.1109/tnnls.2019.2899884] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
In early stages, patients with bipolar disorder are often diagnosed as having unipolar depression in mood disorder diagnosis. Because the long-term monitoring is limited by the delayed detection of mood disorder, an accurate and one-time diagnosis is desirable to avoid delay in appropriate treatment due to misdiagnosis. In this paper, an elicitation-based approach is proposed for realizing a one-time diagnosis by using responses elicited from patients by having them watch six emotion-eliciting videos. After watching each video clip, the conversations, including patient facial expressions and speech responses, between the participant and the clinician conducting the interview were recorded. Next, the hierarchical spectral clustering algorithm was employed to adapt the facial expression and speech response features by using the extended Cohn-Kanade and eNTERFACE databases. A denoizing autoencoder was further applied to extract the bottleneck features of the adapted data. Then, the facial and speech bottleneck features were input into support vector machines to obtain speech emotion profiles (EPs) and the modulation spectrum (MS) of the facial action unit sequence for each elicited response. Finally, a cell-coupled long short-term memory (LSTM) network with an L -skip fusion mechanism was proposed to model the temporal information of all elicited responses and to loosely fuse the EPs and the MS for conducting mood disorder detection. The experimental results revealed that the cell-coupled LSTM with the L -skip fusion mechanism has promising advantages and efficacy for mood disorder detection.
Collapse
|
32
|
Zhang X, Shen J, Din ZU, Liu J, Wang G, Hu B. Multimodal Depression Detection: Fusion of Electroencephalography and Paralinguistic Behaviors Using a Novel Strategy for Classifier Ensemble. IEEE J Biomed Health Inform 2019; 23:2265-2275. [PMID: 31478879 DOI: 10.1109/jbhi.2019.2938247] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Currently, depression has become a common mental disorder and one of the main causes of disability worldwide. Due to the difference in depressive symptoms evoked by individual differences, how to design comprehensive and effective depression detection methods has become an urgent demand. This study explored from physiological and behavioral perspectives simultaneously and fused pervasive electroencephalography (EEG) and vocal signals to make the detection of depression more objective, effective and convenient. After extraction of several effective features for these two types of signals, we trained six representational classifiers on each modality, then denoted diversity and correlation of decisions from different classifiers using co-decision tensor and combined these decisions into the ultimate classification result with multi-agent strategy. Experimental results on 170 (81 depressed patients and 89 normal controls) subjects showed that the proposed multi-modal depression detection strategy is superior to the single-modal classifiers or other typical late fusion strategies in accuracy, f1-score and sensitivity. This work indicates that late fusion of pervasive physiological and behavioral signals is promising for depression detection and the multi-agent strategy can take advantage of diversity and correlation of different classifiers effectively to gain a better final decision.
Collapse
|
33
|
Pan W, Flint J, Shenhav L, Liu T, Liu M, Hu B, Zhu T. Re-examining the robustness of voice features in predicting depression: Compared with baseline of confounders. PLoS One 2019; 14:e0218172. [PMID: 31220113 PMCID: PMC6586278 DOI: 10.1371/journal.pone.0218172] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2018] [Accepted: 05/28/2019] [Indexed: 11/19/2022] Open
Abstract
A large proportion of Depression Disorder patients do not receive an effective diagnosis, which makes it necessary to find a more objective assessment to facilitate a more rapid and accurate diagnosis of depression. Speech data is easy to acquire clinically, its association with depression has been studied, although the actual predictive effect of voice features has not been examined. Thus, we do not have a general understanding of the extent to which voice features contribute to the identification of depression. In this study, we investigated the significance of the association between voice features and depression using binary logistic regression, and the actual classification effect of voice features on depression was re-examined through classification modeling. Nearly 1000 Chinese females participated in this study. Several different datasets was included as test set. We found that 4 voice features (PC1, PC6, PC17, PC24, P<0.05, corrected) made significant contribution to depression, and that the contribution effect of the voice features alone reached 35.65% (Nagelkerke's R2). In classification modeling, voice data based model has consistently higher predicting accuracy(F-measure) than the baseline model of demographic data when tested on different datasets, even across different emotion context. F-measure of voice features alone reached 81%, consistent with existing data. These results demonstrate that voice features are effective in predicting depression and indicate that more sophisticated models based on voice features can be built to help in clinical diagnosis.
Collapse
Affiliation(s)
- Wei Pan
- Institute of Psychology, Chinese Academy of Sciences, Beijing, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Jonathan Flint
- Center for Neurobehavioral Genetics, Semel Institute for Neuroscience and Human Behavior, University of California Los Angeles, Los Angeles, United States of America
| | - Liat Shenhav
- Department of Computer Science, University of California Los Angeles, Los Angeles, United States of America
| | - Tianli Liu
- Institute of Population Research, Peking University, Beijing, China
| | - Mingming Liu
- Institute of Psychology, Chinese Academy of Sciences, Beijing, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Bin Hu
- School of Information Science & Engineering, Lanzhou University, Lanzhou, China
| | - Tingshao Zhu
- Institute of Psychology, Chinese Academy of Sciences, Beijing, China
- * E-mail:
| |
Collapse
|
34
|
König A, Linz N, Zeghari R, Klinge X, Tröger J, Alexandersson J, Robert P. Detecting Apathy in Older Adults with Cognitive Disorders Using Automatic Speech Analysis. J Alzheimers Dis 2019; 69:1183-1193. [DOI: 10.3233/jad-181033] [Citation(s) in RCA: 26] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/14/2023]
Affiliation(s)
- Alexandra König
- CoBTeK (Cognition-Behaviour-Technology) Lab, Memory Center CHU, Université Côte d’Azur, Nice, France
- INRIA Stars Team, Sophia Antipolis, Valbonne, France
| | - Nicklas Linz
- German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany
| | - Radia Zeghari
- CoBTeK (Cognition-Behaviour-Technology) Lab, Memory Center CHU, Université Côte d’Azur, Nice, France
| | - Xenia Klinge
- German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany
| | - Johannes Tröger
- German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany
| | - Jan Alexandersson
- German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany
| | - Philippe Robert
- CoBTeK (Cognition-Behaviour-Technology) Lab, Memory Center CHU, Université Côte d’Azur, Nice, France
| |
Collapse
|
35
|
Rana R, Latif S, Gururajan R, Gray A, Mackenzie G, Humphris G, Dunn J. Automated screening for distress: A perspective for the future. Eur J Cancer Care (Engl) 2019; 28:e13033. [DOI: 10.1111/ecc.13033] [Citation(s) in RCA: 21] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2019] [Revised: 02/05/2019] [Accepted: 02/18/2019] [Indexed: 01/13/2023]
Affiliation(s)
- Rajib Rana
- University of Southern Queensland Springfield Queensland Australia
| | - Siddique Latif
- University of Southern Queensland Springfield Queensland Australia
| | - Raj Gururajan
- University of Southern Queensland Springfield Queensland Australia
| | - Anthony Gray
- University of Southern Queensland Springfield Queensland Australia
| | | | | | - Jeff Dunn
- University of Southern Queensland Springfield Queensland Australia
- Griffith University Brisbane Queensland Australia
- University of Technology Sydney Sydney New South Wales Australia
| |
Collapse
|
36
|
Abstract
Depression is a common and serious medical illness which affects the way how we think, feel and act. Although harmless in its initial stages, it can cause serious problems if detected at a later stage. Due to advancements in technology, it is now possible to detect signs of depression. Different implementation of machine learning algorithms has been worked upon to detect factors causing depression. It is found that speech of a person is dramatically affected and various vocal features are used to classify depression.
Collapse
|
37
|
Gillespie S, Laures-Gore J, Moore E, Farina M, Russell S, Haaland B. Identification of Affective State Change in Adults With Aphasia Using Speech Acoustics. JOURNAL OF SPEECH, LANGUAGE, AND HEARING RESEARCH : JSLHR 2018; 61:2906-2916. [PMID: 30481797 PMCID: PMC6440307 DOI: 10.1044/2018_jslhr-s-17-0057] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/10/2017] [Revised: 05/10/2017] [Accepted: 06/14/2018] [Indexed: 06/09/2023]
Abstract
Purpose The current study aimed to identify objective acoustic measures related to affective state change in the speech of adults with post-stroke aphasia. Method The speech of 20 post-stroke adults with aphasia was recorded during picture description and administration of the Western Aphasia Battery-Revised (Kertesz, 2006). In addition, participants completed the Self-Assessment Manikin (Bradley & Lang, 1994) and the Stress Scale (Tobii Dynavox, 1981-2016) before and after the language tasks. Speech from each participant was used to detect a change in affective state test scores between the beginning and ending speech. Results Machine learning revealed moderate success in classifying depression, minimal success in predicting depression and stress numeric scores, and minimal success in classifying changes in affective state class between the beginning and ending speech. Conclusions The results suggest the existence of objectively measurable aspects of speech that may be used to identify changes in acute affect from adults with aphasia. This work is exploratory and hypothesis-generating; more work will be needed to make conclusive claims. Further work in this area could lead to automated tools to assist clinicians with their diagnoses of stress, depression, and other forms of affect in adults with aphasia.
Collapse
Affiliation(s)
- Stephanie Gillespie
- School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta
| | | | - Elliot Moore
- School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta
| | - Matthew Farina
- Communication Disorders Program, Georgia State University, Atlanta
| | - Scott Russell
- Department of Speech-Language Pathology, Grady Memorial Hospital, Atlanta, GA
| | - Benjamin Haaland
- Department of Population Health Sciences, University of Utah, Salt Lake City
| |
Collapse
|
38
|
Detecting Depression Using an Ensemble Logistic Regression Model Based on Multiple Speech Features. COMPUTATIONAL AND MATHEMATICAL METHODS IN MEDICINE 2018; 2018:6508319. [PMID: 30344616 PMCID: PMC6174772 DOI: 10.1155/2018/6508319] [Citation(s) in RCA: 31] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/14/2018] [Accepted: 08/28/2018] [Indexed: 11/18/2022]
Abstract
Early intervention for depression is very important to ease the disease burden, but current diagnostic methods are still limited. This study investigated automatic depressed speech classification in a sample of 170 native Chinese subjects (85 healthy controls and 85 depressed patients). The classification performances of prosodic, spectral, and glottal speech features were analyzed in recognition of depression. We proposed an ensemble logistic regression model for detecting depression (ELRDD) in speech. The logistic regression, which was superior in recognition of depression, was selected as the base classifier. This ensemble model extracted many speech features from different aspects and ensured diversity of the base classifier. ELRDD provided better classification results than the other compared classifiers. A technique for identifying depression based on ELRDD, ELRDD-E, was here suggested and tested. It offered encouraging outcomes, revealing a high accuracy level of 75.00% for females and 81.82% for males, as well as an advantageous sensitivity/specificity ratio of 79.25%/70.59% for females and 78.13%/85.29% for males.
Collapse
|
39
|
Lind MN, Byrne ML, Wicks G, Smidt AM, Allen NB. The Effortless Assessment of Risk States (EARS) Tool: An Interpersonal Approach to Mobile Sensing. JMIR Ment Health 2018; 5:e10334. [PMID: 30154072 PMCID: PMC6134227 DOI: 10.2196/10334] [Citation(s) in RCA: 41] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/07/2018] [Revised: 07/12/2018] [Accepted: 07/30/2018] [Indexed: 01/08/2023] Open
Abstract
BACKGROUND To predict and prevent mental health crises, we must develop new approaches that can provide a dramatic advance in the effectiveness, timeliness, and scalability of our interventions. However, current methods of predicting mental health crises (eg, clinical monitoring, screening) usually fail on most, if not all, of these criteria. Luckily for us, 77% of Americans carry with them an unprecedented opportunity to detect risk states and provide precise life-saving interventions. Smartphones present an opportunity to empower individuals to leverage the data they generate through their normal phone use to predict and prevent mental health crises. OBJECTIVE To facilitate the collection of high-quality, passive mobile sensing data, we built the Effortless Assessment of Risk States (EARS) tool to enable the generation of predictive machine learning algorithms to solve previously intractable problems and identify risk states before they become crises. METHODS The EARS tool captures multiple indices of a person's social and affective behavior via their naturalistic use of a smartphone. Although other mobile data collection tools exist, the EARS tool places a unique emphasis on capturing the content as well as the form of social communication on the phone. Signals collected include facial expressions, acoustic vocal quality, natural language use, physical activity, music choice, and geographical location. Critically, the EARS tool collects these data passively, with almost no burden on the user. We programmed the EARS tool in Java for the Android mobile platform. In building the EARS tool, we concentrated on two main considerations: (1) privacy and encryption and (2) phone use impact. RESULTS In a pilot study (N=24), participants tolerated the EARS tool well, reporting minimal burden. None of the participants who completed the study reported needing to use the provided battery packs. Current testing on a range of phones indicated that the tool consumed approximately 15% of the battery over a 16-hour period. Installation of the EARS tool caused minimal change in the user interface and user experience. Once installation is completed, the only difference the user notices is the custom keyboard. CONCLUSIONS The EARS tool offers an innovative approach to passive mobile sensing by emphasizing the centrality of a person's social life to their well-being. We built the EARS tool to power cutting-edge research, with the ultimate goal of leveraging individual big data to empower people and enhance mental health.
Collapse
Affiliation(s)
- Monika N Lind
- Center for Digital Mental Health, Department of Psychology, University of Oregon, Eugene, OR, United States
| | - Michelle L Byrne
- Center for Digital Mental Health, Department of Psychology, University of Oregon, Eugene, OR, United States
| | - Geordie Wicks
- Center for Digital Mental Health, Department of Psychology, University of Oregon, Eugene, OR, United States
| | - Alec M Smidt
- Center for Digital Mental Health, Department of Psychology, University of Oregon, Eugene, OR, United States
| | - Nicholas B Allen
- Center for Digital Mental Health, Department of Psychology, University of Oregon, Eugene, OR, United States
| |
Collapse
|
40
|
Pan Z, Gui C, Zhang J, Zhu J, Cui D. Detecting Manic State of Bipolar Disorder Based on Support Vector Machine and Gaussian Mixture Model Using Spontaneous Speech. Psychiatry Investig 2018; 15:695-700. [PMID: 29969852 PMCID: PMC6056700 DOI: 10.30773/pi.2017.12.15] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/06/2017] [Accepted: 12/15/2017] [Indexed: 11/27/2022] Open
Abstract
OBJECTIVE This study was aimed to compare the accuracy of Support Vector Machine (SVM) and Gaussian Mixture Model (GMM) in the detection of manic state of bipolar disorders (BD) of single patients and multiple patients. METHODS 21 hospitalized BD patients (14 females, average age 34.5±15.3) were recruited after admission. Spontaneous speech was collected through a preloaded smartphone. Firstly, speech features [pitch, formants, mel-frequency cepstrum coefficients (MFCC), linear prediction cepstral coefficient (LPCC), gamma-tone frequency cepstral coefficients (GFCC) etc.] were preprocessed and extracted. Then, speech features were selected using the features of between-class variance and within-class variance. The manic state of patients was then detected by SVM and GMM methods. RESULTS LPCC demonstrated the best discrimination efficiency. The accuracy of manic state detection for single patients was much better using SVM method than GMM method. The detection accuracy for multiple patients was higher using GMM method than SVM method. CONCLUSION SVM provided an appropriate tool for detecting manic state for single patients, whereas GMM worked better for multiple patients' manic state detection. Both of them could help doctors and patients for better diagnosis and mood state monitoring in different situations.
Collapse
Affiliation(s)
- Zhongde Pan
- Shanghai Key Laboratory of Psychotic Disorders, Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China.,Shanghai Key Laboratory of Forensic Medicine, Institute of Forensic Science, Ministry of Justice, Shanghai, China
| | - Chao Gui
- School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Jing Zhang
- Jiading District Mental Health Center, Shanghai, China
| | - Jie Zhu
- School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Donghong Cui
- Shanghai Key Laboratory of Psychotic Disorders, Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China.,Brain Science and Technology Research Center, Shanghai Jiao Tong University, Shanghai, China
| |
Collapse
|
41
|
Salekin A, Eberle JW, Glenn JJ, Teachman BA, Stankovic JA. A Weakly Supervised Learning Framework for Detecting Social Anxiety and Depression. PROCEEDINGS OF THE ACM ON INTERACTIVE, MOBILE, WEARABLE AND UBIQUITOUS TECHNOLOGIES 2018; 2:81. [PMID: 31187083 PMCID: PMC6559734 DOI: 10.1145/3214284] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/01/2017] [Accepted: 04/01/2018] [Indexed: 01/04/2023]
Abstract
Although social anxiety and depression are common, they are often underdiagnosed and undertreated, in part due to difficulties identifying and accessing individuals in need of services. Current assessments rely on client self-report and clinician judgment, which are vulnerable to social desirability and other subjective biases. Identifying objective, nonburdensome markers of these mental health problems, such as features of speech, could help advance assessment, prevention, and treatment approaches. Prior research examining speech detection methods has focused on fully supervised learning approaches employing strongly labeled data. However, strong labeling of individuals high in symptoms or state affect in speech audio data is impractical, in part because it is not possible to identify with high confidence which regions of a long speech indicate the person's symptoms or affective state. We propose a weakly supervised learning framework for detecting social anxiety and depression from long audio clips. Specifically, we present a novel feature modeling technique named NN2Vec that identifies and exploits the inherent relationship between speakers' vocal states and symptoms/affective states. Detecting speakers high in social anxiety or depression symptoms using NN2Vec features achieves F-1 scores 17% and 13% higher than those of the best available baselines. In addition, we present a new multiple instance learning adaptation of a BLSTM classifier, named BLSTM-MIL. Our novel framework of using NN2Vec features with the BLSTM-MIL classifier achieves F-1 scores of 90.1% and 85.44% in detecting speakers high in social anxiety and depression symptoms.
Collapse
Affiliation(s)
- Asif Salekin
- Department of Computer Science, University of Virginia, Charlottesville, VA, 22903, USA
| | - Jeremy W Eberle
- Department of Psychology, University of Virginia, Charlottesville, VA, 22903, USA
| | - Jeffrey J Glenn
- Department of Psychology, University of Virginia, Charlottesville, VA, 22903, USA
| | - Bethany A Teachman
- Department of Psychology, University of Virginia, Charlottesville, VA, 22903, USA
| | - John A Stankovic
- Department of Computer Science, University of Virginia, Charlottesville, VA, 22903, USA
| |
Collapse
|
42
|
Shinkawa K, Yamada Y. Word Repetition in Separate Conversations for Detecting Dementia:A Preliminary Evaluation on Data of Regular Monitoring Service. AMIA JOINT SUMMITS ON TRANSLATIONAL SCIENCE PROCEEDINGS. AMIA JOINT SUMMITS ON TRANSLATIONAL SCIENCE 2018; 2017:206-215. [PMID: 29888073 PMCID: PMC5961820] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
For detecting early signs of dementia, monitoring technology has been actively investigated due to the low diagnostic coverage as well as the requirement for early intervention. Although language features have been used for detecting the language dysfunctions resulting from dementia in neuropsychological tests, features that can be extracted by regular conversations remain unexplored. Here, we propose a feature to characterize the atypical repetition of words on different days which is observed in patients with dementia. We tested it on data obtained from a daily monitoring service for eight elderly people, including two who had been diagnosed with dementia. We found that our feature outperformed the existing linguistic features used in previous studies, such as vocabulary richness and repetitiveness, in terms of effect size and AUC score. The results suggest that the use of our proposed feature holds promise for improving detection performance in everyday situations such as regular monitoring.
Collapse
|
43
|
Zhang J, Pan Z, Gui C, Xue T, Lin Y, Zhu J, Cui D. Analysis on speech signal features of manic patients. J Psychiatr Res 2018; 98:59-63. [PMID: 29291581 DOI: 10.1016/j.jpsychires.2017.12.012] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/06/2017] [Revised: 12/15/2017] [Accepted: 12/18/2017] [Indexed: 10/18/2022]
Abstract
Given the lack of effective biological markers for early diagnosis of bipolar mania, and the tendency for voice fluctuation during transition between mood states, this study aimed to investigate the speech features of manic patients to identify a potential set of biomarkers for diagnosis of bipolar mania. 30 manic patients and 30 healthy controls were recruited and their corresponding speech features were collected during natural dialogue using the Automatic Voice Collecting System. Bech-Rafaelsdn Mania Rating Scale (BRMS) and Clinical impression rating scale (CGI) were used to assess illness. The speech features were compared between two groups: mood group (mania vs remission) and bipolar group (manic patients vs healthy individuals). We found that the characteristic speech signals differed between mood groups and bipolar groups. The fourth formant (F4) and Linear Prediction Coefficient (LPC) (P < .05) were significantly differed when patients transmitted from manic to remission state. The first formant (F1), the second formant (F2), and LPC (P < .05) also played key roles in distinguishing between patients and healthy individuals. In addition, there was a significantly correlation between LPC and BRMS, indicating that LPC may play an important role in diagnosis of bipolar mania. In this study we traced speech features of bipolar mania during natural dialogue (conversation), which is an accessible approach in clinic practice. Such specific indicators may respectively serve as promising biomarkers for benefiting the diagnosis and clinical therapeutic evaluation of bipolar mania.
Collapse
Affiliation(s)
- Jing Zhang
- Shanghai Key Laboratory of Psychotic Disorders, Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China; Shanghai Jiading Mental Health Center, Shanghai, China
| | - Zhongde Pan
- Shanghai Key Laboratory of Psychotic Disorders, Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Chao Gui
- Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Ting Xue
- Shanghai Key Laboratory of Psychotic Disorders, Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Yezhe Lin
- Shanghai Key Laboratory of Psychotic Disorders, Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Jie Zhu
- Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai, China.
| | - Donghong Cui
- Shanghai Key Laboratory of Psychotic Disorders, Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China; Brain Science and Technology Research Center, Shanghai Jiao Tong University, China.
| |
Collapse
|
44
|
Asgari M, Shafran I. Improvements to harmonic model for extracting better speech features in clinical applications. COMPUT SPEECH LANG 2018. [DOI: 10.1016/j.csl.2017.08.005] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
45
|
Taguchi T, Tachikawa H, Nemoto K, Suzuki M, Nagano T, Tachibana R, Nishimura M, Arai T. Major depressive disorder discrimination using vocal acoustic features. J Affect Disord 2018; 225:214-220. [PMID: 28841483 DOI: 10.1016/j.jad.2017.08.038] [Citation(s) in RCA: 54] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/15/2017] [Revised: 06/01/2017] [Accepted: 08/14/2017] [Indexed: 11/26/2022]
Abstract
BACKGROUND The voice carries various information produced by vibrations of the vocal cords and the vocal tract. Though many studies have reported a relationship between vocal acoustic features and depression, including mel-frequency cepstrum coefficients (MFCCs) which applied to speech recognition, there have been few studies in which acoustic features allowed discrimination of patients with depressive disorder. Vocal acoustic features as biomarker of depression could make differential diagnosis of patients with depressive state. In order to achieve differential diagnosis of depression, in this preliminary study, we examined whether vocal acoustic features could allow discrimination between depressive patients and healthy controls. METHODS Subjects were 36 patients who met the criteria for major depressive disorder and 36 healthy controls with no current or past psychiatric disorders. Voices of reading out digits before and after verbal fluency task were recorded. Voices were analyzed using OpenSMILE. The extracted acoustic features, including MFCCs, were used for group comparison and discriminant analysis between patients and controls. RESULTS The second dimension of MFCC (MFCC 2) was significantly different between groups and allowed the discrimination between patients and controls with a sensitivity of 77.8% and a specificity of 86.1%. The difference in MFCC 2 between the two groups reflected an energy difference of frequency around 2000-3000Hz. CONCLUSIONS The MFCC 2 was significantly different between depressive patients and controls. This feature could be a useful biomarker to detect major depressive disorder. LIMITATIONS Sample size was relatively small. Psychotropics could have a confounding effect on voice.
Collapse
Affiliation(s)
- Takaya Taguchi
- Department of Psychiatry, Graduate School of Comprehensive Human Sciences, University of Tsukuba, Japan; University of Tsukuba Hospital, Japan
| | - Hirokazu Tachikawa
- Department of Psychiatry, Graduate School of Comprehensive Human Sciences, University of Tsukuba, Japan; Department of Psychiatry, Faculty of Medicine, University of Tsukuba, Japan.
| | - Kiyotaka Nemoto
- Department of Psychiatry, Graduate School of Comprehensive Human Sciences, University of Tsukuba, Japan; Department of Psychiatry, Faculty of Medicine, University of Tsukuba, Japan
| | | | | | | | - Masafumi Nishimura
- Graduate School of Integrated Science and Technology, Shizuoka University, Japan
| | - Tetsuaki Arai
- Department of Psychiatry, Graduate School of Comprehensive Human Sciences, University of Tsukuba, Japan; Department of Psychiatry, Faculty of Medicine, University of Tsukuba, Japan
| |
Collapse
|
46
|
Liu Z, Hu B, Li X, Liu F, Wang G, Yang J. Detecting Depression in Speech Under Different Speaking Styles and Emotional Valences. Brain Inform 2017. [DOI: 10.1007/978-3-319-70772-3_25] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022] Open
|
47
|
Guidi A, Schoentgen J, Bertschy G, Gentili C, Landini L, Scilingo EP, Vanello N. Voice quality in patients suffering from bipolar disease. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2016; 2015:6106-9. [PMID: 26737685 DOI: 10.1109/embc.2015.7319785] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
People suffering from bipolar disease are more and more common. Such pathology can severely affect patients' lifestyle by wide, and sometimes extreme, mood swings. Biosignals can be very useful to understand this disease. Specifically, speech-related features have been seen to vary in depressed people with respect to healthy subjects. Usually prosodic, spectral and energy-related features are studied. Some further information, instead, can be provided studying voice quality. According to Laver's model, voice quality is sensitive and depends on both anatomic/physiologic issues and long-term muscular adjustments of the larynx or the supraglottal vocal tract. A pilot study on both bipolar patients and healthy control subjects, performed by means of the Long-Term Average Spectrum (LTAS) is presented. The effects on LTAS estimation of a F0-correction procedure are discussed. Pairwise statistical comparisons between subjects in euthymic and depressed states and euthymic and hypomanic states were performed. Significant differences were found in some frequency intervals in both cases. The F0-correction procedure modified the values of the significant frequency intervals in the euthymic/depressed comparison, that also was characterized by a change of F0. Noticeably, no statistically significant differences were found in control subjects acquired in the same mood state. Though the number of subjects is small, the results are encouraging given their coherence across patients and the lack of differences in the control group. Finally, this work suggests that particular vocal settings might be involved in different mood states.
Collapse
|
48
|
Chaspari T, Soldatos C, Maragos P. The development of the Athens Emotional States Inventory (AESI): collection, validation and automatic processing of emotionally loaded sentences. World J Biol Psychiatry 2016; 16:312-22. [PMID: 25797829 DOI: 10.3109/15622975.2015.1012228] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/13/2022]
Abstract
OBJECTIVES The development of ecologically valid procedures for collecting reliable and unbiased emotional data towards computer interfaces with social and affective intelligence targeting patients with mental disorders. METHODS Following its development, presented with, the Athens Emotional States Inventory (AESI) proposes the design, recording and validation of an audiovisual database for five emotional states: anger, fear, joy, sadness and neutral. The items of the AESI consist of sentences each having content indicative of the corresponding emotion. Emotional content was assessed through a survey of 40 young participants with a questionnaire following the Latin square design. The emotional sentences that were correctly identified by 85% of the participants were recorded in a soundproof room with microphones and cameras. A preliminary validation of AESI is performed through automatic emotion recognition experiments from speech. RESULTS The resulting database contains 696 recorded utterances in Greek language by 20 native speakers and has a total duration of approximately 28 min. Speech classification results yield accuracy up to 75.15% for automatically recognizing the emotions in AESI. CONCLUSIONS These results indicate the usefulness of our approach for collecting emotional data with reliable content, balanced across classes and with reduced environmental variability.
Collapse
Affiliation(s)
- Theodora Chaspari
- University of Southern California, Ming Hsieh Department of Electrical Engineering , Los Angeles, CA , USA
| | | | | |
Collapse
|
49
|
Guidi A, Salvi S, Ottaviano M, Gentili C, Bertschy G, de Rossi D, Scilingo EP, Vanello N. Smartphone Application for the Analysis of Prosodic Features in Running Speech with a Focus on Bipolar Disorders: System Performance Evaluation and Case Study. SENSORS 2015; 15:28070-87. [PMID: 26561811 PMCID: PMC4701269 DOI: 10.3390/s151128070] [Citation(s) in RCA: 32] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/31/2015] [Revised: 09/26/2015] [Accepted: 10/26/2015] [Indexed: 11/16/2022]
Abstract
Bipolar disorder is one of the most common mood disorders characterized by large and invalidating mood swings. Several projects focus on the development of decision support systems that monitor and advise patients, as well as clinicians. Voice monitoring and speech signal analysis can be exploited to reach this goal. In this study, an Android application was designed for analyzing running speech using a smartphone device. The application can record audio samples and estimate speech fundamental frequency, F0, and its changes. F0-related features are estimated locally on the smartphone, with some advantages with respect to remote processing approaches in terms of privacy protection and reduced upload costs. The raw features can be sent to a central server and further processed. The quality of the audio recordings, algorithm reliability and performance of the overall system were evaluated in terms of voiced segment detection and features estimation. The results demonstrate that mean F0 from each voiced segment can be reliably estimated, thus describing prosodic features across the speech sample. Instead, features related to F0 variability within each voiced segment performed poorly. A case study performed on a bipolar patient is presented.
Collapse
Affiliation(s)
- Andrea Guidi
- Dipartimento di Ingegneria dell'Informazione, University of Pisa, Via G. Caruso 16, Pisa 56122, Italy.
- Research Center "E. Piaggio", University of Pisa, Largo L. Lazzarino 1, Pisa 56122, Italy.
| | - Sergio Salvi
- Life Supporting Technologies, Universidad Politécnica de Madrid , Avd. Complutense 30, Madrid 28040, Spain.
| | - Manuel Ottaviano
- Life Supporting Technologies, Universidad Politécnica de Madrid , Avd. Complutense 30, Madrid 28040, Spain.
| | - Claudio Gentili
- Department of Surgical, Medical, Molecular Pathology and Critical Care, University of Pisa, Via Savi 10, Pisa 56126, Italy.
- Department of General Psychology, University of Padua, Via Venezia 8, Padua 35131, Italy.
| | - Gilles Bertschy
- Department of Psychiatry and Mental Health, Strasbourg University Hospital, INSERM U1114, Translational Medicine Federation, University of Strasbourg, Strasbourg 67000, France.
| | - Danilo de Rossi
- Dipartimento di Ingegneria dell'Informazione, University of Pisa, Via G. Caruso 16, Pisa 56122, Italy.
- Research Center "E. Piaggio", University of Pisa, Largo L. Lazzarino 1, Pisa 56122, Italy.
| | - Enzo Pasquale Scilingo
- Dipartimento di Ingegneria dell'Informazione, University of Pisa, Via G. Caruso 16, Pisa 56122, Italy.
- Research Center "E. Piaggio", University of Pisa, Largo L. Lazzarino 1, Pisa 56122, Italy.
| | - Nicola Vanello
- Dipartimento di Ingegneria dell'Informazione, University of Pisa, Via G. Caruso 16, Pisa 56122, Italy.
- Research Center "E. Piaggio", University of Pisa, Largo L. Lazzarino 1, Pisa 56122, Italy.
| |
Collapse
|
50
|
Abstract
Objective: The aim of this study was to explore the life stories of depressive adolescents and compare them with non-clinical adolescents' life stories. Methods: For this purpose, we compared 20 life stories of hospitalized adolescents suffering from major depressive episode with 40 life stories of adolescents attending school divided into two groups: 20 non-depressed and 20 depressed adolescents. Results: Results showed that life stories differed as a function of psychopathology. Depressed hospitalized adolescents spoke about their disease and defined themselves by their depression. The depressed adolescents in school concentrated on schooling and school achievements, while the non-depressed group defined themselves by their family, friends and inclusion in a peer group. Conclusion: These analyses allowed us to highlight specific themes mentioned by each of the three groups of adolescents. Although life stories are personal and unique, analysis of such stories allows us to better understand the daily reality of depressive adolescents and the relationships between the life events they experience, daily stressors, depression and how they construct their personal history.
Collapse
Affiliation(s)
- Aurore Boulard
- Département de psychologies et cliniques des systèmes humains, University of Liège, Belgium
| |
Collapse
|