1
|
Aina J, Akinniyi O, Rahman MM, Odero-Marah V, Khalifa F. A Hybrid Learning-Architecture for Mental Disorder Detection Using Emotion Recognition. IEEE ACCESS : PRACTICAL INNOVATIONS, OPEN SOLUTIONS 2024; 12:91410-91425. [PMID: 39054996 PMCID: PMC11270886 DOI: 10.1109/access.2024.3421376] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 07/27/2024]
Abstract
Mental illness has grown to become a prevalent and global health concern that affects individuals across various demographics. Timely detection and accurate diagnosis of mental disorders are crucial for effective treatment and support as late diagnosis could result in suicidal, harmful behaviors and ultimately death. To this end, the present study introduces a novel pipeline for the analysis of facial expressions, leveraging both the AffectNet and 2013 Facial Emotion Recognition (FER) datasets. Consequently, this research goes beyond traditional diagnostic methods by contributing a system capable of generating a comprehensive mental disorder dataset and concurrently predicting mental disorders based on facial emotional cues. Particularly, we introduce a hybrid architecture for mental disorder detection leveraging the state-of-the-art object detection algorithm, YOLOv8 to detect and classify visual cues associated with specific mental disorders. To achieve accurate predictions, an integrated learning architecture based on the fusion of Convolution Neural Networks (CNNs) and Visual Transformer (ViT) models is developed to form an ensemble classifier that predicts the presence of mental illness (e.g., depression, anxiety, and other mental disorder). The overall accuracy is improved to about 81% using the proposed ensemble technique. To ensure transparency and interpretability, we integrate techniques such as Gradient-weighted Class Activation Mapping (Grad-CAM) and saliency maps to highlight the regions in the input image that significantly contribute to the model's predictions thus providing healthcare professionals with a clear understanding of the features influencing the system's decisions thereby enhancing trust and more informed diagnostic process.
Collapse
Affiliation(s)
- Joseph Aina
- Electrical and Computer Engineering Department, School of Engineering, Morgan State University, Baltimore, MD 21251, USA
| | - Oluwatunmise Akinniyi
- Electrical and Computer Engineering Department, School of Engineering, Morgan State University, Baltimore, MD 21251, USA
| | - Md Mahmudur Rahman
- Department of Computer Science, School of Computer, Mathematical and Natural Sciences, Morgan State University, Baltimore, MD 21251, USA
| | - Valerie Odero-Marah
- Center for Urban Health Disparities Research and Innovation, Department of Biology, Morgan State University, Baltimore, MD 21251, USA
| | - Fahmi Khalifa
- Electrical and Computer Engineering Department, School of Engineering, Morgan State University, Baltimore, MD 21251, USA
- Electronics and Communications Engineering Department, Mansoura University, Mansoura 35516, Egypt
| |
Collapse
|
2
|
Zhang Z, Zhang S, Ni D, Wei Z, Yang K, Jin S, Huang G, Liang Z, Zhang L, Li L, Ding H, Zhang Z, Wang J. Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data. SENSORS (BASEL, SWITZERLAND) 2024; 24:3714. [PMID: 38931497 PMCID: PMC11207438 DOI: 10.3390/s24123714] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/19/2024] [Revised: 05/30/2024] [Accepted: 06/06/2024] [Indexed: 06/28/2024]
Abstract
Depression is a major psychological disorder with a growing impact worldwide. Traditional methods for detecting the risk of depression, predominantly reliant on psychiatric evaluations and self-assessment questionnaires, are often criticized for their inefficiency and lack of objectivity. Advancements in deep learning have paved the way for innovations in depression risk detection methods that fuse multimodal data. This paper introduces a novel framework, the Audio, Video, and Text Fusion-Three Branch Network (AVTF-TBN), designed to amalgamate auditory, visual, and textual cues for a comprehensive analysis of depression risk. Our approach encompasses three dedicated branches-Audio Branch, Video Branch, and Text Branch-each responsible for extracting salient features from the corresponding modality. These features are subsequently fused through a multimodal fusion (MMF) module, yielding a robust feature vector that feeds into a predictive modeling layer. To further our research, we devised an emotion elicitation paradigm based on two distinct tasks-reading and interviewing-implemented to gather a rich, sensor-based depression risk detection dataset. The sensory equipment, such as cameras, captures subtle facial expressions and vocal characteristics essential for our analysis. The research thoroughly investigates the data generated by varying emotional stimuli and evaluates the contribution of different tasks to emotion evocation. During the experiment, the AVTF-TBN model has the best performance when the data from the two tasks are simultaneously used for detection, where the F1 Score is 0.78, Precision is 0.76, and Recall is 0.81. Our experimental results confirm the validity of the paradigm and demonstrate the efficacy of the AVTF-TBN model in detecting depression risk, showcasing the crucial role of sensor-based data in mental health detection.
Collapse
Affiliation(s)
- Zhenwei Zhang
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen 518060, China; (Z.Z.); (D.N.); (G.H.); (Z.L.); (L.Z.); (L.L.); (H.D.)
- Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China
| | - Shengming Zhang
- Affiliated Mental Health Center, Southern University of Science and Technology, Shenzhen 518055, China;
| | - Dong Ni
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen 518060, China; (Z.Z.); (D.N.); (G.H.); (Z.L.); (L.Z.); (L.L.); (H.D.)
- Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China
| | - Zhaoguo Wei
- Shenzhen Kangning Hospital, Shenzhen 518020, China; (Z.W.); (K.Y.); (S.J.)
- Shenzhen Mental Health Center, Shenzhen 518020, China
| | - Kongjun Yang
- Shenzhen Kangning Hospital, Shenzhen 518020, China; (Z.W.); (K.Y.); (S.J.)
- Shenzhen Mental Health Center, Shenzhen 518020, China
| | - Shan Jin
- Shenzhen Kangning Hospital, Shenzhen 518020, China; (Z.W.); (K.Y.); (S.J.)
- Shenzhen Mental Health Center, Shenzhen 518020, China
| | - Gan Huang
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen 518060, China; (Z.Z.); (D.N.); (G.H.); (Z.L.); (L.Z.); (L.L.); (H.D.)
- Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China
| | - Zhen Liang
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen 518060, China; (Z.Z.); (D.N.); (G.H.); (Z.L.); (L.Z.); (L.L.); (H.D.)
- Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China
| | - Li Zhang
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen 518060, China; (Z.Z.); (D.N.); (G.H.); (Z.L.); (L.Z.); (L.L.); (H.D.)
- Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China
| | - Linling Li
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen 518060, China; (Z.Z.); (D.N.); (G.H.); (Z.L.); (L.Z.); (L.L.); (H.D.)
- Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China
| | - Huijun Ding
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen 518060, China; (Z.Z.); (D.N.); (G.H.); (Z.L.); (L.Z.); (L.L.); (H.D.)
- Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China
| | - Zhiguo Zhang
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China
- Peng Cheng Laboratory, Shenzhen 518055, China
| | - Jianhong Wang
- Shenzhen Kangning Hospital, Shenzhen 518020, China; (Z.W.); (K.Y.); (S.J.)
- Shenzhen Mental Health Center, Shenzhen 518020, China
| |
Collapse
|
3
|
Yasin S, Othmani A, Raza I, Hussain SA. Machine learning based approaches for clinical and non-clinical depression recognition and depression relapse prediction using audiovisual and EEG modalities: A comprehensive review. Comput Biol Med 2023; 159:106741. [PMID: 37105109 DOI: 10.1016/j.compbiomed.2023.106741] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2022] [Revised: 02/25/2023] [Accepted: 03/02/2023] [Indexed: 03/07/2023]
Abstract
Mental disorders are rapidly increasing each year and have become a major challenge affecting the social and financial well-being of individuals. There is a need for phenotypic characterization of psychiatric disorders with biomarkers to provide a rich signature for Major Depressive Disorder, improving the understanding of the pathophysiological mechanisms underlying these mental disorders. This comprehensive review focuses on depression and relapse detection modalities such as self-questionnaires, audiovisuals, and EEG, highlighting noteworthy publications in the last ten years. The article concentrates on the literature that adopts machine learning by audiovisual and EEG signals. It also outlines preprocessing, feature extraction, and public datasets for depression detection. The review concludes with recommendations that will help improve the reliability of developed models and the determinism of computational intelligence-based systems in psychiatry. To the best of our knowledge, this survey is the first comprehensive review on depression and relapse prediction by self-questionnaires, audiovisual, and EEG-based approaches. The findings of this review will serve as a useful and structured starting point for researchers studying clinical and non-clinical depression recognition and relapse through machine learning-based approaches.
Collapse
Affiliation(s)
- Sana Yasin
- Department of Computer Science, COMSATS University Islamabad, Lahore Campus Lahore, Pakistan; Department of Computer Science, University of Okara, Okara, Pakistan.
| | - Alice Othmani
- Université Paris-Est Créteil (UPEC), LISSI, Vitry sur Seine, 94400, France.
| | - Imran Raza
- Department of Computer Science, COMSATS University Islamabad, Lahore Campus Lahore, Pakistan.
| | - Syed Asad Hussain
- Department of Computer Science, COMSATS University Islamabad, Lahore Campus Lahore, Pakistan.
| |
Collapse
|
4
|
Francese R, Attanasio P. Emotion detection for supporting depression screening. MULTIMEDIA TOOLS AND APPLICATIONS 2022; 82:12771-12795. [PMID: 36570729 PMCID: PMC9761032 DOI: 10.1007/s11042-022-14290-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/22/2021] [Revised: 10/14/2022] [Accepted: 12/03/2022] [Indexed: 06/17/2023]
Abstract
Depression is the most prevalent mental disorder in the world. One of the most adopted tools for depression screening is the Beck Depression Inventory-II (BDI-II) questionnaire. Patients may minimize or exaggerate their answers. Thus, to further examine the patient's mood while filling in the questionnaire, we propose a mobile application that captures the BDI-II patient's responses together with their images and speech. Deep learning techniques such as Convolutional Neural Networks analyze the patient's audio and image data. The application displays the correlation between the patient's emotional scores and DBI-II scores to the clinician at the end of the questionnaire, indicating the relationship between the patient's emotional state and the depression screening score. We conducted a preliminary evaluation involving clinicians and patients to assess (i) the acceptability of proposed application for use in clinics and (ii) the patient user experience. The participants were eight clinicians who tried the tool with 21 of their patients. The results seem to confirm the acceptability of the app in clinical practice.
Collapse
Affiliation(s)
- Rita Francese
- Computer Science Department, Università degli Studi di Salerno, Via Giovanni Paolo II, 132, Fisciano, 84084 (SA) Italy
| | | |
Collapse
|
5
|
Gupta S, Goel L, Singh A, Prasad A, Ullah MA. Psychological Analysis for Depression Detection from Social Networking Sites. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:4395358. [PMID: 35432513 PMCID: PMC9007657 DOI: 10.1155/2022/4395358] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Revised: 02/28/2022] [Accepted: 03/24/2022] [Indexed: 11/23/2022]
Abstract
Rapid technological advancements are altering people's communication styles. With the growth of the Internet, social networks (Twitter, Facebook, Telegram, and Instagram) have become popular forums for people to share their thoughts, psychological behavior, and emotions. Psychological analysis analyzes text and extracts facts, features, and important information from the opinions of users. Researchers working on psychological analysis rely on social networks for the detection of depression-related behavior and activity. Social networks provide innumerable data on mindsets of a person's onset of depression, such as low sociology and activities such as undergoing medical treatment, a primary emphasis on oneself, and a high rate of activity during the day and night. In this paper, we used five machine learning classifiers-decision trees, K-nearest neighbor, support vector machines, logistic regression, and LSTM-for depression detection in tweets. The dataset is collected in two forms-balanced and imbalanced-where the oversampling of techniques is studied technically. The results show that the LSTM classification model outperforms the other baseline models in the depression detection healthcare approach for both balanced and imbalanced data.
Collapse
Affiliation(s)
- Sonam Gupta
- Department of Computer Science and Engineering, Ajay Kumar Garg Engineering College, Ghaziabad, India
| | - Lipika Goel
- Gokaraju Rangaraju Institute of Engineering and Technology, Hyderabad, India
| | - Arjun Singh
- School of Computing and Information Technology, Manipal University Jaipur, Jaipur, India
| | - Ajay Prasad
- University of Petroleum and Energy Studies, Dehradun, India
| | - Mohammad Aman Ullah
- Department of Computer Science and Engineering, International Islamic University Chittagong, Chittagong, Bangladesh
| |
Collapse
|
6
|
2-level hierarchical depression recognition method based on task-stimulated and integrated speech features. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
7
|
Shen J, Zhang S, Tong Y, Dong X, Wang X, Fu G, Zhao L, Wu M, Yin Y, Wang Y, Liu NH, Wu J, Li J. Establishment and psychometric characteristics of emotional words list for suicidal risk assessment in speech emotion recognition. Front Psychiatry 2022; 13:1022036. [PMID: 36440401 PMCID: PMC9691664 DOI: 10.3389/fpsyt.2022.1022036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/18/2022] [Accepted: 10/27/2022] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND Emotional disturbance is an important risk factor of suicidal behaviors. To ensure speech emotion recognition (SER), a novel technique to evaluate emotional characteristics of speech, precision in labeling emotional words is a prerequisite. Currently, a list of suicide-related emotional word is absent. The aims of this study were to establish an Emotional Words List for Suicidal Risk Assessment (EWLSRA) and test the reliability and validity of the list in a suicide-related SER task. METHODS Suicide-related emotion words were nominated and discussed by 10 suicide prevention professionals. Sixty-five tape-recordings of calls to a large psychological support hotline in China were selected to test psychometric characteristics of the EWLSRA. RESULTS The results shows that the EWLSRA consists of 11 emotion words which were highly associated with suicide risk scores and suicide attempts. Results of exploratory factor analysis support one-factor model of this list. The Fleiss' Kappa value of 0.42 indicated good inter-rater reliability of the list. In terms of criteria validities, indices of despair (Spearman ρ = 0.54, P < 0.001), sadness (ρ = 0.37, P = 0.006), helplessness (ρ = 0.45, P = 0.001), and numbness (ρ = 0.35, P = 0.009) were significantly associated with suicidal risk scores. The index of the emotional word of numbness in callers with suicide attempt during the 12-month follow-up was significantly higher than that in callers without suicide attempt during the follow-up (P = 0.049). CONCLUSION This study demonstrated that the EWLSRA has adequate psychometric performance in identifying suicide-related emotional words of recording of hotline callers to a national wide suicide prevention line. This list can be useful for SER in future studies on suicide prevention.
Collapse
Affiliation(s)
- Juan Shen
- Beijing Suicide Research and Prevention Center, Beijing Huilongguan Hospital, Beijing, China.,WHO Collaborating Center for Research and Training in Suicide Prevention, Beijing, China
| | - Shuo Zhang
- Faculty of Information Technology, Beijing University of Technology, Beijing, China
| | - Yongsheng Tong
- Beijing Suicide Research and Prevention Center, Beijing Huilongguan Hospital, Beijing, China.,WHO Collaborating Center for Research and Training in Suicide Prevention, Beijing, China.,Peking University Huilongguan Clinical Medical School, Beijing, China
| | - Xiangmin Dong
- Faculty of Information Technology, Beijing University of Technology, Beijing, China
| | - Xuelian Wang
- Beijing Suicide Research and Prevention Center, Beijing Huilongguan Hospital, Beijing, China.,WHO Collaborating Center for Research and Training in Suicide Prevention, Beijing, China.,Peking University Huilongguan Clinical Medical School, Beijing, China
| | - Guanghui Fu
- Sorbonne Université, Institut du Cerveau-Paris Brain Institute-ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié-Salpêtrière, Paris, France
| | - Liting Zhao
- Beijing Suicide Research and Prevention Center, Beijing Huilongguan Hospital, Beijing, China.,WHO Collaborating Center for Research and Training in Suicide Prevention, Beijing, China
| | - Mengjie Wu
- Beijing Suicide Research and Prevention Center, Beijing Huilongguan Hospital, Beijing, China.,WHO Collaborating Center for Research and Training in Suicide Prevention, Beijing, China.,Peking University Huilongguan Clinical Medical School, Beijing, China
| | - Yi Yin
- Beijing Suicide Research and Prevention Center, Beijing Huilongguan Hospital, Beijing, China.,WHO Collaborating Center for Research and Training in Suicide Prevention, Beijing, China.,Peking University Huilongguan Clinical Medical School, Beijing, China
| | - Yuehua Wang
- Beijing Suicide Research and Prevention Center, Beijing Huilongguan Hospital, Beijing, China.,WHO Collaborating Center for Research and Training in Suicide Prevention, Beijing, China
| | - Nancy H Liu
- Department of Psychology, University of California, Berkeley, Berkeley, CA, United States
| | - Jianlan Wu
- Beijing Suicide Research and Prevention Center, Beijing Huilongguan Hospital, Beijing, China.,WHO Collaborating Center for Research and Training in Suicide Prevention, Beijing, China.,Peking University Huilongguan Clinical Medical School, Beijing, China
| | - Jianqiang Li
- Faculty of Information Technology, Beijing University of Technology, Beijing, China
| |
Collapse
|