Automatic depression recognition using CNN with attention mechanism from videos

Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

Depression is a major psychological disorder with a growing impact worldwide. Traditional methods for detecting the risk of depression, predominantly reliant on psychiatric evaluations and self-assessment questionnaires, are often criticized for their inefficiency and lack of objectivity. Advancements in deep learning have paved the way for innovations in depression risk detection methods that fuse multimodal data. This paper introduces a novel framework, the Audio, Video, and Text Fusion-Three Branch Network (AVTF-TBN), designed to amalgamate auditory, visual, and textual cues for a comprehensive analysis of depression risk. Our approach encompasses three dedicated branches-Audio Branch, Video Branch, and Text Branch-each responsible for extracting salient features from the corresponding modality. These features are subsequently fused through a multimodal fusion (MMF) module, yielding a robust feature vector that feeds into a predictive modeling layer. To further our research, we devised an emotion elicitation paradigm based on two distinct tasks-reading and interviewing-implemented to gather a rich, sensor-based depression risk detection dataset. The sensory equipment, such as cameras, captures subtle facial expressions and vocal characteristics essential for our analysis. The research thoroughly investigates the data generated by varying emotional stimuli and evaluates the contribution of different tasks to emotion evocation. During the experiment, the AVTF-TBN model has the best performance when the data from the two tasks are simultaneously used for detection, where the F1 Score is 0.78, Precision is 0.76, and Recall is 0.81. Our experimental results confirm the validity of the paradigm and demonstrate the efficacy of the AVTF-TBN model in detecting depression risk, showcasing the crucial role of sensor-based data in mental health detection.

Convolutional neural network with parallel convolution scale attention module and ResCBAM for breast histology image classification

Breast cancer is the most common cause of female morbidity and death worldwide. Compared with other cancers, early detection of breast cancer is more helpful to improve the prognosis of patients. In order to achieve early diagnosis and treatment, clinical treatment requires rapid and accurate diagnosis. Therefore, the development of an automatic detection system for breast cancer suitable for patient imaging is of great significance for assisting clinical treatment. Accurate classification of pathological images plays a key role in computer-aided medical diagnosis and prognosis. However, in the automatic recognition and classification methods of breast cancer pathological images, the scale information, the loss of image information caused by insufficient feature fusion, and the enormous structure of the model may lead to inaccurate or inefficient classification. To minimize the impact, we proposed a lightweight PCSAM-ResCBAM model based on two-stage convolutional neural network. The model included a Parallel Convolution Scale Attention Module network (PCSAM-Net) and a Residual Convolutional Block Attention Module network (ResCBAM-Net). The first-level convolutional network was built through a 4-layer PCSAM module to achieve prediction and classification of patches extracted from images. To optimize the network's ability to represent global features of images, we proposed a tiled feature fusion method to fuse patch features from the same image, and proposed a residual convolutional attention module. Based on the above, the second-level convolutional network was constructed to achieve predictive classification of images. We evaluated the performance of our proposed model on the ICIAR2018 dataset and the BreakHis dataset, respectively. Furthermore, through model ablation studies, we found that scale attention and dilated convolution play an important role in improving model performance. Our proposed model outperforms the existing state-of-the-art models on 200 × and 400 × magnification datasets with a maximum accuracy of 98.74 %.

Automatic recognition of depression based on audio and video: A review

Depression is a common mental health disorder. With current depression detection methods, specialized physicians often engage in conversations and physiological examinations based on standardized scales as auxiliary measures for depression assessment. Non-biological markers-typically classified as verbal or non-verbal and deemed crucial evaluation criteria for depression-have not been effectively utilized. Specialized physicians usually require extensive training and experience to capture changes in these features. Advancements in deep learning technology have provided technical support for capturing non-biological markers. Several researchers have proposed automatic depression estimation (ADE) systems based on sounds and videos to assist physicians in capturing these features and conducting depression screening. This article summarizes commonly used public datasets and recent research on audio- and video-based ADE based on three perspectives: Datasets, deficiencies in existing research, and future development directions.

SFTNet: A microexpression-based method for depression detection

BACKGROUND AND OBJECTIVES

Depression is a typical mental illness, and early screening can effectively prevent exacerbation of the condition. Many studies have found that the expressions of depressed patients are different from those of other subjects, and microexpressions have been used in the clinical detection of mental illness. However, there are few methods for the automatic detection of depression based on microexpressions.

METHODS

A new dataset of 156 participants (76 in the case group and 80 in the control group) was created. All data were collected in the context of a new emotional stimulation experiment and doctor-patient conversation. We first analyzed the Average Number of Occurrences (ANO) and Average Duration (AD) of facial expressions in the case group and the control group. Then, we proposed a two-stream model SFTNet for identifying depression based on microexpressions, which consists of a single-temporal network (STNet) and a full-temporal network (FTNet). STNet is used to extract features from facial images at a single time node, FTNet is used to extract features from all-time nodes, and the decision network combines the two features to identify depression through decision fusion. The code for SFTNet is available at https://github.com/muzixingyun/SFTNet.

RESULTS

We found that the AD of all subjects was less than 20 frames (2/3 seconds) and that the facial expressions of the control group were richer. SFTNet achieved excellent results on the emotional stimulus experimental dataset, with Accuracy, Precision and Recall of 0.873, 0.888 and 0.846, respectively. We also conducted experiments on the doctor-patient conversation dataset, and the Accuracy, Precision and Recall were 0.829, 0.817 and 0.837, respectively. SFTNet can also be applied to microexpression detection task with more accuracy than SOTA models.

CONCLUSIONS

In the emotional stimulation experiment, the subjects in the case group are more likely to show negative emotions. Compared to SOTA models, our depression detection method is more accurate and can assist doctors in the diagnosis of depression.

Enhancing the efficacy of depression detection system using optimal feature selection from EHR

Diagnosing depression at an early stage is crucial and majorly depends on the clinician's skill. The present work aims to develop an automated tool for assisting the diagnostic procedure of depression using multiple machine-learning techniques. The dataset of sample size 4184 used in this study contains biometric and demographic information of individuals with or without depression, accessed from the University of Nice Sophia-Antipolis. The Artificial Neural Network (ANN), Support Vector Machine (SVM), Random Forest (RF) and Extreme Gradient Boosting (XGBoost) are used for classifying the depressed from the control group. To enhance the computational efficiency, various feature selection algorithms like Recursive Feature Elimination (RFE), Mutual Information (MI) and three bio-inspired techniques, viz. Particle Swarm Optimization (PSO), Genetic Algorithm (GA) and Firefly Algorithms (FA) have been incorporated. To enhance the feature selection process further, majority voting is carried out in all possible combinations of three, four and five feature selection techniques. These feature selection techniques bring down the feature set size significantly to a mean of 33 from the actual size of 61 which is a reduction of 45.90%. The classification accuracy of the enhanced model varies between 84.18% and 88.46%, which is a significant improvement in performance as compared to the pre-existing models (83.76-85.89%). The proposed predictive models outperform the pre-existing classification models without feature selection and thereby enhancing both the performance and efficiency of the diagnostic process.

MHA: a multimodal hierarchical attention model for depression detection in social media

As a serious mental disease, depression causes great harm to the physical and mental health of individuals, and becomes an important cause of suicide. Therefore, it is necessary to accurately identify and treat depressed patients. Compared with traditional clinical diagnosis methods, a large amount of real and different types of data on social media provides new ideas for depression detection research. In this paper, we construct a depression detection data set based on Weibo, and propose a Multimodal Hierarchical Attention (MHA) model for social media depression detection. Multimodal data is fed into the model and the attention mechanism is applied within and between modalities at the same time. Experimental results show that the proposed model achieves the best classification performance. In addition, we propose a distribution normalization method, which can optimize the data distribution and improve the accuracy of depression detection.

A facial depression recognition method based on hybrid multi-head cross attention network

Introduction

Deep-learn methods based on convolutional neural networks (CNNs) have demonstrated impressive performance in depression analysis. Nevertheless, some critical challenges need to be resolved in these methods: (1) It is still difficult for CNNs to learn long-range inductive biases in the low-level feature extraction of different facial regions because of the spatial locality. (2) It is difficult for a model with only a single attention head to concentrate on various parts of the face simultaneously, leading to less sensitivity to other important facial regions associated with depression. In the case of facial depression recognition, many of the clues come from a few areas of the face simultaneously, e.g., the mouth and eyes.

Methods

To address these issues, we present an end-to-end integrated framework called Hybrid Multi-head Cross Attention Network (HMHN), which includes two stages. The first stage consists of the Grid-Wise Attention block (GWA) and Deep Feature Fusion block (DFF) for the low-level visual depression feature learning. In the second stage, we obtain the global representation by encoding high-order interactions among local features with Multi-head Cross Attention block (MAB) and Attention Fusion block (AFB).

Results

We experimented on AVEC2013 and AVEC2014 depression datasets. The results of AVEC 2013 (RMSE = 7.38, MAE = 6.05) and AVEC 2014 (RMSE = 7.60, MAE = 6.01) demonstrated the efficacy of our method and outperformed most of the state-of-the-art video-based depression recognition approaches.

Discussion

We proposed a deep learning hybrid model for depression recognition by capturing the higher-order interactions between the depression features of multiple facial regions, which can effectively reduce the error in depression recognition and gives great potential for clinical experiments.

PRA-Net: Part-and-Relation Attention Network for depression recognition from facial expression

Artificial intelligence methods are widely applied to depression recognition and provide an objective solution. Many effective automated methods for detecting depression use facial expressions, which are strong indicators to reflect psychiatric disorders. However, these methods suffer from insufficient representations of depression. To this end, we propose a novel Part-and-Relation Attention Network (PRA-Net), which can enhance depression representations by accurately focusing on features that are highly correlated with depression. Specifically, we first perform partition on the feature map instead of the original image, in order to obtain part features rich in semantic information. Afterwards, self-attention is used to calculate the weight of each part feature. Following, the relationship between the part feature and the global content representation is explored by relation attention to refine the weight. Finally, all features are aggregated into a more compact and depression-informative representation via both weights for depression score prediction. Extensive experiments demonstrate the superiority of our method. Compared to other end-to-end methods, our method achieves state-of-the-art performance on AVEC2013 and AVEC2014.

A Model of Normality Inspired Deep Learning Framework for Depression Relapse Prediction Using Audiovisual Data

BACKGROUND

Depression (Major Depressive Disorder) is one of the most common mental illnesses. According to the World Health Organization, more than 300 million people in the world are affected. A first depressive episode can be solved by a spontaneous remission within 6 to 12 months. It has been shown that depression affects speech production and facial expressions. Although numerous studies are proposed in the literature for depression recognition using audiovisual cues, depression relapse using audiovisual cues has not been studied in the literature.

METHOD

In this paper, we propose a deep learning-based approach for depression recognition and depression relapse prediction using audiovisual data. For more versatility and reusability, the proposed approach is based on a Model of Normality inspired framework where we define depression relapse by the closeness of the audiovisual patterns of a subject after a symptom-free period to the audiovisual patterns of depressed subjects. A model of Normality is an anomaly detection distance-based approach that computes a distance of normality between the deep audiovisual encoding of a test sample and a learned representation from audiovisual encodings of anomaly-free data.

RESULTS

The proposed approach shows a very promising results with an accuracy of 87.4% and a F1-score of 82.3% for relapse/depression prediction using a Leave-One-Subject-Out training strategy on the DAIC-Woz dataset.

CONCLUSION

The proposed model of normality-based framework is accurate in detecting depression and in predicting depression relapse. A prospective monitoring system is proposed for assisting depressed patients. The proposed framework is easily extensible and others modalities will be integrated in future works.

Can We Quantify Aging-Associated Postural Changes Using Photogrammetry? A Systematic Review

BACKGROUND

Aging is widely known to be associated with changes in standing posture. Recent advancements in the field of computerized image processing have allowed for improved analyses of several health conditions using photographs. However, photogrammetry's potential for assessing aging-associated postural changes is yet unclear. Thus, the aim of this review is to evaluate the potential of photogrammetry in quantifying age-related postural changes.

MATERIALS AND METHODS

We searched the databases PubMed Central, Scopus, Embase, and SciELO from the beginning of records to March 2021. Inclusion criteria were: (a) participants were older adults aged ≥60; (b) standing posture was assessed by photogrammetric means. PRISMA guidelines were followed. We used the Newcastle-Ottawa Scale to assess methodological quality.

RESULTS

Of 946 articles reviewed, after screening and the removal of duplicates, 11 reports were found eligible for full-text assessment, of which 5 full studies met the inclusion criteria. Significant changes occurring with aging included deepening of thoracic kyphosis, flattening of lumbar lordosis, and increased sagittal inclination.

CONCLUSIONS

These changes agree with commonly described aging-related postural changes. However, detailed quantification of these changes was not found; the photogrammetrical methods used were often unvalidated and did not adhere to known protocols. These methodological difficulties call for further studies using validated photogrammetrical methods and improved research methodologies.

Artificial Intelligence-Based Human-Computer Interaction Technology Applied in Consumer Behavior Analysis and Experiential Education

In the course of consumer behavior, it is necessary to study the relationship between the characteristics of psychological activities and the laws of behavior when consumers acquire and use products or services. With the development of the Internet and mobile terminals, electronic commerce (E-commerce) has become an important form of consumption for people. In order to conduct experiential education in E-commerce combined with consumer behavior, courses to understand consumer satisfaction. From the perspective of E-commerce companies, this study proposes to use artificial intelligence (AI) image recognition technology to recognize and analyze consumer facial expressions. First, it analyzes the way of human-computer interaction (HCI) in the context of E-commerce and obtains consumer satisfaction with the product through HCI technology. Then, a deep neural network (DNN) is used to predict the psychological behavior and consumer psychology of consumers to realize personalized product recommendations. In the course education of consumer behavior, it helps to understand consumer satisfaction and make a reasonable design. The experimental results show that consumers are highly satisfied with the products recommended by the system, and the degree of sanctification reaches 93.2%. It is found that the DNN model can learn consumer behavior rules during evaluation, and its prediction effect is increased by 10% compared with the traditional model, which confirms the effectiveness of the recommendation system under the DNN model. This study provides a reference for consumer psychological behavior analysis based on HCI in the context of AI, which is of great significance to help understand consumer satisfaction in consumer behavior education in the context of E-commerce.

"Looking beneath the surface": A visual-physical feature hybrid approach for unattended gauging of construction waste composition

There are various scenarios challenging human experts to judge the interior of something based on limited surface information. Likewise, at waste disposal facilities around the world, human inspectors are often challenged to gauge the composition of waste bulks to determine admissibility and chargeable levy. Manual approaches are laborious, hazardous, and prone to carelessness and fatigue, making unattended gauging of construction waste composition using simple surface information highly desired. This research attempts to contribute to automated waste composition gauging by harnessing a valuable dataset from Hong Kong. Firstly, visual features, called visual inert probability (VIP), characterizing inert and non-inert materials are extracted from 1127 photos of waste bulks using a fine-tuned convolutional neural network (CNN). Then, these visual features together with easy-to-obtain physical features (e.g., weight and depth) are fed to a tailor-made support vector machine (SVM) model to determine waste composition as measured by the proportions of inert and non-inert materials. The visual-physical feature hybrid model achieved a waste composition gauging accuracy of 94% in the experiments. This high performance implies that the model, with proper adaption and integration, could replace human inspectors to smooth the operation of the waste disposal facilities.