1
|
Pereira R, Mendes C, Ribeiro J, Ribeiro R, Miragaia R, Rodrigues N, Costa N, Pereira A. Systematic Review of Emotion Detection with Computer Vision and Deep Learning. SENSORS (BASEL, SWITZERLAND) 2024; 24:3484. [PMID: 38894274 PMCID: PMC11175284 DOI: 10.3390/s24113484] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/29/2024] [Revised: 05/20/2024] [Accepted: 05/24/2024] [Indexed: 06/21/2024]
Abstract
Emotion recognition has become increasingly important in the field of Deep Learning (DL) and computer vision due to its broad applicability by using human-computer interaction (HCI) in areas such as psychology, healthcare, and entertainment. In this paper, we conduct a systematic review of facial and pose emotion recognition using DL and computer vision, analyzing and evaluating 77 papers from different sources under Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Our review covers several topics, including the scope and purpose of the studies, the methods employed, and the used datasets. The scope of this work is to conduct a systematic review of facial and pose emotion recognition using DL methods and computer vision. The studies were categorized based on a proposed taxonomy that describes the type of expressions used for emotion detection, the testing environment, the currently relevant DL methods, and the datasets used. The taxonomy of methods in our review includes Convolutional Neural Network (CNN), Faster Region-based Convolutional Neural Network (R-CNN), Vision Transformer (ViT), and "Other NNs", which are the most commonly used models in the analyzed studies, indicating their trendiness in the field. Hybrid and augmented models are not explicitly categorized within this taxonomy, but they are still important to the field. This review offers an understanding of state-of-the-art computer vision algorithms and datasets for emotion recognition through facial expressions and body poses, allowing researchers to understand its fundamental components and trends.
Collapse
Affiliation(s)
- Rafael Pereira
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - Carla Mendes
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - José Ribeiro
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - Roberto Ribeiro
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - Rolando Miragaia
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - Nuno Rodrigues
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - Nuno Costa
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
| | - António Pereira
- Computer Science and Communications Research Centre, School of Technology and Management, Polytechnic of Leiria, 2411-901 Leiria, Portugal; (R.P.); (C.M.); (J.R.); (R.R.); (R.M.); (N.R.); (N.C.)
- INOV INESC Inovação, Institute of New Technologies, Leiria Office, 2411-901 Leiria, Portugal
| |
Collapse
|
2
|
Yin R, Dou Z, Wang Y, Zhang Q, Guo Y, Wang Y, Chen Y, Zhang C, Li H, Jian X, Qi L, Ma W. Preoperative CECT-Based Multitask Model Predicts Peritoneal Recurrence and Disease-Free Survival in Advanced Ovarian Cancer: A Multicenter Study. Acad Radiol 2024:S1076-6332(24)00231-9. [PMID: 38693025 DOI: 10.1016/j.acra.2024.04.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2024] [Revised: 04/13/2024] [Accepted: 04/14/2024] [Indexed: 05/03/2024]
Abstract
RATIONALE AND OBJECTIVES Peritoneal recurrence is the predominant pattern of recurrence in advanced ovarian cancer (AOC) and portends a dismal prognosis. Accurate prediction of peritoneal recurrence and disease-free survival (DFS) is crucial to identify patients who might benefit from intensive treatment. We aimed to develop a predictive model for peritoneal recurrence and prognosis in AOC. METHODS In this retrospective multi-institution study of 515 patients, an end-to-end multi-task convolutional neural network (MCNN) comprising a segmentation convolutional neural network (CNN) and a classification CNN was developed and tested using preoperative CT images, and MCNN-score was generated to indicate the peritoneal recurrence and DFS status in patients with AOC. We evaluated the accuracy of the model for automatic segmentation and predict prognosis. RESULTS The MCNN achieved promising segmentation performances with a mean Dice coefficient of 84.3% (range: 78.8%-87.0%). The MCNN was able to predict peritoneal recurrence in the training (AUC 0.87; 95% CI 0.82-0.90), internal test (0.88; 0.85-0.92), and external test set (0.82; 0.78-0.86). Similarly, MCNN demonstrated consistently high accuracy in predicting recurrence, with an AUC of 0.85; 95% CI 0.82-0.88, 0.83; 95% CI 0.80-0.86, and 0.85; 95% CI 0.83-0.88. For patients with a high MCNN-score of recurrence, it was associated with poorer DFS with P < 0.0001 and hazard ratios of 0.1964 (95% CI: 0.1439-0.2680), 0.3249 (95% CI: 0.1896-0.5565), and 0.3458 (95% CI: 0.2582-0.4632). CONCLUSION The MCNN approach demonstrated high performance in predicting peritoneal recurrence and DFS in patients with AOC.
Collapse
Affiliation(s)
- Rui Yin
- National Clinical Research Center for Cancer, Key Laboratory of Cancer Prevention and Therapy, Tianjin's Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin 300060, China; School of Biomedical Engineering & Technology, Tianjin Medical University, Tianjin 300203, China
| | - Zhaoxiang Dou
- Department of Breast Imaging, National Clinical Research Center for Cancer, Key Laboratory of Cancer Prevention and Therapy, Tianjin's Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin 300060, China
| | - Yanyan Wang
- Department of CT and MRI, Shanxi Tumor Hospital, Taiyuan 030013, China
| | - Qian Zhang
- Department of Radiology, Baoding No. 1 Central Hospital, Baoding 071030, China
| | - Yijun Guo
- Department of Breast Imaging, National Clinical Research Center for Cancer, Key Laboratory of Cancer Prevention and Therapy, Tianjin's Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin 300060, China
| | - Yigeng Wang
- Department of Radiology, National Clinical Research Center for Cancer, Key Laboratory of Cancer Prevention and Therapy, Tianjin's Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin 300060, China
| | - Ying Chen
- Department of Gynecologic Oncology, National Clinical Research Center for Cancer, Key Laboratory of Cancer Prevention and Therapy, Tianjin's Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin 300060, China
| | - Chao Zhang
- Department of Bone Cancer, National Clinical Research Center for Cancer, Key Laboratory of Cancer Prevention and Therapy, Tianjin's Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin 300060, China
| | - Huiyang Li
- Department of Gynecology and Obstetrics, Tianjin Medical University General Hospital, Tianjin 300052, China
| | - Xiqi Jian
- School of Biomedical Engineering & Technology, Tianjin Medical University, Tianjin 300203, China
| | - Lisha Qi
- Department of Pathology, National Clinical Research Center for Cancer, Key Laboratory of Cancer Prevention and Therapy, Tianjin's Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin 300060, China
| | - Wenjuan Ma
- Department of Breast Imaging, National Clinical Research Center for Cancer, Key Laboratory of Cancer Prevention and Therapy, Tianjin's Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin 300060, China.
| |
Collapse
|
3
|
Improving Facial Emotion Recognition Using Residual Autoencoder Coupled Affinity Based Overlapping Reduction. MATHEMATICS 2022. [DOI: 10.3390/math10030406] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
Emotion recognition using facial images has been a challenging task in computer vision. Recent advancements in deep learning has helped in achieving better results. Studies have pointed out that multiple facial expressions may present in facial images of a particular type of emotion. Thus, facial images of a category of emotion may have similarity to other categories of facial images, leading towards overlapping of classes in feature space. The problem of class overlapping has been studied primarily in the context of imbalanced classes. Few studies have considered imbalanced facial emotion recognition. However, to the authors’ best knowledge, no study has been found on the effects of overlapped classes on emotion recognition. Motivated by this, in the current study, an affinity-based overlap reduction technique (AFORET) has been proposed to deal with the overlapped class problem in facial emotion recognition. Firstly, a residual variational autoencoder (RVA) model has been used to transform the facial images to a latent vector form. Next, the proposed AFORET method has been applied on these overlapped latent vectors to reduce the overlapping between classes. The proposed method has been validated by training and testing various well known classifiers and comparing their performance in terms of a well known set of performance indicators. In addition, the proposed AFORET method is compared with already existing overlap reduction techniques, such as the OSM, ν-SVM, and NBU methods. Experimental results have shown that the proposed AFORET algorithm, when used with the RVA model, boosts classifier performance to a greater extent in predicting human emotion using facial images.
Collapse
|