1
|
Huang J, Vong CM, Chen CLP, Zhou Y. Accurate and Efficient Large-Scale Multi-Label Learning With Reduced Feature Broad Learning System Using Label Correlation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023; 34:10240-10253. [PMID: 35436203 DOI: 10.1109/tnnls.2022.3165299] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Multi-label learning for large-scale data is a grand challenge because of a large number of labels with a complex data structure. Hence, the existing large-scale multi-label methods either have unsatisfactory classification performance or are extremely time-consuming for training utilizing a massive amount of data. A broad learning system (BLS), a flat network with the advantages of succinct structures, is appropriate for addressing large-scale tasks. However, existing BLS models are not directly applicable for large-scale multi-label learning due to the large and complex label space. In this work, a novel multi-label classifier based on BLS (called BLS-MLL) is proposed with two new mechanisms: kernel-based feature reduction module and correlation-based label thresholding. The kernel-based feature reduction module contains three layers, namely, the feature mapping layer, enhancement nodes layer, and feature reduction layer. The feature mapping layer employs elastic network regularization to solve the randomness of features in order to improve performance. In the enhancement nodes layer, the kernel method is applied for high-dimensional nonlinear conversion to achieve high efficiency. The newly constructed feature reduction layer is used to further significantly improve both the training efficiency and accuracy when facing high-dimensionality with abundant or noisy information embedded in large-scale data. The correlation-based label thresholding enables BLS-MLL to generate a label-thresholding function for effective conversion of the final decision values to logical outputs, thus, improving the classification performance. Finally, experimental comparisons among six state-of-the-art multi-label classifiers on ten datasets demonstrate the effectiveness of the proposed BLS-MLL. The results of the classification performance show that BLS-MLL outperforms the compared algorithms in 86% of cases with better training efficiency in 90% of cases.
Collapse
|
2
|
Borgalli RA, Surve S. Review on learning framework for facial expression recognition. THE IMAGING SCIENCE JOURNAL 2023. [DOI: 10.1080/13682199.2023.2172526] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/17/2023]
Affiliation(s)
- Rohan Appasaheb Borgalli
- Department of Electronics Engineering, Fr. Conceicao Rodrigues College of Engineering, Bandra, University of Mumbai, Mumbai, Maharashtra, India
| | - Sunil Surve
- Department of Computer Engineering, Fr. Conceicao Rodrigues College of Engineering, Bandra, University of Mumbai, Mumbai, Maharashtra, India
| |
Collapse
|
3
|
Shahid AR, Yan H. SqueezExpNet: Dual-stage convolutional neural network for accurate facial expression recognition with attention mechanism. Knowl Based Syst 2023. [DOI: 10.1016/j.knosys.2023.110451] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/28/2023]
|
4
|
Wang J, Geng X. Label Distribution Learning by Exploiting Label Distribution Manifold. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023; 34:839-852. [PMID: 34398766 DOI: 10.1109/tnnls.2021.3103178] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Label correlation is helpful to alleviate the overwhelming output space of label distribution learning (LDL). However, existing studies either only consider one of global and local label correlations or exploit label correlation by some prior knowledge (e.g., low-rank assumption, which may not hold sometimes). To efficiently exploit both global and local label correlations in a data-driven way, we propose in this article a new LDL method called label distribution learning by exploiting label distribution manifold (LDL-LDM). Our basic idea is that the underlying manifold structure of label distribution may encode the correlations among labels. LDL-LDM works as follows. First, to exploit global label correlation, we learn the label distribution manifold and encourage the outputs of our model to lie in the same manifold. Second, we learn the label distribution manifold of different clusters of samples to consider local label correlations. Third, to handle incomplete label distribution learning (incomplete LDL), we jointly learn label distribution and label distribution manifold. Theoretical analysis demonstrates the generalization of our method. Finally, experimental results validate the effectiveness of LDL-LDM in both full and incomplete LDL cases.
Collapse
|
5
|
Yan WJ, Ruan QN, Fu X, Sun YQ. Perceived emotions and AU combinations in ambiguous facial expressions. Pattern Recognit Lett 2022. [DOI: 10.1016/j.patrec.2022.10.018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
6
|
Gong W, Qian Y, Fan Y. MPCSAN: multi-head parallel channel-spatial attention network for facial expression recognition in the wild. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-08040-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
7
|
Feature selection for label distribution learning using dual-similarity based neighborhood fuzzy entropy. Inf Sci (N Y) 2022. [DOI: 10.1016/j.ins.2022.10.054] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
8
|
Wang J, Geng X, Xue H. Re-Weighting Large Margin Label Distribution Learning for Classification. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:5445-5459. [PMID: 34018929 DOI: 10.1109/tpami.2021.3082623] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Label ambiguity has attracted quite some attention among the machine learning community. The latterly proposed Label Distribution Learning (LDL) can handle label ambiguity and has found wide applications in real classification problems. In the training phase, an LDL model is learned first. In the test phase, the top label(s) in the label distribution predicted by the learned LDL model is (are) then regarded as the predicted label(s). That is, LDL considers the whole label distribution in the training phase, but only the top label(s) in the test phase, which likely leads to objective inconsistency. To avoid such inconsistency, we propose a new LDL method Re-Weighting Large Margin Label Distribution Learning (RWLM-LDL). First, we prove that the expected L1-norm loss of LDL bounds the classification error probability, and thus apply L1-norm loss as the learning metric. Second, re-weighting schemes are put forward to alleviate the inconsistency. Third, large margin is introduced to further solve the inconsistency. The theoretical results are presented to showcase the generalization and discrimination of RWLM-LDL. Finally, experimental results show the statistically superior performance of RWLM-LDL against other comparing methods.
Collapse
|
9
|
Battini Sonmez E, Han H, Karadeniz O, Dalyan T, Sarioglu B. EMRES: A New EMotional RESpondent Robot. IEEE Trans Cogn Dev Syst 2022. [DOI: 10.1109/tcds.2021.3120562] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
| | - Hasan Han
- Department of Electrical and Electronics Engineering, Istanbul Bilgi University, Istanbul, Turkey
| | - Oguzcan Karadeniz
- Department of Electrical and Electronics Engineering, Istanbul Bilgi University, Istanbul, Turkey
| | - Tugba Dalyan
- Department of Computer Engineering, Istanbul Bilgi University, Istanbul, Turkey
| | - Baykal Sarioglu
- Department of Electrical and Electronics Engineering, Istanbul Bilgi University, Istanbul, Turkey
| |
Collapse
|
10
|
Owusu E, Appati JK, Okae P. Robust facial expression recognition system in higher poses. Vis Comput Ind Biomed Art 2022; 5:14. [PMID: 35575952 PMCID: PMC9110625 DOI: 10.1186/s42492-022-00109-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2021] [Accepted: 04/19/2022] [Indexed: 11/25/2022] Open
Abstract
Facial expression recognition (FER) has numerous applications in computer security, neuroscience, psychology, and engineering. Owing to its non-intrusiveness, it is considered a useful technology for combating crime. However, FER is plagued with several challenges, the most serious of which is its poor prediction accuracy in severe head poses. The aim of this study, therefore, is to improve the recognition accuracy in severe head poses by proposing a robust 3D head-tracking algorithm based on an ellipsoidal model, advanced ensemble of AdaBoost, and saturated vector machine (SVM). The FER features are tracked from one frame to the next using the ellipsoidal tracking model, and the visible expressive facial key points are extracted using Gabor filters. The ensemble algorithm (Ada-AdaSVM) is then used for feature selection and classification. The proposed technique is evaluated using the Bosphorus, BU-3DFE, MMI, CK + , and BP4D-Spontaneous facial expression databases. The overall performance is outstanding.
Collapse
Affiliation(s)
- Ebenezer Owusu
- Department of Computer Science, University of Ghana, P. O. Box LG 163, Accra, Ghana
| | - Justice Kwame Appati
- Department of Computer Science, University of Ghana, P. O. Box LG 163, Accra, Ghana.
| | - Percy Okae
- Department of Computer Engineering, University of Ghana, P. O. Box LG 77, Accra, Ghana
| |
Collapse
|
11
|
Zhou L, Wang Y, Lei B, Yang W. Regional Self-Attention Convolutional Neural Network for Facial Expression Recognition. INT J PATTERN RECOGN 2022. [DOI: 10.1142/s0218001422560134] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
12
|
Abstract
Summary
We consider Fréchet sufficient dimension reduction with responses being complex random objects in a metric space and high dimension Euclidean predictors. We propose a novel approach, called the weighted inverse regression ensemble method for linear Fréchet sufficient dimension reduction. The method is further generalized as a new operator defined on reproducing kernel Hilbert spaces for nonlinear Fréchet sufficient dimension reduction. We provide theoretical guarantees for the new method via asymptotic analysis. Intensive simulation studies verify the performance of our proposals, and we apply our methods to analyse the handwritten digits data and the real-world affective faces data to demonstrate its use in real applications.
Collapse
Affiliation(s)
- Chao Ying
- Key Laboratory of Advanced Theory and Application in Statistics and Data Science - MOE, School of Statistics, East China Normal University, Shanghai 200241, China
| | - Zhou Yu
- Key Laboratory of Advanced Theory and Application in Statistics and Data Science - MOE, School of Statistics, East China Normal University, Shanghai 200241, China
| |
Collapse
|
13
|
|
14
|
Abiram RN, Vincent PMDR. Identity preserving multi-pose facial expression recognition using fine tuned VGG on the latent space vector of generative adversarial network. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2021; 18:3699-3717. [PMID: 34198408 DOI: 10.3934/mbe.2021186] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Facial expression is the crucial component for human beings to express their mental state and it has become one of the prominent areas of research in computer vision. However, the task becomes challenging when the given facial image is non-frontal. The influence of poses on facial images is alleviated using an encoder of a generative adversarial network capable of learning pose invariant representations. State-of-art results for image generation are achieved using styleGAN architecture. An efficient model is proposed to embed the given image into the latent vector space of styleGAN. The encoder extracts high-level features of the facial image and encodes them into the latent space. Rigorous analysis of semantics hidden in the latent space of styleGAN is performed. Based on the analysis, the facial image is synthesized, and facial expressions are recognized using an expression recognition neural network. The original image is recovered from the features encoded in the latent space. Semantic editing operations like face rotation, style transfer, face aging, image morphing and expression transfer can be performed on the image obtained from the image generated using the features encoded latent space of styleGAN. L2 feature-wise loss is applied to warrant the quality of the rebuilt image. The facial image is then fed into the attribute classifier to extract high-level features, and the features are concatenated to perform facial expression classification. Evaluations are performed on the generated results to demonstrate that state-of-art results are achieved using the proposed method.
Collapse
Affiliation(s)
- R Nandhini Abiram
- School of Information Technology, Vellore Institute of Technology, Vellore, Tamilnadu, India
| | - P M Durai Raj Vincent
- School of Information Technology, Vellore Institute of Technology, Vellore, Tamilnadu, India
| |
Collapse
|
15
|
Jia S, Wang S, Hu C, Webster PJ, Li X. Detection of Genuine and Posed Facial Expressions of Emotion: Databases and Methods. Front Psychol 2021; 11:580287. [PMID: 33519600 PMCID: PMC7844089 DOI: 10.3389/fpsyg.2020.580287] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2020] [Accepted: 12/09/2020] [Indexed: 11/18/2022] Open
Abstract
Facial expressions of emotion play an important role in human social interactions. However, posed expressions of emotion are not always the same as genuine feelings. Recent research has found that facial expressions are increasingly used as a tool for understanding social interactions instead of personal emotions. Therefore, the credibility assessment of facial expressions, namely, the discrimination of genuine (spontaneous) expressions from posed (deliberate/volitional/deceptive) ones, is a crucial yet challenging task in facial expression understanding. With recent advances in computer vision and machine learning techniques, rapid progress has been made in recent years for automatic detection of genuine and posed facial expressions. This paper presents a general review of the relevant research, including several spontaneous vs. posed (SVP) facial expression databases and various computer vision based detection methods. In addition, a variety of factors that will influence the performance of SVP detection methods are discussed along with open issues and technical challenges in this nascent field.
Collapse
Affiliation(s)
- Shan Jia
- State Key Laboratory of Information Engineering in Surveying Mapping and Remote Sensing, Wuhan University, Wuhan, China.,Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, United States
| | - Shuo Wang
- Department of Chemical and Biomedical Engineering, West Virginia University, Morgantown, WV, United States
| | - Chuanbo Hu
- Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, United States
| | - Paula J Webster
- Department of Chemical and Biomedical Engineering, West Virginia University, Morgantown, WV, United States
| | - Xin Li
- Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, United States
| |
Collapse
|
16
|
Masson A, Cazenave G, Trombini J, Batt M. The current challenges of automatic recognition of facial expressions: A systematic review. AI COMMUN 2020. [DOI: 10.3233/aic-200631] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
In recent years, due to its great economic and social potential, the recognition of facial expressions linked to emotions has become one of the most flourishing applications in the field of artificial intelligence, and has been the subject of many developments. However, despite significant progress, this field is still subject to many theoretical debates and technical challenges. It therefore seems important to make a general inventory of the different lines of research and to present a synthesis of recent results in this field. To this end, we have carried out a systematic review of the literature according to the guidelines of the PRISMA method. A search of 13 documentary databases identified a total of 220 references over the period 2014–2019. After a global presentation of the current systems and their performance, we grouped and analyzed the selected articles in the light of the main problems encountered in the field of automated facial expression recognition. The conclusion of this review highlights the strengths, limitations and main directions for future research in this field.
Collapse
Affiliation(s)
- Audrey Masson
- Interpsy – GRC, University of Lorraine, France. E-mails: ,
- Two-I, France. E-mails: ,
| | | | | | - Martine Batt
- Interpsy – GRC, University of Lorraine, France. E-mails: ,
| |
Collapse
|
17
|
Zhang Z, Lai C, Liu H, Li YF. Infrared facial expression recognition via Gaussian-based label distribution learning in the dark illumination environment for human emotion detection. Neurocomputing 2020. [DOI: 10.1016/j.neucom.2020.05.081] [Citation(s) in RCA: 27] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
18
|
Abstract
Facial Expression Recognition (FER), as the primary processing method for non-verbal intentions, is an important and promising field of computer vision and artificial intelligence, and one of the subject areas of symmetry. This survey is a comprehensive and structured overview of recent advances in FER. We first categorise the existing FER methods into two main groups, i.e., conventional approaches and deep learning-based approaches. Methodologically, to highlight the differences and similarities, we propose a general framework of a conventional FER approach and review the possible technologies that can be employed in each component. As for deep learning-based methods, four kinds of neural network-based state-of-the-art FER approaches are presented and analysed. Besides, we introduce seventeen commonly used FER datasets and summarise four FER-related elements of datasets that may influence the choosing and processing of FER approaches. Evaluation methods and metrics are given in the later part to show how to assess FER algorithms, along with subsequent performance comparisons of different FER approaches on the benchmark datasets. At the end of the survey, we present some challenges and opportunities that need to be addressed in future.
Collapse
|