Personalized Age Progression with Bi-Level Aging Dictionary Learning

RAgE: Robust Age Estimation Through Subject Anchoring With Consistency Regularisation

Modern facial age estimation systems can achieve high accuracy when training and test datasets are identically distributed and captured under similar conditions. However, domain shifts in data, encountered in practice, lead to a sharp drop in accuracy of most existing age estimation algorithms. In this article, we propose a novel method, namely RAgE, to improve the robustness and reduce the uncertainty of age estimates by leveraging unlabelled data through a subject anchoring strategy and a novel consistency regularisation term. First, we propose an similarity-preserving pseudo-labelling algorithm by which the model generates pseudo-labels for a cohort of unlabelled images belonging to the same subject, while taking into account the similarity among age labels. In order to improve the robustness of the system, a consistency regularisation term is then used to simultaneously encourage the model to produce invariant outputs for the images in the cohort with respect to an anchor image. We propose a novel consistency regularisation term the noise-tolerant property of which effectively mitigates the so-called confirmation bias caused by incorrect pseudo-labels. Experiments on multiple benchmark ageing datasets demonstrate substantial improvements over the state-of-the-art methods and robustness to confounding external factors, including subject's head pose, illumination variation and appearance of expression in the face image.

Toward Fine-Grained Talking Face Generation

Talking face generation is the process of synthesizing a lip-synchronized video when given a reference portrait and an audio clip. However, generating a fine-grained talking video is nontrivial due to several challenges: 1) capturing vivid facial expressions, such as muscle movements; 2) ensuring smooth transitions between consecutive frames; and 3) preserving the details of the reference portrait. Existing efforts have only focused on modeling rigid lip movements, resulting in low-fidelity videos with jerky facial muscle deformations. To address these challenges, we propose a novel Fine-gRained mOtioN moDel (FROND), consisting of three components. In the first component, we adopt a two-stream encoder to capture local facial movement keypoints and embed their overall motion context as the global code. In the second component, we design a motion estimation module to predict audio-driven movements. This enables the learning of local key point motion in the continuous trajectory space to achieve smooth temporal facial movements. Additionally, the local and global motions are fused to estimate a continuous dense motion field, resulting in spatially smooth movements. In the third component, we devise a novel implicit image decoder based on an implicit neural network. This decoder recovers high-frequency information from the input image, resulting in a high-fidelity talking face. In summary, the FROND refines the motion trajectories of facial keypoints into a continuous dense motion field, which is followed by a decoder that fully exploits the inherent smoothness of the motion. We conduct quantitative and qualitative model evaluations on benchmark datasets. The experimental results show that our proposed FROND significantly outperforms several state-of-the-art baselines.

Deep Order-Preserving Learning With Adaptive Optimal Transport Distance

We consider a framework for taking into consideration the relative importance (ordinality) of object labels in the process of learning a label predictor function. The commonly used loss functions are not well matched to this problem, as they exhibit deficiencies in capturing natural correlations of the labels and the corresponding data. We propose to incorporate such correlations into our learning algorithm using an optimal transport formulation. Our approach is to learn the ground metric, which is partly involved in forming the optimal transport distance, by leveraging ordinality as a general form of side information in its formulation. Based on this idea, we then develop a novel loss function for training deep neural networks. A highly efficient alternating learning method is then devised to alternatively optimise the ground metric and the deep model in an end-to-end learning manner. This scheme allows us to adaptively adjust the shape of the ground metric, and consequently the shape of the loss function for each application. We back up our approach by theoretical analysis and verify the performance of our proposed scheme by applying it to two learning tasks, i.e. chronological age estimation from the face and image aesthetic assessment. The numerical results on several benchmark datasets demonstrate the superiority of the proposed algorithm.

AutoMER: Spatiotemporal Neural Architecture Search for Microexpression Recognition

Facial microexpressions offer useful insights into subtle human emotions. This unpremeditated emotional leakage exhibits the true emotions of a person. However, the minute temporal changes in the video sequences are very difficult to model for accurate classification. In this article, we propose a novel spatiotemporal architecture search algorithm, AutoMER for microexpression recognition (MER). Our main contribution is a new parallelogram design-based search space for efficient architecture search. We introduce a spatiotemporal feature module named 3-D singleton convolution for cell-level analysis. Furthermore, we present four such candidate operators and two 3-D dilated convolution operators to encode the raw video sequences in an end-to-end manner. To the best of our knowledge, this is the first attempt to discover 3-D convolutional neural network (CNN) architectures with a network-level search for MER. The searched models using the proposed AutoMER algorithm are evaluated over five microexpression data sets: CASME-I, SMIC, CASME-II, CAS(ME) ∧2 , and SAMM. The proposed generated models quantitatively outperform the existing state-of-the-art approaches. The AutoMER is further validated with different configurations, such as downsampling rate factor, multiscale singleton 3-D convolution, parallelogram, and multiscale kernels. Overall, five ablation experiments were conducted to analyze the operational insights of the proposed AutoMER.

A Novel Ground Metric for Optimal Transport-Based Chronological Age Estimation

Label distribution learning (LDL) is the state-of-the-art approach to dealing with a number of real-world applications, such as chronological age estimation from a face image, where there is an inherent similarity among adjacent age labels. LDL takes into account the semantic similarity by assigning a label distribution to each instance. The well-known Kullback-Leibler (KL) divergence is the widely used loss function for the LDL framework. However, the KL divergence does not fully and effectively capture the semantic similarity among age labels, thus leading to suboptimal performance. In this article, we propose a novel loss function based on the optimal transport theory for the LDL-based age estimation. A ground metric function plays an important role in the optimal transport formulation. It should be carefully determined based on the underlying geometric structure of the label space of the application in-hand. The label space in the age estimation problem has a specific geometric structure, that is, closer ages have more inherent semantic relationships. Inspired by this, we devise a novel ground metric function, which enables the loss function to increase the influence of highly correlated ages; thus exploiting the semantic similarity among ages more effectively than the existing loss functions. We then use the proposed loss function, namely, γ -Wasserstein loss, for training a deep neural network (DNN). This leads to a notoriously computationally expensive and nonconvex optimization problem. Following the standard methodology, we formulate the optimization function as a convex problem and then use an efficient iterative algorithm to update the parameters of the DNN. Extensive experiments in age estimation on different benchmark datasets validate the effectiveness of the proposed method, which consistently outperforms state-of-the-art approaches.

Deep Multiview Union Learning Network for Multisource Image Classification

With the development of the imaging technology of various sensors, multisource image classification has become a key challenge in the field of image interpretation. In this article, a novel classification method, called the deep multiview union learning network (DMULN), is proposed to classify multisensor data. First, an associated feature extractor is designed to process the multisource data by canonical correlation analysis (CCA) in the head of the network. Second, an improved deep learning architecture with two branches is presented to extract high-level view features from the associated features. Third, a novel pooling, called view union pooling, is proposed to fuse the multiview feature from the deep model. Finally, the fused feature is fed into the classifier. The proposed framework is easy to optimize since it is an end-to-end network. Extensive experiments and analysis on the datasets IEEE_grss_dfc_2017 and IEEE_grss_dfc_2018 show that the proposed method achieves comparable results. Our results demonstrate that abundant multisource information can improve the classification performance.

Understanding the Research Landscape of Deep Learning in Biomedical Science: Scientometric Analysis

BACKGROUND

Advances in biomedical research using deep learning techniques have generated a large volume of related literature. However, there is a lack of scientometric studies that provide a bird's-eye view of them. This absence has led to a partial and fragmented understanding of the field and its progress.

OBJECTIVE

This study aimed to gain a quantitative and qualitative understanding of the scientific domain by analyzing diverse bibliographic entities that represent the research landscape from multiple perspectives and levels of granularity.

METHODS

We searched and retrieved 978 deep learning studies in biomedicine from the PubMed database. A scientometric analysis was performed by analyzing the metadata, content of influential works, and cited references.

RESULTS

In the process, we identified the current leading fields, major research topics and techniques, knowledge diffusion, and research collaboration. There was a predominant focus on applying deep learning, especially convolutional neural networks, to radiology and medical imaging, whereas a few studies focused on protein or genome analysis. Radiology and medical imaging also appeared to be the most significant knowledge sources and an important field in knowledge diffusion, followed by computer science and electrical engineering. A coauthorship analysis revealed various collaborations among engineering-oriented and biomedicine-oriented clusters of disciplines.

CONCLUSIONS

This study investigated the landscape of deep learning research in biomedicine and confirmed its interdisciplinary nature. Although it has been successful, we believe that there is a need for diverse applications in certain areas to further boost the contributions of deep learning in addressing biomedical research problems. We expect the results of this study to help researchers and communities better align their present and future work.

Age-Oriented Face Synthesis With Conditional Discriminator Pool and Adversarial Triplet Loss

The vanilla Generative Adversarial Networks (GANs) are commonly used to generate realistic images depicting aged and rejuvenated faces. However, the performance of such vanilla GANs in the age-oriented face synthesis task is often compromised by the mode collapse issue, which may produce poorly synthesized faces with indistinguishable visual variations. In addition, recent age-oriented face synthesis methods use the L1 or L2 constraint to preserve the identity information in synthesized faces, which implicitly limits the identity permanence capabilities when these constraints are associated with a trivial weighting factor. In this paper, we propose a method for the age-oriented face synthesis task that achieves high synthesis accuracy with strong identity permanence capabilities. Specifically, to achieve high synthesis accuracy, our method tackles the mode collapse issue with a novel Conditional Discriminator Pool, which consists of multiple discriminators, each targeting one particular age category. To achieve strong identity permanence capabilities, our method uses a novel Adversarial Triplet loss. This loss, which is based on the Triplet loss, adds a ranking operation to further pull the positive embedding towards the anchor embedding to significantly reduce intra-class variances in the feature space. Through extensive experiments, we show that our proposed method outperforms state-of-the-art methods in terms of synthesis accuracy and identity permanence capabilities, both qualitatively and quantitatively.

Low-rank analysis-synthesis dictionary learning with adaptively ordinal locality

Analysis dictionary learning (ADL) has been successfully applied to a variety of learning systems. However, the ordinal locality of analysis dictionary has rarely been explored in constructing discriminative terms. In this paper, a discriminative low-rank analysis-synthesis dictionary learning (LR-ASDL) algorithm with the adaptively ordinal locality is proposed for object classification. Specifically, we first explicitly introduce the relations between the analysis atoms and profiles (i.e., row vectors of the coefficients matrix). That is, the similarity between two profiles depends on that between the corresponding analysis atoms. Moreover, an adaptively ordinal locality preserving(AOLP) term is constructed by simultaneously exploiting the profiles and analysis atoms, which can be learned in a supervised way. In this way, the neighborhood correlations between analysis atoms and the high-order ranking information of each analysis atom's neighbors can be simultaneously preserved in the learning process. Particularly, this helps to uncover the intrinsic underlying data factors and inherit the geometry structure information of training samples. Furthermore, the low-rank model is imposed on the synthesis atoms to further facilitate the learned dictionaries to be more discriminative. Extensive experimental results on eight databases demonstrate that the LR-ASDL algorithm clearly outperforms some analysis and synthesis dictionary learning algorithms using deep and hand-crafted features.

Discriminative Sparsity Graph Embedding for Unconstrained Face Recognition

In this paper, we propose a new dimensionality reduction method named Discriminative Sparsity Graph Embedding (DSGE) which considers the local structure information and the global distribution information simultaneously. Firstly, we adopt the intra-class compactness constraint to automatically construct the intrinsic adjacent graph, which enhances the reconstruction relationship between the given sample and the non-neighbor samples with the same class. Meanwhile, the inter-class compactness constraint is exploited to construct the penalty adjacent graph, which reduces the reconstruction influence between the given sample and the pseudo-neighbor samples with the different classes. Then, the global distribution constraints are introduced to the projection objective function for seeking the optimal subspace which compacts intra-classes samples and alienates inter-classes samples at the same time. Extensive experiments are carried out on AR, Extended Yale B, LFW and PubFig databases which are four representative face datasets, and the corresponding experimental results illustrate the effectiveness of our proposed method.

Simulation of facial growth based on longitudinal data: Age progression and age regression between 7 and 17 years of age using 3D surface data

Modelling of the development of facial morphology during childhood and adolescence is highly useful in forensic and biomedical practice. However, most studies in this area fail to capture the essence of the face as a three-dimensional structure. The main aims of our present study were (1) to construct ageing trajectories for the female and male face between 7 and 17 years of age and (2) to propose a three-dimensional age progression (age -regression) system focused on real growth-related facial changes. Our approach was based on an assessment of a total of 522 three-dimensional (3D) facial scans of Czech children (39 boys, 48 girls) that were longitudinally studied between the ages of 7 to 12 and 12 to 17 years. Facial surface scans were obtained using a Vectra-3D scanner and evaluated using geometric morphometric methods (CPD-DCA, PCA, Hotelling’s T² tests). We observed very similar growth rates between 7 and 10 years in both sexes, followed by an increase in growth velocity in both sexes, with maxima between 11 and 12 years in girls and 11 to 13 years in boys, which are connected with the different timing of the onset of puberty. Based on these partly different ageing trajectories for girls and boys, we simulated the effects of age progression (age regression) on facial scans. In girls, the mean error was 1.81 mm at 12 years and 1.7 mm at 17 years. In boys, the prediction system was slightly less successful: 2.0 mm at 12 years and 1.94 mm at 17 years. The areas with the greatest deviations between predicted and real facial morphology were not important for facial recognition. Changes of body mass index percentiles in children throughout the observation period had no significant influence on the accuracy of the age progression models for both sexes.