An Automatic 3D Facial Landmarking Algorithm Using 2D Gabor Wavelets

Mapping genes for human face shape: exploration of univariate phenotyping strategies

Human facial shape, while strongly heritable, involves both genetic and structural complexity, necessitating precise phenotyping for accurate assessment. Common phenotyping strategies include simplifying 3D facial features into univariate traits such as anthropometric measurements (e.g., inter-landmark distances), unsupervised dimensionality reductions (e.g., principal component analysis (PCA) and auto-encoder (AE) approaches), and assessing resemblance to particular facial gestalts (e.g., syndromic facial archetypes). This study provides a comparative assessment of these strategies in genome-wide association studies (GWASs) of 3D facial shape. Specifically, we investigated inter-landmark distances, PCA and AE-derived latent dimensions, and facial resemblance to random, extreme, and syndromic gestalts within a GWAS of 8,426 individuals of recent European ancestry. Inter-landmark distances exhibit the highest SNP-based heritability as estimated via LD score regression, followed by AE dimensions. Conversely, resemblance scores to extreme and syndromic facial gestalts display the lowest heritability, in line with expectations. Notably, the aggregation of multiple GWASs on facial resemblance to random gestalts reveals the highest number of independent genetic loci. This novel, easy-to-implement phenotyping approach holds significant promise for capturing genetically relevant morphological traits derived from complex biomedical imaging datasets, and its applications extend beyond faces. Nevertheless, these different phenotyping strategies capture different genetic influences on craniofacial shape. Thus, it remains valuable to explore these strategies individually and in combination to gain a more comprehensive understanding of the genetic factors underlying craniofacial shape and related traits.

Author Summary

Advancements linking variation in the human genome to phenotypes have rapidly evolved in recent decades and have revealed that most human traits are influenced by genetic variants to at least some degree. While many traits, such as stature, are straightforward to acquire and investigate, the multivariate and multipartite nature of facial shape makes quantification more challenging. In this study, we compared the impact of different facial phenotyping approaches on gene mapping outcomes. Our findings suggest that the choice of facial phenotyping method has an impact on apparent trait heritability and the ability to detect genetic association signals. These results offer valuable insights into the importance of phenotyping in genetic investigations, especially when dealing with highly complex morphological traits.

Data-driven trait heritability-based extraction of human facial phenotypes

A genome-wide association study (GWAS) of a complex, multi-dimensional morphological trait, such as the human face, typically relies on predefined and simplified phenotypic measurements, such as inter-landmark distances and angles. These measures are predominantly designed by human experts based on perceived biological or clinical knowledge. To avoid use handcrafted phenotypes (i.e., a priori expert-identified phenotypes), alternative automatically extracted phenotypic descriptors, such as features derived from dimension reduction techniques (e.g., principal component analysis), are employed. While the features generated by such computational algorithms capture the geometric variations of the biological shape, they are not necessarily genetically relevant. Therefore, genetically informed data-driven phenotyping is desirable. Here, we propose an approach where phenotyping is done through a data-driven optimization of trait heritability, defined as the degree of variation in a phenotypic trait in a population that is due to genetic variation. The resulting phenotyping process consists of two steps: 1) constructing a feature space that models shape variations using dimension reduction techniques, and 2) searching for directions in the feature space exhibiting high trait heritability using a genetic search algorithm (i.e., heuristic inspired by natural selection). We show that the phenotypes resulting from the proposed trait heritability-optimized training differ from those of principal components in the following aspects: 1) higher trait heritability, 2) higher SNP heritability, and 3) identification of the same number of independent genetic loci with a smaller number of effective traits. Our results demonstrate that data-driven trait heritability-based optimization enables the automatic extraction of genetically relevant phenotypes, as shown by their increased power in genome-wide association scans.

Advancement in Human Face Prediction Using DNA

The rapid improvements in identifying the genetic factors contributing to facial morphology have enabled the early identification of craniofacial syndromes. Similarly, this technology can be vital in forensic cases involving human identification from biological traces or human remains, especially when reference samples are not available in the deoxyribose nucleic acid (DNA) database. This review summarizes the currently used methods for predicting human phenotypes such as age, ancestry, pigmentation, and facial features based on genetic variations. To identify the facial features affected by DNA, various two-dimensional (2D)- and three-dimensional (3D)-scanning techniques and analysis tools are reviewed. A comparison between the scanning technologies is also presented in this review. Face-landmarking techniques and face-phenotyping algorithms are discussed in chronological order. Then, the latest approaches in genetic to 3D face shape analysis are emphasized. A systematic review of the current markers that passed the threshold of a genome-wide association (GWAS) of single nucleotide polymorphism (SNP)-face traits from the GWAS Catalog is also provided using the preferred reporting items for systematic reviews and meta-analyses (PRISMA), approach. Finally, the current challenges in forensic DNA phenotyping are analyzed and discussed.

Decoding the Human Face: Progress and Challenges in Understanding the Genetics of Craniofacial Morphology

Variations in the form of the human face, which plays a role in our individual identities and societal interactions, have fascinated scientists and artists alike. Here, we review our current understanding of the genetics underlying variation in craniofacial morphology and disease-associated dysmorphology, synthesizing decades of progress on Mendelian syndromes in addition to more recent results from genome-wide association studies of human facial shape and disease risk. We also discuss the various approaches used to phenotype and quantify facial shape, which are of particular importance due to the complex, multipartite nature of the craniofacial form. We close by discussing how experimental studies have contributed and will further contribute to our understanding of human genetic variation and then proposing future directions and applications for the field.

Static and Motion Facial Analysis for Craniofacial Assessment and Diagnosing Diseases

Deviation from a normal facial shape and symmetry can arise from numerous sources, including physical injury and congenital birth defects. Such abnormalities can have important aesthetic and functional consequences. Furthermore, in clinical genetics distinctive facial appearances are often associated with clinical or genetic diagnoses; the recognition of a characteristic facial appearance can substantially narrow the search space of potential diagnoses for the clinician. Unusual patterns of facial movement and expression can indicate disturbances to normal mechanical functioning or emotional affect. Computational analyses of static and moving 2D and 3D images can serve clinicians and researchers by detecting and describing facial structural, mechanical, and affective abnormalities objectively. In this review we survey traditional and emerging methods of facial analysis, including statistical shape modeling, syndrome classification, modeling clinical face phenotype spaces, and analysis of facial motion and affect. Expected final online publication date for the Annual Review of Biomedical Data Science, Volume 5 is August 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Heart Infection Prognosis Analysis by Two-dimensional Spot Tracking Imaging

Cardiovascular death is one of the leading causes worldwide; an accurate identification followed by diagnosing the cardiovascular disease increases the chance of a better recovery. Among different demonstrated strategies, imaging on cardiac infections yields a visible result and highly reliable compared to other analytical methods. Two-dimensional spot tracking imaging is the emerging new technology that has been used to study the function and structure of the heart and test the deformation and movement of the myocardium. Particularly, it helps to capture the images of each segment in different directions of myocardial strain values, such as valves of radial strain, longitudinal strain, and circumferential strain. In this overview, we discussed the imaging of infections in the heart by using the two-dimensional spot tracking.

Human Emotion Recognition Based on Face and Facial Expression Detection Using Deep Belief Network Under Complicated Backgrounds

The human emotion recognition based on facial expression has a significant meaning in the application of intelligent man–machine interaction. However, the human face images vary largely in real environments due to the complex backgrounds and luminance. To solve this problem, this paper proposes a robust face detection method based on skin color enhancement model and a facial expression recognition algorithm with block principal component analysis (PCA). First, the luminance range of human face image is broadened and the contrast ratio of skin color is strengthened by the homomorphic filter. Second, the skin color enhancement model is established using YCbCr color space components to locate the face area. Third, the feature based on differential horizontal integral projection is extracted from the face. Finally, the block PCA with deep neural network is used to accomplish the facial expression recognition. The experimental results indicate that in the case of weaker illumination and more complicated backgrounds, both the face detection and facial expression recognition can be achieved effectively by the proposed algorithm, meanwhile the mean recognition rate obtained by the facial expression recognition method is improved by 2.7% comparing with the traditional Local Binary Patterns (LBPs) method.

Novel genetic loci affecting facial shape variation in humans

The human face represents a combined set of highly heritable phenotypes, but knowledge on its genetic architecture remains limited, despite the relevance for various fields. A series of genome-wide association studies on 78 facial shape phenotypes quantified from 3-dimensional facial images of 10,115 Europeans identified 24 genetic loci reaching study-wide suggestive association (p < 5 × 10-8), among which 17 were previously unreported. A follow-up multi-ethnic study in additional 7917 individuals confirmed 10 loci including six unreported ones (padjusted < 2.1 × 10-3). A global map of derived polygenic face scores assembled facial features in major continental groups consistent with anthropological knowledge. Analyses of epigenomic datasets from cranial neural crest cells revealed abundant cis-regulatory activities at the face-associated genetic loci. Luciferase reporter assays in neural crest progenitor cells highlighted enhancer activities of several face-associated DNA variants. These results substantially advance our understanding of the genetic basis underlying human facial variation and provide candidates for future in-vivo functional studies.

An automatic approach for classification and categorisation of lip morphological traits

Classification of facial traits (e.g., lip shape) is an important area of medical research, for example, in determining associations between lip traits and genetic variants which may lead to a cleft lip. In clinical situations, classification of facial traits is usually performed subjectively directly on the individual or recorded later from a three-dimensional image, which is time consuming and prone to operator errors. The present study proposes, for the first time, an automatic approach for the classification and categorisation of lip area traits. Our approach uses novel three-dimensional geometric features based on surface curvatures measured along geodesic paths between anthropometric landmarks. Different combinations of geodesic features are analysed and compared. The effect of automatically identified categories on the face is visualised using a partial least squares method. The method was applied to the classification and categorisation of six lip shape traits (philtrum, Cupid’s bow, lip contours, lip-chin, and lower lip tone) in a large sample of 4747 faces of normal British Western European descents. The proposed method demonstrates correct automatic classification rate of up to 90%.

Quantification of Facial Traits

Measuring facial traits by quantitative means is a prerequisite to investigate epidemiological, clinical, and forensic questions. This measurement process has received intense attention in recent years. We divided this process into the registration of the face, landmarking, morphometric quantification, and dimension reduction. Face registration is the process of standardizing pose and landmarking annotates positions in the face with anatomic description or mathematically defined properties (pseudolandmarks). Morphometric quantification computes pre-specified transformations such as distances. Landmarking: We review face registration methods which are required by some landmarking methods. Although similar, face registration and landmarking are distinct problems. The registration phase can be seen as a pre-processing step and can be combined independently with a landmarking solution. Existing approaches for landmarking differ in their data requirements, modeling approach, and training complexity. In this review, we focus on 3D surface data as captured by commercial surface scanners but also cover methods for 2D facial pictures, when methodology overlaps. We discuss the broad categories of active shape models, template based approaches, recent deep-learning algorithms, and variations thereof such as hybrid algorithms. The type of algorithm chosen depends on the availability of pre-trained models for the data at hand, availability of an appropriate landmark set, accuracy characteristics, and training complexity. Quantification: Landmarking of anatomical landmarks is usually augmented by pseudo-landmarks, i.e., indirectly defined landmarks that densely cover the scan surface. Such a rich data set is not amenable to direct analysis but is reduced in dimensionality for downstream analysis. We review classic dimension reduction techniques used for facial data and face specific measures, such as geometric measurements and manifold learning. Finally, we review symmetry registration and discuss reliability.

MeshMonk: Open-source large-scale intensive 3D phenotyping

Dense surface registration, commonly used in computer science, could aid the biological sciences in accurate and comprehensive quantification of biological phenotypes. However, few toolboxes exist that are openly available, non-expert friendly, and validated in a way relevant to biologists. Here, we report a customizable toolbox for reproducible high-throughput dense phenotyping of 3D images, specifically geared towards biological use. Given a target image, a template is first oriented, repositioned, and scaled to the target during a scaled rigid registration step, then transformed further to fit the specific shape of the target using a non-rigid transformation. As validation, we use n = 41 3D facial images to demonstrate that the MeshMonk registration is accurate, with 1.26 mm average error, across 19 landmarks, between placements from manual observers and using the MeshMonk toolbox. We also report no variation in landmark position or centroid size significantly attributable to landmarking method used. Though validated using 19 landmarks, the MeshMonk toolbox produces a dense mesh of vertices across the entire surface, thus facilitating more comprehensive investigations of 3D shape variation. This expansion opens up exciting avenues of study in assessing biological shapes to better understand their phenotypic variation, genetic and developmental underpinnings, and evolutionary history.

Three-dimensional soft tissue effects of mandibular midline distraction and surgically assisted rapid maxillary expansion: an automatic stereophotogrammetry landmarking analysis

Studies on mandibular midline distraction (MMD) are mostly performed using conventional research methods. Concerning surgically assisted rapid maxillary expansion (SARME), more research is conducted using three-dimensional (3D) techniques. Research on bimaxillary expansion, the combination of MMD and SARME, is reported sparsely. The main objective of this study was to provide a 3D evaluation of soft tissue effects following SARME and/or MMD. Patients who underwent SARME and/or MMD between 2008 and 2013 were included. Stereophotogrammetry was undertaken at the following time points: preoperative (T1), immediately post-distraction (T2), 1year postoperative (T3). An automatic 3D facial landmarking algorithm using two-dimensional Gabor wavelets was applied for the analysis. Twenty patients who had undergone SARME were included, 12 of whom had undergone bimaxillary expansion. Age at the time of surgery ranged from 16 to 47 years. There was a significant downward displacement of soft tissue pogonion. Furthermore, there was a significant mean increase of 2.20mm in inter-alar width and a non-significant mean increase of 1.77mm in inter-alar curvature point width. In conclusion, automatic stereophotogrammetry landmarking analysis of soft tissue effects showed downward displacement of soft tissue pogonion following bimaxillary expansion and transverse widening of the inter-alar width and a tendency towards an increase in inter-alar curvature point width after SARME.

Spatially Dense 3D Facial Heritability and Modules of Co-heritability in a Father-Offspring Design

Introduction: The human face is a complex trait displaying a strong genetic component as illustrated by various studies on facial heritability. Most of these start from sparse descriptions of facial shape using a limited set of landmarks. Subsequently, facial features are preselected as univariate measurements or principal components and the heritability is estimated for each of these features separately. However, none of these studies investigated multivariate facial features, nor the co-heritability between different facial features. Here we report a spatially dense multivariate analysis of facial heritability and co-heritability starting from data from fathers and their children available within ALSPAC. Additionally, we provide an elaborate overview of related craniofacial heritability studies.

Methods: In total, 3D facial images of 762 father-offspring pairs were retained after quality control. An anthropometric mask was applied to these images to establish spatially dense quasi-landmark configurations. Partial least squares regression was performed and the (co-)heritability for all quasi-landmarks (∼7160) was computed as twice the regression coefficient. Subsequently, these were used as input to a hierarchical facial segmentation, resulting in the definition of facial modules that are internally integrated through the biological mechanisms of inheritance. Finally, multivariate heritability estimates were obtained for each of the resulting modules.

Results: Nearly all modular estimates reached statistical significance under 1,000,000 permutations and after multiple testing correction (p ≤ 1.3889 × 10^-3), displaying low to high heritability scores. Particular facial areas showing the greatest heritability were similar for both sons and daughters. However, higher estimates were obtained in the former. These areas included the global face, upper facial part (encompassing the nasion, zygomas and forehead) and nose, with values reaching 82% in boys and 72% in girls. The lower parts of the face only showed low to moderate levels of heritability.

Conclusion: In this work, we refrain from reducing facial variation to a series of individual measurements and analyze the heritability and co-heritability from spatially dense landmark configurations at multiple levels of organization. Finally, a multivariate estimation of heritability for global-to-local facial segments is reported. Knowledge of the genetic determination of facial shape is useful in the identification of genetic variants that underlie normal-range facial variation.

Automated human skull landmarking with 2D Gabor wavelets

Landmarking of CT scans is an important step in the alignment of skulls that is key in surgery planning, pre-/post-surgery comparisons, and morphometric studies. We present a novel method for automatically locating anatomical landmarks on the surface of cone beam CT-based image models of human skulls using 2D Gabor wavelets and ensemble learning. The algorithm is validated via human inter- and intra-rater comparisons on a set of 39 scans and a skull superimposition experiment with an established surgery planning software (Maxilim). Automatic landmarking results in an accuracy of 1-2 mm for a subset of landmarks around the nose area as compared to a gold standard derived from human raters. These landmarks are located in eye sockets and lower jaw, which is competitive with or surpasses inter-rater variability. The well-performing landmark subsets allow for the automation of skull superimposition in clinical applications. Our approach delivers accurate results, has modest training requirements (training set size of 30-40 items) and is generic, so that landmark sets can be easily expanded or modified to accommodate shifting landmark interests, which are important requirements for the landmarking of larger cohorts.

Ensemble landmarking of 3D facial surface scans

Landmarking of 3D facial surface scans is an important analysis step in medical and biological applications, such as genome-wide association studies (GWAS). Manual landmarking is often employed with considerable cost and rater dependent variability. Landmarking automatically with minimal training is therefore desirable. We apply statistical ensemble methods to improve automated landmarking of 3D facial surface scans. Base landmarking algorithms using features derived from 3D surface scans are combined using either bagging or stacking. A focus is on low training complexity of maximal 40 training samples with template based landmarking algorithms that have proved successful in such applications. Additionally, we use correlations between landmark coordinates by introducing a search strategy guided by principal components (PCs) of training landmarks. We found that bagging has no useful impact, while stacking strongly improves accuracy to an average error of 1.7 mm across all 21 landmarks in this study, a 22% improvement as compared to a previous, comparable algorithm. Heritability estimates in twin pairs also show improvements when using facial distances from landmarks. Ensemble methods allow improvement of automatic, accurate landmarking of 3D facial images with minimal training which is advantageous in large cohort studies for GWAS and when landmarking needs change or data quality varies.