3-D Facial Landmark Localization With Asymmetry Patterns and Shape Regression from Incomplete Local Features

Patch-based convolutional neural networks for automatic landmark detection of 3D facial images in clinical settings

BACKGROUND

The facial landmark annotation of 3D facial images is crucial in clinical orthodontics and orthognathic surgeries for accurate diagnosis and treatment planning. While manual landmarking has traditionally been the gold standard, it is labour-intensive and prone to variability.

OBJECTIVE

This study presents a framework for automated landmark detection in 3D facial images within a clinical context, using convolutional neural networks (CNNs), and it assesses its accuracy in comparison to that of ground-truth data.

MATERIAL AND METHODS

Initially, an in-house dataset of 408 3D facial images, each annotated with 37 landmarks by an expert, was constructed. Subsequently, a 2.5D patch-based CNN architecture was trained using this dataset to detect the same set of landmarks automatically.

RESULTS

The developed CNN model demonstrated high accuracy, with an overall mean localization error of 0.83 ± 0.49 mm. The majority of the landmarks had low localization errors, with 95% exhibiting a mean error of less than 1 mm across all axes. Moreover, the method achieved a high success detection rate, with 88% of detections having an error below 1.5 mm and 94% below 2 mm.

CONCLUSION

The automated method used in this study demonstrated accuracy comparable to that achieved with manual annotations within clinical settings. In addition, the proposed framework for automatic landmark localization exhibited improved accuracy over existing models in the literature. Despite these advancements, it is important to acknowledge the limitations of this research, such as that it was based on a single-centre study and a single annotator. Future work should address computational time challenges to achieve further enhancements. This approach has significant potential to improve the efficiency and accuracy of orthodontic and orthognathic procedures.

Accuracy and repeatability of fetal facial measurements in 3D ultrasound: A longitudinal study

OBJECTIVE

Fetal face measurements in prenatal ultrasound can aid in identifying craniofacial abnormalities in the developing fetus. However, the accuracy and reliability of ultrasound measurements can be affected by factors such as fetal position, image quality, and the sonographer's expertise. This study assesses the accuracy and reliability of fetal facial measurements in prenatal ultrasound. Additionally, the temporal evolution of measurements is studied, comparing prenatal and postnatal measurements.

METHODS

Three different experts located up to 23 facial landmarks in 49 prenatal 3D ultrasound scans from normal Caucasian fetuses at weeks 20, 26, and 35 of gestation. Intra- and inter-observer variability was obtained. Postnatal facial measurements were also obtained at 15 days and 1 month postpartum.

RESULTS

Most facial landmarks exhibited low errors, with overall intra- and inter-observer errors of 1.01 mm and 1.60 mm, respectively. Landmarks on the nose were found to be the most reliable, while the most challenging ones were those located on the ears and eyes. Overall, scans obtained at 26 weeks of gestation presented the best trade-off between observer variability and landmark visibility. The temporal evolution of the measurements revealed that the lower face area had the highest rate of growth throughout the latest stages of pregnancy.

CONCLUSIONS

Craniofacial landmarks can be evaluated using 3D fetal ultrasound, especially those located on the nose, mouth, and chin. Despite its limitations, this study provides valuable insights into prenatal and postnatal biometric changes over time, which could aid in developing predictive models for postnatal measurements based on prenatal data.

Facial augmented reality based on hierarchical optimization of similarity aspect graph

BACKGROUND

The existing face matching method requires a point cloud to be drawn on the real face for registration, which results in low registration accuracy due to the irregular deformation of the patient's skin that makes the point cloud have many outlier points.

METHODS

This work proposes a non-contact pose estimation method based on similarity aspect graph hierarchical optimization. The proposed method constructs a distance-weighted and triangular-constrained similarity measure to describe the similarity between views by automatically identifying the 2D and 3D feature points of the face. A mutual similarity clustering method is proposed to construct a hierarchical aspect graph with 3D pose as nodes. A Monte Carlo tree search strategy is used to search the hierarchical aspect graph for determining the optimal pose of the facial 3D model, so as to realize the accurate registration of the facial 3D model and the real face.

RESULTS

The proposed method was used to conduct accuracy verification experiments on the phantoms and volunteers, which were compared with four advanced pose calibration methods. The proposed method obtained average fusion errors of 1.13 ± 0.20 mm and 0.92 ± 0.08 mm in head phantom and volunteer experiments, respectively, which exhibits the best fusion performance among all comparison methods.

CONCLUSIONS

Our experiments proved the effectiveness of the proposed pose estimation method in facial augmented reality.

Two-Stream Spatial-Temporal Graph Convolutional Networks for Driver Drowsiness Detection

Convolutional neural networks (CNNs) have achieved remarkable performance in driver drowsiness detection based on the extraction of deep features of drivers' faces. However, the performance of driver drowsiness detection methods decreases sharply when complications, such as illumination changes in the cab, occlusions and shadows on the driver's face, and variations in the driver's head pose, occur. In addition, current driver drowsiness detection methods are not capable of distinguishing between driver states, such as talking versus yawning or blinking versus closing eyes. Therefore, technical challenges remain in driver drowsiness detection. In this article, we propose a novel and robust two-stream spatial-temporal graph convolutional network (2s-STGCN) for driver drowsiness detection to solve the above-mentioned challenges. To take advantage of the spatial and temporal features of the input data, we use a facial landmark detection method to extract the driver's facial landmarks from real-time videos and then obtain the driver drowsiness detection result by 2s-STGCN. Unlike existing methods, our proposed method uses videos rather than consecutive video frames as processing units. This is the first effort to exploit these processing units in the field of driver drowsiness detection. Moreover, the two-stream framework not only models both the spatial and temporal features but also models both the first-order and second-order information simultaneously, thereby notably improving driver drowsiness detection. Extensive experiments have been performed on the yawn detection dataset (YawDD) and the National TsingHua University drowsy driver detection (NTHU-DDD) dataset. The experimental results validate the feasibility of the proposed method. This method achieves an average accuracy of 93.4% on the YawDD dataset and an average accuracy of 92.7% on the evaluation set of the NTHU-DDD dataset.

A Sparse and Locally Coherent Morphable Face Model for Dense Semantic Correspondence Across Heterogeneous 3D Faces

The 3D Morphable Model (3DMM) is a powerful statistical tool for representing 3D face shapes. To build a 3DMM, a training set of face scans in full point-to-point correspondence is required, and its modeling capabilities directly depend on the variability contained in the training data. Thus, to increase the descriptive power of the 3DMM, establishing a dense correspondence across heterogeneous scans with sufficient diversity in terms of identities, ethnicities, or expressions becomes essential. In this manuscript, we present a fully automatic approach that leverages a 3DMM to transfer its dense semantic annotation across raw 3D faces, establishing a dense correspondence between them. We propose a novel formulation to learn a set of sparse deformation components with local support on the face that, together with an original non-rigid deformation algorithm, allow the 3DMM to precisely fit unseen faces and transfer its semantic annotation. We extensively experimented our approach, showing it can effectively generalize to highly diverse samples and accurately establish a dense correspondence even in presence of complex facial expressions. The accuracy of the dense registration is demonstrated by building a heterogeneous, large-scale 3DMM from more than 9,000 fully registered scans obtained by joining three large datasets together.

Anthropometric Landmark Detection in 3D Head Surfaces Using a Deep Learning Approach

Landmark labeling in 3D head surfaces is an important and routine task in clinical practice to evaluate head shape, namely to analyze cranial deformities or growth evolution. However, manual labeling is still applied, being a tedious and time-consuming task, highly prone to intra-/inter-observer variability, and can mislead the diagnose. Thus, automatic methods for anthropometric landmark detection in 3D models have a high interest in clinical practice. In this paper, a novel framework is proposed to accurately detect landmarks in 3D infant's head surfaces. The proposed method is divided into two stages: (i) 2D representation of the 3D head surface; and (ii) landmark detection through a deep learning strategy. Moreover, a 3D data augmentation method to create shape models based on the expected head variability is proposed. The proposed framework was evaluated in synthetic and real datasets, achieving accurate detection results. Furthermore, the data augmentation strategy proved its added value, increasing the method's performance. Overall, the obtained results demonstrated the robustness of the proposed method and its potential to be used in clinical practice for head shape analysis.

Simultaneous multi-descent regression and feature learning for facial landmarking in depth images

Face alignment (or facial landmarking) is an important task in many face-related applications, ranging from registration, tracking, and animation to higher-level classification problems such as face, expression, or attribute recognition. While several solutions have been presented in the literature for this task so far, reliably locating salient facial features across a wide range of posses still remains challenging. To address this issue, we propose in this paper a novel method for automatic facial landmark localization in 3D face data designed specifically to address appearance variability caused by significant pose variations. Our method builds on recent cascaded regression-based methods to facial landmarking and uses a gating mechanism to incorporate multiple linear cascaded regression models each trained for a limited range of poses into a single powerful landmarking model capable of processing arbitrary-posed input data. We develop two distinct approaches around the proposed gating mechanism: (1) the first uses a gated multiple ridge descent mechanism in conjunction with established (hand-crafted) histogram of gradients features for face alignment and achieves state-of-the-art landmarking performance across a wide range of facial poses and (2) the second simultaneously learns multiple-descent directions as well as binary features that are optimal for the alignment tasks and in addition to competitive landmarking results also ensures extremely rapid processing. We evaluate both approaches in rigorous experiments on several popular datasets of 3D face images, i.e., the FRGCv2 and Bosphorus 3D face datasets and image collections F and G from the University of Notre Dame. The results of our evaluation show that both approaches compare favorably to the state-of-the-art, while exhibiting considerable robustness to pose variations.

A Coarse-to-Fine Approach for 3D Facial Landmarking by Using Deep Feature Fusion

Facial landmarking locates the key facial feature points on facial data, which provides not only information on semantic facial structures, but also prior knowledge for other kinds of facial analysis. However, most of the existing works still focus on the 2D facial image which may suffer from lighting condition variations. In order to address this limitation, this paper presents a coarse-to-fine approach to accurately and automatically locate the facial landmarks by using deep feature fusion on 3D facial geometry data. Specifically, the 3D data is converted to 2D attribute maps firstly. Then, the global estimation network is trained to predict facial landmarks roughly by feeding the fused CNN (Convolutional Neural Network) features extracted from facial attribute maps. After that, input the local fused CNN features extracted from the local patch around each landmark estimated previously, and other local models are trained separately to refine the locations. Tested on the Bosphorus and BU-3DFE datasets, the experimental results demonstrated effectiveness and accuracy of the proposed method for locating facial landmarks. Compared with existed methods, our results have achieved state-of-the-art performance.

Dense 3D Face Correspondence

We present an algorithm that automatically establishes dense correspondences between a large number of 3D faces. Starting from automatically detected sparse correspondences on the outer boundary of 3D faces, the algorithm triangulates existing correspondences and expands them iteratively by matching points of distinctive surface curvature along the triangle edges. After exhausting keypoint matches, further correspondences are established by generating evenly distributed points within triangles by evolving level set geodesic curves from the centroids of large triangles. A deformable model (K3DM) is constructed from the dense corresponded faces and an algorithm is proposed for morphing the K3DM to fit unseen faces. This algorithm iterates between rigid alignment of an unseen face followed by regularized morphing of the deformable model. We have extensively evaluated the proposed algorithms on synthetic data and real 3D faces from the FRGCv2, Bosphorus, BU3DFE and UND Ear databases using quantitative and qualitative benchmarks. Our algorithm achieved dense correspondences with a mean localisation error of 1.28 mm on synthetic faces and detected 14 anthropometric landmarks on unseen real faces from the FRGCv2 database with 3 mm precision. Furthermore, our deformable model fitting algorithm achieved 98.5 percent face recognition accuracy on the FRGCv2 and 98.6 percent on Bosphorus database. Our dense model is also able to generalize to unseen datasets.

Locally Linear Approximation Approach for Incomplete Data

The matrix completion problem is restoring a given matrix with missing entries when handling incomplete data. In many existing researches, rank minimization plays a central role in matrix completion. In this paper, noticing that the locally linear reconstruction can be used to approximate the missing entries, we view the problem from a new perspective and propose an algorithm called locally linear approximation (LLA). The LLA method tries to keep the local structure of the data space while restoring the missing entries from row angle and column angle simultaneously. The experimental results have demonstrated the effectiveness of the proposed method.

Tensor Rank Preserving Discriminant Analysis for Facial Recognition

Facial recognition, one of the basic topics in computer vision and pattern recognition, has received substantial attention in recent years. However, for those traditional facial recognition algorithms, the facial images are reshaped to a long vector, thereby losing part of the original spatial constraints of each pixel. In this paper, a new tensor-based feature extraction algorithm termed tensor rank preserving discriminant analysis (TRPDA) for facial image recognition is proposed; the proposed method involves two stages: in the first stage, the low-dimensional tensor subspace of the original input tensor samples was obtained; in the second stage, discriminative locality alignment was utilized to obtain the ultimate vector feature representation for subsequent facial recognition. On the one hand, the proposed TRPDA algorithm fully utilizes the natural structure of the input samples, and it applies an optimization criterion that can directly handle the tensor spectral analysis problem, thereby decreasing the computation cost compared those traditional tensor-based feature selection algorithms. On the other hand, the proposed TRPDA algorithm extracts feature by finding a tensor subspace that preserves most of the rank order information of the intra-class input samples. Experiments on the three facial databases are performed here to determine the effectiveness of the proposed TRPDA algorithm.

Translational Genetic Modelling of 3D Craniofacial Dysmorphology: Elaborating the Facial Phenotype of Neurodevelopmental Disorders Through the "Prism" of Schizophrenia

Purpose of Review

In the context of human developmental conditions, we review the conceptualisation of schizophrenia as a neurodevelopmental disorder, the status of craniofacial dysmorphology as a clinically accessible index of brain dysmorphogenesis, the ability of genetically modified mouse models of craniofacial dysmorphology to inform on the underlying dysmorphogenic process and how geometric morphometric techniques in mutant mice can extend quantitative analysis.

Recent Findings

Mutant mice with disruption of neuregulin-1, a gene associated meta-analytically with risk for schizophrenia, constitute proof-of-concept studies of murine facial dysmorphology in a manner analogous to clinical studies in schizophrenia. Geometric morphometric techniques informed on the topography of facial dysmorphology and identified asymmetry therein.

Summary

Targeted disruption in mice of genes involved in individual components of developmental processes and analysis of resultant facial dysmorphology using geometric morphometrics can inform on mechanisms of dysmorphogenesis at levels of incisiveness not possible in human subjects.

The definitions of three-dimensional landmarks on the human face: an interdisciplinary view

The analysis of shape is a key part of anatomical research and in the large majority of cases landmarks provide a standard starting point. However, while the technology of image capture has developed rapidly and in particular three‐dimensional imaging is widely available, the definitions of anatomical landmarks remain rooted in their two‐dimensional origins. In the important case of the human face, standard definitions often require careful orientation of the subject. This paper considers the definitions of facial landmarks from an interdisciplinary perspective, including biological and clinical motivations, issues associated with imaging and subsequent analysis, and the mathematical definition of surface shape using differential geometry. This last perspective provides a route to definitions of landmarks based on surface curvature, often making use of ridge and valley curves, which is genuinely three‐dimensional and is independent of orientation. Specific definitions based on curvature are proposed. These are evaluated, along with traditional definitions, in a study that uses a hierarchical (random effects) model to estimate the error variation that is present at several different levels within the image capture process. The estimates of variation at these different levels are of interest in their own right but, in addition, evidence is provided that variation is reduced at the observer level when the new landmark definitions are used.