A Registration and Deep Learning Approach to Automated Landmark Detection for Geometric Morphometrics

Three-dimensional magnetic resonance imaging-based statistical shape analysis and machine learning-based prediction of patellofemoral instability

This study performed three-dimensional (3D) magnetic resonance imaging (MRI)-based statistical shape analysis (SSA) by comparing patellofemoral instability (PFI) and normal femur models, and developed a machine learning (ML)-based prediction model. Twenty (19 patients) and 31 MRI scans (30 patients) of femurs with PFI and normal femurs, respectively, were used. Bone and cartilage segmentation of the distal femurs was performed and subsequently converted into 3D reconstructed models. The pointwise distance map showed anterior elevation of the trochlea, particularly at the central floor of the proximal trochlea, in the PFI models compared with the normal models. Principal component analysis examined shape variations in the PFI group, and several principal components exhibited shape variations in the trochlear floor and intercondylar width. Multivariate analysis showed that these shape components were significantly correlated with the PFI/non-PFI distinction after adjusting for age and sex. Our ML-based prediction model for PFI achieved a strong predictive performance with an accuracy of 0.909 ± 0.015, and an area under the curve of 0.939 ± 0.009 when using a support vector machine with a linear kernel. This study demonstrated that 3D MRI-based SSA can realistically visualize statistical results on surface models and may facilitate the understanding of complex shape features.

Classifying high-dimensional phenotypes with ensemble learning

Classification is a fundamental task in biology used to assign members to a class. While linear discriminant functions have long been effective, advances in phenotypic data collection are yielding increasingly high-dimensional datasets with more classes, unequal class covariances, and non-linear distributions. Numerous studies have deployed machine learning techniques to classify such distributions, but they are often restricted to a particular organism, a limited set of algorithms, and/or a specific classification task. In addition, the utility of ensemble learning or the strategic combination of models has not been fully explored.We performed a meta-analysis of 33 algorithms across 20 datasets containing over 20,000 high-dimensional shape phenotypes using an ensemble learning framework. Both binary (e.g., sex, environment) and multi-class (e.g., species, genotype, population) classification tasks were considered. The ensemble workflow contains functions for preprocessing, training individual learners and ensembles, and model evaluation. We evaluated algorithm performance within and among datasets. Furthermore, we quantified the extent to which various dataset and phenotypic properties impact performance.We found that discriminant analysis variants and neural networks were the most accurate base learners on average. However, their performance varied substantially between datasets. Ensemble models achieved the highest performance on average, both within and among datasets, increasing average accuracy by up to 3% over the top base learner. Higher class R2 values, mean class shape distances, and between- vs. within-class variances were positively associated with performance, whereas higher class covariance distances were negatively associated. Class balance and total sample size were not predictive.Learning-based classification is a complex task driven by many hyperparameters. We demonstrate that selecting and optimizing an algorithm based on the results of another study is a flawed strategy. Ensemble models instead offer a flexible approach that is data agnostic and exceptionally accurate. By assessing the impact of various dataset and phenotypic properties on classification performance, we also offer potential explanations for variation in performance. Researchers interested in maximizing performance stand to benefit from the simplicity and effectiveness of our approach made accessible via the R package pheble.

Quantifying the relationship between cell proliferation and morphology during development of the face

Morphogenesis requires highly coordinated, complex interactions between cellular processes: proliferation, migration, and apoptosis, along with physical tissue interactions. How these cellular and tissue dynamics drive morphogenesis remains elusive. Three dimensional (3D) microscopic imaging poses great promise, and generates elegant images. However, generating even moderate through-put quantified images is challenging for many reasons. As a result, the association between morphogenesis and cellular processes in 3D developing tissues has not been fully explored. To address this critical gap, we have developed an imaging and image analysis pipeline to enable 3D quantification of cellular dynamics along with 3D morphology for the same individual embryo. Specifically, we focus on how 3D distribution of proliferation relates to morphogenesis during mouse facial development. Our method involves imaging with light-sheet microscopy, automated segmentation of cells and tissues using machine learning-based tools, and quantification of external morphology via geometric morphometrics. Applying this framework, we show that changes in proliferation are tightly correlated to changes in morphology over the course of facial morphogenesis. These analyses illustrate the potential of this pipeline to investigate mechanistic relationships between cellular dynamics and morphogenesis during embryonic development.

An automated landmark method to describe geometric changes in the human mandible during growth

OBJECTIVE

The principal aim of this study was to assess an automatic landmarking approach to human mandibles based on the atlas method. The secondary aim was to identify the areas of greatest variation in the mandibles of middle-aged to older adults.

DESIGN

Our sample consisted of 160 mandibles from computed tomography scans of 80 men and 80 women aged between 40 and 79 years. Eleven anatomical landmarks were placed manually on mandibles. The automated landmarking through point cloud alignment and correspondence (ALPACA) method implemented in 3D Slicer was used to automatically place landmarks to all meshes. Euclidean distances, normalized centroid size, and Procrustes ANOVA were calculated for both methods. A pseudo-landmarks approach was followed using ALPACA to identify areas of changes among our sample.

RESULTS

The ALPACA method showed significant differences in Euclidean distances for all landmarks compared to the manual method. A mean Euclidean distance of 1.7 mm was found for the ALPACA method and 0.99 mm for the manual method. Both methods found that sex, age, and size had a significant effect on mandibular shape. The greatest variations were observed in the condyle, ramus, and symphysis regions.

CONCLUSION

The results obtained using the ALPACA method are acceptable and promising. This approach can automatically place landmarks with an average accuracy of less than 2 mm, which may be sufficient in most anthropometric analyses. In the light of our results, however, odontological application such as occlusal analysis is not recommended.

Automatic landmark correspondence detection in medical images with an application to deformable image registration

Purpose

Deformable image registration (DIR) can benefit from additional guidance using corresponding landmarks in the images. However, the benefits thereof are largely understudied, especially due to the lack of automatic landmark detection methods for three-dimensional (3D) medical images.

Approach

We present a deep convolutional neural network (DCNN), called DCNN-Match, that learns to predict landmark correspondences in 3D images in a self-supervised manner. We trained DCNN-Match on pairs of computed tomography (CT) scans containing simulated deformations. We explored five variants of DCNN-Match that use different loss functions and assessed their effect on the spatial density of predicted landmarks and the associated matching errors. We also tested DCNN-Match variants in combination with the open-source registration software Elastix to assess the impact of predicted landmarks in providing additional guidance to DIR.

Results

We tested our approach on lower abdominal CT scans from cervical cancer patients: 121 pairs containing simulated deformations and 11 pairs demonstrating clinical deformations. The results showed significant improvement in DIR performance when landmark correspondences predicted by DCNN-Match were used in the case of simulated ( p = 0 e 0 ) as well as clinical deformations ( p = 0.030 ). We also observed that the spatial density of the automatic landmarks with respect to the underlying deformation affect the extent of improvement in DIR. Finally, DCNN-Match was found to generalize to magnetic resonance imaging scans without requiring retraining, indicating easy applicability to other datasets.

Conclusions

DCNN-match learns to predict landmark correspondences in 3D medical images in a self-supervised manner, which can improve DIR performance.

The impacts of fine-tuning, phylogenetic distance, and sample size on big-data bioacoustics

Vocalizations in animals, particularly birds, are critically important behaviors that influence their reproductive fitness. While recordings of bioacoustic data have been captured and stored in collections for decades, the automated extraction of data from these recordings has only recently been facilitated by artificial intelligence methods. These have yet to be evaluated with respect to accuracy of different automation strategies and features. Here, we use a recently published machine learning framework to extract syllables from ten bird species ranging in their phylogenetic relatedness from 1 to 85 million years, to compare how phylogenetic relatedness influences accuracy. We also evaluate the utility of applying trained models to novel species. Our results indicate that model performance is best on conspecifics, with accuracy progressively decreasing as phylogenetic distance increases between taxa. However, we also find that the application of models trained on multiple distantly related species can improve the overall accuracy to levels near that of training and analyzing a model on the same species. When planning big-data bioacoustics studies, care must be taken in sample design to maximize sample size and minimize human labor without sacrificing accuracy.

Automated landmarking via multiple templates

Manually collecting landmarks for quantifying complex morphological phenotypes can be laborious and subject to intra and interobserver errors. However, most automated landmarking methods for efficiency and consistency fall short of landmarking highly variable samples due to the bias introduced by the use of a single template. We introduce a fast and open source automated landmarking pipeline (MALPACA) that utilizes multiple templates for accommodating large-scale variations. We also introduce a K-means method of choosing the templates that can be used in conjunction with MALPACA, when no prior information for selecting templates is available. Our results confirm that MALPACA significantly outperforms single-template methods in landmarking both single and multi-species samples. K-means based template selection can also avoid choosing the worst set of templates when compared to random template selection. We further offer an example of post-hoc quality check for each individual template for further refinement. In summary, MALPACA is an efficient and reproducible method that can accommodate large morphological variability, such as those commonly found in evolutionary studies. To support the research community, we have developed open-source and user-friendly software tools for performing K-means multi-templates selection and MALPACA.

Thirty years of geometric morphometrics: Achievements, challenges, and the ongoing quest for biological meaningfulness

The foundations of geometric morphometrics were worked out about 30 years ago and have continually been refined and extended. What has remained as a central thrust and source of debate in the morphometrics community is the shared goal of meaningful biological inference through a tight connection between biological theory, measurement, multivariate biostatistics, and geometry. Here we review the building blocks of modern geometric morphometrics: the representation of organismal geometry by landmarks and semilandmarks, the computation of shape or form variables via superimposition, the visualization of statistical results as actual shapes or forms, the decomposition of shape variation into symmetric and asymmetric components and into different spatial scales, the interpretation of various geometries in shape or form space, and models of the association between shape or form and other variables, such as environmental, genetic, or behavioral data. We focus on recent developments and current methodological challenges, especially those arising from the increasing number of landmarks and semilandmarks, and emphasize the importance of thorough exploratory multivariate analyses rather than single scalar summary statistics. We outline promising directions for further research and for the evaluation of new developments, such as "landmark-free" approaches. To illustrate these methods, we analyze three-dimensional human face shape based on data from the Avon Longitudinal Study of Parents and Children (ALSPAC).

MusMorph, a database of standardized mouse morphology data for morphometric meta-analyses

Complex morphological traits are the product of many genes with transient or lasting developmental effects that interact in anatomical context. Mouse models are a key resource for disentangling such effects, because they offer myriad tools for manipulating the genome in a controlled environment. Unfortunately, phenotypic data are often obtained using laboratory-specific protocols, resulting in self-contained datasets that are difficult to relate to one another for larger scale analyses. To enable meta-analyses of morphological variation, particularly in the craniofacial complex and brain, we created MusMorph, a database of standardized mouse morphology data spanning numerous genotypes and developmental stages, including E10.5, E11.5, E14.5, E15.5, E18.5, and adulthood. To standardize data collection, we implemented an atlas-based phenotyping pipeline that combines techniques from image registration, deep learning, and morphometrics. Alongside stage-specific atlases, we provide aligned micro-computed tomography images, dense anatomical landmarks, and segmentations (if available) for each specimen (N = 10,056). Our workflow is open-source to encourage transparency and reproducible data collection. The MusMorph data and scripts are available on FaceBase ( www.facebase.org , https://doi.org/10.25550/3-HXMC ) and GitHub ( https://github.com/jaydevine/MusMorph ).

The genetic basis of neurocranial size and shape across varied lab mouse populations

Brain and skull tissues interact through molecular signalling and mechanical forces during head development, leading to a strong correlation between the neurocranium and the external brain surface. Therefore, when brain tissue is unavailable, neurocranial endocasts are often used to approximate brain size and shape. Evolutionary changes in brain morphology may have resulted in secondary changes to neurocranial morphology, but the developmental and genetic processes underlying this relationship are not well understood. Using automated phenotyping methods, we quantified the genetic basis of endocast variation across large genetically varied populations of laboratory mice in two ways: (1) to determine the contributions of various genetic factors to neurocranial form and (2) to help clarify whether a neurocranial variation is based on genetic variation that primarily impacts bone development or on genetic variation that primarily impacts brain development, leading to secondary changes in bone morphology. Our results indicate that endocast size is highly heritable and is primarily determined by additive genetic factors. In addition, a non-additive inbreeding effect led to founder strains with lower neurocranial size, but relatively large brains compared to skull size; suggesting stronger canalization of brain size and/or a general allometric effect. Within an outbred sample of mice, we identified a locus on mouse chromosome 1 that is significantly associated with variation in several positively correlated endocast size measures. Because the protein-coding genes at this locus have been previously associated with brain development and not with bone development, we propose that genetic variation at this locus leads primarily to variation in brain volume that secondarily leads to changes in neurocranial globularity. We identify a strain-specific missense mutation within Akt3 that is a strong causal candidate for this genetic effect. Whilst it is not appropriate to generalize our hypothesis for this single locus to all other loci that also contribute to the complex trait of neurocranial skull morphology, our results further reveal the genetic basis of neurocranial variation and highlight the importance of the mechanical influence of brain growth in determining skull morphology.

Testing the accuracy of 3D automatic landmarking via genome-wide association studies

Various advances in 3D automatic phenotyping and landmark-based geometric morphometric methods have been made. While it is generally accepted that automatic landmarking compromises the capture of the biological variation, no studies have directly tested the actual impact of such landmarking approaches in analyses requiring a large number of specimens and for which the precision of phenotyping is crucial to extract an actual biological signal adequately. Here, we use a recently developed 3D atlas-based automatic landmarking method to test its accuracy in detecting QTLs associated with craniofacial development of the house mouse skull and lower jaws for a large number of specimens (circa 700) that were previously phenotyped via a semiautomatic landmarking method complemented with manual adjustment. We compare both landmarking methods with univariate and multivariate mapping of the skull and the lower jaws. We find that most significant SNPs and QTLs are not recovered based on the data derived from the automatic landmarking method. Our results thus confirm the notion that information is lost in the automated landmarking procedure although somewhat dependent on the analyzed structure. The automatic method seems to capture certain types of structures slightly better, such as lower jaws whose shape is almost entirely summarized by its outline and could be assimilated as a 2D flat object. By contrast, the more apparent 3D features exhibited by a structure such as the skull are not adequately captured by the automatic method. We conclude that using 3D atlas-based automatic landmarking methods requires careful consideration of the experimental question.

Phenotypic systems biology for organisms: Concepts, methods and case studies

Design principles of phenotypes in organisms are fundamental issues in physical biology. So far, understanding “systems” of living organisms have been chiefly promoted by understanding the underlying biomolecules such as genes and proteins, and their intra- and inter-relationships and regulations. After a long period of sophistication, biophysics and molecular biology have established a general framework for understanding ‘molecular systems’ in organisms without regard to species, so that the findings of fly studies can be applied to mouse studies. However, little attention has been paid to exploring “phenotypic systems” in organisms, and thus its general framework remains poorly understood. Here I review concepts, methods, and case studies using butterfly and moth wing patterns to explore phenotypes as systems. First, I present a unifying framework for phenotypic traits as systems, termed multi-component systems. Second, I describe how to define components of phenotypic systems, and also show how to quantify interactions among phenotypic parts. Subsequently, I introduce the concept of the macro-evolutionary process, which illustrates how to generate complex traits. In this point, I also introduce mathematical methods, “phylogenetic comparative methods”, which provide stochastic processes along molecular phylogeny as bifurcated paths to quantify trait evolution. Finally, I would like to propose two key concepts, macro-evolutionary pathways and genotype-phenotype loop (GP loop), which must be needed for the next directions. I hope these efforts on phenotypic biology will become one major target in biophysics and create the next generations of textbooks. This review article is an extended version of the Japanese article, Biological Physics in Phenotypic Systems of Living Organisms, published in SEIBUTSU-BUTSURI Vol. 61, p. 31–35 (2021).

ALPACA: A fast and accurate computer vision approach for automated landmarking of three‐dimensional biological structures

Landmark‐based geometric morphometrics has emerged as an essential discipline for the quantitative analysis of size and shape in ecology and evolution. With the ever‐increasing density of digitized landmarks, the possible development of a fully automated method of landmark placement has attracted considerable attention. Despite the recent progress in image registration techniques, which could provide a pathway to automation, three‐dimensional (3D) morphometric data are still mainly gathered by trained experts. For the most part, the large infrastructure requirements necessary to perform image‐based registration, together with its system specificity and its overall speed, have prevented its wide dissemination.

Here, we propose and implement a general and lightweight point cloud‐based approach to automatically collect high‐dimensional landmark data in 3D surfaces (Automated Landmarking through Point cloud Alignment and Correspondence Analysis). Our framework possesses several advantages compared with image‐based approaches. First, it presents comparable landmarking accuracy, despite relying on a single, random reference specimen and much sparser sampling of the structure's surface. Second, it can be efficiently run on consumer‐grade personal computers. Finally, it is general and can be applied at the intraspecific level to any biological structure of interest, regardless of whether anatomical atlases are available.

Our validation procedures indicate that the method can recover intraspecific patterns of morphological variation that are largely comparable to those obtained by manual digitization, indicating that the use of an automated landmarking approach should not result in different conclusions regarding the nature of multivariate patterns of morphological variation.

The proposed point cloud‐based approach has the potential to increase the scale and reproducibility of morphometrics research. To allow ALPACA to be used out‐of‐the‐box by users with no prior programming experience, we implemented it as a SlicerMorph module. SlicerMorph is an extension that enables geometric morphometrics data collection and 3D specimen analysis within the open‐source 3D Slicer biomedical visualization ecosystem. We expect that convenient access to this platform will make ALPACA broadly applicable within ecology and evolution.

Landmark-free, parametric hypothesis tests regarding two-dimensional contour shapes using coherent point drift registration and statistical parametric mapping

This paper proposes a computational framework for automated, landmark-free hypothesis testing of 2D contour shapes (i.e., shape outlines), and implements one realization of that framework. The proposed framework consists of point set registration, point correspondence determination, and parametric full-shape hypothesis testing. The results are calculated quickly (<2 s), yield morphologically rich detail in an easy-to-understand visualization, and are complimented by parametrically (or nonparametrically) calculated probability values. These probability values represent the likelihood that, in the absence of a true shape effect, smooth, random Gaussian shape changes would yield an effect as large as the observed one. This proposed framework nevertheless possesses a number of limitations, including sensitivity to algorithm parameters. As a number of algorithms and algorithm parameters could be substituted at each stage in the proposed data processing chain, sensitivity analysis would be necessary for robust statistical conclusions. In this paper, the proposed technique is applied to nine public datasets using a two-sample design, and an ANCOVA design is then applied to a synthetic dataset to demonstrate how the proposed method generalizes to the family of classical hypothesis tests. Extension to the analysis of 3D shapes is discussed.

Selection for increased tibia length in mice alters skull shape through parallel changes in developmental mechanisms

Bones in the vertebrate cranial base and limb skeleton grow by endochondral ossification, under the control of growth plates. Mechanisms of endochondral ossification are conserved across growth plates, which increases covariation in size and shape among bones, and in turn may lead to correlated changes in skeletal traits not under direct selection. We used micro-CT and geometric morphometrics to characterize shape changes in the cranium of the Longshanks mouse, which was selectively bred for longer tibiae. We show that Longshanks skulls became longer, flatter, and narrower in a stepwise process. Moreover, we show that these morphological changes likely resulted from developmental changes in the growth plates of the Longshanks cranial base, mirroring changes observed in its tibia. Thus, indirect and non-adaptive morphological changes can occur due to developmental overlap among distant skeletal elements, with important implications for interpreting the evolutionary history of vertebrate skeletal form.

Wnt Signaling Drives Correlated Changes in Facial Morphology and Brain Shape

Canonical Wnt signaling plays multiple roles critical to normal craniofacial development while its dysregulation is known to be involved in structural birth defects of the face. However, when and how Wnt signaling influences phenotypic variation, including those associated with disease, remains unclear. One potential mechanism is via Wnt signaling’s role in the patterning of an early facial signaling center, the frontonasal ectodermal zone (FEZ), and its subsequent regulation of early facial morphogenesis. For example, Wnt signaling may directly alter the shape and/or magnitude of expression of the sonic hedgehog (SHH) domain in the FEZ. To test this idea, we used a replication-competent avian sarcoma retrovirus (RCAS) encoding Wnt3a to modulate its expression in the facial mesenchyme. We then quantified and compared ontogenetic changes in treated to untreated embryos in the three-dimensional (3D) shape of both the SHH expression domain of the FEZ, and the morphology of the facial primordia and brain using iodine-contrast microcomputed tomography imaging and 3D geometric morphometrics (3DGM). We found that increased Wnt3a expression in early stages of head development produces correlated variation in shape between both structural and signaling levels of analysis. In addition, altered Wnt3a activation disrupted the integration between the forebrain and other neural tube derivatives. These results show that activation of Wnt signaling influences facial shape through its impact on the forebrain and SHH expression in the FEZ, and highlights the close relationship between morphogenesis of the forebrain and midface.

A landmark-free morphometrics pipeline for high-resolution phenotyping: application to a mouse model of Down syndrome

Characterising phenotypes often requires quantification of anatomical shape. Quantitative shape comparison (morphometrics) traditionally uses manually located landmarks and is limited by landmark number and operator accuracy. Here, we apply a landmark-free method to characterise the craniofacial skeletal phenotype of the Dp1Tyb mouse model of Down syndrome and a population of the Diversity Outbred (DO) mouse model, comparing it with a landmark-based approach. We identified cranial dysmorphologies in Dp1Tyb mice, especially smaller size and brachycephaly (front-back shortening), homologous to the human phenotype. Shape variation in the DO mice was partly attributable to allometry (size-dependent shape variation) and sexual dimorphism. The landmark-free method performed as well as, or better than, the landmark-based method but was less labour-intensive, required less user training and, uniquely, enabled fine mapping of local differences as planar expansion or shrinkage. Its higher resolution pinpointed reductions in interior mid-snout structures and occipital bones in both the models that were not otherwise apparent. We propose that this landmark-free pipeline could make morphometrics widely accessible beyond its traditional niches in zoology and palaeontology, especially in characterising developmental mutant phenotypes.

Bayesian Estimation of Geometric Morphometric Landmarks for Simultaneous Localization of Multiple Anatomies in Cardiac CT Images

We propose a robust method to simultaneously localize multiple objects in cardiac computed tomography angiography (CTA) images. The relative prior distributions of the multiple objects in the three-dimensional (3D) space can be obtained through integrating the geometric morphological relationship of each target object to some reference objects. In cardiac CTA images, the cross-sections of ascending and descending aorta can play the role of the reference objects. We employed the maximum a posteriori (MAP) estimator that utilizes anatomic prior knowledge to address this problem of localizing multiple objects. We propose a new feature for each pixel using the relative distances, which can define any objects that have unclear boundaries. Our experimental results targeting four pulmonary veins (PVs) and the left atrial appendage (LAA) in cardiac CTA images demonstrate the robustness of the proposed method. The method could also be extended to localize other multiple objects in different applications.