An optimal transportation approach for nuclear structure-based pathology

A novel statistical methodology for quantifying the spatial arrangements of axons in peripheral nerves

A thorough understanding of the neuroanatomy of peripheral nerves is required for a better insight into their function and the development of neuromodulation tools and strategies. In biophysical modeling, it is commonly assumed that the complex spatial arrangement of myelinated and unmyelinated axons in peripheral nerves is random, however, in reality the axonal organization is inhomogeneous and anisotropic. Present quantitative neuroanatomy methods analyze peripheral nerves in terms of the number of axons and the morphometric characteristics of the axons, such as area and diameter. In this study, we employed spatial statistics and point process models to describe the spatial arrangement of axons and Sinkhorn distances to compute the similarities between these arrangements (in terms of first- and second-order statistics) in various vagus and pelvic nerve cross-sections. We utilized high-resolution transmission electron microscopy (TEM) images that have been segmented using a custom-built high-throughput deep learning system based on a highly modified U-Net architecture. Our findings show a novel and innovative approach to quantifying similarities between spatial point patterns using metrics derived from the solution to the optimal transport problem. We also present a generalizable pipeline for quantitative analysis of peripheral nerve architecture. Our data demonstrate differences between male- and female-originating samples and similarities between the pelvic and abdominal vagus nerves.

On a Linear Gromov-Wasserstein Distance

Gromov-Wasserstein distances are generalization of Wasserstein distances, which are invariant under distance preserving transformations. Although a simplified version of optimal transport in Wasserstein spaces, called linear optimal transport (LOT), was successfully used in practice, there does not exist a notion of linear Gromov-Wasserstein distances so far. In this paper, we propose a definition of linear Gromov-Wasserstein distances. We motivate our approach by a generalized LOT model, which is based on barycentric projection maps of transport plans. Numerical examples illustrate that the linear Gromov-Wasserstein distances, similarly as LOT, can replace the expensive computation of pairwise Gromov-Wasserstein distances in applications like shape classification.

The Diagnostic Efficiency of Ultrasound Computer-Aided Diagnosis in Differentiating Thyroid Nodules: A Systematic Review and Narrative Synthesis

Computer-aided diagnosis (CAD) techniques have emerged to complement qualitative assessment in the diagnosis of benign and malignant thyroid nodules. The aim of this review was to summarize the current evidence on the diagnostic performance of various ultrasound CAD in characterizing thyroid nodules. PUBMED, EMBASE and Cochrane databases were searched for studies published until August 2019. The Quality Assessment of Studies of Diagnostic Accuracy included in Systematic Review 2 (QUADAS-2) tool was used to assess the methodological quality of the studies. Reported diagnostic performance data were analyzed and discussed. Fourteen studies with 2232 patients and 2675 thyroid nodules met the inclusion criteria. The study quality based on QUADAS-2 assessment was moderate. At best performance, grey scale CAD had a sensitivity of 96.7% while Doppler CAD was 90%. Combined techniques of qualitative grey scale features and Doppler CAD assessment resulted in overall increased sensitivity (92%) and optimal specificity (85.1%). The experience of the CAD user, nodule size and the thyroid malignancy risk stratification system used for interpretation were the main potential factors affecting diagnostic performance outcomes. The diagnostic performance of CAD of thyroid ultrasound is comparable to that of qualitative visual assessment; however, combined techniques have the potential for better optimized diagnostic accuracy.

Tumor Malignancy Detection Using Histopathology Imaging

Image segmentation and classification in the biomedical imaging field has high worth in cancer diagnosis and grading. The proposed method classifies the images based on a combination of handcrafted features and shape features using bag of visual words (BoW). The multistage segmentation technique to localize the nuclei in histopathology images includes the stain decomposition and histogram equalization to highlight the nucleus region, followed by the nuclei key point extraction using fast radial symmetry transform, normalized graph cut based on the nuclei region estimation, and nuclei boundary estimation using modified gradient. Subsequently, features from localized regions termed as handcrafted features and the shape features using BoW are extracted for classification. The experiments are performed using both the handcrafted features and BoW to take the advantages of both local nuclei features and globally spatial features. The simulation is performed on the Bisque and BreakHis data sets (with corresponding average accuracies of 93.87% and 96.96%, respectively) and confirms better diagnosis performance using the proposed method.

Pixel-to-Pixel Learning With Weak Supervision for Single-Stage Nucleus Recognition in Ki67 Images

OBJECTIVE

Nucleus recognition is a critical yet challenging step in histopathology image analysis, for example, in Ki67 immunohistochemistry stained images. Although many automated methods have been proposed, most use a multi-stage processing pipeline to categorize nuclei, leading to cumbersome, low-throughput, and error-prone assessments. To address this issue, we propose a novel deep fully convolutional network for single-stage nucleus recognition.

METHODS

Instead of conducting direct pixel-wise classification, we formulate nucleus identification as a deep structured regression model. For each input image, it produces multiple proximity maps, each of which corresponds to one nucleus category and exhibits strong responses in central regions of the nuclei. In addition, by taking into consideration the nucleus distribution in histopathology images, we further introduce an auxiliary task, region of interest (ROI) extraction, to assist and boost the nucleus quantification with weak ROI annotation. The proposed network can be learned in an end-to-end, pixel-to-pixel manner for simultaneous nucleus detection and classification.

RESULTS

We have evaluated this network on a pancreatic neuroendocrine tumor Ki67 image dataset, and the experiments demonstrate that our method outperforms recent state-of-the-art approaches.

CONCLUSION

We present a new, pixel-to-pixel deep neural network with two sibling branches for effective nucleus recognition and observe that learning with another relevant task, ROI extraction, can further boost individual nucleus localization and classification.

SIGNIFICANCE

Our method provides a clean, single-stage nucleus recognition pipeline for histopathology image analysis, especially a new perspective for Ki67 image quantification, which would potentially benefit individual object quantification in whole-slide images.

Discovery and visualization of structural biomarkers from MRI using transport-based morphometry

Disease in the brain is often associated with subtle, spatially diffuse, or complex tissue changes that may lie beneath the level of gross visual inspection, even on magnetic resonance imaging (MRI). Unfortunately, current computer-assisted approaches that examine pre-specified features, whether anatomically-defined (i.e. thalamic volume, cortical thickness) or based on pixelwise comparison (i.e. deformation-based methods), are prone to missing a vast array of physical changes that are not well-encapsulated by these metrics. In this paper, we have developed a technique for automated pattern analysis that can fully determine the relationship between brain structure and observable phenotype without requiring any a priori features. Our technique, called transport-based morphometry (TBM), is an image transformation that maps brain images losslessly to a domain where they become much more separable. The new approach is validated on structural brain images of healthy older adult subjects where even linear models for discrimination, regression, and blind source separation enable TBM to independently discover the characteristic changes of aging and highlight potential mechanisms by which aerobic fitness may mediate brain health later in life. TBM is a generative approach that can provide visualization of physically meaningful shifts in tissue distribution through inverse transformation. The proposed framework is a powerful technique that can potentially elucidate genotype-structural-behavioral associations in myriad diseases.

Chromatin changes predict recurrence after radical prostatectomy

Background:

Pathological evaluations give the best prognostic markers for prostate cancer patients after radical prostatectomy, but the observer variance is substantial. These risk assessments should be supported and supplemented by objective methods for identifying patients at increased risk of recurrence. Markers of epigenetic aberrations have shown promising results in several cancer types and can be assessed by automatic analysis of chromatin organisation in tumour cell nuclei.

Methods:

A consecutive series of 317 prostate cancer patients treated with radical prostatectomy at a national hospital between 1987 and 2005 were followed for a median of 10 years (interquartile range, 7–14). On average three tumour block samples from each patient were included to account for tumour heterogeneity. We developed a novel marker, termed Nucleotyping, based on automatic assessment of disordered chromatin organisation, and validated its ability to predict recurrence after radical prostatectomy.

Results:

Nucleotyping predicted recurrence with a hazard ratio (HR) of 3.3 (95% confidence interval (CI), 2.1–5.1). With adjustment for clinical and pathological characteristics, the HR was 2.5 (95% CI, 1.5–4.1). An updated stratification into three risk groups significantly improved the concordance with patient outcome compared with a state-of-the-art risk-stratification tool (P<0.001). The prognostic impact was most evident for the patients who were high-risk by clinical and pathological characteristics and for patients with Gleason score 7.

Conclusion:

A novel assessment of epigenetic aberrations was capable of improving risk stratification after radical prostatectomy.

A continuous linear optimal transport approach for pattern analysis in image datasets

We present a new approach to facilitate the application of the optimal transport metric to pattern recognition on image databases. The method is based on a linearized version of the optimal transport metric, which provides a linear embedding for the images. Hence, it enables shape and appearance modeling using linear geometric analysis techniques in the embedded space. In contrast to previous work, we use Monge's formulation of the optimal transport problem, which allows for reasonably fast computation of the linearized optimal transport embedding for large images. We demonstrate the application of the method to recover and visualize meaningful variations in a supervised-learning setting on several image datasets, including chromatin distribution in the nuclei of cells, galaxy morphologies, facial expressions, and bird species identification. We show that the new approach allows for high-resolution construction of modes of variations and discrimination and can enhance classification accuracy in a variety of image discrimination problems.

The Radon Cumulative Distribution Transform and Its Application to Image Classification

Invertible image representation methods (transforms) are routinely employed as low-level image processing operations based on which feature extraction and recognition algorithms are developed. Most transforms in current use (e.g., Fourier, wavelet, and so on) are linear transforms and, by themselves, are unable to substantially simplify the representation of image classes for classification. Here, we describe a nonlinear, invertible, low-level image processing transform based on combining the well-known Radon transform for image data, and the 1D cumulative distribution transform proposed earlier. We describe a few of the properties of this new transform, and with both theoretical and experimental results show that it can often render certain problems linearly separable in a transform space.

New methods for quantifying and visualizing information from images of cells: An overview

New microscopy imaging techniques have enabled the acquisition of cellular and sub-cellular information with unprecedented accuracy and specificity. Fluorescence techniques have enabled labeling of numerous, previously inaccessible, molecules and organelles, while Raman spectrographic techniques, for example, have enabled label free acquisition. Together with the development of high throughput techniques, these technologies now allow for the acquisition of a significant amount of information about cellular processes and have enabled high throughput and high content screening. Beyond image formation and acquisition, computational techniques comprise an important part of the process of obtaining biological understanding from such experiments. Here we review the pros and cons of the main approaches that have been used to extract information from digital images of cells. In addition, we also offer an overview of modern computational techniques that beyond allowing for discrimination between two hypothesis, also allow for modeling, visualization, and understanding of biological phenomena.

Cancer diagnosis by nuclear morphometry using spatial information

Methods for extracting quantitative information regarding nuclear morphology from histopathology images have been long used to aid pathologists in determining the degree of differentiation in numerous malignancies. Most methods currently in use, however, employ the naïve Bayes approach to classify a set of nuclear measurements extracted from one patient. Hence, the statistical dependency between the samples (nuclear measurements) is often not directly taken into account. Here we describe a method that makes use of statistical dependency between samples in thyroid tissue to improve patient classification accuracies with respect to standard naïve Bayes approaches. We report results in two sample diagnostic challenges.

Accurate diagnosis of thyroid follicular lesions from nuclear morphology using supervised learning

Follicular lesions of the thyroid remain significant diagnostic challenges in surgical pathology and cytology. The diagnosis often requires considerable resources and ancillary tests including immunohistochemistry, molecular studies, and expert consultation. Visual analyses of nuclear morphological features, generally speaking, have not been helpful in distinguishing this group of lesions. Here we describe a method for distinguishing between follicular lesions of the thyroid based on nuclear morphology. The method utilizes an optimal transport-based linear embedding for segmented nuclei, together with an adaptation of existing classification methods. We show the method outputs assignments (classification results) which are near perfectly correlated with the clinical diagnosis of several lesion types' lesions utilizing a database of 94 patients in total. Experimental comparisons also show the new method can significantly outperform standard numerical feature-type methods in terms of agreement with the clinical diagnosis gold standard. In addition, the new method could potentially be used to derive insights into biologically meaningful nuclear morphology differences in these lesions. Our methods could be incorporated into a tool for pathologists to aid in distinguishing between follicular lesions of the thyroid. In addition, these results could potentially provide nuclear morphological correlates of biological behavior and reduce health care costs by decreasing histotechnician and pathologist time and obviating the need for ancillary testing.

Interpolation of Longitudinal Shape and Image Data via Optimal Mass Transport

Longitudinal analysis of medical imaging data has become central to the study of many disorders. Unfortunately, various constraints (study design, patient availability, technological limitations) restrict the acquisition of data to only a few time points, limiting the study of continuous disease/treatment progression. Having the ability to produce a sensible time interpolation of the data can lead to improved analysis, such as intuitive visualizations of anatomical changes, or the creation of more samples to improve statistical analysis. In this work, we model interpolation of medical image data, in particular shape data, using the theory of optimal mass transport (OMT), which can construct a continuous transition from two time points while preserving "mass" (e.g., image intensity, shape volume) during the transition. The theory even allows a short extrapolation in time and may help predict short-term treatment impact or disease progression on anatomical structure. We apply the proposed method to the hippocampus-amygdala complex in schizophrenia, the heart in atrial fibrillation, and full head MR images in traumatic brain injury.

Detecting and visualizing cell phenotype differences from microscopy images using transport-based morphometry

Modern microscopic imaging devices are able to extract more information regarding the subcellular organization of different molecules and proteins than can be obtained by visual inspection. Predetermined numerical features (descriptors) often used to quantify cells extracted from these images have long been shown useful for discriminating cell populations (e.g., normal vs. diseased). Direct visual or biological interpretation of results obtained, however, is often not a trivial task. We describe an approach for detecting and visualizing phenotypic differences between classes of cells based on the theory of optimal mass transport. The method is completely automated, does not require the use of predefined numerical features, and at the same time allows for easily interpretable visualizations of the most significant differences. Using this method, we demonstrate that the distribution pattern of peripheral chromatin in the nuclei of cells extracted from liver and thyroid specimens is associated with malignancy. We also show the method can correctly recover biologically interpretable and statistically significant differences in translocation imaging assays in a completely automated fashion.

Machine-based morphologic analysis of glioblastoma using whole-slide pathology images uncovers clinically relevant molecular correlates

Pathologic review of tumor morphology in histologic sections is the traditional method for cancer classification and grading, yet human review has limitations that can result in low reproducibility and inter-observer agreement. Computerized image analysis can partially overcome these shortcomings due to its capacity to quantitatively and reproducibly measure histologic structures on a large-scale. In this paper, we present an end-to-end image analysis and data integration pipeline for large-scale morphologic analysis of pathology images and demonstrate the ability to correlate phenotypic groups with molecular data and clinical outcomes. We demonstrate our method in the context of glioblastoma (GBM), with specific focus on the degree of the oligodendroglioma component. Over 200 million nuclei in digitized pathology slides from 117 GBMs in the Cancer Genome Atlas were quantitatively analyzed, followed by multiplatform correlation of nuclear features with molecular and clinical data. For each nucleus, a Nuclear Score (NS) was calculated based on the degree of oligodendroglioma appearance, using a regression model trained from the optimal feature set. Using the frequencies of neoplastic nuclei in low and high NS intervals, we were able to cluster patients into three well-separated disease groups that contained low, medium, or high Oligodendroglioma Component (OC). We showed that machine-based classification of GBMs with high oligodendroglioma component uncovered a set of tumors with strong associations with PDGFRA amplification, proneural transcriptional class, and expression of the oligodendrocyte signature genes MBP, HOXD1, PLP1, MOBP and PDGFRA. Quantitative morphologic features within the GBMs that correlated most strongly with oligodendrocyte gene expression were high nuclear circularity and low eccentricity. These findings highlight the potential of high throughput morphologic analysis to complement and inform human-based pathologic review.

A linear optimal transportation framework for quantifying and visualizing variations in sets of images

Transportation-based metrics for comparing images have long been applied to analyze images, especially where one can interpret the pixel intensities (or derived quantities) as a distribution of 'mass' that can be transported without strict geometric constraints. Here we describe a new transportation-based framework for analyzing sets of images. More specifically, we describe a new transportation-related distance between pairs of images, which we denote as linear optimal transportation (LOT). The LOT can be used directly on pixel intensities, and is based on a linearized version of the Kantorovich-Wasserstein metric (an optimal transportation distance, as is the earth mover's distance). The new framework is especially well suited for computing all pairwise distances for a large database of images efficiently, and thus it can be used for pattern recognition in sets of images. In addition, the new LOT framework also allows for an isometric linear embedding, greatly facilitating the ability to visualize discriminant information in different classes of images. We demonstrate the application of the framework to several tasks such as discriminating nuclear chromatin patterns in cancer cells, decoding differences in facial expressions, galaxy morphologies, as well as sub cellular protein distributions.

Integrative, multimodal analysis of glioblastoma using TCGA molecular data, pathology images, and clinical outcomes

Multimodal, multiscale data synthesis is becoming increasingly critical for successful translational biomedical research. In this letter, we present a large-scale investigative initiative on glioblastoma, a high-grade brain tumor, with complementary data types using in silico approaches. We integrate and analyze data from The Cancer Genome Atlas Project on glioblastoma that includes novel nuclear phenotypic data derived from microscopic slides, genotypic signatures described by transcriptional class and genetic alterations, and clinical outcomes defined by response to therapy and patient survival. Our preliminary results demonstrate numerous clinically and biologically significant correlations across multiple data types, revealing the power of in silico multimodal data integration for cancer research.

Penalized Fisher Discriminant Analysis and Its Application to Image-Based Morphometry

Image-based morphometry is an important area of pattern recognition research, with numerous applications in science and technology (including biology and medicine). Fisher Linear Discriminant Analysis (FLDA) techniques are often employed to elucidate and visualize important information that discriminates between two or more populations. We demonstrate that the direct application of FLDA can lead to undesirable errors in characterizing such information and that the reason for such errors is not necessarily the ill conditioning in the resulting generalized eigenvalue problem, as usually assumed. We show that the regularized eigenvalue decomposition often used is related to solving a modified FLDA criterion that includes a least-squares-type representation penalty, and derive the relationship explicitly. We demonstrate the concepts by applying this modified technique to several problems in image-based morphometry, and build discriminant representative models for different data sets.

Computational image analysis of nuclear morphology associated with various nuclear-specific aging disorders

Computational image analysis is used in many areas of biological and medical research, but advanced techniques including machine learning remain underutilized. Here, we used automated segmentation and shape analyses, with pre-defined features and with computer generated components, to compare nuclei from various premature aging disorders caused by alterations in nuclear proteins. We considered cells from patients with Hutchinson-Gilford progeria syndrome (HGPS) with an altered nucleoskeletal protein; a mouse model of XFE progeroid syndrome caused by a deficiency of ERCC1-XPF DNA repair nuclease; and patients with Werner syndrome (WS) lacking a functional WRN exonuclease and helicase protein. Using feature space analysis, including circularity, eccentricity, and solidity, we found that XFE nuclei were larger and significantly more elongated than control nuclei. HGPS nuclei were smaller and rounder than the control nuclei with features suggesting small bumps. WS nuclei did not show any significant shape changes from control. We also performed principle component analysis (PCA) and a geometric, contour based metric. PCA allowed direct visualization of morphological changes in diseased nuclei, whereas standard, feature-based approaches required pre-defined parameters and indirect interpretation of multiple parameters. Both methods yielded similar results, but PCA proves to be a powerful pre-analysis methodology for unknown systems.