Penalized Fisher Discriminant Analysis and Its Application to Image-Based Morphometry

Discovering the gene-brain-behavior link in autism via generative machine learning

Autism is traditionally diagnosed behaviorally but has a strong genetic basis. A genetics-first approach could transform understanding and treatment of autism. However, isolating the gene-brain-behavior relationship from confounding sources of variability is a challenge. We demonstrate a novel technique, 3D transport-based morphometry (TBM), to extract the structural brain changes linked to genetic copy number variation (CNV) at the 16p11.2 region. We identified two distinct endophenotypes. In data from the Simons Variation in Individuals Project, detection of these endophenotypes enabled 89 to 95% test accuracy in predicting 16p11.2 CNV from brain images alone. Then, TBM enabled direct visualization of the endophenotypes driving accurate prediction, revealing dose-dependent brain changes among deletion and duplication carriers. These endophenotypes are sensitive to articulation disorders and explain a portion of the intelligence quotient variability. Genetic stratification combined with TBM could reveal new brain endophenotypes in many neurodevelopmental disorders, accelerating precision medicine, and understanding of human neurodiversity.

Empowering breast cancer diagnosis and radiology practice: advances in artificial intelligence for contrast-enhanced mammography

Artificial intelligence (AI) applications in breast imaging span a wide range of tasks including decision support, risk assessment, patient management, quality assessment, treatment response assessment and image enhancement. However, their integration into the clinical workflow has been slow due to the lack of a consensus on data quality, benchmarked robust implementation, and consensus-based guidelines to ensure standardization and generalization. Contrast-enhanced mammography (CEM) has improved sensitivity and specificity compared to current standards of breast cancer diagnostic imaging i.e., mammography (MG) and/or conventional ultrasound (US), with comparable accuracy to MRI (current diagnostic imaging benchmark), but at a much lower cost and higher throughput. This makes CEM an excellent tool for widespread breast lesion characterization for all women, including underserved and minority women. Underlining the critical need for early detection and accurate diagnosis of breast cancer, this review examines the limitations of conventional approaches and reveals how AI can help overcome them. The Methodical approaches, such as image processing, feature extraction, quantitative analysis, lesion classification, lesion segmentation, integration with clinical data, early detection, and screening support have been carefully analysed in recent studies addressing breast cancer detection and diagnosis. Recent guidelines described by Checklist for Artificial Intelligence in Medical Imaging (CLAIM) to establish a robust framework for rigorous evaluation and surveying has inspired the current review criteria.

Predicting Malignancy of Breast Imaging Findings Using Quantitative Analysis of Contrast-Enhanced Mammography (CEM)

We sought to develop new quantitative approaches to characterize the spatial distribution of mammographic density and contrast enhancement of suspicious contrast-enhanced mammography (CEM) findings to improve malignant vs. benign classifications of breast lesions. We retrospectively analyzed all breast lesions that underwent CEM imaging and tissue sampling at our institution from 2014–2020 in this IRB-approved study. A penalized linear discriminant analysis was used to classify lesions based on the averaged histograms of radial distributions of mammographic density and contrast enhancement. T-tests were used to compare the classification accuracies of density, contrast, and concatenated density and contrast histograms. Logistic regression and AUC-ROC analyses were used to assess if adding demographic and clinical data improved the model accuracy. A total of 159 suspicious findings were evaluated. Density histograms were more accurate in classifying lesions as malignant or benign than a random classifier (62.37% vs. 48%; p < 0.001), but the concatenated density and contrast histograms demonstrated a higher accuracy (71.25%; p < 0.001) than the density histograms alone. Including the demographic and clinical data in our models led to a higher AUC-ROC than concatenated density and contrast images (0.81 vs. 0.70; p < 0.001). In the classification of invasive vs. non-invasive malignancy, the concatenated density and contrast histograms demonstrated no significant improvement in accuracy over the density histograms alone (77.63% vs. 78.59%; p = 0.504). Our findings suggest that quantitative differences in the radial distribution of mammographic density could be used to discriminate malignant from benign breast findings; however, classification accuracy was significantly improved with the addition of contrast-enhanced imaging data from CEM. Adding patient demographic and clinical information further improved the classification accuracy.

Human vs. Machine Learning Based Detection of Facial Weakness Using Video Analysis

Background

Current EMS stroke screening tools facilitate early detection and triage, but the tools' accuracy and reliability are limited and highly variable. An automated stroke screening tool could improve stroke outcomes by facilitating more accurate prehospital diagnosis and delivery. We hypothesize that a machine learning algorithm using video analysis can detect common signs of stroke. As a proof-of-concept study, we trained a computer algorithm to detect presence and laterality of facial weakness in publically available videos with comparable accuracy, sensitivity, and specificity to paramedics.

Methods and Results

We curated videos of people with unilateral facial weakness (n = 93) and with a normal smile (n = 96) from publicly available web-based sources. Three board certified vascular neurologists categorized the videos according to the presence or absence of weakness and laterality. Three paramedics independently analyzed each video with a mean accuracy, sensitivity and specificity of 92.6% [95% CI 90.1–94.7%], 87.8% [95% CI 83.9–91.7%] and 99.3% [95% CI 98.2–100%]. Using a 5-fold cross validation scheme, we trained a computer vision algorithm to analyze the same videos producing an accuracy, sensitivity and specificity of 88.9% [95% CI 83.5–93%], 90.3% [95% CI 82.4–95.5%] and 87.5 [95% CI 79.2–93.4%].

Conclusions

These preliminary results suggest that a machine learning algorithm using computer vision analysis can detect unilateral facial weakness in pre-recorded videos with an accuracy and sensitivity comparable to trained paramedics. Further research is warranted to pursue the concept of augmented facial weakness detection and external validation of this algorithm in independent data sets and prospective patient encounters.

Video-Based Facial Weakness Analysis

OBJECTIVE

Facial weakness is a common sign of neurological diseases such as Bell's palsy and stroke. However, recognizing facial weakness still remains as a challenge, because it requires experience and neurological training.

METHODS

We propose a framework for facial weakness detection, which models the temporal dynamics of both shape and appearance-based features of each target frame through a bi-directional long short-term memory network (Bi-LSTM). The system is evaluated on a "in-the-wild"video dataset that is verified by three board-certified neurologists. In addition, three emergency medical services (EMS) personnel and three upper level residents rated the dataset. We compare the evaluation of the proposed algorithm with other comparison methods as well as the human raters.

RESULTS

Experimental evaluation demonstrates that: (1) the proposed algorithm achieves the accuracy, sensitivity, and specificity of 94.3%, 91.4%, and 95.7%, which outperforms other comparison methods and achieves the equal performance to paramedics; (2) the framework can provide visualizable and interpretable results that increases model transparency and interpretability; (3) a prototype is implemented as a proof-of-concept showcase to show the feasibility of an inexpensive solution for facial weakness detection.

CONCLUSION

The experiment results suggest that the proposed framework can identify facial weakness effectively.

SIGNIFICANCE

We provide a proof-of-concept study, showing that such technology could be used by non-neurologists to more readily identify facial weakness in the field, leading to increasing coverage and earlier treatment.

Enabling early detection of osteoarthritis from presymptomatic cartilage texture maps via transport-based learning

Many diseases have no visual cues in the early stages, eluding image-based detection. Today, osteoarthritis (OA) is detected after bone damage has occurred, at an irreversible stage of the disease. Currently no reliable method exists for OA detection at a reversible stage. We present an approach that enables sensitive OA detection in presymptomatic individuals. Our approach combines optimal mass transport theory with statistical pattern recognition. Eighty-six healthy individuals were selected from the Osteoarthritis Initiative, with no symptoms or visual signs of disease on imaging. On 3-y follow-up, a subset of these individuals had progressed to symptomatic OA. We trained a classifier to differentiate progressors and nonprogressors on baseline cartilage texture maps, which achieved a robust test accuracy of 78% in detecting future symptomatic OA progression 3 y prior to symptoms. This work demonstrates that OA detection may be possible at a potentially reversible stage. A key contribution of our work is direct visualization of the cartilage phenotype defining predictive ability as our technique is generative. We observe early biochemical patterns of fissuring in cartilage that define future onset of OA. In the future, coupling presymptomatic OA detection with emergent clinical therapies could modify the outcome of a disease that costs the United States healthcare system $16.5 billion annually. Furthermore, our technique is broadly applicable to earlier image-based detection of many diseases currently diagnosed at advanced stages today.

SetSVM: An Approach to Set Classification in Nuclei-Based Cancer Detection

Due to the importance of nuclear structure in cancer diagnosis, several predictive models have been described for diagnosing a wide variety of cancers based on nuclear morphology. In many computer-aided diagnosis (CAD) systems, cancer detection tasks can be generally formulated as set classification problems, which can not be directly solved by classifying single instances. In this paper, we propose a novel set classification approach SetSVM to build a predictive model by considering any nuclei set as a whole without specific assumptions. SetSVM features highly discriminative power in cancer detection challenges in the sense that it not only optimizes the classifier decision boundary but also transfers discriminative information to set representation learning. During model training, these two processes are unified in the support vector machine (SVM) maximum separation margin problem. Experiment results show that SetSVM provides significant improvements compared with five commonly used approaches in cancer detection tasks utilizing 260 patients in total across three different cancer types, namely, thyroid cancer, liver cancer, and melanoma. In addition, we show that SetSVM enables visual interpretation of discriminative nuclear characteristics representing the nuclei set. These features make SetSVM a potentially practical tool in building accurate and interpretable CAD systems for cancer detection.

Discovery and visualization of structural biomarkers from MRI using transport-based morphometry

Disease in the brain is often associated with subtle, spatially diffuse, or complex tissue changes that may lie beneath the level of gross visual inspection, even on magnetic resonance imaging (MRI). Unfortunately, current computer-assisted approaches that examine pre-specified features, whether anatomically-defined (i.e. thalamic volume, cortical thickness) or based on pixelwise comparison (i.e. deformation-based methods), are prone to missing a vast array of physical changes that are not well-encapsulated by these metrics. In this paper, we have developed a technique for automated pattern analysis that can fully determine the relationship between brain structure and observable phenotype without requiring any a priori features. Our technique, called transport-based morphometry (TBM), is an image transformation that maps brain images losslessly to a domain where they become much more separable. The new approach is validated on structural brain images of healthy older adult subjects where even linear models for discrimination, regression, and blind source separation enable TBM to independently discover the characteristic changes of aging and highlight potential mechanisms by which aerobic fitness may mediate brain health later in life. TBM is a generative approach that can provide visualization of physically meaningful shifts in tissue distribution through inverse transformation. The proposed framework is a powerful technique that can potentially elucidate genotype-structural-behavioral associations in myriad diseases.

Predictive Nuclear Chromatin Characteristics of Melanoma and Dysplastic Nevi

Background:

The diagnosis of malignant melanoma (MM) is among the diagnostic challenges pathologists encounter on a routine basis. Melanoma may arise in patients with preexisting dysplastic nevi (DN) and it is still the cause of 1.7% of all cancer-related deaths. Melanomas often have overlapping histological features with DN, especially those with severe dysplasia. Nucleotyping for identifying nuclear textural features can analyze nuclear DNA structure and organization. The aim of this study is to differentiate MM and DN using these methodologies.

Methods:

Dermatopathology slides diagnosed as MM and DN were retrieved. The glass slides were scanned using an Aperio ScanScopeXT at ×40 (0.25 μ/pixel). Whole slide images (WSI) were annotated for nuclei selection. Nuclear features to distinguish between MM and DN based on chromatin distributions were extracted from the WSI. The morphological characteristics for each nucleus were quantified with the optimal transport-based linear embedding in the continuous domain. Label predictions for individual cell nucleus are achieved through a modified version of linear discriminant analysis, coupled with the k-nearest neighbor classifier. Label for an unknown patient was set by the voting strategy with its pertaining cell nuclei.

Results:

Nucleotyping of 139 patient cases of melanoma (n = 67) and DN (n = 72) showed that our method had superior classification accuracy of 81.29%. This is a 6.4% gain in differentiating MM and DN, compared with numerical feature-based method. The distribution differences in nuclei morphology between MM and DN can be visualized with biological interpretation.

Conclusions:

Nucleotyping using quantitative and qualitative analyses may provide enough information for differentiating MM from DN using pixel image data. Our method to segment cell nuclei may offer a practical and inexpensive solution in aiding in the accurate diagnosis of melanoma.

A continuous linear optimal transport approach for pattern analysis in image datasets

We present a new approach to facilitate the application of the optimal transport metric to pattern recognition on image databases. The method is based on a linearized version of the optimal transport metric, which provides a linear embedding for the images. Hence, it enables shape and appearance modeling using linear geometric analysis techniques in the embedded space. In contrast to previous work, we use Monge's formulation of the optimal transport problem, which allows for reasonably fast computation of the linearized optimal transport embedding for large images. We demonstrate the application of the method to recover and visualize meaningful variations in a supervised-learning setting on several image datasets, including chromatin distribution in the nuclei of cells, galaxy morphologies, facial expressions, and bird species identification. We show that the new approach allows for high-resolution construction of modes of variations and discrimination and can enhance classification accuracy in a variety of image discrimination problems.

The Radon Cumulative Distribution Transform and Its Application to Image Classification

Invertible image representation methods (transforms) are routinely employed as low-level image processing operations based on which feature extraction and recognition algorithms are developed. Most transforms in current use (e.g., Fourier, wavelet, and so on) are linear transforms and, by themselves, are unable to substantially simplify the representation of image classes for classification. Here, we describe a nonlinear, invertible, low-level image processing transform based on combining the well-known Radon transform for image data, and the 1D cumulative distribution transform proposed earlier. We describe a few of the properties of this new transform, and with both theoretical and experimental results show that it can often render certain problems linearly separable in a transform space.

New methods for quantifying and visualizing information from images of cells: An overview

New microscopy imaging techniques have enabled the acquisition of cellular and sub-cellular information with unprecedented accuracy and specificity. Fluorescence techniques have enabled labeling of numerous, previously inaccessible, molecules and organelles, while Raman spectrographic techniques, for example, have enabled label free acquisition. Together with the development of high throughput techniques, these technologies now allow for the acquisition of a significant amount of information about cellular processes and have enabled high throughput and high content screening. Beyond image formation and acquisition, computational techniques comprise an important part of the process of obtaining biological understanding from such experiments. Here we review the pros and cons of the main approaches that have been used to extract information from digital images of cells. In addition, we also offer an overview of modern computational techniques that beyond allowing for discrimination between two hypothesis, also allow for modeling, visualization, and understanding of biological phenomena.

Detection of malignant mesothelioma using nuclear structure of mesothelial cells in effusion cytology specimens

Mesothelioma is a form of cancer generally caused from previous exposure to asbestos. Although it was considered a rare neoplasm in the past, its incidence is increasing worldwide due to extensive use of asbestos. In the current practice of medicine, the gold standard for diagnosing mesothelioma is through a pleural biopsy with subsequent histologic examination of the tissue. The diagnostic tissue should demonstrate the invasion by the tumor and is obtained through thoracoscopy or open thoracotomy, both being highly invasive surgical operations. On the other hand, thoracocentesis, which is removal of effusion fluid from the pleural space, is a far less invasive procedure that can provide material for cytological examination. In this study, we aim at detecting and classifying malignant mesothelioma based on the nuclear chromatin distribution from digital images of mesothelial cells in effusion cytology specimens. Accordingly, a computerized method is developed to determine whether a set of nuclei belonging to a patient is benign or malignant. The quantification of chromatin distribution is performed by using the optimal transport-based linear embedding for segmented nuclei in combination with the modified Fisher discriminant analysis. Classification is then performed through a k-nearest neighborhood approach and a basic voting strategy. Our experiments on 34 different human cases result in 100% accurate predictions computed with blind cross validation. Experimental comparisons also show that the new method can significantly outperform standard numerical feature-type methods in terms of agreement with the clinical diagnosis gold standard. According to our results, we conclude that nuclear structure of mesothelial cells alone may contain enough information to separate malignant mesothelioma from benign mesothelial proliferations.

Accurate diagnosis of thyroid follicular lesions from nuclear morphology using supervised learning

Follicular lesions of the thyroid remain significant diagnostic challenges in surgical pathology and cytology. The diagnosis often requires considerable resources and ancillary tests including immunohistochemistry, molecular studies, and expert consultation. Visual analyses of nuclear morphological features, generally speaking, have not been helpful in distinguishing this group of lesions. Here we describe a method for distinguishing between follicular lesions of the thyroid based on nuclear morphology. The method utilizes an optimal transport-based linear embedding for segmented nuclei, together with an adaptation of existing classification methods. We show the method outputs assignments (classification results) which are near perfectly correlated with the clinical diagnosis of several lesion types' lesions utilizing a database of 94 patients in total. Experimental comparisons also show the new method can significantly outperform standard numerical feature-type methods in terms of agreement with the clinical diagnosis gold standard. In addition, the new method could potentially be used to derive insights into biologically meaningful nuclear morphology differences in these lesions. Our methods could be incorporated into a tool for pathologists to aid in distinguishing between follicular lesions of the thyroid. In addition, these results could potentially provide nuclear morphological correlates of biological behavior and reduce health care costs by decreasing histotechnician and pathologist time and obviating the need for ancillary testing.

Detecting and visualizing cell phenotype differences from microscopy images using transport-based morphometry

Modern microscopic imaging devices are able to extract more information regarding the subcellular organization of different molecules and proteins than can be obtained by visual inspection. Predetermined numerical features (descriptors) often used to quantify cells extracted from these images have long been shown useful for discriminating cell populations (e.g., normal vs. diseased). Direct visual or biological interpretation of results obtained, however, is often not a trivial task. We describe an approach for detecting and visualizing phenotypic differences between classes of cells based on the theory of optimal mass transport. The method is completely automated, does not require the use of predefined numerical features, and at the same time allows for easily interpretable visualizations of the most significant differences. Using this method, we demonstrate that the distribution pattern of peripheral chromatin in the nuclei of cells extracted from liver and thyroid specimens is associated with malignancy. We also show the method can correctly recover biologically interpretable and statistically significant differences in translocation imaging assays in a completely automated fashion.

A linear optimal transportation framework for quantifying and visualizing variations in sets of images

Transportation-based metrics for comparing images have long been applied to analyze images, especially where one can interpret the pixel intensities (or derived quantities) as a distribution of 'mass' that can be transported without strict geometric constraints. Here we describe a new transportation-based framework for analyzing sets of images. More specifically, we describe a new transportation-related distance between pairs of images, which we denote as linear optimal transportation (LOT). The LOT can be used directly on pixel intensities, and is based on a linearized version of the Kantorovich-Wasserstein metric (an optimal transportation distance, as is the earth mover's distance). The new framework is especially well suited for computing all pairwise distances for a large database of images efficiently, and thus it can be used for pattern recognition in sets of images. In addition, the new LOT framework also allows for an isometric linear embedding, greatly facilitating the ability to visualize discriminant information in different classes of images. We demonstrate the application of the framework to several tasks such as discriminating nuclear chromatin patterns in cancer cells, decoding differences in facial expressions, galaxy morphologies, as well as sub cellular protein distributions.