GestAltNet: aggregation and attention to improve deep learning of gestational age from placental whole-slide images

Cross-modal contrastive learning for unified placenta analysis using photographs

The placenta is vital to maternal and child health but often overlooked in pregnancy studies. Addressing the need for a more accessible and cost-effective method of placental assessment, our study introduces a computational tool designed for the analysis of placental photographs. Leveraging images and pathology reports collected from sites in the United States and Uganda over a 12-year period, we developed a cross-modal contrastive learning algorithm consisting of pre-alignment, distillation, and retrieval modules. Moreover, the proposed robustness evaluation protocol enables statistical assessment of performance improvements, provides deeper insight into the impact of different features on predictions, and offers practical guidance for its application in a variety of settings. Through extensive experimentation, our tool demonstrates an average area under the receiver operating characteristic curve score of over 82% in both internal and external validations, which underscores the potential of our tool to enhance clinical care across diverse environments.

An interpretable deep learning model for detecting BRCA pathogenic variants of breast cancer from hematoxylin and eosin-stained pathological images

Background

Determining the status of breast cancer susceptibility genes (BRCA) is crucial for guiding breast cancer treatment. Nevertheless, the need for BRCA genetic testing among breast cancer patients remains unmet due to high costs and limited resources. This study aimed to develop a Bi-directional Self-Attention Multiple Instance Learning (BiAMIL) algorithm to detect BRCA status from hematoxylin and eosin (H&E) pathological images.

Methods

A total of 319 histopathological slides from 254 breast cancer patients were included, comprising two dependent cohorts. Following image pre-processing, 633,484 tumor tiles from the training dataset were employed to train the self-developed deep-learning model. The performance of the network was evaluated in the internal and external test sets.

Results

BiAMIL achieved AUC values of 0.819 (95% CI [0.673-0.965]) in the internal test set, and 0.817 (95% CI [0.712-0.923]) in the external test set. To explore the relationship between BRCA status and interpretable morphological features in pathological images, we utilized Class Activation Mapping (CAM) technique and cluster analysis to investigate the connections between BRCA gene mutation status and tissue and cell features. Significantly, we observed that tumor-infiltrating lymphocytes and the morphological characteristics of tumor cells appeared to be potential features associated with BRCA status.

Conclusions

An interpretable deep neural network model based on the attention mechanism was developed to predict the BRCA status in breast cancer. Keywords: Breast cancer, BRCA, deep learning, self-attention, interpretability.

Quantitative Modeling to Characterize Maternal Inflammatory Response of Histologic Chorioamnionitis in Placental Membranes

PROBLEM

The placental membranes are a key barrier to fetal and uterine infection. Inflammation of the membranes, diagnosed as maternal inflammatory response (MIR) or alternatively as acute chorioamnionitis, is associated with adverse maternal-fetal outcomes. MIR is staged 1-3, with higher stages indicating more hazardous inflammation. However, the diagnosis relies upon subjective evaluation and has not been deeply characterized. The goal of this work is to develop a cell classifier for eight placental membrane cells and quantitatively characterize MIR1-2.

METHOD OF STUDY

Hematoxylin and eosin (H&E)-stained placental membrane slides were digitized. A convolutional neural network was trained on a dataset of hand-annotated and machine learning-identified cells. Overall cell class-level metrics were calculated. The model was applied to 20 control, 20 MIR1, and 23 MIR2 placental membrane cases. MIR cell composition and neutrophil distribution were assessed via density and Ripley's cross K-function. Clinical data were compared to neutrophil density and distribution.

RESULTS

The classification model achieved a test-set accuracy of 0.845, with high precision and recall for amniocytes, decidual cells, endothelial cells, and trophoblasts. Using this model to classify 53 073 cells from healthy and MIR1-2 placental membranes, we found that (1) MIR1-2 have higher neutrophil density and fewer decidual cells and trophoblasts, (2) Neutrophils colocalize heavily around decidual cells in healthy placental membranes and around trophoblasts in MIR1, (3) Neutrophil density impacts distribution in MIR, and (4) Neutrophil metrics correlate with features of clinical chorioamnionitis.

CONCLUSIONS

This paper introduces cell classification into the placental membranes and quantifies cell composition and neutrophil spatial distributions in MIR.

Identifying First-Trimester Risk Factors for SGA-LGA Using Weighted Inheritance Voting Ensemble Learning

The classification of fetuses as Small for Gestational Age (SGA) and Large for Gestational Age (LGA) is a critical aspect of neonatal health assessment. SGA and LGA, terms used to describe fetal weights that fall below or above the expected weights for Appropriate for Gestational Age (AGA) fetuses, indicate intrauterine growth restriction and excessive fetal growth, respectively. Early prediction and assessment of latent risk factors associated with these classifications can facilitate timely medical interventions, thereby optimizing the health outcomes for both the infant and the mother. This study aims to leverage first-trimester data to achieve these objectives. This study analyzed data from 7943 pregnant women, including 424 SGA, 928 LGA, and 6591 AGA cases, collected from 2015 to 2021 at the Third Affiliated Hospital of Sun Yat-sen University in Guangzhou, China. We propose a novel algorithm, named the Weighted Inheritance Voting Ensemble Learning Algorithm (WIVELA), to predict the classification of fetuses into SGA, LGA, and AGA categories based on biochemical parameters, maternal factors, and morbidity during pregnancy. Additionally, we proposed algorithms for relevance determination based on the classifier to ascertain the importance of features associated with SGA and LGA. The proposed classification solution demonstrated a notable average accuracy rate of 92.12% on 10-fold cross-validation over 100 loops, outperforming five state-of-the-art machine learning algorithms. Furthermore, we identified significant latent maternal risk factors directly associated with SGA and LGA conditions, such as weight change during the first trimester, prepregnancy weight, height, age, and obstetric factors like fetal growth restriction and birthing LGA baby. This study also underscored the importance of biomarker features at the end of the first trimester, including HDL, TG, OGTT-1h, OGTT-0h, OGTT-2h, TC, FPG, and LDL, which reflect the status of SGA or LGA fetuses. This study presents innovative solutions for classifying and identifying relevant attributes, offering valuable tools for medical teams in the clinical monitoring of fetuses predisposed to SGA and LGA conditions during the initial stage of pregnancy. These proposed solutions facilitate early intervention in nutritional care and prenatal healthcare, thereby contributing to enhanced strategies for managing the health and well-being of both the fetus and the expectant mother.

Clinical value of placental examination for paediatricians

The placenta contains valuable clinical information that is linked to fetal development, neonatal morbidity and mortality, and future health outcomes. Both gross inspection and histopathological examination of the placenta may identify intrinsic or secondary placental lesions, which can contribute directly to adverse neonatal outcomes or indicate the presence of an unfavourable intrauterine environment. Placental examination therefore forms an essential component of the care of high-risk neonates and at perinatal post-mortem examination. In this article, we describe the clinical value of placental examination for paediatricians and perinatal clinicians. We discuss common pathological findings on general inspection of the placenta with photographic examples and provide an overview of the placental pathological examination, including how to interpret key findings. We also address the medico-legal and financial implications of placental examinations and describe current and future clinical considerations for clinicians in regard to placental examination.

Mapping cell-to-tissue graphs across human placenta histology whole slide images using deep learning with HAPPY

Accurate placenta pathology assessment is essential for managing maternal and newborn health, but the placenta's heterogeneity and temporal variability pose challenges for histology analysis. To address this issue, we developed the 'Histology Analysis Pipeline.PY' (HAPPY), a deep learning hierarchical method for quantifying the variability of cells and micro-anatomical tissue structures across placenta histology whole slide images. HAPPY differs from patch-based features or segmentation approaches by following an interpretable biological hierarchy, representing cells and cellular communities within tissues at a single-cell resolution across whole slide images. We present a set of quantitative metrics from healthy term placentas as a baseline for future assessments of placenta health and we show how these metrics deviate in placentas with clinically significant placental infarction. HAPPY's cell and tissue predictions closely replicate those from independent clinical experts and placental biology literature.

Tissue Contamination Challenges the Credibility of Machine Learning Models in Real World Digital Pathology

Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. Although human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole-slide models. Three operate in placenta for the following functions: (1) detection of decidual arteriopathy, (2) estimation of gestational age, and (3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in the t-distributed stochastic neighbor embedding feature space. Every model showed performance degradation in response to one or more tissue contaminants. Decidual arteriopathy detection--balanced accuracy decreased from 0.74 to 0.69 ± 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant, raised the mean absolute error in estimating gestational age from 1.626 weeks to 2.371 ± 0.003 weeks. Blood, incorporated into placental sections, induced false-negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033 mm2, and resulted in a 97% false-positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.

Automated detection of microscopic placental features indicative of maternal vascular malperfusion using machine learning

INTRODUCTION

Hypertensive disorders of pregnancy (HDP) and fetal growth restriction (FGR) are common obstetrical complications, often with pathological features of maternal vascular malperfusion (MVM) in the placenta. Currently, clinical placental pathology methods involve a manual visual examination of histology sections, a practice that can be resource-intensive and demonstrates moderate-to-poor inter-pathologist agreement on diagnostic outcomes, dependant on the degree of pathologist sub-specialty training.

METHODS

This study aims to apply machine learning (ML) feature extraction methods to classify digital images of placental histopathology specimens, collected from cases of HDP [pregnancy induced hypertension (PIH), preeclampsia (PE), PE + FGR], normotensive FGR, and healthy pregnancies, according to the presence or absence of MVM lesions. 159 digital images were captured from histological placental specimens, manually scored for MVM lesions (MVM- or MVM+) and used to develop a support vector machine (SVM) classifier model, using features extracted from pre-trained ResNet18. The model was trained with data augmentation and shuffling, with the performance assessed for patch-level and image-level classification through measurements of accuracy, precision, and recall using confusion matrices.

RESULTS

The SVM model demonstrated accuracies of 70 % and 79 % for patch-level and image-level MVM classification, respectively, with poorest performance observed on images with borderline MVM presence, as determined through post hoc observation.

DISCUSSION

The results are promising for the integration of ML methods into the placental histopathological examination process. Using this study as a proof-of-concept will lead our group and others to carry ML models further in placental histopathology.

Enhancing Automatic Placenta Analysis through Distributional Feature Recomposition in Vision-Language Contrastive Learning

The placenta is a valuable organ that can aid in understanding adverse events during pregnancy and predicting issues post-birth. Manual pathological examination and report generation, however, are laborious and resource-intensive. Limitations in diagnostic accuracy and model efficiency have impeded previous attempts to automate placenta analysis. This study presents a novel framework for the automatic analysis of placenta images that aims to improve accuracy and efficiency. Building on previous vision-language contrastive learning (VLC) methods, we propose two enhancements, namely Pathology Report Feature Recomposition and Distributional Feature Recomposition, which increase representation robustness and mitigate feature suppression. In addition, we employ efficient neural networks as image encoders to achieve model compression and inference acceleration. Experiments validate that the proposed approach outperforms prior work in both performance and efficiency by significant margins. The benefits of our method, including enhanced efficacy and deployability, may have significant implications for reproductive healthcare, particularly in rural areas or low- and middle-income countries.

Comparison of Placental Pathology Reports From Spontaneous Preterm Births Finalized by General Surgical Pathologists Versus Perinatal Pathologist: A Call to Action

Placental examination, frequently performed by general surgical pathologists, plays an important role in understanding patient outcomes and explaining the underlying mechanisms leading to preterm birth (PTB). This secondary analysis of a larger study recurrent PTB aimed to compare diagnoses between general surgical pathologists (GSP) and a perinatal pathologist (PP) in preterm placentas examined between 2009 and 2018 at a single institution. Pathology diagnoses were coded into 4 categories (acute inflammation [AI], chronic inflammation, fetal vascular malperfusion, maternal vascular malperfusion) based on original reports for the GSP and second review by the single PP. A total of 331 placentas were included, representing placentas finalized by 17 GSPs. The prevalence of all 4 placental diagnostic categories was higher for the PP, and nearly half (49.2%) of placentas finalized by GSP had no diagnostic findings. Agreement was highest for AI at κ=0.50 (weak agreement). However, there was no agreement for maternal vascular malperfusion (κ=0.063), chronic inflammation (κ=0.0026), and fetal vascular malperfusion (κ = -0.018). Chronic basal deciduitis with plasma cells had the highest false-negative rate (missed in 107 cases by GSP). Villous infarction had the highest false-positive rate (overcalled in 28/41 [68%] cases) with the majority of the "infarcts" representing intervillous thrombi. In conclusion, there is no agreement between GSP and PP when assessing placental pathology other than AI, and weak agreement even for AI. These findings are a call to action to implement educational efforts and structural/organizational changes to improve consistency of placental pathology reporting.

Efficient diagnosis of IDH-mutant gliomas: 1p/19qNET assesses 1p/19q codeletion status using weakly-supervised learning

Accurate identification of molecular alterations in gliomas is crucial for their diagnosis and treatment. Although, fluorescence in situ hybridization (FISH) allows for the observation of diverse and heterogeneous alterations, it is inherently time-consuming and challenging due to the limitations of the molecular method. Here, we report the development of 1p/19qNET, an advanced deep-learning network designed to predict fold change values of 1p and 19q chromosomes and classify isocitrate dehydrogenase (IDH)-mutant gliomas from whole-slide images. We trained 1p/19qNET on next-generation sequencing data from a discovery set (DS) of 288 patients and utilized a weakly-supervised approach with slide-level labels to reduce bias and workload. We then performed validation on an independent validation set (IVS) comprising 385 samples from The Cancer Genome Atlas, a comprehensive cancer genomics resource. 1p/19qNET outperformed traditional FISH, achieving R2 values of 0.589 and 0.547 for the 1p and 19q arms, respectively. As an IDH-mutant glioma classifier, 1p/19qNET attained AUCs of 0.930 and 0.837 in the DS and IVS, respectively. The weakly-supervised nature of 1p/19qNET provides explainable heatmaps for the results. This study demonstrates the successful use of deep learning for precise determination of 1p/19q codeletion status and classification of IDH-mutant gliomas as astrocytoma or oligodendroglioma. 1p/19qNET offers comparable results to FISH and provides informative spatial information. This approach has broader applications in tumor classification.

A Patch-Based Deep Learning Approach for Detecting Rib Fractures on Frontal Radiographs in Young Children

Chest radiography is the modality of choice for the identification of rib fractures in young children and there is value for the development of computer-aided rib fracture detection in this age group. However, the automated identification of rib fractures on chest radiographs can be challenging due to the need for high spatial resolution in deep learning frameworks. A patch-based deep learning algorithm was developed to automatically detect rib fractures on frontal chest radiographs in children under 2 years old. A total of 845 chest radiographs of children 0-2 years old (median: 4 months old) were manually segmented for rib fractures by radiologists and served as the ground-truth labels. Image analysis utilized a patch-based sliding-window technique, to meet the high-resolution requirements for fracture detection. Standard transfer learning techniques used ResNet-50 and ResNet-18 architectures. Area-under-curve for precision-recall (AUC-PR) and receiver-operating-characteristic (AUC-ROC), along with patch and whole-image classification metrics, were reported. On the test patches, the ResNet-50 model showed AUC-PR and AUC-ROC of 0.25 and 0.77, respectively, and the ResNet-18 showed an AUC-PR of 0.32 and AUC-ROC of 0.76. On the whole-radiograph level, the ResNet-50 had an AUC-ROC of 0.74 with 88% sensitivity and 43% specificity in identifying rib fractures, and the ResNet-18 had an AUC-ROC of 0.75 with 75% sensitivity and 60% specificity in identifying rib fractures. This work demonstrates the utility of patch-based analysis for detection of rib fractures in children under 2 years old. Future work with large cohorts of multi-institutional data will improve the generalizability of these findings to patients with suspicion of child abuse.

Tissue contamination challenges the credibility of machine learning models in real world digital pathology

Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. While human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole slide models. Three operate in placenta for 1) detection of decidual arteriopathy (DA), 2) estimation of gestational age (GA), and 3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in T-distributed Stochastic Neighbor Embedding (tSNE) feature space. Every model showed performance degradation in response to one or more tissue contaminants. DA detection balanced accuracy decreased from 0.74 to 0.69 +/- 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant raised the mean absolute error in estimating gestation age from 1.626 weeks to 2.371 +/ 0.003 weeks. Blood, incorporated into placental sections, induced false negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033mm2, resulted in a 97% false positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.

Machine learning classification of placental villous infarction, perivillous fibrin deposition, and intervillous thrombus

INTRODUCTION

Placental parenchymal lesions are commonly encountered and carry significant clinical associations. However, they are frequently missed or misclassified by general practice pathologists. Interpretation of pathology slides has emerged as one of the most successful applications of machine learning (ML) in medicine with applications ranging from cancer detection and prognostication to transplant medicine. The goal of this study was to use a whole-slide learning model to identify and classify placental parenchymal lesions including villous infarctions, intervillous thrombi (IVT), and perivillous fibrin deposition (PVFD).

METHODS

We generated whole slide images from placental discs examined at our institution with infarct, IVT, PVFD, or no macroscopic lesion. Slides were analyzed as a set of overlapping patches. We extracted feature vectors from each patch using a pretrained convolutional neural network (EfficientNetV2L). We trained a model to assign attention to each vector and used the attentions as weights to produce a pooled feature vector. The pooled vector was classified as normal or 1 of 3 lesions using a fully connected network. Patch attention was plotted to highlight informative areas of the slide.

RESULTS

Overall balanced accuracy in a test set of held-out slides was 0.86 with receiver-operator characteristic areas under the curve of 0.917-0.993. Cases of PVFD were frequently miscalled as normal or infarcts, the latter possibly due to the perivillous fibrin found at the periphery of infarctions. We used attention maps to further understand some errors, including one most likely due to poor tissue fixation and processing.

DISCUSSION

We used a whole-slide learning paradigm to train models to recognize three of the most common placental parenchymal lesions. We used attention maps to gain insight into model function, which differed from intuitive explanations.

Automated segmentation and morphological characterization of placental intervillous space based on a single labeled image

In this study, a novel method of data augmentation has been presented for the segmentation of placental histological images when the labeled data are scarce. This method generates new realizations of the placenta intervillous morphology while maintaining the general textures and orientations. As a result, a diversified artificial dataset of images is generated that can be used for training deep learning segmentation models. We have observed that on average the presented method of data augmentation led to a 42% decrease in the binary cross-entropy loss of the validation dataset compared to the common approach in the literature. Additionally, the morphology of the intervillous space is studied under the effect of the proposed image reconstruction technique, and the diversity of the artificially generated population is quantified. We have demonstrated that the proposed method results in a more accurate morphological characterization of the placental intervillous space with an average feature relative error of 6.5%, which is significantly lower than the 11.5% error observed with conventional augmentation techniques. Due to the high resemblance of the generated images to the real ones, applications of the proposed method may not be limited to placental histological images, and it is recommended that other types of tissue be investigated in future studies.

Application of Digital Imaging and Artificial Intelligence to Pathology of the Placenta

Digital imaging, including the use of artificial intelligence, has been increasingly applied to investigate the placenta and its related pathology. However, there has been no comprehensive review of this body of work to date. The aim of this study was to therefore review the literature regarding digital pathology of the placenta. A systematic literature search was conducted in several electronic databases. Studies involving the application of digital imaging and artificial intelligence techniques to human placental samples were retrieved and analyzed. Relevant articles were categorized by digital image technique and their relevance to studying normal and diseased placenta. Of 2008 retrieved articles, 279 were included. Digital imaging research related to the placenta was often coupled with immunohistochemistry, confocal microscopy, 3D reconstruction, and/or deep learning algorithms. By significantly increasing pathologists' ability to recognize potentially prognostic relevant features and by lessening inter-observer variability, published data overall indicate that the application of digital pathology to placental and perinatal diseases, along with clinical and radiology correlation, has great potential to improve fetal and maternal health care including the selection of targeted therapy in high-risk pregnancy.

Artificial Intelligence in Pediatric Pathology: The Extinction of a Medical Profession or the Key to a Bright Future?

Artificial Intelligence (AI) has become of increasing interest over the past decade. While digital image analysis (DIA) is already being used in radiology, it is still in its infancy in pathology. One of the reasons is that large-scale digitization of glass slides has only recently become available. With the advent of digital slide scanners, that digitize glass slides into whole slide images, many labs are now in a transition phase towards digital pathology. However, only few departments worldwide are currently fully digital. Digital pathology provides the ability to annotate large datasets and train computers to develop and validate robust algorithms, similar to radiology. In this opinionated overview, we will give a brief introduction into AI in pathology, discuss the potential positive and negative implications and speculate about the future role of AI in the field of pediatric pathology.

Attention-based multiple instance learning with self-supervision to predict microsatellite instability in colorectal cancer from histology whole-slide images

Microsatellite instability (MSI) is a clinically important characteristic of colorectal cancer. Standard diagnosis of MSI is performed via genetic analyses, however these tests are not always included in routine care. Histopathology whole-slide images (WSIs) are the gold-standard for colorectal cancer diagnosis and are routinely collected. This study develops a model to predict MSI directly from WSIs. Making use of both weakly- and self-supervised deep learning techniques, the proposed model shows improved performance over conventional deep learning models. Additionally, the proposed framework allows for visual interpretation of model decisions. These results are validated in internal and external testing datasets.

Project SWAVE 2.0: An overview of the study design for multimodal placental image acquisition and alignment

Development of non-invasive and in utero placenta imaging techniques can potentially identify biomarkers of placental health. Correlative imaging using multiple multiscale modalities is particularly important to advance the understanding of placenta structure, function and their relationship. The objective of the project SWAVE 2.0 was to understand human placental structure and function and thereby identify quantifiable measures of placental health using a multimodal correlative approach. In this paper, we present a multimodal image acquisition protocol designed to acquire and align data from ex vivo placenta specimens derived from both healthy and complicated pregnancies. Qualitative and quantitative validation of the alignment method were performed. The qualitative analysis showed good correlation between findings in the MRI, ultrasound and histopathology images. The proposed protocol would enable future studies on comprehensive analysis of placental anatomy, function and their relationship.

● An overview of a novel multimodal placental image acquisition protocol is presented.

● A co-registration method using surface markers and external fiducials is described.

● A preliminary correlative imaging analysis for a placenta specimen is presented.

Training of deep learning pipelines on memory-constrained GPUs via segmented fused-tiled execution

Training models with massive inputs is a significant challenge in the development of Deep Learning pipelines to process very large digital image datasets as required by Whole Slide Imaging (WSI) in computational pathology and analysis of brain fMRI images in computational neuroscience. Graphics Processing Units (GPUs) represent the primary workhorse in training and inference of Deep Learning models. In order to use GPUs to run inference or training on a neural network pipeline, state-of-the-art machine learning frameworks like PyTorch and TensorFlow currently require that the collective memory on the GPUs must be larger than the size of the activations at any stage in the pipeline. Therefore, existing Deep Learning pipelines for these use cases have been forced to develop sub-optimal “patch-based” modeling approaches, where images are processed in small segments of an image. In this paper, we present a solution to this problem by employing tiling in conjunction with check-pointing, thereby enabling arbitrarily large images to be directly processed, irrespective of the size of global memory on a GPU and the number of available GPUs. Experimental results using PyTorch demonstrate enhanced functionality/performance over existing frameworks.

Genetic mutation and biological pathway prediction based on whole slide images in breast carcinoma using deep learning

Breast carcinoma is the most common cancer among women worldwide that consists of a heterogeneous group of subtype diseases. The whole-slide images (WSIs) can capture the cell-level heterogeneity, and are routinely used for cancer diagnosis by pathologists. However, key driver genetic mutations related to targeted therapies are identified by genomic analysis like high-throughput molecular profiling. In this study, we develop a deep-learning model to predict the genetic mutations and biological pathway activities directly from WSIs. Our study offers unique insights into WSI visual interactions between mutation and its related pathway, enabling a head-to-head comparison to reinforce our major findings. Using the histopathology images from the Genomic Data Commons Database, our model can predict the point mutations of six important genes (AUC 0.68-0.85) and copy number alteration of another six genes (AUC 0.69-0.79). Additionally, the trained models can predict the activities of three out of ten canonical pathways (AUC 0.65-0.79). Next, we visualized the weight maps of tumor tiles in WSI to understand the decision-making process of deep-learning models via a self-attention mechanism. We further validated our models on liver and lung cancers that are related to metastatic breast cancer. Our results provide insights into the association between pathological image features, molecular outcomes, and targeted therapies for breast cancer patients.