Cancer Digital Slide Archive: an informatics resource to support integrated in silico analysis of TCGA pathology data

Masked hypergraph learning for weakly supervised histopathology whole slide image classification

BACKGROUND AND OBJECTIVES

Graph neural network (GNN) has been extensively used in histopathology whole slide image (WSI) analysis due to the efficiency and flexibility in modelling relationships among entities. However, most existing GNN-based WSI analysis methods only consider the pairwise correlation of patches from one single perspective (e.g. spatial affinity or embedding similarity) yet ignore the intrinsic non-pairwise relationships present in gigapixel WSI, which are likely to contribute to feature learning and downstream tasks. The objective of this study is therefore to explore the non-pairwise relationships in histopathology WSI and exploit them to guide the learning of slide-level representations for better classification performance.

METHODS

In this paper, we propose a novel Masked HyperGraph Learning (MaskHGL) framework for weakly supervised histopathology WSI classification. Compared with most GNN-based WSI classification methods, MaskHGL exploits the non-pairwise correlations between patches with hypergraph and global message passing conducted by hypergraph convolution. Concretely, multi-perspective hypergraphs are first built for each WSI, then hypergraph attention is introduced into the jointed hypergraph to propagate the non-pairwise relationships and thus yield more discriminative node representation. More importantly, a masked hypergraph reconstruction module is devised to guide the hypergraph learning which can generate more powerful robustness and generalization than the method only using hypergraph modelling. Additionally, a self-attention-based node aggregator is also applied to explore the global correlation of patches in WSI and produce the slide-level representation for classification.

RESULTS

The proposed method is evaluated on two public TCGA benchmark datasets and one in-house dataset. On the public TCGA-LUNG (1494 WSIs) and TCGA-EGFR (696 WSIs) test set, the area under receiver operating characteristic (ROC) curve (AUC) were 0.9752±0.0024 and 0.7421±0.0380, respectively. On the USTC-EGFR (754 WSIs) dataset, MaskHGL achieved significantly better performance with an AUC of 0.8745±0.0100, which surpassed the second-best state-of-the-art method SlideGraph+ 2.64%.

CONCLUSIONS

MaskHGL shows a great improvement, brought by considering the intrinsic non-pairwise relationships within WSI, in multiple downstream WSI classification tasks. In particular, the designed masked hypergraph reconstruction module promisingly alleviates the data scarcity and greatly enhances the robustness and classification ability of our MaskHGL. Notably, it has shown great potential in cancer subtyping and fine-grained lung cancer gene mutation prediction from hematoxylin and eosin (H&E) stained WSIs.

AI powered quantification of nuclear morphology in cancers enables prediction of genome instability and prognosis

While alterations in nucleus size, shape, and color are ubiquitous in cancer, comprehensive quantification of nuclear morphology across a whole-slide histologic image remains a challenge. Here, we describe the development of a pan-tissue, deep learning-based digital pathology pipeline for exhaustive nucleus detection, segmentation, and classification and the utility of this pipeline for nuclear morphologic biomarker discovery. Manually-collected nucleus annotations were used to train an object detection and segmentation model for identifying nuclei, which was deployed to segment nuclei in H&E-stained slides from the BRCA, LUAD, and PRAD TCGA cohorts. Interpretable features describing the shape, size, color, and texture of each nucleus were extracted from segmented nuclei and compared to measurements of genomic instability, gene expression, and prognosis. The nuclear segmentation and classification model trained herein performed comparably to previously reported models. Features extracted from the model revealed differences sufficient to distinguish between BRCA, LUAD, and PRAD. Furthermore, cancer cell nuclear area was associated with increased aneuploidy score and homologous recombination deficiency. In BRCA, increased fibroblast nuclear area was indicative of poor progression-free and overall survival and was associated with gene expression signatures related to extracellular matrix remodeling and anti-tumor immunity. Thus, we developed a powerful pan-tissue approach for nucleus segmentation and featurization, enabling the construction of predictive models and the identification of features linking nuclear morphology with clinically-relevant prognostic biomarkers across multiple cancer types.

Deep multiple instance learning versus conventional deep single instance learning for interpretable oral cancer detection

The current medical standard for setting an oral cancer (OC) diagnosis is histological examination of a tissue sample taken from the oral cavity. This process is time-consuming and more invasive than an alternative approach of acquiring a brush sample followed by cytological analysis. Using a microscope, skilled cytotechnologists are able to detect changes due to malignancy; however, introducing this approach into clinical routine is associated with challenges such as a lack of resources and experts. To design a trustworthy OC detection system that can assist cytotechnologists, we are interested in deep learning based methods that can reliably detect cancer, given only per-patient labels (thereby minimizing annotation bias), and also provide information regarding which cells are most relevant for the diagnosis (thereby enabling supervision and understanding). In this study, we perform a comparison of two approaches suitable for OC detection and interpretation: (i) conventional single instance learning (SIL) approach and (ii) a modern multiple instance learning (MIL) method. To facilitate systematic evaluation of the considered approaches, we, in addition to a real OC dataset with patient-level ground truth annotations, also introduce a synthetic dataset-PAP-QMNIST. This dataset shares several properties of OC data, such as image size and large and varied number of instances per bag, and may therefore act as a proxy model of a real OC dataset, while, in contrast to OC data, it offers reliable per-instance ground truth, as defined by design. PAP-QMNIST has the additional advantage of being visually interpretable for non-experts, which simplifies analysis of the behavior of methods. For both OC and PAP-QMNIST data, we evaluate performance of the methods utilizing three different neural network architectures. Our study indicates, somewhat surprisingly, that on both synthetic and real data, the performance of the SIL approach is better or equal to the performance of the MIL approach. Visual examination by cytotechnologist indicates that the methods manage to identify cells which deviate from normality, including malignant cells as well as those suspicious for dysplasia. We share the code as open source.

Current status and prospects of artificial intelligence in breast cancer pathology: convolutional neural networks to prospective Vision Transformers

Breast cancer is the most prevalent cancer among women, and its diagnosis requires the accurate identification and classification of histological features for effective patient management. Artificial intelligence, particularly through deep learning, represents the next frontier in cancer diagnosis and management. Notably, the use of convolutional neural networks and emerging Vision Transformers (ViT) has been reported to automate pathologists' tasks, including tumor detection and classification, in addition to improving the efficiency of pathology services. Deep learning applications have also been extended to the prediction of protein expression, molecular subtype, mutation status, therapeutic efficacy, and outcome prediction directly from hematoxylin and eosin-stained slides, bypassing the need for immunohistochemistry or genetic testing. This review explores the current status and prospects of deep learning in breast cancer diagnosis with a focus on whole-slide image analysis. Artificial intelligence applications are increasingly applied to many tasks in breast pathology ranging from disease diagnosis to outcome prediction, thus serving as valuable tools for assisting pathologists and supporting breast cancer management.

Deep learning-based identification of esophageal cancer subtypes through analysis of high-resolution histopathology images

Esophageal cancer (EC) remains a significant health challenge globally, with increasing incidence and high mortality rates. Despite advances in treatment, there remains a need for improved diagnostic methods and understanding of disease progression. This study addresses the significant challenges in the automatic classification of EC, particularly in distinguishing its primary subtypes: adenocarcinoma and squamous cell carcinoma, using histopathology images. Traditional histopathological diagnosis, while being the gold standard, is subject to subjectivity and human error and imposes a substantial burden on pathologists. This study proposes a binary class classification system for detecting EC subtypes in response to these challenges. The system leverages deep learning techniques and tissue-level labels for enhanced accuracy. We utilized 59 high-resolution histopathological images from The Cancer Genome Atlas (TCGA) Esophageal Carcinoma dataset (TCGA-ESCA). These images were preprocessed, segmented into patches, and analyzed using a pre-trained ResNet101 model for feature extraction. For classification, we employed five machine learning classifiers: Support Vector Classifier (SVC), Logistic Regression (LR), Decision Tree (DT), AdaBoost (AD), Random Forest (RF), and a Feed-Forward Neural Network (FFNN). The classifiers were evaluated based on their prediction accuracy on the test dataset, yielding results of 0.88 (SVC and LR), 0.64 (DT and AD), 0.82 (RF), and 0.94 (FFNN). Notably, the FFNN classifier achieved the highest Area Under the Curve (AUC) score of 0.92, indicating its superior performance, followed closely by SVC and LR, with a score of 0.87. This suggested approach holds promising potential as a decision-support tool for pathologists, particularly in regions with limited resources and expertise. The timely and precise detection of EC subtypes through this system can substantially enhance the likelihood of successful treatment, ultimately leading to reduced mortality rates in patients with this aggressive cancer.

tRigon: an R package and Shiny App for integrative (path-)omics data analysis

BACKGROUND

Pathomics facilitates automated, reproducible and precise histopathology analysis and morphological phenotyping. Similar to molecular omics, pathomics datasets are high-dimensional, but also face large outlier variability and inherent data missingness, making quick and comprehensible data analysis challenging. To facilitate pathomics data analysis and interpretation as well as support a broad implementation we developed tRigon (Toolbox foR InteGrative (path-)Omics data aNalysis), a Shiny application for fast, comprehensive and reproducible pathomics analysis.

RESULTS

tRigon is available via the CRAN repository ( https://cran.r-project.org/web/packages/tRigon ) with its source code available on GitLab ( https://git-ce.rwth-aachen.de/labooratory-ai/trigon ). The tRigon package can be installed locally and its application can be executed from the R console via the command 'tRigon::run_tRigon()'. Alternatively, the application is hosted online and can be accessed at https://labooratory.shinyapps.io/tRigon . We show fast computation of small, medium and large datasets in a low- and high-performance hardware setting, indicating broad applicability of tRigon.

CONCLUSIONS

tRigon allows researchers without coding abilities to perform exploratory feature analyses of pathomics and non-pathomics datasets on their own using a variety of hardware.

MMIR: an open-source software for the registration of multimodal histological images

BACKGROUND

Multimodal histology image registration is a process that transforms into a common coordinate system two or more images obtained from different microscopy modalities. The combination of information from various modalities can contribute to a comprehensive understanding of tissue specimens, aiding in more accurate diagnoses, and improved research insights. Multimodal image registration in histology samples presents a significant challenge due to the inherent differences in characteristics and the need for tailored optimization algorithms for each modality.

RESULTS

We developed MMIR a cloud-based system for multimodal histological image registration, which consists of three main modules: a project manager, an algorithm manager, and an image visualization system.

CONCLUSION

Our software solution aims to simplify image registration tasks with a user-friendly approach. It facilitates effective algorithm management, responsive web interfaces, supports multi-resolution images, and facilitates batch image registration. Moreover, its adaptable architecture allows for the integration of custom algorithms, ensuring that it aligns with the specific requirements of each modality combination. Beyond image registration, our software enables the conversion of segmented annotations from one modality to another.

Deep Learning Glioma Grading with the Tumor Microenvironment Analysis Protocol for Comprehensive Learning, Discovering, and Quantifying Microenvironmental Features

Gliomas are primary brain tumors that arise from neural stem cells, or glial precursors. Diagnosis of glioma is based on histological evaluation of pathological cell features and molecular markers. Gliomas are infiltrated by myeloid cells that accumulate preferentially in malignant tumors, and their abundance inversely correlates with survival, which is of interest for cancer immunotherapies. To avoid time-consuming and laborious manual examination of images, a deep learning approach for automatic multiclass classification of tumor grades was proposed. As an alternative way of investigating characteristics of brain tumor grades, we implemented a protocol for learning, discovering, and quantifying tumor microenvironment elements on our glioma dataset. Using only single-stained biopsies we derived characteristic differentiating tumor microenvironment phenotypic neighborhoods. The study was complicated by the small size of the available human leukocyte antigen stained on glioma tissue microarray dataset - 206 images of 5 classes - as well as imbalanced data distribution. This challenge was addressed by image augmentation for underrepresented classes. In practice, we considered two scenarios, a whole slide supervised learning classification, and an unsupervised cell-to-cell analysis looking for patterns of the microenvironment. In the supervised learning investigation, we evaluated 6 distinct model architectures. Experiments revealed that a DenseNet121 architecture surpasses the baseline's accuracy by a significant margin of 9% for the test set, achieving a score of 69%, increasing accuracy in discerning challenging WHO grade 2 and 3 cases. All experiments have been carried out in a cross-validation manner. The tumor microenvironment analysis suggested an important role for myeloid cells and their accumulation in the context of characterizing glioma grades. Those promising approaches can be used as an additional diagnostic tool to improve assessment during intraoperative examination or subtyping tissues for treatment selection, potentially easing the workflow of pathologists and oncologists.

Built to Last? Reproducibility and Reusability of Deep Learning Algorithms in Computational Pathology

Recent progress in computational pathology has been driven by deep learning. While code and data availability are essential to reproduce findings from preceding publications, ensuring a deep learning model's reusability is more challenging. For that, the codebase should be well-documented and easy to integrate into existing workflows and models should be robust toward noise and generalizable toward data from different sources. Strikingly, only a few computational pathology algorithms have been reused by other researchers so far, let alone employed in a clinical setting. To assess the current state of reproducibility and reusability of computational pathology algorithms, we evaluated peer-reviewed articles available in PubMed, published between January 2019 and March 2021, in 5 use cases: stain normalization; tissue type segmentation; evaluation of cell-level features; genetic alteration prediction; and inference of grading, staging, and prognostic information. We compiled criteria for data and code availability and statistical result analysis and assessed them in 160 publications. We found that only one-quarter (41 of 160 publications) made code publicly available. Among these 41 studies, three-quarters (30 of 41) analyzed their results statistically, half of them (20 of 41) released their trained model weights, and approximately a third (16 of 41) used an independent cohort for evaluation. Our review is intended for both pathologists interested in deep learning and researchers applying algorithms to computational pathology challenges. We provide a detailed overview of publications with published code in the field, list reusable data handling tools, and provide criteria for reproducibility and reusability.

Antigen presentation deficiency, mesenchymal differentiation, and resistance to immunotherapy in the murine syngeneic CT2A tumor model

Background

The GL261 and CT2A syngeneic tumor lines are frequently used as immunocompetent orthotopic mouse models of human glioblastoma (huGBM) but demonstrate distinct differences in their responses to immunotherapy.

Methods

To decipher the cell-intrinsic mechanisms that drive immunotherapy resistance in CT2A-luc and to define the aspects of human cancer biology that these lines can best model, we systematically compared their characteristics using whole exome and transcriptome sequencing, and protein analysis through immunohistochemistry, Western blot, flow cytometry, immunopeptidomics, and phosphopeptidomics.

Results

The transcriptional profiles of GL261-luc2 and CT2A-luc tumors resembled those of some huGBMs, despite neither line sharing the essential genetic or histologic features of huGBM. Both models exhibited striking hypermutation, with clonal hotspot mutations in RAS genes (Kras p.G12C in GL261-luc2 and Nras p.Q61L in CT2A-luc). CT2A-luc distinctly displayed mesenchymal differentiation, upregulated angiogenesis, and multiple defects in antigen presentation machinery (e.g. Tap1 p.Y488C and Psmb8 p.A275P mutations) and interferon response pathways (e.g. copy number losses of loci including IFN genes and reduced phosphorylation of JAK/STAT pathway members). The defect in MHC class I expression could be overcome in CT2A-luc by interferon-γ treatment, which may underlie the modest efficacy of some immunotherapy combinations. Additionally, CT2A-luc demonstrated substantial baseline secretion of the CCL-2, CCL-5, and CCL-22 chemokines, which play important roles as myeloid chemoattractants.

Conclusion

Although the clinical contexts that can be modeled by GL261 and CT2A for huGBM are limited, CT2A may be an informative model of immunotherapy resistance due to its deficits in antigen presentation machinery and interferon response pathways.

Immunohistochemical Stain-Aided Annotation Accelerates Machine Learning and Deep Learning Model Development in the Pathologic Diagnosis of Nasopharyngeal Carcinoma

Nasopharyngeal carcinoma (NPC) is an epithelial cancer originating in the nasopharynx epithelium. Nevertheless, annotating pathology slides remains a bottleneck in the development of AI-driven pathology models and applications. In the present study, we aim to demonstrate the feasibility of using immunohistochemistry (IHC) for annotation by non-pathologists and to develop an efficient model for distinguishing NPC without the time-consuming involvement of pathologists. For this study, we gathered NPC slides from 251 different patients, comprising hematoxylin and eosin (H&E) slides, pan-cytokeratin (Pan-CK) IHC slides, and Epstein-Barr virus-encoded small RNA (EBER) slides. The annotation of NPC regions in the H&E slides was carried out by a non-pathologist trainee who had access to corresponding Pan-CK IHC slides, both with and without EBER slides. The training process utilized ResNeXt, a deep neural network featuring a residual and inception architecture. In the validation set, NPC exhibited an AUC of 0.896, with a sensitivity of 0.919 and a specificity of 0.878. This study represents a significant breakthrough: the successful application of deep convolutional neural networks to identify NPC without the need for expert pathologist annotations. Our results underscore the potential of laboratory techniques to substantially reduce the workload of pathologists.

TGMIL: A hybrid multi-instance learning model based on the Transformer and the Graph Attention Network for whole-slide images classification of renal cell carcinoma

BACKGROUND AND OBJECTIVES

The pathological diagnosis of renal cell carcinoma is crucial for treatment. Currently, the multi-instance learning method is commonly used for whole-slide image classification of renal cell carcinoma, which is mainly based on the assumption of independent identical distribution. But this is inconsistent with the need to consider the correlation between different instances in the diagnosis process. Furthermore, the problem of high resource consumption of pathology images is still urgent to be solved. Therefore, we propose a new multi-instance learning method to solve this problem.

METHODS

In this study, we proposed a hybrid multi-instance learning model based on the Transformer and the Graph Attention Network, called TGMIL, to achieve whole-slide image of renal cell carcinoma classification without pixel-level annotation or region of interest extraction. Our approach is divided into three steps. First, we designed a feature pyramid with the multiple low magnifications of whole-slide image named MMFP. It makes the model incorporates richer information, and reduces memory consumption as well as training time compared to the highest magnification. Second, TGMIL amalgamates the Transformer and the Graph Attention's capabilities, adeptly addressing the loss of instance contextual and spatial. Within the Graph Attention network stream, an easy and efficient approach employing max pooling and mean pooling yields the graph adjacency matrix, devoid of extra memory consumption. Finally, the outputs of two streams of TGMIL are aggregated to achieve the classification of renal cell carcinoma.

RESULTS

On the TCGA-RCC validation set, a public dataset for renal cell carcinoma, the area under a receiver operating characteristic (ROC) curve (AUC) and accuracy of TGMIL were 0.98±0.0015,0.9191±0.0062, respectively. It showcased remarkable proficiency on the private validation set of renal cell carcinoma pathology images, attaining AUC of 0.9386±0.0162 and ACC of 0.9197±0.0124. Furthermore, on the public breast cancer whole-slide image test dataset, CAMELYON 16, our model showed good classification performance with an accuracy of 0.8792.

CONCLUSIONS

TGMIL models the diagnostic process of pathologists and shows good classification performance on multiple datasets. Concurrently, the MMFP module efficiently diminishes resource requirements, offering a novel angle for exploring computational pathology images.

Clinical applications of graph neural networks in computational histopathology: A review

Pathological examination is the optimal approach for diagnosing cancer, and with the advancement of digital imaging technologies, it has spurred the emergence of computational histopathology. The objective of computational histopathology is to assist in clinical tasks through image processing and analysis techniques. In the early stages, the technique involved analyzing histopathology images by extracting mathematical features, but the performance of these models was unsatisfactory. With the development of artificial intelligence (AI) technologies, traditional machine learning methods were applied in this field. Although the performance of the models improved, there were issues such as poor model generalization and tedious manual feature extraction. Subsequently, the introduction of deep learning techniques effectively addressed these problems. However, models based on traditional convolutional architectures could not adequately capture the contextual information and deep biological features in histopathology images. Due to the special structure of graphs, they are highly suitable for feature extraction in tissue histopathology images and have achieved promising performance in numerous studies. In this article, we review existing graph-based methods in computational histopathology and propose a novel and more comprehensive graph construction approach. Additionally, we categorize the methods and techniques in computational histopathology according to different learning paradigms. We summarize the common clinical applications of graph-based methods in computational histopathology. Furthermore, we discuss the core concepts in this field and highlight the current challenges and future research directions.

Domain-Specific Pre-training Improves Confidence in Whole Slide Image Classification

Whole Slide Images (WSIs) or histopathology images are used in digital pathology. WSIs pose great challenges to deep learning models for clinical diagnosis, owing to their size and lack of pixel-level annotations. With the recent advancements in computational pathology, newer multiple-instance learning-based models have been proposed. Multiple-instance learning for WSIs necessitates creating patches and uses the encoding of these patches for diagnosis. These models use generic pre-trained models (ResNet-50 pre-trained on ImageNet) for patch encoding. The recently proposed KimiaNet, a DenseNet121 model pre-trained on TCGA slides, is a domain-specific pre-trained model. This paper shows the effect of domain-specific pre-training on WSI classification. To investigate the effect of domain-specific pre-training, we considered the current state-of-the-art multiple-instance learning models, 1) CLAM, an attention-based model, and 2) TransMIL, a self-attention-based model, and evaluated the models' confidence and predictive performance in detecting primary brain tumors - gliomas. Domain-specific pre-training improves the confidence of the models and also achieves a new state-of-the-art performance of WSI-based glioma subtype classification, showing a high clinical applicability in assisting glioma diagnosis. We will publicly share our code and experimental results at https://github.com/soham-chitnis10/WSI-domain-specific.

A state-of-the-art survey of artificial neural networks for Whole-slide Image analysis: From popular Convolutional Neural Networks to potential visual transformers

In recent years, with the advancement of computer-aided diagnosis (CAD) technology and whole slide image (WSI), histopathological WSI has gradually played a crucial aspect in the diagnosis and analysis of diseases. To increase the objectivity and accuracy of pathologists' work, artificial neural network (ANN) methods have been generally needed in the segmentation, classification, and detection of histopathological WSI. However, the existing review papers only focus on equipment hardware, development status and trends, and do not summarize the art neural network used for full-slide image analysis in detail. In this paper, WSI analysis methods based on ANN are reviewed. Firstly, the development status of WSI and ANN methods is introduced. Secondly, we summarize the common ANN methods. Next, we discuss publicly available WSI datasets and evaluation metrics. These ANN architectures for WSI processing are divided into classical neural networks and deep neural networks (DNNs) and then analyzed. Finally, the application prospect of the analytical method in this field is discussed. The important potential method is Visual Transformers.

Biased data, biased AI: deep networks predict the acquisition site of TCGA images

BACKGROUND

Deep learning models applied to healthcare applications including digital pathology have been increasing their scope and importance in recent years. Many of these models have been trained on The Cancer Genome Atlas (TCGA) atlas of digital images, or use it as a validation source. One crucial factor that seems to have been widely ignored is the internal bias that originates from the institutions that contributed WSIs to the TCGA dataset, and its effects on models trained on this dataset.

METHODS

8,579 paraffin-embedded, hematoxylin and eosin stained, digital slides were selected from the TCGA dataset. More than 140 medical institutions (acquisition sites) contributed to this dataset. Two deep neural networks (DenseNet121 and KimiaNet were used to extract deep features at 20× magnification. DenseNet was pre-trained on non-medical objects. KimiaNet has the same structure but trained for cancer type classification on TCGA images. The extracted deep features were later used to detect each slide's acquisition site, and also for slide representation in image search.

RESULTS

DenseNet's deep features could distinguish acquisition sites with 70% accuracy whereas KimiaNet's deep features could reveal acquisition sites with more than 86% accuracy. These findings suggest that there are acquisition site specific patterns that could be picked up by deep neural networks. It has also been shown that these medically irrelevant patterns can interfere with other applications of deep learning in digital pathology, namely image search. This study shows that there are acquisition site specific patterns that can be used to identify tissue acquisition sites without any explicit training. Furthermore, it was observed that a model trained for cancer subtype classification has exploited such medically irrelevant patterns to classify cancer types. Digital scanner configuration and noise, tissue stain variation and artifacts, and source site patient demographics are among factors that likely account for the observed bias. Therefore, researchers should be cautious of such bias when using histopathology datasets for developing and training deep networks.

Privacy risks of whole-slide image sharing in digital pathology

Access to large volumes of so-called whole-slide images-high-resolution scans of complete pathological slides-has become a cornerstone of the development of novel artificial intelligence methods in pathology for diagnostic use, education/training of pathologists, and research. Nevertheless, a methodology based on risk analysis for evaluating the privacy risks associated with sharing such imaging data and applying the principle "as open as possible and as closed as necessary" is still lacking. In this article, we develop a model for privacy risk analysis for whole-slide images which focuses primarily on identity disclosure attacks, as these are the most important from a regulatory perspective. We introduce a taxonomy of whole-slide images with respect to privacy risks and mathematical model for risk assessment and design . Based on this risk assessment model and the taxonomy, we conduct a series of experiments to demonstrate the risks using real-world imaging data. Finally, we develop guidelines for risk assessment and recommendations for low-risk sharing of whole-slide image data.

Machine learning-based integration develops an immune-related risk model for predicting prognosis of high-grade serous ovarian cancer and providing therapeutic strategies

Background

High-grade serous ovarian cancer (HGSOC) is a highly lethal gynecological cancer that requires accurate prognostic models and personalized treatment strategies. The tumor microenvironment (TME) is crucial for disease progression and treatment. Machine learning-based integration is a powerful tool for identifying predictive biomarkers and developing prognostic models. Hence, an immune-related risk model developed using machine learning-based integration could improve prognostic prediction and guide personalized treatment for HGSOC.

Methods

During the bioinformatic study in HGSOC, we performed (i) consensus clustering to identify immune subtypes based on signatures of immune and stromal cells, (ii) differentially expressed genes and univariate Cox regression analysis to derive TME- and prognosis-related genes, (iii) machine learning-based procedures constructed by ten independent machine learning algorithms to screen and construct a TME-related risk score (TMErisk), and (iv) evaluation of the effect of TMErisk on the deconstruction of TME, indication of genomic instability, and guidance of immunotherapy and chemotherapy.

Results

We identified two different immune microenvironment phenotypes and a robust and clinically practicable prognostic scoring system. TMErisk demonstrated superior performance over most clinical features and other published signatures in predicting HGSOC prognosis across cohorts. The low TMErisk group with a notably favorable prognosis was characterized by BRCA1 mutation, activation of immunity, and a better immune response. Conversely, the high TMErisk group was significantly associated with C-X-C motif chemokine ligands deletion and carcinogenic activation pathways. Additionally, low TMErisk group patients were more responsive to eleven candidate agents.

Conclusion

Our study developed a novel immune-related risk model that predicts the prognosis of ovarian cancer patients using machine learning-based integration. Additionally, the study not only depicts the diversity of cell components in the TME of HGSOC but also guides the development of potential therapeutic techniques for addressing tumor immunosuppression and enhancing the response to cancer therapy.

Bias reduction in representation of histopathology images using deep feature selection

Appearing traces of bias in deep networks is a serious reliability issue which can play a significant role in ethics and generalization related concerns. Recent studies report that the deep features extracted from the histopathology images of The Cancer Genome Atlas (TCGA), the largest publicly available archive, are surprisingly able to accurately classify the whole slide images (WSIs) based on their acquisition site while these features are extracted to primarily discriminate cancer types. This is clear evidence that the utilized Deep Neural Networks (DNNs) unexpectedly detect the specific patterns of the source site, i.e, the hospital of origin, rather than histomorphologic patterns, a biased behavior resulting in degraded trust and generalization. This observation motivated us to propose a method to alleviate the destructive impact of hospital bias through a novel feature selection process. To this effect, we have proposed an evolutionary strategy to select a small set of optimal features to not only accurately represent the histological patterns of tissue samples but also to eliminate the features contributing to internal bias toward the institution. The defined objective function for an optimal subset selection of features is to minimize the accuracy of the model to classify the source institutions which is basically defined as a bias indicator. By the conducted experiments, the selected features extracted by the state-of-the-art network trained on TCGA images (i.e., the KimiaNet), considerably decreased the institutional bias, while improving the quality of features to discriminate the cancer types. In addition, the selected features could significantly improve the results of external validation compared to the entire set of features which has been negatively affected by bias. The proposed scheme is a model-independent approach which can be employed when it is possible to define a bias indicator as a participating objective in a feature selection process; even with unknown bias sources.

An integrative web-based software tool for multi-dimensional pathology whole-slide image analytics

Objective.In the era of precision medicine, human tumor atlas-oriented studies have been significantly facilitated by high-resolution, multi-modal tissue based microscopic pathology image analytics. To better support such tissue-based investigations, we have developed Digital Pathology Laboratory (DPLab), a publicly available web-based platform, to assist biomedical research groups, non-technical end users, and clinicians for pathology whole-slide image visualization, annotation, analysis, and sharing via web browsers.Approach.A major advancement of this work is the easy-to-follow methods to reconstruct three-dimension (3D) tissue image volumes by registering two-dimension (2D) whole-slide pathology images of serial tissue sections stained by hematoxylin and eosin (H&E), and immunohistochemistry (IHC). The integration of these serial slides stained by different methods provides cellular phenotype and pathophysiologic states in the context of a 3D tissue micro-environment. DPLab is hosted on a publicly accessible server and connected to a backend computational cluster for intensive image analysis computations, with results visualized, downloaded, and shared via a web interface.Main results.Equipped with an analysis toolbox of numerous image processing algorithms, DPLab supports continued integration of community-contributed algorithms and presents an effective solution to improve the accessibility and dissemination of image analysis algorithms by research communities.Significance.DPLab represents the first step in making next generation tissue investigation tools widely available to the research community, enabling and facilitating discovery of clinically relevant disease mechanisms in a digital 3D tissue space.

Deep Learning Approaches in Histopathology

The revolution of artificial intelligence and its impacts on our daily life has led to tremendous interest in the field and its related subtypes: machine learning and deep learning. Scientists and developers have designed machine learning- and deep learning-based algorithms to perform various tasks related to tumor pathologies, such as tumor detection, classification, grading with variant stages, diagnostic forecasting, recognition of pathological attributes, pathogenesis, and genomic mutations. Pathologists are interested in artificial intelligence to improve the diagnosis precision impartiality and to minimize the workload combined with the time consumed, which affects the accuracy of the decision taken. Regrettably, there are already certain obstacles to overcome connected to artificial intelligence deployments, such as the applicability and validation of algorithms and computational technologies, in addition to the ability to train pathologists and doctors to use these machines and their willingness to accept the results. This review paper provides a survey of how machine learning and deep learning methods could be implemented into health care providers' routine tasks and the obstacles and opportunities for artificial intelligence application in tumor morphology.

Computational pathology in ovarian cancer

Histopathologic evaluations of tissue sections are key to diagnosing and managing ovarian cancer. Pathologists empirically assess and integrate visual information, such as cellular density, nuclear atypia, mitotic figures, architectural growth patterns, and higher-order patterns, to determine the tumor type and grade, which guides oncologists in selecting appropriate treatment options. Latent data embedded in pathology slides can be extracted using computational imaging. Computers can analyze digital slide images to simultaneously quantify thousands of features, some of which are visible with a manual microscope, such as nuclear size and shape, while others, such as entropy, eccentricity, and fractal dimensions, are quantitatively beyond the grasp of the human mind. Applications of artificial intelligence and machine learning tools to interpret digital image data provide new opportunities to explore and quantify the spatial organization of tissues, cells, and subcellular structures. In comparison to genomic, epigenomic, transcriptomic, and proteomic patterns, morphologic and spatial patterns are expected to be more informative as quantitative biomarkers of complex and dynamic tumor biology. As computational pathology is not limited to visual data, nuanced subvisual alterations that occur in the seemingly “normal” pre-cancer microenvironment could facilitate research in early cancer detection and prevention. Currently, efforts to maximize the utility of computational pathology are focused on integrating image data with other -omics platforms that lack spatial information, thereby providing a new way to relate the molecular, spatial, and microenvironmental characteristics of cancer. Despite a dire need for improvements in ovarian cancer prevention, early detection, and treatment, the ovarian cancer field has lagged behind other cancers in the application of computational pathology. The intent of this review is to encourage ovarian cancer research teams to apply existing and/or develop additional tools in computational pathology for ovarian cancer and actively contribute to advancing this important field.

Software tools and platforms in Digital Pathology: a review for clinicians and computer scientists

At the end of the twentieth century, a new technology was developed that allowed an entire tissue section to be scanned on an objective slide. Originally called virtual microscopy, this technology is now known as Whole Slide Imaging (WSI). WSI presents new challenges for reading, visualization, storage, and analysis. For this reason, several technologies have been developed to facilitate the handling of these images. In this paper, we analyze the most widely used technologies in the field of digital pathology, ranging from specialized libraries for the reading of these images to complete platforms that allow reading, visualization, and analysis. Our aim is to provide the reader, whether a pathologist or a computational scientist, with the knowledge to choose the technologies to use for new studies, development, or research.

Automatic Recognition of Ragged Red Fibers in Muscle Biopsy from Patients with Mitochondrial Disorders

Mitochondrial dysfunction is considered to be a major cause of primary mitochondrial myopathy in children and adults, as reduced mitochondrial respiration and morphological changes such as ragged red fibers (RRFs) are observed in muscle biopsies. However, it is also possible to hypothesize the role of mitochondrial dysfunction in aging muscle or in secondary mitochondrial dysfunctions. The recognition of true histological patterns of mitochondrial myopathy can avoid unnecessary genetic investigations. The aim of our study was to develop and validate machine-learning methods for RRF detection in light microscopy images of skeletal muscle tissue. We used image sets of 489 color images captured from representative areas of Gomori’s trichrome-stained tissue retrieved from light microscopy images at a 20× magnification. We compared the performance of random forest, gradient boosting machine, and support vector machine classifiers. Our results suggested that the advent of scanning technologies, combined with the development of machine-learning models for image classification, make neuromuscular disorders’ automated diagnostic systems a concrete possibility.

A community-based approach to image analysis of cells, tissues and tumors

Emerging multiplexed imaging platforms provide an unprecedented view of an increasing number of molecular markers at subcellular resolution and the dynamic evolution of tumor cellular composition. As such, they are capable of elucidating cell-to-cell interactions within the tumor microenvironment that impact clinical outcome and therapeutic response. However, the rapid development of these platforms has far outpaced the computational methods for processing and analyzing the data they generate. While being technologically disparate, all imaging assays share many computational requirements for post-collection data processing. As such, our Image Analysis Working Group (IAWG), composed of researchers in the Cancer Systems Biology Consortium (CSBC) and the Physical Sciences - Oncology Network (PS-ON), convened a workshop on "Computational Challenges Shared by Diverse Imaging Platforms" to characterize these common issues and a follow-up hackathon to implement solutions for a selected subset of them. Here, we delineate these areas that reflect major axes of research within the field, including image registration, segmentation of cells and subcellular structures, and identification of cell types from their morphology. We further describe the logistical organization of these events, believing our lessons learned can aid others in uniting the imaging community around self-identified topics of mutual interest, in designing and implementing operational procedures to address those topics and in mitigating issues inherent in image analysis (e.g., sharing exemplar images of large datasets and disseminating baseline solutions to hackathon challenges through open-source code repositories).

Molecular analysis of TCGA breast cancer histologic types

Breast cancer is classified into multiple distinct histologic types, and many of the rarer types have limited characterization. Here, we extend The Cancer Genome Atlas Breast Cancer (TCGA-BRCA) dataset with additional histologic type annotations, in a total of 1063 breast cancers. We analyze this extended dataset to define transcriptomic and genomic profiles of six rare special histologic types: cribriform, micropapillary, mucinous, papillary, metaplastic, and invasive carcinoma with medullary pattern. We show the broader applicability of our constructed special histologic type gene signatures in the TCGA Pan-Cancer Atlas dataset with a predictive model that detects mucinous histologic type across cancers of other organ systems. Using a normal mammary cell differentiation score analysis, we order histologic types into a continuum from stem cell-like to luminal progenitor-like to mature luminal-like. Finally, we classify TCGA-BRCA into 12 consensus groups based on integrated genomic and histological features. We present a rich openly accessible resource of histologic and genomic characterization of TCGA-BRCA to enable studies of the range of breast cancers.

Navigating Multi-Scale Cancer Systems Biology Towards Model-Driven Clinical Oncology and Its Applications in Personalized Therapeutics

Rapid advancements in high-throughput omics technologies and experimental protocols have led to the generation of vast amounts of scale-specific biomolecular data on cancer that now populates several online databases and resources. Cancer systems biology models built using this data have the potential to provide specific insights into complex multifactorial aberrations underpinning tumor initiation, development, and metastasis. Furthermore, the annotation of these single- and multi-scale models with patient data can additionally assist in designing personalized therapeutic interventions as well as aid in clinical decision-making. Here, we have systematically reviewed the emergence and evolution of (i) repositories with scale-specific and multi-scale biomolecular cancer data, (ii) systems biology models developed using this data, (iii) associated simulation software for the development of personalized cancer therapeutics, and (iv) translational attempts to pipeline multi-scale panomics data for data-driven in silico clinical oncology. The review concludes that the absence of a generic, zero-code, panomics-based multi-scale modeling pipeline and associated software framework, impedes the development and seamless deployment of personalized in silico multi-scale models in clinical settings.

IDH1 and IDH2 Mutations in Colorectal Cancers

OBJECTIVES

To elucidate clinicopathologic and molecular characteristics of IDH1 and IDH2 (IDH1/2) mutations in colorectal cancers (CRCs).

METHODS

We evaluated IDH1/2 mutations in 1,623 CRCs using a next-generation sequencing assay.

RESULTS

IDH1/2 mutations, predominantly IDH1 p.R132C, were detected in 15 (0.9%) CRCs and in 5 (3.0%) of 167 BRAF p.V600E-mutated CRCs. Three IDH1/2-mutated CRCs were associated with inflammatory bowel disease. They were significantly associated with old age, mucinous or signet ring cell adenocarcinoma, and high-grade histomorphology. Concordance of variant allele frequency between IDH1/2 mutants and other trunk drivers in CRCs and presence of IDH1/2 mutation in the adenoma and early adenocarcinoma indicated IDH1/2 mutations could be trunk drivers suitable for targeted therapy.

CONCLUSIONS

IDH1/2 mutations in CRCs were uncommon but enriched in BRAF p.V600E-mutated CRCs and perhaps colitis-associated CRCs. Further studies on IDH1/2-mutated CRCs are needed to clarify their clinicopathologic features and implications for targeted therapy.

Visual Analytics for Hypothesis-Driven Exploration in Computational Pathology

Recent advances in computational and algorithmic power are evolving the field of medical imaging rapidly. In cancer research, many new directions are sought to characterize patients with additional imaging features derived from radiology and pathology images. The emerging field of Computational Pathology targets the high-throughput extraction and analysis of the spatial distribution of cells from digital histopathology images. The associated morphological and architectural features allow researchers to quantify and characterize new imaging biomarkers for cancer diagnosis, prognosis, and treatment decisions. However, while the image feature space grows, exploration and analysis become more difficult and ineffective. There is a need for dedicated interfaces for interactive data manipulation and visual analysis of computational pathology and clinical data. For this purpose, we present IIComPath, a visual analytics approach that enables clinical researchers to formulate hypotheses and create computational pathology pipelines involving cohort construction, spatial analysis of image-derived features, and cohort analysis. We demonstrate our approach through use cases that investigate the prognostic value of current diagnostic features and new computational pathology biomarkers.

Pan-cancer analysis of pathway-based gene expression pattern at the individual level reveals biomarkers of clinical prognosis

Identifying biomarkers to predict the clinical outcomes of individual patients is a fundamental problem in clinical oncology. Multiple single-gene biomarkers have already been identified and used in clinics. However, multiple oncogenes or tumor-suppressor genes are involved during the process of tumorigenesis. Additionally, the efficacy of single-gene biomarkers is limited by the extensively variable expression levels measured by high-throughput assays. In this study, we hypothesize that in individual tumor samples, the disruption of transcription homeostasis in key pathways or gene sets plays an important role in tumorigenesis and has profound implications for the patient's clinical outcome. We devised a computational method named iPath to identify, at the individual-sample level, which pathways or gene sets significantly deviate from their norms. We conducted a pan-cancer analysis and demonstrated that iPath is capable of identifying highly predictive biomarkers for clinical outcomes, including overall survival, tumor subtypes, and tumor-stage classifications.

The impact of site-specific digital histology signatures on deep learning model accuracy and bias

The Cancer Genome Atlas (TCGA) is one of the largest biorepositories of digital histology. Deep learning (DL) models have been trained on TCGA to predict numerous features directly from histology, including survival, gene expression patterns, and driver mutations. However, we demonstrate that these features vary substantially across tissue submitting sites in TCGA for over 3,000 patients with six cancer subtypes. Additionally, we show that histologic image differences between submitting sites can easily be identified with DL. Site detection remains possible despite commonly used color normalization and augmentation methods, and we quantify the image characteristics constituting this site-specific digital histology signature. We demonstrate that these site-specific signatures lead to biased accuracy for prediction of features including survival, genomic mutations, and tumor stage. Furthermore, ethnicity can also be inferred from site-specific signatures, which must be accounted for to ensure equitable application of DL. These site-specific signatures can lead to overoptimistic estimates of model performance, and we propose a quadratic programming method that abrogates this bias by ensuring models are not trained and validated on samples from the same site.

Fine-Tuning and training of densenet for histopathology image representation using TCGA diagnostic slides

Feature vectors provided by pre-trained deep artificial neural networks have become a dominant source for image representation in recent literature. Their contribution to the performance of image analysis can be improved through fine-tuning. As an ultimate solution, one might even train a deep network from scratch with the domain-relevant images, a highly desirable option which is generally impeded in pathology by lack of labeled images and the computational expense. In this study, we propose a new network, namely KimiaNet, that employs the topology of the DenseNet with four dense blocks, fine-tuned and trained with histopathology images in different configurations. We used more than 240,000 image patches with 1000×1000 pixels acquired at 20× magnification through our proposed "high-cellularity mosaic" approach to enable the usage of weak labels of 7126 whole slide images of formalin-fixed paraffin-embedded human pathology samples publicly available through The Cancer Genome Atlas (TCGA) repository. We tested KimiaNet using three public datasets, namely TCGA, endometrial cancer images, and colorectal cancer images by evaluating the performance of search and classification when corresponding features of different networks are used for image representation. As well, we designed and trained multiple convolutional batch-normalized ReLU (CBR) networks. The results show that KimiaNet provides superior results compared to the original DenseNet and smaller CBR networks when used as feature extractor to represent histopathology images.

Artificial intelligence and algorithmic computational pathology: an introduction with renal allograft examples

Whole slide imaging, which is an important technique in the field of digital pathology, has recently been the subject of increased interest and avenues for utilisation, and with more widespread whole slide image (WSI) utilisation, there will also be increased interest in and implementation of image analysis (IA) techniques. IA includes artificial intelligence (AI) and targeted or hypothesis-driven algorithms. In the overall pathology field, the number of citations related to these topics has increased in recent years. Renal pathology is one anatomical pathology subspecialty that has utilised WSIs and IA algorithms; it can be argued that renal transplant pathology could be particularly suited for whole slide imaging and IA, as renal transplant pathology is frequently classified by use of the semiquantitative Banff classification of renal allograft pathology. Hypothesis-driven/targeted algorithms have been used in the past for the assessment of a variety of features in the kidney (e.g. interstitial fibrosis, tubular atrophy, inflammation); in recent years, the amount of research has particularly increased in the area of AI/machine learning for the identification of glomeruli, for histological segmentation, and for other applications. Deep learning is the form of machine learning that is most often used for such AI approaches to the 'big data' of pathology WSIs, and deep learning methods such as artificial neural networks (ANNs)/convolutional neural networks (CNNs) are utilised. Unsupervised and supervised AI algorithms can be employed to accomplish image or semantic classification. In this review, AI and other IA algorithms applied to WSIs are discussed, and examples from renal pathology are covered, with an emphasis on renal transplant pathology.

Histopathology classification and localization of colorectal cancer using global labels by weakly supervised deep learning

Colorectal cancer (CRC) is the second leading cause of cancer-related mortality worldwide. In coping with it, histopathology image analysis (HIA) provides key information for clinical diagnosis of CRC. Nowadays, the deep learning methods are widely used in improving cancer classification and localization of tumor-regions in HIA. However, these efforts are both time-consuming and labor-intensive due to the manual annotation of tumor-regions in the whole slide images (WSIs). Furthermore, classical deep learning methods to analyze thousands of patches extracted from WSIs may cause loss of integrated information of image. Herein, a novel method was developed, which used only global labels to achieve WSI classification and localization of carcinoma by combining features from different magnifications of WSIs. The model was trained and tested using 1346 colorectal cancer WSIs from the Cancer Genome Atlas (TCGA). Our method classified colorectal cancer with an accuracy of 94.6 %, which slightly outperforms most of the existing methods. Its cancerous-location probability maps were in good agreement with annotations from three individual expert pathologists. Independent tests on 50 newly-collected colorectal cancer WSIs from hospitals produced 92.0 % accuracy and cancerous-location probability maps were in good agreement with the three pathologists. The results thereby demonstrated that the method sufficiently achieved WSI classification and localization utilizing only global labels. This weakly supervised deep learning method is effective in time and cost, as it delivered a better performance in comparison with the state-of-the-art methods.

Yottixel - An Image Search Engine for Large Archives of Histopathology Whole Slide Images

With the emergence of digital pathology, searching for similar images in large archives has gained considerable attention. Image retrieval can provide pathologists with unprecedented access to the evidence embodied in already diagnosed and treated cases from the past. This paper proposes a search engine specialized for digital pathology, called Yottixel, a portmanteau for "one yotta pixel," alluding to the big-data nature of histopathology images. The most impressive characteristic of Yottixel is its ability to represent whole slide images (WSIs) in a compact manner. Yottixel can perform millions of searches in real-time with a high search accuracy and low storage profile. Yottixel uses an intelligent indexing algorithm capable of representing WSIs with a mosaic of patches which are then converted into barcodes, called "Bunch of Barcodes" (BoB), the most prominent performance enabler of Yottixel. The performance of the prototype platform is qualitatively tested using 300 WSIs from the University of Pittsburgh Medical Center (UPMC) and 2,020 WSIs from The Cancer Genome Atlas Program (TCGA) provided by the National Cancer Institute. Both datasets amount to more than 4,000,000 patches of 1000 × 1000 pixels. We report three sets of experiments that show that Yottixel can accurately retrieve organs and malignancies, and its semantic ordering shows good agreement with the subjective evaluation of human observers.

IDH1 and IDH2 mutations in lung adenocarcinomas: Evidences of subclonal evolution

Background

Selective IDH1 and IDH2 inhibitors have been approved for targeted therapy of acute myeloid leukemia. Clinical trials for solid tumors with IDH1 and IDH2 (IDH1/2) mutations are ongoing. Reports of IDH1/2‐mutated non–small cell lung cancers (NSCLCs), however, are limited.

Methods

We evaluated IDH1/2 mutations in 1,924 NSCLC specimens (92% adenocarcinoma) using a next‐generation sequencing assay.

Results

Retrospective quality assessments identified false detection of IDH1 c.395G>A (p.R132H) resulting from cytosine deamination (C:G→T:A) artifact in one specimen. IDH1/2 mutations were detected in 9 (0.5%) adenocarcinomas taken by fine‐needle aspiration (n = 3), thoracentesis (n = 2) or core biopsy (n = 4). All nine adenocarcinomas showed high‐grade features. Extensive clear cell change, however, was not observed. High expression (50% or greater) of PD‐L1 was observed in two of five specimens examined. IDH1/2 mutations were associated with old age, smoking history, and coexisting KRAS mutation. Lower than expected variant allele frequency of IDH1/2 mutants and coexistence of IDH1/2 mutations with known trunk drivers in the BRAF, EGFR, and KRAS genes suggest they could be branching drivers leading to subclonal evolution in lung adenocarcinomas. Multiregional analysis of an adenocarcinoma harboring two IDH2 mutations revealed parallel evolution originating from a KRAS‐mutated lineage, further supporting subclonal evolution promoted by IDH1/2 mutations.

Conclusions

IDH1/2 mutations in NSCLCs are uncommon. They occur in adenocarcinomas with high‐grade features and may be branching drivers leading to subclonal evolution. Accumulation of more IDH1/2‐mutated NSCLCs is needed to clarify their clinicopathological characteristics and implications for targeted therapy.

A survey and evaluation of Web-based tools/databases for variant analysis of TCGA data

The Cancer Genome Atlas (TCGA) is a publicly funded project that aims to catalog and discover major cancer-causing genomic alterations with the goal of creating a comprehensive ‘atlas’ of cancer genomic profiles. The availability of this genome-wide information provides an unprecedented opportunity to expand our knowledge of tumourigenesis. Computational analytics and mining are frequently used as effective tools for exploring this byzantine series of biological and biomedical data. However, some of the more advanced computational tools are often difficult to understand or use, thereby limiting their application by scientists who do not have a strong computational background. Hence, it is of great importance to build user-friendly interfaces that allow both computational scientists and life scientists without a computational background to gain greater biological and medical insights. To that end, this survey was designed to systematically present available Web-based tools and facilitate the use TCGA data for cancer research.

Emerging role of deep learning-based artificial intelligence in tumor pathology

The development of digital pathology and progression of state-of-the-art algorithms for computer vision have led to increasing interest in the use of artificial intelligence (AI), especially deep learning (DL)-based AI, in tumor pathology. The DL-based algorithms have been developed to conduct all kinds of work involved in tumor pathology, including tumor diagnosis, subtyping, grading, staging, and prognostic prediction, as well as the identification of pathological features, biomarkers and genetic changes. The applications of AI in pathology not only contribute to improve diagnostic accuracy and objectivity but also reduce the workload of pathologists and subsequently enable them to spend additional time on high-level decision-making tasks. In addition, AI is useful for pathologists to meet the requirements of precision oncology. However, there are still some challenges relating to the implementation of AI, including the issues of algorithm validation and interpretability, computing systems, the unbelieving attitude of pathologists, clinicians and patients, as well as regulators and reimbursements. Herein, we present an overview on how AI-based approaches could be integrated into the workflow of pathologists and discuss the challenges and perspectives of the implementation of AI in tumor pathology.

The value of lncRNA FENDRR and FOXF1 as a prognostic factor for survival of lung adenocarcinoma

It is increasingly evident that non-coding RNAs play a significant role in tumour development. However, we still have a limited knowledge of the clinical significance of long non-coding RNAs (lncRNAs) in lung cancer. The FENDRR is a long coding RNA (also named FOXF1-AS1) located in the vicinity of the protein-coding gene FOXF1 at 16q24.1 chromosomal region. The present study aimed to define the clinic pathological significance of the long-non-coding RNA FENDRR in lung adenocarcinomas. FENDRR expression measured by quantitative PCR was found significantly downregulated (p<0.001) in lung adenocarcinoma samples in comparison with their normal adjacent tissues (n=70). RNA in situ hybridization (RNA-FISH) corroborated independently the down-regulation of FENDRR. Interestingly, the expression of FENDRR correlated positively (p<0.001) with the expression of its protein-coding neighbor gene FOXF1. Additionally, FOXF1 expression was also found downregulated in adenocarcinomas compared to normal samples (p<0.001) and its expression was significantly correlated with overall survival alone (p=0.003) or in combination with FENDRR expression (p=0.01). In conclusion, our data support that FENDRR and FOXF1 expression is decreased in lung adenocarcinoma and should be considered as new potential diagnostic/prognosis biomarkers.

Histo-genomics: digital pathology at the forefront of precision medicine

The toughest challenge OMICs face is that they provide extremely high molecular resolution but poor spatial information. Understanding the cellular/histological context of the overwhelming genetic data is critical for a full understanding of the clinical behavior of a malignant tumor. Digital pathology can add an extra layer of information to help visualize in a spatial and microenvironmental context the molecular information of cancer. Thus, histo-genomics provide a unique chance for data integration. In the era of a precision medicine, a four-dimensional (4D) (temporal/spatial) analysis of cancer aided by digital pathology can be a critical step to understand the evolution/progression of different cancers and consequently tailor individual treatment plans. For instance, the integration of molecular biomarkers expression into a three-dimensional (3D) image of a digitally scanned tumor can offer a better understanding of its subtype, behavior, host immune response and prognosis. Using advanced digital image analysis, a larger spectrum of parameters can be analyzed as potential predictors of clinical behavior. Correlation between morphological features and host immune response can be also performed with therapeutic implications. Radio-histomics, or the interface of radiological images and histology is another emerging exciting field which encompasses the integration of radiological imaging with digital pathological images, genomics, and clinical data to portray a more holistic approach to understating and treating disease. These advances in digital slide scanning are not without technical challenges, which will be addressed carefully in this review with quick peek at its future.

Integrative analysis of cross-modal features for the prognosis prediction of clear cell renal cell carcinoma

Motivation

As a highly heterogeneous disease, clear cell renal cell carcinoma (ccRCC) has quite variable clinical behaviors. The prognostic biomarkers play a crucial role in stratifying patients suffering from ccRCC to avoid over- and under-treatment. Researches based on hand-crafted features and single-modal data have been widely conducted to predict the prognosis of ccRCC. However, these experience-dependent methods, neglecting the synergy among multimodal data, have limited capacity to perform accurate prediction. Inspired by complementary information among multimodal data and the successful application of convolutional neural networks (CNNs) in medical image analysis, a novel framework was proposed to improve prediction performance.

Results

We proposed a cross-modal feature-based integrative framework, in which deep features extracted from computed tomography/histopathological images by using CNNs were combined with eigengenes generated from functional genomic data, to construct a prognostic model for ccRCC. Results showed that our proposed model can stratify high- and low-risk subgroups with significant difference (P-value &lt; 0.05) and outperform the predictive performance of those models based on single-modality features in the independent testing cohort [C-index, 0.808 (0.728–0.888)]. In addition, we also explored the relationship between deep image features and eigengenes, and make an attempt to explain deep image features from the view of genomic data. Notably, the integrative framework is available to the task of prognosis prediction of other cancer with matched multimodal data.

Availability and implementation

https://github.com/zhang-de-lab/zhang-lab? from=singlemessage

Supplementary information

Supplementary data are available at Bioinformatics online.

Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology

In the past decade, advances in precision oncology have resulted in an increased demand for predictive assays that enable the selection and stratification of patients for treatment. The enormous divergence of signalling and transcriptional networks mediating the crosstalk between cancer, stromal and immune cells complicates the development of functionally relevant biomarkers based on a single gene or protein. However, the result of these complex processes can be uniquely captured in the morphometric features of stained tissue specimens. The possibility of digitizing whole-slide images of tissue has led to the advent of artificial intelligence (AI) and machine learning tools in digital pathology, which enable mining of subvisual morphometric phenotypes and might, ultimately, improve patient management. In this Perspective, we critically evaluate various AI-based computational approaches for digital pathology, focusing on deep neural networks and 'hand-crafted' feature-based methodologies. We aim to provide a broad framework for incorporating AI and machine learning tools into clinical oncology, with an emphasis on biomarker development. We discuss some of the challenges relating to the use of AI, including the need for well-curated validation datasets, regulatory approval and fair reimbursement strategies. Finally, we present potential future opportunities for precision oncology.

Automated clear cell renal carcinoma grade classification with prognostic significance

We developed an automated 2-tiered Fuhrman's grading system for clear cell renal cell carcinoma (ccRCC). Whole slide images (WSI) and clinical data were retrieved for 395 The Cancer Genome Atlas (TCGA) ccRCC cases. Pathologist 1 reviewed and selected regions of interests (ROIs). Nuclear segmentation was performed. Quantitative morphological, intensity, and texture features (n = 72) were extracted. Features associated with grade were identified by constructing a Lasso model using data from cases with concordant 2-tiered Fuhrman's grades between TCGA and Pathologist 1 (training set n = 235; held-out test set n = 42). Discordant cases (n = 118) were additionally reviewed by Pathologist 2. Cox proportional hazard model evaluated the prognostic efficacy of the predicted grades in an extended test set which was created by combining the test set and discordant cases (n = 160). The Lasso model consisted of 26 features and predicted grade with 84.6% sensitivity and 81.3% specificity in the test set. In the extended test set, predicted grade was significantly associated with overall survival after adjusting for age and gender (Hazard Ratio 2.05; 95% CI 1.21-3.47); manual grades were not prognostic. Future work can adapt our computational system to predict WHO/ISUP grades, and validating this system on other ccRCC cohorts.

Structured crowdsourcing enables convolutional segmentation of histology images

MOTIVATION

While deep-learning algorithms have demonstrated outstanding performance in semantic image segmentation tasks, large annotation datasets are needed to create accurate models. Annotation of histology images is challenging due to the effort and experience required to carefully delineate tissue structures, and difficulties related to sharing and markup of whole-slide images.

RESULTS

We recruited 25 participants, ranging in experience from senior pathologists to medical students, to delineate tissue regions in 151 breast cancer slides using the Digital Slide Archive. Inter-participant discordance was systematically evaluated, revealing low discordance for tumor and stroma, and higher discordance for more subjectively defined or rare tissue classes. Feedback provided by senior participants enabled the generation and curation of 20 000+ annotated tissue regions. Fully convolutional networks trained using these annotations were highly accurate (mean AUC=0.945), and the scale of annotation data provided notable improvements in image classification accuracy.

AVAILABILITY AND IMPLEMENTATION

Dataset is freely available at: https://goo.gl/cNM4EL.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

An augmented reality microscope with real-time artificial intelligence integration for cancer diagnosis

The microscopic assessment of tissue samples is instrumental for the diagnosis and staging of cancer, and thus guides therapy. However, these assessments demonstrate considerable variability and many regions of the world lack access to trained pathologists. Though artificial intelligence (AI) promises to improve the access and quality of healthcare, the costs of image digitization in pathology and difficulties in deploying AI solutions remain as barriers to real-world use. Here we propose a cost-effective solution: the augmented reality microscope (ARM). The ARM overlays AI-based information onto the current view of the sample in real time, enabling seamless integration of AI into routine workflows. We demonstrate the utility of ARM in the detection of metastatic breast cancer and the identification of prostate cancer, with latency compatible with real-time use. We anticipate that the ARM will remove barriers towards the use of AI designed to improve the accuracy and efficiency of cancer diagnosis.

Pan-Renal Cell Carcinoma classification and survival prediction from histopathology images using deep learning

Histopathological images contain morphological markers of disease progression that have diagnostic and predictive values. In this study, we demonstrate how deep learning framework can be used for an automatic classification of Renal Cell Carcinoma (RCC) subtypes, and for identification of features that predict survival outcome from digital histopathological images. Convolutional neural networks (CNN's) trained on whole-slide images distinguish clear cell and chromophobe RCC from normal tissue with a classification accuracy of 93.39% and 87.34%, respectively. Further, a CNN trained to distinguish clear cell, chromophobe and papillary RCC achieves a classification accuracy of 94.07%. Here, we introduced a novel support vector machine-based method that helped to break the multi-class classification task into multiple binary classification tasks which not only improved the performance of the model but also helped to deal with data imbalance. Finally, we extracted the morphological features from high probability tumor regions identified by the CNN to predict patient survival outcome of most common clear cell RCC. The generated risk index based on both tumor shape and nuclei features are significantly associated with patient survival outcome. These results highlight that deep learning can play a role in both cancer diagnosis and prognosis.