On image search in histopathology

Pathology images of histopathology can be acquired from camera-mounted microscopes or whole-slide scanners. Utilizing similarity calculations to match patients based on these images holds significant potential in research and clinical contexts. Recent advancements in search technologies allow for implicit quantification of tissue morphology across diverse primary sites, facilitating comparisons, and enabling inferences about diagnosis, and potentially prognosis, and predictions for new patients when compared against a curated database of diagnosed and treated cases. In this article, we comprehensively review the latest developments in image search technologies for histopathology, offering a concise overview tailored for computational pathology researchers seeking effective, fast, and efficient image search methods in their work.

Publicly available datasets of breast histopathology H&E whole-slide images: A scoping review

Advancements in digital pathology and computing resources have made a significant impact in the field of computational pathology for breast cancer diagnosis and treatment. However, access to high-quality labeled histopathological images of breast cancer is a big challenge that limits the development of accurate and robust deep learning models. In this scoping review, we identified the publicly available datasets of breast H&E-stained whole-slide images (WSIs) that can be used to develop deep learning algorithms. We systematically searched 9 scientific literature databases and 9 research data repositories and found 17 publicly available datasets containing 10 385 H&E WSIs of breast cancer. Moreover, we reported image metadata and characteristics for each dataset to assist researchers in selecting proper datasets for specific tasks in breast cancer computational pathology. In addition, we compiled 2 lists of breast H&E patches and private datasets as supplementary resources for researchers. Notably, only 28% of the included articles utilized multiple datasets, and only 14% used an external validation set, suggesting that the performance of other developed models may be susceptible to overestimation. The TCGA-BRCA was used in 52% of the selected studies. This dataset has a considerable selection bias that can impact the robustness and generalizability of the trained algorithms. There is also a lack of consistent metadata reporting of breast WSI datasets that can be an issue in developing accurate deep learning models, indicating the necessity of establishing explicit guidelines for documenting breast WSI dataset characteristics and metadata.

Classifying breast cancer and fibroadenoma tissue biopsies from paraffined stain-free slides by fractal biomarkers in Fourier Ptychographic Microscopy

Breast cancer is one of the most spread and monitored pathologies in high-income countries. After breast biopsy, histological tissue is stored in paraffin, sectioned and mounted. Conventional inspection of tissue slides under benchtop light microscopes involves paraffin removal and staining, typically with H&E. Then, expert pathologists are called to judge the stained slides. However, paraffin removal and staining are operator-dependent, time and resources consuming processes that can generate ambiguities due to non-uniform staining. Here we propose a novel method that can work directly on paraffined stain-free slides. We use Fourier Ptychography as a quantitative phase-contrast microscopy method, which allows accessing a very wide field of view (i.e., mm2) in one single image while guaranteeing high lateral resolution (i.e., 0.5 µm). This imaging method is multi-scale, since it enables looking at the big picture, i.e. the complex tissue structure and connections, with the possibility to zoom-in up to the single-cell level. To handle this informative image content, we introduce elements of fractal geometry as multi-scale analysis method. We show the effectiveness of fractal features in describing and classifying fibroadenoma and breast cancer tissue slides from ten patients with very high accuracy. We reach 94.0 ± 4.2% test accuracy in classifying single images. Above all, we show that combining the decisions of the single images, each patient's slide can be classified with no error. Besides, fractal geometry returns a guide map to help pathologist to judge the different tissue portions based on the likelihood these can be associated to a breast cancer or fibroadenoma biomarker. The proposed automatic method could significantly simplify the steps of tissue analysis and make it independent from the sample preparation, the skills of the lab operator and the pathologist.

ProDiv: Prototype-driven consistent pseudo-bag division for whole-slide image classification

BACKGROUND AND OBJECTIVE

Pathology image classification is one of the most essential auxiliary processes in cancer diagnosis. To overcome the problem of inadequate Whole-Slide Image (WSI) samples with weak labels, pseudo-bag-based multiple instance learning (MIL) methods have attracted wide attention in pathology image classification. In this type of method, the division scheme of pseudo-bags is usually a primary factor affecting classification performance. In order to improve the division of WSI pseudo-bags on existing random/clustering approaches, this paper proposes a new Prototype-driven Division (ProDiv) scheme for the pseudo-bag-based MIL classification framework on pathology images.

METHODS

This scheme first designs an attention-based method to generate a bag prototype for each slide. On this basis, it further groups WSI patch instances into a series of instance clusters according to the feature similarities between the prototype and patches. Finally, pseudo-bags are obtained by randomly combining the non-overlapping patch instances of different instance clusters. Moreover, the design scheme of our ProDiv considers practicality, and it could be smoothly assembled with almost all the MIL-based WSI classification methods in recent years.

RESULTS

Empirical results show that our ProDiv, when integrated with several existing methods, can deliver classification AUC improvements of up to 7.3% and 10.3%, respectively on two public WSI datasets.

CONCLUSIONS

ProDiv could almost always bring obvious performance improvements to compared MIL models on typical metrics, which suggests the effectiveness of our scheme. Experimental visualization also visually interprets the correctness of the proposed ProDiv.

A diagnostic strategy for pulmonary fat embolism based on routine H&E staining using computational pathology

Pulmonary fat embolism (PFE) as a cause of death often occurs in trauma cases such as fractures and soft tissue contusions. Traditional PFE diagnosis relies on subjective methods and special stains like oil red O. This study utilizes computational pathology, combining digital pathology and deep learning algorithms, to precisely quantify fat emboli in whole slide images using conventional hematoxylin-eosin (H&E) staining. The results demonstrate deep learning's ability to identify fat droplet morphology in lung microvessels, achieving an area under the receiver operating characteristic (ROC) curve (AUC) of 0.98. The AI-quantified fat globules generally matched the Falzi scoring system with oil red O staining. The relative quantity of fat emboli against lung area was calculated by the algorithm, determining a diagnostic threshold of 8.275% for fatal PFE. A diagnostic strategy based on this threshold achieved a high AUC of 0.984, similar to manual identification with special stains but surpassing H&E staining. This demonstrates computational pathology's potential as an affordable, rapid, and precise method for fatal PFE diagnosis in forensic practice.

LESS: Label-efficient multi-scale learning for cytological whole slide image screening

In computational pathology, multiple instance learning (MIL) is widely used to circumvent the computational impasse in giga-pixel whole slide image (WSI) analysis. It usually consists of two stages: patch-level feature extraction and slide-level aggregation. Recently, pretrained models or self-supervised learning have been used to extract patch features, but they suffer from low effectiveness or inefficiency due to overlooking the task-specific supervision provided by slide labels. Here we propose a weakly-supervised Label-Efficient WSI Screening method, dubbed LESS, for cytological WSI analysis with only slide-level labels, which can be effectively applied to small datasets. First, we suggest using variational positive-unlabeled (VPU) learning to uncover hidden labels of both benign and malignant patches. We provide appropriate supervision by using slide-level labels to improve the learning of patch-level features. Next, we take into account the sparse and random arrangement of cells in cytological WSIs. To address this, we propose a strategy to crop patches at multiple scales and utilize a cross-attention vision transformer (CrossViT) to combine information from different scales for WSI classification. The combination of our two steps achieves task-alignment, improving effectiveness and efficiency. We validate the proposed label-efficient method on a urine cytology WSI dataset encompassing 130 samples (13,000 patches) and a breast cytology dataset FNAC 2019 with 212 samples (21,200 patches). The experiment shows that the proposed LESS reaches 84.79%, 85.43%, 91.79% and 78.30% on the urine cytology WSI dataset, and 96.88%, 96.86%, 98.95%, 97.06% on the breast cytology high-resolution-image dataset in terms of accuracy, AUC, sensitivity and specificity. It outperforms state-of-the-art MIL methods on pathology WSIs and realizes automatic cytological WSI cancer screening.

Mitosis detection, fast and slow: Robust and efficient detection of mitotic figures

Counting of mitotic figures is a fundamental step in grading and prognostication of several cancers. However, manual mitosis counting is tedious and time-consuming. In addition, variation in the appearance of mitotic figures causes a high degree of discordance among pathologists. With advances in deep learning models, several automatic mitosis detection algorithms have been proposed but they are sensitive to domain shift often seen in histology images. We propose a robust and efficient two-stage mitosis detection framework, which comprises mitosis candidate segmentation (Detecting Fast) and candidate refinement (Detecting Slow) stages. The proposed candidate segmentation model, termed EUNet, is fast and accurate due to its architectural design. EUNet can precisely segment candidates at a lower resolution to considerably speed up candidate detection. Candidates are then refined using a deeper classifier network, EfficientNet-B7, in the second stage. We make sure both stages are robust against domain shift by incorporating domain generalization methods. We demonstrate state-of-the-art performance and generalizability of the proposed model on the three largest publicly available mitosis datasets, winning the two mitosis domain generalization challenge contests (MIDOG21 and MIDOG22). Finally, we showcase the utility of the proposed algorithm by processing the TCGA breast cancer cohort (1,124 whole-slide images) to generate and release a repository of more than 620K potential mitotic figures (not exhaustively validated).

Oral epithelial dysplasia detection and grading in oral leukoplakia using deep learning

BACKGROUND

The grading of oral epithelial dysplasia is often time-consuming for oral pathologists and the results are poorly reproducible between observers. In this study, we aimed to establish an objective, accurate and useful detection and grading system for oral epithelial dysplasia in the whole-slides of oral leukoplakia.

METHODS

Four convolutional neural networks were compared using the image patches from 56 whole-slide of oral leukoplakia labeled by pathologists as the gold standard. Sequentially, feature detection models were trained, validated and tested with 1,000 image patches using the optimal network. Lastly, a comprehensive system named E-MOD-plus was established by combining feature detection models and a multiclass logistic model.

RESULTS

EfficientNet-B0 was selected as the optimal network to build feature detection models. In the internal dataset of whole-slide images, the prediction accuracy of E-MOD-plus was 81.3% (95% confidence interval: 71.4-90.5%) and the area under the receiver operating characteristic curve was 0.793 (95% confidence interval: 0.650 to 0.925); in the external dataset of 229 tissue microarray images, the prediction accuracy was 86.5% (95% confidence interval: 82.4-90.0%) and the area under the receiver operating characteristic curve was 0.669 (95% confidence interval: 0.496 to 0.843).

CONCLUSIONS

E-MOD-plus was objective and accurate in the detection of pathological features as well as the grading of oral epithelial dysplasia, and had potential to assist pathologists in clinical practice.

Deep neural networks can differentiate thyroid pathologies on infrared hyperspectral images

BACKGROUND AND OBJECTIVE

The thyroid is a gland responsible for producing important body hormones. Several pathologies can affect this gland, such as thyroiditis, hypothyroidism, and thyroid cancer. The visual histological analysis of thyroid specimens is a valuable process that enables pathologists to detect diseases with high efficiency, providing the patient with a better prognosis. Existing computer vision systems developed to aid in the analysis of histological samples have limitations in distinguishing pathologies with similar characteristics or samples containing multiple diseases. To overcome this challenge, hyperspectral images are being studied to represent biological samples based on their molecular interaction with light.

METHODS

In this study, we address the acquisition of infrared absorbance spectra from each voxel of histological specimens. This data is then used for the development of a multiclass fully-connected neural network model that discriminates spectral patterns, enabling the classification of voxels as healthy, cancerous, or goiter.

RESULTS

Through experiments using the k-fold cross-validation protocol, we obtained an average accuracy of 93.66 %, a sensitivity of 93.47 %, and a specificity of 96.93 %. Our results demonstrate the feasibility of using infrared hyperspectral imaging to characterize healthy tissue and thyroid pathologies using absorbance measurements. The proposed deep learning model has the potential to improve diagnostic efficiency and enhance patient outcomes.

Immune subtyping of melanoma whole slide images using multiple instance learning

Determining early-stage prognostic markers and stratifying patients for effective treatment are two key challenges for improving outcomes for melanoma patients. Previous studies have used tumour transcriptome data to stratify patients into immune subgroups, which were associated with differential melanoma specific survival and potential predictive biomarkers. However, acquiring transcriptome data is a time-consuming and costly process. Moreover, it is not routinely used in the current clinical workflow. Here, we attempt to overcome this by developing deep learning models to classify gigapixel haematoxylin and eosin (H&E) stained pathology slides, which are well established in clinical workflows, into these immune subgroups. We systematically assess six different multiple instance learning (MIL) frameworks, using five different image resolutions and three different feature extraction methods. We show that pathology-specific self-supervised models using 10x resolution patches generate superior representations for the classification of immune subtypes. In addition, in a primary melanoma dataset, we achieve a mean area under the receiver operating characteristic curve (AUC) of 0.80 for classifying histopathology images into 'high' or 'low immune' subgroups and a mean AUC of 0.82 in an independent TCGA melanoma dataset. Furthermore, we show that these models are able to stratify patients into 'high' and 'low immune' subgroups with significantly different melanoma specific survival outcomes (log rank test, P< 0.005). We anticipate that MIL methods will allow us to find new biomarkers of high importance, act as a tool for clinicians to infer the immune landscape of tumours and stratify patients, without needing to carry out additional expensive genetic tests.

Attention De-sparsification Matters: Inducing diversity in digital pathology representation learning

We propose DiRL, a Diversity-inducing Representation Learning technique for histopathology imaging. Self-supervised learning (SSL) techniques, such as contrastive and non-contrastive approaches, have been shown to learn rich and effective representations of digitized tissue samples with limited pathologist supervision. Our analysis of vanilla SSL-pretrained models' attention distribution reveals an insightful observation: sparsity in attention, i.e, models tends to localize most of their attention to some prominent patterns in the image. Although attention sparsity can be beneficial in natural images due to these prominent patterns being the object of interest itself, this can be sub-optimal in digital pathology; this is because, unlike natural images, digital pathology scans are not object-centric, but rather a complex phenotype of various spatially intermixed biological components. Inadequate diversification of attention in these complex images could result in crucial information loss. To address this, we leverage cell segmentation to densely extract multiple histopathology-specific representations, and then propose a prior-guided dense pretext task, designed to match the multiple corresponding representations between the views. Through this, the model learns to attend to various components more closely and evenly, thus inducing adequate diversification in attention for capturing context-rich representations. Through quantitative and qualitative analysis on multiple tasks across cancer types, we demonstrate the efficacy of our method and observe that the attention is more globally distributed.

Digital counting of tissue cells for molecular analysis: the QuANTUM pipeline

The estimation of tumor cellular fraction (TCF) is a crucial step in predictive molecular pathology, representing an entry adequacy criterion also in the next-generation sequencing (NGS) era. However, heterogeneity of quantification practices and inter-pathologist variability hamper the robustness of its evaluation, stressing the need for more reliable results. Here, 121 routine histological samples from non-small cell lung cancer (NSCLC) cases with complete NGS profiling were used to evaluate TCF interobserver variability among three different pathologists (pTCF), developing a computational tool (cTCF) and assessing its reliability vs ground truth (GT) tumor cellularity and potential impact on the final molecular results. Inter-pathologist reproducibility was fair to good, with overall Wk ranging between 0.46 and 0.83 (avg. 0.59). The obtained cTCF was comparable to the GT (p = 0.129, 0.502, and 0.130 for surgical, biopsies, and cell block, respectively) and demonstrated good reliability if elaborated by different pathologists (Wk = 0.9). Overall cTCF was lower as compared to pTCF (30 ± 10 vs 52 ± 19, p < 0.001), with more cases < 20% (17, 14%, p = 0.690), but none containing < 100 cells for the algorithm. Similarities were noted between tumor area estimation and pTCF (36 ± 29, p < 0.001), partly explaining variability in the human assessment of tumor cellularity. Finally, the cTCF allowed a reduction of the copy number variations (CNVs) called (27 vs 29, - 6.9%) with an increase of effective CNVs detection (13 vs 7, + 85.7%), some with potential clinical impact previously undetected with pTCF. An automated computational pipeline (Qupath Analysis of Nuclei from Tumor to Uniform Molecular tests, QuANTUM) has been created and is freely available as a QuPath extension. The computational method used in this study has the potential to improve efficacy and reliability of TCF estimation in NSCLC, with demonstrated impact on the final molecular results.

Automatic data augmentation to improve generalization of deep learning in H&E stained histopathology

In histopathology practice, scanners, tissue processing, staining, and image acquisition protocols vary from center to center, resulting in subtle variations in images. Vanilla convolutional neural networks are sensitive to such domain shifts. Data augmentation is a popular way to improve domain generalization. Currently, state-of-the-art domain generalization in computational pathology is achieved using a manually curated set of augmentation transforms. However, manual tuning of augmentation parameters is time-consuming and can lead to sub-optimal generalization performance. Meta-learning frameworks can provide efficient ways to find optimal training hyper-parameters, including data augmentation. In this study, we hypothesize that an automated search of augmentation hyper-parameters can provide superior generalization performance and reduce experimental optimization time. We select four state-of-the-art automatic augmentation methods from general computer vision and investigate their capacity to improve domain generalization in histopathology. We analyze their performance on data from 25 centers across two different tasks: tumor metastasis detection in lymph nodes and breast cancer tissue type classification. On tumor metastasis detection, most automatic augmentation methods achieve comparable performance to state-of-the-art manual augmentation. On breast cancer tissue type classification, the leading automatic augmentation method significantly outperforms state-of-the-art manual data augmentation.

Are you sure it's an artifact? Artifact detection and uncertainty quantification in histological images

Modern cancer diagnostics involves extracting tissue specimens from suspicious areas and conducting histotechnical procedures to prepare a digitized glass slide, called Whole Slide Image (WSI), for further examination. These procedures frequently introduce different types of artifacts in the obtained WSI, and histological artifacts might influence Computational Pathology (CPATH) systems further down to a diagnostic pipeline if not excluded or handled. Deep Convolutional Neural Networks (DCNNs) have achieved promising results for the detection of some WSI artifacts, however, they do not incorporate uncertainty in their predictions. This paper proposes an uncertainty-aware Deep Kernel Learning (DKL) model to detect blurry areas and folded tissues, two types of artifacts that can appear in WSIs. The proposed probabilistic model combines a CNN feature extractor and a sparse Gaussian Processes (GPs) classifier, which improves the performance of current state-of-the-art artifact detection DCNNs and provides uncertainty estimates. We achieved 0.996 and 0.938 F1 scores for blur and folded tissue detection on unseen data, respectively. In extensive experiments, we validated the DKL model on unseen data from external independent cohorts with different staining and tissue types, where it outperformed DCNNs. Interestingly, the DKL model is more confident in the correct predictions and less in the wrong ones. The proposed DKL model can be integrated into the preprocessing pipeline of CPATH systems to provide reliable predictions and possibly serve as a quality control tool.

Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview

The capability to gather heterogeneous data, alongside the increasing power of artificial intelligence to examine it, leading a revolution in harnessing multimodal data in the life sciences. However, most approaches are limited to unimodal data, leaving integrated approaches across modalities relatively underdeveloped in computational pathology. Pathogenomics, as an invasive method to integrate advanced molecular diagnostics from genomic data, morphological information from histopathological imaging, and codified clinical data enable the discovery of new multimodal cancer biomarkers to propel the field of precision oncology in the coming decade. In this perspective, we offer our opinions on synthesizing complementary modalities of data with emerging multimodal artificial intelligence methods in pathogenomics. It includes correlation between the pathological and genomic profile of cancer, fusion of histology, and genomics profile of cancer. We also present challenges, opportunities, and avenues for future work.

CoNIC Challenge: Pushing the frontiers of nuclear detection, segmentation, classification and counting

Nuclear detection, segmentation and morphometric profiling are essential in helping us further understand the relationship between histology and patient outcome. To drive innovation in this area, we setup a community-wide challenge using the largest available dataset of its kind to assess nuclear segmentation and cellular composition. Our challenge, named CoNIC, stimulated the development of reproducible algorithms for cellular recognition with real-time result inspection on public leaderboards. We conducted an extensive post-challenge analysis based on the top-performing models using 1,658 whole-slide images of colon tissue. With around 700 million detected nuclei per model, associated features were used for dysplasia grading and survival analysis, where we demonstrated that the challenge's improvement over the previous state-of-the-art led to significant boosts in downstream performance. Our findings also suggest that eosinophils and neutrophils play an important role in the tumour microevironment. We release challenge models and WSI-level results to foster the development of further methods for biomarker discovery.

Consistency regularisation in varying contexts and feature perturbations for semi-supervised semantic segmentation of histology images

Semantic segmentation of various tissue and nuclei types in histology images is fundamental to many downstream tasks in the area of computational pathology (CPath). In recent years, Deep Learning (DL) methods have been shown to perform well on segmentation tasks but DL methods generally require a large amount of pixel-wise annotated data. Pixel-wise annotation sometimes requires expert's knowledge and time which is laborious and costly to obtain. In this paper, we present a consistency based semi-supervised learning (SSL) approach that can help mitigate this challenge by exploiting a large amount of unlabelled data for model training thus alleviating the need for a large annotated dataset. However, SSL models might also be susceptible to changing context and features perturbations exhibiting poor generalisation due to the limited training data. We propose an SSL method that learns robust features from both labelled and unlabelled images by enforcing consistency against varying contexts and feature perturbations. The proposed method incorporates context-aware consistency by contrasting pairs of overlapping images in a pixel-wise manner from changing contexts resulting in robust and context invariant features. We show that cross-consistency training makes the encoder features invariant to different perturbations and improves the prediction confidence. Finally, entropy minimisation is employed to further boost the confidence of the final prediction maps from unlabelled data. We conduct an extensive set of experiments on two publicly available large datasets (BCSS and MoNuSeg) and show superior performance compared to the state-of-the-art methods.

Digital pathology in cardiac transplant diagnostics: from biopsies to algorithms

In the field of heart transplantation, the ability to accurately and promptly diagnose cardiac allograft rejection is crucial. This comprehensive review explores the transformative role of digital pathology and computational pathology, especially through machine learning, in this critical domain. These methodologies harness large datasets to extract subtle patterns and valuable information that extend beyond human perceptual capabilities, potentially enhancing diagnostic outcomes. Current research indicates that these computer-based systems could offer accuracy and performance matching, or even exceeding, that of expert pathologists, thereby introducing more objectivity and reducing observer variability. Despite promising results, several challenges such as limited sample sizes, diverse data sources, and the absence of standardized protocols pose significant barriers to the widespread adoption of these techniques. The future of digital pathology in heart transplantation diagnostics depends on utilizing larger, more diverse patient cohorts, standardizing data collection, processing, and evaluation protocols, and fostering collaborative research efforts. The integration of various data types, including clinical, demographic, and imaging information, could further refine diagnostic precision. As researchers address these challenges and promote collaborative efforts, digital pathology has the potential to become an integral part of clinical practice, ultimately improving patient care in heart transplantation.

AdvMIL: Adversarial multiple instance learning for the survival analysis on whole-slide images

The survival analysis on histological whole-slide images (WSIs) is one of the most important means to estimate patient prognosis. Although many weakly-supervised deep learning models have been developed for gigapixel WSIs, their potential is generally restricted by classical survival analysis rules and fully-supervised learning requirements. As a result, these models provide patients only with a completely-certain point estimation of time-to-event, and they could only learn from the labeled WSI data currently at a small scale. To tackle these problems, we propose a novel adversarial multiple instance learning (AdvMIL) framework. This framework is based on adversarial time-to-event modeling, and integrates the multiple instance learning (MIL) that is much necessary for WSI representation learning. It is a plug-and-play one, so that most existing MIL-based end-to-end methods can be easily upgraded by applying this framework, gaining the improved abilities of survival distribution estimation and semi-supervised learning. Our extensive experiments show that AdvMIL not only could often bring performance improvement to mainstream WSI survival analysis methods at a relatively low computational cost, but also enables these methods to effectively utilize unlabeled data via semi-supervised learning. Moreover, it is observed that AdvMIL could help improving the robustness of models against patch occlusion and two representative image noises. The proposed AdvMIL framework could promote the research of survival analysis in computational pathology with its novel adversarial MIL paradigm.

SynCLay: Interactive synthesis of histology images from bespoke cellular layouts

Automated synthesis of histology images has several potential applications in computational pathology. However, no existing method can generate realistic tissue images with a bespoke cellular layout or user-defined histology parameters. In this work, we propose a novel framework called SynCLay (Synthesis from Cellular Layouts) that can construct realistic and high-quality histology images from user-defined cellular layouts along with annotated cellular boundaries. Tissue image generation based on bespoke cellular layouts through the proposed framework allows users to generate different histological patterns from arbitrary topological arrangement of different types of cells (e.g., neutrophils, lymphocytes, epithelial cells and others). SynCLay generated synthetic images can be helpful in studying the role of different types of cells present in the tumor microenvironment. Additionally, they can assist in balancing the distribution of cellular counts in tissue images for designing accurate cellular composition predictors by minimizing the effects of data imbalance. We train SynCLay in an adversarial manner and integrate a nuclear segmentation and classification model in its training to refine nuclear structures and generate nuclear masks in conjunction with synthetic images. During inference, we combine the model with another parametric model for generating colon images and associated cellular counts as annotations given the grade of differentiation and cellularities (cell densities) of different cells. We assess the generated images quantitatively using the Frechet Inception Distance and report on feedback from trained pathologists who assigned realism scores to a set of images generated by the framework. The average realism score across all pathologists for synthetic images was as high as that for the real images. Moreover, with the assistance from pathologists, we showcase the ability of the generated images to accurately differentiate between benign and malignant tumors, thus reinforcing their reliability. We demonstrate that the proposed framework can be used to add new cells to a tissue images and alter cellular positions. We also show that augmenting limited real data with the synthetic data generated by our framework can significantly boost prediction performance of the cellular composition prediction task. The implementation of the proposed SynCLay framework is available at https://github.com/Srijay/SynCLay-Framework.

The NCI Imaging Data Commons as a platform for reproducible research in computational pathology

BACKGROUND AND OBJECTIVES

Reproducibility is a major challenge in developing machine learning (ML)-based solutions in computational pathology (CompPath). The NCI Imaging Data Commons (IDC) provides >120 cancer image collections according to the FAIR principles and is designed to be used with cloud ML services. Here, we explore its potential to facilitate reproducibility in CompPath research.

METHODS

Using the IDC, we implemented two experiments in which a representative ML-based method for classifying lung tumor tissue was trained and/or evaluated on different datasets. To assess reproducibility, the experiments were run multiple times with separate but identically configured instances of common ML services.

RESULTS

The results of different runs of the same experiment were reproducible to a large extent. However, we observed occasional, small variations in AUC values, indicating a practical limit to reproducibility.

CONCLUSIONS

We conclude that the IDC facilitates approaching the reproducibility limit of CompPath research (i) by enabling researchers to reuse exactly the same datasets and (ii) by integrating with cloud ML services so that experiments can be run in identically configured computing environments.

Artificial Intelligence-Enabled Gastric Cancer Interpretations: Are We There yet?

The integration of digital pathology and artificial intelligence (AI) is revolutionizing pathology by providing pathologists with new tools to improve workflow, enhance diagnostic accuracy, and undertake novel discovery. The capability of AI to recognize patterns and features in digital images beyond human perception is particularly valuable, thereby providing additional information for prognostic and predictive purposes. AI-based tools diagnose gastric carcinoma in digital images, detect gastric carcinoma metastases in lymph nodes, automate Ki-67 scoring in gastric neuroendocrine tumors, and quantify tumor-infiltrating lymphocytes. This article provides an overview of all of these applications of AI pertaining to gastric cancer.

Clinical Application of Digital and Computational Pathology in Renal Cell Carcinoma: A Systematic Review

CONTEXT

Computational pathology is a new interdisciplinary field that combines traditional pathology with modern technologies such as digital imaging and machine learning to better understand the diagnosis, prognosis, and natural history of many diseases.

OBJECTIVE

To provide an overview of digital and computational pathology and its current and potential applications in renal cell carcinoma (RCC).

EVIDENCE ACQUISITION

A systematic review of the English-language literature was conducted using the PubMed, Web of Science, and Scopus databases in December 2022 according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (PROSPERO ID: CRD42023389282). Risk of bias was assessed according to the Prediction Model Study Risk of Bias Assessment Tool.

EVIDENCE SYNTHESIS

In total, 20 articles were included in the review. All the studies used a retrospective design, and all digital pathology techniques were implemented retrospectively. The studies were classified according to their primary objective: detection, tumor characterization, and patient outcome. Regarding the transition to clinical practice, several studies showed promising potential. However, none presented a comprehensive assessment of clinical utility and implementation. Notably, there was substantial heterogeneity for both the strategies used for model building and the performance metrics reported.

CONCLUSIONS

This review highlights the vast potential of digital and computational pathology for the detection, classification, and assessment of oncological outcomes in RCC. Preliminary work in this field has yielded promising results. However, these models have not yet reached a stage where they can be integrated into routine clinical practice.

PATIENT SUMMARY

Computational pathology combines traditional pathology and technologies such as digital imaging and artificial intelligence to improve diagnosis of disease and identify prognostic factors and new biomarkers. The number of studies exploring its potential in kidney cancer is rapidly increasing. However, despite the surge in research activity, computational pathology is not yet ready for widespread routine use.

CaMeL-Net: Centroid-aware metric learning for efficient multi-class cancer classification in pathology images

BACKGROUND AND OBJECTIVE

Cancer grading in pathology image analysis is a major task due to its importance in patient care, treatment, and management. The recent developments in artificial neural networks for computational pathology have demonstrated great potential to improve the accuracy and quality of cancer diagnosis. These improvements are generally ascribable to the advance in the architecture of the networks, often leading to increase in the computation and resources. In this work, we propose an efficient convolutional neural network that is designed to conduct multi-class cancer classification in an accurate and robust manner via metric learning.

METHODS

We propose a centroid-aware metric learning network for an improved cancer grading in pathology images. The proposed network utilizes centroids of different classes within the feature embedding space to optimize the relative distances between pathology images, which manifest the innate similarities/dissimilarities between them. For improved optimization, we introduce a new loss function and a training strategy that are tailored to the proposed network and metric learning.

RESULTS

We evaluated the proposed approach on multiple datasets of colorectal and gastric cancers. For the colorectal cancer, two different datasets were employed that were collected from different acquisition settings. the proposed method achieved an accuracy, F1-score, quadratic weighted kappa of 88.7%, 0.849, and 0.946 for the first dataset and 83.3%, 0.764, and 0.907 for the second dataset, respectively. For the gastric cancer, the proposed method obtained an accuracy of 85.9%, F1-score of 0.793, and quadratic weighted kappa of 0.939. We also found that the proposed method outperforms other competing models and is computationally efficient.

CONCLUSIONS

The experimental results demonstrate that the prediction results by the proposed network are both accurate and reliable. The proposed network not only outperformed other related methods in cancer classification but also achieved superior computational efficiency during training and inference. The future study will entail further development of the proposed method and the application of the method to other problems and domains.

Weakly supervised joint whole-slide segmentation and classification in prostate cancer

The identification and segmentation of histological regions of interest can provide significant support to pathologists in their diagnostic tasks. However, segmentation methods are constrained by the difficulty in obtaining pixel-level annotations, which are tedious and expensive to collect for whole-slide images (WSI). Though several methods have been developed to exploit image-level weak-supervision for WSI classification, the task of segmentation using WSI-level labels has received very little attention. The research in this direction typically require additional supervision beyond image labels, which are difficult to obtain in real-world practice. In this study, we propose WholeSIGHT, a weakly-supervised method that can simultaneously segment and classify WSIs of arbitrary shapes and sizes. Formally, WholeSIGHT first constructs a tissue-graph representation of WSI, where the nodes and edges depict tissue regions and their interactions, respectively. During training, a graph classification head classifies the WSI and produces node-level pseudo-labels via post-hoc feature attribution. These pseudo-labels are then used to train a node classification head for WSI segmentation. During testing, both heads simultaneously render segmentation and class prediction for an input WSI. We evaluate the performance of WholeSIGHT on three public prostate cancer WSI datasets. Our method achieves state-of-the-art weakly-supervised segmentation performance on all datasets while resulting in better or comparable classification with respect to state-of-the-art weakly-supervised WSI classification methods. Additionally, we assess the generalization capability of our method in terms of segmentation and classification performance, uncertainty estimation, and model calibration. Our code is available at: https://github.com/histocartography/wholesight.

Histopathological Image Deep Feature Representation for CBIR in Smart PACS

Pathological Anatomy is moving toward computerizing processes mainly due to the extensive digitization of histology slides that resulted in the availability of many Whole Slide Images (WSIs). Their use is essential, especially in cancer diagnosis and research, and raises the pressing need for increasingly influential information archiving and retrieval systems. Picture Archiving and Communication Systems (PACSs) represent an actual possibility to archive and organize this growing amount of data. The design and implementation of a robust and accurate methodology for querying them in the pathology domain using a novel approach are mandatory. In particular, the Content-Based Image Retrieval (CBIR) methodology can be involved in the PACSs using a query-by-example task. In this context, one of many crucial points of CBIR concerns the representation of images as feature vectors, and the accuracy of retrieval mainly depends on feature extraction. Thus, our study explored different representations of WSI patches by features extracted from pre-trained Convolution Neural Networks (CNNs). In order to perform a helpful comparison, we evaluated features extracted from different layers of state-of-the-art CNNs using different dimensionality reduction techniques. Furthermore, we provided a qualitative analysis of obtained results. The evaluation showed encouraging results for our proposed framework.

Advances in the application of computational pathology in diagnosis, immunomicroenvironment recognition, and immunotherapy evaluation of breast cancer: a narrative review

BACKGROUND

Breast cancer (BC) is a prevalent and highly lethal malignancy affecting women worldwide. Immunotherapy has emerged as a promising therapeutic strategy for BC, offering potential improvements in patient survival. Neoadjuvant therapy (NAT) has also gained significant clinical traction. With the advancement of computer technology, Artificial Intelligence (AI) has been increasingly applied in pathology research, expanding and redefining the scope of the field. This narrative review aims to provide a comprehensive overview of the current literature on the application of computational pathology in BC, specifically focusing on diagnosis, immune microenvironment recognition, and the evaluation of immunotherapy and NAT response.

METHODS

A thorough examination of relevant literature was conducted, focusing on studies investigating the role of computational pathology in BC diagnosis, immune microenvironment recognition, and immunotherapy and NAT assessment.

RESULTS

The application of computational pathology has shown significant potential in BC management. AI-based techniques enable improved diagnosis and classification of BC subtypes, enhance the identification and characterization of the immune microenvironment, and facilitate the evaluation of immunotherapy and NAT response. However, challenges related to data quality, standardization, and algorithm development still need to be addressed.

CONCLUSION

The integration of computational pathology and AI has transformative implications for BC patient care. By leveraging AI-based technologies, clinicians can make more informed decisions in diagnosis, treatment planning, and therapeutic response assessment. Future research should focus on refining AI algorithms, addressing technical challenges, and conducting large-scale clinical validation studies to facilitate the translation of computational pathology into routine clinical practice for BC patients.

Diagnosis of diabetic kidney disease in whole slide images via AI-driven quantification of pathological indicators

Diagnosis of diabetic kidney disease (DKD) mainly relies on screening the morphological variations and internal lesions of glomeruli from pathological kidney biopsy. The prominent pathological alterations of glomeruli for DKD include glomerular hypertrophy and nodular mesangial sclerosis. However, the qualitative judgment of these alterations is inaccurate and inconstant due to the intra- and inter-subject variability of pathologists. It is necessary to design artificial intelligence (AI) methods for accurate quantification of these pathological alterations and outcome prediction of DKD. In this work, we present an AI-driven framework to quantify the volume of glomeruli and degree of nodular mesangial sclerosis, respectively, based on an instance segmentation module and a novel weakly supervised Macro-Micro Aggregation (MMA) module. Subsequently, we construct classic machine learning models to predict the degree of DKD based on three selected pathological indicators via factor analysis. These corresponding modules are trained and tested on a total of 281 whole slide images (WSIs) digitized from two hospitals with different scanners. Our designed AI framework achieved inspiring results with 0.926 mIoU for glomerulus segmentation, and 0.899 F1 score for glomerulus classification in the external testing dataset. Meantime, the visualized results of the MMA module could reflect the location of the lesions. The performance of predicting disease achieved the F1 score of 0.917, which further proved the effectiveness of our AI-driven quantification of pathological indicators. Additionally, the interpretation of the machine learning model with the SHAP method showed similar accordance with the development of DKD in pathology. In conclusion, the proposed auxiliary diagnostic technologies have the feasibility for quantitative analysis of glomerular pathological tissues and alterations in DKD. Pathological quantitative indicators will also make it more convenient to provide doctors with assistance in clinical practice.

Malignant Mesothelioma subtyping via sampling driven multiple instance prediction on tissue image and cell morphology data

Malignant Mesothelioma is a difficult to diagnose and highly lethal cancer usually associated with asbestos exposure. It can be broadly classified into three subtypes: Epithelioid, Sarcomatoid, and a hybrid Biphasic subtype in which significant components of both of the previous subtypes are present. Early diagnosis and identification of the subtype informs treatment and can help improve patient outcome. However, the subtyping of malignant mesothelioma, and specifically the recognition of transitional features from routine histology slides has a high level of inter-observer variability. In this work, we propose an end-to-end multiple instance learning (MIL) approach for malignant mesothelioma subtyping. This uses an adaptive instance-based sampling scheme for training deep convolutional neural networks on bags of image patches that allows learning on a wider range of relevant instances compared to max or top-N based MIL approaches. We also investigate augmenting the instance representation to include aggregate cellular morphology features from cell segmentation. The proposed MIL approach enables identification of malignant mesothelial subtypes of specific tissue regions. From this a continuous characterisation of a sample according to predominance of sarcomatoid vs epithelioid regions is possible, thus avoiding the arbitrary and highly subjective categorisation by currently used subtypes. Instance scoring also enables studying tumor heterogeneity and identifying patterns associated with different subtypes. We have evaluated the proposed method on a dataset of 234 tissue micro-array cores with an AUROC of 0.89±0.05 for this task. The dataset and developed methodology is available for the community at: https://github.com/measty/PINS.

Modelling digital health data: The ExaMode ontology for computational pathology

Computational pathology can significantly benefit from ontologies to standardize the employed nomenclature and help with knowledge extraction processes for high-quality annotated image datasets. The end goal is to reach a shared model for digital pathology to overcome data variability and integration problems. Indeed, data annotation in such a specific domain is still an unsolved challenge and datasets cannot be steadily reused in diverse contexts due to heterogeneity issues of the adopted labels, multilingualism, and different clinical practices.

Material and methods

This paper presents the ExaMode ontology, modeling the histopathology process by considering 3 key cancer diseases (colon, cervical, and lung tumors) and celiac disease. The ExaMode ontology has been designed bottom-up in an iterative fashion with continuous feedback and validation from pathologists and clinicians. The ontology is organized into 5 semantic areas that defines an ontological template to model any disease of interest in histopathology.

Results

The ExaMode ontology is currently being used as a common semantic layer in: (i) an entity linking tool for the automatic annotation of medical records; (ii) a web-based collaborative annotation tool for histopathology text reports; and (iii) a software platform for building holistic solutions integrating multimodal histopathology data.

Discussion

The ontology ExaMode is a key means to store data in a graph database according to the RDF data model. The creation of an RDF dataset can help develop more accurate algorithms for image analysis, especially in the field of digital pathology. This approach allows for seamless data integration and a unified query access point, from which we can extract relevant clinical insights about the considered diseases using SPARQL queries.

DeepBLS: Deep Feature-Based Broad Learning System for Tissue Phenotyping in Colorectal Cancer WSIs

Tissue phenotyping is a fundamental step in computational pathology for the analysis of tumor micro-environment in whole slide images (WSIs). Automatic tissue phenotyping in whole slide images (WSIs) of colorectal cancer (CRC) assists pathologists in better cancer grading and prognostication. In this paper, we propose a novel algorithm for the identification of distinct tissue components in colon cancer histology images by blending a comprehensive learning system with deep features extraction in the current work. Firstly, we extracted the features from the pre-trained VGG19 network which are then transformed into mapped features space for nodes enhancement generation. Utilizing both mapped features and enhancement nodes, the proposed algorithm classifies seven distinct tissue components including stroma, tumor, complex stroma, necrotic, normal benign, lymphocytes, and smooth muscle. To validate our proposed model, the experiments are performed on two publicly available colorectal cancer histology datasets. We showcase that our approach achieves a remarkable performance boost surpassing existing state-of-the-art methods by (1.3% AvTP, 2% F1) and (7% AvTP, 6% F1) on CRCD-1, and CRCD-2, respectively.

An aggregation of aggregation methods in computational pathology

Image analysis and machine learning algorithms operating on multi-gigapixel whole-slide images (WSIs) often process a large number of tiles (sub-images) and require aggregating predictions from the tiles in order to predict WSI-level labels. In this paper, we present a review of existing literature on various types of aggregation methods with a view to help guide future research in the area of computational pathology (CPath). We propose a general CPath workflow with three pathways that consider multiple levels and types of data and the nature of computation to analyse WSIs for predictive modelling. We categorize aggregation methods according to the context and representation of the data, features of computational modules and CPath use cases. We compare and contrast different methods based on the principle of multiple instance learning, perhaps the most commonly used aggregation method, covering a wide range of CPath literature. To provide a fair comparison, we consider a specific WSI-level prediction task and compare various aggregation methods for that task. Finally, we conclude with a list of objectives and desirable attributes of aggregation methods in general, pros and cons of the various approaches, some recommendations and possible future directions.

Gigapixel end-to-end training using streaming and attention

Current hardware limitations make it impossible to train convolutional neural networks on gigapixel image inputs directly. Recent developments in weakly supervised learning, such as attention-gated multiple instance learning, have shown promising results, but often use multi-stage or patch-wise training strategies risking suboptimal feature extraction, which can negatively impact performance. In this paper, we propose to train a ResNet-34 encoder with an attention-gated classification head in an end-to-end fashion, which we call StreamingCLAM, using a streaming implementation of convolutional layers. This allows us to train end-to-end on 4-gigapixel microscopic images using only slide-level labels. We achieve a mean area under the receiver operating characteristic curve of 0.9757 for metastatic breast cancer detection (CAMELYON16), close to fully supervised approaches using pixel-level annotations. Our model can also detect MYC-gene translocation in histologic slides of diffuse large B-cell lymphoma, achieving a mean area under the ROC curve of 0.8259. Furthermore, we show that our model offers a degree of interpretability through the attention mechanism.

Multilayer outperforms single-layer slide scanning in AI-based classification of whole slide images with low-burden acid-fast mycobacteria (AFB)

Manual screening of Ziehl-Neelsen (ZN)-stained slides that are negative or contain rare acid-fast mycobacteria (AFB) is labor-intensive and requires repetitive refocusing to visualize AFB candidates under the microscope. Whole slide image (WSI) scanners have enabled implementation of AI to classify digital ZN-stained slides as AFB+ or AFB-. By default, these scanners acquire a single-layer WSI. However, some scanners can acquire a multilayer WSI with a z-stack and an extended focus image layer embedded. We developed a parameterized WSI classification pipeline to assess whether multilayer imaging improves ZN-stained slide classification accuracy. A CNN built into the pipeline classified tiles in each image layer to form an AFB probability score heatmap. Features extracted from the heatmap were then entered into a WSI classifier. 46 AFB+ and 88 AFB- single-layer WSIs were used for the classifier training. 15 AFB+ (with rare microorganisms) and 5 AFB- multilayer WSIs comprised the test set. Parameters in the pipeline included: (a) a WSI representation: z-stack of image layers, middle image layer (a single image layer equivalent) or an extended focus image layer, (b) 4 methods of aggregating AFB probability scores across the z-stack, (c) 3 classifiers, (d) 3 AFB probability thresholds, and (e) 9 feature vector types extracted from the aggregated AFB probability heatmaps. Balanced accuracy (BACC) was used to measure the pipeline performance for all parameter combinations. Analysis of Covariance (ANCOVA) was used to statistically evaluate the effect of each parameter on the BACC. After adjusting for other factors, a significant effect of the WSI representation (p-value < 1.99E-76), classifier type (p-value < 1.73E-21), and AFB threshold (p-value = 0.003) was observed on the BACC. The feature type had no significant effect (p-value = 0.459) on the BACC. WSIs represented by the middle layer, extended focus layer and the z-stack followed by the weighted averaging of AFB probability scores were classified with the average BACC of 58.80%, 68.64%, and 77.28%, respectively. The multilayer WSIs represented by the z-stack with the weighted averaging of AFB probability scores were classified by a Random Forest classifier with the average BACC of 83.32%. Low classification accuracy of WSIs represented by the middle layer suggests that they contain fewer features permitting identification of AFB than the multilayer WSIs. Our results indicate that single-layer acquisition can introduce a bias (sampling error) into the WSI. This bias can be mitigated by the multilayer or the extended focus acquisitions.

ChampKit: A framework for rapid evaluation of deep neural networks for patch-based histopathology classification

BACKGROUND AND OBJECTIVE

Histopathology is the gold standard for diagnosis of many cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for many tasks, including the detection of immune cells and microsatellite instability. However, it remains difficult to identify optimal models and training configurations for different histopathology classification tasks due to the abundance of available architectures and the lack of systematic evaluations. Our objective in this work is to present a software tool that addresses this need and enables robust, systematic evaluation of neural network models for patch classification in histology in a light-weight, easy-to-use package for both algorithm developers and biomedical researchers.

METHODS

Here we present ChampKit (Comprehensive Histopathology Assessment of Model Predictions toolKit): an extensible, fully reproducible evaluation toolkit that is a one-stop-shop to train and evaluate deep neural networks for patch classification. ChampKit curates a broad range of public datasets. It enables training and evaluation of models supported by timm directly from the command line, without the need for users to write any code. External models are enabled through a straightforward API and minimal coding. As a result, Champkit facilitates the evaluation of existing and new models and deep learning architectures on pathology datasets, making it more accessible to the broader scientific community. To demonstrate the utility of ChampKit, we establish baseline performance for a subset of possible models that could be employed with ChampKit, focusing on several popular deep learning models, namely ResNet18, ResNet50, and R26-ViT, a hybrid vision transformer. In addition, we compare each model trained either from random weight initialization or with transfer learning from ImageNet pretrained models. For ResNet18, we also consider transfer learning from a self-supervised pretrained model.

RESULTS

The main result of this paper is the ChampKit software. Using ChampKit, we were able to systemically evaluate multiple neural networks across six datasets. We observed mixed results when evaluating the benefits of pretraining versus random intialization, with no clear benefit except in the low data regime, where transfer learning was found to be beneficial. Surprisingly, we found that transfer learning from self-supervised weights rarely improved performance, which is counter to other areas of computer vision.

CONCLUSIONS

Choosing the right model for a given digital pathology dataset is nontrivial. ChampKit provides a valuable tool to fill this gap by enabling the evaluation of hundreds of existing (or user-defined) deep learning models across a variety of pathology tasks. Source code and data for the tool are freely accessible at https://github.com/SBU-BMI/champkit.

E2EFP-MIL: End-to-end and high-generalizability weakly supervised deep convolutional network for lung cancer classification from whole slide image

Efficient and accurate distinction of histopathological subtype of lung cancer is quite critical for the individualized treatment. So far, artificial intelligence techniques have been developed, whose performance yet remained debatable on more heterogenous data, hindering their clinical deployment. Here, we propose an end-to-end, well-generalized and data-efficient weakly supervised deep learning-based method. The method, end-to-end feature pyramid deep multi-instance learning model (E2EFP-MIL), contains an iterative sampling module, a trainable feature pyramid module and a robust feature aggregation module. E2EFP-MIL uses end-to-end learning to extract generalized morphological features automatically and identify discriminative histomorphological patterns. This method is trained with 1007 whole slide images (WSIs) of lung cancer from TCGA, with AUCs of 0.95-0.97 in test sets. We validated E2EFP-MIL in 5 real-world external heterogenous cohorts including nearly 1600 WSIs from both United States and China with AUCs of 0.94-0.97, and found that 100-200 training images are enough to achieve an AUC of >0.9. E2EFP-MIL overperforms multiple state-of-the-art MIL-based methods with high accuracy and low hardware requirements. Excellent and robust results prove generalizability and effectiveness of E2EFP-MIL in clinical practice. Our code is available at https://github.com/raycaohmu/E2EFP-MIL.

Valuing vicinity: Memory attention framework for context-based semantic segmentation in histopathology

The segmentation of histopathological whole slide images into tumourous and non-tumourous types of tissue is a challenging task that requires the consideration of both local and global spatial contexts to classify tumourous regions precisely. The identification of subtypes of tumour tissue complicates the issue as the sharpness of separation decreases and the pathologist's reasoning is even more guided by spatial context. However, the identification of detailed tissue types is crucial for providing personalized cancer therapies. Due to the high resolution of whole slide images, existing semantic segmentation methods, restricted to isolated image sections, are incapable of processing context information beyond. To take a step towards better context comprehension, we propose a patch neighbour attention mechanism to query the neighbouring tissue context from a patch embedding memory bank and infuse context embeddings into bottleneck hidden feature maps. Our memory attention framework (MAF) mimics a pathologist's annotation procedure - zooming out and considering surrounding tissue context. The framework can be integrated into any encoder-decoder segmentation method. We evaluate the MAF on two public breast cancer and liver cancer data sets and an internal kidney cancer data set using famous segmentation models (U-Net, DeeplabV3) and demonstrate the superiority over other context-integrating algorithms - achieving a substantial improvement of up to 17% on Dice score. The code is publicly available at https://github.com/tio-ikim/valuing-vicinity.

Prediction of heart transplant rejection from routine pathology slides with self-supervised deep learning

Aims

One of the most important complications of heart transplantation is organ rejection, which is diagnosed on endomyocardial biopsies by pathologists. Computer-based systems could assist in the diagnostic process and potentially improve reproducibility. Here, we evaluated the feasibility of using deep learning in predicting the degree of cellular rejection from pathology slides as defined by the International Society for Heart and Lung Transplantation (ISHLT) grading system.

Methods and results

We collected 1079 histopathology slides from 325 patients from three transplant centres in Germany. We trained an attention-based deep neural network to predict rejection in the primary cohort and evaluated its performance using cross-validation and by deploying it to three cohorts. For binary prediction (rejection yes/no), the mean area under the receiver operating curve (AUROC) was 0.849 in the cross-validated experiment and 0.734, 0.729, and 0.716 in external validation cohorts. For a prediction of the ISHLT grade (0R, 1R, 2/3R), AUROCs were 0.835, 0.633, and 0.905 in the cross-validated experiment and 0.764, 0.597, and 0.913; 0.631, 0.633, and 0.682; and 0.722, 0.601, and 0.805 in the validation cohorts, respectively. The predictions of the artificial intelligence model were interpretable by human experts and highlighted plausible morphological patterns.

Conclusion

We conclude that artificial intelligence can detect patterns of cellular transplant rejection in routine pathology, even when trained on small cohorts.

Whole slide image features predict pathologic complete response and poor clinical outcomes in triple-negative breast cancer

INTRODUCTION

Breast cancers are complex ecosystem like networks of malignant cells and their associated microenvironment. Applications for machine intelligence and the tumoral microenvironment are expanding frontiers in pathology. Previously, computational approaches have been developed to quantify and spatially analyze immune cells, proportionate stroma, and detect tumor budding. Little work has been done to analyze different types of tumor-associated stromata both quantitatively and computationally in relation to clinical endpoints.

METHODS

We aimed to quantify stromal features from whole slide images (WSI) including stromata (myxoid, collagenous, immune) and tumoral components and combined them with traditional clinical and pathologic parameters in 120 triple-negative breast cancer (TNBC) patients treated with neoadjuvant chemotherapy (NAC) to predict pathologic complete response (pCR) and poor clinical outcomes.

RESULTS

High collagenous stroma on WSI was best associated with lower rates of pCR, while combined high proportionated stroma (myxoid, collagenous, and immune) most optimally predicted worse clinical survival outcomes. When combining clinical, pathologic, and WSI features, Receiver Operator Characteristics (ROC) curves for LASSO features was up to 0.67 for pCR and 0.77 for poor outcomes.

CONCLUSION

The techniques demonstrated in the present study can be performed with appropriate quality assurance. Future trials are needed to demonstrate whether coupling applications for machine intelligence, inclusive of the tumor mesenchyme, can improve outcomes prediction for patients with breast cancer.

Handcrafted Histological Transformer (H2T): Unsupervised representation of whole slide images

Diagnostic, prognostic and therapeutic decision-making of cancer in pathology clinics can now be carried out based on analysis of multi-gigapixel tissue images, also known as whole-slide images (WSIs). Recently, deep convolutional neural networks (CNNs) have been proposed to derive unsupervised WSI representations; these are attractive as they rely less on expert annotation which is cumbersome. However, a major trade-off is that higher predictive power generally comes at the cost of interpretability, posing a challenge to their clinical use where transparency in decision-making is generally expected. To address this challenge, we present a handcrafted framework based on deep CNN for constructing holistic WSI-level representations. Building on recent findings about the internal working of the Transformer in the domain of natural language processing, we break down its processes and handcraft them into a more transparent framework that we term as the Handcrafted Histological Transformer or H2T. Based on our experiments involving various datasets consisting of a total of 10,042 WSIs, the results demonstrate that H2T based holistic WSI-level representations offer competitive performance compared to recent state-of-the-art methods and can be readily utilized for various downstream analysis tasks. Finally, our results demonstrate that the H2T framework can be up to 14 times faster than the Transformer models.

Prototypical multiple instance learning for predicting lymph node metastasis of breast cancer from whole-slide pathological images

Computerized identification of lymph node metastasis of breast cancer (BCLNM) from whole-slide pathological images (WSIs) can largely benefit therapy decision and prognosis analysis. Besides the general challenges of computational pathology, like extra-high resolution, very expensive fine-grained annotation, etc., two particular difficulties with this task lie in (1) modeling the significant inter-tumoral heterogeneity in BCLNM pathological images, and (2) identifying micro-metastases, i.e., metastasized tumors with tiny foci. Towards this end, this paper presents a novel weakly supervised method, termed as Prototypical Multiple Instance Learning (PMIL), to learn to predict BCLNM from WSIs with slide-level class labels only. PMIL introduces the well-established vocabulary-based multiple instance learning (MIL) paradigm into computational pathology, which is characterized by utilizing the so-called prototypes to model pathological data and construct WSI features. PMIL mainly consists of two innovatively designed modules, i.e., the prototype discovery module which acquires prototypes from training data by unsupervised clustering, and the prototype-based slide embedding module which builds WSI features by matching constitutive patches against the prototypes. Relative to existing MIL methods for WSI classification, PMIL has two substantial merits: (1) being more explicit and interpretable in modeling the inter-tumoral heterogeneity in BCLNM pathological images, and (2) being more effective in identifying micro-metastases. Evaluation is conducted on two datasets, i.e., the public Camelyon16 dataset and the Zbraln dataset created by ourselves. PMIL achieves an AUC of 88.2% on Camelyon16 and 98.4% on Zbraln (at 40x magnification factor), which consistently outperforms other compared methods. Comprehensive analysis will also be carried out to further reveal the effectiveness and merits of the proposed method.

GraphLSurv: A scalable survival prediction network with adaptive and sparse structure learning for histopathological whole-slide images

BACKGROUND AND OBJECTIVE

Predicting patients' survival from gigapixel Whole-Slide Images (WSIs) has always been a challenging task. To learn effective WSI representations for survival prediction, existing deep learning methods have explored utilizing graphs to describe the complex structure inner WSIs, where graph node is respective to WSI patch. However, these graphs are often densely-connected or static, leading to some redundant or missing patch correlations. Moreover, these methods cannot be directly scaled to the very-large WSI with more than 10,000 patches. To address these, this paper proposes a scalable graph convolution network, GraphLSurv, which can efficiently learn adaptive and sparse structures to better characterize WSIs for survival prediction.

METHODS

GraphLSurv has three highlights in methodology: (1) it generates adaptive and sparse structures for patches so that latent patch correlations could be captured and adjusted dynamically according to prediction tasks; (2) based on the generated structure and a given graph, GraphLSurv further aggregates local microenvironmental cues into a non-local embedding using the proposed hybrid message passing network; (3) to make this network suitable for very large-scale graphs, it adopts an anchor-based technique to reduce theorical computation complexity.

RESULTS

The experiments on 2268 WSIs show that GraphLSurv achieves a concordance-index of 0.66132 and 0.68348, with an improvement of 3.79% and 3.41% compared to existing methods, on NLST and TCGA-BRCA, respectively.

CONCLUSIONS

GraphLSurv could often perform better than previous methods, which suggests that GraphLSurv could provide an important and effective means for WSI survival prediction. Moreover, this work empirically shows that adaptive and sparse structures could be more suitable than static or dense ones for modeling WSIs.

Artificial intelligence-assisted cancer diagnosis improves the efficiency of pathologists in prostatic biopsies

Paige Prostate is a clinical-grade artificial intelligence tool designed to assist the pathologist in detecting, grading, and quantifying prostate cancer. In this work, a cohort of 105 prostate core needle biopsies (CNBs) was evaluated through digital pathology. Then, we compared the diagnostic performance of four pathologists diagnosing prostatic CNB unaided and, in a second phase, assisted by Paige Prostate. In phase 1, pathologists had a diagnostic accuracy for prostate cancer of 95.00%, maintaining their performance in phase 2 (93.81%), with an intraobserver concordance rate between phases of 98.81%. In phase 2, pathologists reported atypical small acinar proliferation (ASAP) less often (about 30% less). Additionally, they requested significantly fewer immunohistochemistry (IHC) studies (about 20% less) and second opinions (about 40% less). The median time required for reading and reporting each slide was about 20% lower in phase 2, in both negative and cancer cases. Lastly, the average total agreement with the software performance was observed in about 70% of the cases, being significantly higher in negative cases (about 90%) than in cancer cases (about 30%). Most of the diagnostic discordances occurred in distinguishing negative cases with ASAP from small foci of well-differentiated (less than 1.5 mm) acinar adenocarcinoma. In conclusion, the synergic usage of Paige Prostate contributes to a significant decrease in IHC studies, second opinion requests, and time for reporting while maintaining highly accurate diagnostic standards.

The devil is in the details: a small-lesion sensitive weakly supervised learning framework for prostate cancer detection and grading

Prostate cancer (PCa) is a significant health concern in aging males, and the diagnosis depends primarily on histopathological assessments to determine tumor size and Gleason score. This process is highly time-consuming, subjective, and relies on the extensive experience of the pathologists. Deep learning based artificial intelligence shows an ability to match pathologists on many prostate cancer diagnostic scenarios. However, it is easy to make mistakes on some hard cases with small tumor areas considering the extensively high-resolution of whole slide images (WSIs). The absence of fine-grained and large-scale annotations of such small tumor lesions makes this problem more challenging. Existing methods usually perform uniform cropping of the foreground of WSI and then use convolutional neural networks as the backbone network to predict the classification results. However, cropping can damage the structure of tiny tumors, which affects classification accuracy. To solve this problem, we propose an Intensive-Sampling Multiple Instance Learning Framework (ISMIL), which focuses on tumor regions and improves the recognition of small tumor regions by intensively sampling the crucial regions. Experiments of prostate cancer detection show that our method achieves an area under the receiver operating characteristic curve (AUC) of 0.987 on the PANDA sets, which improves recall by at least 33% with higher specificity over the current primary methods for hard cases. The ISMIL also demonstrates comparable abilities to human experts on the prostate cancer grading task. Moreover, ISMIL has shown good robustness in independent cohorts, which makes it a potential tool to improve the diagnostic efficiency of pathologists.

The status of digital pathology and associated infrastructure within Alzheimer's Disease Centers

Digital pathology (DP) has transformative potential, especially for Alzheimer disease and related disorders. However, infrastructure barriers may limit adoption. To provide benchmarks and insights into implementation barriers, a survey was conducted in 2019 within National Institutes of Health's Alzheimer's Disease Centers (ADCs). Questions covered infrastructure, funding sources, and data management related to digital pathology. Of the 35 ADCs to which the survey was sent, 33 responded. Most respondents (81%) stated that their ADC had digital slide scanner access, with the most frequent brand being Aperio/Leica (62.9%). Approximately a third of respondents stated there were fees to utilize the scanner. For DP and machine learning (ML) resources, 41% of respondents stated none was supported by their ADC. For scanner purchasing and operations, 50% of respondents stated they received institutional support. Some were unsure of the file size of scanned digital images (37%) and total amount of storage space files occupied (50%). Most (76%) were aware of other departments at their institution working with ML; a similar (76%) percentage were unaware of multiuniversity or industry partnerships. These results demonstrate many ADCs have access to a digital slide scanner; additional investigations are needed to further understand hurdles to implement DP and ML workflows.

Expectation-maximization algorithm leads to domain adaptation for a perineural invasion and nerve extraction task in whole slide digital pathology images

In addition to lymphatic and vascular channels, tumor cells can also spread via nerves, i.e., perineural invasion (PNI). PNI serves as an independent prognostic indicator in many malignancies. As a result, identifying and determining the extent of PNI is an important yet extremely tedious task in surgical pathology. In this work, we present a computational approach to extract nerves and PNI from whole slide histopathology images. We make manual annotations on selected prostate cancer slides once but then apply the trained model for nerve segmentation to both prostate cancer slides and head and neck cancer slides. For the purpose of multi-domain learning/prediction and investigation on the generalization capability of deep neural network, an expectation-maximization (EM)-based domain adaptation approach is proposed to improve the segmentation performance, in particular for the head and neck cancer slides. Experiments are conducted to demonstrate the segmentation performances. The average Dice coefficient for prostate cancer slides is 0.82 and 0.79 for head and neck cancer slides. Comparisons are then made for segmentations with and without the proposed EM-based domain adaptation on prostate cancer and head and neck cancer whole slide histopathology images from The Cancer Genome Atlas (TCGA) database and significant improvements are observed.

Interpretable classification of pathology whole-slide images using attention based context-aware graph convolutional neural network

BACKGROUND AND OBJECTIVE

Whole slide image (WSI) classification and lesion localization within giga-pixel slide are challenging tasks in computational pathology that requires context-aware representations of histological features to adequately infer nidus. The existing weakly supervised learning methods mainly treat different locations in the slide as independent regions and cannot learn potential nonlinear interactions between instances based on i.i.d assumption, resulting in the model unable to effectively utilize context-ware information to predict the labels of WSIs and locate the region of interest (ROI).

METHODS

Here, we propose an interpretable classification model named bidirectional Attention-based Multiple Instance Learning Graph Convolutional Network (ABMIL-GCN), which hierarchically aggregates context-aware features of instances into a global representation in a topology fashion to predict the slide labels and localize the region of lymph node metastasis in WSIs.

RESULTS

We verified the superiority of this method on the Camelyon16 dataset, and the results show that the average predicted ACC and AUC of the proposed model after flooding optimization can reach 90.89% and 0.9149, respectively. The average accuracy and ACC score are improved by more than 7% and 4% compared with the existing state-of-the-art algorithms.

CONCLUSIONS

The results demonstrate that context-aware GCN outperforms existing weakly supervised learning methods by introducing spatial correlations between the neighbor image patches, which also addresses the 'accuracy-interpretability trade-off' problem. The framework provides a novel paradigm for the clinical application of computer-aided diagnosis and intelligent systems.

Validation of automated positive cell and region detection of immunohistochemically stained laryngeal tumor tissue using digital image analysis

Objectives

This study aimed to validate a digital image analysis (DIA) workflow for automatic positive cell detection and positive region delineation for immunohistochemical hypoxia markers with a nuclear (hypoxia-inducible factor 1α [HIF-1α]) and a cytoplasmic (pimonidazole [PIMO]) staining pattern.

Materials and methods

101 tissue fragments from 44 laryngeal tumor biopsies were immunohistochemically stained for HIF-1α and PIMO. QuPath was used to determine the percentage of positive cells and to delineate positive regions automatically. For HIF-1α, only cells with strong staining were considered positive. Three dedicated head and neck pathologists scored the percentage of positive cells using three categories (0: <1%; 1: 1%-33%; 2: >33%;). The pathologists also delineated the positive regions on 14 corresponding PIMO and HIF-1α-stained fragments. The consensus between observers was used as the reference standard and was compared to the automatic delineation.

Results

Agreement between categorical positivity scores was 76.2% and 65.4% for PIMO and HIF-1α, respectively. In all cases of disagreement in HIF-1α fragments, the DIA underestimated the percentage of positive cells. As for the region detection, the DIA correctly detected most positive regions on PIMO fragments (false positive area=3.1%, false negative area=0.7%). In HIF-1α, the DIA missed some positive regions (false positive area=1.3%, false negative area=9.7%).

Conclusions

Positive cell and region detection on biopsy material is feasible, but further optimization is needed before unsupervised use. Validation at varying DAB staining intensities is hampered by lack of reliability of the gold standard (i.e., visual human interpretation). Nevertheless, the DIA method has the potential to be used as a tool to assist pathologists in the analysis of IHC staining.