Weakly supervised bilayer convolutional network in segmentation of HER2 related cells to guide HER2 targeted therapies

Ensemble transformer-based multiple instance learning to predict pathological subtypes and tumor mutational burden from histopathological whole slide images of endometrial and colorectal cancer

In endometrial cancer (EC) and colorectal cancer (CRC), in addition to microsatellite instability, tumor mutational burden (TMB) has gradually gained attention as a genomic biomarker that can be used clinically to determine which patients may benefit from immune checkpoint inhibitors. High TMB is characterized by a large number of mutated genes, which encode aberrant tumor neoantigens, and implies a better response to immunotherapy. Hence, a part of EC and CRC patients associated with high TMB may have higher chances to receive immunotherapy. TMB measurement was mainly evaluated by whole-exome sequencing or next-generation sequencing, which was costly and difficult to be widely applied in all clinical cases. Therefore, an effective, efficient, low-cost and easily accessible tool is urgently needed to distinguish the TMB status of EC and CRC patients. In this study, we present a deep learning framework, namely Ensemble Transformer-based Multiple Instance Learning with Self-Supervised Learning Vision Transformer feature encoder (ETMIL-SSLViT), to predict pathological subtype and TMB status directly from the H&E stained whole slide images (WSIs) in EC and CRC patients, which is helpful for both pathological classification and cancer treatment planning. Our framework was evaluated on two different cancer cohorts, including an EC cohort with 918 histopathology WSIs from 529 patients and a CRC cohort with 1495 WSIs from 594 patients from The Cancer Genome Atlas. The experimental results show that the proposed methods achieved excellent performance and outperforming seven state-of-the-art (SOTA) methods in cancer subtype classification and TMB prediction on both cancer datasets. Fisher's exact test further validated that the associations between the predictions of the proposed models and the actual cancer subtype or TMB status are both extremely strong (p<0.001). These promising findings show the potential of our proposed methods to guide personalized treatment decisions by accurately predicting the EC and CRC subtype and the TMB status for effective immunotherapy planning for EC and CRC patients.

Dual attention model with reinforcement learning for classification of histology whole-slide images

Digital whole slide images (WSIs) are generally captured at microscopic resolution and encompass extensive spatial data (several billions of pixels per image). Directly feeding these images to deep learning models is computationally intractable due to memory constraints, while downsampling the WSIs risks incurring information loss. Alternatively, splitting the WSIs into smaller patches (or tiles) may result in a loss of important contextual information. In this paper, we propose a novel dual attention approach, consisting of two main components, both inspired by the visual examination process of a pathologist: The first soft attention model processes a low magnification view of the WSI to identify relevant regions of interest (ROIs), followed by a custom sampling method to extract diverse and spatially distinct image tiles from the selected ROIs. The second component, the hard attention classification model further extracts a sequence of multi-resolution glimpses from each tile for classification. Since hard attention is non-differentiable, we train this component using reinforcement learning to predict the location of the glimpses. This approach allows the model to focus on essential regions instead of processing the entire tile, thereby aligning with a pathologist's way of diagnosis. The two components are trained in an end-to-end fashion using a joint loss function to demonstrate the efficacy of the model. The proposed model was evaluated on two WSI-level classification problems: Human epidermal growth factor receptor 2 (HER2) scoring on breast cancer histology images and prediction of Intact/Loss status of two Mismatch Repair (MMR) biomarkers from colorectal cancer histology images. We show that the proposed model achieves performance better than or comparable to the state-of-the-art methods while processing less than 10% of the WSI at the highest magnification and reducing the time required to infer the WSI-level label by more than 75%. The code is available at github.

Performance of a HER2 testing algorithm tailored for urothelial bladder cancer: A Bi-centre study

Aims

This study aimed to develop an AI algorithm for automated HER2 scoring in urothelial bladder cancer (UBCa) and assess the interobserver agreement using both manual and AI-assisted methods based on breast cancer criteria.

Methods and Results

We utilized 330 slides from two institutions for initial AI development and selected 200 slides for the ring study, involving six pathologists (3 senior, 3 junior). Our AI algorithm achieved high accuracy in two independent tests, with accuracies of 0.94 and 0.92. In the ring study, the AI-assisted method improved both accuracy (0.66 vs 0.94) and consistency (kappa=0.48; 95 % CI, 0.443-0.526 vs kappa=0.87; 95 % CI, 0.852-0.885) compared to manual scoring, especially in HER2-low cases (F1-scores: 0.63 vs 0.92). Additionally, in 62.3 % of heterogeneous HER2-positive cases, the interpretation accuracy significantly improved (0.49 vs 0.93). Pathologists, particularly junior ones, experienced enhanced accuracy and consistency with AI assistance.

Conclusions

This is the first study to provide a quantification algorithm for HER2 scoring in UBCa to assist pathologists in diagnosis. The ring study demonstrated that HER2 scoring based on breast cancer criteria can be effectively applied to UBCa. Furthermore, AI assistance significantly improves the accuracy and consistency of interpretations among pathologists with varying levels of experience, even in heterogeneous cases.

Deep learning to assess microsatellite instability directly from histopathological whole slide images in endometrial cancer

Molecular classification, particularly microsatellite instability-high (MSI-H), has gained attention for immunotherapy in endometrial cancer (EC). MSI-H is associated with DNA mismatch repair defects and is a crucial treatment predictor. The NCCN guidelines recommend pembrolizumab and nivolumab for advanced or recurrent MSI-H/mismatch repair deficient (dMMR) EC. However, evaluating MSI in all cases is impractical due to time and cost constraints. To overcome this challenge, we present an effective and efficient deep learning-based model designed to accurately and rapidly assess MSI status of EC using H&E-stained whole slide images. Our framework was evaluated on a comprehensive dataset of gigapixel histopathology images of 529 patients from the Cancer Genome Atlas (TCGA). The experimental results have shown that the proposed method achieved excellent performances in assessing MSI status, obtaining remarkably high results with 96%, 94%, 93% and 100% for endometrioid carcinoma G1G2, respectively, and 87%, 84%, 81% and 94% for endometrioid carcinoma G3, in terms of F-measure, accuracy, precision and sensitivity, respectively. Furthermore, the proposed deep learning framework outperforms four state-of-the-art benchmarked methods by a significant margin (p < 0.001) in terms of accuracy, precision, sensitivity and F-measure, respectively. Additionally, a run time analysis demonstrates that the proposed method achieves excellent quantitative results with high efficiency in AI inference time (1.03 seconds per slide), making the proposed framework viable for practical clinical usage. These results highlight the efficacy and efficiency of the proposed model to assess MSI status of EC directly from histopathological slides.

HoLy-Net: Segmentation of histological images of diffuse large B-cell lymphoma

Over the last years, there has been large progress in automated segmentation and classification methods in histological whole slide images (WSIs) stained with hematoxylin and eosin (H&E). Current state-of-the-art (SOTA) techniques are based on diverse datasets of H&E-stained WSIs of different types of predominantly solid cancer. However, there is a scarcity of methods and datasets enabling segmentation of tumors of the lymphatic system (lymphomas). Here, we propose a solution for segmentation of diffuse large B-cell lymphoma (DLBCL), the most common non-Hodgkin's lymphoma. Our method applies to both H&E-stained slides and to a broad range of markers stained with immunohistochemistry (IHC). While IHC staining is an important tool in cancer diagnosis and treatment decisions, there are few automated segmentation and classification methods for IHC-stained WSIs. To address the challenges of nuclei segmentation in H&E- and IHC-stained DLBCL images, we propose HoLy-Net - a HoVer-Net-based deep learning model for lymphoma segmentation. We train two different models, one for segmenting H&E- and one for IHC-stained images and compare the test results with the SOTA methods as well as with the original version of HoVer-Net. Subsequently, we segment patient WSIs and perform single cell-level analysis of different cell types to identify patient-specific tumor characteristics such as high level of immune infiltration. Our method outperforms general-purpose segmentation methods for H&E staining in lymphoma WSIs (with an F1 score of 0.899) and is also a unique automated method for IHC slide segmentation (with an F1 score of 0.913). With our solution, we provide a new dataset we denote LyNSeC (lymphoma nuclear segmentation and classification) containing 73,931 annotated cell nuclei from H&E and 87,316 from IHC slides. Our method and dataset open up new avenues for quantitative, large-scale studies of morphology and microenvironment of lymphomas overlooked by the current automated segmentation methods.