Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard

Computational pathology: A survey review and the way forward

Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.

Federated learning for medical image analysis: A survey

Machine learning in medical imaging often faces a fundamental dilemma, namely, the small sample size problem. Many recent studies suggest using multi-domain data pooled from different acquisition sites/centers to improve statistical power. However, medical images from different sites cannot be easily shared to build large datasets for model training due to privacy protection reasons. As a promising solution, federated learning, which enables collaborative training of machine learning models based on data from different sites without cross-site data sharing, has attracted considerable attention recently. In this paper, we conduct a comprehensive survey of the recent development of federated learning methods in medical image analysis. We have systematically gathered research papers on federated learning and its applications in medical image analysis published between 2017 and 2023. Our search and compilation were conducted using databases from IEEE Xplore, ACM Digital Library, Science Direct, Springer Link, Web of Science, Google Scholar, and PubMed. In this survey, we first introduce the background of federated learning for dealing with privacy protection and collaborative learning issues. We then present a comprehensive review of recent advances in federated learning methods for medical image analysis. Specifically, existing methods are categorized based on three critical aspects of a federated learning system, including client end, server end, and communication techniques. In each category, we summarize the existing federated learning methods according to specific research problems in medical image analysis and also provide insights into the motivations of different approaches. In addition, we provide a review of existing benchmark medical imaging datasets and software platforms for current federated learning research. We also conduct an experimental study to empirically evaluate typical federated learning methods for medical image analysis. This survey can help to better understand the current research status, challenges, and potential research opportunities in this promising research field.

Harnessing artificial intelligence for prostate cancer management

Prostate cancer (PCa) is a common malignancy in males. The pathology review of PCa is crucial for clinical decision-making, but traditional pathology review is labor intensive and subjective to some extent. Digital pathology and whole-slide imaging enable the application of artificial intelligence (AI) in pathology. This review highlights the success of AI in detecting and grading PCa, predicting patient outcomes, and identifying molecular subtypes. We propose that AI-based methods could collaborate with pathologists to reduce workload and assist clinicians in formulating treatment recommendations. We also introduce the general process and challenges in developing AI pathology models for PCa. Importantly, we summarize publicly available datasets and open-source codes to facilitate the utilization of existing data and the comparison of the performance of different models to improve future studies.

HistoEM: A Pathologist-Guided and Explainable Workflow Using Histogram Embedding for Gland Classification

Pathologists have, over several decades, developed criteria for diagnosing and grading prostate cancer. However, this knowledge has not, so far, been included in the design of convolutional neural networks (CNN) for prostate cancer detection and grading. Further, it is not known whether the features learned by machine-learning algorithms coincide with diagnostic features used by pathologists. We propose a framework that enforces algorithms to learn the cellular and subcellular differences between benign and cancerous prostate glands in digital slides from hematoxylin and eosin-stained tissue sections. After accurate gland segmentation and exclusion of the stroma, the central component of the pipeline, named HistoEM, utilizes a histogram embedding of features from the latent space of the CNN encoder. Each gland is represented by 128 feature-wise histograms that provide the input into a second network for benign vs cancer classification of the whole gland. Cancer glands are further processed by a U-Net structured network to separate low-grade from high-grade cancer. Our model demonstrates similar performance compared with other state-of-the-art prostate cancer grading models with gland-level resolution. To understand the features learned by HistoEM, we first rank features based on the distance between benign and cancer histograms and visualize the tissue origins of the 2 most important features. A heatmap of pixel activation by each feature is generated using Grad-CAM and overlaid on nuclear segmentation outlines. We conclude that HistoEM, similar to pathologists, uses nuclear features for the detection of prostate cancer. Altogether, this novel approach can be broadly deployed to visualize computer-learned features in histopathology images.

MMIR: an open-source software for the registration of multimodal histological images

BACKGROUND

Multimodal histology image registration is a process that transforms into a common coordinate system two or more images obtained from different microscopy modalities. The combination of information from various modalities can contribute to a comprehensive understanding of tissue specimens, aiding in more accurate diagnoses, and improved research insights. Multimodal image registration in histology samples presents a significant challenge due to the inherent differences in characteristics and the need for tailored optimization algorithms for each modality.

RESULTS

We developed MMIR a cloud-based system for multimodal histological image registration, which consists of three main modules: a project manager, an algorithm manager, and an image visualization system.

CONCLUSION

Our software solution aims to simplify image registration tasks with a user-friendly approach. It facilitates effective algorithm management, responsive web interfaces, supports multi-resolution images, and facilitates batch image registration. Moreover, its adaptable architecture allows for the integration of custom algorithms, ensuring that it aligns with the specific requirements of each modality combination. Beyond image registration, our software enables the conversion of segmented annotations from one modality to another.

Responding to the healthcare workforce shortage: A scoping review exploring anatomical pathologists' professional identities over time

Anatomical pathology (AP) is an anatomy-centric medical specialty devoted to tissue-based diagnosis of disease. The field faces a current and predicted workforce shortage, likely increasing diagnostic wait times and delaying patient access to urgent treatment. A lack of AP exposure is proposed to preclude recruitment to the field, as medical students are afforded only a limited understanding of who a pathologist is and what they do (their professional identity/PI and role). Anatomical sciences educators may be well placed to increase student understanding of anatomical pathologists' PI features, but until features of anatomical pathologists' PI are understood, recommendations for anatomy educators are premature. Thus, this scoping review asked: "What are the professional identity features of anatomical pathologists reported in the literature, and how have these changed over time?" A six-stage scoping review was performed. Medline and PubMed, Global Health, and Embase were used to identify relevant studies (n = 74). Team-based framework analysis identified that features of anatomical pathologists' professional identity encompass five overarching themes: professional practice, views about the role, training and education, personal implications, and technology. Technology was identified as an important theme of anatomical pathologists' PI, as it intersected with many other PI feature themes, including diagnosis and collaboration. This review found that pathologists may sometimes perceive professional competition with technology, such as artificial intelligence. These findings suggest unique opportunities for integrating AP-specific PI features into anatomy teaching, which may foster student interest in AP, and potentially increase recruitment into the field.

Image-based multiplex immune profiling of cancer tissues: translational implications. A report of the International Immuno-oncology Biomarker Working Group on Breast Cancer

Recent advances in the field of immuno-oncology have brought transformative changes in the management of cancer patients. The immune profile of tumours has been found to have key value in predicting disease prognosis and treatment response in various cancers. Multiplex immunohistochemistry and immunofluorescence have emerged as potent tools for the simultaneous detection of multiple protein biomarkers in a single tissue section, thereby expanding opportunities for molecular and immune profiling while preserving tissue samples. By establishing the phenotype of individual tumour cells when distributed within a mixed cell population, the identification of clinically relevant biomarkers with high-throughput multiplex immunophenotyping of tumour samples has great potential to guide appropriate treatment choices. Moreover, the emergence of novel multi-marker imaging approaches can now provide unprecedented insights into the tumour microenvironment, including the potential interplay between various cell types. However, there are significant challenges to widespread integration of these technologies in daily research and clinical practice. This review addresses the challenges and potential solutions within a structured framework of action from a regulatory and clinical trial perspective. New developments within the field of immunophenotyping using multiplexed tissue imaging platforms and associated digital pathology are also described, with a specific focus on translational implications across different subtypes of cancer. © 2024 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Binary semantic segmentation for detection of prostate adenocarcinoma using an ensemble with attention and residual U-Net architectures

An accurate determination of the Gleason Score (GS) or Gleason Pattern (GP) is crucial in the diagnosis of prostate cancer (PCa) because it is one of the criterion used to guide treatment decisions for prognostic-risk groups. However, the manually designation of GP by a pathologist using a microscope is prone to error and subject to significant inter-observer variability. Deep learning has been used to automatically differentiate GP on digitized slides, aiding pathologists and reducing inter-observer variability, especially in the early GP of cancer. This article presents a binary semantic segmentation for the GP of prostate adenocarcinoma. The segmentation separates benign and malignant tissues, with the malignant class consisting of adenocarcinoma GP3 and GP4 tissues annotated from 50 unique digitized whole slide images (WSIs) of prostate needle core biopsy specimens stained with hematoxylin and eosin. The pyramidal digitized WSIs were extracted into image patches with a size of 256 × 256 pixels at a magnification of 20×. An ensemble approach is proposed combining U-Net-based architectures, including traditional U-Net, attention-based U-Net, and residual attention-based U-Net. This work initially considers a PCa tissue analysis using a combination of attention gate units with residual convolution units. The performance evaluation revealed a mean Intersection-over-Union of 0.79 for the two classes, 0.88 for the benign class, and 0.70 for the malignant class. The proposed method was then used to produce pixel-level segmentation maps of PCa adenocarcinoma tissue slides in the testing set. We developed a screening tool to discriminate between benign and malignant prostate tissue in digitized images of needle biopsy samples using an AI approach. We aimed to identify malignant adenocarcinoma tissues from our own collected, annotated, and organized dataset. Our approach returned the performance which was accepted by the pathologists.

A machine learning approach toward automating spatial identification of LAG3+/CD3+ cells in ulcerative colitis

Over the past decade, automation of digital image analysis has become commonplace in both research and clinical settings. Spurred by recent advances in artificial intelligence and machine learning (AI/ML), tissue sub-compartments and cellular phenotypes within those compartments can be identified with higher throughput and accuracy than ever before. Recently, immune checkpoints have emerged as potential targets for auto-immune diseases. As such, spatial identification of these proteins along with immune cell markers (e.g., CD3+/LAG3+ T-cells) is a crucial step in understanding the potential and/or efficacy of such treatments. Here, we describe a semi-automated imaging and analysis pipeline that identifies CD3+/LAG3+ cells in colorectal tissue sub-compartments. While chromogenic staining has been a clinical mainstay and the resulting brightfield images have been utilized in AI/ML approaches in the past, there are associated drawbacks in phenotyping algorithms that can be overcome by fluorescence imaging. To address these tradeoffs, we developed an analysis pipeline combining the strengths of brightfield and fluorescence images. In this assay, immunofluorescence imaging was conducted to identify phenotypes followed by coverslip removal and hematoxylin and eosin staining of the same section to inform an AI/ML tissue segmentation algorithm. This assay proved to be robust in both tissue segmentation and phenotyping, was compatible with automated workflows, and revealed presence of LAG3+ T-cells in ulcerative colitis biopsies with spatial context preserved.

A systematic review and meta-analysis of artificial intelligence diagnostic accuracy in prostate cancer histology identification and grading

BACKGROUND

Artificial intelligence (AI) is a promising tool in pathology, including cancer diagnosis, subtyping, grading, and prognostic prediction.

METHODS

The aim of the study is to assess AI application in prostate cancer (PCa) histology. We carried out a systematic literature search in 3 databases. Primary outcome was AI accuracy in differentiating between PCa and benign hyperplasia. Secondary outcomes were AI accuracy in determining Gleason grade and agreement among AI and pathologists.

RESULTS

Our final sample consists of 24 studies conducted from 2007 to 2021. They aggregate data from roughly 8000 cases of prostate biopsy and 458 cases of radical prostatectomy (RP). Sensitivity for PCa diagnostic exceeded 90% and ranged from 87% to 100%, and specificity varied from 68% to 99%. Overall accuracy ranged from 83.7% to 98.3% with AUC reaching 0.99. The meta-analysis using the Mantel-Haenszel method showed pooled sensitivity of 0.96 with I2 = 80.7% and pooled specificity of 0.95 with I2 = 86.1%. Pooled positive likehood ratio was 15.3 with I2 = 87.3% and negative - was 0.04 with I2 = 78.6%. SROC (symmetric receiver operating characteristics) curve represents AUC = 0.99. For grading the accuracy of AI was lower: sensitivity for Gleason grading ranged from 77% to 87%, and specificity from 82% to 90%.

CONCLUSIONS

The accuracy of AI for PCa identification and grading is comparable to expert pathologists. This is a promising approach which has several possible clinical applications resulting in expedite and optimize pathology reports. AI introduction into common practice may be limited by difficult and time-consuming convolutional neural network training and tuning.

Pathologist Computer-Aided Diagnostic Scoring of Tumor Cell Fraction: A Swiss National Study

Tumor cell fraction (TCF) estimation is a common clinical task with well-established large interobserver variability. It thus provides an ideal test bed to evaluate potential impacts of employing a tumor cell fraction computer-aided diagnostic (TCFCAD) tool to support pathologists' evaluation. During a National Slide Seminar event, pathologists (n = 69) were asked to visually estimate TCF in 10 regions of interest (ROIs) from hematoxylin and eosin colorectal cancer images intentionally curated for diverse tissue compositions, cellularity, and stain intensities. Next, they re-evaluated the same ROIs while being provided a TCFCAD-created overlay highlighting predicted tumor vs nontumor cells, together with the corresponding TCF percentage. Participants also reported confidence levels in their assessments using a 5-tier scale, indicating no confidence to high confidence, respectively. The TCF ground truth (GT) was defined by manual cell-counting by experts. When assisted, interobserver variability significantly decreased, showing estimates converging to the GT. This improvement remained even when TCFCAD predictions deviated slightly from the GT. The standard deviation (SD) of the estimated TCF to the GT across ROIs was 9.9% vs 5.8% with TCFCAD (P < .0001). The intraclass correlation coefficient increased from 0.8 to 0.93 (95% CI, 0.65-0.93 vs 0.86-0.98), and pathologists stated feeling more confident when aided (3.67 ± 0.81 vs 4.17 ± 0.82 with the computer-aided diagnostic [CAD] tool). TCFCAD estimation support demonstrated improved scoring accuracy, interpathologist agreement, and scoring confidence. Interestingly, pathologists also expressed more willingness to use such a CAD tool at the end of the survey, highlighting the importance of training/education to increase adoption of CAD systems.

Automating Ground Truth Annotations for Gland Segmentation Through Immunohistochemistry

Microscopic evaluation of glands in the colon is of utmost importance in the diagnosis of inflammatory bowel disease and cancer. When properly trained, deep learning pipelines can provide a systematic, reproducible, and quantitative assessment of disease-related changes in glandular tissue architecture. The training and testing of deep learning models require large amounts of manual annotations, which are difficult, time-consuming, and expensive to obtain. Here, we propose a method for automated generation of ground truth in digital hematoxylin and eosin (H&E)-stained slides using immunohistochemistry (IHC) labels. The image processing pipeline generates annotations of glands in H&E histopathology images from colon biopsy specimens by transfer of gland masks from KRT8/18, CDX2, or EPCAM IHC. The IHC gland outlines are transferred to coregistered H&E images for training of deep learning models. We compared the performance of the deep learning models to that of manual annotations using an internal held-out set of biopsy specimens as well as 2 public data sets. Our results show that EPCAM IHC provides gland outlines that closely match manual gland annotations (Dice = 0.89) and are resilient to damage by inflammation. In addition, we propose a simple data sampling technique that allows models trained on data from several sources to be adapted to a new data source using just a few newly annotated samples. The best performing models achieved average Dice scores of 0.902 and 0.89 on Gland Segmentation and Colorectal Adenocarcinoma Gland colon cancer public data sets, respectively, when trained with only 10% of annotated cases from either public cohort. Altogether, the performances of our models indicate that automated annotations using cell type-specific IHC markers can safely replace manual annotations. Automated IHC labels from single-institution cohorts can be combined with small numbers of hand-annotated cases from multi-institutional cohorts to train models that generalize well to diverse data sources.

Deep learning approach for accurate prostate cancer identification and stratification using combined immunostaining of cytokeratin, p63, and racemase

BACKGROUND

Prostate cancer (PCa) is the most frequently diagnosed cancer in men worldwide, affecting around 1.4 million individuals. Current PCa diagnosis relies on histological analysis of prostate biopsy samples, an activity that is both time-consuming and prone to observer bias. Previous studies have demonstrated that immunostaining of cytokeratin, p63, and racemase can significantly improve the sensitivity and the specificity of PCa detection compared to traditional H&E staining.

METHODS

This study introduces a novel approach that combines diagnosis-specific immunohistochemical (IHC) staining and deep learning techniques to provide reliable stratification of prostate glands. Our approach leverages a customized segmentation network, called K-PPM, that incorporates adaptive kernels and multiscale feature integration to enhance the functional information of IHC. To address the high class-imbalance problem in the dataset, we propose a weighted adaptive patch-extraction and specific-class kernel update.

RESULTS

Our system achieved noteworthy results, with a mean Dice Score Coefficient of 90.36% and a mean absolute error of 1.64 % in specific-class gland quantification on whole slides. These findings demonstrate the potential of our system as a valuable support tool for pathologists, reducing workload and decreasing diagnostic inter-observer variability.

CONCLUSIONS

Our study presents innovative approaches that have broad applicability to other digital pathology areas beyond PCa diagnosis. As a fully automated system, this model can serve as a framework for improving the histological and IHC diagnosis of other types of cancer.

Deep Learning Methodologies Applied to Digital Pathology in Prostate Cancer: A Systematic Review

Deep learning (DL), often called artificial intelligence (AI), has been increasingly used in Pathology thanks to the use of scanners to digitize slides which allow us to visualize them on monitors and process them with AI algorithms. Many articles have focused on DL applied to prostate cancer (PCa). This systematic review explains the DL applications and their performances for PCa in digital pathology. Article research was performed using PubMed and Embase to collect relevant articles. A Risk of Bias (RoB) was assessed with an adaptation of the QUADAS-2 tool. Out of the 77 included studies, eight focused on pre-processing tasks such as quality assessment or staining normalization. Most articles (n = 53) focused on diagnosis tasks like cancer detection or Gleason grading. Fifteen articles focused on prediction tasks, such as recurrence prediction or genomic correlations. Best performances were reached for cancer detection with an Area Under the Curve (AUC) up to 0.99 with algorithms already available for routine diagnosis. A few biases outlined by the RoB analysis are often found in these articles, such as the lack of external validation. This review was registered on PROSPERO under CRD42023418661.

Deep learning for histopathological segmentation of smooth muscle in the urinary bladder

BACKGROUND

Histological assessment of smooth muscle is a critical step particularly in staging malignant tumors in various internal organs including  the urinary bladder. Nonetheless, manual segmentation and classification of muscular tissues by pathologists is often challenging. Therefore, a fully automated and reliable smooth muscle image segmentation system is in high demand.

METHODS

To characterize muscle fibers in the urinary bladder, including muscularis mucosa (MM) and muscularis propria (MP), we assessed 277 histological images from surgical specimens, using two well-known deep learning (DL) model groups, one including VGG16, ResNet18, SqueezeNet, and MobileNetV2, considered as a patch-based approach, and the other including U-Net, MA-Net, DeepLabv3 + , and FPN, considered as a pixel-based approach. All the trained models in both the groups were evaluated at pixel-level for their performance.

RESULTS

For segmenting MP and non-MP (including MM) regions, MobileNetV2, in the patch-based approach and U-Net, in the pixel-based approach outperformed their peers in the groups with mean Jaccard Index equal to 0.74 and 0.79, and mean Dice co-efficient equal to 0.82 and 0.88, respectively. We also demonstrated the strengths and weaknesses of the models in terms of speed and prediction accuracy.

CONCLUSIONS

This work not only creates a benchmark for future development of tools for the histological segmentation of smooth muscle but also provides an effective DL-based diagnostic system for accurate pathological staging of bladder cancer.

Composite Biomarker Image for Advanced Visualization in Histopathology

Immunohistochemistry (IHC) biomarkers are essential tools for reliable cancer diagnosis and subtyping. It requires cross-staining comparison among Whole Slide Images (WSIs) of IHCs and hematoxylin and eosin (H&E) slides. Currently, pathologists examine the visually co-localized areas across IHC and H&E glass slides for a final diagnosis, which is a tedious and challenging task. Moreover, visually inspecting different IHC slides back and forth to analyze local co-expressions is inherently subjective and prone to error, even when carried out by experienced pathologists. Relying on digital pathology, we propose "Composite Biomarker Image" (CBI) in this work. CBI is a single image that can be composed using different filtered IHC biomarker images for better visualization. We present a CBI image produced in two steps by the proposed solution for better visualization and hence more efficient clinical workflow. In the first step, IHC biomarker images are aligned with the H&E images using one coordinate system and orientation. In the second step, the positive or negative IHC regions from each biomarker image (based on the pathologists' recommendation) are filtered and combined into one image using a fuzzy inference system. For evaluation, the resulting CBI images, from the proposed system, were evaluated qualitatively by the expert pathologists. The CBI concept helps the pathologists to identify the suspected target tissues more easily, which could be further assessed by examining the actual WSIs at the same suspected regions.

Digital labeling for 3D histology: segmenting blood vessels without a vascular contrast agent using deep learning

Recent advances in optical tissue clearing and three-dimensional (3D) fluorescence microscopy have enabled high resolution in situ imaging of intact tissues. Using simply prepared samples, we demonstrate here "digital labeling," a method to segment blood vessels in 3D volumes solely based on the autofluorescence signal and a nuclei stain (DAPI). We trained a deep-learning neural network based on the U-net architecture using a regression loss instead of a commonly used segmentation loss to achieve better detection of small vessels. We achieved high vessel detection accuracy and obtained accurate vascular morphometrics such as vessel length density and orientation. In the future, such digital labeling approach could easily be transferred to other biological structures.

Classifying Malignancy in Prostate Glandular Structures from Biopsy Scans with Deep Learning

Histopathological classification in prostate cancer remains a challenge with high dependence on the expert practitioner. We develop a deep learning (DL) model to identify the most prominent Gleason pattern in a highly curated data cohort and validate it on an independent dataset. The histology images are partitioned in tiles (14,509) and are curated by an expert to identify individual glandular structures with assigned primary Gleason pattern grades. We use transfer learning and fine-tuning approaches to compare several deep neural network architectures that are trained on a corpus of camera images (ImageNet) and tuned with histology examples to be context appropriate for histopathological discrimination with small samples. In our study, the best DL network is able to discriminate cancer grade (GS3/4) from benign with an accuracy of 91%, F₁-score of 0.91 and AUC 0.96 in a baseline test (52 patients), while the cancer grade discrimination of the GS3 from GS4 had an accuracy of 68% and AUC of 0.71 (40 patients).

Applications of Artificial Intelligence in Philadelphia-Negative Myeloproliferative Neoplasms

Philadelphia-negative (Ph-) myeloproliferative neoplasms (MPNs) are a group of hematopoietic malignancies identified by clonal proliferation of blood cell lineages and encompasses polycythemia vera (PV), essential thrombocythemia (ET), and primary myelofibrosis (PMF). The clinical and laboratory features of Philadelphia-negative MPNs are similar, making them difficult to diagnose, especially in the preliminary stages. Because treatment goals and progression risk differ amongst MPNs, accurate classification and prognostication are critical for optimal management. Artificial intelligence (AI) and machine learning (ML) algorithms provide a plethora of possible tools to clinicians in general, and particularly in the field of malignant hematology, to better improve diagnosis, prognosis, therapy planning, and fundamental knowledge. In this review, we summarize the literature discussing the application of AI and ML algorithms in patients with diagnosed or suspected Philadelphia-negative MPNs. A literature search was conducted on PubMed/MEDLINE, Embase, Scopus, and Web of Science databases and yielded 125 studies, out of which 17 studies were included after screening. The included studies demonstrated the potential for the practical use of ML and AI in the diagnosis, prognosis, and genomic landscaping of patients with Philadelphia-negative MPNs.

Restaining-based annotation for cancer histology segmentation to overcome annotation-related limitations among pathologists

Numerous cancer histopathology specimens have been collected and digitized over the past few decades. A comprehensive evaluation of the distribution of various cells in tumor tissue sections can provide valuable information for understanding cancer. Deep learning is suitable for achieving these goals; however, the collection of extensive, unbiased training data is hindered, thus limiting the production of accurate segmentation models. This study presents SegPath-the largest annotation dataset (>10 times larger than publicly available annotations)-for the segmentation of hematoxylin and eosin (H&E)-stained sections for eight major cell types in cancer tissue. The SegPath generating pipeline used H&E-stained sections that were destained and subsequently immunofluorescence-stained with carefully selected antibodies. We found that SegPath is comparable with, or outperforms, pathologist annotations. Moreover, annotations by pathologists are biased toward typical morphologies. However, the model trained on SegPath can overcome this limitation. Our results provide foundational datasets for machine-learning research in histopathology.

PathNarratives: Data annotation for pathological human-AI collaborative diagnosis

Pathology is the gold standard of clinical diagnosis. Artificial intelligence (AI) in pathology becomes a new trend, but it is still not widely used due to the lack of necessary explanations for pathologists to understand the rationale. Clinic-compliant explanations besides the diagnostic decision of pathological images are essential for AI model training to provide diagnostic suggestions assisting pathologists practice. In this study, we propose a new annotation form, PathNarratives, that includes a hierarchical decision-to-reason data structure, a narrative annotation process, and a multimodal interactive annotation tool. Following PathNarratives, we recruited 8 pathologist annotators to build a colorectal pathological dataset, CR-PathNarratives, containing 174 whole-slide images (WSIs). We further experiment on the dataset with classification and captioning tasks to explore the clinical scenarios of human-AI-collaborative pathological diagnosis. The classification tasks show that fine-grain prediction enhances the overall classification accuracy from 79.56 to 85.26%. In Human-AI collaboration experience, the trust and confidence scores from 8 pathologists raised from 3.88 to 4.63 with providing more details. Results show that the classification and captioning tasks achieve better results with reason labels, provide explainable clues for doctors to understand and make the final decision and thus can support a better experience of human-AI collaboration in pathological diagnosis. In the future, we plan to optimize the tools for the annotation process, and expand the datasets with more WSIs and covering more pathological domains.

Deepfake Histologic Images for Enhancing Digital Pathology

A pathologist's optical microscopic examination of thinly cut, stained tissue on glass slides prepared from a formalin-fixed paraffin-embedded tissue blocks is the gold standard for tissue diagnostics. In addition, the diagnostic abilities and expertise of pathologists is dependent on their direct experience with common and rarer variant morphologies. Recently, deep learning approaches have been used to successfully show a high level of accuracy for such tasks. However, obtaining expert-level annotated images is an expensive and time-consuming task, and artificially synthesized histologic images can prove greatly beneficial. In this study, we present an approach to not only generate histologic images that reproduce the diagnostic morphologic features of common disease but also provide a user ability to generate new and rare morphologies. Our approach involves developing a generative adversarial network model that synthesizes pathology images constrained by class labels. We investigated the ability of this framework in synthesizing realistic prostate and colon tissue images and assessed the utility of these images in augmenting the diagnostic ability of machine learning methods and their usability by a panel of experienced anatomic pathologists. Synthetic data generated by our framework performed similar to real data when training a deep learning model for diagnosis. Pathologists were not able to distinguish between real and synthetic images, and their analyses showed a similar level of interobserver agreement for prostate cancer grading. We extended the approach to significantly more complex images from colon biopsies and showed that the morphology of the complex microenvironment in such tissues can be reproduced. Finally, we present the ability for a user to generate deepfake histologic images using a simple markup of sematic labels.

Expression site agnostic histopathology image segmentation framework by self supervised domain adaption

MOTIVATION

With the sites of antigen expression different, the segmentation of immunohistochemical (IHC) histopathology images is challenging, due to the visual variances. With H&E images highlighting the tissue structure and cell distribution more broadly, transferring more salient features from H&E images can achieve considerable performance on expression site agnostic IHC images segmentation.

METHODS

To the best of our knowledge, this is the first work that focuses on domain adaptive segmentation for different expression sites. We propose an expression site agnostic domain adaptive histopathology image semantic segmentation framework (ESASeg). In ESASeg, multi-level feature alignment encodes expression site invariance by learning generic representations of global and multi-scale local features. Moreover, self-supervision enhances domain adaptation to perceive high-level semantics by predicting pseudo-labels.

RESULTS

We construct a dataset with three IHCs (Her2 with membrane stained, Ki67 with nucleus stained, GPC3 with cytoplasm stained) with different expression sites from two diseases (breast and liver cancer). Intensive experiments on tumor region segmentation illustrate that ESASeg performs best across all metrics, and the implementation of each module proves to achieve impressive improvements.

CONCLUSION

The performance of ESASeg on the tumor region segmentation demonstrates the efficiency of the proposed framework, which provides a novel solution on expression site agnostic IHC related tasks. Moreover, the proposed domain adaption and self-supervision module can improve feature domain adaption and extraction without labels. In addition, ESASeg lays the foundation to perform joint analysis and information interaction for IHCs with different expression sites.

A Deep-Learning-Based Artificial Intelligence System for the Pathology Diagnosis of Uterine Smooth Muscle Tumor

We aimed to develop an artificial intelligence (AI) diagnosis system for uterine smooth muscle tumors (UMTs) by using deep learning. We analyzed the morphological features of UMTs on whole-slide images (233, 108, and 30 digital slides of leiomyosarcomas, leiomyomas, and smooth muscle tumors of uncertain malignant potential stained with hematoxylin and eosin, respectively). Aperio ImageScope software randomly selected ≥10 areas of the total field of view. Pathologists randomly selected a marked region in each section that was no smaller than the total area of 10 high-power fields in which necrotic, vascular, collagenous, and mitotic areas were labeled. We constructed an automatic identification algorithm for cytological atypia and necrosis by using ResNet and constructed an automatic detection algorithm for mitosis by using YOLOv5. A logical evaluation algorithm was then designed to obtain an automatic UMT diagnostic aid that can "study and synthesize" a pathologist's experience. The precision, recall, and F1 index reached more than 0.920. The detection network could accurately detect the mitoses (0.913 precision, 0.893 recall). For the prediction ability, the AI system had a precision of 0.90. An AI-assisted system for diagnosing UMTs in routine practice scenarios is feasible and can improve the accuracy and efficiency of diagnosis.

Multi-Stage Classification-Based Deep Learning for Gleason System Grading Using Histopathological Images

In this work, we introduced an automated diagnostic system for Gleason system grading and grade groups (GG) classification using whole slide images (WSIs) of digitized prostate biopsy specimens (PBSs). Our system first classifies the Gleason pattern (GP) from PBSs and then identifies the Gleason score (GS) and GG. We developed a comprehensive DL-based approach to develop a grading pipeline system for the digitized PBSs and consider GP as a classification problem (not segmentation) compared to current research studies (deals with as a segmentation problem). A multilevel binary classification was implemented to enhance the segmentation accuracy for GP. Also, we created three levels of analysis (pyramidal levels) to extract different types of features. Each level has four shallow binary CNN to classify five GP labels. A majority fusion is applied for each pixel that has a total of 39 labeled images to create the final output for GP. The proposed framework is trained, validated, and tested on 3080 WSIs of PBS. The overall diagnostic accuracy for each CNN is evaluated using several metrics: precision (PR), recall (RE), and accuracy, which are documented by the confusion matrices.The results proved our system's potential for classifying all five GP and, thus, GG. The overall accuracy for the GG is evaluated using two metrics, PR and RE. The grade GG results are between 50% to 92% for RE and 50% to 92% for PR. Also, a comparison between our CNN architecture and the standard CNN (ResNet50) highlights our system's advantage. Finally, our deep-learning system achieved an agreement with the consensus grade groups.

Computer-Assisted Annotation of Digital H&E/SOX10 Dual Stains Generates High-Performing Convolutional Neural Network for Calculating Tumor Burden in H&E-Stained Cutaneous Melanoma

Deep learning for the analysis of H&E stains requires a large annotated training set. This may form a labor-intensive task involving highly skilled pathologists. We aimed to optimize and evaluate computer-assisted annotation based on digital dual stains of the same tissue section. H&E stains of primary and metastatic melanoma (N = 77) were digitized, re-stained with SOX10, and re-scanned. Because images were aligned, annotations of SOX10 image analysis were directly transferred to H&E stains of the training set. Based on 1,221,367 annotated nuclei, a convolutional neural network for calculating tumor burden (CNN_TB) was developed. For primary melanomas, precision of annotation was 100% (95%CI, 99% to 100%) for tumor cells and 99% (95%CI, 98% to 100%) for normal cells. Due to low or missing tumor-cell SOX10 positivity, precision for normal cells was markedly reduced in lymph-node and organ metastases compared with primary melanomas (p < 0.001). Compared with stereological counts within skin lesions, mean difference in tumor burden was 6% (95%CI, -1% to 13%, p = 0.10) for CNN_TB and 16% (95%CI, 4% to 28%, p = 0.02) for pathologists. Conclusively, the technique produced a large annotated H&E training set with high quality within a reasonable timeframe for primary melanomas and subcutaneous metastases. For these lesion types, the training set generated a high-performing CNN_TB, which was superior to the routine assessments of pathologists.

Cell2Grid: an efficient, spatial, and convolutional neural network-ready representation of cell segmentation data

Purpose

Cell segmentation algorithms are commonly used to analyze large histologic images as they facilitate interpretation, but on the other hand they complicate hypothesis-free spatial analysis. Therefore, many applications train convolutional neural networks (CNNs) on high-resolution images that resolve individual cells instead, but their practical application is severely limited by computational resources. In this work, we propose and investigate an alternative spatial data representation based on cell segmentation data for direct training of CNNs.

Approach

We introduce and analyze the properties of Cell2Grid, an algorithm that generates compact images from cell segmentation data by placing individual cells into a low-resolution grid and resolves possible cell conflicts. For evaluation, we present a case study on colorectal cancer relapse prediction using fluorescent multiplex immunohistochemistry images.

Results

We could generate Cell2Grid images at 5 - μ m resolution that were 100 times smaller than the original ones. Cell features, such as phenotype counts and nearest-neighbor cell distances, remain similar to those of original cell segmentation tables ( p < 0.0001 ). These images could be directly fed to a CNN for predicting colon cancer relapse. Our experiments showed that test set error rate was reduced by 25% compared with CNNs trained on images rescaled to 5 μ m with bilinear interpolation. Compared with images at 1 - μ m resolution (bilinear rescaling), our method reduced CNN training time by 85%.

Conclusions

Cell2Grid is an efficient spatial data representation algorithm that enables the use of conventional CNNs on cell segmentation data. Its cell-based representation additionally opens a door for simplified model interpretation and synthetic image generation.

Global Research Trends of Artificial Intelligence on Histopathological Images: A 20-Year Bibliometric Analysis

Cancer has become a major threat to global health care. With the development of computer science, artificial intelligence (AI) has been widely applied in histopathological images (HI) analysis. This study analyzed the publications of AI in HI from 2001 to 2021 by bibliometrics, exploring the research status and the potential popular directions in the future. A total of 2844 publications from the Web of Science Core Collection were included in the bibliometric analysis. The country/region, institution, author, journal, keyword, and references were analyzed by using VOSviewer and CiteSpace. The results showed that the number of publications has grown rapidly in the last five years. The USA is the most productive and influential country with 937 publications and 23,010 citations, and most of the authors and institutions with higher numbers of publications and citations are from the USA. Keyword analysis showed that breast cancer, prostate cancer, colorectal cancer, and lung cancer are the tumor types of greatest concern. Co-citation analysis showed that classification and nucleus segmentation are the main research directions of AI-based HI studies. Transfer learning and self-supervised learning in HI is on the rise. This study performed the first bibliometric analysis of AI in HI from multiple indicators, providing insights for researchers to identify key cancer types and understand the research trends of AI application in HI.

Prediction of Adverse Pathology at Radical Prostatectomy in Grade Group 2 and 3 Prostate Biopsies Using Machine Learning

PURPOSE

There is ongoing clinical need to improve estimates of disease outcome in prostate cancer. Machine learning (ML) approaches to pathologic diagnosis and prognosis are a promising and increasingly used strategy. In this study, we use an ML algorithm for prediction of adverse outcomes at radical prostatectomy (RP) using whole-slide images (WSIs) of prostate biopsies with Grade Group (GG) 2 or 3 disease.

METHODS

We performed a retrospective review of prostate biopsies collected at our institution which had corresponding RP, GG 2 or 3 disease one or more cores, and no biopsies with higher than GG 3 disease. A hematoxylin and eosin-stained core needle biopsy from each site with GG 2 or 3 disease was scanned and used as the sole input for the algorithm. The ML pipeline had three phases: image preprocessing, feature extraction, and adverse outcome prediction. First, patches were extracted from each biopsy scan. Subsequently, the pre-trained Visual Geometry Group-16 convolutional neural network was used for feature extraction. A representative feature vector was then used as input to an Extreme Gradient Boosting classifier for predicting the binary adverse outcome. We subsequently assessed patient clinical risk using CAPRA score for comparison with the ML pipeline results.

RESULTS

The data set included 361 WSIs from 107 patients (56 with adverse pathology at RP). The area under the receiver operating characteristic curves for the ML classification were 0.72 (95% CI, 0.62 to 0.81), 0.65 (95% CI, 0.53 to 0.79) and 0.89 (95% CI, 0.79 to 1.00) for the entire cohort, and GG 2 and GG 3 patients, respectively, similar to the performance of the CAPRA clinical risk assessment.

CONCLUSION

We provide evidence for the potential of ML algorithms to use WSIs of needle core prostate biopsies to estimate clinically relevant prostate cancer outcomes.

A machine learning model for separating epithelial and stromal regions in oral cavity squamous cell carcinomas using H&E-stained histology images: A multi-center, retrospective study

OBJECTIVE

Tissue slides from Oral cavity squamous cell carcinoma (OC-SCC), particularly the epithelial regions, hold morphologic features that are both diagnostic and prognostic. Yet, previously developed approaches for automated epithelium segmentation in OC-SCC have not been independently tested in a multi-center setting. In this study, we aimed to investigate the effectiveness and applicability of a convolutional neural network (CNN) model to perform epithelial segmentation using digitized H&E-stained diagnostic slides from OC-SCC patients in a multi-center setting.

METHODS

A CNN model was developed to segment the epithelial regions of digitized slides (n = 810), retrospectively collected from five different centers. Deep learning models were trained and validated using well-annotated tissue microarray (TMA) images (n = 212) at various magnifications. The best performing model was locked down and used for independent testing with a total of 478 whole-slide images (WSIs). Manually annotated epithelial regions were used as the reference standard for evaluation. We also compared the model generated results with IHC-stained epithelium (n = 120) as the reference.

RESULTS

The locked-down CNN model trained on the TMA image training cohorts with 10x magnification achieved the best segmentation performance. The locked-down model performed consistently and yielded Pixel Accuracy, Recall Rate, Precision Rate, and Dice Coefficient that ranged from 95.8% to 96.6%, 79.1% to 93.8%, 85.7% to 89.3%, and 82.3% to 89.0%, respectively for the three independent testing WSI cohorts.

CONCLUSION

The automated model achieved a consistently accurate performance for automated epithelial region segmentation compared to manual annotations. This model could be integrated into a computer-aided diagnosis or prognosis system.

Weakly supervised deep learning for prediction of treatment effectiveness on ovarian cancer from histopathology images

Despite the progress made during the last two decades in the surgery and chemotherapy of ovarian cancer, more than 70 % of advanced patients are with recurrent cancer and decease. Surgical debulking of tumors following chemotherapy is the conventional treatment for advanced carcinoma, but patients with such treatment remain at great risk for recurrence and developing drug resistance, and only about 30 % of the women affected will be cured. Bevacizumab is a humanized monoclonal antibody, which blocks VEGF signaling in cancer, inhibits angiogenesis and causes tumor shrinkage, and has been recently approved by FDA as a monotherapy for advanced ovarian cancer in combination with chemotherapy. Considering the cost, potential toxicity, and finding that only a portion of patients will benefit from these drugs, the identification of new predictive method for the treatment of ovarian cancer remains an urgent unmet medical need. In this study, we develop weakly supervised deep learning approaches to accurately predict therapeutic effect for bevacizumab of ovarian cancer patients from histopathological hematoxylin and eosin stained whole slide images, without any pathologist-provided locally annotated regions. To the authors' best knowledge, this is the first model demonstrated to be effective for prediction of the therapeutic effect of patients with epithelial ovarian cancer to bevacizumab. Quantitative evaluation of a whole section dataset shows that the proposed method achieves high accuracy, 0.882 ± 0.06; precision, 0.921 ± 0.04, recall, 0.912 ± 0.03; F-measure, 0.917 ± 0.07 using 5-fold cross validation and outperforms two state-of-the art deep learning approaches Coudray et al. (2018), Campanella et al. (2019). For an independent TMA testing set, the three proposed methods obtain promising results with high recall (sensitivity) 0.946, 0.893 and 0.964, respectively. The results suggest that the proposed method could be useful for guiding treatment by assisting in filtering out patients without positive therapeutic response to suffer from further treatments while keeping patients with positive response in the treatment process. Furthermore, according to the statistical analysis of the Cox Proportional Hazards Model, patients who were predicted to be invalid by the proposed model had a very high risk of cancer recurrence (hazard ratio = 13.727) than patients predicted to be effective with statistical signifcance (p < 0.05).

A Novel Method Based on GAN Using a Segmentation Module for Oligodendroglioma Pathological Image Generation

Digital pathology analysis using deep learning has been the subject of several studies. As with other medical data, pathological data are not easily obtained. Because deep learning-based image analysis requires large amounts of data, augmentation techniques are used to increase the size of pathological datasets. This study proposes a novel method for synthesizing brain tumor pathology data using a generative model. For image synthesis, we used embedding features extracted from a segmentation module in a general generative model. We also introduce a simple solution for training a segmentation model in an environment in which the masked label of the training dataset is not supplied. As a result of this experiment, the proposed method did not make great progress in quantitative metrics but showed improved results in the confusion rate of more than 70 subjects and the quality of the visual output.

Automated Detection of Portal Fields and Central Veins in Whole-Slide Images of Liver Tissue

Many physiological processes and pathological phenomena in the liver tissue are spatially heterogeneous. At a local scale, biomarkers can be quantified along the axis of the blood flow, from portal fields (PFs) to central veins (CVs), i.e., in zonated form. This requires detecting PFs and CVs. However, manually annotating these structures in multiple whole-slide images is a tedious task. We describe and evaluate a fully automated method, based on a convolutional neural network, for simultaneously detecting PFs and CVs in a single stained section. Trained on scans of hematoxylin and eosin-stained liver tissue, the detector performed well with an F1 score of 0.81 compared to annotation by a human expert. It does, however, not generalize well to previously unseen scans of steatotic liver tissue with an F1 score of 0.59. Automated PF and CV detection eliminates the bottleneck of manual annotation for subsequent automated analyses, as illustrated by two proof-of-concept applications: We computed lobulus sizes based on the detected PF and CV positions, where results agreed with published lobulus sizes. Moreover, we demonstrate the feasibility of zonated quantification of biomarkers detected in different stainings based on lobuli and zones obtained from the detected PF and CV positions. A negative control (hematoxylin and eosin) showed the expected homogeneity, a positive control (glutamine synthetase) was quantified to be strictly pericentral, and a plausible zonation for a heterogeneous F4/80 staining was obtained. Automated detection of PFs and CVs is one building block for automatically quantifying physiologically relevant heterogeneity of liver tissue biomarkers. Perspectively, a more robust and automated assessment of zonation from whole-slide images will be valuable for parameterizing spatially resolved models of liver metabolism and to provide diagnostic information.

Prostate Cancer Risk Stratification via Nondestructive 3D Pathology with Deep Learning-Assisted Gland Analysis

Prostate cancer treatment planning is largely dependent upon examination of core-needle biopsies. The microscopic architecture of the prostate glands forms the basis for prognostic grading by pathologists. Interpretation of these convoluted three-dimensional (3D) glandular structures via visual inspection of a limited number of two-dimensional (2D) histology sections is often unreliable, which contributes to the under- and overtreatment of patients. To improve risk assessment and treatment decisions, we have developed a workflow for nondestructive 3D pathology and computational analysis of whole prostate biopsies labeled with a rapid and inexpensive fluorescent analogue of standard hematoxylin and eosin (H&E) staining. This analysis is based on interpretable glandular features and is facilitated by the development of image translation-assisted segmentation in 3D (ITAS3D). ITAS3D is a generalizable deep learning-based strategy that enables tissue microstructures to be volumetrically segmented in an annotation-free and objective (biomarker-based) manner without requiring immunolabeling. As a preliminary demonstration of the translational value of a computational 3D versus a computational 2D pathology approach, we imaged 300 ex vivo biopsies extracted from 50 archived radical prostatectomy specimens, of which, 118 biopsies contained cancer. The 3D glandular features in cancer biopsies were superior to corresponding 2D features for risk stratification of patients with low- to intermediate-risk prostate cancer based on their clinical biochemical recurrence outcomes. The results of this study support the use of computational 3D pathology for guiding the clinical management of prostate cancer. SIGNIFICANCE: An end-to-end pipeline for deep learning-assisted computational 3D histology analysis of whole prostate biopsies shows that nondestructive 3D pathology has the potential to enable superior prognostic stratification of patients with prostate cancer.

Digital pathology and artificial intelligence in translational medicine and clinical practice

Traditional pathology approaches have played an integral role in the delivery of diagnosis, semi-quantitative or qualitative assessment of protein expression, and classification of disease. Technological advances and the increased focus on precision medicine have recently paved the way for the development of digital pathology-based approaches for quantitative pathologic assessments, namely whole slide imaging and artificial intelligence (AI)-based solutions, allowing us to explore and extract information beyond human visual perception. Within the field of immuno-oncology, the application of such methodologies in drug development and translational research have created invaluable opportunities for deciphering complex pathophysiology and the discovery of novel biomarkers and drug targets. With an increasing number of treatment options available for any given disease, practitioners face the growing challenge of selecting the most appropriate treatment for each patient. The ever-increasing utilization of AI-based approaches substantially expands our understanding of the tumor microenvironment, with digital approaches to patient stratification and selection for diagnostic assays supporting the identification of the optimal treatment regimen based on patient profiles. This review provides an overview of the opportunities and limitations around implementing AI-based methods in biomarker discovery and patient selection and discusses how advances in digital pathology and AI should be considered in the current landscape of translational medicine, touching on challenges this technology may face if adopted in clinical settings. The traditional role of pathologists in delivering accurate diagnoses or assessing biomarkers for companion diagnostics may be enhanced in precision, reproducibility, and scale by AI-powered analysis tools.

Spectral decoupling for training transferable neural networks in medical imaging

Many neural networks for medical imaging generalize poorly to data unseen during training. Such behavior can be caused by overfitting easy-to-learn features while disregarding other potentially informative features. A recent implicit bias mitigation technique called spectral decoupling provably encourages neural networks to learn more features by regularizing the networks' unnormalized prediction scores with an L2 penalty. We show that spectral decoupling increases the networks′ robustness for data distribution shifts and prevents overfitting on easy-to-learn features in medical images. To validate our findings, we train networks with and without spectral decoupling to detect prostate cancer on tissue slides and COVID-19 in chest radiographs. Networks trained with spectral decoupling achieve up to 9.5 percent point higher performance on external datasets. Spectral decoupling alleviates generalization issues associated with neural networks and can be used to complement or replace computationally expensive explicit bias mitigation methods, such as stain normalization in histological images.

•

We evaluate the first implicit bias mitigation method in medical imaging

•

Spectral decoupling increases robustness for distribution shifts and shortcut learning

•

Complement or replace explicit mitigation methods, such as color normalization

•

Up to 9.5% point higher accuracy on external datasets with one line of code

Building tools for machine learning and artificial intelligence in cancer research: best practices and a case study with the PathML toolkit for computational pathology

Imaging datasets in cancer research are growing exponentially in both quantity and information density. These massive datasets may enable derivation of insights for cancer research and clinical care, but only if researchers are equipped with the tools to leverage advanced computational analysis approaches such as machine learning and artificial intelligence. In this work, we highlight three themes to guide development of such computational tools: scalability, standardization, and ease of use. We then apply these principles to develop PathML, a general-purpose research toolkit for computational pathology. We describe the design of the PathML framework and demonstrate applications in diverse use-cases. PathML is publicly available at www.pathml.com.

A Polarization-Imaging-Based Machine Learning Framework for Quantitative Pathological Diagnosis of Cervical Precancerous Lesions

Polarization images encode high resolution microstructural information even at low resolution. We propose a framework combining polarization imaging and traditional microscopy imaging, constructing a dual-modality machine learning framework that is not only accurate but also generalizable and interpretable. We demonstrate the viability of our proposed framework using the cervical intraepithelial neoplasia grading task, providing a polarimetry feature parameter to quantitatively characterize microstructural variations with lesion progression in hematoxylin-eosin-stained pathological sections of cervical precancerous tissues. By taking advantages of polarization imaging techniques and machine learning methods, the model enables interpretable and quantitative diagnosis of cervical precancerous lesion cases with improved sensitivity and accuracy in a low-resolution and wide-field system. The proposed framework applies routine image-analysis technology to identify the macro-structure and segment the target region in H&E-stained pathological images, and then employs emerging polarization method to extract the micro-structure information of the target region, which intends to expand the boundary of the current image-heavy digital pathology, bringing new possibilities for quantitative medical diagnosis.

A Classification-Guided Segmentation Algorithm Based on Deep Learning for Epithelium Segmentation in Histopathological Images of Radicular Cysts

In histopathological analysis of radicular cysts (RCs), lesions in epithelium can provide pathologists with rich information on pathologic degree, which is helpful to determine the type of periapical lesions and make precise treatment planning. Automatic segmentation and localization of epithelium from whole slide images (WSIs) can assist pathologists to complete pathological diagnosis more quickly. However, the class imbalance problem caused by the small proportion of fragmented epithelium in RCs imposes challenge on the typical automatic one-stage segmentation method. In this paper, we proposed a classification-guided segmentation algorithm (CGSA) for accurate segmentation. Our method was a two-stage model, including a classification network for region of interest (ROI) location and a segmentation network guided by classification. The classification stage eliminated most irrelevant areas and alleviated the class imbalance problem faced by the segmentation model. The results of 5-fold cross validation demonstrated that CGSA outperformed the one-stage segmentation method which was lacking in prior epithelium localization information. The epithelium segmentation achieved an overall Dice's coefficient of 0.722, and intersection over union (IoU) of 0.593, which improved by 5.5% and 5.9% respectively compared with the one-stage segmentation method using UNet.Clinical Relevance- This work presents a framework for automatic epithelium segmentation in histopathological images of RCs. It can be applied to make up for the shortcomings of manual annotation which is labor-intensive, time-consuming and objective.

Artificial Intelligence-Based Segmentation of Residual Tumor in Histopathology of Pancreatic Cancer after Neoadjuvant Treatment

Simple Summary

The use of neoadjuvant therapy (NAT) in patients with pancreatic ductal adenocarcinoma (PDAC) is increasing. Objective quantification of the histopathological response to NAT may be used to guide adjuvant treatment and compare the efficacy of neoadjuvant regimens. However, current tumor response scoring (TRS) systems suffer from interobserver variability, originating from subjective definitions, the sometimes challenging histology, and response heterogeneity throughout the tumor bed. This study investigates if artificial intelligence-based segmentation of residual tumor burden in histopathology of PDAC after NAT may offer a more objective and reproducible TRS solution.

Abstract

Background: Histologic examination of resected pancreatic cancer after neoadjuvant therapy (NAT) is used to assess the effect of NAT and may guide the choice for adjuvant treatment. However, evaluating residual tumor burden in pancreatic cancer is challenging given tumor response heterogeneity and challenging histomorphology. Artificial intelligence techniques may offer a more reproducible approach. Methods: From 64 patients, one H&E-stained slide of resected pancreatic cancer after NAT was digitized. Three separate classes were manually outlined in each slide (i.e., tumor, normal ducts, and remaining epithelium). Corresponding segmentation masks and patches were generated and distributed over training, validation, and test sets. Modified U-nets with varying encoders were trained, and F1 scores were obtained to express segmentation accuracy. Results: The highest mean segmentation accuracy was obtained using modified U-nets with a DenseNet161 encoder. Tumor tissue was segmented with a high mean F1 score of 0.86, while the overall multiclass average F1 score was 0.82. Conclusions: This study shows that artificial intelligence-based assessment of residual tumor burden is feasible given the promising obtained F1 scores for tumor segmentation. This model could be developed into a tool for the objective evaluation of the response to NAT and may potentially guide the choice for adjuvant treatment.

Deep learning-based virtual cytokeratin staining of gastric carcinomas to measure tumor-stroma ratio

The tumor-stroma ratio (TSR) determined by pathologists is subject to intra- and inter-observer variability. We aimed to develop a computational quantification method of TSR using deep learning-based virtual cytokeratin staining algorithms. Patients with 373 advanced (stage III [n = 171] and IV [n = 202]) gastric cancers were analyzed for TSR. Moderate agreement was observed, with a kappa value of 0.623, between deep learning metrics (dTSR) and visual measurement by pathologists (vTSR) and the area under the curve of receiver operating characteristic of 0.907. Moreover, dTSR was significantly associated with the overall survival of the patients (P = 0.0024). In conclusion, we developed a virtual cytokeratin staining and deep learning-based TSR measurement, which may aid in the diagnosis of TSR in gastric cancer.

Domain Adaptation-Based Deep Learning for Automated Tumor Cell (TC) Scoring and Survival Analysis on PD-L1 Stained Tissue Images

We report the ability of two deep learning-based decision systems to stratify non-small cell lung cancer (NSCLC) patients treated with checkpoint inhibitor therapy into two distinct survival groups. Both systems analyze functional and morphological properties of epithelial regions in digital histopathology whole slide images stained with the SP263 PD-L1 antibody. The first system learns to replicate the pathologist assessment of the Tumor Cell (TC) score with a cut-point for positivity at 25% for patient stratification. The second system is free from assumptions related to TC scoring and directly learns patient stratification from the overall survival time and event information. Both systems are built on a novel unpaired domain adaptation deep learning solution for epithelial region segmentation. This approach significantly reduces the need for large pixel-precise manually annotated datasets while superseding serial sectioning or re-staining of slides to obtain ground truth by cytokeratin staining. The capacity of the first system to replicate the TC scoring by pathologists is evaluated on 703 unseen cases, with an addition of 97 cases from an independent cohort. Our results show Lin's concordance values of 0.93 and 0.96 against pathologist scoring, respectively. The ability of the first and second system to stratify anti-PD-L1 treated patients is evaluated on 151 clinical samples. Both systems show similar stratification powers (first system: HR = 0.539, p = 0.004 and second system: HR = 0.525, p = 0.003) compared to TC scoring by pathologists (HR = 0.574, p = 0.01).

DeepHistoClass: A Novel Strategy for Confident Classification of Immunohistochemistry Images Using Deep Learning

A multitude of efforts worldwide aim to create a single-cell reference map of the human body, for fundamental understanding of human health, molecular medicine, and targeted treatment. Antibody-based proteomics using immunohistochemistry (IHC) has proven to be an excellent technology for integration with large-scale single-cell transcriptomics datasets. The golden standard for evaluation of IHC staining patterns is manual annotation, which is expensive and may lead to subjective errors. Artificial intelligence holds much promise for efficient and accurate pattern recognition, but confidence in prediction needs to be addressed. Here, the aim was to present a reliable and comprehensive framework for automated annotation of IHC images. We developed a multilabel classification of 7848 complex IHC images of human testis corresponding to 2794 unique proteins, generated as part of the Human Protein Atlas (HPA) project. Manual annotation data for eight different cell types was generated as a basis for training and testing a proposed Hybrid Bayesian Neural Network. By combining the deep learning model with a novel uncertainty metric, DeepHistoClass (DHC) Confidence Score, the average diagnostic performance improved from 86.9% to 96.3%. This metric not only reveals which images are reliably classified by the model, but can also be utilized for identification of manual annotation errors. The proposed streamlined workflow can be developed further for other tissue types in health and disease and has important implications for digital pathology initiatives or large-scale protein mapping efforts such as the HPA project.

•

A novel method for automated annotation of immunohistochemistry images.

•

Introduction of an uncertainty metric, the DeepHistoClass (DHC) confidence score.

•

Increased accuracy of automated image predictions.

•

Identification of manual annotation errors.

Tissue Multiplex Analyte Detection in Anatomic Pathology - Pathways to Clinical Implementation

Background: Multiplex tissue analysis has revolutionized our understanding of the tumor microenvironment (TME) with implications for biomarker development and diagnostic testing. Multiplex labeling is used for specific clinical situations, but there remain barriers to expanded use in anatomic pathology practice.

Methods: We review immunohistochemistry (IHC) and related assays used to localize molecules in tissues, with reference to United States regulatory and practice landscapes. We review multiplex methods and strategies used in clinical diagnosis and in research, particularly in immuno-oncology. Within the framework of assay design and testing phases, we examine the suitability of multiplex immunofluorescence (mIF) for clinical diagnostic workflows, considering its advantages and challenges to implementation.

Results: Multiplex labeling is poised to radically transform pathologic diagnosis because it can answer questions about tissue-level biology and single-cell phenotypes that cannot be addressed with traditional IHC biomarker panels. Widespread implementation will require improved detection chemistry, illustrated by InSituPlex technology (Ultivue, Inc., Cambridge, MA) that allows coregistration of hematoxylin and eosin (H&E) and mIF images, greater standardization and interoperability of workflow and data pipelines to facilitate consistent interpretation by pathologists, and integration of multichannel images into digital pathology whole slide imaging (WSI) systems, including interpretation aided by artificial intelligence (AI). Adoption will also be facilitated by evidence that justifies incorporation into clinical practice, an ability to navigate regulatory pathways, and adequate health care budgets and reimbursement. We expand the brightfield WSI system “pixel pathway” concept to multiplex workflows, suggesting that adoption might be accelerated by data standardization centered on cell phenotypes defined by coexpression of multiple molecules.

Conclusion: Multiplex labeling has the potential to complement next generation sequencing in cancer diagnosis by allowing pathologists to visualize and understand every cell in a tissue biopsy slide. Until mIF reagents, digital pathology systems including fluorescence scanners, and data pipelines are standardized, we propose that diagnostic labs will play a crucial role in driving adoption of multiplex tissue diagnostics by using retrospective data from tissue collections as a foundation for laboratory-developed test (LDT) implementation and use in prospective trials as companion diagnostics (CDx).

Automated Quantification of sTIL Density with H&E-Based Digital Image Analysis Has Prognostic Potential in Triple-Negative Breast Cancers

Triple-negative breast cancer (TNBC) is an aggressive and difficult-to-treat cancer type that represents approximately 15% of all breast cancers. Recently, stromal tumor-infiltrating lymphocytes (sTIL) resurfaced as a strong prognostic biomarker for overall survival (OS) for TNBC patients. Manual assessment has innate limitations that hinder clinical adoption, and the International Immuno-Oncology Biomarker Working Group (TIL-WG) has therefore envisioned that computational assessment of sTIL could overcome these limitations and recommended that any algorithm should follow the manual guidelines where appropriate. However, no existing studies capture all the concepts of the guideline or have shown the same prognostic evidence as manual assessment. In this study, we present a fully automated digital image analysis pipeline and demonstrate that our hematoxylin and eosin (H&E)-based pipeline can provide a quantitative and interpretable score that correlates with the manual pathologist-derived sTIL status, and importantly, can stratify a retrospective cohort into two significant distinct prognostic groups. We found our score to be prognostic for OS (HR: 0.81 CI: 0.72-0.92 p = 0.001) independent of age, tumor size, nodal status, and tumor type in statistical modeling. While prior studies have followed fragments of the TIL-WG guideline, our approach is the first to follow all complex aspects, where appropriate, supporting the TIL-WG vision of computational assessment of sTIL in the future clinical setting.

How can artificial intelligence models assist PD-L1 expression scoring in breast cancer: results of multi-institutional ring studies

Programmed death ligand-1 (PD-L1) expression is a key biomarker to screen patients for PD-1/PD-L1-targeted immunotherapy. However, a subjective assessment guide on PD-L1 expression of tumor-infiltrating immune cells (IC) scoring is currently adopted in clinical practice with low concordance. Therefore, a repeatable and quantifiable PD-L1 IC scoring method of breast cancer is desirable. In this study, we propose a deep learning-based artificial intelligence-assisted (AI-assisted) model for PD-L1 IC scoring. Three rounds of ring studies (RSs) involving 31 pathologists from 10 hospitals were carried out, using the current guideline in the first two rounds (RS1, RS2) and our AI scoring model in the last round (RS3). A total of 109 PD-L1 (Ventana SP142) immunohistochemistry (IHC) stained images were assessed and the role of the AI-assisted model was evaluated. With the assistance of AI, the scoring concordance across pathologists was boosted to excellent in RS3 (0.950, 95% confidence interval (CI): 0.936-0.962) from moderate in RS1 (0.674, 95% CI: 0.614-0.735) and RS2 (0.736, 95% CI: 0.683-0.789). The 2- and 4-category scoring accuracy were improved by 4.2% (0.959, 95% CI: 0.953-0.964) and 13% (0.815, 95% CI: 0.803-0.827) (p < 0.001). The AI results were generally accepted by pathologists with 61% "fully accepted" and 91% "almost accepted". The proposed AI-assisted method can help pathologists at all levels to improve the PD-L1 assay (SP-142) IC assessment in breast cancer in terms of both accuracy and concordance. The AI tool provides a scheme to standardize the PD-L1 IC scoring in clinical practice.

Artificial intelligence-based morphological fingerprinting of megakaryocytes: a new tool for assessing disease in MPN patients

Accurate diagnosis and classification of myeloproliferative neoplasms (MPNs) requires integration of clinical, morphological, and genetic findings. Despite major advances in our understanding of the molecular and genetic basis of MPNs, the morphological assessment of bone marrow trephines (BMT) is critical in differentiating MPN subtypes and their reactive mimics. However, morphological assessment is heavily constrained by a reliance on subjective, qualitative, and poorly reproducible criteria. To improve the morphological assessment of MPNs, we have developed a machine learning approach for the automated identification, quantitative analysis, and abstract representation of megakaryocyte features using reactive/nonneoplastic BMT samples (n = 43) and those from patients with established diagnoses of essential thrombocythemia (n = 45), polycythemia vera (n = 18), or myelofibrosis (n = 25). We describe the application of an automated workflow for the identification and delineation of relevant histological features from routinely prepared BMTs. Subsequent analysis enabled the tissue diagnosis of MPN with a high predictive accuracy (area under the curve = 0.95) and revealed clear evidence of the potential to discriminate between important MPN subtypes. Our method of visually representing abstracted megakaryocyte features in the context of analyzed patient cohorts facilitates the interpretation and monitoring of samples in a manner that is beyond conventional approaches. The automated BMT phenotyping approach described here has significant potential as an adjunct to standard genetic and molecular testing in established or suspected MPN patients, either as part of the routine diagnostic pathway or in the assessment of disease progression/response to treatment.

Deep learning in histopathology: the path to the clinic

Machine learning techniques have great potential to improve medical diagnostics, offering ways to improve accuracy, reproducibility and speed, and to ease workloads for clinicians. In the field of histopathology, deep learning algorithms have been developed that perform similarly to trained pathologists for tasks such as tumor detection and grading. However, despite these promising results, very few algorithms have reached clinical implementation, challenging the balance between hope and hype for these new techniques. This Review provides an overview of the current state of the field, as well as describing the challenges that still need to be addressed before artificial intelligence in histopathology can achieve clinical value.