Using deep convolutional neural networks to identify and classify tumor-associated stroma in diagnostic breast biopsies

Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review

BACKGROUND

Over the past decade, the development of molecular high-throughput methods (omics) increased rapidly and provided new insights for cancer research. In parallel, deep learning approaches revealed the enormous potential for medical image analysis, especially in digital pathology. Combining image and omics data with deep learning tools may enable the discovery of new cancer biomarkers and a more precise prediction of patient prognosis. This systematic review addresses different multimodal fusion methods of convolutional neural network-based image analyses with omics data, focussing on the impact of data combination on the classification performance.

METHODS

PubMed was screened for peer-reviewed articles published in English between January 2015 and June 2021 by two independent researchers. Search terms related to deep learning, digital pathology, omics, and multimodal fusion were combined.

RESULTS

We identified a total of 11 studies meeting the inclusion criteria, namely studies that used convolutional neural networks for haematoxylin and eosin image analysis of patients with cancer in combination with integrated omics data. Publications were categorised according to their endpoints: 7 studies focused on survival analysis and 4 studies on prediction of cancer subtypes, malignancy or microsatellite instability with spatial analysis.

CONCLUSIONS

Image-based classifiers already show high performances in prognostic and predictive cancer diagnostics. The integration of omics data led to improved performance in all studies described here. However, these are very early studies that still require external validation to demonstrate their generalisability and robustness. Further and more comprehensive studies with larger sample sizes are needed to evaluate performance and determine clinical benefits.

Mammary collagen architecture and its association with mammographic density and lesion severity among women undergoing image-guided breast biopsy

Background

Elevated mammographic breast density is a strong breast cancer risk factor with poorly understood etiology. Increased deposition of collagen, one of the main fibrous proteins present in breast stroma, has been associated with increased mammographic density. Collagen fiber architecture has been linked to poor outcomes in breast cancer. However, relationships of quantitative collagen fiber features assessed in diagnostic biopsies with mammographic density and lesion severity are not well-established.

Methods

Clinically indicated breast biopsies from 65 in situ or invasive breast cancer cases and 73 frequency matched-controls with a benign biopsy result were used to measure collagen fiber features (length, straightness, width, alignment, orientation and density (fibers/µm²)) using second harmonic generation microscopy in up to three regions of interest (ROIs) per biopsy: normal, benign breast disease, and cancer. Local and global mammographic density volumes were quantified in the ipsilateral breast in pre-biopsy full-field digital mammograms. Associations of fibrillar collagen features with mammographic density and severity of biopsy diagnosis were evaluated using generalized estimating equation models with an independent correlation structure to account for multiple ROIs within each biopsy section.

Results

Collagen fiber density was positively associated with the proportion of stroma on the biopsy slide (p < 0.001) and with local percent mammographic density volume at both the biopsy target (p = 0.035) and within a 2 mm perilesional ring (p = 0.02), but not with global mammographic density measures. As severity of the breast biopsy diagnosis increased at the ROI level, collagen fibers tended to be less dense, shorter, straighter, thinner, and more aligned with one another (p < 0.05).

Conclusions

Collagen fiber density was positively associated with local, but not global, mammographic density, suggesting that collagen microarchitecture may not translate into macroscopic mammographic features. However, collagen fiber features may be markers of cancer risk and/or progression among women referred for biopsy based on abnormal breast imaging.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13058-021-01482-z.

Convolutional neural network for human cancer types prediction by integrating protein interaction networks and omics data

Many studies have proven the power of gene expression profile in cancer identification, however, the explosive growth of genomics data increasing needs of tools for cancer diagnosis and prognosis in high accuracy and short times. Here, we collected 6136 human samples from 11 cancer types, and integrated their gene expression profiles and protein-protein interaction (PPI) network to generate 2D images with spectral clustering method. To predict normal samples and 11 cancer tumor types, the images of these 6136 human cancer network were separated into training and validation dataset to develop convolutional neural network (CNN). Our model showed 97.4% and 95.4% accuracies in identification of normal versus tumors and 11 cancer types, respectively. We also provided the results that tumors located in neighboring tissues or in the same cell types, would induce machine make error classification due to the similar gene expression profiles. Furthermore, we observed some patients may exhibit better prognosis if their tumors often misjudged into normal samples. As far as we know, we are the first to generate thousands of cancer networks to predict and classify multiple cancer types with CNN architecture. We believe that our model not only can be applied to cancer diagnosis and prognosis, but also promote the discovery of multiple cancer biomarkers.

Deep learning in cancer diagnosis, prognosis and treatment selection

Deep learning is a subdiscipline of artificial intelligence that uses a machine learning technique called artificial neural networks to extract patterns and make predictions from large data sets. The increasing adoption of deep learning across healthcare domains together with the availability of highly characterised cancer datasets has accelerated research into the utility of deep learning in the analysis of the complex biology of cancer. While early results are promising, this is a rapidly evolving field with new knowledge emerging in both cancer biology and deep learning. In this review, we provide an overview of emerging deep learning techniques and how they are being applied to oncology. We focus on the deep learning applications for omics data types, including genomic, methylation and transcriptomic data, as well as histopathology-based genomic inference, and provide perspectives on how the different data types can be integrated to develop decision support tools. We provide specific examples of how deep learning may be applied in cancer diagnosis, prognosis and treatment management. We also assess the current limitations and challenges for the application of deep learning in precision oncology, including the lack of phenotypically rich data and the need for more explainable deep learning models. Finally, we conclude with a discussion of how current obstacles can be overcome to enable future clinical utilisation of deep learning.

Machine Learning and Artificial Intelligence-driven Spatial Analysis of the Tumor Immune Microenvironment in Pathology Slides

A better understanding of the tumor immune microenvironment (TIME) could lead to accurate diagnosis, prognosis, and treatment stratification. Although molecular analyses at the tissue and/or single cell level could reveal the cellular status of the tumor microenvironment, these approaches lack information related to spatial-level cellular distribution, co-organization, and cell-cell interaction in the TIME. With the emergence of computational pathology coupled with machine learning (ML) and artificial intelligence (AI), ML- and AI-driven spatial TIME analyses of pathology images could revolutionize our understanding of the highly heterogeneous and complex molecular architecture of the TIME. In this review we highlight recent studies on spatial TIME analysis of pathology slides using state-of-the-art ML and AI algorithms. PATIENT SUMMARY: This mini-review reports recent advances in machine learning and artificial intelligence for spatial analysis of the tumor immune microenvironment in pathology slides. This information can help in understanding the spatial heterogeneity and organization of cells in patient tumors.

Quantitative Image Analysis for Tissue Biomarker Use: A White Paper From the Digital Pathology Association

Tissue biomarkers have been of increasing utility for scientific research, diagnosing disease, and treatment response prediction. There has been a steady shift away from qualitative assessment toward providing more quantitative scores for these biomarkers. The application of quantitative image analysis has thus become an indispensable tool for in-depth tissue biomarker interrogation in these contexts. This white paper reviews current technologies being employed for quantitative image analysis, their application and pitfalls, regulatory framework demands, and guidelines established for promoting their safe adoption in clinical practice.

Artificial Intelligence for solid tumor diagnosis in digital pathology

Tumor diagnosis relies on the visual examination of histological slides by pathologists through a microscope eyepiece. Digital pathology, the digitalization of histological slides at high magnification with slides scanners, has raised the opportunity to extract quantitative information thanks to image analysis. In the last decade, medical image analysis has made exceptional progress due to the development of artificial intelligence (AI) algorithms. AI has been successfully used in the field of medical imaging and more recently in digital pathology. The feasibility and usefulness of AI assisted pathology tasks have been demonstrated in the very last years and we can expect those developments to be applied on routine histopathology in the future. In this review, we will describe and illustrate this technique and present the most recent applications in the field of tumor histopathology.

A Computational Tumor-Infiltrating Lymphocyte Assessment Method Comparable with Visual Reporting Guidelines for Triple-Negative Breast Cancer

Background

Tumor-infiltrating lymphocytes (TILs) are clinically significant in triple-negative breast cancer (TNBC). Although a standardized methodology for visual TILs assessment (VTA) exists, it has several inherent limitations. We established a deep learning-based computational TIL assessment (CTA) method broadly following VTA guideline and compared it with VTA for TNBC to determine the prognostic value of the CTA and a reasonable CTA workflow for clinical practice.

Methods

We trained three deep neural networks for nuclei segmentation, nuclei classification and necrosis classification to establish a CTA workflow. The automatic TIL (aTIL) score generated was compared with manual TIL (mTIL) scores provided by three pathologists in an Asian (n = 184) and a Caucasian (n = 117) TNBC cohort to evaluate scoring concordance and prognostic value.

Findings

The intraclass correlations (ICCs) between aTILs and mTILs varied from 0.40 to 0.70 in two cohorts. Multivariate Cox proportional hazards analysis revealed that the aTIL score was associated with disease free survival (DFS) in both cohorts, as either a continuous [hazard ratio (HR)=0.96, 95% CI 0.94–0.99] or dichotomous variable (HR=0.29, 95% CI 0.12–0.72). A higher C-index was observed in a composite mTIL/aTIL three-tier stratification model than in the dichotomous model, using either mTILs or aTILs alone.

Interpretation

The current study provides a useful tool for stromal TIL assessment and prognosis evaluation for patients with TNBC. A workflow integrating both VTA and CTA may aid pathologists in performing risk management and decision-making tasks.

Deep learning for the standardized classification of Ki-67 in vulva carcinoma: A feasibility study

Background

The aim of this study is to demonstrate the feasibility of automatic classification of Ki-67 histological immunostainings in patients with squamous cell carcinoma of the vulva using a deep convolutional neural network (dCNN).

Material and methods

For evaluation of the dCNN, we used 55 well characterized squamous cell carcinomas of the vulva in a tissue microarray (TMA) format in this retrospective study. The tumor specimens were classified in 3 different categories C1 (0-2%), C2 (2-20%) and C3 (>20%), representing the relation of the number of KI-67 positive tumor cells to all cancer cells on the TMA spot. Representative areas of the spots were manually labeled by extracting images of 351 × 280 pixels. A dCNN with 13 convolutional layers was used for the evaluation. Two independent pathologists classified 45 labeled images in order to compare the dCNN's results to human readouts.

Results

Using a small labeled dataset with 1020 images with equal distribution among classes, the dCNN reached an accuracy of 90.9% (93%) for the training (validation) data. Applying a larger dataset with additional 1017 labeled images resulted in an accuracy of 96.1% (91.4%) for the training (validation) dataset. For the human readout, there were no significant differences between the pathologists and the dCNN in Ki-67 classification results.

Conclusion

The dCNN is capable of a standardized classification of Ki-67 staining in vulva carcinoma; therefore, it may be suitable for quality control and standardization in the assessment of tumor grading.

Deep Learning Techniques for the Classification of Colorectal Cancer Tissue

It is very important to make an objective evaluation of colorectal cancer histological images. Current approaches are generally based on the use of different combinations of textual features and classifiers to assess the classification performance, or transfer learning to classify different organizational types. However, since histological images contain multiple tissue types and characteristics, classification is still challenging. In this study, we proposed the best classification methodology based on the selected optimizer and modified the parameters of CNN methods. Then, we used deep learning technology to distinguish between healthy and diseased large intestine tissues. Firstly, we trained a neural network and compared the network architecture optimizers. Secondly, we modified the parameters of the network layer to optimize the superior architecture. Finally, we compared our well-trained deep learning methods on two different histological image open datasets, which comprised 5000 H&E images of colorectal cancer. The other dataset was composed of nine organizational categories of 100,000 images with an external validation of 7180 images. The results showed that the accuracy of the recognition of histopathological images was significantly better than that of existing methods. Therefore, this method is expected to have great potential to assist physicians to make clinical diagnoses and reduce the number of disparate assessments based on the use of artificial intelligence to classify colorectal cancer tissue.

Mitotic nuclei analysis in breast cancer histopathology images using deep ensemble classifier

Mitotic nuclei estimation in breast tumour samples has a prognostic significance in analysing tumour aggressiveness and grading system. The automated assessment of mitotic nuclei is challenging because of their high similarity with non-mitotic nuclei and heteromorphic appearance. In this work, we have proposed a new Deep Convolutional Neural Network (CNN) based Heterogeneous Ensemble technique "DHE-Mit-Classifier" for analysis of mitotic nuclei in breast histopathology images. The proposed technique in the first step detects candidate mitotic patches from the histopathological biopsy regions, whereas, in the second step, these patches are classified into mitotic and non-mitotic nuclei using the proposed DHE-Mit-Classifier. For the development of a heterogeneous ensemble, five different deep CNNs are designed and used as base-classifiers. These deep CNNs have varying architectural designs to capture the structural, textural, and morphological properties of the mitotic nuclei. The developed base-classifiers exploit different ideas, including (i) region homogeneity and feature invariance, (ii) asymmetric split-transform-merge, (iii) dilated convolution based multi-scale transformation, (iv) spatial and channel attention, and (v) residual learning. Multi-layer-perceptron is used as a meta-classifier to develop a robust and accurate classifier for providing the final decision. The performance of the proposed ensemble "DHE-Mit-Classifier" is evaluated against state-of-the-art CNNs. The performance evaluation on the test set suggests the superiority of the proposed ensemble with an F-score (0.77), recall (0.71), precision (0.83), and area under the precision-recall curve (0.80). The good generalisation of the proposed ensemble with a considerably high F-score and precision suggests its potential use in the development of an assistance tool for pathologists.

Machine learning-based image analysis for accelerating the diagnosis of complicated preneoplastic and neoplastic ductal lesions in breast biopsy tissues

PURPOSE

Diagnosis of breast preneoplastic and neoplastic lesions is difficult due to their similar morphology in breast biopsy specimens. To diagnose these lesions, pathologists perform immunohistochemical analysis and consult with expert breast pathologists. These additional examinations are time-consuming and expensive. Artificial intelligence (AI)-based image analysis has recently improved, and may help in ordinal pathological diagnosis. Here, we showed the significance of machine learning-based image analysis of breast preneoplastic and neoplastic lesions for facilitating high-throughput diagnosis.

METHODS

Images were obtained from normal mammary glands, hyperplastic lesions, preneoplastic lesions and neoplastic lesions, such as usual ductal hyperplasia (UDH), columnar cell lesion (CCL), ductal carcinoma in situ (DCIS), and DCIS with comedo necrosis (comedo DCIS) in breast biopsy specimens. The original enhanced convoluted neural network (CNN) system was used for analyzing the pathological images.

RESULTS

The AI-based image analysis provided the following area under the curve values (AUC): normal lesion versus DCIS, 0.9902; DCIS versus comedo DCIS, 0.9942; normal lesion versus CCL, 0.9786; and UDH versus DCIS, 1.000. Multiple comparison analysis showed precision and recall scores similar to those of single comparison analysis. Based on the gradient-weighted class activation mapping (Grad-CAM) used to visualize the important regions reflecting the result of CNN analysis, the ratio of stromal tissue in the whole weighted area was significantly higher in UDH and CCL than that in DCIS.

CONCLUSIONS

These analyses may provide a more accurate and rapid pathological diagnosis of patients. Moreover, Grad-CAM identifies uncharted important histological characteristics for newer pathological findings and targets of research for understanding diseases.

Improving DCIS diagnosis and predictive outcome by applying artificial intelligence

Breast ductal carcinoma in situ (DCIS) is a preinvasive lesion that is considered to be a precursor to invasive breast cancer. Nevertheless, not all DCIS will progress to invasion. Current histopathological classification systems are unable to predict which cases will or will not progress, and therefore many women with DCIS may be overtreated. Artificial intelligence (AI) image-based analysis methods have potential to identify and analyze novel features that may facilitate tumor identification, prediction of disease outcome and response to treatment. Indeed, these methods prove promising for accurately identifying DCIS lesions, and show potential clinical utility in the therapeutic stratification of DCIS patients. Here, we review how AI techniques in histopathology may aid diagnosis and clinical decisions in regards to DCIS, and how such techniques could be incorporated into clinical practice.

Melanoma Prognosis: Accuracy of the American Joint Committee on Cancer Staging Manual Eighth Edition

BACKGROUND

The American Joint Committee on Cancer (AJCC) maintains that the eighth edition of its Staging Manual (AJCC8) has improved accuracy compared with the seventh (AJCC7). However, there are concerns that implementation may disrupt analysis of active clinical trials for stage III patients. We used an independent cohort of melanoma patients to test the extent to which AJCC8 has improved prognostic accuracy compared with AJCC7.

METHODS

We analyzed a cohort of 1315 prospectively enrolled patients. We assessed primary tumor and nodal classification of stage I-III patients using AJCC7 and AJCC8 to assign disease stages at diagnosis. We compared recurrence-free (RFS) and overall survival (OS) using Kaplan-Meier curves and log-rank tests. We then compared concordance indices of discriminatory prognostic ability and area under the curve of 5-year survival to predict RFS and OS. All statistical tests were two-sided.

RESULTS

Stage IIC patients continued to have worse outcomes than stage IIIA patients, with a 5-year RFS of 26.5% (95% confidence interval [CI] = 12.8% to 55.1%) vs 56.0% (95% CI = 37.0% to 84.7%) by AJCC8 (P = .002). For stage I, removing mitotic index as a T classification factor decreased its prognostic value, although not statistically significantly (RFS concordance index [C-index] = 0.63, 95% CI = 0.56 to 0.69; to 0.56, 95% CI = 0.49 to 0.63, P = .07; OS C-index = 0.48, 95% CI = 0.38 to 0.58; to 0.48, 95% CI = 0.41 to 0.56, P = .90). For stage II, prognostication remained constant (RFS C-index = 0.65, 95% CI = 0.57 to 0.72; OS C-index = 0.61, 95% CI = 0.50 to 0.72), and for stage III, AJCC8 yielded statistically significantly enhanced prognostication for RFS (C-index = 0.65, 95% CI = 0.60 to 0.70; to 0.70, 95% CI = 0.66 to 0.75, P = .01).

CONCLUSIONS

Compared with AJCC7, we demonstrate that AJCC8 enables more accurate prognosis for patients with stage III melanoma. Restaging a large cohort of patients can enhance the analysis of active clinical trials.

Automated identification of glomeruli and synchronised review of special stains in renal biopsies by machine learning and slide registration: a cross-institutional study

AIMS

Machine learning in digital pathology can improve efficiency and accuracy via prescreening with automated feature identification. Studies using uniform histological material have shown promise. Generalised application requires validation on slides from multiple institutions. We used machine learning to identify glomeruli on renal biopsies and compared performance between single and multiple institutions.

METHODS AND RESULTS

Randomly selected, adequately sampled renal core biopsy cases (71) consisting of four stains each (haematoxylin and eosin, trichrome, silver, periodic acid Schiff) from three institutions were digitised at ×40. Glomeruli were manually annotated by three renal pathologists using a digital tool. Cases were divided into training/validation (n = 52) and evaluation (n = 19) cohorts. An algorithm was trained to develop three convolutional neural network (CNN) models which tested case cohorts intra- and inter-institutionally. Raw CNN search data from each of the four slides per case were merged into composite regions of interest containing putative glomeruli. The sensitivity and modified specificity of glomerulus detection (versus annotated truth) were calculated for each model/cohort. Intra-institutional (3) sensitivity ranged from 90 to 93%, with modified specificity from 86 to 98%. Interinstitutional (1) sensitivity was 77%, with modified specificity 97%. Combined intra- and inter-institutional (1) sensitivity was 86%, with modified specificity 92%.

CONCLUSIONS

Feature detection sensitivity degrades when training and test material originate from different sites. Training using a combined set of digital slides from three institutions improves performance. Differing histology methods probably account for algorithm performance contrasts. Our data highlight the need for diverse training sets for the development of generalisable machine learning histology algorithms.

Artificial Intelligence in Cancer Research and Precision Medicine

Artificial intelligence (AI) is rapidly reshaping cancer research and personalized clinical care. Availability of high-dimensionality datasets coupled with advances in high-performance computing, as well as innovative deep learning architectures, has led to an explosion of AI use in various aspects of oncology research. These applications range from detection and classification of cancer, to molecular characterization of tumors and their microenvironment, to drug discovery and repurposing, to predicting treatment outcomes for patients. As these advances start penetrating the clinic, we foresee a shifting paradigm in cancer care becoming strongly driven by AI. SIGNIFICANCE: AI has the potential to dramatically affect nearly all aspects of oncology-from enhancing diagnosis to personalizing treatment and discovering novel anticancer drugs. Here, we review the recent enormous progress in the application of AI to oncology, highlight limitations and pitfalls, and chart a path for adoption of AI in the cancer clinic.

Deep learning-based grading of ductal carcinoma in situ in breast histopathology images

Ductal carcinoma in situ (DCIS) is a non-invasive breast cancer that can progress into invasive ductal carcinoma (IDC). Studies suggest DCIS is often overtreated since a considerable part of DCIS lesions may never progress into IDC. Lower grade lesions have a lower progression speed and risk, possibly allowing treatment de-escalation. However, studies show significant inter-observer variation in DCIS grading. Automated image analysis may provide an objective solution to address high subjectivity of DCIS grading by pathologists. In this study, we developed and evaluated a deep learning-based DCIS grading system. The system was developed using the consensus DCIS grade of three expert observers on a dataset of 1186 DCIS lesions from 59 patients. The inter-observer agreement, measured by quadratic weighted Cohen's kappa, was used to evaluate the system and compare its performance to that of expert observers. We present an analysis of the lesion-level and patient-level inter-observer agreement on an independent test set of 1001 lesions from 50 patients. The deep learning system (dl) achieved on average slightly higher inter-observer agreement to the three observers (o1, o2 and o3) (κo1,dl = 0.81, κo2,dl = 0.53 and κo3,dl = 0.40) than the observers amongst each other (κo1,o2 = 0.58, κo1,o3 = 0.50 and κo2,o3 = 0.42) at the lesion-level. At the patient-level, the deep learning system achieved similar agreement to the observers (κo1,dl = 0.77, κo2,dl = 0.75 and κo3,dl = 0.70) as the observers amongst each other (κo1,o2 = 0.77, κo1,o3 = 0.75 and κo2,o3 = 0.72). The deep learning system better reflected the grading spectrum of DCIS than two of the observers. In conclusion, we developed a deep learning-based DCIS grading system that achieved a performance similar to expert observers. To the best of our knowledge, this is the first automated system for the grading of DCIS that could assist pathologists by providing robust and reproducible second opinions on DCIS grade.

Two Ensemble-CNN Approaches for Colorectal Cancer Tissue Type Classification

In recent years, automatic tissue phenotyping has attracted increasing interest in the Digital Pathology (DP) field. For Colorectal Cancer (CRC), tissue phenotyping can diagnose the cancer and differentiate between different cancer grades. The development of Whole Slide Images (WSIs) has provided the required data for creating automatic tissue phenotyping systems. In this paper, we study different hand-crafted feature-based and deep learning methods using two popular multi-classes CRC-tissue-type databases: Kather-CRC-2016 and CRC-TP. For the hand-crafted features, we use two texture descriptors (LPQ and BSIF) and their combination. In addition, two classifiers are used (SVM and NN) to classify the texture features into distinct CRC tissue types. For the deep learning methods, we evaluate four Convolutional Neural Network (CNN) architectures (ResNet-101, ResNeXt-50, Inception-v3, and DenseNet-161). Moreover, we propose two Ensemble CNN approaches: Mean-Ensemble-CNN and NN-Ensemble-CNN. The experimental results show that the proposed approaches outperformed the hand-crafted feature-based methods, CNN architectures and the state-of-the-art methods in both databases.

Histological Severity Risk Factors Identification in Juvenile-Onset Recurrent Respiratory Papillomatosis: How Immunohistochemistry and AI Algorithms Can Help?

Juvenile-onset recurrent respiratory papillomatosis (JoRRP) is a condition characterized by the repeated growth of benign exophytic papilloma in the respiratory tract. The course of the disease remains unpredictable: some children experience minor symptoms, while others require multiple interventions due to florid growth. Our study aimed to identify histologic severity risk factors in patients with JoRRP. Forty-eight children from two French pediatric centers were included retrospectively. Criteria for a severe disease were: annual rate of surgical endoscopy ≥ 5, spread to the lung, carcinomatous transformation or death. We conducted a multi-stage study with image analysis. First, with Hematoxylin and eosin (HE) digital slides of papilloma, we searched for morphological patterns associated with a severe JoRRP using a deep-learning algorithm. Then, immunohistochemistry with antibody against p53 and p63 was performed on sections of FFPE samples of laryngeal papilloma obtained between 2008 and 2018. Immunostainings were quantified according to the staining intensity through two automated workflows: one using machine learning, the other using deep learning. Twenty-four patients had severe disease. For the HE analysis, no significative results were obtained with cross-validation. For immunostaining with anti-p63 antibody, we found similar results between the two image analysis methods. Using machine learning, we found 23.98% of stained nuclei for medium intensity for mild JoRRP vs. 36.1% for severe JoRRP (p = 0.041); and for medium and strong intensity together, 24.14% for mild JoRRP vs. 36.9% for severe JoRRP (p = 0.048). Using deep learning, we found 58.32% for mild JoRRP vs. 67.45% for severe JoRRP (p = 0.045) for medium and strong intensity together. Regarding p53, we did not find any significant difference in the number of nuclei stained between the two groups of patients. In conclusion, we highlighted that immunochemistry with the anti-p63 antibody is a potential biomarker to predict the severity of the JoRRP.

Deep Multi-Magnification Networks for multi-class breast cancer image segmentation

Pathologic analysis of surgical excision specimens for breast carcinoma is important to evaluate the completeness of surgical excision and has implications for future treatment. This analysis is performed manually by pathologists reviewing histologic slides prepared from formalin-fixed tissue. In this paper, we present Deep Multi-Magnification Network trained by partial annotation for automated multi-class tissue segmentation by a set of patches from multiple magnifications in digitized whole slide images. Our proposed architecture with multi-encoder, multi-decoder, and multi-concatenation outperforms other single and multi-magnification-based architectures by achieving the highest mean intersection-over-union, and can be used to facilitate pathologists' assessments of breast cancer.

Tumor Microenvironment and the Role of Artificial Intelligence in Breast Cancer Detection and Prognosis

A critical knowledge gap has been noted in breast cancer detection, prognosis, and evaluation between tumor microenvironment and associated neoplasm. Artificial intelligence (AI) has multiple subsets or methods for data extraction and evaluation, including artificial neural networking, which allows computational foundations, similar to neurons, to make connections and new neural pathways during data set training. Deep machine learning and AI hold great potential to accurately assess tumor microenvironment models employing vast data management techniques. Despite the significant potential AI holds, there is still much debate surrounding the appropriate and ethical curation of medical data from picture archiving and communication systems. AI output's clinical significance depends on its human predecessor's data training sets. Integration between biomarkers, risk factors, and imaging data will allow the best predictor models for patient-based outcomes.

Deriving Polarimetry Feature Parameters to Characterize Microstructural Features in Histological Sections of Breast Tissues

OBJECTIVE

Mueller matrix polarimetry technique has been regarded as a powerful tool for probing the microstructural information of tissues. The multiplying of cells and remodeling of collagen fibers in breast carcinoma tissues have been reported to be related to patient survival and prognosis, and they give rise to observable patterns in hematoxylin and eosin (H&E) sections of typical breast tissues (TBTs) that the pathologist can label as three distinctive pathological features (DPFs)-cell nuclei, aligned collagen, and disorganized collagen. The aim of this paper is to propose a pixel-based extraction approach of polarimetry feature parameters (PFPs) using a linear discriminant analysis (LDA) classifier. These parameters provide quantitative characterization of the three DPFs in four types of TBTs.

METHODS

The LDA-based training method learns to find the most simplified linear combination from polarimetry basis parameters (PBPs) constrained under the accuracy remains constant to characterize the specific microstructural feature quantitatively in TBTs.

RESULTS

We present results from a cohort of 32 clinical patients with analysis of 224 regions-of-interest. The characterization accuracy for PFPs ranges from 0.82 to 0.91.

CONCLUSION

This work demonstrates the ability of PFPs to quantitatively characterize the DPFs in the H&E pathological sections of TBTs.

SIGNIFICANCE

This technique paves the way for automatic and quantitative evaluation of specific microstructural features in histopathological digitalization and computer-aided diagnosis.

Aggregation of cohorts for histopathological diagnosis with deep morphological analysis

There have been substantial efforts in using deep learning (DL) to diagnose cancer from digital images of pathology slides. Existing algorithms typically operate by training deep neural networks either specialized in specific cohorts or an aggregate of all cohorts when there are only a few images available for the target cohort. A trade-off between decreasing the number of models and their cancer detection performance was evident in our experiments with The Cancer Genomic Atlas dataset, with the former approach achieving higher performance at the cost of having to acquire large datasets from the cohort of interest. Constructing annotated datasets for individual cohorts is extremely time-consuming, with the acquisition cost of such datasets growing linearly with the number of cohorts. Another issue associated with developing cohort-specific models is the difficulty of maintenance: all cohort-specific models may need to be adjusted when a new DL algorithm is to be used, where training even a single model may require a non-negligible amount of computation, or when more data is added to some cohorts. In resolving the sub-optimal behavior of a universal cancer detection model trained on an aggregate of cohorts, we investigated how cohorts can be grouped to augment a dataset without increasing the number of models linearly with the number of cohorts. This study introduces several metrics which measure the morphological similarities between cohort pairs and demonstrates how the metrics can be used to control the trade-off between performance and the number of models.

Introduction to Artificial Intelligence and Machine Learning for Pathology

CONTEXT.—

Recent developments in machine learning have stimulated intense interest in software that may augment or replace human experts. Machine learning may impact pathology practice by offering new capabilities in analysis, interpretation, and outcomes prediction using images and other data. The principles of operation and management of machine learning systems are unfamiliar to pathologists, who anticipate a need for additional education to be effective as expert users and managers of the new tools.

OBJECTIVE.—

To provide a background on machine learning for practicing pathologists, including an overview of algorithms, model development, and performance evaluation; to examine the current status of machine learning in pathology and consider possible roles and requirements for pathologists in local deployment and management of machine learning systems; and to highlight existing challenges and gaps in deployment methodology and regulation.

DATA SOURCES.—

Sources include the biomedical and engineering literature, white papers from professional organizations, government reports, electronic resources, and authors' experience in machine learning. References were chosen when possible for accessibility to practicing pathologists without specialized training in mathematics, statistics, or software development.

CONCLUSIONS.—

Machine learning offers an array of techniques that in recent published results show substantial promise. Data suggest that human experts working with machine learning tools outperform humans or machines separately, but the optimal form for this combination in pathology has not been established. Significant questions related to the generalizability of machine learning systems, local site verification, and performance monitoring remain to be resolved before a consensus on best practices and a regulatory environment can be established.

Deep neural network models for computational histopathology: A survey

Histopathological images contain rich phenotypic information that can be used to monitor underlying mechanisms contributing to disease progression and patient survival outcomes. Recently, deep learning has become the mainstream methodological choice for analyzing and interpreting histology images. In this paper, we present a comprehensive review of state-of-the-art deep learning approaches that have been used in the context of histopathological image analysis. From the survey of over 130 papers, we review the field's progress based on the methodological aspect of different machine learning strategies such as supervised, weakly supervised, unsupervised, transfer learning and various other sub-variants of these methods. We also provide an overview of deep learning based survival models that are applicable for disease-specific prognosis tasks. Finally, we summarize several existing open datasets and highlight critical challenges and limitations with current deep learning approaches, along with possible avenues for future research.

Deep learning-enabled breast cancer hormonal receptor status determination from base-level H&E stains

For newly diagnosed breast cancer, estrogen receptor status (ERS) is a key molecular marker used for prognosis and treatment decisions. During clinical management, ERS is determined by pathologists from immunohistochemistry (IHC) staining of biopsied tissue for the targeted receptor, which highlights the presence of cellular surface antigens. This is an expensive, time-consuming process which introduces discordance in results due to variability in IHC preparation and pathologist subjectivity. In contrast, hematoxylin and eosin (H&E) staining—which highlights cellular morphology—is quick, less expensive, and less variable in preparation. Here we show that machine learning can determine molecular marker status, as assessed by hormone receptors, directly from cellular morphology. We develop a multiple instance learning-based deep neural network that determines ERS from H&E-stained whole slide images (WSI). Our algorithm—trained strictly with WSI-level annotations—is accurate on a varied, multi-country dataset of 3,474 patients, achieving an area under the curve (AUC) of 0.92 for sensitivity and specificity. Our approach has the potential to augment clinicians’ capabilities in cancer prognosis and theragnosis by harnessing biological signals imperceptible to the human eye.

Determination of estrogen receptor status (ERS) in breast cancer tissue requires immunohistochemistry, which is sensitive to the vagaries of sample processing and the subjectivity of pathologists. Here the authors present a deep learning model that determines ERS from H&E stained tissue, which could improve oncology decisions in under-resourced settings.

Deep-learning convolutional neural networks with transfer learning accurately classify COVID-19 lung infection on portable chest radiographs

Portable chest X-ray (pCXR) has become an indispensable tool in the management of Coronavirus Disease 2019 (COVID-19) lung infection. This study employed deep-learning convolutional neural networks to classify COVID-19 lung infections on pCXR from normal and related lung infections to potentially enable more timely and accurate diagnosis. This retrospect study employed deep-learning convolutional neural network (CNN) with transfer learning to classify based on pCXRs COVID-19 pneumonia (N = 455) on pCXR from normal (N = 532), bacterial pneumonia (N = 492), and non-COVID viral pneumonia (N = 552). The data was randomly split into 75% training and 25% testing, randomly. A five-fold cross-validation was used for the testing set separately. Performance was evaluated using receiver-operating curve analysis. Comparison was made with CNN operated on the whole pCXR and segmented lungs. CNN accurately classified COVID-19 pCXR from those of normal, bacterial pneumonia, and non-COVID-19 viral pneumonia patients in a multiclass model. The overall sensitivity, specificity, accuracy, and AUC were 0.79, 0.93, and 0.79, 0.85 respectively (whole pCXR), and were 0.91, 0.93, 0.88, and 0.89 (CXR of segmented lung). The performance was generally better using segmented lungs. Heatmaps showed that CNN accurately localized areas of hazy appearance, ground glass opacity and/or consolidation on the pCXR. Deep-learning convolutional neural network with transfer learning accurately classifies COVID-19 on portable chest X-ray against normal, bacterial pneumonia or non-COVID viral pneumonia. This approach has the potential to help radiologists and frontline physicians by providing more timely and accurate diagnosis.

Reimagining T Staging Through Artificial Intelligence and Machine Learning Image Processing Approaches in Digital Pathology

Tumor stage and grade, visually assessed by pathologists from evaluation of pathology images in conjunction with radiographic imaging techniques, have been linked to outcome, progression, and survival for a number of cancers. The gold standard of staging in oncology has been the TNM (tumor-node-metastasis) staging system. Though histopathological grading has shown prognostic significance, it is subjective and limited by interobserver variability even among experienced surgical pathologists. Recently, artificial intelligence (AI) approaches have been applied to pathology images toward diagnostic-, prognostic-, and treatment prediction-related tasks in cancer. AI approaches have the potential to overcome the limitations of conventional TNM staging and tumor grading approaches, providing a direct prognostic prediction of disease outcome independent of tumor stage and grade. Broadly speaking, these AI approaches involve extracting patterns from images that are then compared against previously defined disease signatures. These patterns are typically categorized as either (1) handcrafted, which involve domain-inspired attributes, such as nuclear shape, or (2) deep learning (DL)-based representations, which tend to be more abstract. DL approaches have particularly gained considerable popularity because of the minimal domain knowledge needed for training, mostly only requiring annotated examples corresponding to the categories of interest. In this article, we discuss AI approaches for digital pathology, especially as they relate to disease prognosis, prediction of genomic and molecular alterations in the tumor, and prediction of treatment response in oncology. We also discuss some of the potential challenges with validation, interpretability, and reimbursement that must be addressed before widespread clinical deployment. The article concludes with a brief discussion of potential future opportunities in the field of AI for digital pathology and oncology.

Fine-Grained Breast Cancer Classification With Bilinear Convolutional Neural Networks (BCNNs)

Classification of histopathological images of cancer is challenging even for well-trained professionals, due to the fine-grained variability of the disease. Deep Convolutional Neural Networks (CNNs) showed great potential for classification of a number of the highly variable fine-grained objects. In this study, we introduce a Bilinear Convolutional Neural Networks (BCNNs) based deep learning method for fine-grained classification of breast cancer histopathological images. We evaluated our model by comparison with several deep learning algorithms for fine-grained classification. We used bilinear pooling to aggregate a large number of orderless features without taking into consideration the disease location. The experimental results on BreaKHis, a publicly available breast cancer dataset, showed that our method is highly accurate with 99.24% and 95.95% accuracy in binary and in fine-grained classification, respectively.

Virtual Molecular Projections and Convolutional Neural Networks for the End-to-End Modeling of Nanoparticle Activities and Properties

Digitalizing complex nanostructures into data structures suitable for machine learning modeling without losing nanostructure information has been a major challenge. Deep learning frameworks, particularly convolutional neural networks (CNNs), are especially adept at handling multidimensional and complex inputs. In this study, CNNs were applied for the modeling of nanoparticle activities exclusively from nanostructures. The nanostructures were represented by virtual molecular projections, a multidimensional digitalization of nanostructures, and used as input data to train CNNs. To this end, 77 nanoparticles with various activities and/or physicochemical property results were used for modeling. The resulting CNN model predictions show high correlations with the experimental results. An analysis of a trained CNN quantitatively showed that neurons were able to recognize distinct nanostructure features critical to activities and physicochemical properties. This "end-to-end" deep learning approach is well suited to digitalize complex nanostructures for data-driven machine learning modeling and can be broadly applied to rationally design nanoparticles with desired activities.

Integration of geoscience frameworks into digital pathology analysis permits quantification of microarchitectural relationships in histological landscapes

Although gold-standard histological assessment is subjective it remains central to diagnosis and clinical trial protocols and is crucial for the evaluation of any preclinical disease model. Objectivity and reproducibility are enhanced by quantitative analysis of histological images but current methods require application-specific algorithm training and fail to extract understanding from the histological context of observable features. We reinterpret histopathological images as disease landscapes to describe a generalisable framework defining topographic relationships in tissue using geoscience approaches. The framework requires no user-dependent training to operate on all image datasets in a classifier-agnostic manner but is adaptable and scalable, able to quantify occult abnormalities, derive mechanistic insights, and define a new feature class for machine-learning diagnostic classification. We demonstrate application to inflammatory, fibrotic and neoplastic disease in multiple organs, including the detection and quantification of occult lobular enlargement in the liver secondary to hilar obstruction. We anticipate this approach will provide a robust class of histological data for trial stratification or endpoints, provide quantitative endorsement of experimental models of disease, and could be incorporated within advanced approaches to clinical diagnostic pathology.

Interobserver Variability in Ductal Carcinoma In Situ of the Breast

OBJECTIVES

Since most patients with ductal carcinoma in situ (DCIS) of the breast are treated upon diagnosis, evidence on its natural progression to invasive carcinoma is limited. It is estimated that around half of the screen-detected DCIS lesions would have remained indolent if they had never been detected. Many patients with DCIS are therefore probably overtreated. Four ongoing randomized noninferiority trials explore active surveillance as a treatment option. Eligibility for these trials is mainly based on histopathologic features. Hence, the call for reproducible histopathologic assessment has never sounded louder.

METHODS

Here, the available classification systems for DCIS are discussed in depth.

RESULTS

This comprehensive review illustrates that histopathologic evaluation of DCIS is characterized by significant interobserver variability. Future digitalization of pathology, combined with development of deep learning algorithms or so-called artificial intelligence, may be an innovative solution to tackle this problem. However, implementation of digital pathology is not within reach for each laboratory worldwide. An alternative classification system could reduce the disagreement among histopathologists who use "conventional" light microscopy: the introduction of dichotomous histopathologic assessment is likely to increase interobserver concordance.

CONCLUSIONS

Reproducible histopathologic assessment is a prerequisite for robust risk stratification and adequate clinical decision-making. Two-tier histopathologic assessment might enhance the quality of care.

Multiplex Cellular Communities in Multi-Gigapixel Colorectal Cancer Histology Images for Tissue Phenotyping

In computational pathology, automated tissue phenotyping in cancer histology images is a fundamental tool for profiling tumor microenvironments. Current tissue phenotyping methods use features derived from image patches which may not carry biological significance. In this work, we propose a novel multiplex cellular community-based algorithm for tissue phenotyping integrating cell-level features within a graph-based hierarchical framework. We demonstrate that such integration offers better performance compared to prior deep learning and texture-based methods as well as to cellular community based methods using uniplex networks. To this end, we construct celllevel graphs using texture, alpha diversity and multi-resolution deep features. Using these graphs, we compute cellular connectivity features which are then employed for the construction of a patch-level multiplex network. Over this network, we compute multiplex cellular communities using a novel objective function. The proposed objective function computes a low-dimensional subspace from each cellular network and subsequently seeks a common low-dimensional subspace using the Grassmann manifold. We evaluate our proposed algorithm on three publicly available datasets for tissue phenotyping, demonstrating a significant improvement over existing state-of-the-art methods.

Automatic detection of squamous cell carcinoma metastasis in esophageal lymph nodes using semantic segmentation

Esophageal squamous cell carcinoma (ESCC) is more prevalent than esophageal adenocarcinoma in Asia, especially in China, where more than half of ESCC cases occur worldwide. Many studies have reported that the automatic detection of lymph node metastasis using semantic segmentation shows good performance in breast cancer and other adenocarcinomas. However, the detection of squamous cell carcinoma metastasis in hematoxylin-eosin (H&E)-stained slides has never been reported. We collected a training set of 110 esophageal lymph node slides with metastasis and 132 lymph node slides without metastasis. An iPad-based annotation system was used to draw the contours of the cancer metastasis region. A DeepLab v3 model was trained to achieve the best fit with the training data. The learned model could estimate the probability of metastasis. To evaluate the effectiveness of the detection model of learned metastasis, we used another large cohort of clinical H&E-stained esophageal lymph node slides containing 795 esophageal lymph nodes from 154 esophageal cancer patients. The basic authenticity label for each slide was confirmed by experienced pathologists. After filtering isolated noise in the prediction, we obtained an accuracy of 94%. Furthermore, we applied the learned model to throat and lung lymph node squamous cell carcinoma metastases and achieved the following promising results: an accuracy of 96.7% in throat cancer and an accuracy of 90% in lung cancer. In this work, we organized an annotated dataset of H&E-stained esophageal lymph node and trained a deep neural network to detect lymph node metastasis in H&E-stained slides of squamous cell carcinoma automatically. Moreover, it is possible to use this model to detect lymph nodes metastasis in squamous cell carcinoma from other organs. This study directly demonstrates the potential for determining the localization of squamous cell carcinoma metastases in lymph node and assisting in pathological diagnosis.

Deep transfer learning artificial intelligence accurately stages COVID-19 lung disease severity on portable chest radiographs

This study employed deep-learning convolutional neural networks to stage lung disease severity of Coronavirus Disease 2019 (COVID-19) infection on portable chest x-ray (CXR) with radiologist score of disease severity as ground truth. This study consisted of 131 portable CXR from 84 COVID-19 patients (51M 55.1±14.9yo; 29F 60.1±14.3yo; 4 missing information). Three expert chest radiologists scored the left and right lung separately based on the degree of opacity (0-3) and geographic extent (0-4). Deep-learning convolutional neural network (CNN) was used to predict lung disease severity scores. Data were split into 80% training and 20% testing datasets. Correlation analysis between AI-predicted versus radiologist scores were analyzed. Comparison was made with traditional and transfer learning. The average opacity score was 2.52 (range: 0-6) with a standard deviation of 0.25 (9.9%) across three readers. The average geographic extent score was 3.42 (range: 0-8) with a standard deviation of 0.57 (16.7%) across three readers. The inter-rater agreement yielded a Fleiss' Kappa of 0.45 for opacity score and 0.71 for extent score. AI-predicted scores strongly correlated with radiologist scores, with the top model yielding a correlation coefficient (R2) of 0.90 (range: 0.73-0.90 for traditional learning and 0.83-0.90 for transfer learning) and a mean absolute error of 8.5% (ranges: 17.2-21.0% and 8.5%-15.5, respectively). Transfer learning generally performed better. In conclusion, deep-learning CNN accurately stages disease severity on portable chest x-ray of COVID-19 lung infection. This approach may prove useful to stage lung disease severity, prognosticate, and predict treatment response and survival, thereby informing risk management and resource allocation.

Explainable AI (xAI) for Anatomic Pathology

Pathologists are adopting whole slide images (WSIs) for diagnosis, thanks to recent FDA approval of WSI systems as class II medical devices. In response to new market forces and recent technology advances outside of pathology, a new field of computational pathology has emerged that applies artificial intelligence (AI) and machine learning algorithms to WSIs. Computational pathology has great potential for augmenting pathologists' accuracy and efficiency, but there are important concerns regarding trust of AI due to the opaque, black-box nature of most AI algorithms. In addition, there is a lack of consensus on how pathologists should incorporate computational pathology systems into their workflow. To address these concerns, building computational pathology systems with explainable AI (xAI) mechanisms is a powerful and transparent alternative to black-box AI models. xAI can reveal underlying causes for its decisions; this is intended to promote safety and reliability of AI for critical tasks such as pathology diagnosis. This article outlines xAI enabled applications in anatomic pathology workflow that improves efficiency and accuracy of the practice. In addition, we describe HistoMapr-Breast, an initial xAI enabled software application for breast core biopsies. HistoMapr-Breast automatically previews breast core WSIs and recognizes the regions of interest to rapidly present the key diagnostic areas in an interactive and explainable manner. We anticipate xAI will ultimately serve pathologists as an interactive computational guide for computer-assisted primary diagnosis.

Artificial intelligence as the next step towards precision pathology

Pathology is the cornerstone of cancer care. The need for accuracy in histopathologic diagnosis of cancer is increasing as personalized cancer therapy requires accurate biomarker assessment. The appearance of digital image analysis holds promise to improve both the volume and precision of histomorphological evaluation. Recently, machine learning, and particularly deep learning, has enabled rapid advances in computational pathology. The integration of machine learning into routine care will be a milestone for the healthcare sector in the next decade, and histopathology is right at the centre of this revolution. Examples of potential high-value machine learning applications include both model-based assessment of routine diagnostic features in pathology, and the ability to extract and identify novel features that provide insights into a disease. Recent groundbreaking results have demonstrated that applications of machine learning methods in pathology significantly improves metastases detection in lymph nodes, Ki67 scoring in breast cancer, Gleason grading in prostate cancer and tumour-infiltrating lymphocyte (TIL) scoring in melanoma. Furthermore, deep learning models have also been demonstrated to be able to predict status of some molecular markers in lung, prostate, gastric and colorectal cancer based on standard HE slides. Moreover, prognostic (survival outcomes) deep neural network models based on digitized HE slides have been demonstrated in several diseases, including lung cancer, melanoma and glioma. In this review, we aim to present and summarize the latest developments in digital image analysis and in the application of artificial intelligence in diagnostic pathology.

The use of artificial intelligence, machine learning and deep learning in oncologic histopathology

BACKGROUND

Recently, there has been a momentous drive to apply advanced artificial intelligence (AI) technologies to diagnostic medicine. The introduction of AI has provided vast new opportunities to improve health care and has introduced a new wave of heightened precision in oncologic pathology. The impact of AI on oncologic pathology has now become apparent, and its use with respect to oral oncology is still in the nascent stage.

DISCUSSION

A foundational overview of AI classification systems used in medicine and a review of common terminology used in machine learning and computational pathology will be presented. This paper provides a focused review on the recent advances in AI and deep learning in oncologic histopathology and oral oncology. In addition, specific emphasis on recent studies that have applied these technologies to oral cancer prognostication will also be discussed.

CONCLUSION

Machine and deep learning methods designed to enhance prognostication of oral cancer have been proposed with much of the work focused on prediction models on patient survival and locoregional recurrences in patients with oral squamous cell carcinomas (OSCC). Few studies have explored machine learning methods on OSCC digital histopathologic images. It is evident that further research at the whole slide image level is needed and future collaborations with computer scientists may progress the field of oral oncology.

Classification and mutation prediction based on histopathology H&E images in liver cancer using deep learning

Hepatocellular carcinoma (HCC) is the most common subtype of liver cancer, and assessing its histopathological grade requires visual inspection by an experienced pathologist. In this study, the histopathological H&E images from the Genomic Data Commons Databases were used to train a neural network (inception V3) for automatic classification. According to the evaluation of our model by the Matthews correlation coefficient, the performance level was close to the ability of a 5-year experience pathologist, with 96.0% accuracy for benign and malignant classification, and 89.6% accuracy for well, moderate, and poor tumor differentiation. Furthermore, the model was trained to predict the ten most common and prognostic mutated genes in HCC. We found that four of them, including CTNNB1, FMN2, TP53, and ZFX4, could be predicted from histopathology images, with external AUCs from 0.71 to 0.89. The findings demonstrated that convolutional neural networks could be used to assist pathologists in the classification and detection of gene mutation in liver cancer.

Development and applications of computer image analysis algorithms for scoring of PD-L1 immunohistochemistry

Immune checkpoint inhibitors targeting programmed cell death 1 (PD-1) and programmed cell death ligand 1 (PD-L1) have rapidly become integral to standard-of-care therapy for non-small cell lung cancer and other cancers. Immunohistochemical (IHC) staining of PD-L1 is currently the accepted and approved diagnostic assay for selecting patients for PD-L1/PD-1 axis therapies in certain indications. However, the inherent biological complexity of PD-L1 and the availability of several PD-L1 assays – each with different detection systems, platforms, scoring algorithms and cut-offs – have created challenges to ensure reliable and reproducible results based on subjective visual assessment by pathologists. The increasing adoption of computer technologies into the daily workflow of pathology provides an opportunity to leverage these tools towards improving the clinical value of PD-L1 IHC assays. This review describes several image analysis software programs of computer-aided PD-L1 scoring in the hope of driving further discussion and technological advancement in digital pathology and artificial intelligence approaches, particularly as precision medicine evolves to encompass accurate simultaneous assessment of multiple features of cancer cells and their interactions with the tumor microenvironment.

Impact of image analysis and artificial intelligence in thyroid pathology, with particular reference to cytological aspects

OBJECTIVE

Thyroid pathology has great potential for automated/artificial intelligence algorithm application as the incidence of thyroid nodules is increasing and the indeterminate interpretation rate of fine-needle aspiration remains relatively high. The aim of the study is to review the published literature on automated image analysis and artificial intelligence applications to thyroid pathology with whole-slide imaging.

METHODS

Systematic search was carried out in electronic databases. Studies dealing with thyroid pathology and use of automated algorithms applied to whole-slide imaging were included. Quality of studies was assessed with a modified QUADAS-2 tool.

RESULTS

Of 919 retrieved articles, 19 were included. The main themes addressed were the comparison of automated assessment of immunohistochemical staining with manual pathologist's assessment, quantification of differences in cellular and nuclear parameters among tumour entities, and discrimination between benign and malignant nodules. Correlation coefficients with manual assessment were higher than 0.76 and diagnostic performance of automated models was comparable with an expert pathologist diagnosis. Computational difficulties were related to the large size of whole-slide images.

CONCLUSIONS

Overall, the results are promising and it is likely that, with the resolution of technical issues, the application of automated algorithms in thyroid pathology will increase and be adopted following suitable validation studies.

Review of the current state of digital image analysis in breast pathology

Advances in digital image analysis have the potential to transform the practice of breast pathology. In the near future, a move to a digital workflow offers improvements in efficiency. Coupled with artificial intelligence (AI), digital pathology can assist pathologist interpretation, automate time-consuming tasks, and discover novel morphologic patterns. Opportunities for digital enhancements abound in breast pathology, from increasing reproducibility in grading and biomarker interpretation, to discovering features that correlate with patient outcome and treatment. Our objective is to review the most recent developments in digital pathology with clear impact to breast pathology practice. Although breast pathologists currently undertake limited adoption of digital methods, the field is rapidly evolving. Care is needed to validate emerging technologies for effective patient care.

Deep learning assessment of breast terminal duct lobular unit involution: Towards automated prediction of breast cancer risk

Terminal duct lobular unit (TDLU) involution is the regression of milk-producing structures in the breast. Women with less TDLU involution are more likely to develop breast cancer. A major bottleneck in studying TDLU involution in large cohort studies is the need for labor-intensive manual assessment of TDLUs. We developed a computational pathology solution to automatically capture TDLU involution measures. Whole slide images (WSIs) of benign breast biopsies were obtained from the Nurses' Health Study. A set of 92 WSIs was annotated for acini, TDLUs and adipose tissue to train deep convolutional neural network (CNN) models for detection of acini, and segmentation of TDLUs and adipose tissue. These networks were integrated into a single computational method to capture TDLU involution measures including number of TDLUs per tissue area, median TDLU span and median number of acini per TDLU. We validated our method on 40 additional WSIs by comparing with manually acquired measures. Our CNN models detected acini with an F1 score of 0.73±0.07, and segmented TDLUs and adipose tissue with Dice scores of 0.84±0.13 and 0.87±0.04, respectively. The inter-observer ICC scores for manual assessments on 40 WSIs of number of TDLUs per tissue area, median TDLU span, and median acini count per TDLU were 0.71, 0.81 and 0.73, respectively. Intra-observer reliability was evaluated on 10/40 WSIs with ICC scores of >0.8. Inter-observer ICC scores between automated results and the mean of the two observers were: 0.80 for number of TDLUs per tissue area, 0.57 for median TDLU span, and 0.80 for median acini count per TDLU. TDLU involution measures evaluated by manual and automated assessment were inversely associated with age and menopausal status. We developed a computational pathology method to measure TDLU involution. This technology eliminates the labor-intensiveness and subjectivity of manual TDLU assessment, and can be applied to future breast cancer risk studies.

Emerging role of deep learning-based artificial intelligence in tumor pathology

The development of digital pathology and progression of state-of-the-art algorithms for computer vision have led to increasing interest in the use of artificial intelligence (AI), especially deep learning (DL)-based AI, in tumor pathology. The DL-based algorithms have been developed to conduct all kinds of work involved in tumor pathology, including tumor diagnosis, subtyping, grading, staging, and prognostic prediction, as well as the identification of pathological features, biomarkers and genetic changes. The applications of AI in pathology not only contribute to improve diagnostic accuracy and objectivity but also reduce the workload of pathologists and subsequently enable them to spend additional time on high-level decision-making tasks. In addition, AI is useful for pathologists to meet the requirements of precision oncology. However, there are still some challenges relating to the implementation of AI, including the issues of algorithm validation and interpretability, computing systems, the unbelieving attitude of pathologists, clinicians and patients, as well as regulators and reimbursements. Herein, we present an overview on how AI-based approaches could be integrated into the workflow of pathologists and discuss the challenges and perspectives of the implementation of AI in tumor pathology.

Image analysis and artificial intelligence in infectious disease diagnostics

BACKGROUND

Microbiologists are valued for their time-honed skills in image analysis, including identification of pathogens and inflammatory context in Gram stains, ova and parasite preparations, blood smears and histopathologic slides. They also must classify colony growth on a variety of agar plates for triage and assessment. Recent advances in image analysis, in particular application of artificial intelligence (AI), have the potential to automate these processes and support more timely and accurate diagnoses.

OBJECTIVES

To review current AI-based image analysis as applied to clinical microbiology; and to discuss future trends in the field.

SOURCES

Material sourced for this review included peer-reviewed literature annotated in the PubMed or Google Scholar databases and preprint articles from bioRxiv. Articles describing use of AI for analysis of images used in infectious disease diagnostics were reviewed.

CONTENT

We describe application of machine learning towards analysis of different types of microbiologic image data. Specifically, we outline progress in smear and plate interpretation as well as the potential for AI diagnostic applications in the clinical microbiology laboratory.

IMPLICATIONS

Combined with automation, we predict that AI algorithms will be used in the future to prescreen and preclassify image data, thereby increasing productivity and enabling more accurate diagnoses through collaboration between the AI and the microbiologist. Once developed, image-based AI analysis is inexpensive and amenable to local and remote diagnostic use.

Successful Identification of Nasopharyngeal Carcinoma in Nasopharyngeal Biopsies Using Deep Learning

Pathologic diagnosis of nasopharyngeal carcinoma (NPC) can be challenging since most cases are nonkeratinizing carcinoma with little differentiation and many admixed lymphocytes. Our aim was to evaluate the possibility to identify NPC in nasopharyngeal biopsies using deep learning. A total of 726 nasopharyngeal biopsies were included. Among them, 100 cases were randomly selected as the testing set, 20 cases as the validation set, and all other 606 cases as the training set. All three datasets had equal numbers of NPC cases and benign cases. Manual annotation was performed. Cropped square image patches of 256 × 256 pixels were used for patch-level training, validation, and testing. The final patch-level algorithm effectively identified NPC patches, with an area under the receiver operator characteristic curve (AUC) of 0.9900. Using gradient-weighted class activation mapping, we demonstrated that the identification of NPC patches was based on morphologic features of tumor cells. At the second stage, whole-slide images were sequentially cropped into patches, inferred with the patch-level algorithm, and reconstructed into images with a smaller size for training, validation, and testing. Finally, the AUC was 0.9848 for slide-level identification of NPC. Our result shows for the first time that deep learning algorithms can identify NPC.

Tailored for Real-World: A Whole Slide Image Classification System Validated on Uncurated Multi-Site Data Emulating the Prospective Pathology Workload

Standard of care diagnostic procedure for suspected skin cancer is microscopic examination of hematoxylin & eosin stained tissue by a pathologist. Areas of high inter-pathologist discordance and rising biopsy rates necessitate higher efficiency and diagnostic reproducibility. We present and validate a deep learning system which classifies digitized dermatopathology slides into 4 categories. The system is developed using 5,070 images from a single lab, and tested on an uncurated set of 13,537 images from 3 test labs, using whole slide scanners manufactured by 3 different vendors. The system's use of deep-learning-based confidence scoring as a criterion to consider the result as accurate yields an accuracy of up to 98%, and makes it adoptable in a real-world setting. Without confidence scoring, the system achieved an accuracy of 78%. We anticipate that our deep learning system will serve as a foundation enabling faster diagnosis of skin cancer, identification of cases for specialist review, and targeted diagnostic classifications.

Resolution-agnostic tissue segmentation in whole-slide histopathology images with convolutional neural networks

Modern pathology diagnostics is being driven toward large scale digitization of microscopic tissue sections. A prerequisite for its safe implementation is the guarantee that all tissue present on a glass slide can also be found back in the digital image. Whole-slide scanners perform a tissue segmentation in a low resolution overview image to prevent inefficient high-resolution scanning of empty background areas. However, currently applied algorithms can fail in detecting all tissue regions.

In this study, we developed convolutional neural networks to distinguish tissue from background. We collected 100 whole-slide images of 10 tissue samples—staining categories from five medical centers for development and testing. Additionally, eight more images of eight unfamiliar categories were collected for testing only. We compared our fully-convolutional neural networks to three traditional methods on a range of resolution levels using Dice score and sensitivity.

We also tested whether a single neural network can perform equivalently to multiple networks, each specialized in a single resolution. Overall, our solutions outperformed the traditional methods on all the tested resolutions. The resolution-agnostic network achieved average Dice scores between 0.97 and 0.98 across the tested resolution levels, only 0.0069 less than the resolution-specific networks. Finally, its excellent generalization performance was demonstrated by achieving averages of 0.98 Dice score and 0.97 sensitivity on the eight unfamiliar images. A future study should test this network prospectively.