A machine learning algorithm for simulating immunohistochemistry: development of SOX10 virtual IHC and evaluation on primarily melanocytic neoplasms

An initial game-theoretic assessment of enhanced tissue preparation and imaging protocols for improved deep learning inference of spatial transcriptomics from tissue morphology

The application of deep learning to spatial transcriptomics (ST) can reveal relationships between gene expression and tissue architecture. Prior work has demonstrated that inferring gene expression from tissue histomorphology can discern these spatial molecular markers to enable population scale studies, reducing the fiscal barriers associated with large-scale spatial profiling. However, while most improvements in algorithmic performance have focused on improving model architectures, little is known about how the quality of tissue preparation and imaging can affect deep learning model training for spatial inference from morphology and its potential for widespread clinical adoption. Prior studies for ST inference from histology typically utilize manually stained frozen sections with imaging on non-clinical grade scanners. Training such models on ST cohorts is also costly. We hypothesize that adopting tissue processing and imaging practices that mirror standards for clinical implementation (permanent sections, automated tissue staining, and clinical grade scanning) can significantly improve model performance. An enhanced specimen processing and imaging protocol was developed for deep learning-based ST inference from morphology. This protocol featured the Visium CytAssist assay to permit automated hematoxylin and eosin staining (e.g. Leica Bond), 40×-resolution imaging, and joining of multiple patients' tissue sections per capture area prior to ST profiling. Using a cohort of 13 pathologic T Stage-III stage colorectal cancer patients, we compared the performance of models trained on slide prepared using enhanced versus traditional (i.e. manual staining and low-resolution imaging) protocols. Leveraging Inceptionv3 neural networks, we predicted gene expression across serial, histologically-matched tissue sections using whole slide images (WSI) from both protocols. The data Shapley was used to quantify and compare marginal performance gains on a patient-by-patient basis attributed to using the enhanced protocol versus the actual costs of spatial profiling. Findings indicate that training and validating on WSI acquired through the enhanced protocol as opposed to the traditional method resulted in improved performance at lower fiscal cost. In the realm of ST, the enhancement of deep learning architectures frequently captures the spotlight; however, the significance of specimen processing and imaging is often understated. This research, informed through a game-theoretic lens, underscores the substantial impact that specimen preparation/imaging can have on spatial transcriptomic inference from morphology. It is essential to integrate such optimized processing protocols to facilitate the identification of prognostic markers at a larger scale.

Deep learning model shows pathologist-level detection of sentinel node metastasis of melanoma and intra-nodal nevi on whole slide images

Introduction

Nodal metastasis (NM) in sentinel node biopsies (SNB) is crucial for melanoma staging. However, an intra-nodal nevus (INN) may often be misclassified as NM, leading to potential misdiagnosis and incorrect staging. There is high discordance among pathologists in assessing SNB positivity, which may lead to false staging. Digital whole slide imaging offers the potential for implementing artificial intelligence (AI) in digital pathology. In this study, we assessed the capability of AI to detect NM and INN in SNBs.

Methods

A total of 485 hematoxylin and eosin whole slide images (WSIs), including NM and INN from 196 SNBs, were collected and divided into training (279 WSIs), validation (89 WSIs), and test sets (117 WSIs). A deep learning model was trained with 5,956 manual pixel-wise annotations. The AI and three blinded dermatopathologists assessed the test set, with immunohistochemistry serving as the reference standard.

Results

The AI model showed excellent performance with an area under the curve receiver operating characteristic (AUC) of 0.965 for detecting NM. In comparison, the AUC for NM detection among dermatopathologists ranged between 0.94 and 0.98. For the detection of INN, the AUC was lower for both AI (0.781) and dermatopathologists (range of 0.63-0.79).

Discussion

In conclusion, the deep learning AI model showed excellent accuracy in detecting NM, achieving dermatopathologist-level performance in detecting both NM and INN. Importantly, the AI model showed the potential to differentiate between these two entities. However, further validation is warranted.

Utilizing deep learning model for assessing melanocytic density in resection margins of lentigo maligna

BACKGROUND

Surgical excision with clear histopathological margins is the preferred treatment to prevent progression of lentigo maligna (LM) to invasive melanoma. However, the assessment of resection margins on sun-damaged skin is challenging. We developed a deep learning model for detection of melanocytes in resection margins of LM.

METHODS

In total, 353 whole slide images (WSIs) were included. 295 WSIs were used for training and 58 for validation and testing. The algorithm was trained with 3,973 manual pixel-wise annotations. The AI analyses were compared to those of three blinded dermatopathologists and two pathology residents, who performed their evaluations without AI and AI-assisted. Immunohistochemistry (SOX10) served as the reference standard. We used a dichotomized cutoff for low and high risk of recurrence (≤ 25 melanocytes in an area of 0.5 mm for low risk and > 25 for high risk).

RESULTS

The AI model achieved an area under the receiver operating characteristic curve (AUC) of 0.84 in discriminating margins with low and high recurrence risk. In comparison, the AUC for dermatopathologists ranged from 0.72 to 0.90 and for the residents in pathology, 0.68 to 0.80. Additionally, with aid of the AI model the performance of two pathologists significantly improved.

CONCLUSIONS

The deep learning showed notable accuracy in detecting resection margins of LM with a high versus low risk of recurrence. Furthermore, the use of AI improved the performance of 2/5 pathologists. This automated tool could aid pathologists in the assessment or pre-screening of LM margins.

Training immunophenotyping deep learning models with the same-section ground truth cell label derivation method improves virtual staining accuracy

Introduction

Deep learning (DL) models predicting biomarker expression in images of hematoxylin and eosin (H&E)-stained tissues can improve access to multi-marker immunophenotyping, crucial for therapeutic monitoring, biomarker discovery, and personalized treatment development. Conventionally, these models are trained on ground truth cell labels derived from IHC-stained tissue sections adjacent to H&E-stained ones, which might be less accurate than labels from the same section. Although many such DL models have been developed, the impact of ground truth cell label derivation methods on their performance has not been studied.

Methodology

In this study, we assess the impact of cell label derivation on H&E model performance, with CD3+ T-cells in lung cancer tissues as a proof-of-concept. We compare two Pix2Pix generative adversarial network (P2P-GAN)-based virtual staining models: one trained with cell labels obtained from the same tissue section as the H&E-stained section (the 'same-section' model) and one trained on cell labels from an adjacent tissue section (the 'serial-section' model).

Results

We show that the same-section model exhibited significantly improved prediction performance compared to the 'serial-section' model. Furthermore, the same-section model outperformed the serial-section model in stratifying lung cancer patients within a public lung cancer cohort based on survival outcomes, demonstrating its potential clinical utility.

Discussion

Collectively, our findings suggest that employing ground truth cell labels obtained through the same-section approach boosts immunophenotyping DL solutions.

Immunocytochemical markers, molecular testing and digital cytopathology for aspiration cytology of metastatic breast carcinoma

Fine-needle aspiration cytology (FNAC) is a versatile diagnostic procedure uniquely suited for tissue biopsy of breast carcinomas and axillary metastases and/or recurrences. With the expanding treatment options and accompanying theragnostic tests, it is crucial to recognize the developments on ancillary testing and digital cytopathology techniques related to aspiration cytology of metastatic breast carcinoma. In this review, we aim to summarize and update the evidence of immunocytochemistry, for the detection of carcinoma cells (epithelial markers), confirmation of breast primary (breast-specific markers), assessment of surrogate immunostains (hormone receptors, ki-67 proliferative index and HER2) and theragnostic biomarkers, with discussion on potential diagnostic pitfalls, followed by the application of molecular tests, and digital cytopathologic techniques for assessing metastatic breast carcinoma in cytology.

Artificial Intelligence in Melanoma Dermatopathology: A Review of Literature

ABSTRACT

Pathology serves as a promising field to integrate artificial intelligence into clinical practice as a powerful screening tool. Melanoma is a common skin cancer with high mortality and morbidity, requiring timely and accurate histopathologic diagnosis. This review explores applications of artificial intelligence in melanoma dermatopathology, including differential diagnostics, prognosis prediction, and personalized medicine decision-making.

Intraoperative margin assessment for basal cell carcinoma with deep learning and histologic tumor mapping to surgical site

Successful treatment of solid cancers relies on complete surgical excision of the tumor either for definitive treatment or before adjuvant therapy. Intraoperative and postoperative radial sectioning, the most common form of margin assessment, can lead to incomplete excision and increase the risk of recurrence and repeat procedures. Mohs Micrographic Surgery is associated with complete removal of basal cell and squamous cell carcinoma through real-time margin assessment of 100% of the peripheral and deep margins. Real-time assessment in many tumor types is constrained by tissue size, complexity, and specimen processing / assessment time during general anesthesia. We developed an artificial intelligence platform to reduce the tissue preprocessing and histological assessment time through automated grossing recommendations, mapping and orientation of tumor to the surgical specimen. Using basal cell carcinoma as a model system, results demonstrate that this approach can address surgical laboratory efficiency bottlenecks for rapid and complete intraoperative margin assessment.

The Overlooked Role of Specimen Preparation in Bolstering Deep Learning-Enhanced Spatial Transcriptomics Workflows

The application of deep learning methods to spatial transcriptomics has shown promise in unraveling the complex relationships between gene expression patterns and tissue architecture as they pertain to various pathological conditions. Deep learning methods that can infer gene expression patterns directly from tissue histomorphology can expand the capability to discern spatial molecular markers within tissue slides. However, current methods utilizing these techniques are plagued by substantial variability in tissue preparation and characteristics, which can hinder the broader adoption of these tools. Furthermore, training deep learning models using spatial transcriptomics on small study cohorts remains a costly endeavor. Necessitating novel tissue preparation processes enhance assay reliability, resolution, and scalability. This study investigated the impact of an enhanced specimen processing workflow for facilitating a deep learning-based spatial transcriptomics assessment. The enhanced workflow leveraged the flexibility of the Visium CytAssist assay to permit automated H&E staining (e.g., Leica Bond) of tissue slides, whole-slide imaging at 40x-resolution, and multiplexing of tissue sections from multiple patients within individual capture areas for spatial transcriptomics profiling. Using a cohort of thirteen pT3 stage colorectal cancer (CRC) patients, we compared the efficacy of deep learning models trained on slide prepared using an enhanced workflow as compared to the traditional workflow which leverages manual tissue staining and standard imaging of tissue slides. Leveraging Inceptionv3 neural networks, we aimed to predict gene expression patterns across matched serial tissue sections, each stemming from a distinct workflow but aligned based on persistent histological structures. Findings indicate that the enhanced workflow considerably outperformed the traditional spatial transcriptomics workflow. Gene expression profiles predicted from enhanced tissue slides also yielded expression patterns more topologically consistent with the ground truth. This led to enhanced statistical precision in pinpointing biomarkers associated with distinct spatial structures. These insights can potentially elevate diagnostic and prognostic biomarker detection by broadening the range of spatial molecular markers linked to metastasis and recurrence. Future endeavors will further explore these findings to enrich our comprehension of various diseases and uncover molecular pathways with greater nuance. Combining deep learning with spatial transcriptomics provides a compelling avenue to enrich our understanding of tumor biology and improve clinical outcomes. For results of the highest fidelity, however, effective specimen processing is crucial, and fostering collaboration between histotechnicians, pathologists, and genomics specialists is essential to herald this new era in spatial transcriptomics-driven cancer research.

Deep learning in computational dermatopathology of melanoma: A technical systematic literature review

Deep learning (DL) has become one of the major approaches in computational dermatopathology, evidenced by a significant increase in this topic in the current literature. We aim to provide a structured and comprehensive overview of peer-reviewed publications on DL applied to dermatopathology focused on melanoma. In comparison to well-published DL methods on non-medical images (e.g., classification on ImageNet), this field of application comprises a specific set of challenges, such as staining artifacts, large gigapixel images, and various magnification levels. Thus, we are particularly interested in the pathology-specific technical state-of-the-art. We also aim to summarize the best performances achieved thus far with respect to accuracy, along with an overview of self-reported limitations. Accordingly, we conducted a systematic literature review of peer-reviewed journal and conference articles published between 2012 and 2022 in the databases ACM Digital Library, Embase, IEEE Xplore, PubMed, and Scopus, expanded by forward and backward searches to identify 495 potentially eligible studies. After screening for relevance and quality, a total of 54 studies were included. We qualitatively summarized and analyzed these studies from technical, problem-oriented, and task-oriented perspectives. Our findings suggest that the technical aspects of DL for histopathology in melanoma can be further improved. The DL methodology was adopted later in this field, and still lacks the wider adoption of DL methods already shown to be effective for other applications. We also discuss upcoming trends toward ImageNet-based feature extraction and larger models. While DL has achieved human-competitive accuracy in routine pathological tasks, its performance on advanced tasks is still inferior to wet-lab testing (for example). Finally, we discuss the challenges impeding the translation of DL methods to clinical practice and provide insight into future research directions.

A snapshot of artificial intelligence research 2019-2021: is it replacing or assisting physicians?

Artificial intelligence (AI) has the potential to be a disruptive technology in healthcare. Recently, there is increased speculation that AI may be used to replace healthcare providers in the future. To answer this question, we reviewed over 21 000 articles published in medical specialty journals between 2019 and 2021 to evaluate whether these AI models were intended to assist or replace healthcare providers. We also evaluated whether all Food and Drug Administration (FDA)-approved AI models were used to assist or replace healthcare providers. We find that most AI models published in this time period were intended to assist rather than replace healthcare providers, and that most of the published AI models performed tasks that could not be done by healthcare providers.

Recent Advances in Melanoma Diagnosis and Prognosis Using Machine Learning Methods

PURPOSE OF REVIEW

The purpose was to summarize the current role and state of artificial intelligence and machine learning in the diagnosis and management of melanoma.

RECENT FINDINGS

Deep learning algorithms can identify melanoma from clinical, dermoscopic, and whole slide pathology images with increasing accuracy. Efforts to provide more granular annotation to datasets and to identify new predictors are ongoing. There have been many incremental advances in both melanoma diagnostics and prognostic tools using artificial intelligence and machine learning. Higher quality input data will further improve these models' capabilities.

Restaining-based annotation for cancer histology segmentation to overcome annotation-related limitations among pathologists

Numerous cancer histopathology specimens have been collected and digitized over the past few decades. A comprehensive evaluation of the distribution of various cells in tumor tissue sections can provide valuable information for understanding cancer. Deep learning is suitable for achieving these goals; however, the collection of extensive, unbiased training data is hindered, thus limiting the production of accurate segmentation models. This study presents SegPath-the largest annotation dataset (>10 times larger than publicly available annotations)-for the segmentation of hematoxylin and eosin (H&E)-stained sections for eight major cell types in cancer tissue. The SegPath generating pipeline used H&E-stained sections that were destained and subsequently immunofluorescence-stained with carefully selected antibodies. We found that SegPath is comparable with, or outperforms, pathologist annotations. Moreover, annotations by pathologists are biased toward typical morphologies. However, the model trained on SegPath can overcome this limitation. Our results provide foundational datasets for machine-learning research in histopathology.

Computer-Assisted Annotation of Digital H&E/SOX10 Dual Stains Generates High-Performing Convolutional Neural Network for Calculating Tumor Burden in H&E-Stained Cutaneous Melanoma

Deep learning for the analysis of H&E stains requires a large annotated training set. This may form a labor-intensive task involving highly skilled pathologists. We aimed to optimize and evaluate computer-assisted annotation based on digital dual stains of the same tissue section. H&E stains of primary and metastatic melanoma (N = 77) were digitized, re-stained with SOX10, and re-scanned. Because images were aligned, annotations of SOX10 image analysis were directly transferred to H&E stains of the training set. Based on 1,221,367 annotated nuclei, a convolutional neural network for calculating tumor burden (CNN_TB) was developed. For primary melanomas, precision of annotation was 100% (95%CI, 99% to 100%) for tumor cells and 99% (95%CI, 98% to 100%) for normal cells. Due to low or missing tumor-cell SOX10 positivity, precision for normal cells was markedly reduced in lymph-node and organ metastases compared with primary melanomas (p < 0.001). Compared with stereological counts within skin lesions, mean difference in tumor burden was 6% (95%CI, -1% to 13%, p = 0.10) for CNN_TB and 16% (95%CI, 4% to 28%, p = 0.02) for pathologists. Conclusively, the technique produced a large annotated H&E training set with high quality within a reasonable timeframe for primary melanomas and subcutaneous metastases. For these lesion types, the training set generated a high-performing CNN_TB, which was superior to the routine assessments of pathologists.

Global Research Trends of Artificial Intelligence on Histopathological Images: A 20-Year Bibliometric Analysis

Cancer has become a major threat to global health care. With the development of computer science, artificial intelligence (AI) has been widely applied in histopathological images (HI) analysis. This study analyzed the publications of AI in HI from 2001 to 2021 by bibliometrics, exploring the research status and the potential popular directions in the future. A total of 2844 publications from the Web of Science Core Collection were included in the bibliometric analysis. The country/region, institution, author, journal, keyword, and references were analyzed by using VOSviewer and CiteSpace. The results showed that the number of publications has grown rapidly in the last five years. The USA is the most productive and influential country with 937 publications and 23,010 citations, and most of the authors and institutions with higher numbers of publications and citations are from the USA. Keyword analysis showed that breast cancer, prostate cancer, colorectal cancer, and lung cancer are the tumor types of greatest concern. Co-citation analysis showed that classification and nucleus segmentation are the main research directions of AI-based HI studies. Transfer learning and self-supervised learning in HI is on the rise. This study performed the first bibliometric analysis of AI in HI from multiple indicators, providing insights for researchers to identify key cancer types and understand the research trends of AI application in HI.

HEROHE Challenge: Predicting HER2 Status in Breast Cancer from Hematoxylin–Eosin Whole-Slide Imaging

Breast cancer is the most common malignancy in women worldwide, and is responsible for more than half a million deaths each year. The appropriate therapy depends on the evaluation of the expression of various biomarkers, such as the human epidermal growth factor receptor 2 (HER2) transmembrane protein, through specialized techniques, such as immunohistochemistry or in situ hybridization. In this work, we present the HER2 on hematoxylin and eosin (HEROHE) challenge, a parallel event of the 16th European Congress on Digital Pathology, which aimed to predict the HER2 status in breast cancer based only on hematoxylin–eosin-stained tissue samples, thus avoiding specialized techniques. The challenge consisted of a large, annotated, whole-slide images dataset (509), specifically collected for the challenge. Models for predicting HER2 status were presented by 21 teams worldwide. The best-performing models are presented by detailing the network architectures and key parameters. Methods are compared and approaches, core methodologies, and software choices contrasted. Different evaluation metrics are discussed, as well as the performance of the presented models for each of these metrics. Potential differences in ranking that would result from different choices of evaluation metrics highlight the need for careful consideration at the time of their selection, as the results show that some metrics may misrepresent the true potential of a model to solve the problem for which it was developed. The HEROHE dataset remains publicly available to promote advances in the field of computational pathology.

From Immunohistochemistry to New Digital Ecosystems: A State-of-the-Art Biomarker Review for Precision Breast Cancer Medicine

Breast cancers represent complex ecosystem-like networks of malignant cells and their associated microenvironment. Estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2) are biomarkers ubiquitous to clinical practice in evaluating prognosis and predicting response to therapy. Recent feats in breast cancer have led to a new digital era, and advanced clinical trials have resulted in a growing number of personalized therapies with corresponding biomarkers. In this state-of-the-art review, we included the latest 10-year updated recommendations for ER, PR, and HER2, along with the most salient information on tumor-infiltrating lymphocytes (TILs), Ki-67, PD-L1, and several prognostic/predictive biomarkers at genomic, transcriptomic, and proteomic levels recently developed for selection and optimization of breast cancer treatment. Looking forward, the multi-omic landscape of the tumor ecosystem could be integrated with computational findings from whole slide images and radiomics in predictive machine learning (ML) models. These are new digital ecosystems on the road to precision breast cancer medicine.

Deep learning features encode interpretable morphologies within histological images

Convolutional neural networks (CNNs) are revolutionizing digital pathology by enabling machine learning-based classification of a variety of phenotypes from hematoxylin and eosin (H&E) whole slide images (WSIs), but the interpretation of CNNs remains difficult. Most studies have considered interpretability in a post hoc fashion, e.g. by presenting example regions with strongly predicted class labels. However, such an approach does not explain the biological features that contribute to correct predictions. To address this problem, here we investigate the interpretability of H&E-derived CNN features (the feature weights in the final layer of a transfer-learning-based architecture). While many studies have incorporated CNN features into predictive models, there has been little empirical study of their properties. We show such features can be construed as abstract morphological genes (“mones”) with strong independent associations to biological phenotypes. Many mones are specific to individual cancer types, while others are found in multiple cancers especially from related tissue types. We also observe that mone-mone correlations are strong and robustly preserved across related cancers. Importantly, linear mone-based classifiers can very accurately separate 38 distinct classes (19 tumor types and their adjacent normals, AUC = \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$97.1\% \pm 2.8\%$$\end{document}97.1%±2.8% for each class prediction), and linear classifiers are also highly effective for universal tumor detection (AUC = \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$99.2\% \pm 0.12\%$$\end{document}99.2%±0.12%). This linearity provides evidence that individual mones or correlated mone clusters may be associated with interpretable histopathological features or other patient characteristics. In particular, the statistical similarity of mones to gene expression values allows integrative mone analysis via expression-based bioinformatics approaches. We observe strong correlations between individual mones and individual gene expression values, notably mones associated with collagen gene expression in ovarian cancer. Mone-expression comparisons also indicate that immunoglobulin expression can be identified using mones in colon adenocarcinoma and that immune activity can be identified across multiple cancer types, and we verify these findings by expert histopathological review. Our work demonstrates that mones provide a morphological H&E decomposition that can be effectively associated with diverse phenotypes, analogous to the interpretability of transcription via gene expression values. Our work also demonstrates mones can be interpreted without using a classifier as a proxy.

AI-enabled in silico immunohistochemical characterization for Alzheimer's disease

We develop a deep learning approach, in silico immunohistochemistry (IHC), which takes routinely collected histochemical-stained samples as input and computationally generates virtual IHC slide images. We apply in silico IHC to Alzheimer's disease samples, where several hallmark changes are conventionally identified using IHC staining across many regions of the brain. In silico IHC computationally identifies neurofibrillary tangles, β-amyloid plaques, and neuritic plaques at a high spatial resolution directly from the histochemical images, with areas under the receiver operating characteristic curve of between 0.88 and 0.92. In silico IHC learns to identify subtle cellular morphologies associated with these lesions and can generate in silico IHC slides that capture key features of the actual IHC.

A deep learning model for molecular label transfer that enables cancer cell identification from histopathology images

Deep-learning classification systems have the potential to improve cancer diagnosis. However, development of these computational approaches so far depends on prior pathological annotations and large training datasets. The manual annotation is low-resolution, time-consuming, highly variable and subject to observer variance. To address this issue, we developed a method, H&E Molecular neural network (HEMnet). HEMnet utilizes immunohistochemistry as an initial molecular label for cancer cells on a H&E image and trains a cancer classifier on the overlapping clinical histopathological images. Using this molecular transfer method, HEMnet successfully generated and labeled 21,939 tumor and 8782 normal tiles from ten whole-slide images for model training. After building the model, HEMnet accurately identified colorectal cancer regions, which achieved 0.84 and 0.73 of ROC AUC values compared to p53 staining and pathological annotations, respectively. Our validation study using histopathology images from TCGA samples accurately estimated tumor purity, which showed a significant correlation (regression coefficient of 0.8) with the estimation based on genomic sequencing data. Thus, HEMnet contributes to addressing two main challenges in cancer deep-learning analysis, namely the need to have a large number of images for training and the dependence on manual labeling by a pathologist. HEMnet also predicts cancer cells at a much higher resolution compared to manual histopathologic evaluation. Overall, our method provides a path towards a fully automated delineation of any type of tumor so long as there is a cancer-oriented molecular stain available for subsequent learning. Software, tutorials and interactive tools are available at: https://github.com/BiomedicalMachineLearning/HEMnet.

DeepHistoClass: A Novel Strategy for Confident Classification of Immunohistochemistry Images Using Deep Learning

A multitude of efforts worldwide aim to create a single-cell reference map of the human body, for fundamental understanding of human health, molecular medicine, and targeted treatment. Antibody-based proteomics using immunohistochemistry (IHC) has proven to be an excellent technology for integration with large-scale single-cell transcriptomics datasets. The golden standard for evaluation of IHC staining patterns is manual annotation, which is expensive and may lead to subjective errors. Artificial intelligence holds much promise for efficient and accurate pattern recognition, but confidence in prediction needs to be addressed. Here, the aim was to present a reliable and comprehensive framework for automated annotation of IHC images. We developed a multilabel classification of 7848 complex IHC images of human testis corresponding to 2794 unique proteins, generated as part of the Human Protein Atlas (HPA) project. Manual annotation data for eight different cell types was generated as a basis for training and testing a proposed Hybrid Bayesian Neural Network. By combining the deep learning model with a novel uncertainty metric, DeepHistoClass (DHC) Confidence Score, the average diagnostic performance improved from 86.9% to 96.3%. This metric not only reveals which images are reliably classified by the model, but can also be utilized for identification of manual annotation errors. The proposed streamlined workflow can be developed further for other tissue types in health and disease and has important implications for digital pathology initiatives or large-scale protein mapping efforts such as the HPA project.

•

A novel method for automated annotation of immunohistochemistry images.

•

Introduction of an uncertainty metric, the DeepHistoClass (DHC) confidence score.

•

Increased accuracy of automated image predictions.

•

Identification of manual annotation errors.

How the variability between computer-assisted analysis procedures evaluating immune markers can influence patients' outcome prediction

Differences between computer-assisted image analysis (CAI) algorithms may cause discrepancies in the identification of immunohistochemically stained immune biomarkers in biopsies of breast cancer patients. These discrepancies have implications for their association with disease outcome. This study aims to compare three CAI procedures (A, B and C) to measure positive marker areas in post-neoadjuvant chemotherapy biopsies of patients with triple-negative breast cancer (TNBC) and to explore the differences in their performance in determining the potential association with relapse in these patients. A total of 3304 digital images of biopsy tissue obtained from 118 TNBC patients were stained for seven immune markers using immunohistochemistry (CD4, CD8, FOXP3, CD21, CD1a, CD83, HLA-DR) and were analyzed with procedures A, B and C. The three methods measure the positive pixel markers in the total tissue areas. The extent of agreement between paired CAI procedures, a principal component analysis (PCA) and Cox multivariate analysis was assessed. Comparisons of paired procedures showed close agreement for most of the immune markers at low concentration. The probability of differences between the paired procedures B/C and B/A was generally higher than those observed in C/A. The principal component analysis, largely based on data from CD8, CD1a and HLA-DR, identified two groups of patients with a significantly lower probability of relapse than the others. The multivariate regression models showed similarities in the factors associated with relapse for procedures A and C, as opposed to those obtained with procedure B. General agreement among the results of CAI procedures would not guarantee that the same predictive breast cancer markers were consistently identified. These results highlight the importance of developing additional strategies to improve the sensitivity of CAI procedures.

A large-scale internal validation study of unsupervised virtual trichrome staining technologies on nonalcoholic steatohepatitis liver biopsies

Non-alcoholic steatohepatitis (NASH) is a fatty liver disease characterized by accumulation of fat in hepatocytes with concurrent inflammation and is associated with morbidity, cirrhosis and liver failure. After extraction of a liver core biopsy, tissue sections are stained with hematoxylin and eosin (H&E) to grade NASH activity, and stained with trichrome to stage fibrosis. Methods to computationally transform one stain into another on digital whole slide images (WSI) can lessen the need for additional physical staining besides H&E, reducing personnel, equipment, and time costs. Generative adversarial networks (GAN) have shown promise for virtual staining of tissue. We conducted a large-scale validation study of the viability of GANs for H&E to trichrome conversion on WSI (n = 574). Pathologists were largely unable to distinguish real images from virtual/synthetic images given a set of twelve Turing Tests. We report high correlation between staging of real and virtual stains ([Formula: see text]; 95% CI: 0.84-0.88). Stages assigned to both virtual and real stains correlated similarly with a number of clinical biomarkers and progression to End Stage Liver Disease (Hazard Ratio HR = 2.06, 95% CI: 1.36-3.12, p < 0.001 for real stains; HR = 2.02, 95% CI: 1.40-2.92, p < 0.001 for virtual stains). Our results demonstrate that virtual trichrome technologies may offer a software solution that can be employed in the clinical setting as a diagnostic decision aid.

Harnessing non-destructive 3D pathology

High-throughput methods for slide-free three-dimensional (3D) pathological analyses of whole biopsies and surgical specimens offer the promise of modernizing traditional histology workflows and delivering improvements in diagnostic performance. Advanced optical methods now enable the interrogation of orders of magnitude more tissue than previously possible, where volumetric imaging allows for enhanced quantitative analyses of cell distributions and tissue structures that are prognostic and predictive. Non-destructive imaging processes can simplify laboratory workflows, potentially reducing costs, and can ensure that samples are available for subsequent molecular assays. However, the large size of the feature-rich datasets that they generate poses challenges for data management and computer-aided analysis. In this Perspective, we provide an overview of the imaging technologies that enable 3D pathology, and the computational tools-machine learning, in particular-for image processing and interpretation. We also discuss the integration of various other diagnostic modalities with 3D pathology, along with the challenges and opportunities for clinical adoption and regulatory approval.

SHIFT: speedy histological-to-immunofluorescent translation of a tumor signature enabled by deep learning

Spatially-resolved molecular profiling by immunostaining tissue sections is a key feature in cancer diagnosis, subtyping, and treatment, where it complements routine histopathological evaluation by clarifying tumor phenotypes. In this work, we present a deep learning-based method called speedy histological-to-immunofluorescent translation (SHIFT) which takes histologic images of hematoxylin and eosin (H&E)-stained tissue as input, then in near-real time returns inferred virtual immunofluorescence (IF) images that estimate the underlying distribution of the tumor cell marker pan-cytokeratin (panCK). To build a dataset suitable for learning this task, we developed a serial staining protocol which allows IF and H&E images from the same tissue to be spatially registered. We show that deep learning-extracted morphological feature representations of histological images can guide representative sample selection, which improved SHIFT generalizability in a small but heterogenous set of human pancreatic cancer samples. With validation in larger cohorts, SHIFT could serve as an efficient preliminary, auxiliary, or substitute for panCK IF by delivering virtual panCK IF images for a fraction of the cost and in a fraction of the time required by traditional IF.