Data integration from pathology slides for quantitative imaging of multiple cell types within the tumor immune cell infiltrate

A Pipeline for Evaluation of Machine Learning/Artificial Intelligence Models to Quantify Programmed Death Ligand 1 Immunohistochemistry

Immunohistochemistry (IHC) is used to guide treatment decisions in multiple cancer types. For treatment with checkpoint inhibitors, programmed death ligand 1 (PD-L1) IHC is used as a companion diagnostic. However, the scoring of PD-L1 is complicated by its expression in cancer and immune cells. Separation of cancer and noncancer regions is needed to calculate tumor proportion scores (TPS) of PD-L1, which is based on the percentage of PD-L1-positive cancer cells. Evaluation of PD-L1 expression requires highly experienced pathologists and is often challenging and time-consuming. Here, we used a multi-institutional cohort of 77 lung cancer cases stained centrally with the PD-L1 22C3 clone. We developed a 4-step pipeline for measuring TPS that includes the coregistration of hematoxylin and eosin, PD-L1, and negative control (NC) digital slides for exclusion of necrosis, segmentation of cancer regions, and quantification of PD-L1+ cells. As cancer segmentation is a challenging step for TPS generation, we trained DeepLab V3 in the Visiopharm software package to outline cancer regions in PD-L1 and NC images and evaluated the model performance by mean intersection over union (mIoU) against manual outlines. Only 14 cases were required to accomplish a mIoU of 0.82 for cancer segmentation in hematoxylin-stained NC cases. For PD-L1-stained slides, a model trained on PD-L1 tiles augmented by registered NC tiles achieved a mIoU of 0.79. In segmented cancer regions from whole slide images, the digital TPS achieved an accuracy of 75% against the manual TPS scores from the pathology report. Major reasons for algorithmic inaccuracies include the inclusion of immune cells in cancer outlines and poor nuclear segmentation of cancer cells. Our transparent and stepwise approach and performance metrics can be applied to any IHC assay to provide pathologists with important insights on when to apply and how to evaluate commercial automated IHC scoring systems.

Towards Artificial Intelligence Applications in Next Generation Cytopathology

Over the last 20 years we have seen an increase in techniques in the field of computational pathology and machine learning, improving our ability to analyze and interpret imaging. Neural networks, in particular, have been used for more than thirty years, starting with the computer assisted smear test using early generation models. Today, advanced machine learning, working on large image data sets, has been shown to perform classification, detection, and segmentation with remarkable accuracy and generalization in several domains. Deep learning algorithms, as a branch of machine learning, are thus attracting attention in digital pathology and cytopathology, providing feasible solutions for accurate and efficient cytological diagnoses, ranging from efficient cell counts to automatic classification of anomalous cells and queries over large clinical databases. The integration of machine learning with related next-generation technologies powered by AI, such as augmented/virtual reality, metaverse, and computational linguistic models are a focus of interest in health care digitalization, to support education, diagnosis, and therapy. In this work we will consider how all these innovations can help cytopathology to go beyond the microscope and to undergo a hyper-digitalized transformation. We also discuss specific challenges to their applications in the field, notably, the requirement for large-scale cytopathology datasets, the necessity of new protocols for sharing information, and the need for further technological training for pathologists.

Assessing the role of tumour-associated macrophage subsets in breast cancer subtypes using digital image analysis

PURPOSE

The number of M1-like and M2-like tumour-associated macrophages (TAMs) and their ratio can play a role in breast cancer development and progression. Early clinical trials using macrophage targeting compounds are currently ongoing. However, the most optimal detection method of M1-like and M2-like macrophage subsets and their clinical relevance in breast cancer is still unclear. We aimed to optimize the assessment of TAM subsets in different breast cancer subtypes, and therefore related TAM subset numbers and ratio to clinicopathological characteristics and clinical outcome.

METHODS

Tissue microarrays of 347 consecutive primary Luminal-A, Luminal-B, HER2-positive and triple-negative tumours of patients with early-stage breast cancer were serially sectioned and immunohistochemically stained for the pan-macrophage marker CD68 and the M2-like macrophage markers CD163, CSF-1R and CD206. TAM numbers were quantified using a digital image analysis algorithm. M1-like macrophage numbers were calculated by subtracting M2-like TAM numbers from the total TAM number.

RESULTS

M2-like markers CD163 and CSF-1R showed a moderate positive association with each other and with CD68 (r ≥ 0.47), but only weakly with CD206 (r ≤ 0.06). CD68 + , CD163 + and CSF-1R + macrophages correlated with tumour grade in Luminal-B tumours (P < 0.001). Total or subset TAM numbers did not correlate with disease outcome in any breast cancer subtype.

CONCLUSION

In conclusion, macrophages and their subsets can be detected by means of a panel of TAM markers and are related to unfavourable clinicopathological characteristics in Luminal-B breast cancer. However, their impact on outcome remains unclear. Preferably, this should be determined in prospective series.

Automated scoring of total inflammation in renal allograft biopsies

BACKGROUND

Computer-assisted scoring is gaining prominence in the evaluation of renal histology; however, much of the focus has been on identifying larger objects such as glomeruli. Total inflammation impacts graft outcome, and its quantification requires tools to identify objects at the cellular level or smaller. The goal of the current study was to use CD45 stained slides coupled with image analysis tools to quantify the amount of non-glomerular inflammation within the cortex.

METHODS

Sixty renal transplant whole slide images were used for digital image analysis. Multiple thresholding methods using pixel intensity and object size were used to identify inflammation in the cortex. Additionally, convolutional neural networks were used to separate glomeruli from other objects in the cortex. This combined measure of inflammation was then correlated with rescored Banff total inflammation classification and outcomes.

RESULTS

Identification of glomeruli on biopsies had high fidelity (mean pixelwise dice coefficient of .858). Continuous total inflammation scores correlated well with Banff rescoring (maximum Pearson correlation .824). A separate set of thresholds resulted in a significant correlation with alloimmune graft loss.

CONCLUSIONS

Automated scoring of inflammation showed a high correlation with Banff scoring. Digital image analysis provides a powerful tool for analysis of renal pathology, not only because it is reproducible and can be automated, but also because it provides much more granular data for studies.

Dissecting Tumor-Immune Microenvironment in Breast Cancer at a Spatial and Multiplex Resolution

The tumor immune microenvironment (TIME) is an important player in breast cancer pathophysiology. Surrogates for antitumor immune response have been explored as predictive biomarkers to immunotherapy, though with several limitations. Immunohistochemistry for programmed death ligand 1 suffers from analytical problems, immune signatures are devoid of spatial information and histopathological evaluation of tumor infiltrating lymphocytes exhibits interobserver variability. Towards improved understanding of the complex interactions in TIME, several emerging multiplex in situ methods are being developed and gaining much attention for protein detection. They enable the simultaneous evaluation of multiple targets in situ, detection of cell densities/subpopulations as well as estimations of functional states of immune infiltrate. Furthermore, they can characterize spatial organization of TIME—by cell-to-cell interaction analyses and the evaluation of distribution within different regions of interest and tissue compartments—while digital imaging and image analysis software allow for reproducibility of the various assays. In this review, we aim to provide an overview of the different multiplex in situ methods used in cancer research with special focus on breast cancer TIME at the neoadjuvant, adjuvant and metastatic setting. Spatial heterogeneity of TIME and importance of longitudinal evaluation of TIME changes under the pressure of therapy and metastatic progression are also addressed.

Objective Diagnosis for Histopathological Images Based on Machine Learning Techniques: Classical Approaches and New Trends

Histopathology refers to the examination by a pathologist of biopsy samples. Histopathology images are captured by a microscope to locate, examine, and classify many diseases, such as different cancer types. They provide a detailed view of different types of diseases and their tissue status. These images are an essential resource with which to define biological compositions or analyze cell and tissue structures. This imaging modality is very important for diagnostic applications. The analysis of histopathology images is a prolific and relevant research area supporting disease diagnosis. In this paper, the challenges of histopathology image analysis are evaluated. An extensive review of conventional and deep learning techniques which have been applied in histological image analyses is presented. This review summarizes many current datasets and highlights important challenges and constraints with recent deep learning techniques, alongside possible future research avenues. Despite the progress made in this research area so far, it is still a significant area of open research because of the variety of imaging techniques and disease-specific characteristics.

Immune contexture analysis in immuno-oncology: applications and challenges of multiplex fluorescent immunohistochemistry

The tumor microenvironment is an integral player in cancer initiation, tumor progression, response and resistance to anti-cancer therapy. Understanding the complex interactions of tumor immune architecture (referred to as 'immune contexture') has therefore become increasingly desirable to guide our approach to patient selection, clinical trial design, combination therapies, and patient management. Quantitative image analysis based on multiplexed fluorescence immunohistochemistry and deep learning technologies are rapidly developing to enable researchers to interrogate complex information from the tumor microenvironment and find predictive insights into treatment response. Herein, we discuss current developments in multiplexed fluorescence immunohistochemistry for immune contexture analysis, and their application in immuno-oncology, and discuss challenges to effectively use this technology in clinical settings. We also present a multiplexed image analysis workflow to analyse fluorescence multiplexed stained tumor sections using the Vectra Automated Digital Pathology System together with FCS express flow cytometry software. The benefit of this strategy is that the spectral unmixing accurately generates and analyses complex arrays of multiple biomarkers, which can be helpful for diagnosis, risk stratification, and guiding clinical management of oncology patients.

Deep learning-based image analysis methods for brightfield-acquired multiplex immunohistochemistry images

BACKGROUND

Multiplex immunohistochemistry (mIHC) permits the labeling of six or more distinct cell types within a single histologic tissue section. The classification of each cell type requires detection of the unique colored chromogens localized to cells expressing biomarkers of interest. The most comprehensive and reproducible method to evaluate such slides is to employ digital pathology and image analysis pipelines to whole-slide images (WSIs). Our suite of deep learning tools quantitatively evaluates the expression of six biomarkers in mIHC WSIs. These methods address the current lack of readily available methods to evaluate more than four biomarkers and circumvent the need for specialized instrumentation to spectrally separate different colors. The use case application for our methods is a study that investigates tumor immune interactions in pancreatic ductal adenocarcinoma (PDAC) with a customized mIHC panel.

METHODS

Six different colored chromogens were utilized to label T-cells (CD3, CD4, CD8), B-cells (CD20), macrophages (CD16), and tumor cells (K17) in formalin-fixed paraffin-embedded (FFPE) PDAC tissue sections. We leveraged pathologist annotations to develop complementary deep learning-based methods: (1) ColorAE is a deep autoencoder which segments stained objects based on color; (2) U-Net is a convolutional neural network (CNN) trained to segment cells based on color, texture and shape; and ensemble methods that employ both ColorAE and U-Net, collectively referred to as (3) ColorAE:U-Net. We assessed the performance of our methods using: structural similarity and DICE score to evaluate segmentation results of ColorAE against traditional color deconvolution; F1 score, sensitivity, positive predictive value, and DICE score to evaluate the predictions from ColorAE, U-Net, and ColorAE:U-Net ensemble methods against pathologist-generated ground truth. We then used prediction results for spatial analysis (nearest neighbor).

RESULTS

We observed that (1) the performance of ColorAE is comparable to traditional color deconvolution for single-stain IHC images (note: traditional color deconvolution cannot be used for mIHC); (2) ColorAE and U-Net are complementary methods that detect 6 different classes of cells with comparable performance; (3) combinations of ColorAE and U-Net into ensemble methods outperform using either ColorAE and U-Net alone; and (4) ColorAE:U-Net ensemble methods can be employed for detailed analysis of the tumor microenvironment (TME). We developed a suite of scalable deep learning methods to analyze 6 distinctly labeled cell populations in mIHC WSIs. We evaluated our methods and found that they reliably detected and classified cells in the PDAC tumor microenvironment. We also present a use case, wherein we apply the ColorAE:U-Net ensemble method across 3 mIHC WSIs and use the predictions to quantify all stained cell populations and perform nearest neighbor spatial analysis. Thus, we provide proof of concept that these methods can be employed to quantitatively describe the spatial distribution immune cells within the tumor microenvironment. These complementary deep learning methods are readily deployable for use in clinical research studies.

Overview of multiplex immunohistochemistry/immunofluorescence techniques in the era of cancer immunotherapy

Conventional immunohistochemistry (IHC) is a widely used diagnostic technique in tissue pathology. However, this technique is associated with a number of limitations, including high inter-observer variability and the capacity to label only one marker per tissue section. This review details various highly multiplexed techniques that have emerged to circumvent these constraints, allowing simultaneous detection of multiple markers on a single tissue section and the comprehensive study of cell composition, cellular functional and cell-cell interactions. Among these techniques, multiplex Immunohistochemistry/Immunofluorescence (mIHC/IF) has emerged to be particularly promising. mIHC/IF provides high-throughput multiplex staining and standardized quantitative analysis for highly reproducible, efficient and cost-effective tissue studies. This technique has immediate potential for translational research and clinical practice, particularly in the era of cancer immunotherapy.

Notch Signaling Pathway Expression in the Skin of Leprosy Patients: Association With Skin and Neural Damage

Introduction: Leprosy is an infectious disease caused by Mycobacterium leprae, a debilitating disease that affects the skin and peripheral nerves. It is possible that tissue changes during infection with leprosy are related to alterations in the activity of the Notch signaling pathway, an innate signaling pathway in the physiology of the skin and peripheral nerves.

Methods: This is a descriptive observational study. Thirty skin biopsies from leprosy patients and 15 from individuals with no history of this disease were evaluated. In these samples, gene expressions of cellular components associated with the Notch signaling pathway, Hes-1, Hey-1, Runx-1 Jagged-1, Notch-1, and Numb, were evaluated using q-PCR, and protein expression was evaluated using immunohistochemistry of Runx-1 and Hes-1.

Results: Changes were observed in the transcription of Notch signaling pathway components; Hes-1 was downregulated and Runx-1 upregulated in the skin of infected patients. These results were confirmed by immunohistochemistry, where reduction of Hes-1 expression was found in the epidermis, eccrine glands, and hair follicles. Increased expression of Runx-1 was found in inflammatory cells in the dermis of infected patients; however, it is not related to tissue changes. With these results, a multivariate analysis was performed to determine the causes of transcription factor Hes-1 reduction. It was concluded that tissue inflammation was the main cause.

Conclusions: The tissue changes found in the skin of infected patients could be associated with a reduction in the expression of Hes-1, a situation that would promote the survival and proliferation of M. leprae in this tissue.

Pathology Image Analysis Using Segmentation Deep Learning Algorithms

With the rapid development of image scanning techniques and visualization software, whole slide imaging (WSI) is becoming a routine diagnostic method. Accelerating clinical diagnosis from pathology images and automating image analysis efficiently and accurately remain significant challenges. Recently, deep learning algorithms have shown great promise in pathology image analysis, such as in tumor region identification, metastasis detection, and patient prognosis. Many machine learning algorithms, including convolutional neural networks, have been proposed to automatically segment pathology images. Among these algorithms, segmentation deep learning algorithms such as fully convolutional networks stand out for their accuracy, computational efficiency, and generalizability. Thus, deep learning-based pathology image segmentation has become an important tool in WSI analysis. In this review, the pathology image segmentation process using deep learning algorithms is described in detail. The goals are to provide quick guidance for implementing deep learning into pathology image analysis and to provide some potential ways of further improving segmentation performance. Although there have been previous reviews on using machine learning methods in digital pathology image analysis, this is the first in-depth review of the applications of deep learning algorithms for segmentation in WSI analysis.

Method for coregistration of optical measurements of breast tissue with histopathology: the importance of accounting for tissue deformations

For the validation of optical diagnostic technologies, experimental results need to be benchmarked against the gold standard. Currently, the gold standard for tissue characterization is assessment of hematoxylin and eosin (H&E)-stained sections by a pathologist. When processing tissue into H&E sections, the shape of the tissue deforms with respect to the initial shape when it was optically measured. We demonstrate the importance of accounting for these tissue deformations when correlating optical measurement with routinely acquired histopathology. We propose a method to register the tissue in the H&E sections to the optical measurements, which corrects for these tissue deformations. We compare the registered H&E sections to H&E sections that were registered with an algorithm that does not account for tissue deformations by evaluating both the shape and the composition of the tissue and using microcomputer tomography data as an independent measure. The proposed method, which did account for tissue deformations, was more accurate than the method that did not account for tissue deformations. These results emphasize the need for a registration method that accounts for tissue deformations, such as the method presented in this study, which can aid in validating optical techniques for clinical use.

State-of-the-Art of Profiling Immune Contexture in the Era of Multiplexed Staining and Digital Analysis to Study Paraffin Tumor Tissues

Multiplexed platforms for multiple epitope detection have emerged in the last years as very powerful tools to study tumor tissues. These revolutionary technologies provide important visual techniques for tumor examination in formalin-fixed paraffin-embedded specimens to improve the understanding of the tumor microenvironment, promote new treatment discoveries, aid in cancer prevention, as well as allowing translational studies to be carried out. The aim of this review is to highlight the more recent methodologies that use multiplexed staining to study simultaneous protein identification in formalin-fixed paraffin-embedded tumor tissues for immune profiling, clinical research, and potential translational analysis. New multiplexed methodologies, which permit the identification of several proteins at the same time in one single tissue section, have been developed in recent years with the ability to study different cell populations, cells by cells, and their spatial distribution in different tumor specimens including whole sections, core needle biopsies, and tissue microarrays. Multiplexed technologies associated with image analysis software can be performed with a high-quality throughput assay to study cancer specimens and are important tools for new discoveries. The different multiplexed technologies described in this review have shown their utility in the study of cancer tissues and their advantages for translational research studies and application in cancer prevention and treatments.

Spatial Mapping of Myeloid Cells and Macrophages by Multiplexed Tissue Staining

An array of phenotypically diverse myeloid cells and macrophages (MC&M) resides in the tumor microenvironment, requiring multiplexed detection systems for visualization. Here we report an automated, multiplexed staining approach, named PLEXODY, that consists of five MC&M-related fluorescently-tagged antibodies (anti - CD68, - CD163, - CD206, - CD11b, and - CD11c), and three chromogenic antibodies, reactive with high- and low-molecular weight cytokeratins and CD3, highlighting tumor regions, benign glands and T cells. The staining prototype and image analysis methods which include a pixel/area-based quantification were developed using tissues from inflamed colon and tonsil and revealed a unique tissue-specific composition of 14 MC&M-associated pixel classes. As a proof-of-principle, PLEXODY was applied to three cases of pancreatic, prostate and renal cancers. Across digital images from these cancer types we observed 10 MC&M-associated pixel classes at frequencies greater than 3%. Cases revealed higher frequencies of single positive compared to multi-color pixels and a high abundance of CD68+/CD163+ and CD68+/CD163+/CD206+ pixels. Significantly more CD68+ and CD163+ vs. CD11b+ and CD11c+ pixels were in direct contact with tumor cells and T cells. While the greatest percentage (~70%) of CD68+ and CD163+ pixels was 0–20 microns away from tumor and T cell borders, CD11b+ and CD11c+ pixels were detected up to 240 microns away from tumor/T cell masks. Together, these data demonstrate significant differences in densities and spatial organization of MC&M-associated pixel classes, but surprising similarities between the three cancer types.

Precision immunoprofiling by image analysis and artificial intelligence

Clinical success of immunotherapy is driving the need for new prognostic and predictive assays to inform patient selection and stratification. This requirement can be met by a combination of computational pathology and artificial intelligence. Here, we critically assess computational approaches supporting the development of a standardized methodology in the assessment of immune-oncology biomarkers, such as PD-L1 and immune cell infiltrates. We examine immunoprofiling through spatial analysis of tumor-immune cell interactions and multiplexing technologies as a predictor of patient response to cancer treatment. Further, we discuss how integrated bioinformatics can enable the amalgamation of complex morphological phenotypes with the multiomics datasets that drive precision medicine. We provide an outline to machine learning (ML) and artificial intelligence tools and illustrate fields of application in immune-oncology, such as pattern-recognition in large and complex datasets and deep learning approaches for survival analysis. Synergies of surgical pathology and computational analyses are expected to improve patient stratification in immuno-oncology. We propose that future clinical demands will be best met by (1) dedicated research at the interface of pathology and bioinformatics, supported by professional societies, and (2) the integration of data sciences and digital image analysis in the professional education of pathologists.

Machine Learning Methods for Histopathological Image Analysis

Abundant accumulation of digital histopathological images has led to the increased demand for their analysis, such as computer-aided diagnosis using machine learning techniques. However, digital pathological images and related tasks have some issues to be considered. In this mini-review, we introduce the application of digital pathological image analysis using machine learning algorithms, address some problems specific to such analysis, and propose possible solutions.