An Image Analysis Resource for Cancer Research: PIIP-Pathology Image Informatics Platform for Visualization, Analysis, and Management

Personalizable AI platform for universal access to research and diagnosis in digital pathology

BACKGROUND AND MOTIVATION

Digital pathology has been evolving over the last years, proposing significant workflow advantages that have fostered its adoption in professional environments. Patient clinical and image data are readily available in remote data banks that can be consumed efficiently over standard communication technologies. The appearance of new imaging techniques and advanced artificial intelligence algorithms has significantly reduced the burden on medical professionals by speeding up the screening process. Despite these advancements, the usage of digital pathology in professional environments has been slowed down by poor interoperability between services resulting from a lack of standard interfaces and integrative solutions. This work addresses this issue by proposing a cloud-based digital pathology platform built on standard and open interfaces.

METHODS

The work proposes and describes a vendor-neutral platform that provides interfaces for managing digital slides, and medical reports, and integrating digital image analysis services compatible with existing standards. The solution integrates the open-source plugin-based Dicoogle PACS for interoperability and extensibility, which grants the proposed solution great feature customization.

RESULTS

The solution was developed in collaboration with iPATH research project partners, including the validation by medical pathologists. The result is a pure Web collaborative framework that supports both research and production environments. A total of 566 digital slides from different pathologies were successfully uploaded to the platform. Using the integration interfaces, a mitosis detection algorithm was successfully installed into the platform, and it was trained with 2400 annotations collected from breast carcinoma images.

CONCLUSION

Interoperability is a key factor when discussing digital pathology solutions, as it facilitates their integration into existing institutions' information systems. Moreover, it improves data sharing and integration of third-party services such as image analysis services, which have become relevant in today's digital pathology workflow. The proposed solution fully embraces the DICOM standard for digital pathology, presenting an interoperable cloud-based solution that provides great feature customization thanks to its extensible architecture.

Targeting tumour-associated macrophages in hodgkin lymphoma using engineered extracellular matrix-mimicking cryogels

Tumour-associated macrophages are linked with poor prognosis and resistance to therapy in Hodgkin lymphoma; however, there are no suitable preclinical models to identify macrophage-targeting therapeutics. We used primary human tumours to guide the development of a mimetic cryogel, wherein Hodgkin (but not Non-Hodgkin) lymphoma cells promoted primary human macrophage invasion. In an invasion inhibitor screen, we identified five drug hits that significantly reduced tumour-associated macrophage invasion: marimastat, batimastat, AS1517499, ruxolitinib, and PD-169316. Importantly, ruxolitinib has demonstrated recent success in Hodgkin lymphoma clinical trials. Both ruxolitinib and PD-169316 (a p38 mitogen-activated protein kinase (p38 MAPK) inhibitor) decreased the percent of M2-like macrophages; however, only PD-169316 enhanced the percentage of M1-like macrophages. We validated p38 MAPK as an anti-invasion drug target with five additional drugs using a high-content imaging platform. With our biomimetic cryogel, we modeled macrophage invasion in Hodgkin lymphoma and then used it for target discovery and drug screening, ultimately identifying potential future therapeutics.

Annotating for Artificial Intelligence Applications in Digital Pathology: A Practical Guide for Pathologists and Researchers

Training machine learning models for artificial intelligence (AI) applications in pathology often requires extensive annotation by human experts, but there is little guidance on the subject. In this work, we aimed to describe our experience and provide a simple, useful, and practical guide addressing annotation strategies for AI development in computational pathology. Annotation methodology will vary significantly depending on the specific study's objectives, but common difficulties will be present across different settings. We summarize key aspects and issue guiding principles regarding team interaction, ground-truth quality assessment, different annotation types, and available software and hardware options and address common difficulties while annotating. This guide was specifically designed for pathology annotation, intending to help pathologists, other researchers, and AI developers with this process.

Machine learning in renal pathology

Introduction

When assessing kidney biopsies, pathologists use light microscopy, immunofluorescence, and electron microscopy to describe and diagnose glomerular lesions and diseases. These methods can be laborious, costly, fraught with inter-observer variability, and can have delays in turn-around time. Thus, computational approaches can be designed as screening and/or diagnostic tools, potentially relieving pathologist time, healthcare resources, while also having the ability to identify novel biomarkers, including subvisual features.

Methods

Here, we implement our recently published biomarker feature extraction (BFE) model along with 3 pre-trained deep learning models (VGG16, VGG19, and InceptionV3) to diagnose 3 glomerular diseases using PAS-stained digital pathology images alone. The BFE model extracts a panel of 233 explainable features related to underlying pathology, which are subsequently narrowed down to 10 morphological and microstructural texture features for classification with a linear discriminant analysis machine learning classifier. 45 patient renal biopsies (371 glomeruli) from minimal change disease (MCD), membranous nephropathy (MN), and thin-basement membrane nephropathy (TBMN) were split into training/validation and held out sets. For the 3 deep learningmodels, data augmentation and Grad-CAM were used for better performance and interpretability.

Results

The BFE model showed glomerular validation accuracy of 67.6% and testing accuracy of 76.8%. All deep learning approaches had higher validation accuracies (most for VGG16 at 78.5%) but lower testing accuracies. The highest testing accuracy at the glomerular level was VGG16 at 71.9%, while at the patient-level was InceptionV3 at 73.3%.

Discussion

The results highlight the potential of both traditional machine learning and deep learning-based approaches for kidney biopsy evaluation.

Advantages of manual and automatic computer-aided compared to traditional histopathological diagnosis of melanoma: A pilot study

BACKGROUND

Cutaneous malignant melanoma (CMM) accounts for the highest mortality rate among all skin cancers. Traditional histopathologic diagnosis may be limited by the pathologists' subjectivity. Second-opinion strategies and multidisciplinary consultations are usually performed to overcome this issue. An available solution in the future could be the use of automated solutions based on a computational algorithm that could help the pathologist in everyday practice. The aim of this pilot study was to investigate the potential diagnostic aid of a machine-based algorithm in the histopathologic diagnosis of CMM.

METHODS

We retrospectively examined excisional biopsies of 50 CMM and 20 benign congenital compound nevi. Hematoxylin and eosin (H&E) stained WSI were reviewed independently by two expert dermatopathologists. A fully automated pipeline for WSI processing to support the estimation and prioritization of the melanoma areas was developed.

RESULTS

The spatial distribution of the nuclei in the sample provided a multi-scale overview of the tumor. A global overview of the lesion's silhouette was achieved and, by increasing the magnification, the topological distribution of the nuclei and the most informative areas of interest for the CMM diagnosis were identified and highlighted. These silhouettes allow the histopathologist to discriminate between nevus and CMM with an accuracy of 96% without any extra information.

CONCLUSION

In this study we proposed an easy-to-use model that produces segmentations of CMM silhouettes at fine detail level.

Quantitative digital histopathology and machine learning to predict pathological complete response to chemotherapy in breast cancer patients using pre-treatment tumor biopsies

Complete pathological response (pCR) to neoadjuvant chemotherapy (NAC) is a prognostic factor for breast cancer (BC) patients and is correlated with improved survival. However, pCR rates are variable to standard NAC, depending on BC subtype. This study investigates quantitative digital histopathology coupled with machine learning (ML) to predict NAC response a priori. Clinicopathologic data and digitized slides of BC core needle biopsies were collected from 149 patients treated with NAC. The nuclei within the tumor regions were segmented on the histology images of biopsy samples using a weighted U-Net model. Five pathomic feature subsets were extracted from segmented digitized samples, including the morphological, intensity-based, texture, graph-based and wavelet features. Seven ML experiments were conducted with different feature sets to develop a prediction model of therapy response using a gradient boosting machine with decision trees. The models were trained and optimized using a five-fold cross validation on the training data and evaluated using an unseen independent test set. The prediction model developed with the best clinical features (tumor size, tumor grade, age, and ER, PR, HER2 status) demonstrated an area under the ROC curve (AUC) of 0.73. Various pathomic feature subsets resulted in models with AUCs in the range of 0.67 and 0.87, with the best results associated with the graph-based and wavelet features. The selected features among all subsets of the pathomic and clinicopathologic features included four wavelet and three graph-based features and no clinical features. The predictive model developed with these features outperformed the other models, with an AUC of 0.90, a sensitivity of 85% and a specificity of 82% on the independent test set. The results demonstrated the potential of quantitative digital histopathology features integrated with ML methods in predicting BC response to NAC. This study is a step forward towards precision oncology for BC patients to potentially guide future therapies.

Software tools and platforms in Digital Pathology: a review for clinicians and computer scientists

At the end of the twentieth century, a new technology was developed that allowed an entire tissue section to be scanned on an objective slide. Originally called virtual microscopy, this technology is now known as Whole Slide Imaging (WSI). WSI presents new challenges for reading, visualization, storage, and analysis. For this reason, several technologies have been developed to facilitate the handling of these images. In this paper, we analyze the most widely used technologies in the field of digital pathology, ranging from specialized libraries for the reading of these images to complete platforms that allow reading, visualization, and analysis. Our aim is to provide the reader, whether a pathologist or a computational scientist, with the knowledge to choose the technologies to use for new studies, development, or research.

General Roadmap and Core Steps for the Development of AI Tools in Digital Pathology

Integrating Artificial Intelligence (AI) tools in the tissue diagnostic workflow will benefit the pathologist and, ultimately, the patient. The generation of such AI tools has two parallel yet interconnected processes, namely the definition of the pathologist’s task to be delivered in silico, and the software development requirements. In this review paper, we demystify this process, from a viewpoint that joins experienced pathologists and data scientists, by proposing a general pathway and describing the core steps to build an AI digital pathology tool. In doing so, we highlight the importance of the collaboration between AI scientists and pathologists, from the initial formulation of the hypothesis to the final, ready-to-use product.

Automatic Recognition of Ragged Red Fibers in Muscle Biopsy from Patients with Mitochondrial Disorders

Mitochondrial dysfunction is considered to be a major cause of primary mitochondrial myopathy in children and adults, as reduced mitochondrial respiration and morphological changes such as ragged red fibers (RRFs) are observed in muscle biopsies. However, it is also possible to hypothesize the role of mitochondrial dysfunction in aging muscle or in secondary mitochondrial dysfunctions. The recognition of true histological patterns of mitochondrial myopathy can avoid unnecessary genetic investigations. The aim of our study was to develop and validate machine-learning methods for RRF detection in light microscopy images of skeletal muscle tissue. We used image sets of 489 color images captured from representative areas of Gomori’s trichrome-stained tissue retrieved from light microscopy images at a 20× magnification. We compared the performance of random forest, gradient boosting machine, and support vector machine classifiers. Our results suggested that the advent of scanning technologies, combined with the development of machine-learning models for image classification, make neuromuscular disorders’ automated diagnostic systems a concrete possibility.

Cell segmentation for immunofluorescence multiplexed images using two-stage domain adaptation and weakly labeled data for pre-training

Cellular profiling with multiplexed immunofluorescence (MxIF) images can contribute to a more accurate patient stratification for immunotherapy. Accurate cell segmentation of the MxIF images is an essential step. We propose a deep learning pipeline to train a Mask R-CNN model (deep network) for cell segmentation using nuclear (DAPI) and membrane (Na+K+ATPase) stained images. We used two-stage domain adaptation by first using a weakly labeled dataset followed by fine-tuning with a manually annotated dataset. We validated our method against manual annotations on three different datasets. Our method yields comparable results to the multi-observer agreement on an ovarian cancer dataset and improves on state-of-the-art performance on a publicly available dataset of mouse pancreatic tissues. Our proposed method, using a weakly labeled dataset for pre-training, showed superior performance in all of our experiments. When using smaller training sample sizes for fine-tuning, the proposed method provided comparable performance to that obtained using much larger training sample sizes. Our results demonstrate that using two-stage domain adaptation with a weakly labeled dataset can effectively boost system performance, especially when using a small training sample size. We deployed the model as a plug-in to CellProfiler, a widely used software platform for cellular image analysis.

MiNuGAN: Dual Segmentation of Mitoses and Nuclei Using Conditional GANs on Multi-center Breast H&E Images

Breast cancer is the second most commonly diagnosed type of cancer among women as of 2021. Grading of histopathological images is used to guide breast cancer treatment decisions and a critical component of this is a mitotic score, which is related to tumor aggressiveness. Manual mitosis counting is an extremely tedious manual task, but automated approaches can be used to overcome inefficiency and subjectivity. In this paper, we propose an automatic mitosis and nuclear segmentation method for a diverse set of H&E breast cancer pathology images. The method is based on a conditional generative adversarial network to segment both mitoses and nuclei at the same time. Architecture optimizations are investigated, including hyper parameters and the addition of a focal loss. The accuracy of the proposed method is investigated using images from multiple centers and scanners, including TUPAC16, ICPR14 and ICPR12 datasets. In TUPAC16, we use 618 carefully annotated images of size 256×256 scanned at 40×. TUPAC16 is used to train the model, and segmentation performance is measured on the test set for both nuclei and mitoses. Results on 200 held-out testing images from the TUPAC16 dataset were mean DSC = 0.784 and 0.721 for nuclear and mitosis, respectively. On 202 ICPR12 images, mitosis segmentation accuracy had a mean DSC = 0.782, indicating the model generalizes well to unseen datasets. For datasets that had mitosis centroid annotations, which included 200 TUPAC16, 202 ICPR12 and 524 ICPR14, a mean F1-score of 0.854 was found indicating high mitosis detection accuracy.

Explainable Biomarkers for Automated Glomerular and Patient-Level Disease Classification

Pathologists use multiple microscopy modalities to assess renal biopsy specimens. Besides usual diagnostic features, some changes are too subtle to be properly defined. Computational approaches have the potential to systematically quantitate subvisual clues, provide pathogenetic insight, and link to clinical outcomes. To this end, a proof-of-principle study is presented demonstrating that explainable biomarkers through machine learning can distinguish between glomerular disorders at the light-microscopy level. The proposed system used image analysis techniques and extracted 233 explainable biomarkers related to color, morphology, and microstructural texture. Traditional machine learning was then used to classify minimal change disease (MCD), membranous nephropathy (MN), and thin basement membrane nephropathy (TBMN) diseases on a glomerular and patient-level basis. The final model combined the Gini feature importance set and linear discriminant analysis classifier. Six morphologic (nuclei-to-glomerular tuft area, nuclei-to-glomerular area, glomerular tuft thickness greater than ten, glomerular tuft thickness greater than three, total glomerular tuft thickness, and glomerular circularity) and four microstructural texture features (luminal contrast using wavelets, nuclei energy using wavelets, nuclei variance using color vector LBP, and glomerular correlation using GLCM) were, together, the best performing biomarkers. Accuracies of 77% and 87% were obtained for glomerular and patient-level classification, respectively. Computational methods, using explainable glomerular biomarkers, have diagnostic value and are compatible with our existing knowledge of disease pathogenesis. Furthermore, this algorithm can be applied to clinical datasets for novel prognostic and mechanistic biomarker discovery.

Quick Annotator: an open-source digital pathology based rapid image annotation tool

Image-based biomarker discovery typically requires accurate segmentation of histologic structures (e.g. cell nuclei, tubules, and epithelial regions) in digital pathology whole slide images (WSIs). Unfortunately, annotating each structure of interest is laborious and often intractable even in moderately sized cohorts. Here, we present an open-source tool, Quick Annotator (QA), designed to improve annotation efficiency of histologic structures by orders of magnitude. While the user annotates regions of interest (ROIs) via an intuitive web interface, a deep learning (DL) model is concurrently optimized using these annotations and applied to the ROI. The user iteratively reviews DL results to either (1) accept accurately annotated regions or (2) correct erroneously segmented structures to improve subsequent model suggestions, before transitioning to other ROIs. We demonstrate the effectiveness of QA over comparable manual efforts via three use cases. These include annotating (1) 337,386 nuclei in 5 pancreatic WSIs, (2) 5,692 tubules in 10 colorectal WSIs, and (3) 14,187 regions of epithelium in 10 breast WSIs. Efficiency gains in terms of annotations per second of 102×, 9×, and 39× were, respectively, witnessed while retaining f-scores >0.95, suggesting that QA may be a valuable tool for efficiently fully annotating WSIs employed in downstream biomarker studies.

SPIE-AAPM-NCI BreastPathQ challenge: an image analysis challenge for quantitative tumor cellularity assessment in breast cancer histology images following neoadjuvant treatment

Purpose: The breast pathology quantitative biomarkers (BreastPathQ) challenge was a grand challenge organized jointly by the International Society for Optics and Photonics (SPIE), the American Association of Physicists in Medicine (AAPM), the U.S. National Cancer Institute (NCI), and the U.S. Food and Drug Administration (FDA). The task of the BreastPathQ challenge was computerized estimation of tumor cellularity (TC) in breast cancer histology images following neoadjuvant treatment. Approach: A total of 39 teams developed, validated, and tested their TC estimation algorithms during the challenge. The training, validation, and testing sets consisted of 2394, 185, and 1119 image patches originating from 63, 6, and 27 scanned pathology slides from 33, 4, and 18 patients, respectively. The summary performance metric used for comparing and ranking algorithms was the average prediction probability concordance (PK) using scores from two pathologists as the TC reference standard. Results: Test PK performance ranged from 0.497 to 0.941 across the 100 submitted algorithms. The submitted algorithms generally performed well in estimating TC, with high-performing algorithms obtaining comparable results to the average interrater PK of 0.927 from the two pathologists providing the reference TC scores. Conclusions: The SPIE-AAPM-NCI BreastPathQ challenge was a success, indicating that artificial intelligence/machine learning algorithms may be able to approach human performance for cellularity assessment and may have some utility in clinical practice for improving efficiency and reducing reader variability. The BreastPathQ challenge can be accessed on the Grand Challenge website.

CytoBrowser: a browser-based collaborative annotation platform for whole slide images

We present CytoBrowser, an open-source (GPLv3) JavaScript and Node.js driven environment for fast and accessible collaborative online visualization, assessment, and annotation of very large microscopy images, including, but not limited to, z-stacks (focus stacks) of cytology or histology whole slide images. CytoBrowser provides a web-based viewer for high-resolution zoomable images and facilitates easy remote collaboration, with options for joint-view visualization and simultaneous collaborative annotation of very large datasets. It delivers a unique combination of functionalities not found in other software solutions, making it a preferred tool for large scale annotation of whole slide image data. The web browser interface is directly accessible on any modern computer or even on a mobile phone, without need for additional software. By sharing a "session", several remote users can interactively explore and jointly annotate whole slide image data, thereby reaching improved data understanding and annotation quality, effortless project scaling and distribution of resources to/from remote locations, efficient creation of "ground truth" annotations for methods' evaluation and training of machine learning-based approaches, a user-friendly learning environment for medical students, to just name a few. Rectangle and polygon region annotations complement point-based annotations, each with a selectable annotation-class as well as free-form text fields. The default setting of CytoBrowser presents an interface for the Bethesda cancer grading system, while other annotation schemes can easily be incorporated. Automatic server side storage of annotations is complemented by JSON-based import/export options facilitating easy interoperability with other tools. CytoBrowser is available here: https://mida-group.github.io/CytoBrowser/.

Harnessing non-destructive 3D pathology

High-throughput methods for slide-free three-dimensional (3D) pathological analyses of whole biopsies and surgical specimens offer the promise of modernizing traditional histology workflows and delivering improvements in diagnostic performance. Advanced optical methods now enable the interrogation of orders of magnitude more tissue than previously possible, where volumetric imaging allows for enhanced quantitative analyses of cell distributions and tissue structures that are prognostic and predictive. Non-destructive imaging processes can simplify laboratory workflows, potentially reducing costs, and can ensure that samples are available for subsequent molecular assays. However, the large size of the feature-rich datasets that they generate poses challenges for data management and computer-aided analysis. In this Perspective, we provide an overview of the imaging technologies that enable 3D pathology, and the computational tools-machine learning, in particular-for image processing and interpretation. We also discuss the integration of various other diagnostic modalities with 3D pathology, along with the challenges and opportunities for clinical adoption and regulatory approval.

Machine learning techniques for mitoses classification

BACKGROUND

Pathologists analyze biopsy material at both the cellular and structural level to determine diagnosis and cancer stage. Mitotic figures are surrogate biomarkers of cellular proliferation that can provide prognostic information; thus, their precise detection is an important factor for clinical care. Convolutional Neural Networks (CNNs) have shown remarkable performance on several recognition tasks. Utilizing CNNs for mitosis classification may aid pathologists to improve the detection accuracy.

METHODS

We studied two state-of-the-art CNN-based models, ESPNet and DenseNet, for mitosis classification on six whole slide images of skin biopsies and compared their quantitative performance in terms of sensitivity, specificity, and F-score. We used raw RGB images of mitosis and non-mitosis samples with their corresponding labels as training input. In order to compare with other work, we studied the performance of these classifiers and two other architectures, ResNet and ShuffleNet, on the publicly available MITOS breast biopsy dataset and compared the performance of all four in terms of precision, recall, and F-score (which are standard for this data set), architecture, training time and inference time.

RESULTS

The ESPNet and DenseNet results on our primary melanoma dataset had a sensitivity of 0.976 and 0.968, and a specificity of 0.987 and 0.995, respectively, with F-scores of .968 and .976, respectively. On the MITOS dataset, ESPNet and DenseNet showed a sensitivity of 0.866 and 0.916, and a specificity of 0.973 and 0.980, respectively. The MITOS results using DenseNet had a precision of 0.939, recall of 0.916, and F-score of 0.927. The best published result on MITOS (Saha et al. 2018) reported precision of 0.92, recall of 0.88, and F-score of 0.90. In our architecture comparisons on MITOS, we found that DenseNet beats the others in terms of F-Score (DenseNet 0.927, ESPNet 0.890, ResNet 0.865, ShuffleNet 0.847) and especially Recall (DenseNet 0.916, ESPNet 0.866, ResNet 0.807, ShuffleNet 0.753), while ResNet and ESPNet have much faster inference times (ResNet 6 s, ESPNet 8 s, DenseNet 31 s). ResNet is faster than ESPNet, but ESPNet has a higher F-Score and Recall than ResNet, making it a good compromise solution.

CONCLUSION

We studied several state-of-the-art CNNs for detecting mitotic figures in whole slide biopsy images. We evaluated two CNNs on a melanoma cancer dataset and then compared four CNNs on a public breast cancer data set, using the same methodology on both. Our methodology and architecture for mitosis finding in both melanoma and breast cancer whole slide images has been thoroughly tested and is likely to be useful for finding mitoses in any whole slide biopsy images.

piNET-An Automated Proliferation Index Calculator Framework for Ki67 Breast Cancer Images

In this work, a novel proliferation index (PI) calculator for Ki67 images called piNET is proposed. It is successfully tested on four datasets, from three scanners comprised of patches, tissue microarrays (TMAs) and whole slide images (WSI), representing a diverse multi-centre dataset for evaluating Ki67 quantification. Compared to state-of-the-art methods, piNET consistently performs the best over all datasets with an average PI difference of 5.603%, PI accuracy rate of 86% and correlation coefficient R = 0.927. The success of the system can be attributed to several innovations. Firstly, this tool is built based on deep learning, which can adapt to wide variability of medical images-and it was posed as a detection problem to mimic pathologists' workflow which improves accuracy and efficiency. Secondly, the system is trained purely on tumor cells, which reduces false positives from non-tumor cells without needing the usual pre-requisite tumor segmentation step for Ki67 quantification. Thirdly, the concept of learning background regions through weak supervision is introduced, by providing the system with ideal and non-ideal (artifact) patches that further reduces false positives. Lastly, a novel hotspot analysis is proposed to allow automated methods to score patches from WSI that contain "significant" activity.

AI in Medical Imaging Informatics: Current Challenges and Future Directions

This paper reviews state-of-the-art research solutions across the spectrum of medical imaging informatics, discusses clinical translation, and provides future directions for advancing clinical practice. More specifically, it summarizes advances in medical imaging acquisition technologies for different modalities, highlighting the necessity for efficient medical data management strategies in the context of AI in big healthcare data analytics. It then provides a synopsis of contemporary and emerging algorithmic methods for disease classification and organ/ tissue segmentation, focusing on AI and deep learning architectures that have already become the de facto approach. The clinical benefits of in-silico modelling advances linked with evolving 3D reconstruction and visualization applications are further documented. Concluding, integrative analytics approaches driven by associate research branches highlighted in this study promise to revolutionize imaging informatics as known today across the healthcare continuum for both radiology and digital pathology applications. The latter, is projected to enable informed, more accurate diagnosis, timely prognosis, and effective treatment planning, underpinning precision medicine.

MLCD: A Unified Software Package for Cancer Diagnosis

PURPOSE

Machine Learning Package for Cancer Diagnosis (MLCD) is the result of a National Institutes of Health/National Cancer Institute (NIH/NCI)-sponsored project for developing a unified software package from state-of-the-art breast cancer biopsy diagnosis and machine learning algorithms that can improve the quality of both clinical practice and ongoing research.

METHODS

Whole-slide images of 240 well-characterized breast biopsy cases, initially assembled under R01 CA140560, were used for developing the algorithms and training the machine learning models. This software package is based on the methodology developed and published under our recent NIH/NCI-sponsored research grant (R01 CA172343) for finding regions of interest (ROIs) in whole-slide breast biopsy images, for segmenting ROIs into histopathologic tissue types and for using this segmentation in classifiers that can suggest final diagnoses.

RESULT

The package provides an ROI detector for whole-slide images and modules for semantic segmentation into tissue classes and diagnostic classification into 4 classes (benign, atypia, ductal carcinoma in situ, invasive cancer) of the ROIs. It is available through the GitHub repository under the Massachusetts Institute of Technology license and will later be distributed with the Pathology Image Informatics Platform system. A Web page provides instructions for use.

CONCLUSION

Our tools have the potential to provide help to other cancer researchers and, ultimately, to practicing physicians and will motivate future research in this field. This article describes the methodology behind the software development and gives sample outputs to guide those interested in using this package.

Automated detection and quantification of breast cancer brain metastases in an animal model using democratized machine learning tools

Advances in digital whole-slide imaging and machine learning (ML) provide new opportunities for automated examination and quantification of histopathological slides to support pathologists and biologists. However, implementation of ML tools often requires advanced skills in computer science that may not be immediately available in the traditional wet-lab environment. Here, we propose a simple and accessible workflow to automate detection and quantification of brain epithelial metastases on digitized histological slides. We leverage 100 Hematoxylin & Eosin (H&E)-stained whole slide images (WSIs) from 25 Balb/c mice with various level of brain metastatic tumor burden. A supervised training of the Trainable Weka Segmentation (TWS) from Fiji was achieved from annotated WSIs. Upon comparison with manually drawn regions, it is apparent that the algorithm learned to identify and segment cancer cell-specific nuclei and normal brain tissue. Our approach resulted in a robust and highly concordant correlation between automated metastases quantification of brain metastases and manual human assessment (R² = 0.8783; P < 0.0001). This simple approach is amenable to other similar analyses, including that of human tissues. Widespread adoption of these tools aims to democratize ML and improve precision in traditionally qualitative tasks in histopathology-based research.

IHC Color Histograms for Unsupervised Ki67 Proliferation Index Calculation

Automated image analysis tools for Ki67 breast cancer digital pathology images would have significant value if integrated into diagnostic pathology workflows. Such tools would reduce the workload of pathologists, while improving efficiency, and accuracy. Developing tools that are robust and reliable to multicentre data is challenging, however, differences in staining protocols, digitization equipment, staining compounds, and slide preparation can create variabilities in image quality and color across digital pathology datasets. In this work, a novel unsupervised color separation framework based on the IHC color histogram (IHCCH) is proposed for the robust analysis of Ki67 and hematoxylin stained images in multicentre datasets. An "overstaining" threshold is implemented to adjust for background overstaining, and an automated nuclei radius estimator is designed to improve nuclei detection. Proliferation index and F1 scores were compared between the proposed method and manually labeled ground truth data for 30 TMA cores that have ground truths for Ki67+ and Ki67- nuclei. The method accurately quantified the PI over the dataset, with an average proliferation index difference of 3.25%. To ensure the method generalizes to new, diverse datasets, 50 Ki67 TMAs from the Protein Atlas were used to test the validated approach. As the ground truth for this dataset is PI ranges, the automated result was compared to the PI range. The proposed method correctly classified 74 out of 80 TMA images, resulting in a 92.5% accuracy. In addition to these validations experiments, performance was compared to two color-deconvolution based methods, and to six machine learning classifiers. In all cases, the proposed work maintained more consistent (reproducible) results, and higher PI quantification accuracy.

Automated and Manual Quantification of Tumour Cellularity in Digital Slides for Tumour Burden Assessment

The residual cancer burden index is an important quantitative measure used for assessing treatment response following neoadjuvant therapy for breast cancer. It has shown to be predictive of overall survival and is composed of two key metrics: qualitative assessment of lymph nodes and the percentage of invasive or in situ tumour cellularity (TC) in the tumour bed (TB). Currently, TC is assessed through eye-balling of routine histopathology slides estimating the proportion of tumour cells within the TB. With the advances in production of digitized slides and increasing availability of slide scanners in pathology laboratories, there is potential to measure TC using automated algorithms with greater precision and accuracy. We describe two methods for automated TC scoring: 1) a traditional approach to image analysis development whereby we mimic the pathologists’ workflow, and 2) a recent development in artificial intelligence in which features are learned automatically in deep neural networks using image data alone. We show strong agreements between automated and manual analysis of digital slides. Agreements between our trained deep neural networks and experts in this study (0.82) approach the inter-rater agreements between pathologists (0.89). We also reveal properties that are captured when we apply deep neural network to whole slide images, and discuss the potential of using such visualisations to improve upon TC assessment in the future.

Deep Learning-Based Retrieval System for Gigapixel Histopathology Cases and the Open Access Literature

Background:

The introduction of digital pathology into clinical practice has led to the development of clinical workflows with digital images, in connection with pathology reports. Still, most of the current work is time-consuming manual analysis of image areas at different scales. Links with data in the biomedical literature are rare, and a need for search based on visual similarity within whole slide images (WSIs) exists.

Objectives:

The main objective of the work presented is to integrate content-based visual retrieval with a WSI viewer in a prototype. Another objective is to connect cases analyzed in the viewer with cases or images from the biomedical literature, including the search through visual similarity and text.

Methods:

An innovative retrieval system for digital pathology is integrated with a WSI viewer, allowing to define regions of interest (ROIs) in images as queries for finding visually similar areas in the same or other images and to zoom in/out to find structures at varying magnification levels. The algorithms are based on a multimodal approach, exploiting both text information and content-based image features.

Results:

The retrieval system allows viewing WSIs and searching for regions that are visually similar to manually defined ROIs in various data sources (proprietary and public datasets, e.g., scientific literature). The system was tested by pathologists, highlighting its capabilities and suggesting ways to improve it and make it more usable in clinical practice.

Conclusions:

The developed system can enhance the practice of pathologists by enabling them to use their experience and knowledge to control artificial intelligence tools for navigating repositories of images for clinical decision support and teaching, where the comparison with visually similar cases can help to avoid misinterpretations. The system is available as open source, allowing the scientific community to test, ideate and develop similar systems for research and clinical practice.

Biomedical Image Processing with Containers and Deep Learning: An Automated Analysis Pipeline: Data architecture, artificial intelligence, automated processing, containerization, and clusters orchestration ease the transition from data acquisition to insights in medium-to-large datasets

Here, a streamlined, scalable, laboratory approach is discussed that enables medium-to-large dataset analysis. The presented approach combines data management, artificial intelligence, containerization, cluster orchestration, and quality control in a unified analytic pipeline. The unique combination of these individual building blocks creates a new and powerful analysis approach that can readily be applied to medium-to-large datasets by researchers to accelerate the pace of research. The proposed framework is applied to a project that counts the number of plasmonic nanoparticles bound to peripheral blood mononuclear cells in dark-field microscopy images. By using the techniques presented in this article, the images are automatically processed overnight, without user interaction, streamlining the path from experiment to conclusions.

Dual-Personality DICOM-TIFF for Whole Slide Images: A Migration Technique for Legacy Software

Despite recently organized Digital Imaging and Communications in Medicine (DICOM) testing and demonstration events involving numerous participating vendors, it is still the case that scanner manufacturers, software developers, and users continue to depend on proprietary file formats rather than adopting the standard DICOM whole slide microscopic image object. Many proprietary formats are Tagged Image File Format (TIFF) based, and existing applications and libraries can read tiled TIFF files. The sluggish adoption of DICOM for whole slide image encoding can be temporarily mitigated by the use of dual-personality DICOM-TIFF files. These are compatible with the installed base of TIFF-based software, as well as newer DICOM-based software. The DICOM file format was deliberately designed to support this dual-personality capability for such transitional situations, although it is rarely used. Furthermore, existing TIFF files can be converted into dual-personality DICOM-TIFF without changing the pixel data. This paper demonstrates the feasibility of extending the dual-personality concept to multiframe-tiled pyramidal whole slide images and explores the issues encountered. Open source code and sample converted images are provided for testing.

Advanced Morphologic Analysis for Diagnosing Allograft Rejection: The Case of Cardiac Transplant Rejection

Allograft rejection remains a significant concern after all solid organ transplants. Although qualitative morphologic analysis with histologic grading of biopsy samples is the main tool employed for diagnosing allograft rejection, this standard has significant limitations in precision and accuracy that affect patient care. The use of endomyocardial biopsy to diagnose cardiac allograft rejection illustrates the significant shortcomings of current approaches for diagnosing allograft rejection. Despite disappointing interobserver variability, concerns about discordance with clinical trajectories, attempts at revising the histologic criteria and efforts to establish new diagnostic tools with imaging and gene expression profiling, no method has yet supplanted endomyocardial biopsy as the diagnostic gold standard. In this context, automated approaches to complex data analysis problems—often referred to as “machine learning”—represent promising strategies to improve overall diagnostic accuracy. By focusing on cardiac allograft rejection, where tissue sampling is relatively frequent, this review highlights the limitations of the current approach to diagnosing allograft rejection, introduces the basic methodology behind machine learning and automated image feature detection, and highlights the initial successes of these approaches within cardiovascular medicine.

The authors review “machine learning“ and “automated image feature detection“ to improve the diagnostic accuracy of endomyocardial biopsies to diagnose cardiac allograft rejection.

AACR 2019 — Congrès de l’association américaine de recherche contre le cancer

Dans ce numéro spécial de la revueOncologie, les principaux points discutés au congrès de l’Association américaine pour la recherche sur le cancer (AACR) sont rapportés. L’objectif ici est de présenter de manière concise des exposés qui méritent une attention toute particulière. Le programme de la réunion de l’AACR de cette année, qui a eu lieu à Atlanta, a couvert les dernières découvertes de tout le spectre de la recherche sur le cancer — des sciences de la population à la prévention ; biologie du cancer, études translationnelles et cliniques ; à la survie et à la défense des droits — et souligne le travail des meilleurs esprits en matière de recherche et de médecine d’institutions du monde entier. Le congrès qui a duré cinq jours a proposé un programme multidisciplinaire couvrant tous les aspects de la recherche sur le cancer depuis ses bases fondamentales jusqu’à ses applications translationnelles et cliniques. Grâce à notre compréhension accrue des bases moléculaires du cancer, de nombreuses thérapies ciblées nouvelles ont émergé. Ainsi, notre compréhension sur la façon dont les tumeurs échappent aux attaques du système immunitaire a conduit au développement de nouvelles thérapies. Compte tenu de l’importance accrue de l’immunothérapie dans le traitement du cancer, nous présentons ici les dernières avancées dans ce domaine. Enfin, d’autres approches telles que l’étude du microbiome, l’épigénétique et l’intelligence artificielle comme un outil dans la recherche sur le cancer ont aussi été discutées au congrès de l’AACR 2019.

Computer-Aided Laser Dissection: A Microdissection Workflow Leveraging Image Analysis Tools

Introduction

The development and application of new molecular diagnostic assays based on next-generation sequencing and proteomics require improved methodologies for procurement of target cells from histological sections. Laser microdissection can successfully isolate distinct cells from tissue specimens based on visual selection for many research and clinical applications. However, this can be a daunting task when a large number of cells are required for molecular analysis or when a sizeable number of specimens need to be evaluated.

Materials and Methods

To improve the efficiency of the cellular identification process, we describe a microdissection workflow that leverages recently developed and open source image analysis algorithms referred to as computer-aided laser dissection (CALD). CALD permits a computer algorithm to identify the cells of interest and drive the dissection process.

Results

We describe several "use cases" that demonstrate the integration of image analytic tools probabilistic pairwise Markov model, ImageJ, spatially invariant vector quantization (SIVQ), and eSeg onto the ThermoFisher Scientific ArcturusXT and Leica LMD7000 microdissection platforms.

Conclusions

The CALD methodology demonstrates the integration of image analysis tools with the microdissection workflow and shows the potential impact to clinical and life science applications.

Twenty Years of Digital Pathology: An Overview of the Road Travelled, What is on the Horizon, and the Emergence of Vendor-Neutral Archives

Almost 20 years have passed since the commercial introduction of whole-slide imaging (WSI) scanners. During this time, the creation of various WSI devices with the ability to digitize an entire glass slide has transformed the field of pathology. Parallel advances in computational technology and storage have permitted rapid processing of large-scale WSI datasets. This article provides an overview of important past and present efforts related to WSI. An account of how the virtual microscope evolved from the need to visualize and manage satellite data for earth science applications is provided. The article also discusses important milestones beginning from the first WSI scanner designed by Bacus to the Food and Drug Administration approval of the first digital pathology system for primary diagnosis in surgical pathology. As pathology laboratories commit to going fully digitalize, the need has emerged to include WSIs into an enterprise-level vendor-neutral archive (VNA). The different types of VNAs available are reviewed as well as how best to implement them and how pathology can benefit from participating in this effort. Differences between traditional image algorithms that extract pixel-, object-, and semantic-level features versus deep learning methods are highlighted. The need for large-scale data management, analysis, and visualization in computational pathology is also addressed.

Serum-derived carcinoembryonic antigen (CEA) activates fibroblasts to induce a local re-modeling of the extracellular matrix that favors the engraftment of CEA-expressing tumor cells

Elevated levels of the carcinoembryonic antigen (CEA; CEACAM5) in the serum of colorectal cancer (CRC) patients represent a clinical biomarker that correlates with disease recurrence. However, a mechanistic role for soluble CEA (sCEA) in tumor progression and metastasis remains to be established. In our study, we report that sCEA acts as a paracrine factor, activating human fibroblasts by signaling through both the STAT3 and AKT1-mTORC1 pathways, promoting their transition to a cancer-associated fibroblast (CaF) phenotype. sCEA-activated fibroblasts express and secrete higher levels of fibronectin, including cellular EDA⁺ -fibronectin (Fn-EDA) that selectively promote the implantation and adherence of CEA-expressing cancer cells. Immunohistochemical analyses of liver tissues derived from CRC patients with elevated levels of sCEA reveal that the expression of cellular Fn-EDA co-registers with CEA-expressing liver metastases. Taken together, these findings indicate a direct role for sCEA as a human fibroblast activation factor, in priming target tissues for the engraftment of CEA-expressing cancer cells, through the differentiation of tissue-resident fibroblasts, resulting in a local change in composition of the extracellular matrix.

A Cluster-then-label Semi-supervised Learning Approach for Pathology Image Classification

Completely labeled pathology datasets are often challenging and time-consuming to obtain. Semi-supervised learning (SSL) methods are able to learn from fewer labeled data points with the help of a large number of unlabeled data points. In this paper, we investigated the possibility of using clustering analysis to identify the underlying structure of the data space for SSL. A cluster-then-label method was proposed to identify high-density regions in the data space which were then used to help a supervised SVM in finding the decision boundary. We have compared our method with other supervised and semi-supervised state-of-the-art techniques using two different classification tasks applied to breast pathology datasets. We found that compared with other state-of-the-art supervised and semi-supervised methods, our SSL method is able to improve classification performance when a limited number of labeled data instances are made available. We also showed that it is important to examine the underlying distribution of the data space before applying SSL techniques to ensure semi-supervised learning assumptions are not violated by the data.

The transition module: a method for preventing overfitting in convolutional neural networks

Digital pathology has advanced substantially over the last decade with the adoption of slide scanners in pathology labs. The use of digital slides to analyse diseases at the microscopic level is both cost-effective and efficient. Identifying complex tumour patterns in digital slides is a challenging problem but holds significant importance for tumour burden assessment, grading and many other pathological assessments in cancer research. The use of convolutional neural networks (CNNs) to analyse such complex images has been well adopted in digital pathology. However, in recent years, the architecture of CNNs has altered with the introduction of inception modules which have shown great promise for classification tasks. In this paper, we propose a modified 'transition' module which encourages generalisation in a deep learning framework with few training samples. In the transition module, filters of varying sizes are used to encourage class-specific filters at multiple spatial resolutions followed by global average pooling. We demonstrate the performance of the transition module in AlexNet and ZFNet, for classifying breast tumours in two independent data-sets of scanned histology sections; the inclusion of the transition module in these CNNs improved performance.