High-throughput adaptive sampling for whole-slide histopathology image analysis (HASHI) via convolutional neural networks: Application to invasive breast cancer detection

Automated and Manual Quantification of Tumour Cellularity in Digital Slides for Tumour Burden Assessment

The residual cancer burden index is an important quantitative measure used for assessing treatment response following neoadjuvant therapy for breast cancer. It has shown to be predictive of overall survival and is composed of two key metrics: qualitative assessment of lymph nodes and the percentage of invasive or in situ tumour cellularity (TC) in the tumour bed (TB). Currently, TC is assessed through eye-balling of routine histopathology slides estimating the proportion of tumour cells within the TB. With the advances in production of digitized slides and increasing availability of slide scanners in pathology laboratories, there is potential to measure TC using automated algorithms with greater precision and accuracy. We describe two methods for automated TC scoring: 1) a traditional approach to image analysis development whereby we mimic the pathologists’ workflow, and 2) a recent development in artificial intelligence in which features are learned automatically in deep neural networks using image data alone. We show strong agreements between automated and manual analysis of digital slides. Agreements between our trained deep neural networks and experts in this study (0.82) approach the inter-rater agreements between pathologists (0.89). We also reveal properties that are captured when we apply deep neural network to whole slide images, and discuss the potential of using such visualisations to improve upon TC assessment in the future.

Staining Invariant Features for Improving Generalization of Deep Convolutional Neural Networks in Computational Pathology

One of the main obstacles for the implementation of deep convolutional neural networks (DCNNs) in the clinical pathology workflow is their low capability to overcome variability in slide preparation and scanner configuration, that leads to changes in tissue appearance. Some of these variations may not be not included in the training data, which means that the models have a risk to not generalize well. Addressing such variations and evaluating them in reproducible scenarios allows understanding of when the models generalize better, which is crucial for performance improvements and better DCNN models. Staining normalization techniques (often based on color deconvolution and deep learning) and color augmentation approaches have shown improvements in the generalization of the classification tasks for several tissue types. Domain-invariant training of DCNN's is also a promising technique to address the problem of training a single model for different domains, since it includes the source domain information to guide the training toward domain-invariant features, achieving state-of-the-art results in classification tasks. In this article, deep domain adaptation in convolutional networks (DANN) is applied to computational pathology and compared with widely used staining normalization and color augmentation methods in two challenging classification tasks. The classification tasks rely on two openly accessible datasets, targeting Gleason grading in prostate cancer, and mitosis classification in breast tissue. The benchmark of the different techniques and their combination in two DCNN architectures allows us to assess the generalization abilities and advantages of each method in the considered classification tasks. The code for reproducing our experiments and preprocessing the data is publicly available¹. Quantitative and qualitative results show that the use of DANN helps model generalization to external datasets. The combination of several techniques to manage color heterogeneity suggests that several methods together, such as color augmentation methods with DANN training, can generalize even further. The results do not show a single best technique among the considered methods, even when combining them. However, color augmentation and DANN training obtain most often the best results (alone or combined with color normalization and color augmentation). The statistical significance of the results and the embeddings visualizations provide useful insights to design DCNN that generalizes to unseen staining appearances. Furthermore, in this work, we release for the first time code for DANN evaluation in open access datasets for computational pathology. This work opens the possibility for further research on using DANN models together with techniques that can overcome the tissue preparation differences across datasets to tackle limited generalization.

Artificial Intelligence Algorithms to Assess Hormonal Status From Tissue Microarrays in Patients With Breast Cancer

IMPORTANCE

Immunohistochemistry (IHC) is the most widely used assay for identification of molecular biomarkers. However, IHC is time consuming and costly, depends on tissue-handling protocols, and relies on pathologists' subjective interpretation. Image analysis by machine learning is gaining ground for various applications in pathology but has not been proposed to replace chemical-based assays for molecular detection.

OBJECTIVE

To assess the prediction feasibility of molecular expression of biomarkers in cancer tissues, relying only on tissue architecture as seen in digitized hematoxylin-eosin (H&E)-stained specimens.

DESIGN, SETTING, AND PARTICIPANTS

This single-institution retrospective diagnostic study assessed the breast cancer tissue microarrays library of patients from Vancouver General Hospital, British Columbia, Canada. The study and analysis were conducted from July 1, 2015, through July 1, 2018. A machine learning method, termed morphological-based molecular profiling (MBMP), was developed. Logistic regression was used to explore correlations between histomorphology and biomarker expression, and a deep convolutional neural network was used to predict the biomarker expression in examined tissues.

MAIN OUTCOMES AND MEASURES

Positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristics curve measures of MBMP for assessment of molecular biomarkers.

RESULTS

The database consisted of 20 600 digitized, publicly available H&E-stained sections of 5356 patients with breast cancer from 2 cohorts. The median age at diagnosis was 61 years for cohort 1 (412 patients) and 62 years for cohort 2 (4944 patients), and the median follow-up was 12.0 years and 12.4 years, respectively. Tissue histomorphology was significantly correlated with the molecular expression of all 19 biomarkers assayed, including estrogen receptor (ER), progesterone receptor (PR), and ERBB2 (formerly HER2). Expression of ER was predicted for 105 of 207 validation patients in cohort 1 (50.7%) and 1059 of 2046 validation patients in cohort 2 (51.8%), with PPVs of 97% and 98%, respectively, NPVs of 68% and 76%, respectively, and accuracy of 91% and 92%, respectively, which were noninferior to traditional IHC (PPV, 91%-98%; NPV, 51%-78%; and accuracy, 81%-90%). Diagnostic accuracy improved given more data. Morphological analysis of patients with ER-negative/PR-positive status by IHC revealed resemblance to patients with ER-positive status (Bhattacharyya distance, 0.03) and not those with ER-negative/PR-negative status (Bhattacharyya distance, 0.25). This suggests a false-negative IHC finding and warrants antihormonal therapy for these patients.

CONCLUSIONS AND RELEVANCE

For at least half of the patients in this study, MBMP appeared to predict biomarker expression with noninferiority to IHC. Results suggest that prediction accuracy is likely to improve as data used for training expand. Morphological-based molecular profiling could be used as a general approach for mass-scale molecular profiling based on digitized H&E-stained images, allowing quick, accurate, and inexpensive methods for simultaneous profiling of multiple biomarkers in cancer tissues.

Software tools for automated transmission electron microscopy

The demand for high-throughput data collection in electron microscopy is increasing for applications in structural and cellular biology. Here we present a combination of software tools that enable automated acquisition guided by image analysis for a variety of transmission electron microscopy acquisition schemes. SerialEM controls microscopes and detectors and can trigger automated tasks at multiple positions with high flexibility. Py-EM interfaces with SerialEM to enact specimen-specific image-analysis pipelines that enable feedback microscopy. As example applications, we demonstrate dose reduction in cryo-electron microscopy experiments, fully automated acquisition of every cell in a plastic section and automated targeting on serial sections for 3D volume imaging across multiple grids.

BACH: Grand challenge on breast cancer histology images

Breast cancer is the most common invasive cancer in women, affecting more than 10% of women worldwide. Microscopic analysis of a biopsy remains one of the most important methods to diagnose the type of breast cancer. This requires specialized analysis by pathologists, in a task that i) is highly time- and cost-consuming and ii) often leads to nonconsensual results. The relevance and potential of automatic classification algorithms using hematoxylin-eosin stained histopathological images has already been demonstrated, but the reported results are still sub-optimal for clinical use. With the goal of advancing the state-of-the-art in automatic classification, the Grand Challenge on BreAst Cancer Histology images (BACH) was organized in conjunction with the 15th International Conference on Image Analysis and Recognition (ICIAR 2018). BACH aimed at the classification and localization of clinically relevant histopathological classes in microscopy and whole-slide images from a large annotated dataset, specifically compiled and made publicly available for the challenge. Following a positive response from the scientific community, a total of 64 submissions, out of 677 registrations, effectively entered the competition. The submitted algorithms improved the state-of-the-art in automatic classification of breast cancer with microscopy images to an accuracy of 87%. Convolutional neuronal networks were the most successful methodology in the BACH challenge. Detailed analysis of the collective results allowed the identification of remaining challenges in the field and recommendations for future developments. The BACH dataset remains publicly available as to promote further improvements to the field of automatic classification in digital pathology.

Digital pathology and artificial intelligence

In modern clinical practice, digital pathology has a crucial role and is increasingly a technological requirement in the scientific laboratory environment. The advent of whole-slide imaging, availability of faster networks, and cheaper storage solutions has made it easier for pathologists to manage digital slide images and share them for clinical use. In parallel, unprecedented advances in machine learning have enabled the synergy of artificial intelligence and digital pathology, which offers image-based diagnosis possibilities that were once limited only to radiology and cardiology. Integration of digital slides into the pathology workflow, advanced algorithms, and computer-aided diagnostic techniques extend the frontiers of the pathologist's view beyond a microscopic slide and enable true utilisation and integration of knowledge that is beyond human limits and boundaries, and we believe there is clear potential for artificial intelligence breakthroughs in the pathology setting. In this Review, we discuss advancements in digital slide-based image diagnosis for cancer along with some challenges and opportunities for artificial intelligence in digital pathology.

Deep Learning in Image Cytometry: A Review

Artificial intelligence, deep convolutional neural networks, and deep learning are all niche terms that are increasingly appearing in scientific presentations as well as in the general media. In this review, we focus on deep learning and how it is applied to microscopy image data of cells and tissue samples. Starting with an analogy to neuroscience, we aim to give the reader an overview of the key concepts of neural networks, and an understanding of how deep learning differs from more classical approaches for extracting information from image data. We aim to increase the understanding of these methods, while highlighting considerations regarding input data requirements, computational resources, challenges, and limitations. We do not provide a full manual for applying these methods to your own data, but rather review previously published articles on deep learning in image cytometry, and guide the readers toward further reading on specific networks and methods, including new methods not yet applied to cytometry data. © 2018 The Authors. Cytometry Part A published by Wiley Periodicals, Inc. on behalf of International Society for Advancement of Cytometry.

Deep Learning With Sampling in Colon Cancer Histology

This study applied a deep-learning cell identification algorithm to diagnostic images from the colon cancer repository at The Cancer Genome Atlas (TCGA). Within-image sampling improved performance without loss of accuracy. The features thus derived were associated with various clinical variables including metastasis, residual tumor, venous invasion, and lymphatic invasion. The deep-learning algorithm was trained using images from a locally available data set, then applied to the TCGA images by tiling them, and identifying cells in each patch defined by the tiling. In this application the average number of patches containing tissue in an image was ~900. Processing a random sample of patches greatly reduced computation costs. The cell identification algorithm was applied directly to each sampled patch, resulting in a list of cells. Each cell was labeled with its location and classification (“epithelial,” “inflammatory,” “fibroblast,” or “other”). The number of cells of a given type in the patch was calculated, resulting in a patch profile containing four features. A morphological profile that applied to the entire image was obtained by averaging profiles over all patches. Two sampling policies were examined. The first policy was random sampling which samples patches with uniform weighting. The second policy was systematic random sampling which takes spatial dependencies into account. Compared with the processing of complete whole slide images there was a seven-fold improvement in performance when systematic random spatial sampling was used to select 100 tiles from the whole-slide image for processing, with very little loss of accuracy (~4% on average). We found links between the predicted features and clinical variables in the TCGA colon cancer data set. Several significant associations were found: increased fibroblast numbers were associated with the presence of metastasis, venous invasion, lymphatic invasion and residual tumor while decreased numbers of inflammatory cells were associated with mucinous carcinomas. Regarding the four different types of cell, deep learning has generated morphological features that are indicators of cell density. The features are related to cellularity, the numbers, degree, or quality of cells present in a tumor. Cellularity has been reported to be related to patient survival and other diagnostic and prognostic indicators, indicating that the features calculated here may be of general usefulness.

Tissue and host species-specific transcriptional changes in models of experimental visceral leishmaniasis

Background: Human visceral leishmaniasis, caused by infection with Leishmania donovani or L. infantum, is a potentially fatal disease affecting 50,000-90,000 people yearly in 75 disease endemic countries, with more than 20,000 deaths reported. Experimental models of infection play a major role in understanding parasite biology, host-pathogen interaction, disease pathogenesis, and parasite transmission. In addition, they have an essential role in the identification and pre-clinical evaluation of new drugs and vaccines. However, our understanding of these models remains fragmentary. Although the immune response to Leishmania donovani infection in mice has been extensively characterized, transcriptomic analysis capturing the tissue-specific evolution of disease has yet to be reported. Methods: We provide an analysis of the transcriptome of spleen, liver and peripheral blood of BALB/c mice infected with L. donovani. Where possible, we compare our data in murine experimental visceral leishmaniasis with transcriptomic data in the public domain obtained from the study of L. donovani-infected hamsters and patients with human visceral leishmaniasis. Digitised whole slide images showing the histopathology in spleen and liver are made available via a dedicated website, www.leishpathnet.org. Results: Our analysis confirms marked tissue-specific alterations in the transcriptome of infected mice over time and identifies previously unrecognized parallels and differences between murine, hamster and human responses to infection. We show commonality of interferon-regulated genes whilst confirming a greater activation of type 2 immune pathways in infected hamsters compared to mice. Cytokine genes and genes encoding immune checkpoints were markedly tissue specific and dynamic in their expression, and pathways focused on non-immune cells reflected tissue specific immunopathology. Our data also addresses the value of measuring peripheral blood transcriptomics as a potential window into underlying systemic disease.  Conclusions: Our transcriptomic data, coupled with histopathologic analysis of the tissue response, provide an additional resource to underpin future mechanistic studies and to guide clinical research.

Tissue and host species-specific transcriptional changes in models of experimental visceral leishmaniasis

Background: Human visceral leishmaniasis, caused by infection with Leishmania donovani or L. infantum, is a potentially fatal disease affecting 50,000-90,000 people yearly in 75 disease endemic countries, with more than 20,000 deaths reported. Experimental models of infection play a major role in understanding parasite biology, host-pathogen interaction, disease pathogenesis, and parasite transmission. In addition, they have an essential role in the identification and pre-clinical evaluation of new drugs and vaccines. However, our understanding of these models remains fragmentary. Although the immune response to Leishmania donovani infection in mice has been extensively characterized, transcriptomic analysis capturing the tissue-specific evolution of disease has yet to be reported. Methods: We provide an analysis of the transcriptome of spleen, liver and peripheral blood of BALB/c mice infected with L. donovani. Where possible, we compare our data in murine experimental visceral leishmaniasis with transcriptomic data in the public domain obtained from the study of L. donovani-infected hamsters and patients with human visceral leishmaniasis. Digitised whole slide images showing the histopathology in spleen and liver are made available via a dedicated website, www.leishpathnet.org. Results: Our analysis confirms marked tissue-specific alterations in the transcriptome of infected mice over time and identifies previously unrecognized parallels and differences between murine, hamster and human responses to infection. We show commonality of interferon-regulated genes whilst confirming a greater activation of type 2 immune pathways in infected hamsters compared to mice. Cytokine genes and genes encoding immune checkpoints were markedly tissue specific and dynamic in their expression, and pathways focused on non-immune cells reflected tissue specific immunopathology. Our data also addresses the value of measuring peripheral blood transcriptomics as a potential window into underlying systemic disease.  Conclusions: Our transcriptomic data, coupled with histopathologic analysis of the tissue response, provide an additional resource to underpin future mechanistic studies and to guide clinical research.