101
|
Ayyad SM, Shehata M, Shalaby A, Abou El-Ghar M, Ghazal M, El-Melegy M, Abdel-Hamid NB, Labib LM, Ali HA, El-Baz A. Role of AI and Histopathological Images in Detecting Prostate Cancer: A Survey. SENSORS (BASEL, SWITZERLAND) 2021; 21:2586. [PMID: 33917035 PMCID: PMC8067693 DOI: 10.3390/s21082586] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/05/2021] [Revised: 03/29/2021] [Accepted: 04/04/2021] [Indexed: 02/07/2023]
Abstract
Prostate cancer is one of the most identified cancers and second most prevalent among cancer-related deaths of men worldwide. Early diagnosis and treatment are substantial to stop or handle the increase and spread of cancer cells in the body. Histopathological image diagnosis is a gold standard for detecting prostate cancer as it has different visual characteristics but interpreting those type of images needs a high level of expertise and takes too much time. One of the ways to accelerate such an analysis is by employing artificial intelligence (AI) through the use of computer-aided diagnosis (CAD) systems. The recent developments in artificial intelligence along with its sub-fields of conventional machine learning and deep learning provide new insights to clinicians and researchers, and an abundance of research is presented specifically for histopathology images tailored for prostate cancer. However, there is a lack of comprehensive surveys that focus on prostate cancer using histopathology images. In this paper, we provide a very comprehensive review of most, if not all, studies that handled the prostate cancer diagnosis using histopathological images. The survey begins with an overview of histopathological image preparation and its challenges. We also briefly review the computing techniques that are commonly applied in image processing, segmentation, feature selection, and classification that can help in detecting prostate malignancies in histopathological images.
Collapse
Affiliation(s)
- Sarah M. Ayyad
- Computers and Systems Department, Faculty of Engineering, Mansoura University, Mansoura 35511, Egypt; (S.M.A.); (N.B.A.-H.); (L.M.L.); (H.A.A.)
| | - Mohamed Shehata
- BioImaging Laboratory, Bioengineering Department, University of Louisville, Louisville, KY 40292, USA; (M.S.); (A.S.)
| | - Ahmed Shalaby
- BioImaging Laboratory, Bioengineering Department, University of Louisville, Louisville, KY 40292, USA; (M.S.); (A.S.)
| | - Mohamed Abou El-Ghar
- Department of Radiology, Urology and Nephrology Center, Mansoura University, Mansoura 35516, Egypt;
| | - Mohammed Ghazal
- Department of Electrical and Computer Engineering, College of Engineering, Abu Dhabi University, Abu Dhabi 59911, United Arab Emirates;
| | - Moumen El-Melegy
- Department of Electrical Engineering, Assiut University, Assiut 71511, Egypt;
| | - Nahla B. Abdel-Hamid
- Computers and Systems Department, Faculty of Engineering, Mansoura University, Mansoura 35511, Egypt; (S.M.A.); (N.B.A.-H.); (L.M.L.); (H.A.A.)
| | - Labib M. Labib
- Computers and Systems Department, Faculty of Engineering, Mansoura University, Mansoura 35511, Egypt; (S.M.A.); (N.B.A.-H.); (L.M.L.); (H.A.A.)
| | - H. Arafat Ali
- Computers and Systems Department, Faculty of Engineering, Mansoura University, Mansoura 35511, Egypt; (S.M.A.); (N.B.A.-H.); (L.M.L.); (H.A.A.)
| | - Ayman El-Baz
- BioImaging Laboratory, Bioengineering Department, University of Louisville, Louisville, KY 40292, USA; (M.S.); (A.S.)
| |
Collapse
|
102
|
Levy JJ, Azizgolshani N, Andersen MJ, Suriawinata A, Liu X, Lisovsky M, Ren B, Bobak CA, Christensen BC, Vaickus LJ. A large-scale internal validation study of unsupervised virtual trichrome staining technologies on nonalcoholic steatohepatitis liver biopsies. Mod Pathol 2021; 34:808-822. [PMID: 33299110 PMCID: PMC7985027 DOI: 10.1038/s41379-020-00718-1] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2020] [Revised: 11/06/2020] [Accepted: 11/08/2020] [Indexed: 02/07/2023]
Abstract
Non-alcoholic steatohepatitis (NASH) is a fatty liver disease characterized by accumulation of fat in hepatocytes with concurrent inflammation and is associated with morbidity, cirrhosis and liver failure. After extraction of a liver core biopsy, tissue sections are stained with hematoxylin and eosin (H&E) to grade NASH activity, and stained with trichrome to stage fibrosis. Methods to computationally transform one stain into another on digital whole slide images (WSI) can lessen the need for additional physical staining besides H&E, reducing personnel, equipment, and time costs. Generative adversarial networks (GAN) have shown promise for virtual staining of tissue. We conducted a large-scale validation study of the viability of GANs for H&E to trichrome conversion on WSI (n = 574). Pathologists were largely unable to distinguish real images from virtual/synthetic images given a set of twelve Turing Tests. We report high correlation between staging of real and virtual stains ([Formula: see text]; 95% CI: 0.84-0.88). Stages assigned to both virtual and real stains correlated similarly with a number of clinical biomarkers and progression to End Stage Liver Disease (Hazard Ratio HR = 2.06, 95% CI: 1.36-3.12, p < 0.001 for real stains; HR = 2.02, 95% CI: 1.40-2.92, p < 0.001 for virtual stains). Our results demonstrate that virtual trichrome technologies may offer a software solution that can be employed in the clinical setting as a diagnostic decision aid.
Collapse
Affiliation(s)
- Joshua J Levy
- Emerging Diagnostic and Investigative Technologies, Clinical Genomics and Advanced Technologies, Department of Pathology and Laboratory Medicine, Dartmouth Hitchcock Medical Center, Lebanon, NH, 03756, USA.
- Department of Epidemiology, Geisel School of Medicine at Dartmouth, Lebanon, NH, 03756, USA.
- Program in Quantitative Biomedical Sciences, Geisel School of Medicine at Dartmouth, Lebanon, NH, 03756, USA.
| | - Nasim Azizgolshani
- Department of Epidemiology, Geisel School of Medicine at Dartmouth, Lebanon, NH, 03756, USA
| | - Michael J Andersen
- Emerging Diagnostic and Investigative Technologies, Clinical Genomics and Advanced Technologies, Department of Pathology and Laboratory Medicine, Dartmouth Hitchcock Medical Center, Lebanon, NH, 03756, USA
| | - Arief Suriawinata
- Emerging Diagnostic and Investigative Technologies, Clinical Genomics and Advanced Technologies, Department of Pathology and Laboratory Medicine, Dartmouth Hitchcock Medical Center, Lebanon, NH, 03756, USA
| | - Xiaoying Liu
- Emerging Diagnostic and Investigative Technologies, Clinical Genomics and Advanced Technologies, Department of Pathology and Laboratory Medicine, Dartmouth Hitchcock Medical Center, Lebanon, NH, 03756, USA
| | - Mikhail Lisovsky
- Emerging Diagnostic and Investigative Technologies, Clinical Genomics and Advanced Technologies, Department of Pathology and Laboratory Medicine, Dartmouth Hitchcock Medical Center, Lebanon, NH, 03756, USA
| | - Bing Ren
- Emerging Diagnostic and Investigative Technologies, Clinical Genomics and Advanced Technologies, Department of Pathology and Laboratory Medicine, Dartmouth Hitchcock Medical Center, Lebanon, NH, 03756, USA
| | - Carly A Bobak
- Program in Quantitative Biomedical Sciences, Geisel School of Medicine at Dartmouth, Lebanon, NH, 03756, USA
- Thayer School of Engineering, Dartmouth College, Hanover, NH, 03755, USA
- Department of Biomedical Data Science, Geisel School of Medicine at Dartmouth, Lebanon, NH, 03756, USA
- The Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth, Lebanon, NH, 03756, USA
| | - Brock C Christensen
- Department of Epidemiology, Geisel School of Medicine at Dartmouth, Lebanon, NH, 03756, USA
- Department of Molecular and Systems Biology, Geisel School of Medicine at Dartmouth, Lebanon, NH, 03756, USA
- Department of Community and Family Medicine, Geisel School of Medicine at Dartmouth, Lebanon, NH, 03756, USA
| | - Louis J Vaickus
- Emerging Diagnostic and Investigative Technologies, Clinical Genomics and Advanced Technologies, Department of Pathology and Laboratory Medicine, Dartmouth Hitchcock Medical Center, Lebanon, NH, 03756, USA
| |
Collapse
|
103
|
Liu D, Zhang D, Song Y, Zhang F, O'Donnell L, Huang H, Chen M, Cai W. PDAM: A Panoptic-Level Feature Alignment Framework for Unsupervised Domain Adaptive Instance Segmentation in Microscopy Images. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:154-165. [PMID: 32915732 DOI: 10.1109/tmi.2020.3023466] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
In this work, we present an unsupervised domain adaptation (UDA) method, named Panoptic Domain Adaptive Mask R-CNN (PDAM), for unsupervised instance segmentation in microscopy images. Since there currently lack methods particularly for UDA instance segmentation, we first design a Domain Adaptive Mask R-CNN (DAM) as the baseline, with cross-domain feature alignment at the image and instance levels. In addition to the image- and instance-level domain discrepancy, there also exists domain bias at the semantic level in the contextual information. Next, we, therefore, design a semantic segmentation branch with a domain discriminator to bridge the domain gap at the contextual level. By integrating the semantic- and instance-level feature adaptation, our method aligns the cross-domain features at the panoptic level. Third, we propose a task re-weighting mechanism to assign trade-off weights for the detection and segmentation loss functions. The task re-weighting mechanism solves the domain bias issue by alleviating the task learning for some iterations when the features contain source-specific factors. Furthermore, we design a feature similarity maximization mechanism to facilitate instance-level feature adaptation from the perspective of representational learning. Different from the typical feature alignment methods, our feature similarity maximization mechanism separates the domain-invariant and domain-specific features by enlarging their feature distribution dependency. Experimental results on three UDA instance segmentation scenarios with five datasets demonstrate the effectiveness of our proposed PDAM method, which outperforms state-of-the-art UDA methods by a large margin.
Collapse
|
104
|
Papke DJ, Lohmann S, Downing M, Hufnagl P, Mutter GL. Computational augmentation of neoplastic endometrial glands in digital pathology displays. J Pathol 2020; 253:258-267. [PMID: 33165914 DOI: 10.1002/path.5586] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2020] [Revised: 10/14/2020] [Accepted: 11/02/2020] [Indexed: 11/09/2022]
Abstract
The pathologic diagnosis of neoplasia requires localization and classification of lesional tissue, a process that depends on the recognition of an abnormal spatial distribution of neoplastic elements relative to admixed normal background tissue. In endometrial intraepithelial neoplasia (EIN), a pre-cancer usually managed by hysterectomy, a clonally mutated proliferation of cytologically altered glands ('neoplastic-EIN') aggregates in clusters that also contain background non-neoplastic glands ('background-NL'). Here, we used image analysis to classify individual glands within endometrial tissue fragments as neoplastic-EIN or background-NL, and we used the distribution of predictions to localize foci diagnostic of EIN. Nuclear coordinates were automatically assigned and were used as vertices to generate Delaunay triangulations for each gland. Graph statistical variables were used to develop random forest algorithms to classify glands as neoplastic-EIN or background-NL. Individual glands in an independent validation set were scored by a 'ground truth' biomarker (PAX2 immunohistochemistry). We found that exclusion of small glands led to improvement in classification accuracy. Using an inclusion threshold of 200 nuclei per gland, our final model classification accuracy was 77.5% in the validation set, with a positive predictive value of 0.81. We leveraged this high positive predictive value in a point cloud overlay display to assist end-user identification of EIN foci. This study demonstrates that graph theory approaches applied to small-scale anatomic elements in the endometrium allow biologic classification by machine learning, and that spatial superimposition over large-scale tissue expanses can have practical diagnostic utility. We expect this augmented diagnostic approach to be generalizable to commonly encountered problems in other organ systems. © 2020 The Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.
Collapse
Affiliation(s)
- David J Papke
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA.,Harvard Medical School, Boston, MA, USA
| | - Sebastian Lohmann
- Institut für Pathologie, Charité - Universitätsmedizin, Berlin, Germany
| | - Michael Downing
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA
| | - Peter Hufnagl
- Institut für Pathologie, Charité - Universitätsmedizin, Berlin, Germany
| | - George L Mutter
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA.,Harvard Medical School, Boston, MA, USA
| |
Collapse
|
105
|
Safarpoor A, Kalra S, Tizhoosh HR. Generative models in pathology: synthesis of diagnostic quality pathology images †. J Pathol 2020; 253:131-132. [PMID: 33140849 DOI: 10.1002/path.5577] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2020] [Accepted: 10/27/2020] [Indexed: 11/10/2022]
Abstract
Within artificial intelligence and machine learning, a generative model is a powerful tool for learning any kind of data distribution. With the advent of deep learning and its success in image recognition, the field of deep generative models has clearly emerged as one of the promising fields for medical imaging. In a recent issue of The Journal of Pathology, Levine, Peng et al demonstrate the ability of generative models to synthesize high-quality pathology images. They suggested that generative models can serve as an unlimited source of images either for educating freshman pathologists or training machine learning models for diverse image analysis tasks, especially in scarce cases, while resolving patients' privacy and confidentiality concerns. © 2020 The Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.
Collapse
Affiliation(s)
| | - Shivam Kalra
- Kimia Lab, University of Waterloo, Waterloo, ON, Canada
| | | |
Collapse
|
106
|
Pandey P, P PA, Kyatham V, Mishra D, Dastidar TR. Target-Independent Domain Adaptation for WBC Classification Using Generative Latent Search. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:3979-3991. [PMID: 32746144 DOI: 10.1109/tmi.2020.3009029] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Automating the classification of camera-obtained microscopic images of White Blood Cells (WBCs) and related cell subtypes has assumed importance since it aids the laborious manual process of review and diagnosis. Several State-Of-The-Art (SOTA) methods developed using Deep Convolutional Neural Networks suffer from the problem of domain shift - severe performance degradation when they are tested on data (target) obtained in a setting different from that of the training (source). The change in the target data might be caused by factors such as differences in camera/microscope types, lenses, lighting-conditions etc. This problem can potentially be solved using Unsupervised Domain Adaptation (UDA) techniques albeit standard algorithms presuppose the existence of a sufficient amount of unlabelled target data which is not always the case with medical images. In this paper, we propose a method for UDA that is devoid of the need for target data. Given a test image from the target data, we obtain its 'closest-clone' from the source data that is used as a proxy in the classifier. We prove the existence of such a clone given that infinite number of data points can be sampled from the source distribution. We propose a method in which a latent-variable generative model based on variational inference is used to simultaneously sample and find the 'closest-clone' from the source distribution through an optimization procedure in the latent space. We demonstrate the efficacy of the proposed method over several SOTA UDA methods for WBC classification on datasets captured using different imaging modalities under multiple settings.
Collapse
|
107
|
Qu H, Wu P, Huang Q, Yi J, Yan Z, Li K, Riedlinger GM, De S, Zhang S, Metaxas DN. Weakly Supervised Deep Nuclei Segmentation Using Partial Points Annotation in Histopathology Images. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:3655-3666. [PMID: 32746112 DOI: 10.1109/tmi.2020.3002244] [Citation(s) in RCA: 49] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/06/2023]
Abstract
Nuclei segmentation is a fundamental task in histopathology image analysis. Typically, such segmentation tasks require significant effort to manually generate accurate pixel-wise annotations for fully supervised training. To alleviate such tedious and manual effort, in this paper we propose a novel weakly supervised segmentation framework based on partial points annotation, i.e., only a small portion of nuclei locations in each image are labeled. The framework consists of two learning stages. In the first stage, we design a semi-supervised strategy to learn a detection model from partially labeled nuclei locations. Specifically, an extended Gaussian mask is designed to train an initial model with partially labeled data. Then, self-training with background propagation is proposed to make use of the unlabeled regions to boost nuclei detection and suppress false positives. In the second stage, a segmentation model is trained from the detected nuclei locations in a weakly-supervised fashion. Two types of coarse labels with complementary information are derived from the detected points and are then utilized to train a deep neural network. The fully-connected conditional random field loss is utilized in training to further refine the model without introducing extra computational complexity during inference. The proposed method is extensively evaluated on two nuclei segmentation datasets. The experimental results demonstrate that our method can achieve competitive performance compared to the fully supervised counterpart and the state-of-the-art methods while requiring significantly less annotation effort.
Collapse
|
108
|
Tschuchnig ME, Oostingh GJ, Gadermayr M. Generative Adversarial Networks in Digital Pathology: A Survey on Trends and Future Potential. PATTERNS (NEW YORK, N.Y.) 2020; 1:100089. [PMID: 33205132 PMCID: PMC7660380 DOI: 10.1016/j.patter.2020.100089] [Citation(s) in RCA: 38] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Abstract
Image analysis in the field of digital pathology has recently gained increased popularity. The use of high-quality whole-slide scanners enables the fast acquisition of large amounts of image data, showing extensive context and microscopic detail at the same time. Simultaneously, novel machine-learning algorithms have boosted the performance of image analysis approaches. In this paper, we focus on a particularly powerful class of architectures, the so-called generative adversarial networks (GANs) applied to histological image data. Besides improving performance, GANs also enable previously intractable application scenarios in this field. However, GANs could exhibit a potential for introducing bias. Hereby, we summarize the recent state-of-the-art developments in a generalizing notation, present the main applications of GANs, and give an outlook of some chosen promising approaches and their possible future applications. In addition, we identify currently unavailable methods with potential for future applications.
Collapse
Affiliation(s)
- Maximilian E. Tschuchnig
- Department of Information Technologies and Systems Management, Salzburg University of Applied Sciences, 5412 Puch bei Hallein, Austria
- Department of Biomedical Sciences, Salzburg University of Applied Sciences, 5412 Puch bei Hallein, Austria
| | - Gertie J. Oostingh
- Department of Biomedical Sciences, Salzburg University of Applied Sciences, 5412 Puch bei Hallein, Austria
| | - Michael Gadermayr
- Department of Information Technologies and Systems Management, Salzburg University of Applied Sciences, 5412 Puch bei Hallein, Austria
| |
Collapse
|
109
|
Levine AB, Peng J, Farnell D, Nursey M, Wang Y, Naso JR, Ren H, Farahani H, Chen C, Chiu D, Talhouk A, Sheffield B, Riazy M, Ip PP, Parra-Herran C, Mills A, Singh N, Tessier-Cloutier B, Salisbury T, Lee J, Salcudean T, Jones SJ, Huntsman DG, Gilks CB, Yip S, Bashashati A. Synthesis of diagnostic quality cancer pathology images by generative adversarial networks. J Pathol 2020; 252:178-188. [PMID: 32686118 DOI: 10.1002/path.5509] [Citation(s) in RCA: 35] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2020] [Revised: 07/02/2020] [Accepted: 07/10/2020] [Indexed: 12/16/2022]
Abstract
Deep learning-based computer vision methods have recently made remarkable breakthroughs in the analysis and classification of cancer pathology images. However, there has been relatively little investigation of the utility of deep neural networks to synthesize medical images. In this study, we evaluated the efficacy of generative adversarial networks to synthesize high-resolution pathology images of 10 histological types of cancer, including five cancer types from The Cancer Genome Atlas and the five major histological subtypes of ovarian carcinoma. The quality of these images was assessed using a comprehensive survey of board-certified pathologists (n = 9) and pathology trainees (n = 6). Our results show that the real and synthetic images are classified by histotype with comparable accuracies and the synthetic images are visually indistinguishable from real images. Furthermore, we trained deep convolutional neural networks to diagnose the different cancer types and determined that the synthetic images perform as well as additional real images when used to supplement a small training set. These findings have important applications in proficiency testing of medical practitioners and quality assurance in clinical laboratories. Furthermore, training of computer-aided diagnostic systems can benefit from synthetic images where labeled datasets are limited (e.g. rare cancers). We have created a publicly available website where clinicians and researchers can attempt questions from the image survey (http://gan.aimlab.ca/). © 2020 Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.
Collapse
Affiliation(s)
- Adrian B Levine
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Jason Peng
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada.,School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
| | - David Farnell
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Mitchell Nursey
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada.,School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
| | - Yiping Wang
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada.,School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
| | - Julia R Naso
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Hezhen Ren
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Hossein Farahani
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada.,School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
| | - Colin Chen
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada.,School of Biomedical Engineering, University of British Columbia, Vancouver, Canada
| | - Derek Chiu
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Aline Talhouk
- Department of Obstetrics and Gynecology, University of British Columbia, Vancouver, Canada
| | - Brandon Sheffield
- Department of Pathology, William Osler Health Centre-Brampton Civic Hospital, Brampton, Canada
| | - Maziar Riazy
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Philip P Ip
- Department of Pathology, University of Hong Kong, Hong Kong SAR, PR China
| | - Carlos Parra-Herran
- Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Canada
| | - Anne Mills
- Department of Pathology, University of Virginia, Charlottesville, VA, USA
| | - Naveena Singh
- Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
| | - Basile Tessier-Cloutier
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Taylor Salisbury
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Jonathan Lee
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Tim Salcudean
- Electrical & Computer Engineering, University of British Columbia, Vancouver, Canada
| | - Steven Jm Jones
- Canada's Michael Smith Genome Sciences Centre, Vancouver, Canada
| | - David G Huntsman
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - C Blake Gilks
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Stephen Yip
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada
| | - Ali Bashashati
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, Canada.,School of Biomedical Engineering, University of British Columbia, Vancouver, Canada.,Electrical & Computer Engineering, University of British Columbia, Vancouver, Canada
| |
Collapse
|
110
|
Abstract
Artificial intelligence (AI) has contributed substantially to the resolution of a variety of biomedical problems, including cancer, over the past decade. Deep learning, a subfield of AI that is highly flexible and supports automatic feature extraction, is increasingly being applied in various areas of both basic and clinical cancer research. In this review, we describe numerous recent examples of the application of AI in oncology, including cases in which deep learning has efficiently solved problems that were previously thought to be unsolvable, and we address obstacles that must be overcome before such application can become more widespread. We also highlight resources and datasets that can help harness the power of AI for cancer research. The development of innovative approaches to and applications of AI will yield important insights in oncology in the coming decade.
Collapse
Affiliation(s)
- Hideyuki Shimizu
- Department of Molecular and Cellular BiologyMedical Institute of BioregulationKyushu UniversityFukuokaJapan
| | - Keiichi I. Nakayama
- Department of Molecular and Cellular BiologyMedical Institute of BioregulationKyushu UniversityFukuokaJapan
| |
Collapse
|
111
|
Kurc T, Bakas S, Ren X, Bagari A, Momeni A, Huang Y, Zhang L, Kumar A, Thibault M, Qi Q, Wang Q, Kori A, Gevaert O, Zhang Y, Shen D, Khened M, Ding X, Krishnamurthi G, Kalpathy-Cramer J, Davis J, Zhao T, Gupta R, Saltz J, Farahani K. Segmentation and Classification in Digital Pathology for Glioma Research: Challenges and Deep Learning Approaches. Front Neurosci 2020; 14:27. [PMID: 32153349 PMCID: PMC7046596 DOI: 10.3389/fnins.2020.00027] [Citation(s) in RCA: 33] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2019] [Accepted: 01/10/2020] [Indexed: 12/12/2022] Open
Abstract
Biomedical imaging Is an important source of information in cancer research. Characterizations of cancer morphology at onset, progression, and in response to treatment provide complementary information to that gleaned from genomics and clinical data. Accurate extraction and classification of both visual and latent image features Is an increasingly complex challenge due to the increased complexity and resolution of biomedical image data. In this paper, we present four deep learning-based image analysis methods from the Computational Precision Medicine (CPM) satellite event of the 21st International Medical Image Computing and Computer Assisted Intervention (MICCAI 2018) conference. One method Is a segmentation method designed to segment nuclei in whole slide tissue images (WSIs) of adult diffuse glioma cases. It achieved a Dice similarity coefficient of 0.868 with the CPM challenge datasets. Three methods are classification methods developed to categorize adult diffuse glioma cases into oligodendroglioma and astrocytoma classes using radiographic and histologic image data. These methods achieved accuracy values of 0.75, 0.80, and 0.90, measured as the ratio of the number of correct classifications to the number of total cases, with the challenge datasets. The evaluations of the four methods indicate that (1) carefully constructed deep learning algorithms are able to produce high accuracy in the analysis of biomedical image data and (2) the combination of radiographic with histologic image information improves classification performance.
Collapse
Affiliation(s)
- Tahsin Kurc
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, United States
| | - Spyridon Bakas
- Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, PA, United States
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
| | - Xuhua Ren
- Institute for Medical Imaging Technology, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Aditya Bagari
- Department of Engineering Design, Indian Institute of Technology Madras, Chennai, India
| | - Alexandre Momeni
- Department of Medicine and Biomedical Data Science, Stanford University, Stanford, CA, United States
| | - Yue Huang
- School of Informatics, Xiamen University, Xiamen, China
| | - Lichi Zhang
- Institute for Medical Imaging Technology, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Ashish Kumar
- Department of Engineering Design, Indian Institute of Technology Madras, Chennai, India
| | - Marc Thibault
- Department of Medicine and Biomedical Data Science, Stanford University, Stanford, CA, United States
| | - Qi Qi
- School of Informatics, Xiamen University, Xiamen, China
| | - Qian Wang
- Institute for Medical Imaging Technology, School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Avinash Kori
- Department of Engineering Design, Indian Institute of Technology Madras, Chennai, India
| | - Olivier Gevaert
- Department of Medicine and Biomedical Data Science, Stanford University, Stanford, CA, United States
| | - Yunlong Zhang
- School of Informatics, Xiamen University, Xiamen, China
| | - Dinggang Shen
- Department of Radiology and BRIC, The University of North Carolina at Chapel Hill, Chapel Hill, NC, United States
- Department of Brain and Cognitive Engineering, Korea University, Seoul, South Korea
| | - Mahendra Khened
- Department of Engineering Design, Indian Institute of Technology Madras, Chennai, India
| | - Xinghao Ding
- School of Informatics, Xiamen University, Xiamen, China
| | | | - Jayashree Kalpathy-Cramer
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, United States
| | - James Davis
- Department of Pathology, Stony Brook University, Stony Brook, NY, United States
| | - Tianhao Zhao
- Department of Pathology, Stony Brook University, Stony Brook, NY, United States
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, United States
- Department of Pathology, Stony Brook University, Stony Brook, NY, United States
| | - Joel Saltz
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, United States
| | - Keyvan Farahani
- Cancer Imaging Program, National Cancer Institute, National Institutes of Health, Bethesda, MD, United States
| |
Collapse
|
112
|
Enhancing Multi-tissue and Multi-scale Cell Nuclei Segmentation with Deep Metric Learning. APPLIED SCIENCES-BASEL 2020. [DOI: 10.3390/app10020615] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/29/2023]
Abstract
(1) Background: The segmentation of cell nuclei is an essential task in a wide range of biomedical studies and clinical practices. The full automation of this process remains a challenge due to intra- and internuclear variations across a wide range of tissue morphologies, differences in staining protocols and imaging procedures. (2) Methods: A deep learning model with metric embeddings such as contrastive loss and triplet loss with semi-hard negative mining is proposed in order to accurately segment cell nuclei in a diverse set of microscopy images. The effectiveness of the proposed model was tested on a large-scale multi-tissue collection of microscopy image sets. (3) Results: The use of deep metric learning increased the overall segmentation prediction by 3.12% in the average value of Dice similarity coefficients as compared to no metric learning. In particular, the largest gain was observed for segmenting cell nuclei in H&E -stained images when deep learning network and triplet loss with semi-hard negative mining were considered for the task. (4) Conclusion: We conclude that deep metric learning gives an additional boost to the overall learning process and consequently improves the segmentation performance. Notably, the improvement ranges approximately between 0.13% and 22.31% for different types of images in the terms of Dice coefficients when compared to no metric deep learning.
Collapse
|
113
|
Kim M, Yun J, Cho Y, Shin K, Jang R, Bae HJ, Kim N. Deep Learning in Medical Imaging. Neurospine 2019; 16:657-668. [PMID: 31905454 PMCID: PMC6945006 DOI: 10.14245/ns.1938396.198] [Citation(s) in RCA: 101] [Impact Index Per Article: 20.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2019] [Revised: 12/10/2019] [Accepted: 12/12/2019] [Indexed: 12/14/2022] Open
Abstract
The artificial neural network (ANN), one of the machine learning (ML) algorithms, inspired by the human brain system, was developed by connecting layers with artificial neurons. However, due to the low computing power and insufficient learnable data, ANN has suffered from overfitting and vanishing gradient problems for training deep networks. The advancement of computing power with graphics processing units and the availability of large data acquisition, deep neural network outperforms human or other ML capabilities in computer vision and speech recognition tasks. These potentials are recently applied to healthcare problems, including computer-aided detection/diagnosis, disease prediction, image segmentation, image generation, etc. In this review article, we will explain the history, development, and applications in medical imaging.
Collapse
Affiliation(s)
- Mingyu Kim
- Department of Convergence Medicine, Asan Medical Institute of Convergence Science and Technology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Jihye Yun
- Department of Convergence Medicine, Asan Medical Institute of Convergence Science and Technology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Yongwon Cho
- Department of Convergence Medicine, Asan Medical Institute of Convergence Science and Technology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Keewon Shin
- Department of Convergence Medicine, Asan Medical Institute of Convergence Science and Technology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Ryoungwoo Jang
- Department of Convergence Medicine, Asan Medical Institute of Convergence Science and Technology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Hyun-jin Bae
- Department of Convergence Medicine, Asan Medical Institute of Convergence Science and Technology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Namkug Kim
- Department of Convergence Medicine, Asan Medical Institute of Convergence Science and Technology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
- Department of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| |
Collapse
|
114
|
Ihle SJ, Reichmuth AM, Girardin S, Han H, Stauffer F, Bonnin A, Stampanoni M, Pattisapu K, Vörös J, Forró C. Unsupervised data to content transformation with histogram-matching cycle-consistent generative adversarial networks. NAT MACH INTELL 2019. [DOI: 10.1038/s42256-019-0096-2] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
115
|
Munir K, Elahi H, Ayub A, Frezza F, Rizzi A. Cancer Diagnosis Using Deep Learning: A Bibliographic Review. Cancers (Basel) 2019; 11:E1235. [PMID: 31450799 PMCID: PMC6770116 DOI: 10.3390/cancers11091235] [Citation(s) in RCA: 128] [Impact Index Per Article: 25.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2019] [Revised: 06/30/2019] [Accepted: 08/14/2019] [Indexed: 01/06/2023] Open
Abstract
In this paper, we first describe the basics of the field of cancer diagnosis, which includes steps of cancer diagnosis followed by the typical classification methods used by doctors, providing a historical idea of cancer classification techniques to the readers. These methods include Asymmetry, Border, Color and Diameter (ABCD) method, seven-point detection method, Menzies method, and pattern analysis. They are used regularly by doctors for cancer diagnosis, although they are not considered very efficient for obtaining better performance. Moreover, considering all types of audience, the basic evaluation criteria are also discussed. The criteria include the receiver operating characteristic curve (ROC curve), Area under the ROC curve (AUC), F1 score, accuracy, specificity, sensitivity, precision, dice-coefficient, average accuracy, and Jaccard index. Previously used methods are considered inefficient, asking for better and smarter methods for cancer diagnosis. Artificial intelligence and cancer diagnosis are gaining attention as a way to define better diagnostic tools. In particular, deep neural networks can be successfully used for intelligent image analysis. The basic framework of how this machine learning works on medical imaging is provided in this study, i.e., pre-processing, image segmentation and post-processing. The second part of this manuscript describes the different deep learning techniques, such as convolutional neural networks (CNNs), generative adversarial models (GANs), deep autoencoders (DANs), restricted Boltzmann's machine (RBM), stacked autoencoders (SAE), convolutional autoencoders (CAE), recurrent neural networks (RNNs), long short-term memory (LTSM), multi-scale convolutional neural network (M-CNN), multi-instance learning convolutional neural network (MIL-CNN). For each technique, we provide Python codes, to allow interested readers to experiment with the cited algorithms on their own diagnostic problems. The third part of this manuscript compiles the successfully applied deep learning models for different types of cancers. Considering the length of the manuscript, we restrict ourselves to the discussion of breast cancer, lung cancer, brain cancer, and skin cancer. The purpose of this bibliographic review is to provide researchers opting to work in implementing deep learning and artificial neural networks for cancer diagnosis a knowledge from scratch of the state-of-the-art achievements.
Collapse
Affiliation(s)
- Khushboo Munir
- Department of Information Engineering, Electronics and Telecommunications (DIET), Sapienza University of Rome, Via Eudossiana 18, 00184 Rome, Italy.
| | - Hassan Elahi
- Department of Mechanical and Aerospace Engineering (DIMA), Sapienza University of Rome, Via Eudossiana 18, 00184 Rome, Italy
| | - Afsheen Ayub
- Department of Basic and Applied Science for Engineering (SBAI), Sapienza University of Rome, Via Antonio Scarpa 14/16, 00161 Rome, Italy
| | - Fabrizio Frezza
- Department of Information Engineering, Electronics and Telecommunications (DIET), Sapienza University of Rome, Via Eudossiana 18, 00184 Rome, Italy
| | - Antonello Rizzi
- Department of Information Engineering, Electronics and Telecommunications (DIET), Sapienza University of Rome, Via Eudossiana 18, 00184 Rome, Italy
| |
Collapse
|
116
|
Hou L, Agarwal A, Samaras D, Kurc TM, Gupta RR, Saltz JH. Robust Histopathology Image Analysis: to Label or to Synthesize? PROCEEDINGS. IEEE COMPUTER SOCIETY CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION 2019; 2019:8533-8542. [PMID: 34025103 PMCID: PMC8139403 DOI: 10.1109/cvpr.2019.00873] [Citation(s) in RCA: 40] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Detection, segmentation and classification of nuclei are fundamental analysis operations in digital pathology. Existing state-of-the-art approaches demand extensive amount of supervised training data from pathologists and may still perform poorly in images from unseen tissue types. We propose an unsupervised approach for histopathology image segmentation that synthesizes heterogeneous sets of training image patches, of every tissue type. Although our synthetic patches are not always of high quality, we harness the motley crew of generated samples through a generally applicable importance sampling method. This proposed approach, for the first time, re-weighs the training loss over synthetic data so that the ideal (unbiased) generalization loss over the true data distribution is minimized. This enables us to use a random polygon generator to synthesize approximate cellular structures (i.e., nuclear masks) for which no real examples are given in many tissue types, and hence, GAN-based methods are not suited. In addition, we propose a hybrid synthesis pipeline that utilizes textures in real histopathology patches and GAN models, to tackle heterogeneity in tissue textures. Compared with existing state-of-the-art supervised models, our approach generalizes significantly better on cancer types without training data. Even in cancer types with training data, our approach achieves the same performance without supervision cost. We release code and segmentation results on over 5000 Whole Slide Images (WSI) in The Cancer Genome Atlas (TCGA) repository, a dataset that would be orders of magnitude larger than what is available today.
Collapse
Affiliation(s)
| | - Ayush Agarwal
- Stony Brook University
- Stanford University, California
| | | | | | | | | |
Collapse
|