51
|
High-Resolution Histopathological Image Classification Model Based on Fused Heterogeneous Networks with Self-Supervised Feature Representation. BIOMED RESEARCH INTERNATIONAL 2022; 2022:8007713. [PMID: 36046446 PMCID: PMC9420597 DOI: 10.1155/2022/8007713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/20/2022] [Accepted: 08/02/2022] [Indexed: 11/18/2022]
Abstract
Applying machine learning technology to automatic image analysis and auxiliary diagnosis of whole slide image (WSI) may help to improve the efficiency, objectivity, and consistency of pathological diagnosis. Due to its extremely high resolution, it is still a great challenge to directly process WSI through deep neural networks. In this paper, we propose a novel model for the task of classification of WSIs. The model is composed of two parts. The first part is a self-supervised encoding network with a UNet-like architecture. Each patch from a WSI is encoded as a compressed latent representation. These features are placed according to their corresponding patch’s original location in WSI, forming a feature cube. The second part is a classification network fused by 4 famous network blocks with heterogeneous architectures, with feature cube as input. Our model effectively expresses the feature and preserves location information of each patch. The fused network integrates heterogeneous features generated by different networks which yields robust classification results. The model is evaluated on two public datasets with comparison to baseline models. The evaluation results show the effectiveness of the proposed model.
Collapse
|
52
|
Da Q, Huang X, Li Z, Zuo Y, Zhang C, Liu J, Chen W, Li J, Xu D, Hu Z, Yi H, Guo Y, Wang Z, Chen L, Zhang L, He X, Zhang X, Mei K, Zhu C, Lu W, Shen L, Shi J, Li J, S S, Krishnamurthi G, Yang J, Lin T, Song Q, Liu X, Graham S, Bashir RMS, Yang C, Qin S, Tian X, Yin B, Zhao J, Metaxas DN, Li H, Wang C, Zhang S. DigestPath: A benchmark dataset with challenge review for the pathological detection and segmentation of digestive-system. Med Image Anal 2022; 80:102485. [DOI: 10.1016/j.media.2022.102485] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2021] [Revised: 04/08/2022] [Accepted: 05/20/2022] [Indexed: 12/19/2022]
|
53
|
MITNET: a novel dataset and a two-stage deep learning approach for mitosis recognition in whole slide images of breast cancer tissue. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07441-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
Abstract
AbstractMitosis assessment of breast cancer has a strong prognostic importance and is visually evaluated by pathologists. The inter, and intra-observer variability of this assessment is high. In this paper, a two-stage deep learning approach, named MITNET, has been applied to automatically detect nucleus and classify mitoses in whole slide images (WSI) of breast cancer. Moreover, this paper introduces two new datasets. The first dataset is used to detect the nucleus in the WSIs, which contains 139,124 annotated nuclei in 1749 patches extracted from 115 WSIs of breast cancer tissue, and the second dataset consists of 4908 mitotic cells and 4908 non-mitotic cells image samples extracted from 214 WSIs which is used for mitosis classification. The created datasets are used to train the MITNET network, which consists of two deep learning architectures, called MITNET-det and MITNET-rec, respectively, to isolate nuclei cells and identify the mitoses in WSIs. In MITNET-det architecture, to extract features from nucleus images and fuse them, CSPDarknet and Path Aggregation Network (PANet) are used, respectively, and then, a detection strategy using You Look Only Once (scaled-YOLOv4) is employed to detect nucleus at three different scales. In the classification part, the detected isolated nucleus images are passed through proposed MITNET-rec deep learning architecture, to identify the mitosis in the WSIs. Various deep learning classifiers and the proposed classifier are trained with a publicly available mitosis datasets (MIDOG and ATYPIA) and then, validated over our created dataset. The results verify that deep learning-based classifiers trained on MIDOG and ATYPIA have difficulties to recognize mitosis on our dataset which shows that the created mitosis dataset has unique features and characteristics. Besides this, the proposed classifier outperforms the state-of-the-art classifiers significantly and achieves a $$68.7\%$$
68.7
%
F1-score and $$49.0\%$$
49.0
%
F1-score on the MIDOG and the created mitosis datasets, respectively. Moreover, the experimental results reveal that the overall proposed MITNET framework detects the nucleus in WSIs with high detection rates and recognizes the mitotic cells in WSI with high F1-score which leads to the improvement of the accuracy of pathologists’ decision.
Collapse
|
54
|
Amgad M, Atteya LA, Hussein H, Mohammed KH, Hafiz E, Elsebaie MAT, Alhusseiny AM, AlMoslemany MA, Elmatboly AM, Pappalardo PA, Sakr RA, Mobadersany P, Rachid A, Saad AM, Alkashash AM, Ruhban IA, Alrefai A, Elgazar NM, Abdulkarim A, Farag AA, Etman A, Elsaeed AG, Alagha Y, Amer YA, Raslan AM, Nadim MK, Elsebaie MAT, Ayad A, Hanna LE, Gadallah A, Elkady M, Drumheller B, Jaye D, Manthey D, Gutman DA, Elfandy H, Cooper LAD. NuCLS: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer. Gigascience 2022; 11:6586817. [PMID: 35579553 PMCID: PMC9112766 DOI: 10.1093/gigascience/giac037] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2021] [Revised: 12/24/2021] [Accepted: 03/18/2022] [Indexed: 01/20/2023] Open
Abstract
Background Deep learning enables accurate high-resolution mapping of cells and tissue structures that can serve as the foundation of interpretable machine-learning models for computational pathology. However, generating adequate labels for these structures is a critical barrier, given the time and effort required from pathologists. Results This article describes a novel collaborative framework for engaging crowds of medical students and pathologists to produce quality labels for cell nuclei. We used this approach to produce the NuCLS dataset, containing >220,000 annotations of cell nuclei in breast cancers. This builds on prior work labeling tissue regions to produce an integrated tissue region- and cell-level annotation dataset for training that is the largest such resource for multi-scale analysis of breast cancer histology. This article presents data and analysis results for single and multi-rater annotations from both non-experts and pathologists. We present a novel workflow that uses algorithmic suggestions to collect accurate segmentation data without the need for laborious manual tracing of nuclei. Our results indicate that even noisy algorithmic suggestions do not adversely affect pathologist accuracy and can help non-experts improve annotation quality. We also present a new approach for inferring truth from multiple raters and show that non-experts can produce accurate annotations for visually distinctive classes. Conclusions This study is the most extensive systematic exploration of the large-scale use of wisdom-of-the-crowd approaches to generate data for computational pathology applications.
Collapse
Affiliation(s)
- Mohamed Amgad
- Department of Pathology, Northwestern University, 750 N Lake Shore Dr., Chicago, IL 60611, USA
| | - Lamees A Atteya
- Cairo Health Care Administration, Egyptian Ministry of Health, 3 Magles El Shaab Street, Cairo, Postal code 222, Egypt
| | - Hagar Hussein
- Department of Pathology, Nasser institute for research and treatment, 3 Magles El Shaab Street, Cairo, Postal code 222, Egypt
| | - Kareem Hosny Mohammed
- Department of Pathology and Laboratory Medicine, University of Pennsylvania, 3620 Hamilton Walk M163, Philadelphia, PA 19104, USA
| | - Ehab Hafiz
- Department of Clinical Laboratory Research, Theodor Bilharz Research Institute, 1 El-Nile Street, Imbaba Warrak El-Hadar, Giza, Postal code 12411, Egypt
| | - Maha A T Elsebaie
- Department of Medicine, Cook County Hospital, 1969 W Ogden Ave, Chicago, IL 60612, USA
| | - Ahmed M Alhusseiny
- Department of Pathology, Baystate Medical Center, University of Massachusetts, 759 Chestnut St, Springfield, MA 01199, USA
| | - Mohamed Atef AlMoslemany
- Faculty of Medicine, Menoufia University, Gamal Abd El-Nasir, Qism Shebeen El-Kom, Shibin el Kom, Menofia Governorate, Postal code: 32511, Egypt
| | - Abdelmagid M Elmatboly
- Faculty of Medicine, Al-Azhar University, 15 Mohammed Abdou, El-Darb El-Ahmar, Cairo Governorate, Postal code 11651, Egypt
| | - Philip A Pappalardo
- Consultant for The Center for Applied Proteomics and Molecular Medicine (CAPMM), George Mason University, 10920 George Mason Circle Institute for Advanced Biomedical Research Room 2008, MS1A9 Manassas, Virginia 20110, USA
| | - Rokia Adel Sakr
- Department of Pathology, National Liver Institute, Gamal Abd El-Nasir, Qism Shebeen El-Kom, Shibin el Kom, Menofia Governorate, Postal code: 32511, Egypt
| | - Pooya Mobadersany
- Department of Pathology, Northwestern University, 750 N Lake Shore Dr., Chicago, IL 60611, USA
| | - Ahmad Rachid
- Faculty of Medicine, Ain Shams University, 38 Abbassia, Next to the Al-Nour Mosque, Cairo, Postal code: 1181, Egypt
| | - Anas M Saad
- Cleveland Clinic Foundation, 9500 Euclid Ave. Cleveland, Ohio 44195, USA
| | - Ahmad M Alkashash
- Department of Pathology, Indiana University, 635 Barnhill Drive Medical Science Building A-128 Indianapolis, IN 46202, USA
| | - Inas A Ruhban
- Faculty of Medicine, Damascus University, Damascus, Damaskus, PO Box 30621, Syria
| | - Anas Alrefai
- Faculty of Medicine, Ain Shams University, 38 Abbassia, Next to the Al-Nour Mosque, Cairo, Postal code: 1181, Egypt
| | - Nada M Elgazar
- Faculty of Medicine, Mansoura University, 1 El Gomhouria St, Dakahlia Governorate 35516, Egypt
| | - Ali Abdulkarim
- Faculty of Medicine, Cairo University, Kasr Al Ainy Hospitals, Kasr Al Ainy St., Cairo, Postal code: 11562, Egypt
| | - Abo-Alela Farag
- Faculty of Medicine, Ain Shams University, 38 Abbassia, Next to the Al-Nour Mosque, Cairo, Postal code: 1181, Egypt
| | - Amira Etman
- Faculty of Medicine, Menoufia University, Gamal Abd El-Nasir, Qism Shebeen El-Kom, Shibin el Kom, Menofia Governorate, Postal code: 32511, Egypt
| | - Ahmed G Elsaeed
- Faculty of Medicine, Mansoura University, 1 El Gomhouria St, Dakahlia Governorate 35516, Egypt
| | - Yahya Alagha
- Faculty of Medicine, Cairo University, Kasr Al Ainy Hospitals, Kasr Al Ainy St., Cairo, Postal code: 11562, Egypt
| | - Yomna A Amer
- Faculty of Medicine, Menoufia University, Gamal Abd El-Nasir, Qism Shebeen El-Kom, Shibin el Kom, Menofia Governorate, Postal code: 32511, Egypt
| | - Ahmed M Raslan
- Department of Anaesthesia and Critical Care, Menoufia University Hospital, Gamal Abd El-Nasir, Qism Shebeen El-Kom, Shibin el Kom, Menofia Governorate, Postal code: 32511, Egypt
| | - Menatalla K Nadim
- Department of Clinical Pathology, Ain Shams University, 38 Abbassia, Next to the Al-Nour Mosque, Cairo, Postal code: 1181, Egypt
| | - Mai A T Elsebaie
- Faculty of Medicine, Ain Shams University, 38 Abbassia, Next to the Al-Nour Mosque, Cairo, Postal code: 1181, Egypt
| | - Ahmed Ayad
- Research Department, Oncology Consultants, 2130 W. Holcombe Blvd, 10th Floor, Houston, Texas 77030, USA
| | - Liza E Hanna
- Department of Pathology, Nasser institute for research and treatment, 3 Magles El Shaab Street, Cairo, Postal code 222, Egypt
| | - Ahmed Gadallah
- Faculty of Medicine, Ain Shams University, 38 Abbassia, Next to the Al-Nour Mosque, Cairo, Postal code: 1181, Egypt
| | - Mohamed Elkady
- Siparadigm Diagnostic Informatics, 25 Riverside Dr no. 2, Pine Brook, NJ 07058, USA
| | - Bradley Drumheller
- Department of Pathology and Laboratory Medicine, Emory University School of Medicine, 201 Dowman Dr, Atlanta, GA 30322, USA
| | - David Jaye
- Department of Pathology and Laboratory Medicine, Emory University School of Medicine, 201 Dowman Dr, Atlanta, GA 30322, USA
| | - David Manthey
- Kitware Inc., 1712 Route 9. Suite 300. Clifton Park, New York 12065, USA
| | - David A Gutman
- Department of Neurology, Emory University School of Medicine, 201 Dowman Dr, Atlanta, GA 30322, USA
| | - Habiba Elfandy
- Department of Pathology, National Cancer Institute, Kasr Al Eini Street, Fom El Khalig, Cairo, Postal code: 11562, Egypt.,Department of Pathology, Children's Cancer Hospital Egypt (CCHE 57357), 1 Seket Al-Emam Street, El-Madbah El-Kadeem Yard, El-Saida Zenab, Cairo, Postal code: 11562, Egypt
| | - Lee A D Cooper
- Department of Pathology, Northwestern University, 750 N Lake Shore Dr., Chicago, IL 60611, USA.,Lurie Cancer Center, Northwestern University, 675 N St Clair St Fl 21 Ste 100, Chicago, IL 60611, USA.,Center for Computational Imaging and Signal Analytics, Northwestern University Feinberg School of Medicine, 750 N Lake Shore Dr., Chicago, IL 60611, USA
| |
Collapse
|
55
|
NaroNet: Discovery of tumor microenvironment elements from highly multiplexed images. Med Image Anal 2022; 78:102384. [PMID: 35217454 PMCID: PMC9972483 DOI: 10.1016/j.media.2022.102384] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2021] [Revised: 10/26/2021] [Accepted: 02/01/2022] [Indexed: 12/11/2022]
Abstract
Understanding the spatial interactions between the elements of the tumor microenvironment -i.e. tumor cells. fibroblasts, immune cells- and how these interactions relate to the diagnosis or prognosis of a tumor is one of the goals of computational pathology. We present NaroNet, a deep learning framework that models the multi-scale tumor microenvironment from multiplex-stained cancer tissue images and provides patient-level interpretable predictions using a seamless end-to-end learning pipeline. Trained only with multiplex-stained tissue images and their corresponding patient-level clinical labels, NaroNet unsupervisedly learns which cell phenotypes, cell neighborhoods, and neighborhood interactions have the highest influence to predict the correct label. To this end, NaroNet incorporates several novel and state-of-the-art deep learning techniques, such as patch-level contrastive learning, multi-level graph embeddings, a novel max-sum pooling operation, or a metric that quantifies the relevance that each microenvironment element has in the individual predictions. We validate NaroNet using synthetic data simulating multiplex-immunostained images where a patient label is artificially associated to the -adjustable- probabilistic incidence of different microenvironment elements. We then apply our model to two sets of images of human cancer tissues: 336 seven-color multiplex-immunostained images from 12 high-grade endometrial cancer patients; and 382 35-plex mass cytometry images from 215 breast cancer patients. In both synthetic and real datasets, NaroNet provides outstanding predictions of relevant clinical information while associating those predictions to the presence of specific microenvironment elements.
Collapse
|
56
|
Abbet C, Studer L, Fischer A, Dawson H, Zlobec I, Bozorgtabar B, Thiran JP. Self-Rule to Multi-Adapt: Generalized Multi-source Feature Learning Using Unsupervised Domain Adaptation for Colorectal Cancer Tissue Detection. Med Image Anal 2022; 79:102473. [DOI: 10.1016/j.media.2022.102473] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2021] [Revised: 03/07/2022] [Accepted: 05/03/2022] [Indexed: 10/18/2022]
|
57
|
Hwang M, Wu C, Jiang WC, Hung WC. A sequential attention interface with a dense reward function for mitosis detection. INT J MACH LEARN CYB 2022. [DOI: 10.1007/s13042-022-01549-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
58
|
MiNuGAN: Dual Segmentation of Mitoses and Nuclei Using Conditional GANs on Multi-center Breast H&E Images. J Pathol Inform 2022; 13:100002. [PMID: 35242442 PMCID: PMC8860738 DOI: 10.1016/j.jpi.2022.100002] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Accepted: 11/30/2021] [Indexed: 11/21/2022] Open
Abstract
Breast cancer is the second most commonly diagnosed type of cancer among women as of 2021. Grading of histopathological images is used to guide breast cancer treatment decisions and a critical component of this is a mitotic score, which is related to tumor aggressiveness. Manual mitosis counting is an extremely tedious manual task, but automated approaches can be used to overcome inefficiency and subjectivity. In this paper, we propose an automatic mitosis and nuclear segmentation method for a diverse set of H&E breast cancer pathology images. The method is based on a conditional generative adversarial network to segment both mitoses and nuclei at the same time. Architecture optimizations are investigated, including hyper parameters and the addition of a focal loss. The accuracy of the proposed method is investigated using images from multiple centers and scanners, including TUPAC16, ICPR14 and ICPR12 datasets. In TUPAC16, we use 618 carefully annotated images of size 256×256 scanned at 40×. TUPAC16 is used to train the model, and segmentation performance is measured on the test set for both nuclei and mitoses. Results on 200 held-out testing images from the TUPAC16 dataset were mean DSC = 0.784 and 0.721 for nuclear and mitosis, respectively. On 202 ICPR12 images, mitosis segmentation accuracy had a mean DSC = 0.782, indicating the model generalizes well to unseen datasets. For datasets that had mitosis centroid annotations, which included 200 TUPAC16, 202 ICPR12 and 524 ICPR14, a mean F1-score of 0.854 was found indicating high mitosis detection accuracy.
Collapse
|
59
|
Pinckaers H, van Ginneken B, Litjens G. Streaming Convolutional Neural Networks for End-to-End Learning With Multi-Megapixel Images. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:1581-1590. [PMID: 32845835 DOI: 10.1109/tpami.2020.3019563] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Due to memory constraints on current hardware, most convolution neural networks (CNN) are trained on sub-megapixel images. For example, most popular datasets in computer vision contain images much less than a megapixel in size (0.09MP for ImageNet and 0.001MP for CIFAR-10). In some domains such as medical imaging, multi-megapixel images are needed to identify the presence of disease accurately. We propose a novel method to directly train convolutional neural networks using any input image size end-to-end. This method exploits the locality of most operations in modern convolutional neural networks by performing the forward and backward pass on smaller tiles of the image. In this work, we show a proof of concept using images of up to 66-megapixels (8192×8192), saving approximately 50GB of memory per image. Using two public challenge datasets, we demonstrate that CNNs can learn to extract relevant information from these large images and benefit from increasing resolution. We improved the area under the receiver-operating characteristic curve from 0.580 (4MP) to 0.706 (66MP) for metastasis detection in breast cancer (CAMELYON17). We also obtained a Spearman correlation metric approaching state-of-the-art performance on the TUPAC16 dataset, from 0.485 (1MP) to 0.570 (16MP). Code to reproduce a subset of the experiments is available at https://github.com/DIAGNijmegen/StreamingCNN.
Collapse
|
60
|
Ciga O, Xu T, Martel AL. Self supervised contrastive learning for digital histopathology. MACHINE LEARNING WITH APPLICATIONS 2022. [DOI: 10.1016/j.mlwa.2021.100198] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/25/2022] Open
|
61
|
Wang C, Gong W, Cheng J, Qian Y. DBLCNN: Dependency-based lightweight convolutional neural network for multi-classification of breast histopathology images. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103451] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
|
62
|
Sturm B, Creytens D, Smits J, Ooms AHAG, Eijken E, Kurpershoek E, Küsters-Vandevelde HVN, Wauters C, Blokx WAM, van der Laak JAWM. Computer-Aided Assessment of Melanocytic Lesions by Means of a Mitosis Algorithm. Diagnostics (Basel) 2022; 12:diagnostics12020436. [PMID: 35204526 PMCID: PMC8871065 DOI: 10.3390/diagnostics12020436] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2021] [Revised: 12/31/2021] [Accepted: 01/14/2022] [Indexed: 11/16/2022] Open
Abstract
An increasing number of pathology laboratories are now fully digitised, using whole slide imaging (WSI) for routine diagnostics. WSI paves the road to use artificial intelligence (AI) that will play an increasing role in computer-aided diagnosis (CAD). In melanocytic skin lesions, the presence of a dermal mitosis may be an important clue for an intermediate or a malignant lesion and may indicate worse prognosis. In this study a mitosis algorithm primarily developed for breast carcinoma is applied to melanocytic skin lesions. This study aimed to assess whether the algorithm could be used in diagnosing melanocytic lesions, and to study the added value in diagnosing melanocytic lesions in a practical setting. WSI’s of a set of hematoxylin and eosin (H&E) stained slides of 99 melanocytic lesions (35 nevi, 4 intermediate melanocytic lesions, and 60 malignant melanomas, including 10 nevoid melanomas), for which a consensus diagnosis was reached by three academic pathologists, were subjected to a mitosis algorithm based on AI. Two academic and six general pathologists specialized in dermatopathology examined the WSI cases two times, first without mitosis annotations and after a washout period of at least 2 months with mitosis annotations based on the algorithm. The algorithm indicated true mitosis in lesional cells, i.e., melanocytes, and non-lesional cells, i.e., mainly keratinocytes and inflammatory cells. A high number of false positive mitosis was indicated as well, comprising melanin pigment, sebaceous glands nuclei, and spindle cell nuclei such as stromal cells and neuroid differentiated melanocytes. All but one pathologist reported more often a dermal mitosis with the mitosis algorithm, which on a regular basis, was incorrectly attributed to mitoses from mainly inflammatory cells. The overall concordance of the pathologists with the consensus diagnosis for all cases excluding nevoid melanoma (n = 89) appeared to be comparable with and without the use of AI (89% vs. 90%). However, the concordance increased by using AI in nevoid melanoma cases (n = 10) (75% vs. 68%). This study showed that in general cases, pathologists perform similarly with the aid of a mitosis algorithm developed primarily for breast cancer. In nevoid melanoma cases, pathologists perform better with the algorithm. From this study, it can be learned that pathologists need to be aware of potential pitfalls using CAD on H&E slides, e.g., misinterpreting dermal mitoses in non-melanotic cells.
Collapse
Affiliation(s)
- Bart Sturm
- Department of Pathology, Radboud University Medical Center, 6500 HB Nijmegen, The Netherlands;
- Pathan B.V., 3045 PM Rotterdam, The Netherlands; (J.S.); (A.H.A.G.O.); (E.K.)
| | - David Creytens
- Department of Pathology, Ghent University Hospital, 9000 Ghent, Belgium;
| | - Jan Smits
- Pathan B.V., 3045 PM Rotterdam, The Netherlands; (J.S.); (A.H.A.G.O.); (E.K.)
| | | | - Erik Eijken
- Laboratory for Pathology Oost Nederland (LabPON), 7550 AM Hengelo, The Netherlands;
| | - Eline Kurpershoek
- Pathan B.V., 3045 PM Rotterdam, The Netherlands; (J.S.); (A.H.A.G.O.); (E.K.)
| | | | - Carla Wauters
- Department of Pathology, Canisius Wilhelmina Hospital, 6500 GS Nijmegen, The Netherlands; (H.V.N.K.-V.); (C.W.)
| | - Willeke A. M. Blokx
- Division Laboratories, Pharmacy and Biomedical Genetics, University Medical Center Utrecht, 3508 GA Utrecht, The Netherlands;
| | - Jeroen A. W. M. van der Laak
- Department of Pathology, Radboud University Medical Center, 6500 HB Nijmegen, The Netherlands;
- Center for Medical Image Science and Visualization, Linköping University, 581 83 Linköping, Sweden
- Correspondence: ; Tel.: +31-638-814-869
| |
Collapse
|
63
|
A comprehensive review of computer-aided whole-slide image analysis: from datasets to feature extraction, segmentation, classification and detection approaches. Artif Intell Rev 2022. [DOI: 10.1007/s10462-021-10121-0] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
|
64
|
McGenity C, Wright A, Treanor D. AIM in Surgical Pathology. Artif Intell Med 2022. [DOI: 10.1007/978-3-030-64573-1_278] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
65
|
Liang H, Cheng Z, Zhong H, Qu A, Chen L. A region-based convolutional network for nuclei detection and segmentation in microscopy images. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103276] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
66
|
Rehman MU, Akhtar S, Zakwan M, Mahmood MH. Novel architecture with selected feature vector for effective classification of mitotic and non-mitotic cells in breast cancer histology images. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103212] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
67
|
Bertram CA, Aubreville M, Donovan TA, Bartel A, Wilm F, Marzahl C, Assenmacher CA, Becker K, Bennett M, Corner S, Cossic B, Denk D, Dettwiler M, Gonzalez BG, Gurtner C, Haverkamp AK, Heier A, Lehmbecker A, Merz S, Noland EL, Plog S, Schmidt A, Sebastian F, Sledge DG, Smedley RC, Tecilla M, Thaiwong T, Fuchs-Baumgartinger A, Meuten DJ, Breininger K, Kiupel M, Maier A, Klopfleisch R. Computer-assisted mitotic count using a deep learning–based algorithm improves interobserver reproducibility and accuracy. Vet Pathol 2021; 59:211-226. [PMID: 34965805 PMCID: PMC8928234 DOI: 10.1177/03009858211067478] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
The mitotic count (MC) is an important histological parameter for prognostication of malignant neoplasms. However, it has inter- and intraobserver discrepancies due to difficulties in selecting the region of interest (MC-ROI) and in identifying or classifying mitotic figures (MFs). Recent progress in the field of artificial intelligence has allowed the development of high-performance algorithms that may improve standardization of the MC. As algorithmic predictions are not flawless, computer-assisted review by pathologists may ensure reliability. In the present study, we compared partial (MC-ROI preselection) and full (additional visualization of MF candidates and display of algorithmic confidence values) computer-assisted MC analysis to the routine (unaided) MC analysis by 23 pathologists for whole-slide images of 50 canine cutaneous mast cell tumors (ccMCTs). Algorithmic predictions aimed to assist pathologists in detecting mitotic hotspot locations, reducing omission of MFs, and improving classification against imposters. The interobserver consistency for the MC significantly increased with computer assistance (interobserver correlation coefficient, ICC = 0.92) compared to the unaided approach (ICC = 0.70). Classification into prognostic stratifications had a higher accuracy with computer assistance. The algorithmically preselected hotspot MC-ROIs had a consistently higher MCs than the manually selected MC-ROIs. Compared to a ground truth (developed with immunohistochemistry for phosphohistone H3), pathologist performance in detecting individual MF was augmented when using computer assistance (F1-score of 0.68 increased to 0.79) with a reduction in false negatives by 38%. The results of this study demonstrate that computer assistance may lead to more reproducible and accurate MCs in ccMCTs.
Collapse
Affiliation(s)
- Christof A. Bertram
- University of Veterinary Medicine, Vienna, Austria
- Freie Universität Berlin, Berlin, Germany
| | | | | | | | - Frauke Wilm
- Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Christian Marzahl
- Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | | | | | | | | | | | | | | | | | | | | | | | | | - Sophie Merz
- IDEXX Vet Med Labor GmbH, Kornwestheim, Germany
| | | | | | | | | | | | | | - Marco Tecilla
- Roche Pharmaceutical Research and Early Development (pRED), Basel, Switzerland
| | | | | | | | | | | | - Andreas Maier
- Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | | |
Collapse
|
68
|
A Whole-Slide Image Managing Library Based on Fastai for Deep Learning in the Context of Histopathology: Two Use-Cases Explained. APPLIED SCIENCES-BASEL 2021. [DOI: 10.3390/app12010013] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
Background: Processing whole-slide images (WSI) to train neural networks can be intricate and labor intensive. We developed an open-source library dealing with recurrent tasks in the processing of WSI and helping with the training and evaluation of neuronal networks for classification tasks. Methods: Two histopathology use-cases were selected and only hematoxylin and eosin (H&E) stained slides were used. The first use case was a two-class classification problem. We trained a convolutional neuronal network (CNN) to distinguish between dysembryoplastic neuroepithelial tumor (DNET) and ganglioglioma (GG), two neuropathological low-grade epilepsy-associated tumor entities. Within the second use case, we included four clinicopathological disease conditions in a multilabel approach. Here we trained a CNN to predict the hormone expression profile of pituitary adenomas. In the same approach, we also predicted clinically silent corticotroph adenoma. Results: Our DNET-GG classifier achieved an AUC of 1.00 for the ROC curve. For the second use case, the best performing CNN achieved an area under the curve (AUC) of 0.97 for the receiver operating characteristic (ROC) for corticotroph adenoma, 0.86 for silent corticotroph adenoma, and 0.98 for gonadotroph adenoma. All scores were calculated with the help of our library on predictions on a case basis. Conclusions: Our comprehensive and fastai-compatible library is helpful to standardize the workflow and minimize the burden of training a CNN. Indeed, our trained CNNs extracted neuropathologically relevant information from the WSI. This approach will supplement the clinicopathological diagnosis of brain tumors, which is currently based on cost-intensive microscopic examination and variable panels of immunohistochemical stainings.
Collapse
|
69
|
Zafar MM, Rauf Z, Sohail A, Khan AR, Obaidullah M, Khan SH, Lee YS, Khan A. Detection of tumour infiltrating lymphocytes in CD3 and CD8 stained histopathological images using a two-phase deep CNN. Photodiagnosis Photodyn Ther 2021; 37:102676. [PMID: 34890783 DOI: 10.1016/j.pdpdt.2021.102676] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2021] [Revised: 11/10/2021] [Accepted: 12/06/2021] [Indexed: 12/26/2022]
Abstract
BACKGROUND Immuno-score, a prognostic measure for cancer, employed in determining tumor grade and type, is generated by counting the number of Tumour-Infiltrating Lymphocytes (TILs) in CD3 and CD8 stained histopathological tissue samples. Significant stain variations and heterogeneity in lymphocytes' spatial distribution and density make automated counting of TILs' a challenging task. METHODS This work addresses the aforementioned challenges by developing a pipeline "Two-Phase Deep Convolutional Neural Network based Lymphocyte Counter (TDC-LC)" to detect lymphocytes in CD3 and CD8 stained histology images. The proposed pipeline sequentially works by removing hard negative examples (artifacts) in the first phase using a custom CNN "LSATM-Net" that exploits the idea of a split, asymmetric transform, and merge. Whereas, in the second phase, instance segmentation is performed to detect and generate a lymphocyte count against the remaining samples. Furthermore, the effectiveness of the proposed pipeline is measured by comparing it with the state-of-the-art single- and two-stage detectors. The inference code is available at GitHub Repository https://github.com/m-mohsin-zafar/tdc-lc. RESULTS The empirical evaluation on samples from LYSTO dataset shows that the proposed LSTAM-Net can learn variations in the images and precisely remove the hard negative stain artifacts with an F-score of 0.74. The detection analysis shows that the proposed TDC-LC outperforms the existing models in identifying and counting lymphocytes with high Recall (0.87) and F-score (0.89). Moreover, the commendable performance of the proposed TDC-LC in different organs suggests a good generalization. CONCLUSION The promising performance of the proposed pipeline suggests that it can serve as an automated system for detecting and counting lymphocytes from patches of tissue samples thereby reducing the burden on pathologists.
Collapse
Affiliation(s)
- Muhammad Mohsin Zafar
- Pattern Recognition Lab, Department of Computer & Information Sciences, Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan; Faculty of Computer Science and Engineering, Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi 23640, District Swabi, Khyber Pakhtunkhwa, Pakistan
| | - Zunaira Rauf
- Pattern Recognition Lab, Department of Computer & Information Sciences, Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan; PIEAS Artificial Intelligence Center (PAIC), Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan
| | - Anabia Sohail
- Pattern Recognition Lab, Department of Computer & Information Sciences, Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan; PIEAS Artificial Intelligence Center (PAIC), Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan
| | - Abdul Rehman Khan
- Pattern Recognition Lab, Department of Computer & Information Sciences, Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan; Center for Mathematical Sciences, Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan
| | - Muhammad Obaidullah
- Pattern Recognition Lab, Department of Computer & Information Sciences, Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan; Center for Mathematical Sciences, Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan
| | - Saddam Hussain Khan
- Pattern Recognition Lab, Department of Computer & Information Sciences, Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan; PIEAS Artificial Intelligence Center (PAIC), Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan
| | - Yeon Soo Lee
- Deparment of Biomedical Engineering, College of Medical Sciences, Catholic University of Daegu, South Korea.
| | - Asifullah Khan
- Pattern Recognition Lab, Department of Computer & Information Sciences, Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan; PIEAS Artificial Intelligence Center (PAIC), Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan; Deparment of Biomedical Engineering, College of Medical Sciences, Catholic University of Daegu, South Korea; Center for Mathematical Sciences, Pakistan Institute of Engineering & Applied Sciences, Nilore, Islamabad 45650, Pakistan.
| |
Collapse
|
70
|
Mridha MF, Hamid MA, Monowar MM, Keya AJ, Ohi AQ, Islam MR, Kim JM. A Comprehensive Survey on Deep-Learning-Based Breast Cancer Diagnosis. Cancers (Basel) 2021; 13:6116. [PMID: 34885225 PMCID: PMC8656730 DOI: 10.3390/cancers13236116] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2021] [Revised: 11/25/2021] [Accepted: 12/01/2021] [Indexed: 12/11/2022] Open
Abstract
Breast cancer is now the most frequently diagnosed cancer in women, and its percentage is gradually increasing. Optimistically, there is a good chance of recovery from breast cancer if identified and treated at an early stage. Therefore, several researchers have established deep-learning-based automated methods for their efficiency and accuracy in predicting the growth of cancer cells utilizing medical imaging modalities. As of yet, few review studies on breast cancer diagnosis are available that summarize some existing studies. However, these studies were unable to address emerging architectures and modalities in breast cancer diagnosis. This review focuses on the evolving architectures of deep learning for breast cancer detection. In what follows, this survey presents existing deep-learning-based architectures, analyzes the strengths and limitations of the existing studies, examines the used datasets, and reviews image pre-processing techniques. Furthermore, a concrete review of diverse imaging modalities, performance metrics and results, challenges, and research directions for future researchers is presented.
Collapse
Affiliation(s)
- Muhammad Firoz Mridha
- Department of Computer Science and Engineering, Bangladesh University of Business and Technology, Dhaka 1216, Bangladesh; (M.F.M.); (A.J.K.); (A.Q.O.)
| | - Md. Abdul Hamid
- Department of Information Technology, Faculty of Computing & Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia; (M.A.H.); (M.M.M.)
| | - Muhammad Mostafa Monowar
- Department of Information Technology, Faculty of Computing & Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia; (M.A.H.); (M.M.M.)
| | - Ashfia Jannat Keya
- Department of Computer Science and Engineering, Bangladesh University of Business and Technology, Dhaka 1216, Bangladesh; (M.F.M.); (A.J.K.); (A.Q.O.)
| | - Abu Quwsar Ohi
- Department of Computer Science and Engineering, Bangladesh University of Business and Technology, Dhaka 1216, Bangladesh; (M.F.M.); (A.J.K.); (A.Q.O.)
| | - Md. Rashedul Islam
- Department of Computer Science and Engineering, University of Asia Pacific, Dhaka 1216, Bangladesh;
| | - Jong-Myon Kim
- Department of Electrical, Electronics, and Computer Engineering, University of Ulsan, Ulsan 680-749, Korea
| |
Collapse
|
71
|
Rashmi R, Prasad K, Udupa CBK. Breast histopathological image analysis using image processing techniques for diagnostic puposes: A methodological review. J Med Syst 2021; 46:7. [PMID: 34860316 PMCID: PMC8642363 DOI: 10.1007/s10916-021-01786-9] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2021] [Accepted: 10/21/2021] [Indexed: 12/24/2022]
Abstract
Breast cancer in women is the second most common cancer worldwide. Early detection of breast cancer can reduce the risk of human life. Non-invasive techniques such as mammograms and ultrasound imaging are popularly used to detect the tumour. However, histopathological analysis is necessary to determine the malignancy of the tumour as it analyses the image at the cellular level. Manual analysis of these slides is time consuming, tedious, subjective and are susceptible to human errors. Also, at times the interpretation of these images are inconsistent between laboratories. Hence, a Computer-Aided Diagnostic system that can act as a decision support system is need of the hour. Moreover, recent developments in computational power and memory capacity led to the application of computer tools and medical image processing techniques to process and analyze breast cancer histopathological images. This review paper summarizes various traditional and deep learning based methods developed to analyze breast cancer histopathological images. Initially, the characteristics of breast cancer histopathological images are discussed. A detailed discussion on the various potential regions of interest is presented which is crucial for the development of Computer-Aided Diagnostic systems. We summarize the recent trends and choices made during the selection of medical image processing techniques. Finally, a detailed discussion on the various challenges involved in the analysis of BCHI is presented along with the future scope.
Collapse
Affiliation(s)
- R Rashmi
- Manipal School of Information Sciences, Manipal Academy of Higher Education, Manipal, India
| | - Keerthana Prasad
- Manipal School of Information Sciences, Manipal Academy of Higher Education, Manipal, India
| | | |
Collapse
|
72
|
Liew XY, Hameed N, Clos J. An investigation of XGBoost-based algorithm for breast cancer classification. MACHINE LEARNING WITH APPLICATIONS 2021. [DOI: 10.1016/j.mlwa.2021.100154] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022] Open
|
73
|
|
74
|
Zheng Y, Jiang Z, Shi J, Xie F, Zhang H, Luo W, Hu D, Sun S, Jiang Z, Xue C. Encoding histopathology whole slide images with location-aware graphs for diagnostically relevant regions retrieval. Med Image Anal 2021; 76:102308. [PMID: 34856455 DOI: 10.1016/j.media.2021.102308] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2019] [Revised: 10/14/2021] [Accepted: 11/17/2021] [Indexed: 01/18/2023]
Abstract
Content-based histopathological image retrieval (CBHIR) has become popular in recent years in histopathological image analysis. CBHIR systems provide auxiliary diagnosis information for pathologists by searching for and returning regions that are contently similar to the region of interest (ROI) from a pre-established database. It is challenging and yet significant in clinical applications to retrieve diagnostically relevant regions from a database consisting of histopathological whole slide images (WSIs). In this paper, we propose a novel framework for regions retrieval from WSI database based on location-aware graphs and deep hash techniques. Compared to the present CBHIR framework, both structural information and global location information of ROIs in the WSI are preserved by graph convolution and self-attention operations, which makes the retrieval framework more sensitive to regions that are similar in tissue distribution. Moreover, benefited from the graph structure, the proposed framework has good scalability for both the size and shape variation of ROIs. It allows the pathologist to define query regions using free curves according to the appearance of tissue. Thirdly, the retrieval is achieved based on the hash technique, which ensures the framework is efficient and adequate for practical large-scale WSI database. The proposed method was evaluated on an in-house endometrium dataset with 2650 WSIs and the public ACDC-LungHP dataset. The experimental results have demonstrated that the proposed method achieved a mean average precision above 0.667 on the endometrium dataset and above 0.869 on the ACDC-LungHP dataset in the task of irregular region retrieval, which are superior to the state-of-the-art methods. The average retrieval time from a database containing 1855 WSIs is 0.752 ms. The source code is available at https://github.com/zhengyushan/lagenet.
Collapse
Affiliation(s)
- Yushan Zheng
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China
| | - Zhiguo Jiang
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China.
| | - Jun Shi
- School of Software, Hefei University of Technology, Hefei 230601, China.
| | - Fengying Xie
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Haopeng Zhang
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Wei Luo
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Dingyi Hu
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Shujiao Sun
- Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Image Processing Center, School of Astronautics, Beihang University, Beijing 102206, China
| | - Zhongmin Jiang
- Department of Pathology, Tianjin Fifth Central Hospital, Tianjin 300450, China
| | - Chenghai Xue
- Wankangyuan Tianjin Gene Technology, Inc, Tianjin 300220, China; Tianjin Institute of Industrial Biotechnology, Chinese Academy of Sciences, Tianjin 300308, China
| |
Collapse
|
75
|
Yousif M, van Diest PJ, Laurinavicius A, Rimm D, van der Laak J, Madabhushi A, Schnitt S, Pantanowitz L. Artificial intelligence applied to breast pathology. Virchows Arch 2021; 480:191-209. [PMID: 34791536 DOI: 10.1007/s00428-021-03213-3] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2021] [Revised: 09/12/2021] [Accepted: 09/27/2021] [Indexed: 12/12/2022]
Abstract
The convergence of digital pathology and computer vision is increasingly enabling computers to perform tasks performed by humans. As a result, artificial intelligence (AI) is having an astoundingly positive effect on the field of pathology, including breast pathology. Research using machine learning and the development of algorithms that learn patterns from labeled digital data based on "deep learning" neural networks and feature-engineered approaches to analyze histology images have recently provided promising results. Thus far, image analysis and more complex AI-based tools have demonstrated excellent success performing tasks such as the quantification of breast biomarkers and Ki67, mitosis detection, lymph node metastasis recognition, tissue segmentation for diagnosing breast carcinoma, prognostication, computational assessment of tumor-infiltrating lymphocytes, and prediction of molecular expression as well as treatment response and benefit of therapy from routine H&E images. This review critically examines the literature regarding these applications of AI in the area of breast pathology.
Collapse
Affiliation(s)
- Mustafa Yousif
- Department of Pathology, University of Michigan, Ann Arbor, MI, USA.
- Department of Pathology, Vanderbilt University Medical Center, Nashville, TN, USA.
| | - Paul J van Diest
- Department of Pathology, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Arvydas Laurinavicius
- Department of Pathology, Pharmacology and Forensic Medicine, Faculty of Medicine, Vilnius University, and National Center of Pathology, Affiliate of Vilnius University Hospital Santaros Klinikos, Vilnius, Lithuania
| | - David Rimm
- Department of Pathology, Yale University School of Medicine, New Haven, CT, USA
| | - Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Anant Madabhushi
- Department of Biomedical Engineering, Case Western Reserve University, and Louis Stokes Cleveland Veterans Administration Medical Center, Cleveland, OH, USA
- Louis Stokes Cleveland Veterans Administration Medical Center, Cleveland, OH, USA
| | - Stuart Schnitt
- Department of Pathology, Brigham and Women's Hospital and Harvard Medical School, Breast Oncology Program, Dana-Farber/Brigham and Women's Cancer Center, Boston, MA, USA
| | | |
Collapse
|
76
|
Shavlokhova V, Sandhu S, Flechtenmacher C, Koveshazi I, Neumeier F, Padrón-Laso V, Jonke Ž, Saravi B, Vollmer M, Vollmer A, Hoffmann J, Engel M, Ristow O, Freudlsperger C. Deep Learning on Oral Squamous Cell Carcinoma Ex Vivo Fluorescent Confocal Microscopy Data: A Feasibility Study. J Clin Med 2021; 10:5326. [PMID: 34830608 PMCID: PMC8618824 DOI: 10.3390/jcm10225326] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2021] [Revised: 11/11/2021] [Accepted: 11/13/2021] [Indexed: 12/12/2022] Open
Abstract
BACKGROUND Ex vivo fluorescent confocal microscopy (FCM) is a novel and effective method for a fast-automatized histological tissue examination. In contrast, conventional diagnostic methods are primarily based on the skills of the histopathologist. In this study, we investigated the potential of convolutional neural networks (CNNs) for automatized classification of oral squamous cell carcinoma via ex vivo FCM imaging for the first time. MATERIAL AND METHODS Tissue samples from 20 patients were collected, scanned with an ex vivo confocal microscope immediately after resection, and investigated histopathologically. A CNN architecture (MobileNet) was trained and tested for accuracy. RESULTS The model achieved a sensitivity of 0.47 and specificity of 0.96 in the automated classification of cancerous tissue in our study. CONCLUSION In this preliminary work, we trained a CNN model on a limited number of ex vivo FCM images and obtained promising results in the automated classification of cancerous tissue. Further studies using large sample sizes are warranted to introduce this technology into clinics.
Collapse
Affiliation(s)
- Veronika Shavlokhova
- Department of Oral and Maxillofacial Surgery, University Hospital Heidelberg, 69120 Heidelberg, Germany; (S.S.); (M.V.); (A.V.); (J.H.); (M.E.); (O.R.); (C.F.)
| | - Sameena Sandhu
- Department of Oral and Maxillofacial Surgery, University Hospital Heidelberg, 69120 Heidelberg, Germany; (S.S.); (M.V.); (A.V.); (J.H.); (M.E.); (O.R.); (C.F.)
| | | | | | | | | | - Žan Jonke
- Munich Innovation Labs GmbH, 80336 Munich, Germany; (V.P.-L.); (Ž.J.)
| | - Babak Saravi
- Department of Orthopedics and Trauma Surgery, Medical Centre-Albert-Ludwigs-University of Freiburg, Faculty of Medicine, Albert-Ludwigs-University of Freiburg, 79106 Freiburg, Germany;
| | - Michael Vollmer
- Department of Oral and Maxillofacial Surgery, University Hospital Heidelberg, 69120 Heidelberg, Germany; (S.S.); (M.V.); (A.V.); (J.H.); (M.E.); (O.R.); (C.F.)
| | - Andreas Vollmer
- Department of Oral and Maxillofacial Surgery, University Hospital Heidelberg, 69120 Heidelberg, Germany; (S.S.); (M.V.); (A.V.); (J.H.); (M.E.); (O.R.); (C.F.)
| | - Jürgen Hoffmann
- Department of Oral and Maxillofacial Surgery, University Hospital Heidelberg, 69120 Heidelberg, Germany; (S.S.); (M.V.); (A.V.); (J.H.); (M.E.); (O.R.); (C.F.)
| | - Michael Engel
- Department of Oral and Maxillofacial Surgery, University Hospital Heidelberg, 69120 Heidelberg, Germany; (S.S.); (M.V.); (A.V.); (J.H.); (M.E.); (O.R.); (C.F.)
| | - Oliver Ristow
- Department of Oral and Maxillofacial Surgery, University Hospital Heidelberg, 69120 Heidelberg, Germany; (S.S.); (M.V.); (A.V.); (J.H.); (M.E.); (O.R.); (C.F.)
| | - Christian Freudlsperger
- Department of Oral and Maxillofacial Surgery, University Hospital Heidelberg, 69120 Heidelberg, Germany; (S.S.); (M.V.); (A.V.); (J.H.); (M.E.); (O.R.); (C.F.)
| |
Collapse
|
77
|
Deep Learning in Cancer Diagnosis and Prognosis Prediction: A Minireview on Challenges, Recent Trends, and Future Directions. COMPUTATIONAL AND MATHEMATICAL METHODS IN MEDICINE 2021; 2021:9025470. [PMID: 34754327 PMCID: PMC8572604 DOI: 10.1155/2021/9025470] [Citation(s) in RCA: 34] [Impact Index Per Article: 11.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/20/2021] [Revised: 09/30/2021] [Accepted: 10/05/2021] [Indexed: 12/30/2022]
Abstract
Deep learning (DL) is a branch of machine learning and artificial intelligence that has been applied to many areas in different domains such as health care and drug design. Cancer prognosis estimates the ultimate fate of a cancer subject and provides survival estimation of the subjects. An accurate and timely diagnostic and prognostic decision will greatly benefit cancer subjects. DL has emerged as a technology of choice due to the availability of high computational resources. The main components in a standard computer-aided design (CAD) system are preprocessing, feature recognition, extraction and selection, categorization, and performance assessment. Reduction of costs associated with sequencing systems offers a myriad of opportunities for building precise models for cancer diagnosis and prognosis prediction. In this survey, we provided a summary of current works where DL has helped to determine the best models for the cancer diagnosis and prognosis prediction tasks. DL is a generic model requiring minimal data manipulations and achieves better results while working with enormous volumes of data. Aims are to scrutinize the influence of DL systems using histopathology images, present a summary of state-of-the-art DL methods, and give directions to future researchers to refine the existing methods.
Collapse
|
78
|
Rahman A, Rahman M, Kundu D, Karim MR, Band SS, Sookhak M. Study on IoT for SARS-CoV-2 with healthcare: present and future perspective. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2021; 18:9697-9726. [PMID: 34814364 DOI: 10.3934/mbe.2021475] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
The ever-evolving and contagious nature of the Coronavirus (COVID-19) has immobilized the world around us. As the daily number of infected cases increases, the containment of the spread of this virus is proving to be an overwhelming task. Healthcare facilities around the world are overburdened with an ominous responsibility to combat an ever-worsening scenario. To aid the healthcare system, Internet of Things (IoT) technology provides a better solution-tracing, testing of COVID patients efficiently is gaining rapid pace. This study discusses the role of IoT technology in healthcare during the SARS-CoV-2 pandemics. The study overviews different research, platforms, services, products where IoT is used to combat the COVID-19 pandemic. Further, we intelligently integrate IoT and healthcare for COVID-19 related applications. Again, we focus on a wide range of IoT applications in regards to SARS-CoV-2 tracing, testing, and treatment. Finally, we effectively consider further challenges, issues, and some direction regarding IoT in order to uplift the healthcare system during COVID-19 and future pandemics.
Collapse
Affiliation(s)
- Anichur Rahman
- Department of Computer Science and Engineering, National Institute of Textile Engineering and Research (NITER), Constituent Institute of Dhaka University, Savar, Dhaka-1350, Bangladesh
- Department of Computer Science and Engineering, Mawlana Bhashani Science and Technology University, Tangail, Bangladesh
| | - Muaz Rahman
- Department of Electrical and Electronic Engineering, National Institute of Textile Engineering and Research (NITER), Constituent Institute of Dhaka University, Savar, Dhaka-1350, Bangladesh
| | - Dipanjali Kundu
- Department of Computer Science and Engineering, National Institute of Textile Engineering and Research (NITER), Constituent Institute of Dhaka University, Savar, Dhaka-1350, Bangladesh
| | - Md Razaul Karim
- Department of Computer Science and Engineering, Mawlana Bhashani Science and Technology University, Tangail, Bangladesh
| | - Shahab S Band
- Future Technology Research Center, College of Future, National Yunlin University of Science and Technology, 123 University Road, Section 3, Douliou, Yunlin 64002, Taiwan
| | - Mehdi Sookhak
- Dept. of Computer Science, Texas A & M University-Corpus Christi, 6300 Ocean Drive, Corpus Christi, Texas, USA, 78412
| |
Collapse
|
79
|
Towards computationally efficient prediction of molecular signatures from routine histology images. LANCET DIGITAL HEALTH 2021; 3:e752-e753. [PMID: 34686475 DOI: 10.1016/s2589-7500(21)00232-6] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Accepted: 10/01/2021] [Indexed: 02/07/2023]
|
80
|
A Two-Phase Mitosis Detection Approach Based on U-Shaped Network. BIOMED RESEARCH INTERNATIONAL 2021; 2021:1722652. [PMID: 34651044 PMCID: PMC8510810 DOI: 10.1155/2021/1722652] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/29/2021] [Accepted: 09/11/2021] [Indexed: 11/17/2022]
Abstract
This paper proposes a deep learning-based method for mitosis detection in breast histopathology images. A main problem in mitosis detection is that most of the datasets only have weak labels, i.e., only the coordinates indicating the center of the mitosis region. This makes most of the existing powerful object detection methods hardly be used in mitosis detection. Aiming at solving this problem, this paper firstly applies a CNN-based algorithm to pixelwisely segment the mitosis regions, based on which bounding boxes of mitosis are generated as strong labels. Based on the generated bounding boxes, an object detection network is trained to accomplish mitosis detection. Experimental results show that the proposed method is effective in detecting mitosis, and the accuracies outperform state-of-the-art literatures.
Collapse
|
81
|
A Pyramid Architecture-Based Deep Learning Framework for Breast Cancer Detection. BIOMED RESEARCH INTERNATIONAL 2021; 2021:2567202. [PMID: 34631877 PMCID: PMC8500767 DOI: 10.1155/2021/2567202] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/30/2021] [Accepted: 08/20/2021] [Indexed: 11/24/2022]
Abstract
Breast cancer diagnosis is a critical step in clinical decision making, and this is achieved by making a pathological slide and gives a decision by the doctors, which is the method of final decision making for cancer diagnosis. Traditionally, the doctors usually check the pathological images by visual inspection under the microscope. Whole-slide images (WSIs) have supported the state-of-the-art diagnosis results and have been admitted as the gold standard clinically. However, this task is time-consuming and labour-intensive, and all of these limitations make low efficiency in decision making. Medical image processing protocols have been used for this task during the last decades and have obtained satisfactory results under some conditions; especially in the deep learning era, it has exhibited the advantages than those in the shallow learning period. In this paper, we proposed a novel breast cancer region mining framework based on deep pyramid architecture from multilevel and multiscale breast pathological WSIs. We incorporate the tissue- and cell-level information together and integrate these into a LSTM model for the final sequence modelling, which successfully keeps the WSIs' integration and is not mentioned by the prevalence frameworks. The experiment results demonstrated that our proposed framework greatly improved the detection accuracy than that only using tissue-level information.
Collapse
|
82
|
van der Wal D, Jhun I, Laklouk I, Nirschl J, Richer L, Rojansky R, Theparee T, Wheeler J, Sander J, Feng F, Mohamad O, Savarese S, Socher R, Esteva A. Biological data annotation via a human-augmenting AI-based labeling system. NPJ Digit Med 2021; 4:145. [PMID: 34620993 PMCID: PMC8497580 DOI: 10.1038/s41746-021-00520-6] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2021] [Accepted: 09/13/2021] [Indexed: 12/22/2022] Open
Abstract
Biology has become a prime area for the deployment of deep learning and artificial intelligence (AI), enabled largely by the massive data sets that the field can generate. Key to most AI tasks is the availability of a sufficiently large, labeled data set with which to train AI models. In the context of microscopy, it is easy to generate image data sets containing millions of cells and structures. However, it is challenging to obtain large-scale high-quality annotations for AI models. Here, we present HALS (Human-Augmenting Labeling System), a human-in-the-loop data labeling AI, which begins uninitialized and learns annotations from a human, in real-time. Using a multi-part AI composed of three deep learning models, HALS learns from just a few examples and immediately decreases the workload of the annotator, while increasing the quality of their annotations. Using a highly repetitive use-case-annotating cell types-and running experiments with seven pathologists-experts at the microscopic analysis of biological specimens-we demonstrate a manual work reduction of 90.60%, and an average data-quality boost of 4.34%, measured across four use-cases and two tissue stain types.
Collapse
Affiliation(s)
| | - Iny Jhun
- Stanford University, 450 Serra Mall, Stanford, CA, 94305, USA
| | - Israa Laklouk
- University of California, San Francisco, 505 Parnassus Ave, San Francisco, CA, 94143, USA
| | - Jeff Nirschl
- Stanford University, 450 Serra Mall, Stanford, CA, 94305, USA
| | - Lara Richer
- University of California, San Francisco, 505 Parnassus Ave, San Francisco, CA, 94143, USA
| | | | - Talent Theparee
- University of California, San Francisco, 505 Parnassus Ave, San Francisco, CA, 94143, USA
| | - Joshua Wheeler
- Stanford University, 450 Serra Mall, Stanford, CA, 94305, USA
| | - Jörg Sander
- Department of Biomedical Engineering and Physics, Amsterdam University Medical Centers, Meibergdreef 9, 1105 AZ, Amsterdam, Netherlands
| | - Felix Feng
- University of California, San Francisco, 505 Parnassus Ave, San Francisco, CA, 94143, USA
| | - Osama Mohamad
- University of California, San Francisco, 505 Parnassus Ave, San Francisco, CA, 94143, USA
| | - Silvio Savarese
- Salesforce AI Research, 575 High St, Palo Alto, CA, 94301, USA
| | - Richard Socher
- Salesforce AI Research, 575 High St, Palo Alto, CA, 94301, USA
| | - Andre Esteva
- Salesforce AI Research, 575 High St, Palo Alto, CA, 94301, USA.
| |
Collapse
|
83
|
Wang Y, Kartasalo K, Weitz P, Ács B, Valkonen M, Larsson C, Ruusuvuori P, Hartman J, Rantalainen M. Predicting Molecular Phenotypes from Histopathology Images: A Transcriptome-Wide Expression-Morphology Analysis in Breast Cancer. Cancer Res 2021; 81:5115-5126. [PMID: 34341074 PMCID: PMC9397635 DOI: 10.1158/0008-5472.can-21-0482] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2021] [Revised: 04/30/2021] [Accepted: 07/28/2021] [Indexed: 01/07/2023]
Abstract
Molecular profiling is central in cancer precision medicine but remains costly and is based on tumor average profiles. Morphologic patterns observable in histopathology sections from tumors are determined by the underlying molecular phenotype and therefore have the potential to be exploited for prediction of molecular phenotypes. We report here the first transcriptome-wide expression-morphology (EMO) analysis in breast cancer, where individual deep convolutional neural networks were optimized and validated for prediction of mRNA expression in 17,695 genes from hematoxylin and eosin-stained whole slide images. Predicted expressions in 9,334 (52.75%) genes were significantly associated with RNA sequencing estimates. We also demonstrated successful prediction of an mRNA-based proliferation score with established clinical value. The results were validated in independent internal and external test datasets. Predicted spatial intratumor variabilities in expression were validated through spatial transcriptomics profiling. These results suggest that EMO provides a cost-efficient and scalable approach to predict both tumor average and intratumor spatial expression from histopathology images. SIGNIFICANCE: Transcriptome-wide expression morphology deep learning analysis enables prediction of mRNA expression and proliferation markers from routine histopathology whole slide images in breast cancer.
Collapse
Affiliation(s)
- Yinxi Wang
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | - Kimmo Kartasalo
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden.,Faculty of Medicine and Health Technology, Tampere University, Tampere, Finland
| | - Philippe Weitz
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | - Balázs Ács
- Department of Oncology-Pathology, Karolinska Institutet, Stockholm, Sweden.,Department of Clinical Pathology and Cytology, Karolinska University Laboratory, Stockholm, Sweden
| | - Masi Valkonen
- Institute of Biomedicine, Cancer Research Unit and FICAN West Cancer Centre, University of Turku and Turku University Hospital, Turku, Finland
| | - Christer Larsson
- Division of Translational Cancer Research, Department of Laboratory Medicine, Lund University, Lund, Sweden
| | - Pekka Ruusuvuori
- Faculty of Medicine and Health Technology, Tampere University, Tampere, Finland.,Institute of Biomedicine, Cancer Research Unit and FICAN West Cancer Centre, University of Turku and Turku University Hospital, Turku, Finland
| | - Johan Hartman
- Department of Oncology-Pathology, Karolinska Institutet, Stockholm, Sweden.,Department of Clinical Pathology and Cytology, Karolinska University Laboratory, Stockholm, Sweden.,MedTechLabs, BioClinicum, Karolinska University Hospital, Solna, Sweden
| | - Mattias Rantalainen
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden.,MedTechLabs, BioClinicum, Karolinska University Hospital, Solna, Sweden.,Corresponding Author: Mattias Rantalainen, Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, PO Box 281, SE-171 77 Stockholm, Sweden. Phone: 46-0-8-5248-0000, ext. 2465; E-mail:
| |
Collapse
|
84
|
van Dooijeweert C, van Diest PJ, Ellis IO. Grading of invasive breast carcinoma: the way forward. Virchows Arch 2021; 480:33-43. [PMID: 34196797 PMCID: PMC8983621 DOI: 10.1007/s00428-021-03141-2] [Citation(s) in RCA: 26] [Impact Index Per Article: 8.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2021] [Revised: 06/03/2021] [Accepted: 06/10/2021] [Indexed: 12/12/2022]
Abstract
Histologic grading has been a simple and inexpensive method to assess tumor behavior and prognosis of invasive breast cancer grading, thereby identifying patients at risk for adverse outcomes, who may be eligible for (neo)adjuvant therapies. Histologic grading needs to be performed accurately, on properly fixed specimens, and by adequately trained dedicated pathologists that take the time to diligently follow the protocol methodology. In this paper, we review the history of histologic grading, describe the basics of grading, review prognostic value and reproducibility issues, compare performance of grading to gene expression profiles, and discuss how to move forward to improve reproducibility of grading by training, feedback and artificial intelligence algorithms, and special stains to better recognize mitoses. We conclude that histologic grading, when adequately carried out, remains to be of important prognostic value in breast cancer patients.
Collapse
Affiliation(s)
- C van Dooijeweert
- Department of Pathology, University Medical Center Utrecht, Utrecht, Netherlands.,Department of Internal Medicine, Meander Medical Center, Amersfoort, Netherlands
| | - P J van Diest
- Department of Pathology, University Medical Center Utrecht, Utrecht, Netherlands.
| | - I O Ellis
- Department of Histopathology, Nottingham University Hospitals, Nottingham, UK
| |
Collapse
|
85
|
Liew XY, Hameed N, Clos J. A Review of Computer-Aided Expert Systems for Breast Cancer Diagnosis. Cancers (Basel) 2021; 13:2764. [PMID: 34199444 PMCID: PMC8199592 DOI: 10.3390/cancers13112764] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Revised: 05/25/2021] [Accepted: 05/28/2021] [Indexed: 11/18/2022] Open
Abstract
A computer-aided diagnosis (CAD) expert system is a powerful tool to efficiently assist a pathologist in achieving an early diagnosis of breast cancer. This process identifies the presence of cancer in breast tissue samples and the distinct type of cancer stages. In a standard CAD system, the main process involves image pre-processing, segmentation, feature extraction, feature selection, classification, and performance evaluation. In this review paper, we reviewed the existing state-of-the-art machine learning approaches applied at each stage involving conventional methods and deep learning methods, the comparisons within methods, and we provide technical details with advantages and disadvantages. The aims are to investigate the impact of CAD systems using histopathology images, investigate deep learning methods that outperform conventional methods, and provide a summary for future researchers to analyse and improve the existing techniques used. Lastly, we will discuss the research gaps of existing machine learning approaches for implementation and propose future direction guidelines for upcoming researchers.
Collapse
Affiliation(s)
- Xin Yu Liew
- Jubilee Campus, University of Nottingham, Wollaton Road, Nottingham NG8 1BB, UK; (N.H.); (J.C.)
| | | | | |
Collapse
|
86
|
van der Laak J, Litjens G, Ciompi F. Deep learning in histopathology: the path to the clinic. Nat Med 2021; 27:775-784. [PMID: 33990804 DOI: 10.1038/s41591-021-01343-4] [Citation(s) in RCA: 284] [Impact Index Per Article: 94.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2020] [Accepted: 03/31/2021] [Indexed: 02/08/2023]
Abstract
Machine learning techniques have great potential to improve medical diagnostics, offering ways to improve accuracy, reproducibility and speed, and to ease workloads for clinicians. In the field of histopathology, deep learning algorithms have been developed that perform similarly to trained pathologists for tasks such as tumor detection and grading. However, despite these promising results, very few algorithms have reached clinical implementation, challenging the balance between hope and hype for these new techniques. This Review provides an overview of the current state of the field, as well as describing the challenges that still need to be addressed before artificial intelligence in histopathology can achieve clinical value.
Collapse
Affiliation(s)
- Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands. .,Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden.
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Francesco Ciompi
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| |
Collapse
|
87
|
Wetstein SC, Stathonikos N, Pluim JPW, Heng YJ, Ter Hoeve ND, Vreuls CPH, van Diest PJ, Veta M. Deep learning-based grading of ductal carcinoma in situ in breast histopathology images. J Transl Med 2021; 101:525-533. [PMID: 33608619 PMCID: PMC7985025 DOI: 10.1038/s41374-021-00540-6] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2020] [Revised: 01/05/2021] [Accepted: 01/07/2021] [Indexed: 11/08/2022] Open
Abstract
Ductal carcinoma in situ (DCIS) is a non-invasive breast cancer that can progress into invasive ductal carcinoma (IDC). Studies suggest DCIS is often overtreated since a considerable part of DCIS lesions may never progress into IDC. Lower grade lesions have a lower progression speed and risk, possibly allowing treatment de-escalation. However, studies show significant inter-observer variation in DCIS grading. Automated image analysis may provide an objective solution to address high subjectivity of DCIS grading by pathologists. In this study, we developed and evaluated a deep learning-based DCIS grading system. The system was developed using the consensus DCIS grade of three expert observers on a dataset of 1186 DCIS lesions from 59 patients. The inter-observer agreement, measured by quadratic weighted Cohen's kappa, was used to evaluate the system and compare its performance to that of expert observers. We present an analysis of the lesion-level and patient-level inter-observer agreement on an independent test set of 1001 lesions from 50 patients. The deep learning system (dl) achieved on average slightly higher inter-observer agreement to the three observers (o1, o2 and o3) (κo1,dl = 0.81, κo2,dl = 0.53 and κo3,dl = 0.40) than the observers amongst each other (κo1,o2 = 0.58, κo1,o3 = 0.50 and κo2,o3 = 0.42) at the lesion-level. At the patient-level, the deep learning system achieved similar agreement to the observers (κo1,dl = 0.77, κo2,dl = 0.75 and κo3,dl = 0.70) as the observers amongst each other (κo1,o2 = 0.77, κo1,o3 = 0.75 and κo2,o3 = 0.72). The deep learning system better reflected the grading spectrum of DCIS than two of the observers. In conclusion, we developed a deep learning-based DCIS grading system that achieved a performance similar to expert observers. To the best of our knowledge, this is the first automated system for the grading of DCIS that could assist pathologists by providing robust and reproducible second opinions on DCIS grade.
Collapse
Affiliation(s)
- Suzanne C Wetstein
- Medical Image Analysis Group, Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands.
| | - Nikolas Stathonikos
- Department of Pathology, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands
| | - Josien P W Pluim
- Medical Image Analysis Group, Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Yujing J Heng
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Natalie D Ter Hoeve
- Department of Pathology, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands
| | - Celien P H Vreuls
- Department of Pathology, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands
| | - Paul J van Diest
- Department of Pathology, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands
| | - Mitko Veta
- Medical Image Analysis Group, Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
| |
Collapse
|
88
|
Sohail A, Khan A, Wahab N, Zameer A, Khan S. A multi-phase deep CNN based mitosis detection framework for breast cancer histopathological images. Sci Rep 2021; 11:6215. [PMID: 33737632 PMCID: PMC7973714 DOI: 10.1038/s41598-021-85652-1] [Citation(s) in RCA: 34] [Impact Index Per Article: 11.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2020] [Accepted: 02/16/2021] [Indexed: 12/24/2022] Open
Abstract
The mitotic activity index is a key prognostic measure in tumour grading. Microscopy based detection of mitotic nuclei is a significant overhead and necessitates automation. This work proposes deep CNN based multi-phase mitosis detection framework "MP-MitDet" for mitotic nuclei identification in breast cancer histopathological images. The workflow constitutes: (1) label-refiner, (2) tissue-level mitotic region selection, (3) blob analysis, and (4) cell-level refinement. We developed an automatic label-refiner to represent weak labels with semi-sematic information for training of deep CNNs. A deep instance-based detection and segmentation model is used to explore probable mitotic regions on tissue patches. More probable regions are screened based on blob area and then analysed at cell-level by developing a custom CNN classifier "MitosRes-CNN" to filter false mitoses. The performance of the proposed "MitosRes-CNN" is compared with the state-of-the-art CNNs that are adapted to cell-level discrimination through cross-domain transfer learning and by adding task-specific layers. The performance of the proposed framework shows good discrimination ability in terms of F-score (0.75), recall (0.76), precision (0.71) and area under the precision-recall curve (0.78) on challenging TUPAC16 dataset. Promising results suggest good generalization of the proposed framework that can learn characteristic features from heterogenous mitotic nuclei.
Collapse
Affiliation(s)
- Anabia Sohail
- Pattern Recognition Lab, DCIS, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Nilore, Islamabad, 45650, Pakistan
| | - Asifullah Khan
- Pattern Recognition Lab, DCIS, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Nilore, Islamabad, 45650, Pakistan.
- Deep Learning Lab, Centre for Mathematical Sciences, Pakistan Institute of Engineering and Applied Sciences, (PIEAS), Nilore, Islamabad, 45650, Pakistan.
| | - Noorul Wahab
- Department of Computer Science, Tissue Image Analytics (TIA) Lab, University of Warwick, Coventry, UK
| | - Aneela Zameer
- Pattern Recognition Lab, DCIS, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Nilore, Islamabad, 45650, Pakistan
| | - Saranjam Khan
- Department of Physics, Islamia College Peshawar, Peshawar, Pakistan
| |
Collapse
|
89
|
Nateghi R, Danyali H, Helfroush MS. A deep learning approach for mitosis detection: Application in tumor proliferation prediction from whole slide images. Artif Intell Med 2021; 114:102048. [PMID: 33875159 DOI: 10.1016/j.artmed.2021.102048] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2020] [Revised: 02/25/2021] [Accepted: 02/28/2021] [Indexed: 02/07/2023]
Abstract
The tumor proliferation, which is correlated with tumor grade, is a crucial biomarker indicative of breast cancer patients' prognosis. The most commonly used method in predicting tumor proliferation speed is the counting of mitotic figures in Hematoxylin and Eosin (H&E) histological slides. Manual mitosis counting is known to suffer from reproducibility problems. This paper presents a fully automated system for tumor proliferation prediction from whole slide images via mitosis counting. First, by considering the epithelial tissue as mitosis activity regions, we build a deep-learning-based region of interest detection method to select the high mitosis activity regions from whole slide images. Second, we learned a set of deep neural networks to detect mitosis detection from selected areas. The proposed mitosis detection system is designed to effectively overcome the mitosis detection challenges by two novel deep preprocessing and two-step hard negative mining approaches. Third, we trained a Support Vector Machine (SVM) classifier to predict the final tumor proliferation score. The proposed method was evaluated on the dataset of the Tumor Proliferation Assessment Challenge (TUPAC16) and achieved a 73.81 % F-measure and 0.612 weighted kappa score, respectively, outperforming all previous approaches significantly. Experimental results demonstrate that the proposed system considerably improves the tumor proliferation prediction accuracy and provides a reliable automated tool to support health care make-decisions.
Collapse
Affiliation(s)
- Ramin Nateghi
- Department of Electrical and Electronics Engineering, Shiraz University of Technology, Shiraz, Iran.
| | - Habibollah Danyali
- Department of Electrical and Electronics Engineering, Shiraz University of Technology, Shiraz, Iran.
| | - Mohammad Sadegh Helfroush
- Department of Electrical and Electronics Engineering, Shiraz University of Technology, Shiraz, Iran.
| |
Collapse
|
90
|
Hoefling H, Sing T, Hossain I, Boisclair J, Doelemeyer A, Flandre T, Piaia A, Romanet V, Santarossa G, Saravanan C, Sutter E, Turner O, Wuersch K, Moulin P. HistoNet: A Deep Learning-Based Model of Normal Histology. Toxicol Pathol 2021; 49:784-797. [PMID: 33653171 DOI: 10.1177/0192623321993425] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/21/2023]
Abstract
We introduce HistoNet, a deep neural network trained on normal tissue. On 1690 slides with rat tissue samples from 6 preclinical toxicology studies, tissue regions were outlined and annotated by pathologists into 46 different tissue classes. From these annotated regions, we sampled small 224 × 224 pixels images (patches) at 6 different levels of magnification. Using 4 studies as training set and 2 studies as test set, we trained VGG-16, ResNet-50, and Inception-v3 networks separately at each magnification level. Among these model architectures, Inception-v3 and ResNet-50 outperformed VGG-16. Inception-v3 identified the tissue from query images, with an accuracy up to 83.4%. Most misclassifications occurred between histologically similar tissues. Investigation of the features learned by the model (embedding layer) using Uniform Manifold Approximation and Projection revealed not only coherent clusters associated with the individual tissues but also subclusters corresponding to histologically meaningful structures that had not been annotated or trained for. This suggests that the histological representation learned by HistoNet could be useful as the basis of other machine learning algorithms and data mining. Finally, we found that models trained on rat tissues can be used on non-human primate and minipig tissues with minimal retraining.
Collapse
Affiliation(s)
- Holger Hoefling
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| | - Tobias Sing
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| | - Imtiaz Hossain
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| | - Julie Boisclair
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| | - Arno Doelemeyer
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| | - Thierry Flandre
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| | - Alessandro Piaia
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| | - Vincent Romanet
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| | | | | | - Esther Sutter
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| | - Oliver Turner
- Novartis Institutes for BioMedical Research, East Hanover, NJ, the United States
| | - Kuno Wuersch
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| | - Pierre Moulin
- 98560Novartis Institutes for BioMedical Research, Basel, Switzerland
| |
Collapse
|
91
|
Zheng Y, Jiang Z, Xie F, Shi J, Zhang H, Huai J, Cao M, Yang X. Diagnostic Regions Attention Network (DRA-Net) for Histopathology WSI Recommendation and Retrieval. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:1090-1103. [PMID: 33351756 DOI: 10.1109/tmi.2020.3046636] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
The development of whole slide imaging techniques and online digital pathology platforms have accelerated the popularization of telepathology for remote tumor diagnoses. During a diagnosis, the behavior information of the pathologist can be recorded by the platform and then archived with the digital case. The browsing path of the pathologist on the WSI is one of the valuable information in the digital database because the image content within the path is expected to be highly correlated with the diagnosis report of the pathologist. In this article, we proposed a novel approach for computer-assisted cancer diagnosis named session-based histopathology image recommendation (SHIR) based on the browsing paths on WSIs. To achieve the SHIR, we developed a novel diagnostic regions attention network (DRA-Net) to learn the pathology knowledge from the image content associated with the browsing paths. The DRA-Net does not rely on the pixel-level or region-level annotations of pathologists. All the data for training can be automatically collected by the digital pathology platform without interrupting the pathologists' diagnoses. The proposed approaches were evaluated on a gastric dataset containing 983 cases within 5 categories of gastric lesions. The quantitative and qualitative assessments on the dataset have demonstrated the proposed SHIR framework with the novel DRA-Net is effective in recommending diagnostically relevant cases for auxiliary diagnosis. The MRR and MAP for the recommendation are respectively 0.816 and 0.836 on the gastric dataset. The source code of the DRA-Net is available at https://github.com/zhengyushan/dpathnet.
Collapse
|
92
|
Wollmann T, Rohr K. Deep Consensus Network: Aggregating predictions to improve object detection in microscopy images. Med Image Anal 2021; 70:102019. [PMID: 33730623 DOI: 10.1016/j.media.2021.102019] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2019] [Revised: 02/13/2021] [Accepted: 02/19/2021] [Indexed: 12/11/2022]
Abstract
Detection of cells and particles in microscopy images is a common and challenging task. In recent years, detection approaches in computer vision achieved remarkable improvements by leveraging deep learning. Microscopy images pose challenges like small and clustered objects, low signal to noise, and complex shape and appearance, for which current approaches still struggle. We introduce Deep Consensus Network, a new deep neural network for object detection in microscopy images based on object centroids. Our network is trainable end-to-end and comprises a Feature Pyramid Network-based feature extractor, a Centroid Proposal Network, and a layer for ensembling detection hypotheses over all image scales and anchors. We suggest an anchor regularization scheme that favours prior anchors over regressed locations. We also propose a novel loss function based on Normalized Mutual Information to cope with strong class imbalance, which we derive within a Bayesian framework. In addition, we introduce an improved algorithm for Non-Maximum Suppression which significantly reduces the algorithmic complexity. Experiments on synthetic data are performed to provide insights into the properties of the proposed loss function and its robustness. We also applied our method to challenging data from the TUPAC16 mitosis detection challenge and the Particle Tracking Challenge, and achieved results competitive or better than state-of-the-art.
Collapse
Affiliation(s)
- Thomas Wollmann
- Biomedical Computer Vision Group, BioQuant, IPMB, Heidelberg University Im Neuenheimer Feld 267, Heidelberg, Germany.
| | - Karl Rohr
- Biomedical Computer Vision Group, BioQuant, IPMB, Heidelberg University Im Neuenheimer Feld 267, Heidelberg, Germany.
| |
Collapse
|
93
|
Mormont R, Geurts P, Maree R. Multi-Task Pre-Training of Deep Neural Networks for Digital Pathology. IEEE J Biomed Health Inform 2021; 25:412-421. [PMID: 32386169 DOI: 10.1109/jbhi.2020.2992878] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
In this work, we investigate multi-task learning as a way of pre-training models for classification tasks in digital pathology. It is motivated by the fact that many small and medium-size datasets have been released by the community over the years whereas there is no large scale dataset similar to ImageNet in the domain. We first assemble and transform many digital pathology datasets into a pool of 22 classification tasks and almost 900k images. Then, we propose a simple architecture and training scheme for creating a transferable model and a robust evaluation and selection protocol in order to evaluate our method. Depending on the target task, we show that our models used as feature extractors either improve significantly over ImageNet pre-trained models or provide comparable performance. Fine-tuning improves performance over feature extraction and is able to recover the lack of specificity of ImageNet features, as both pre-training sources yield comparable performance.
Collapse
|
94
|
Li Z, Zhang J, Tan T, Teng X, Sun X, Zhao H, Liu L, Xiao Y, Lee B, Li Y, Zhang Q, Sun S, Zheng Y, Yan J, Li N, Hong Y, Ko J, Jung H, Liu Y, Chen YC, Wang CW, Yurovskiy V, Maevskikh P, Khanagha V, Jiang Y, Yu L, Liu Z, Li D, Schuffler PJ, Yu Q, Chen H, Tang Y, Litjens G. Deep Learning Methods for Lung Cancer Segmentation in Whole-Slide Histopathology Images-The ACDC@LungHP Challenge 2019. IEEE J Biomed Health Inform 2021; 25:429-440. [PMID: 33216724 DOI: 10.1109/jbhi.2020.3039741] [Citation(s) in RCA: 31] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Accurate segmentation of lung cancer in pathology slides is a critical step in improving patient care. We proposed the ACDC@LungHP (Automatic Cancer Detection and Classification in Whole-slide Lung Histopathology) challenge for evaluating different computer-aided diagnosis (CADs) methods on the automatic diagnosis of lung cancer. The ACDC@LungHP 2019 focused on segmentation (pixel-wise detection) of cancer tissue in whole slide imaging (WSI), using an annotated dataset of 150 training images and 50 test images from 200 patients. This paper reviews this challenge and summarizes the top 10 submitted methods for lung cancer segmentation. All methods were evaluated using metrics using the precision, accuracy, sensitivity, specificity, and DICE coefficient (DC). The DC ranged from 0.7354 ±0.1149 to 0.8372 ±0.0858. The DC of the best method was close to the inter-observer agreement (0.8398 ±0.0890). All methods were based on deep learning and categorized into two groups: multi-model method and single model method. In general, multi-model methods were significantly better (p 0.01) than single model methods, with mean DC of 0.7966 and 0.7544, respectively. Deep learning based methods could potentially help pathologists find suspicious regions for further analysis of lung cancer in WSI.
Collapse
|
95
|
Tellez D, Litjens G, van der Laak J, Ciompi F. Neural Image Compression for Gigapixel Histopathology Image Analysis. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2021; 43:567-578. [PMID: 31442971 DOI: 10.1109/tpami.2019.2936841] [Citation(s) in RCA: 48] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
We propose Neural Image Compression (NIC), a two-step method to build convolutional neural networks for gigapixel image analysis solely using weak image-level labels. First, gigapixel images are compressed using a neural network trained in an unsupervised fashion, retaining high-level information while suppressing pixel-level noise. Second, a convolutional neural network (CNN) is trained on these compressed image representations to predict image-level labels, avoiding the need for fine-grained manual annotations. We compared several encoding strategies, namely reconstruction error minimization, contrastive training and adversarial feature learning, and evaluated NIC on a synthetic task and two public histopathology datasets. We found that NIC can exploit visual cues associated with image-level labels successfully, integrating both global and local visual information. Furthermore, we visualized the regions of the input gigapixel images where the CNN attended to, and confirmed that they overlapped with annotations from human experts.
Collapse
|
96
|
Vellal AD, Sirinukunwattan K, Kensler KH, Baker GM, Stancu AL, Pyle ME, Collins LC, Schnitt SJ, Connolly JL, Veta M, Eliassen AH, Tamimi RM, Heng YJ. Deep Learning Image Analysis of Benign Breast Disease to Identify Subsequent Risk of Breast Cancer. JNCI Cancer Spectr 2021; 5:pkaa119. [PMID: 33644680 PMCID: PMC7898083 DOI: 10.1093/jncics/pkaa119] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2020] [Revised: 11/04/2020] [Accepted: 12/18/2020] [Indexed: 12/16/2022] Open
Abstract
Background New biomarkers of risk may improve breast cancer (BC) risk prediction. We developed a computational pathology method to segment benign breast disease (BBD) whole slide images into epithelium, fibrous stroma, and fat. We applied our method to the BBD BC nested case-control study within the Nurses' Health Studies to assess whether computer-derived tissue composition or a morphometric signature was associated with subsequent risk of BC. Methods Tissue segmentation and nuclei detection deep-learning networks were established and applied to 3795 whole slide images from 293 cases who developed BC and 1132 controls who did not. Percentages of each tissue region were calculated, and 615 morphometric features were extracted. Elastic net regression was used to create a BC morphometric signature. Associations between BC risk factors and age-adjusted tissue composition among controls were assessed using analysis of covariance. Unconditional logistic regression, adjusting for the matching factors, BBD histological subtypes, parity, menopausal status, and body mass index evaluated the relationship between tissue composition and BC risk. All statistical tests were 2-sided. Results Among controls, direction of associations between BBD subtypes, parity, and number of births with breast composition varied by tissue region; select regions were associated with childhood body size, body mass index, age of menarche, and menopausal status (all P < .05). A higher proportion of epithelial tissue was associated with increased BC risk (odds ratio = 1.39, 95% confidence interval = 0.91 to 2.14, for highest vs lowest quartiles, P trend = .047). No morphometric signature was associated with BC. Conclusions The amount of epithelial tissue may be incorporated into risk assessment models to improve BC risk prediction.
Collapse
Affiliation(s)
- Adithya D Vellal
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Korsuk Sirinukunwattan
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, USA
- Department of Engineering Science, Institute of Biomedical Engineering (IBME), University of Oxford, Oxford, UK
- Big Data Institute, University of Oxford, Li Ka Shing Centre for Health Information and Discovery, Oxford, UK
- NIHR Oxford Biomedical Research Centre, Oxford University NHS Foundation Trust, Oxford, UK
| | - Kevin H Kensler
- Division of Population Sciences, Dana Farber Cancer Institute, Boston, MA, USA
| | - Gabrielle M Baker
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Andreea L Stancu
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Michael E Pyle
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Laura C Collins
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Stuart J Schnitt
- Dana-Farber/Brigham and Women's Cancer Center, Harvard Medical School, Dana-Farber Cancer Institute-Brigham and Women's Hospital, Boston, MA, USA
| | - James L Connolly
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Mitko Veta
- Medical Image Analysis Group, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - A Heather Eliassen
- Channing Division of Network Medicine, Department of Medicine, Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Rulla M Tamimi
- Channing Division of Network Medicine, Department of Medicine, Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA
| | - Yujing J Heng
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, USA
| |
Collapse
|
97
|
A Petri Dish for Histopathology Image Analysis. Artif Intell Med 2021. [DOI: 10.1007/978-3-030-77211-6_2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
98
|
Srinidhi CL, Ciga O, Martel AL. Deep neural network models for computational histopathology: A survey. Med Image Anal 2021; 67:101813. [PMID: 33049577 PMCID: PMC7725956 DOI: 10.1016/j.media.2020.101813] [Citation(s) in RCA: 193] [Impact Index Per Article: 64.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2019] [Revised: 05/12/2020] [Accepted: 08/09/2020] [Indexed: 12/14/2022]
Abstract
Histopathological images contain rich phenotypic information that can be used to monitor underlying mechanisms contributing to disease progression and patient survival outcomes. Recently, deep learning has become the mainstream methodological choice for analyzing and interpreting histology images. In this paper, we present a comprehensive review of state-of-the-art deep learning approaches that have been used in the context of histopathological image analysis. From the survey of over 130 papers, we review the field's progress based on the methodological aspect of different machine learning strategies such as supervised, weakly supervised, unsupervised, transfer learning and various other sub-variants of these methods. We also provide an overview of deep learning based survival models that are applicable for disease-specific prognosis tasks. Finally, we summarize several existing open datasets and highlight critical challenges and limitations with current deep learning approaches, along with possible avenues for future research.
Collapse
Affiliation(s)
- Chetan L Srinidhi
- Physical Sciences, Sunnybrook Research Institute, Toronto, Canada; Department of Medical Biophysics, University of Toronto, Canada.
| | - Ozan Ciga
- Department of Medical Biophysics, University of Toronto, Canada
| | - Anne L Martel
- Physical Sciences, Sunnybrook Research Institute, Toronto, Canada; Department of Medical Biophysics, University of Toronto, Canada
| |
Collapse
|
99
|
AIM in Surgical Pathology. Artif Intell Med 2021. [DOI: 10.1007/978-3-030-58080-3_278-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
100
|
Donovan TA, Moore FM, Bertram CA, Luong R, Bolfa P, Klopfleisch R, Tvedten H, Salas EN, Whitley DB, Aubreville M, Meuten DJ. Mitotic Figures-Normal, Atypical, and Imposters: A Guide to Identification. Vet Pathol 2020; 58:243-257. [PMID: 33371818 DOI: 10.1177/0300985820980049] [Citation(s) in RCA: 19] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022]
Abstract
Counting mitotic figures (MF) in hematoxylin and eosin-stained histologic sections is an integral part of the diagnostic pathologist's tumor evaluation. The mitotic count (MC) is used alone or as part of a grading scheme for assessment of prognosis and clinical decisions. Determining MCs is subjective, somewhat laborious, and has interobserver variation. Proposals for standardizing this parameter in the veterinary field are limited to terminology (use of the term MC) and area (MC is counted in an area measuring 2.37 mm2). Digital imaging techniques are now commonplace and widely used among veterinary pathologists, and field of view area can be easily calculated with digital imaging software. In addition to standardizing the methods of counting MF, the morphologic characteristics of MF and distinguishing atypical mitotic figures (AMF) versus mitotic-like figures (MLF) need to be defined. This article provides morphologic criteria for MF identification and for distinguishing normal phases of MF from AMF and MLF. Pertinent features of digital microscopy and application of computational pathology (CPATH) methods are discussed. Correct identification of MF will improve MC consistency, reproducibility, and accuracy obtained from manual (glass slide or whole-slide imaging) and CPATH approaches.
Collapse
Affiliation(s)
| | | | | | | | - Pompei Bolfa
- 41635Ross University, Basseterre, Saint Kitts and Nevis
| | | | - Harold Tvedten
- 8095Swedish University of Agricultural Sciences, Uppsala, Sweden
| | | | | | | | | |
Collapse
|