1
|
Altini N, Puro E, Taccogna MG, Marino F, De Summa S, Saponaro C, Mattioli E, Zito FA, Bevilacqua V. Tumor Cellularity Assessment of Breast Histopathological Slides via Instance Segmentation and Pathomic Features Explainability. Bioengineering (Basel) 2023; 10:bioengineering10040396. [PMID: 37106583 PMCID: PMC10135772 DOI: 10.3390/bioengineering10040396] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Revised: 03/14/2023] [Accepted: 03/19/2023] [Indexed: 04/29/2023] Open
Abstract
The segmentation and classification of cell nuclei are pivotal steps in the pipelines for the analysis of bioimages. Deep learning (DL) approaches are leading the digital pathology field in the context of nuclei detection and classification. Nevertheless, the features that are exploited by DL models to make their predictions are difficult to interpret, hindering the deployment of such methods in clinical practice. On the other hand, pathomic features can be linked to an easier description of the characteristics exploited by the classifiers for making the final predictions. Thus, in this work, we developed an explainable computer-aided diagnosis (CAD) system that can be used to support pathologists in the evaluation of tumor cellularity in breast histopathological slides. In particular, we compared an end-to-end DL approach that exploits the Mask R-CNN instance segmentation architecture with a two steps pipeline, where the features are extracted while considering the morphological and textural characteristics of the cell nuclei. Classifiers that are based on support vector machines and artificial neural networks are trained on top of these features in order to discriminate between tumor and non-tumor nuclei. Afterwards, the SHAP (Shapley additive explanations) explainable artificial intelligence technique was employed to perform a feature importance analysis, which led to an understanding of the features processed by the machine learning models for making their decisions. An expert pathologist validated the employed feature set, corroborating the clinical usability of the model. Even though the models resulting from the two-stage pipeline are slightly less accurate than those of the end-to-end approach, the interpretability of their features is clearer and may help build trust for pathologists to adopt artificial intelligence-based CAD systems in their clinical workflow. To further show the validity of the proposed approach, it has been tested on an external validation dataset, which was collected from IRCCS Istituto Tumori "Giovanni Paolo II" and made publicly available to ease research concerning the quantification of tumor cellularity.
Collapse
Affiliation(s)
- Nicola Altini
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona n. 4, 70126 Bari, Italy
| | - Emilia Puro
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona n. 4, 70126 Bari, Italy
| | - Maria Giovanna Taccogna
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona n. 4, 70126 Bari, Italy
| | - Francescomaria Marino
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona n. 4, 70126 Bari, Italy
| | - Simona De Summa
- Molecular Diagnostics and Pharmacogenetics Unit, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco n. 65, 70124 Bari, Italy
| | - Concetta Saponaro
- Laboratory of Preclinical and Translational Research, Centro di Riferimento Oncologico della Basilicata (IRCCS-CROB), Via Padre Pio n. 1, 85028 Rionero in Vulture, Italy
| | - Eliseo Mattioli
- Pathology Department, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco n. 65, 70124 Bari, Italy
| | - Francesco Alfredo Zito
- Pathology Department, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco n. 65, 70124 Bari, Italy
| | - Vitoantonio Bevilacqua
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona n. 4, 70126 Bari, Italy
- Apulian Bioengineering s.r.l., Via delle Violette n. 14, 70026 Modugno, Italy
| |
Collapse
|
2
|
Zhou X, Gu M, Cheng Z. Local Integral Regression Network for Cell Nuclei Detection. Entropy (Basel) 2021; 23:e23101336. [PMID: 34682060 PMCID: PMC8535160 DOI: 10.3390/e23101336] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/15/2021] [Accepted: 10/07/2021] [Indexed: 11/16/2022]
Abstract
Nuclei detection is a fundamental task in the field of histopathology image analysis and remains challenging due to cellular heterogeneity. Recent studies explore convolutional neural networks to either isolate them with sophisticated boundaries (segmentation-based methods) or locate the centroids of the nuclei (counting-based approaches). Although these two methods have demonstrated superior success, their fully supervised training demands considerable and laborious pixel-wise annotations manually labeled by pathology experts. To alleviate such tedious effort and reduce the annotation cost, we propose a novel local integral regression network (LIRNet) that allows both fully and weakly supervised learning (FSL/WSL) frameworks for nuclei detection. Furthermore, the LIRNet can output an exquisite density map of nuclei, in which the localization of each nucleus is barely affected by the post-processing algorithms. The quantitative experimental results demonstrate that the FSL version of the LIRNet achieves a state-of-the-art performance compared to other counterparts. In addition, the WSL version has exhibited a competitive detection performance and an effortless data annotation that requires only 17.5% of the annotation effort.
Collapse
|
3
|
Ho DJ, Mas Montserrat D, Fu C, Salama P, Dunn KW, Delp EJ. Sphere estimation network: three-dimensional nuclei detection of fluorescence microscopy images. J Med Imaging (Bellingham) 2020; 7:044003. [PMID: 32904135 PMCID: PMC7451995 DOI: 10.1117/1.jmi.7.4.044003] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2020] [Accepted: 08/07/2020] [Indexed: 02/04/2023] Open
Abstract
Purpose: Fluorescence microscopy visualizes three-dimensional subcellular structures in tissue with two-photon microscopy achieving deeper penetration into tissue. Nuclei detection, which is essential for analyzing tissue for clinical and research purposes, remains a challenging problem due to the spatial variability of nuclei. Recent advancements in deep learning techniques have enabled the analysis of fluorescence microscopy data to localize and segment nuclei. However, these localization or segmentation techniques would require additional steps to extract characteristics of nuclei. We develop a 3D convolutional neural network, called Sphere Estimation Network (SphEsNet), to extract characteristics of nuclei without any postprocessing steps. Approach: To simultaneously estimate the center locations of nuclei and their sizes, SphEsNet is composed of two branches to localize nuclei center coordinates and to estimate their radii. Synthetic microscopy volumes automatically generated using a spatially constrained cycle-consistent adversarial network are used for training the network because manually generating 3D real ground truth volumes would be extremely tedious. Results: Three SphEsNet models based on the size of nuclei were trained and tested on five real fluorescence microscopy data sets from rat kidney and mouse intestine. Our method can successfully detect nuclei in multiple locations with various sizes. In addition, our method was compared with other techniques and outperformed them based on object-level precision, recall, and F 1 score. Our model achieved 89.90% for F 1 score. Conclusions: SphEsNet can simultaneously localize nuclei and estimate their size without additional steps. SphEsNet can be potentially used to extract more information from nuclei in fluorescence microscopy images.
Collapse
Affiliation(s)
- David Joon Ho
- Memorial Sloan Kettering Cancer Center, Department of Pathology, New York, New York, United States
| | - Daniel Mas Montserrat
- Purdue University, School of Electrical and Computer Engineering, Video and Image Processing Laboratory, West Lafayette, Indiana, United States
| | - Chichen Fu
- Purdue University, School of Electrical and Computer Engineering, Video and Image Processing Laboratory, West Lafayette, Indiana, United States
| | - Paul Salama
- Indiana University-Purdue University, Department of Electrical and Computer Engineering, Indianapolis, Indiana, United States
| | - Kenneth W. Dunn
- Indiana University, School of Medicine, Division of Nephrology, Indianapolis, Indiana, United States
| | - Edward J. Delp
- Purdue University, School of Electrical and Computer Engineering, Video and Image Processing Laboratory, West Lafayette, Indiana, United States
| |
Collapse
|
4
|
Zhou H, Machupalli R, Mandal M. Efficient FPGA Implementation of Automatic Nuclei Detection in Histopathology Images. J Imaging 2019; 5:jimaging5010021. [PMID: 34465711 PMCID: PMC8320863 DOI: 10.3390/jimaging5010021] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2018] [Revised: 12/27/2018] [Accepted: 01/11/2019] [Indexed: 11/17/2022] Open
Abstract
Accurate and efficient detection of cell nuclei is an important step towards the development of a pathology-based Computer Aided Diagnosis. Generally, high-resolution histopathology images are very large, in the order of billion pixels, therefore nuclei detection is a highly compute intensive task, and software implementation requires a significant amount of processing time. To assist the doctors in real time, special hardware accelerators, which can reduce the processing time, are required. In this paper, we propose a Field Programmable Gate Array (FPGA) implementation of automated nuclei detection algorithm using generalized Laplacian of Gaussian filters. The experimental results show that the implemented architecture has the potential to provide a significant improvement in processing time without losing detection accuracy.
Collapse
|
5
|
Kost H, Homeyer A, Molin J, Lundström C, Hahn HK. Training Nuclei Detection Algorithms with Simple Annotations. J Pathol Inform 2017; 8:21. [PMID: 28584683 PMCID: PMC5450511 DOI: 10.4103/jpi.jpi_3_17] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2017] [Accepted: 03/17/2017] [Indexed: 12/27/2022] Open
Abstract
BACKGROUND Generating good training datasets is essential for machine learning-based nuclei detection methods. However, creating exhaustive nuclei contour annotations, to derive optimal training data from, is often infeasible. METHODS We compared different approaches for training nuclei detection methods solely based on nucleus center markers. Such markers contain less accurate information, especially with regard to nuclear boundaries, but can be produced much easier and in greater quantities. The approaches use different automated sample extraction methods to derive image positions and class labels from nucleus center markers. In addition, the approaches use different automated sample selection methods to improve the detection quality of the classification algorithm and reduce the run time of the training process. We evaluated the approaches based on a previously published generic nuclei detection algorithm and a set of Ki-67-stained breast cancer images. RESULTS A Voronoi tessellation-based sample extraction method produced the best performing training sets. However, subsampling of the extracted training samples was crucial. Even simple class balancing improved the detection quality considerably. The incorporation of active learning led to a further increase in detection quality. CONCLUSIONS With appropriate sample extraction and selection methods, nuclei detection algorithms trained on the basis of simple center marker annotations can produce comparable quality to algorithms trained on conventionally created training sets.
Collapse
Affiliation(s)
- Henning Kost
- Fraunhofer Institute for Medical Image Computing MEVIS, 28359 Bremen, Germany
| | - André Homeyer
- Fraunhofer Institute for Medical Image Computing MEVIS, 28359 Bremen, Germany
| | - Jesper Molin
- Department of Applied Information Technology, Chalmers University of Technology, 41258 Gothenburg, Sweden.,Sectra AB, 58330 Linköping, Sweden.,Center for Medical Image Science and Visualization, Linköping University, 58183 Linköping, Sweden
| | - Claes Lundström
- Sectra AB, 58330 Linköping, Sweden.,Center for Medical Image Science and Visualization, Linköping University, 58183 Linköping, Sweden
| | - Horst Karl Hahn
- Fraunhofer Institute for Medical Image Computing MEVIS, 28359 Bremen, Germany
| |
Collapse
|
6
|
Irshad H, Montaser-Kouhsari L, Waltz G, Bucur O, Nowak J, Dong F, Knoblauch N, Beck AH. Crowdsourcing image annotation for nucleus detection and segmentation in computational pathology: evaluating experts, automated methods, and the crowd. Pac Symp Biocomput 2015:294-305. [PMID: 25592590 PMCID: PMC4299942 DOI: 10.1142/9789814644730_0029] [Citation(s) in RCA: 23] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/24/2023]
Abstract
The development of tools in computational pathology to assist physicians and biomedical scientists in the diagnosis of disease requires access to high-quality annotated images for algorithm learning and evaluation. Generating high-quality expert-derived annotations is time-consuming and expensive. We explore the use of crowdsourcing for rapidly obtaining annotations for two core tasks in com- putational pathology: nucleus detection and nucleus segmentation. We designed and implemented crowdsourcing experiments using the CrowdFlower platform, which provides access to a large set of labor channel partners that accesses and manages millions of contributors worldwide. We obtained annotations from four types of annotators and compared concordance across these groups. We obtained: crowdsourced annotations for nucleus detection and segmentation on a total of 810 images; annotations using automated methods on 810 images; annotations from research fellows for detection and segmentation on 477 and 455 images, respectively; and expert pathologist-derived annotations for detection and segmentation on 80 and 63 images, respectively. For the crowdsourced annotations, we evaluated performance across a range of contributor skill levels (1, 2, or 3). The crowdsourced annotations (4,860 images in total) were completed in only a fraction of the time and cost required for obtaining annotations using traditional methods. For the nucleus detection task, the research fellow-derived annotations showed the strongest concordance with the expert pathologist- derived annotations (F-M =93.68%), followed by the crowd-sourced contributor levels 1,2, and 3 and the automated method, which showed relatively similar performance (F-M = 87.84%, 88.49%, 87.26%, and 86.99%, respectively). For the nucleus segmentation task, the crowdsourced contributor level 3-derived annotations, research fellow-derived annotations, and automated method showed the strongest concordance with the expert pathologist-derived annotations (F-M = 66.41%, 65.93%, and 65.36%, respectively), followed by the contributor levels 2 and 1 (60.89% and 60.87%, respectively). When the research fellows were used as a gold-standard for the segmentation task, all three con- tributor levels of the crowdsourced annotations significantly outperformed the automated method (F-M = 62.21%, 62.47%, and 65.15% vs. 51.92%). Aggregating multiple annotations from the crowd to obtain a consensus annotation resulted in the strongest performance for the crowd-sourced segmentation. For both detection and segmentation, crowd-sourced performance is strongest with small images (400 × 400 pixels) and degrades significantly with the use of larger images (600 × 600 and 800 × 800 pixels). We conclude that crowdsourcing to non-experts can be used for large-scale labeling microtasks in computational pathology and offers a new approach for the rapid generation of labeled images for algorithm development and evaluation.
Collapse
Affiliation(s)
- H. Irshad
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston USA
| | | | - G. Waltz
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston USA
| | - O. Bucur
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston USA
| | - J.A. Nowak
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston USA
| | - F. Dong
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston USA
| | - N.W. Knoblauch
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston USA
| | - A. H. Beck
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston USA
| |
Collapse
|
7
|
Irshad H. Automated mitosis detection in histopathology using morphological and multi-channel statistics features. J Pathol Inform 2013; 4:10. [PMID: 23858385 PMCID: PMC3709420 DOI: 10.4103/2153-3539.112695] [Citation(s) in RCA: 94] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2013] [Accepted: 03/13/2013] [Indexed: 11/11/2022] Open
Abstract
Context: According to Nottingham grading system, mitosis count plays a critical role in cancer diagnosis and grading. Manual counting of mitosis is tedious and subject to considerable inter- and intra-reader variations. Aims: The aim is to improve the accuracy of mitosis detection by selecting the color channels that better capture the statistical and morphological features, which classify mitosis from other objects. Materials and Methods: We propose a framework that includes comprehensive analysis of statistics and morphological features in selected channels of various color spaces that assist pathologists in mitosis detection. In candidate detection phase, we perform Laplacian of Gaussian, thresholding, morphology and active contour model on blue-ratio image to detect and segment candidates. In candidate classification phase, we extract a total of 143 features including morphological, first order and second order (texture) statistics features for each candidate in selected channels and finally classify using decision tree classifier. Results and Discussion: The proposed method has been evaluated on Mitosis Detection in Breast Cancer Histological Images (MITOS) dataset provided for an International Conference on Pattern Recognition 2012 contest and achieved 74% and 71% detection rate, 70% and 56% precision and 72% and 63% F-Measure on Aperio and Hamamatsu images, respectively. Conclusions and Future Work: The proposed multi-channel features computation scheme uses fixed image scale and extracts nuclei features in selected channels of various color spaces. This simple but robust model has proven to be highly efficient in capturing multi-channels statistical features for mitosis detection, during the MITOS international benchmark. Indeed, the mitosis detection of critical importance in cancer diagnosis is a very challenging visual task. In future work, we plan to use color deconvolution as preprocessing and Hough transform or local extrema based candidate detection in order to reduce the number of candidates in mitosis and non-mitosis classes.
Collapse
Affiliation(s)
- Humayun Irshad
- Mathematics, Science and Information Technology, Computer IPAL CNRS, University of Joseph Fourier, Grenoble, France
| |
Collapse
|
8
|
Schüffler PJ, Fuchs TJ, Ong CS, Wild PJ, Rupp NJ, Buhmann JM. TMARKER: A free software toolkit for histopathological cell counting and staining estimation. J Pathol Inform 2013; 4:S2. [PMID: 23766938 PMCID: PMC3678753 DOI: 10.4103/2153-3539.109804] [Citation(s) in RCA: 45] [Impact Index Per Article: 4.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2012] [Accepted: 01/21/2013] [Indexed: 11/04/2022] Open
Abstract
BACKGROUND Histological tissue analysis often involves manual cell counting and staining estimation of cancerous cells. These assessments are extremely time consuming, highly subjective and prone to error, since immunohistochemically stained cancer tissues usually show high variability in cell sizes, morphological structures and staining quality. To facilitate reproducible analysis in clinical practice as well as for cancer research, objective computer assisted staining estimation is highly desirable. METHODS We employ machine learning algorithms as randomized decision trees and support vector machines for nucleus detection and classification. Superpixels as segmentation over the tissue image are classified into foreground and background and thereafter into malignant and benign, learning from the user's feedback. As a fast alternative without nucleus classification, the existing color deconvolution method is incorporated. RESULTS Our program TMARKER connects already available workflows for computational pathology and immunohistochemical tissue rating with modern active learning algorithms from machine learning and computer vision. On a test dataset of human renal clear cell carcinoma and prostate carcinoma, the performance of the used algorithms is equivalent to two independent pathologists for nucleus detection and classification. CONCLUSION We present a novel, free and operating system independent software package for computational cell counting and staining estimation, supporting IHC stained tissue analysis in clinic and for research. Proprietary toolboxes for similar tasks are expensive, bound to specific commercial hardware (e.g. a microscope) and mostly not quantitatively validated in terms of performance and reproducibility. We are confident that the presented software package will proof valuable for the scientific community and we anticipate a broader application domain due to the possibility to interactively learn models for new image types.
Collapse
Affiliation(s)
- Peter J. Schüffler
- Institute for Computational Science, ETH Zurich, Switzerland
- Competence Center for Systems Physiology and Metabolic Diseases, ETH Zurich, Switzerland
| | | | - Cheng Soon Ong
- NICTA, The University of Melbourne, Parkville VIC 3010, Australia
| | - Peter J. Wild
- Institute of Pathology, University Hospital Zurich, Switzerland
| | - Niels J. Rupp
- Institute of Pathology, University Hospital Zurich, Switzerland
| | - Joachim M. Buhmann
- Institute for Computational Science, ETH Zurich, Switzerland
- Competence Center for Systems Physiology and Metabolic Diseases, ETH Zurich, Switzerland
| |
Collapse
|