101
|
Wahab N, Khan A. Multifaceted fused-CNN based scoring of breast cancer whole-slide histopathology images. Appl Soft Comput 2020. [DOI: 10.1016/j.asoc.2020.106808] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
|
102
|
A completely annotated whole slide image dataset of canine breast cancer to aid human breast cancer research. Sci Data 2020; 7:417. [PMID: 33247116 PMCID: PMC7699627 DOI: 10.1038/s41597-020-00756-z] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2020] [Accepted: 10/29/2020] [Indexed: 01/10/2023] Open
Abstract
Canine mammary carcinoma (CMC) has been used as a model to investigate the pathogenesis of human breast cancer and the same grading scheme is commonly used to assess tumor malignancy in both. One key component of this grading scheme is the density of mitotic figures (MF). Current publicly available datasets on human breast cancer only provide annotations for small subsets of whole slide images (WSIs). We present a novel dataset of 21 WSIs of CMC completely annotated for MF. For this, a pathologist screened all WSIs for potential MF and structures with a similar appearance. A second expert blindly assigned labels, and for non-matching labels, a third expert assigned the final labels. Additionally, we used machine learning to identify previously undetected MF. Finally, we performed representation learning and two-dimensional projection to further increase the consistency of the annotations. Our dataset consists of 13,907 MF and 36,379 hard negatives. We achieved a mean F1-score of 0.791 on the test set and of up to 0.696 on a human breast cancer dataset. Measurement(s) | Mitotic Figure • Slide Image • non-mitotic structures • anatomical phenotype annotation | Technology Type(s) | Pathology Report • hematoxylin and eosin stain • machine learning | Factor Type(s) | breast cancer tissue | Sample Characteristic - Organism | Canis |
Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.13182857
Collapse
|
103
|
|
104
|
Aubreville M, Bertram CA, Marzahl C, Gurtner C, Dettwiler M, Schmidt A, Bartenschlager F, Merz S, Fragoso M, Kershaw O, Klopfleisch R, Maier A. Deep learning algorithms out-perform veterinary pathologists in detecting the mitotically most active tumor region. Sci Rep 2020; 10:16447. [PMID: 33020510 PMCID: PMC7536430 DOI: 10.1038/s41598-020-73246-2] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2020] [Accepted: 09/15/2020] [Indexed: 01/13/2023] Open
Abstract
Manual count of mitotic figures, which is determined in the tumor region with the highest mitotic activity, is a key parameter of most tumor grading schemes. It can be, however, strongly dependent on the area selection due to uneven mitotic figure distribution in the tumor section. We aimed to assess the question, how significantly the area selection could impact the mitotic count, which has a known high inter-rater disagreement. On a data set of 32 whole slide images of H&E-stained canine cutaneous mast cell tumor, fully annotated for mitotic figures, we asked eight veterinary pathologists (five board-certified, three in training) to select a field of interest for the mitotic count. To assess the potential difference on the mitotic count, we compared the mitotic count of the selected regions to the overall distribution on the slide. Additionally, we evaluated three deep learning-based methods for the assessment of highest mitotic density: In one approach, the model would directly try to predict the mitotic count for the presented image patches as a regression task. The second method aims at deriving a segmentation mask for mitotic figures, which is then used to obtain a mitotic density. Finally, we evaluated a two-stage object-detection pipeline based on state-of-the-art architectures to identify individual mitotic figures. We found that the predictions by all models were, on average, better than those of the experts. The two-stage object detector performed best and outperformed most of the human pathologists on the majority of tumor cases. The correlation between the predicted and the ground truth mitotic count was also best for this approach (0.963–0.979). Further, we found considerable differences in position selection between pathologists, which could partially explain the high variance that has been reported for the manual mitotic count. To achieve better inter-rater agreement, we propose to use a computer-based area selection for support of the pathologist in the manual mitotic count.
Collapse
Affiliation(s)
- Marc Aubreville
- Pattern Recognition Lab, Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
| | - Christof A Bertram
- Institute of Veterinary Pathology, Freie Universität Berlin, Berlin, Germany
| | - Christian Marzahl
- Pattern Recognition Lab, Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Corinne Gurtner
- Department of Infectious Diseases and Pathobiology, Vetsuisse Faculty, University of Bern, Bern, Switzerland
| | - Martina Dettwiler
- Department of Infectious Diseases and Pathobiology, Vetsuisse Faculty, University of Bern, Bern, Switzerland
| | - Anja Schmidt
- Vet Med Labor GmbH - Division of IDEXX Laboratories, Ludwigsburg, Germany
| | | | - Sophie Merz
- Institute of Veterinary Pathology, Freie Universität Berlin, Berlin, Germany
| | - Marco Fragoso
- Institute of Veterinary Pathology, Freie Universität Berlin, Berlin, Germany
| | - Olivia Kershaw
- Institute of Veterinary Pathology, Freie Universität Berlin, Berlin, Germany
| | - Robert Klopfleisch
- Institute of Veterinary Pathology, Freie Universität Berlin, Berlin, Germany
| | - Andreas Maier
- Pattern Recognition Lab, Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| |
Collapse
|
105
|
van Steenhoven JEC, Kuijer A, Kornegoor R, van Leeuwen G, van Gorp J, van Dalen T, van Diest PJ. Assessment of tumour proliferation by use of the mitotic activity index, and Ki67 and phosphohistone H3 expression, in early-stage luminal breast cancer. Histopathology 2020; 77:579-587. [PMID: 32557844 PMCID: PMC7539961 DOI: 10.1111/his.14185] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2020] [Accepted: 06/11/2020] [Indexed: 12/16/2022]
Abstract
AIMS Phosphohistone H3 (PhH3) has been proposed as a novel proliferation marker in breast cancer. This study compares the interobserver agreement for assessment of the mitotic activity index (MAI), Ki67 expression, and PhH3 in a cohort of oestrogen receptor (ER)-positive breast cancer patients. METHODS AND RESULTS Tumour samples of 159 luminal breast cancer patients were collected. MAI and PhH3 scores were assessed by three breast cancer pathologists. Ki67 scores were assessed separately by two of the three pathologists. PhH3-positive cells were counted in an area of 2 mm2 , with a threshold of ≥13 positive cells being used to discriminate between low-proliferative and high-proliferative tumours. Ki67 expression was assessed with the global scoring method. Ki67 percentages of <20% were considered to be low. The intraclass correlation coefficient (ICC) and Cohen's κ statistics were used to evaluate interobserver agreement. The impact on histological grading of replacing the MAI with PhH3 was assessed. Counting PhH3-positive cells was highly reproducible among all three observers (ICC of 0.86). The κ scores for the categorical PhH3 count (κ = 0.78, κ = 0.68, and κ = 0.80) reflected substantial agreement among all observers, whereas agreement for the MAI (κ = 0.38, κ = 0.52, and κ = 0.26) and Ki67 (κ = 0.55) was fair to moderate. When PhH3 was used to determine the histological grade, agreement in grading increased (PhH3, κ = 0.52, κ = 0.48, and κ = 0.52; MAI, κ = 0.43, κ = 0.35, and κ = 0.32), and the proportion of grade III tumours increased (14%, 18%, and 27%). CONCLUSION PhH3 seems to outperform Ki67 and the MAI as a reproducible means to measure tumour proliferation in luminal-type breast cancer. Variation in the assessment of histological grade might be reduced by using PhH3, but would result in an increase in the proportion of high-grade cancers.
Collapse
Affiliation(s)
- Julia E C van Steenhoven
- Department of SurgeryDiakonessenhuis UtrechtUtrechtThe Netherlands
- Department of PathologyUniversity Medical Centre UtrechtUtrecht UniversityUtrechtThe Netherlands
| | - Anne Kuijer
- Department of SurgerySt Antonius HospitalNieuwegeinThe Netherlands
| | | | - Gijs van Leeuwen
- Department of PathologySt Antonius HospitalNieuwegeinThe Netherlands
| | - Joost van Gorp
- Department of PathologySt Antonius HospitalNieuwegeinThe Netherlands
| | - Thijs van Dalen
- Department of SurgeryDiakonessenhuis UtrechtUtrechtThe Netherlands
| | - Paul J van Diest
- Department of PathologyUniversity Medical Centre UtrechtUtrecht UniversityUtrechtThe Netherlands
| |
Collapse
|
106
|
Dong Y, Puttapirat P, Deng J, Zhang X, Li C. LibMI: An Open Source Library for Efficient Histopathological Image Processing. J Pathol Inform 2020; 11:26. [PMID: 33042605 PMCID: PMC7518208 DOI: 10.4103/jpi.jpi_11_20] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2020] [Revised: 04/20/2020] [Accepted: 06/25/2020] [Indexed: 12/03/2022] Open
Abstract
BACKGROUND Whole-slide images (WSIs) as a kind of image data are rapidly growing in the digital pathology domain. With unusual high resolution, these images make them hard to be supported by conventional tools or file formats. Thus, it obstructs data sharing and automated analysis. Here, we propose a library, LibMI, along with its open and standardized image file format. They can be used together to efficiently read, write, modify, and annotate large images. MATERIALS AND METHODS LibMI utilizes the concept of pyramid image structure and lazy propagation from a segment tree algorithm to support reading and modifying and to guarantee that both operations have linear time complexity. Further, a cache mechanism was introduced to speed up the program. RESULTS LibMI is an open and efficient library for histopathological image processing. To demonstrate its functions, we applied it to several tasks including image thresholding, microscopic color correction, and storing pixel-wise information on WSIs. The result shows that libMI is particularly suitable for modifying large images. Furthermore, compared with congeneric libraries and file formats, libMI and modifiable multiscale image (MMSI) run 18.237 times faster on read-only tasks. CONCLUSIONS The combination of libMI library and MMSI file format enables developers to efficiently read and modify WSIs, thus can assist in pixel-wise image processing on extremely large images to promote building image processing pipeline. The library together with the data schema is freely available on GitLab: https://gitlab.com/BioAI/libMI.
Collapse
Affiliation(s)
- Yuxin Dong
- School of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an, Shaanxi, China
| | - Pargorn Puttapirat
- School of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an, Shaanxi, China
| | - Jingyi Deng
- School of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an, Shaanxi, China
| | - Xiangrong Zhang
- Institute of Intelligent Information Processing, Xidian University, Xi'an, Shaanxi, China
| | - Chen Li
- School of Electronic and Information Engineering, Xi'an Jiaotong University, Xi'an, Shaanxi, China
| |
Collapse
|
107
|
Duggento A, Conti A, Mauriello A, Guerrisi M, Toschi N. Deep computational pathology in breast cancer. Semin Cancer Biol 2020; 72:226-237. [PMID: 32818626 DOI: 10.1016/j.semcancer.2020.08.006] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2020] [Revised: 08/13/2020] [Accepted: 08/13/2020] [Indexed: 01/07/2023]
Abstract
Deep Learning (DL) algorithms are a set of techniques that exploit large and/or complex real-world datasets for cross-domain and cross-discipline prediction and classification tasks. DL architectures excel in computer vision tasks, and in particular image processing and interpretation. This has prompted a wave of disruptingly innovative applications in medical imaging, where DL strategies have the potential to vastly outperform human experts. This is particularly relevant in the context of histopathology, where whole slide imaging (WSI) of stained tissue in conjuction with DL algorithms for their interpretation, selection and cancer staging are beginning to play an ever increasing role in supporting human operators in visual assessments. This has the potential to reduce everyday workload as well as to increase precision and reproducibility across observers, centers, staining techniques and even pathologies. In this paper we introduce the most common DL architectures used in image analysis, with a focus on histopathological image analysis in general and in breast histology in particular. We briefly review how, state-of-art DL architectures compare to human performance on across a number of critical tasks such as mitotic count, tubules analysis and nuclear pleomorphism analysis. Also, the development of DL algorithms specialized to pathology images have been enormously fueled by a number of world-wide challenges based on large, multicentric image databases which are now publicly available. In turn, this has allowed most recent efforts to shift more and more towards semi-supervised learning methods, which provide greater flexibility and applicability. We also review all major repositories of manually labelled pathology images in breast cancer and provide an in-depth discussion of the challenges specific to training DL architectures to interpret WSI data, as well as a review of the state-of-the-art methods for interpretation of images generated from immunohistochemical analysis of breast lesions. We finally discuss the future challenges and opportunities which the adoption of DL paradigms is most likely to pose in the field of pathology for breast cancer detection, diagnosis, staging and prognosis. This review is intended as a comprehensive stepping stone into the field of modern computational pathology for a transdisciplinary readership across technical and medical disciplines.
Collapse
Affiliation(s)
- Andrea Duggento
- Department of Biomedicine and prevention, University of Rome Tor Vergata, Rome, Italy
| | - Allegra Conti
- Department of Biomedicine and prevention, University of Rome Tor Vergata, Rome, Italy
| | - Alessandro Mauriello
- Department of Experimental Medicine, University of Rome Tor Vergata, Rome, Italy
| | - Maria Guerrisi
- Department of Biomedicine and prevention, University of Rome Tor Vergata, Rome, Italy
| | - Nicola Toschi
- Department of Biomedicine and prevention, University of Rome Tor Vergata, Rome, Italy; A.A. Martinos Center for Biomedical Imaging - Harvard Medical School/MGH, Boston, MA, USA
| |
Collapse
|
108
|
Pantanowitz L, Hartman D, Qi Y, Cho EY, Suh B, Paeng K, Dhir R, Michelow P, Hazelhurst S, Song SY, Cho SY. Accuracy and efficiency of an artificial intelligence tool when counting breast mitoses. Diagn Pathol 2020; 15:80. [PMID: 32622359 PMCID: PMC7335442 DOI: 10.1186/s13000-020-00995-z] [Citation(s) in RCA: 40] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2020] [Accepted: 06/25/2020] [Indexed: 12/17/2022] Open
Abstract
BACKGROUND The mitotic count in breast carcinoma is an important prognostic marker. Unfortunately substantial inter- and intra-laboratory variation exists when pathologists manually count mitotic figures. Artificial intelligence (AI) coupled with whole slide imaging offers a potential solution to this problem. The aim of this study was to accordingly critique an AI tool developed to quantify mitotic figures in whole slide images of invasive breast ductal carcinoma. METHODS A representative H&E slide from 320 breast invasive ductal carcinoma cases was scanned at 40x magnification. Ten expert pathologists from two academic medical centers labeled mitotic figures in whole slide images to train and validate an AI algorithm to detect and count mitoses. Thereafter, 24 readers of varying expertise were asked to count mitotic figures with and without AI support in 140 high-power fields derived from a separate dataset. Their accuracy and efficiency of performing these tasks were calculated and statistical comparisons performed. RESULTS For each experience level the accuracy, precision and sensitivity of counting mitoses by users improved with AI support. There were 21 readers (87.5%) that identified more mitoses using AI support and 13 reviewers (54.2%) that decreased the quantity of falsely flagged mitoses with AI. More time was spent on this task for most participants when not provided with AI support. AI assistance resulted in an overall time savings of 27.8%. CONCLUSIONS This study demonstrates that pathology end-users were more accurate and efficient at quantifying mitotic figures in digital images of invasive breast carcinoma with the aid of AI. Higher inter-pathologist agreement with AI assistance suggests that such algorithms can also help standardize practice. Not surprisingly, there is much enthusiasm in pathology regarding the prospect of using AI in routine practice to perform mundane tasks such as counting mitoses.
Collapse
Affiliation(s)
- Liron Pantanowitz
- Department of Pathology, University of Pittsburgh Medical Center Cancer Pavilion, Suite 201, 5150 Centre Ave, Pittsburgh, PA, 15232, USA.
- Department of Anatomical Pathology, University of the Witwatersrand and National Health Laboratory Services, Johannesburg, South Africa.
| | - Douglas Hartman
- Department of Pathology, University of Pittsburgh Medical Center Cancer Pavilion, Suite 201, 5150 Centre Ave, Pittsburgh, PA, 15232, USA
| | - Yan Qi
- School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA
| | - Eun Yoon Cho
- Department of Pathology, Samsung Medical Center, Seoul, South Korea
| | | | | | - Rajiv Dhir
- Department of Pathology, University of Pittsburgh Medical Center Cancer Pavilion, Suite 201, 5150 Centre Ave, Pittsburgh, PA, 15232, USA
| | - Pamela Michelow
- Department of Anatomical Pathology, University of the Witwatersrand and National Health Laboratory Services, Johannesburg, South Africa
| | - Scott Hazelhurst
- School of Electrical & Information Engineering and Sydney Brenner Institute for Molecular Bioscience, University of the Witwatersrand, Johannesburg, South Africa
| | - Sang Yong Song
- Department of Pathology, Samsung Medical Center, Seoul, South Korea
| | - Soo Youn Cho
- Department of Pathology, Samsung Medical Center, Seoul, South Korea
| |
Collapse
|
109
|
Sultan AS, Elgharib MA, Tavares T, Jessri M, Basile JR. The use of artificial intelligence, machine learning and deep learning in oncologic histopathology. J Oral Pathol Med 2020; 49:849-856. [PMID: 32449232 DOI: 10.1111/jop.13042] [Citation(s) in RCA: 49] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2020] [Revised: 04/29/2020] [Accepted: 05/09/2020] [Indexed: 12/13/2022]
Abstract
BACKGROUND Recently, there has been a momentous drive to apply advanced artificial intelligence (AI) technologies to diagnostic medicine. The introduction of AI has provided vast new opportunities to improve health care and has introduced a new wave of heightened precision in oncologic pathology. The impact of AI on oncologic pathology has now become apparent, and its use with respect to oral oncology is still in the nascent stage. DISCUSSION A foundational overview of AI classification systems used in medicine and a review of common terminology used in machine learning and computational pathology will be presented. This paper provides a focused review on the recent advances in AI and deep learning in oncologic histopathology and oral oncology. In addition, specific emphasis on recent studies that have applied these technologies to oral cancer prognostication will also be discussed. CONCLUSION Machine and deep learning methods designed to enhance prognostication of oral cancer have been proposed with much of the work focused on prediction models on patient survival and locoregional recurrences in patients with oral squamous cell carcinomas (OSCC). Few studies have explored machine learning methods on OSCC digital histopathologic images. It is evident that further research at the whole slide image level is needed and future collaborations with computer scientists may progress the field of oral oncology.
Collapse
Affiliation(s)
- Ahmed S Sultan
- School of Dentistry, University of Maryland, Baltimore, MD, USA
| | | | - Tiffany Tavares
- School of Dentistry, University of Missouri-Kansas City, Kansas City, MO, USA
| | - Maryam Jessri
- Oral Health Centre of Western Australia, Perth, WA, Australia
| | - John R Basile
- School of Dentistry, University of Maryland, Baltimore, MD, USA.,University of Maryland Greenebaum Cancer Center, Baltimore, MD, USA
| |
Collapse
|
110
|
Chang MC, Mrkonjic M. Review of the current state of digital image analysis in breast pathology. Breast J 2020; 26:1208-1212. [PMID: 32342590 DOI: 10.1111/tbj.13858] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2019] [Accepted: 11/05/2019] [Indexed: 01/10/2023]
Abstract
Advances in digital image analysis have the potential to transform the practice of breast pathology. In the near future, a move to a digital workflow offers improvements in efficiency. Coupled with artificial intelligence (AI), digital pathology can assist pathologist interpretation, automate time-consuming tasks, and discover novel morphologic patterns. Opportunities for digital enhancements abound in breast pathology, from increasing reproducibility in grading and biomarker interpretation, to discovering features that correlate with patient outcome and treatment. Our objective is to review the most recent developments in digital pathology with clear impact to breast pathology practice. Although breast pathologists currently undertake limited adoption of digital methods, the field is rapidly evolving. Care is needed to validate emerging technologies for effective patient care.
Collapse
Affiliation(s)
- Martin C Chang
- University of Vermont Cancer Center, Burlington, VT, USA.,Department of Pathology and Laboratory Medicine, Larner College of Medicine at the University of Vermont, Burlington, VT, USA
| | - Miralem Mrkonjic
- Sinai Health System, Toronto, ON, Canada.,Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
111
|
Wetstein SC, Onken AM, Luffman C, Baker GM, Pyle ME, Kensler KH, Liu Y, Bakker B, Vlutters R, van Leeuwen MB, Collins LC, Schnitt SJ, Pluim JPW, Tamimi RM, Heng YJ, Veta M. Deep learning assessment of breast terminal duct lobular unit involution: Towards automated prediction of breast cancer risk. PLoS One 2020; 15:e0231653. [PMID: 32294107 PMCID: PMC7159218 DOI: 10.1371/journal.pone.0231653] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2019] [Accepted: 03/27/2020] [Indexed: 02/07/2023] Open
Abstract
Terminal duct lobular unit (TDLU) involution is the regression of milk-producing structures in the breast. Women with less TDLU involution are more likely to develop breast cancer. A major bottleneck in studying TDLU involution in large cohort studies is the need for labor-intensive manual assessment of TDLUs. We developed a computational pathology solution to automatically capture TDLU involution measures. Whole slide images (WSIs) of benign breast biopsies were obtained from the Nurses' Health Study. A set of 92 WSIs was annotated for acini, TDLUs and adipose tissue to train deep convolutional neural network (CNN) models for detection of acini, and segmentation of TDLUs and adipose tissue. These networks were integrated into a single computational method to capture TDLU involution measures including number of TDLUs per tissue area, median TDLU span and median number of acini per TDLU. We validated our method on 40 additional WSIs by comparing with manually acquired measures. Our CNN models detected acini with an F1 score of 0.73±0.07, and segmented TDLUs and adipose tissue with Dice scores of 0.84±0.13 and 0.87±0.04, respectively. The inter-observer ICC scores for manual assessments on 40 WSIs of number of TDLUs per tissue area, median TDLU span, and median acini count per TDLU were 0.71, 0.81 and 0.73, respectively. Intra-observer reliability was evaluated on 10/40 WSIs with ICC scores of >0.8. Inter-observer ICC scores between automated results and the mean of the two observers were: 0.80 for number of TDLUs per tissue area, 0.57 for median TDLU span, and 0.80 for median acini count per TDLU. TDLU involution measures evaluated by manual and automated assessment were inversely associated with age and menopausal status. We developed a computational pathology method to measure TDLU involution. This technology eliminates the labor-intensiveness and subjectivity of manual TDLU assessment, and can be applied to future breast cancer risk studies.
Collapse
Affiliation(s)
- Suzanne C. Wetstein
- Medical Image Analysis Group, Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Allison M. Onken
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States of America
| | - Christina Luffman
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States of America
| | - Gabrielle M. Baker
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States of America
| | - Michael E. Pyle
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States of America
| | - Kevin H. Kensler
- Division of Population Sciences, Dana Farber Cancer Institute, Boston, Massachusetts, United States of America
| | - Ying Liu
- Division of Public Health Sciences, Department of Surgery, Washington University School of Medicine and Alvin J. Siteman Cancer Center, St Louis, Missouri, United States of America
| | - Bart Bakker
- Philips Research Europe, High Tech Campus, Eindhoven, The Netherlands
| | - Ruud Vlutters
- Philips Research Europe, High Tech Campus, Eindhoven, The Netherlands
| | | | - Laura C. Collins
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States of America
| | - Stuart J. Schnitt
- Dana-Farber/Brigham and Women’s Cancer Center, Harvard Medical School, Dana-Farber Cancer Institute-Brigham and Women’s Hospital, Boston, Massachusetts, United States of America
| | - Josien P. W. Pluim
- Medical Image Analysis Group, Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Rulla M. Tamimi
- Channing Division of Network Medicine, Department of Medicine, Harvard Medical School, Brigham and Women’s Hospital, Boston, Massachusetts, United States of America
| | - Yujing J. Heng
- Department of Pathology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, Massachusetts, United States of America
| | - Mitko Veta
- Medical Image Analysis Group, Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
| |
Collapse
|
112
|
Jiang Y, Yang M, Wang S, Li X, Sun Y. Emerging role of deep learning-based artificial intelligence in tumor pathology. Cancer Commun (Lond) 2020; 40:154-166. [PMID: 32277744 PMCID: PMC7170661 DOI: 10.1002/cac2.12012] [Citation(s) in RCA: 154] [Impact Index Per Article: 38.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2020] [Accepted: 02/06/2020] [Indexed: 12/11/2022] Open
Abstract
The development of digital pathology and progression of state-of-the-art algorithms for computer vision have led to increasing interest in the use of artificial intelligence (AI), especially deep learning (DL)-based AI, in tumor pathology. The DL-based algorithms have been developed to conduct all kinds of work involved in tumor pathology, including tumor diagnosis, subtyping, grading, staging, and prognostic prediction, as well as the identification of pathological features, biomarkers and genetic changes. The applications of AI in pathology not only contribute to improve diagnostic accuracy and objectivity but also reduce the workload of pathologists and subsequently enable them to spend additional time on high-level decision-making tasks. In addition, AI is useful for pathologists to meet the requirements of precision oncology. However, there are still some challenges relating to the implementation of AI, including the issues of algorithm validation and interpretability, computing systems, the unbelieving attitude of pathologists, clinicians and patients, as well as regulators and reimbursements. Herein, we present an overview on how AI-based approaches could be integrated into the workflow of pathologists and discuss the challenges and perspectives of the implementation of AI in tumor pathology.
Collapse
Affiliation(s)
- Yahui Jiang
- Department of PathologyKey Laboratory of Cancer Prevention and TherapyTianjin's Clinical Research Center for CancerNational Clinical Research Center for CancerTianjin Cancer Institute and HospitalTianjin Medical UniversityTianjin300060P. R. China
| | - Meng Yang
- Department Epidemiology and BiostatisticsKey Laboratory of Cancer Prevention and TherapyTianjin's Clinical Research Center for CancerNational Clinical Research Center for CancerTianjin Cancer Institute and HospitalTianjin Medical UniversityTianjin300060P.R. China
| | - Shuhao Wang
- Institute for Interdisciplinary Information SciencesTsinghua UniversityBeijing100084P. R. China
| | - Xiangchun Li
- Department Epidemiology and BiostatisticsKey Laboratory of Cancer Prevention and TherapyTianjin's Clinical Research Center for CancerNational Clinical Research Center for CancerTianjin Cancer Institute and HospitalTianjin Medical UniversityTianjin300060P.R. China
| | - Yan Sun
- Department of PathologyKey Laboratory of Cancer Prevention and TherapyTianjin's Clinical Research Center for CancerNational Clinical Research Center for CancerTianjin Cancer Institute and HospitalTianjin Medical UniversityTianjin300060P. R. China
| |
Collapse
|
113
|
Kucharski D, Kleczek P, Jaworek-Korjakowska J, Dyduch G, Gorgon M. Semi-Supervised Nests of Melanocytes Segmentation Method Using Convolutional Autoencoders. SENSORS 2020; 20:s20061546. [PMID: 32168748 PMCID: PMC7146382 DOI: 10.3390/s20061546] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/29/2019] [Revised: 02/28/2020] [Accepted: 03/04/2020] [Indexed: 11/24/2022]
Abstract
In this research, we present a semi-supervised segmentation solution using convolutional autoencoders to solve the problem of segmentation tasks having a small number of ground-truth images. We evaluate the proposed deep network architecture for the detection of nests of nevus cells in histopathological images of skin specimens is an important step in dermatopathology. The diagnostic criteria based on the degree of uniformity and symmetry of border irregularities are particularly vital in dermatopathology, in order to distinguish between benign and malignant skin lesions. However, to the best of our knowledge, it is the first described method to segment the nests region. The novelty of our approach is not only the area of research, but, furthermore, we address a problem with a small ground-truth dataset. We propose an effective computer-vision based deep learning tool that can perform the nests segmentation based on an autoencoder architecture with two learning steps. Experimental results verified the effectiveness of the proposed approach and its ability to segment nests areas with Dice similarity coefficient 0.81, sensitivity 0.76, and specificity 0.94, which is a state-of-the-art result.
Collapse
Affiliation(s)
- Dariusz Kucharski
- Department of Automatic Control and Robotics, AGH University of Science and Technology, al. A. Mickiewicza 30, 30-059 Krakow, Poland; (P.K.); (J.J.-K.); (M.G.)
- Correspondence:
| | - Pawel Kleczek
- Department of Automatic Control and Robotics, AGH University of Science and Technology, al. A. Mickiewicza 30, 30-059 Krakow, Poland; (P.K.); (J.J.-K.); (M.G.)
| | - Joanna Jaworek-Korjakowska
- Department of Automatic Control and Robotics, AGH University of Science and Technology, al. A. Mickiewicza 30, 30-059 Krakow, Poland; (P.K.); (J.J.-K.); (M.G.)
| | - Grzegorz Dyduch
- Chair of Pathomorphology, Jagiellonian University Medical College, ul. Grzegorzecka 16, 31-531 Krakow, Poland
| | - Marek Gorgon
- Department of Automatic Control and Robotics, AGH University of Science and Technology, al. A. Mickiewicza 30, 30-059 Krakow, Poland; (P.K.); (J.J.-K.); (M.G.)
| |
Collapse
|
114
|
Valkonen M, Isola J, Ylinen O, Muhonen V, Saxlin A, Tolonen T, Nykter M, Ruusuvuori P. Cytokeratin-Supervised Deep Learning for Automatic Recognition of Epithelial Cells in Breast Cancers Stained for ER, PR, and Ki-67. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:534-542. [PMID: 31398111 DOI: 10.1109/tmi.2019.2933656] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
Immunohistochemistry (IHC) of ER, PR, and Ki-67 are routinely used assays in breast cancer diagnostics. Determination of the proportion of stained cells (labeling index) should be restricted on malignant epithelial cells, carefully avoiding tumor infiltrating stroma and inflammatory cells. Here, we developed a deep learning based digital mask for automated epithelial cell detection using fluoro-chromogenic cytokeratin-Ki-67 double staining and sequential hematoxylin-IHC staining as training material. A partially pre-trained deep convolutional neural network was fine-tuned using image batches from 152 patient samples of invasive breast tumors. Validity of the trained digital epithelial cell masks was studied with 366 images captured from 98 unseen samples, by comparing the epithelial cell masks to cytokeratin images and by visual evaluation of the brightfield images performed by two pathologists. A good discrimination of epithelial cells was achieved (AUC of mean ROC = 0.93; defined as the area under mean receiver operating characteristics), and well in concordance with pathologists' visual assessment (4.01/5 and 4.67/5). The effect of epithelial cell masking on the Ki-67 labeling index was substantial. 52 tumor images initially classified as low proliferation (Ki-67 < 14%) without epithelial cell masking were re-classified as high proliferation (Ki-67 ≥ 14%) after applying the deep learning based epithelial cell mask. The digital epithelial cell masks were found applicable also to IHC of ER and PR. We conclude that deep learning can be applied to detect carcinoma cells in breast cancer samples stained with conventional brightfield IHC.
Collapse
|
115
|
Wang T, Wang M, Yang L, Li Z, Loh XJ, Chen X. Cyber-Physiochemical Interfaces. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2020; 32:e1905522. [PMID: 31944425 DOI: 10.1002/adma.201905522] [Citation(s) in RCA: 33] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/25/2019] [Revised: 10/07/2019] [Indexed: 06/10/2023]
Abstract
Living things rely on various physical, chemical, and biological interfaces, e.g., somatosensation, olfactory/gustatory perception, and nervous system response. They help organisms to perceive the world, adapt to their surroundings, and maintain internal and external balance. Interfacial information exchanges are complicated but efficient, delicate but precise, and multimodal but unisonous, which has driven researchers to study the science of such interfaces and develop techniques with potential applications in health monitoring, smart robotics, future wearable devices, and cyber physical/human systems. To understand better the issues in these interfaces, a cyber-physiochemical interface (CPI) that is capable of extracting biophysical and biochemical signals, and closely relating them to electronic, communication, and computing technology, to provide the core for aforementioned applications, is proposed. The scientific and technical progress in CPI is summarized, and the challenges to and strategies for building stable interfaces, including materials, sensor development, system integration, and data processing techniques are discussed. It is hoped that this will result in an unprecedented multi-disciplinary network of scientific collaboration in CPI to explore much uncharted territory for progress, providing technical inspiration-to the development of the next-generation personal healthcare technology, smart sports-technology, adaptive prosthetics and augmentation of human capability, etc.
Collapse
Affiliation(s)
- Ting Wang
- Innovative Center for Flexible Devices (iFLEX), Max Planck - NTU Joint Lab for Artificial Senses, School of Materials Science and Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore, 639798, Singapore
| | - Ming Wang
- Innovative Center for Flexible Devices (iFLEX), Max Planck - NTU Joint Lab for Artificial Senses, School of Materials Science and Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore, 639798, Singapore
| | - Le Yang
- Institute of Materials Research and Engineering, Agency for Science Technology and Research (A*STAR), 2 Fusionopolis Way, Singapore, 138634, Singapore
| | - Zhuyun Li
- Innovative Center for Flexible Devices (iFLEX), Max Planck - NTU Joint Lab for Artificial Senses, School of Materials Science and Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore, 639798, Singapore
| | - Xian Jun Loh
- Institute of Materials Research and Engineering, Agency for Science Technology and Research (A*STAR), 2 Fusionopolis Way, Singapore, 138634, Singapore
| | - Xiaodong Chen
- Innovative Center for Flexible Devices (iFLEX), Max Planck - NTU Joint Lab for Artificial Senses, School of Materials Science and Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore, 639798, Singapore
| |
Collapse
|
116
|
Ibrahim A, Gamble P, Jaroensri R, Abdelsamea MM, Mermel CH, Chen PHC, Rakha EA. Artificial intelligence in digital breast pathology: Techniques and applications. Breast 2020; 49:267-273. [PMID: 31935669 PMCID: PMC7375550 DOI: 10.1016/j.breast.2019.12.007] [Citation(s) in RCA: 78] [Impact Index Per Article: 19.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2019] [Accepted: 12/12/2019] [Indexed: 12/16/2022] Open
Abstract
Breast cancer is the most common cancer and second leading cause of cancer-related death worldwide. The mainstay of breast cancer workup is histopathological diagnosis - which guides therapy and prognosis. However, emerging knowledge about the complex nature of cancer and the availability of tailored therapies have exposed opportunities for improvements in diagnostic precision. In parallel, advances in artificial intelligence (AI) along with the growing digitization of pathology slides for the primary diagnosis are a promising approach to meet the demand for more accurate detection, classification and prediction of behaviour of breast tumours. In this article, we cover the current and prospective uses of AI in digital pathology for breast cancer, review the basics of digital pathology and AI, and outline outstanding challenges in the field.
Collapse
Affiliation(s)
- Asmaa Ibrahim
- Department of Histopathology, Division of Cancer and Stem Cells, School of Medicine, The University of Nottingham and Nottingham University Hospitals NHS Trust, Nottingham City Hospital, Nottingham, NG5 1PB, UK
| | | | | | - Mohammed M Abdelsamea
- School of Computing and Digital Technology, Birmingham City University, Birmingham, UK
| | | | | | - Emad A Rakha
- Department of Histopathology, Division of Cancer and Stem Cells, School of Medicine, The University of Nottingham and Nottingham University Hospitals NHS Trust, Nottingham City Hospital, Nottingham, NG5 1PB, UK.
| |
Collapse
|
117
|
Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media. Mod Pathol 2020; 33:2169-2185. [PMID: 32467650 PMCID: PMC7581495 DOI: 10.1038/s41379-020-0540-1] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2020] [Revised: 03/17/2020] [Accepted: 03/17/2020] [Indexed: 02/07/2023]
Abstract
Pathologists are responsible for rapidly providing a diagnosis on critical health issues. Challenging cases benefit from additional opinions of pathologist colleagues. In addition to on-site colleagues, there is an active worldwide community of pathologists on social media for complementary opinions. Such access to pathologists worldwide has the capacity to improve diagnostic accuracy and generate broader consensus on next steps in patient care. From Twitter we curate 13,626 images from 6,351 tweets from 25 pathologists from 13 countries. We supplement the Twitter data with 113,161 images from 1,074,484 PubMed articles. We develop machine learning and deep learning models to (i) accurately identify histopathology stains, (ii) discriminate between tissues, and (iii) differentiate disease states. Area Under Receiver Operating Characteristic (AUROC) is 0.805-0.996 for these tasks. We repurpose the disease classifier to search for similar disease states given an image and clinical covariates. We report precision@k = 1 = 0.7618 ± 0.0018 (chance 0.397 ± 0.004, mean ±stdev ). The classifiers find that texture and tissue are important clinico-visual features of disease. Deep features trained only on natural images (e.g., cats and dogs) substantially improved search performance, while pathology-specific deep features and cell nuclei features further improved search to a lesser extent. We implement a social media bot (@pathobot on Twitter) to use the trained classifiers to aid pathologists in obtaining real-time feedback on challenging cases. If a social media post containing pathology text and images mentions the bot, the bot generates quantitative predictions of disease state (normal/artifact/infection/injury/nontumor, preneoplastic/benign/low-grade-malignant-potential, or malignant) and lists similar cases across social media and PubMed. Our project has become a globally distributed expert system that facilitates pathological diagnosis and brings expertise to underserved regions or hospitals with less expertise in a particular disease. This is the first pan-tissue pan-disease (i.e., from infection to malignancy) method for prediction and search on social media, and the first pathology study prospectively tested in public on social media. We will share data through http://pathobotology.org . We expect our project to cultivate a more connected world of physicians and improve patient care worldwide.
Collapse
|
118
|
Belciug S. Pathologist at work. Artif Intell Cancer 2020. [DOI: 10.1016/b978-0-12-820201-2.00003-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/19/2022] Open
|
119
|
Tellez D, Litjens G, Bándi P, Bulten W, Bokhorst JM, Ciompi F, van der Laak J. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Med Image Anal 2019; 58:101544. [DOI: 10.1016/j.media.2019.101544] [Citation(s) in RCA: 179] [Impact Index Per Article: 35.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2019] [Revised: 06/25/2019] [Accepted: 08/19/2019] [Indexed: 11/17/2022]
|
120
|
Kleczek P, Jaworek-Korjakowska J, Gorgon M. A novel method for tissue segmentation in high-resolution H&E-stained histopathological whole-slide images. Comput Med Imaging Graph 2019; 79:101686. [PMID: 31816574 DOI: 10.1016/j.compmedimag.2019.101686] [Citation(s) in RCA: 24] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2019] [Revised: 10/28/2019] [Accepted: 11/14/2019] [Indexed: 10/25/2022]
Abstract
Tissue segmentation in whole-slide images is an important task in digital pathology, required for efficient and accurate computer-aided diagnostics. Precise tissue segmentation is particularly significant for a correct diagnosis in cases, when tissue structure of a specimen is very porous, such as skin specimens. In this paper, we addressed the problem of fore- and background segmentation in histopatological images of skin specimens stained with hematoxylin and eosin (H&E), which has not been solved yet, by a novel method based on a combination of statistical analysis, color thresholding, and binary morphology. We validated our algorithm on large extracts from 60 high-resolution whole slide images, with differing staining quality and captured under varying imaging conditions, from three laboratories. The size of extracts varies from 2000×1500 to 20000×30000 pixels and the number of images used in our study matches the number of H&E images used by other research teams. We compared our method to the published ones (GrabCut and FESI) and showed that our approach outperforms its counterparts (Jaccard index of 0.929 vs. 0.776 and 0.695).
Collapse
Affiliation(s)
- Pawel Kleczek
- AGH University of Science and Technology, Department of Automatic Control and Robotics, Krakow, Poland.
| | | | - Marek Gorgon
- AGH University of Science and Technology, Department of Automatic Control and Robotics, Krakow, Poland
| |
Collapse
|
121
|
Dimitriou N, Arandjelović O, Caie PD. Deep Learning for Whole Slide Image Analysis: An Overview. Front Med (Lausanne) 2019; 6:264. [PMID: 31824952 PMCID: PMC6882930 DOI: 10.3389/fmed.2019.00264] [Citation(s) in RCA: 119] [Impact Index Per Article: 23.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2019] [Accepted: 10/29/2019] [Indexed: 12/15/2022] Open
Abstract
The widespread adoption of whole slide imaging has increased the demand for effective and efficient gigapixel image analysis. Deep learning is at the forefront of computer vision, showcasing significant improvements over previous methodologies on visual understanding. However, whole slide images have billions of pixels and suffer from high morphological heterogeneity as well as from different types of artifacts. Collectively, these impede the conventional use of deep learning. For the clinical translation of deep learning solutions to become a reality, these challenges need to be addressed. In this paper, we review work on the interdisciplinary attempt of training deep neural networks using whole slide images, and highlight the different ideas underlying these methodologies.
Collapse
Affiliation(s)
- Neofytos Dimitriou
- School of Computer Science, University of St Andrews, St Andrews, United Kingdom
| | - Ognjen Arandjelović
- School of Computer Science, University of St Andrews, St Andrews, United Kingdom
| | - Peter D Caie
- School of Medicine, University of St Andrews, St Andrews, United Kingdom
| |
Collapse
|
122
|
Bertram CA, Aubreville M, Marzahl C, Maier A, Klopfleisch R. A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Sci Data 2019; 6:274. [PMID: 31754105 PMCID: PMC6872565 DOI: 10.1038/s41597-019-0290-4] [Citation(s) in RCA: 21] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2019] [Accepted: 09/13/2019] [Indexed: 11/09/2022] Open
Abstract
We introduce a novel, large-scale dataset for microscopy cell annotations. The dataset includes 32 whole slide images (WSI) of canine cutaneous mast cell tumors, selected to include both low grade cases as well as high grade cases. The slides have been completely annotated for mitotic figures and we provide secondary annotations for neoplastic mast cells, inflammatory granulocytes, and mitotic figure look-alikes. Additionally to a blinded two-expert manual annotation with consensus, we provide an algorithm-aided dataset, where potentially missed mitotic figures were detected by a deep neural network and subsequently assessed by two human experts. We included 262,481 annotations in total, out of which 44,880 represent mitotic figures. For algorithmic validation, we used a customized RetinaNet approach, followed by a cell classification network. We find F1-Scores of 0.786 and 0.820 for the manually labelled and the algorithm-aided dataset, respectively. The dataset provides, for the first time, WSIs completely annotated for mitotic figures and thus enables assessment of mitosis detection algorithms on complete WSIs as well as region of interest detection algorithms.
Collapse
Affiliation(s)
- Christof A Bertram
- Institute of Veterinary Pathology, Freie Universität Berlin, Berlin, Germany
| | - Marc Aubreville
- Pattern Recognition Lab, Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
| | - Christian Marzahl
- Pattern Recognition Lab, Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Andreas Maier
- Pattern Recognition Lab, Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Robert Klopfleisch
- Institute of Veterinary Pathology, Freie Universität Berlin, Berlin, Germany
| |
Collapse
|
123
|
Jung H, Lodhi B, Kang J. An automatic nuclei segmentation method based on deep convolutional neural networks for histopathology images. BMC Biomed Eng 2019; 1:24. [PMID: 32903361 PMCID: PMC7422516 DOI: 10.1186/s42490-019-0026-8] [Citation(s) in RCA: 25] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2019] [Accepted: 09/02/2019] [Indexed: 12/18/2022] Open
Abstract
BACKGROUND Since nuclei segmentation in histopathology images can provide key information for identifying the presence or stage of a disease, the images need to be assessed carefully. However, color variation in histopathology images, and various structures of nuclei are two major obstacles in accurately segmenting and analyzing histopathology images. Several machine learning methods heavily rely on hand-crafted features which have limitations due to manual thresholding. RESULTS To obtain robust results, deep learning based methods have been proposed. Deep convolutional neural networks (DCNN) used for automatically extracting features from raw image data have been proven to achieve great performance. Inspired by such achievements, we propose a nuclei segmentation method based on DCNNs. To normalize the color of histopathology images, we use a deep convolutional Gaussian mixture color normalization model which is able to cluster pixels while considering the structures of nuclei. To segment nuclei, we use Mask R-CNN which achieves state-of-the-art object segmentation performance in the field of computer vision. In addition, we perform multiple inference as a post-processing step to boost segmentation performance. We evaluate our segmentation method on two different datasets. The first dataset consists of histopathology images of various organ while the other consists histopathology images of the same organ. Performance of our segmentation method is measured in various experimental setups at the object-level and the pixel-level. In addition, we compare the performance of our method with that of existing state-of-the-art methods. The experimental results show that our nuclei segmentation method outperforms the existing methods. CONCLUSIONS We propose a nuclei segmentation method based on DCNNs for histopathology images. The proposed method which uses Mask R-CNN with color normalization and multiple inference post-processing provides robust nuclei segmentation results. Our method also can facilitate downstream nuclei morphological analyses as it provides high-quality features extracted from histopathology images.
Collapse
Affiliation(s)
- Hwejin Jung
- Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea
| | - Bilal Lodhi
- Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea
| | - Jaewoo Kang
- Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea
- Interdisciplinary Graduate Program in Bioinformatics, Korea University, Seoul, Republic of Korea
| |
Collapse
|
124
|
Serag A, Ion-Margineanu A, Qureshi H, McMillan R, Saint Martin MJ, Diamond J, O'Reilly P, Hamilton P. Translational AI and Deep Learning in Diagnostic Pathology. Front Med (Lausanne) 2019; 6:185. [PMID: 31632973 PMCID: PMC6779702 DOI: 10.3389/fmed.2019.00185] [Citation(s) in RCA: 123] [Impact Index Per Article: 24.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2019] [Accepted: 07/30/2019] [Indexed: 12/15/2022] Open
Abstract
There has been an exponential growth in the application of AI in health and in pathology. This is resulting in the innovation of deep learning technologies that are specifically aimed at cellular imaging and practical applications that could transform diagnostic pathology. This paper reviews the different approaches to deep learning in pathology, the public grand challenges that have driven this innovation and a range of emerging applications in pathology. The translation of AI into clinical practice will require applications to be embedded seamlessly within digital pathology workflows, driving an integrated approach to diagnostics and providing pathologists with new tools that accelerate workflow and improve diagnostic consistency and reduce errors. The clearance of digital pathology for primary diagnosis in the US by some manufacturers provides the platform on which to deliver practical AI. AI and computational pathology will continue to mature as researchers, clinicians, industry, regulatory organizations and patient advocacy groups work together to innovate and deliver new technologies to health care providers: technologies which are better, faster, cheaper, more precise, and safe.
Collapse
|
125
|
Otálora S, Atzori M, Andrearczyk V, Khan A, Müller H. Staining Invariant Features for Improving Generalization of Deep Convolutional Neural Networks in Computational Pathology. Front Bioeng Biotechnol 2019; 7:198. [PMID: 31508414 PMCID: PMC6716536 DOI: 10.3389/fbioe.2019.00198] [Citation(s) in RCA: 25] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2019] [Accepted: 08/05/2019] [Indexed: 12/12/2022] Open
Abstract
One of the main obstacles for the implementation of deep convolutional neural networks (DCNNs) in the clinical pathology workflow is their low capability to overcome variability in slide preparation and scanner configuration, that leads to changes in tissue appearance. Some of these variations may not be not included in the training data, which means that the models have a risk to not generalize well. Addressing such variations and evaluating them in reproducible scenarios allows understanding of when the models generalize better, which is crucial for performance improvements and better DCNN models. Staining normalization techniques (often based on color deconvolution and deep learning) and color augmentation approaches have shown improvements in the generalization of the classification tasks for several tissue types. Domain-invariant training of DCNN's is also a promising technique to address the problem of training a single model for different domains, since it includes the source domain information to guide the training toward domain-invariant features, achieving state-of-the-art results in classification tasks. In this article, deep domain adaptation in convolutional networks (DANN) is applied to computational pathology and compared with widely used staining normalization and color augmentation methods in two challenging classification tasks. The classification tasks rely on two openly accessible datasets, targeting Gleason grading in prostate cancer, and mitosis classification in breast tissue. The benchmark of the different techniques and their combination in two DCNN architectures allows us to assess the generalization abilities and advantages of each method in the considered classification tasks. The code for reproducing our experiments and preprocessing the data is publicly available1. Quantitative and qualitative results show that the use of DANN helps model generalization to external datasets. The combination of several techniques to manage color heterogeneity suggests that several methods together, such as color augmentation methods with DANN training, can generalize even further. The results do not show a single best technique among the considered methods, even when combining them. However, color augmentation and DANN training obtain most often the best results (alone or combined with color normalization and color augmentation). The statistical significance of the results and the embeddings visualizations provide useful insights to design DCNN that generalizes to unseen staining appearances. Furthermore, in this work, we release for the first time code for DANN evaluation in open access datasets for computational pathology. This work opens the possibility for further research on using DANN models together with techniques that can overcome the tissue preparation differences across datasets to tackle limited generalization.
Collapse
Affiliation(s)
- Sebastian Otálora
- Institute of Information Systems, HES-SO University of Applied Sciences and Arts Western Switzerland, Sierre, Switzerland.,Computer Science Centre (CUI), University of Geneva, Geneva, Switzerland
| | - Manfredo Atzori
- Institute of Information Systems, HES-SO University of Applied Sciences and Arts Western Switzerland, Sierre, Switzerland
| | - Vincent Andrearczyk
- Institute of Information Systems, HES-SO University of Applied Sciences and Arts Western Switzerland, Sierre, Switzerland
| | - Amjad Khan
- Institute of Information Systems, HES-SO University of Applied Sciences and Arts Western Switzerland, Sierre, Switzerland.,Institute of Pathology, University of Bern, Bern, Switzerland
| | - Henning Müller
- Institute of Information Systems, HES-SO University of Applied Sciences and Arts Western Switzerland, Sierre, Switzerland.,Medical Faculty, University of Geneva, Geneva, Switzerland
| |
Collapse
|
126
|
Shahid AH, Singh M. Computational intelligence techniques for medical diagnosis and prognosis: Problems and current developments. Biocybern Biomed Eng 2019. [DOI: 10.1016/j.bbe.2019.05.010] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022]
|
127
|
Hegde N, Hipp JD, Liu Y, Emmert-Buck M, Reif E, Smilkov D, Terry M, Cai CJ, Amin MB, Mermel CH, Nelson PQ, Peng LH, Corrado GS, Stumpe MC. Similar image search for histopathology: SMILY. NPJ Digit Med 2019; 2:56. [PMID: 31304402 PMCID: PMC6588631 DOI: 10.1038/s41746-019-0131-z] [Citation(s) in RCA: 60] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2019] [Accepted: 05/29/2019] [Indexed: 12/19/2022] Open
Abstract
The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Although these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. As pathology images are extremely large (up to 100,000 pixels in each dimension), further laborious visual search of each image may be needed to find the feature of interest. In this paper, we introduce a deep-learning-based reverse image search tool for histopathology images: Similar Medical Images Like Yours (SMILY). We assessed SMILY's ability to retrieve search results in two ways: using pathologist-provided annotations, and via prospective studies where pathologists evaluated the quality of SMILY search results. As a negative control in the second evaluation, pathologists were blinded to whether search results were retrieved by SMILY or randomly. In both types of assessments, SMILY was able to retrieve search results with similar histologic features, organ site, and prostate cancer Gleason grade compared with the original query. SMILY may be a useful general-purpose tool in the pathologist's arsenal, to improve the efficiency of searching large archives of histopathology images, without the need to develop and implement specific tools for each application.
Collapse
Affiliation(s)
| | | | - Yun Liu
- Google AI Healthcare, Mountain View, CA 94043 USA
| | | | - Emily Reif
- Google AI Healthcare, Mountain View, CA 94043 USA
| | | | | | | | - Mahul B. Amin
- Department of Pathology and Laboratory Medicine, University of Tennessee Health Science Center, Memphis, TN 38163 USA
| | | | | | - Lily H. Peng
- Google AI Healthcare, Mountain View, CA 94043 USA
| | | | - Martin C. Stumpe
- Google AI Healthcare, Mountain View, CA 94043 USA
- Present Address: AI and Data Science, Tempus Labs Inc, Chicago, IL USA
| |
Collapse
|
128
|
Li C, Wang X, Liu W, Latecki LJ, Wang B, Huang J. Weakly supervised mitosis detection in breast histopathology images using concentric loss. Med Image Anal 2019; 53:165-178. [DOI: 10.1016/j.media.2019.01.013] [Citation(s) in RCA: 68] [Impact Index Per Article: 13.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2017] [Revised: 01/19/2019] [Accepted: 01/21/2019] [Indexed: 11/30/2022]
|