1
|
Linmans J, Raya G, van der Laak J, Litjens G. Diffusion models for out-of-distribution detection in digital pathology. Med Image Anal 2024; 93:103088. [PMID: 38228075 DOI: 10.1016/j.media.2024.103088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2023] [Revised: 12/29/2023] [Accepted: 01/10/2024] [Indexed: 01/18/2024]
Abstract
The ability to detect anomalies, i.e. anything not seen during training or out-of-distribution (OOD), in medical imaging applications is essential for successfully deploying machine learning systems. Filtering out OOD data using unsupervised learning is especially promising because it does not require costly annotations. A new class of models called AnoDDPMs, based on denoising diffusion probabilistic models (DDPMs), has recently achieved significant progress in unsupervised OOD detection. This work provides a benchmark for unsupervised OOD detection methods in digital pathology. By leveraging fast sampling techniques, we apply AnoDDPM on a large enough scale for whole-slide image analysis on the complete test set of the Camelyon16 challenge. Based on ROC analysis, we show that AnoDDPMs can detect OOD data with an AUC of up to 94.13 and 86.93 on two patch-level OOD detection tasks, outperforming the other unsupervised methods. We observe that AnoDDPMs alter the semantic properties of inputs, replacing anomalous data with more benign-looking tissue. Furthermore, we highlight the flexibility of AnoDDPM towards different information bottlenecks by evaluating reconstruction errors for inputs with different signal-to-noise ratios. While there is still a significant performance gap with fully supervised learning, AnoDDPMs show considerable promise in the field of OOD detection in digital pathology.
Collapse
Affiliation(s)
- Jasper Linmans
- Department of Pathology, RadboudUMC Graduate School, Radboud University Medical Center, Nijmegen, The Netherlands.
| | - Gabriel Raya
- Jheronimus Academy of Data Science, 's-Hertogenbosch, The Netherlands
| | - Jeroen van der Laak
- Department of Pathology, RadboudUMC Graduate School, Radboud University Medical Center, Nijmegen, The Netherlands; Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | - Geert Litjens
- Department of Pathology, RadboudUMC Graduate School, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
2
|
Geijs DJ, Dooper S, Aswolinskiy W, Hillen LM, Amir AL, Litjens G. Detection and subtyping of basal cell carcinoma in whole-slide histopathology using weakly-supervised learning. Med Image Anal 2024; 93:103063. [PMID: 38194735 DOI: 10.1016/j.media.2023.103063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Revised: 10/15/2023] [Accepted: 12/05/2023] [Indexed: 01/11/2024]
Abstract
The frequency of basal cell carcinoma (BCC) cases is putting an increasing strain on dermatopathologists. BCC is the most common type of skin cancer, and its incidence is increasing rapidly worldwide. AI can play a significant role in reducing the time and effort required for BCC diagnostics and thus improve the overall efficiency of the process. To train such an AI system in a fully-supervised fashion however, would require a large amount of pixel-level annotation by already strained dermatopathologists. Therefore, in this study, our primary objective was to develop a weakly-supervised for the identification of basal cell carcinoma (BCC) and the stratification of BCC into low-risk and high-risk categories within histopathology whole-slide images (WSI). We compared Clustering-constrained Attention Multiple instance learning (CLAM) with StreamingCLAM and hypothesized that the latter would be the superior approach. A total of 5147 images were used to train and validate the models, which were subsequently tested on an internal set of 949 images and an external set of 183 images. The labels for training were automatically extracted from free-text pathology reports using a rule-based approach. All data has been made available through the COBRA dataset. The results showed that both the CLAM and StreamingCLAM models achieved high performance for the detection of BCC, with an area under the ROC curve (AUC) of 0.994 and 0.997, respectively, on the internal test set and 0.983 and 0.993 on the external dataset. Furthermore, the models performed well on risk stratification, with AUC values of 0.912 and 0.931, respectively, on the internal set, and 0.851 and 0.883 on the external set. In every single metric the StreamingCLAM model outperformed the CLAM model or is on par. The performance of both models was comparable to that of two pathologists who scored 240 BCC positive slides. Additionally, in the public test set, StreamingCLAM demonstrated a comparable AUC of 0.958, markedly superior to CLAM's 0.803. This difference was statistically significant and emphasized the strength and better adaptability of the StreamingCLAM approach.
Collapse
Affiliation(s)
- Daan J Geijs
- Department of Pathology, Research Institute for Medical Innovation, Radboud University Medical Center, Nijmegen, The Netherlands.
| | - Stephan Dooper
- Department of Pathology, Research Institute for Medical Innovation, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Witali Aswolinskiy
- Department of Pathology, Research Institute for Medical Innovation, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Lisa M Hillen
- Department of Pathology, GROW-School for Oncology & Developmental Biology, Maastricht University Medical Center, MUMC+, Maastricht, The Netherlands
| | - Avital L Amir
- Department of Pathology, Research Institute for Medical Innovation, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Geert Litjens
- Department of Pathology, Research Institute for Medical Innovation, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
3
|
Faryna K, van der Laak J, Litjens G. Automatic data augmentation to improve generalization of deep learning in H&E stained histopathology. Comput Biol Med 2024; 170:108018. [PMID: 38281317 DOI: 10.1016/j.compbiomed.2024.108018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Revised: 01/03/2024] [Accepted: 01/22/2024] [Indexed: 01/30/2024]
Abstract
In histopathology practice, scanners, tissue processing, staining, and image acquisition protocols vary from center to center, resulting in subtle variations in images. Vanilla convolutional neural networks are sensitive to such domain shifts. Data augmentation is a popular way to improve domain generalization. Currently, state-of-the-art domain generalization in computational pathology is achieved using a manually curated set of augmentation transforms. However, manual tuning of augmentation parameters is time-consuming and can lead to sub-optimal generalization performance. Meta-learning frameworks can provide efficient ways to find optimal training hyper-parameters, including data augmentation. In this study, we hypothesize that an automated search of augmentation hyper-parameters can provide superior generalization performance and reduce experimental optimization time. We select four state-of-the-art automatic augmentation methods from general computer vision and investigate their capacity to improve domain generalization in histopathology. We analyze their performance on data from 25 centers across two different tasks: tumor metastasis detection in lymph nodes and breast cancer tissue type classification. On tumor metastasis detection, most automatic augmentation methods achieve comparable performance to state-of-the-art manual augmentation. On breast cancer tissue type classification, the leading automatic augmentation method significantly outperforms state-of-the-art manual data augmentation.
Collapse
Affiliation(s)
- Khrystyna Faryna
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA, Nijmegen, The Netherlands.
| | - Jeroen van der Laak
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA, Nijmegen, The Netherlands; Center for Medical Image Science and Visualization, Linköping University, SE-581 83, Linköping, Sweden
| | - Geert Litjens
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA, Nijmegen, The Netherlands
| |
Collapse
|
4
|
Solé-Guardia G, Luijten M, Geenen B, Claassen JAHR, Litjens G, de Leeuw FE, Wiesmann M, Kiliaan AJ. Three-dimensional identification of microvascular pathology and neurovascular inflammation in severe white matter hyperintensity: a case report. Sci Rep 2024; 14:5004. [PMID: 38424226 PMCID: PMC10904845 DOI: 10.1038/s41598-024-55733-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Accepted: 02/27/2024] [Indexed: 03/02/2024] Open
Abstract
White matter hyperintensities (WMH) are the most prevalent markers of cerebral small vessel disease (SVD), which is the major vascular risk factor for dementia. Microvascular pathology and neuroinflammation are suggested to drive the transition from normal-appearing white matter (NAWM) to WMH, particularly in individuals with hypertension. However, current imaging techniques cannot capture ongoing NAWM changes. The transition from NAWM into WMH is a continuous process, yet white matter lesions are often examined dichotomously, which may explain their underlying heterogeneity. Therefore, we examined microvascular and neurovascular inflammation pathology in NAWM and severe WMH three-dimensionally, along with gradual magnetic resonance imaging (MRI) fluid-attenuated inversion recovery (FLAIR) signal (sub-)segmentation. In WMH, the vascular network exhibited reduced length and complexity compared to NAWM. Neuroinflammation was more severe in WMH. Vascular inflammation was more pronounced in NAWM, suggesting its potential significance in converting NAWM into WMH. Moreover, the (sub-)segmentation of FLAIR signal displayed varying degrees of vascular pathology, particularly within WMH regions. These findings highlight the intricate interplay between microvascular pathology and neuroinflammation in the transition from NAWM to WMH. Further examination of neurovascular inflammation across MRI-visible alterations could aid deepening our understanding on WMH conversion, and therewith how to improve the prognosis of SVD.
Collapse
Affiliation(s)
- Gemma Solé-Guardia
- Department of Medical Imaging, Anatomy, Donders Institute for Brain, Cognition & Behavior, Preclinical Imaging Center PRIME, Radboud Alzheimer Center, Radboud university medical center, 6525 EZ, Nijmegen, PO Box 9101, The Netherlands
| | - Matthijs Luijten
- Department of Medical Imaging, Anatomy, Donders Institute for Brain, Cognition & Behavior, Preclinical Imaging Center PRIME, Radboud Alzheimer Center, Radboud university medical center, 6525 EZ, Nijmegen, PO Box 9101, The Netherlands
| | - Bram Geenen
- Department of Medical Imaging, Anatomy, Donders Institute for Brain, Cognition & Behavior, Preclinical Imaging Center PRIME, Radboud Alzheimer Center, Radboud university medical center, 6525 EZ, Nijmegen, PO Box 9101, The Netherlands
| | - Jurgen A H R Claassen
- Department of Geriatrics, Donders Institute for Brain, Cognition & Behavior, Radboud Alzheimer Center, Radboud university medical center, Nijmegen, The Netherlands
- Department of Cardiovascular Sciences, University of Leicester, Leicester, UK
| | - Geert Litjens
- Department of Pathology, Radboud university medical center, Nijmegen, The Netherlands
- Computational Pathology Group, Research Institute for Medical Innovation, Radboud university medical center, Nijmegen, The Netherlands
| | - Frank-Erik de Leeuw
- Department of Neurology, Donders Institute for Brain, Cognition & Behavior, Radboud university medical center, Nijmegen, The Netherlands
| | - Maximilian Wiesmann
- Department of Medical Imaging, Anatomy, Donders Institute for Brain, Cognition & Behavior, Preclinical Imaging Center PRIME, Radboud Alzheimer Center, Radboud university medical center, 6525 EZ, Nijmegen, PO Box 9101, The Netherlands
| | - Amanda J Kiliaan
- Department of Medical Imaging, Anatomy, Donders Institute for Brain, Cognition & Behavior, Preclinical Imaging Center PRIME, Radboud Alzheimer Center, Radboud university medical center, 6525 EZ, Nijmegen, PO Box 9101, The Netherlands.
| |
Collapse
|
5
|
Reinke A, Tizabi MD, Baumgartner M, Eisenmann M, Heckmann-Nötzel D, Kavur AE, Rädsch T, Sudre CH, Acion L, Antonelli M, Arbel T, Bakas S, Benis A, Buettner F, Cardoso MJ, Cheplygina V, Chen J, Christodoulou E, Cimini BA, Farahani K, Ferrer L, Galdran A, van Ginneken B, Glocker B, Godau P, Hashimoto DA, Hoffman MM, Huisman M, Isensee F, Jannin P, Kahn CE, Kainmueller D, Kainz B, Karargyris A, Kleesiek J, Kofler F, Kooi T, Kopp-Schneider A, Kozubek M, Kreshuk A, Kurc T, Landman BA, Litjens G, Madani A, Maier-Hein K, Martel AL, Meijering E, Menze B, Moons KGM, Müller H, Nichyporuk B, Nickel F, Petersen J, Rafelski SM, Rajpoot N, Reyes M, Riegler MA, Rieke N, Saez-Rodriguez J, Sánchez CI, Shetty S, Summers RM, Taha AA, Tiulpin A, Tsaftaris SA, Van Calster B, Varoquaux G, Yaniv ZR, Jäger PF, Maier-Hein L. Understanding metric-related pitfalls in image analysis validation. Nat Methods 2024; 21:182-194. [PMID: 38347140 DOI: 10.1038/s41592-023-02150-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2023] [Accepted: 12/12/2023] [Indexed: 02/15/2024]
Abstract
Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.
Collapse
Affiliation(s)
- Annika Reinke
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
- German Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany.
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany.
| | - Minu D Tizabi
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany.
| | - Michael Baumgartner
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
| | - Matthias Eisenmann
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
| | - Doreen Heckmann-Nötzel
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany
| | - A Emre Kavur
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
- German Cancer Research Center (DKFZ) Heidelberg, HI Applied Computer Vision Lab, Heidelberg, Germany
| | - Tim Rädsch
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
- German Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany
| | - Carole H Sudre
- MRC Unit for Lifelong Health and Ageing at UCL and Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK
- School of Biomedical Engineering and Imaging Science, King's College London, London, UK
| | - Laura Acion
- Instituto de Cálculo, CONICET - Universidad de Buenos Aires, Buenos Aires, Argentina
| | - Michela Antonelli
- School of Biomedical Engineering and Imaging Science, King's College London, London, UK
- Centre for Medical Image Computing, University College London, London, UK
| | - Tal Arbel
- Centre for Intelligent Machines and MILA (Quebec Artificial Intelligence Institute), McGill University, Montréal, Quebec, Canada
| | - Spyridon Bakas
- Division of Computational Pathology, Dept of Pathology & Laboratory Medicine, Indiana University School of Medicine, Indianapolis, IN, USA
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
| | - Arriel Benis
- Department of Digital Medical Technologies, Holon Institute of Technology, Holon, Israel
- European Federation for Medical Informatics, Le Mont-sur-Lausanne, Switzerland
| | - Florian Buettner
- German Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Frankfurt am Main, Germany
- German Cancer Research Center (DKFZ) Heidelberg, Heidelberg, Germany
- Goethe University Frankfurt, Department of Medicine, Frankfurt am Main, Germany
- Goethe University Frankfurt, Department of Informatics, Frankfurt am Main, Germany
- Frankfurt Cancer Insititute, Frankfurt am Main, Germany
| | - M Jorge Cardoso
- School of Biomedical Engineering and Imaging Science, King's College London, London, UK
| | - Veronika Cheplygina
- Department of Computer Science, IT University of Copenhagen, Copenhagen, Denmark
| | - Jianxu Chen
- Leibniz-Institut für Analytische Wissenschaften - ISAS - e.V., Dortmund, Germany
| | - Evangelia Christodoulou
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
| | - Beth A Cimini
- Imaging Platform, Broad Institute of MIT and Harvard, Cambridge, MA, USA
| | - Keyvan Farahani
- Center for Biomedical Informatics and Information Technology, National Cancer Institute, Bethesda, MD, USA
| | - Luciana Ferrer
- Instituto de Investigación en Ciencias de la Computación (ICC), CONICET-UBA, Ciudad Autónoma de Buenos Aires, Buenos Aires, Argentina
| | - Adrian Galdran
- Universitat Pompeu Fabra, Barcelona, Spain
- University of Adelaide, Adelaide, South Australia, Australia
| | - Bram van Ginneken
- Fraunhofer MEVIS, Bremen, Germany
- Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Ben Glocker
- Department of Computing, Imperial College London, South Kensington Campus, London, UK
| | - Patrick Godau
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany
| | - Daniel A Hashimoto
- Department of Surgery, Perelman School of Medicine, Philadelphia, PA, USA
- General Robotics Automation Sensing and Perception Laboratory, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA
| | - Michael M Hoffman
- Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada
- Department of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada
| | - Merel Huisman
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Fabian Isensee
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
- German Cancer Research Center (DKFZ) Heidelberg, HI Applied Computer Vision Lab, Heidelberg, Germany
| | - Pierre Jannin
- Laboratoire Traitement du Signal et de l'Image - UMR_S 1099, Université de Rennes 1, Rennes, France
- INSERM, Paris, France
| | - Charles E Kahn
- Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, PA, USA
| | - Dagmar Kainmueller
- Max-Delbrück Center for Molecular Medicine in the Helmholtz Association (MDC), Biomedical Image Analysis and HI Helmholtz Imaging, Berlin, Germany
- University of Potsdam, Digital Engineering Faculty, Potsdam, Germany
| | - Bernhard Kainz
- Department of Computing, Faculty of Engineering, Imperial College London, London, UK
- Department AIBE, Friedrich-Alexander-Universität (FAU), Erlangen-Nürnberg, Germany
| | | | - Jens Kleesiek
- Translational Image-guided Oncology (TIO), Institute for AI in Medicine (IKIM), University Medicine Essen, Essen, Germany
| | | | | | - Annette Kopp-Schneider
- German Cancer Research Center (DKFZ) Heidelberg, Division of Biostatistics, Heidelberg, Germany
| | - Michal Kozubek
- Centre for Biomedical Image Analysis and Faculty of Informatics, Masaryk University, Brno, Czech Republic
| | - Anna Kreshuk
- Cell Biology and Biophysics Unit, European Molecular Biology Laboratory (EMBL), Heidelberg, Germany
| | - Tahsin Kurc
- Department of Biomedical Informatics, Stony Brook University, Health Science Center, Stony Brook, NY, USA
| | | | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Amin Madani
- Department of Surgery, University Health Network, Philadelphia, PA, USA
| | - Klaus Maier-Hein
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
- Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg, Germany
| | - Anne L Martel
- Department of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada
- Physical Sciences, Sunnybrook Research Institute, Toronto, Ontario, Canada
| | - Erik Meijering
- School of Computer Science and Engineering, University of New South Wales, UNSW Sydney, Kensington, New South Wales, Australia
| | - Bjoern Menze
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Henning Müller
- Information Systems Institute, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland
- Medical Faculty, University of Geneva, Geneva, Switzerland
| | - Brennan Nichyporuk
- MILA (Quebec Artificial Intelligence Institute), Montréal, Quebec, Canada
| | - Felix Nickel
- Department of General, Visceral and Thoracic Surgery, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Jens Petersen
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
| | | | - Nasir Rajpoot
- Tissue Image Analytics Laboratory, Department of Computer Science, University of Warwick, Coventry, UK
| | - Mauricio Reyes
- ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
- Department of Radiation Oncology, University Hospital Bern, University of Bern, Bern, Switzerland
| | - Michael A Riegler
- Simula Metropolitan Center for Digital Engineering, Oslo, Norway
- UiT The Arctic University of Norway, Tromsø, Norway
| | | | - Julio Saez-Rodriguez
- Institute for Computational Biomedicine, Heidelberg University, Heidelberg, Germany
- Faculty of Medicine, Heidelberg University Hospital, Heidelberg, Germany
| | - Clara I Sánchez
- Informatics Institute, Faculty of Science, University of Amsterdam, Amsterdam, the Netherlands
| | | | - Ronald M Summers
- National Institutes of Health Clinical Center, Bethesda, MD, USA
| | - Abdel A Taha
- Institute of Information Systems Engineering, TU Wien, Vienna, Austria
| | - Aleksei Tiulpin
- Research Unit of Health Sciences and Technology, Faculty of Medicine, University of Oulu, Oulu, Finland
- Neurocenter Oulu, Oulu University Hospital, Oulu, Finland
| | | | - Ben Van Calster
- Department of Development and Regeneration and EPI-centre, KU Leuven, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
| | - Gaël Varoquaux
- Parietal project team, INRIA Saclay-Île de France, Palaiseau, France
| | - Ziv R Yaniv
- National Institute of Allergy and Infectious Diseases, Bethesda, MD, USA
| | - Paul F Jäger
- German Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany.
- German Cancer Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group, Heidelberg, Germany.
| | - Lena Maier-Hein
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
- German Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany.
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany.
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany.
- Faculty of Medicine, Heidelberg University Hospital, Heidelberg, Germany.
| |
Collapse
|
6
|
Maier-Hein L, Reinke A, Godau P, Tizabi MD, Buettner F, Christodoulou E, Glocker B, Isensee F, Kleesiek J, Kozubek M, Reyes M, Riegler MA, Wiesenfarth M, Kavur AE, Sudre CH, Baumgartner M, Eisenmann M, Heckmann-Nötzel D, Rädsch T, Acion L, Antonelli M, Arbel T, Bakas S, Benis A, Blaschko MB, Cardoso MJ, Cheplygina V, Cimini BA, Collins GS, Farahani K, Ferrer L, Galdran A, van Ginneken B, Haase R, Hashimoto DA, Hoffman MM, Huisman M, Jannin P, Kahn CE, Kainmueller D, Kainz B, Karargyris A, Karthikesalingam A, Kofler F, Kopp-Schneider A, Kreshuk A, Kurc T, Landman BA, Litjens G, Madani A, Maier-Hein K, Martel AL, Mattson P, Meijering E, Menze B, Moons KGM, Müller H, Nichyporuk B, Nickel F, Petersen J, Rajpoot N, Rieke N, Saez-Rodriguez J, Sánchez CI, Shetty S, van Smeden M, Summers RM, Taha AA, Tiulpin A, Tsaftaris SA, Van Calster B, Varoquaux G, Jäger PF. Metrics reloaded: recommendations for image analysis validation. Nat Methods 2024; 21:195-212. [PMID: 38347141 DOI: 10.1038/s41592-023-02151-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2023] [Accepted: 12/12/2023] [Indexed: 02/15/2024]
Abstract
Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.
Collapse
Affiliation(s)
- Lena Maier-Hein
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
- German Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany.
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany.
- Medical Faculty, Heidelberg University, Heidelberg, Germany.
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany.
| | - Annika Reinke
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
- German Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany.
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany.
| | - Patrick Godau
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany
| | - Minu D Tizabi
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany
| | - Florian Buettner
- German Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Frankfurt am Main, Germany
- German Cancer Research Center (DKFZ) Heidelberg, Heidelberg, Germany
- Department of Medicine, Goethe University Frankfurt, Frankfurt am Main, Germany
- Department of Informatics, Goethe University Frankfurt, Frankfurt am Main, Germany
- Frankfurt Cancer Insititute, Frankfurt am Main, Germany
| | - Evangelia Christodoulou
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
| | - Ben Glocker
- Department of Computing, Imperial College London, South Kensington Campus, London, UK
| | - Fabian Isensee
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
- German Cancer Research Center (DKFZ) Heidelberg, HI Applied Computer Vision Lab, Heidelberg, Germany
| | - Jens Kleesiek
- Institute for AI in Medicine, University Medicine Essen, Essen, Germany
| | - Michal Kozubek
- Centre for Biomedical Image Analysis and Faculty of Informatics, Masaryk University, Brno, Czech Republic
| | - Mauricio Reyes
- ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
- Department of Radiation Oncology, University Hospital Bern, University of Bern, Bern, Switzerland
| | - Michael A Riegler
- Simula Metropolitan Center for Digital Engineering, Oslo, Norway
- Department of Computer Science, UiT The Arctic University of Norway, Tromsø, Norway
| | - Manuel Wiesenfarth
- German Cancer Research Center (DKFZ) Heidelberg, Division of Biostatistics, Heidelberg, Germany
| | - A Emre Kavur
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
- German Cancer Research Center (DKFZ) Heidelberg, HI Applied Computer Vision Lab, Heidelberg, Germany
| | - Carole H Sudre
- MRC Unit for Lifelong Health and Ageing at UCL and Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK
- School of Biomedical Engineering and Imaging Science, King's College London, London, UK
| | - Michael Baumgartner
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
| | - Matthias Eisenmann
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
| | - Doreen Heckmann-Nötzel
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany
| | - Tim Rädsch
- German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany
- German Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany
| | - Laura Acion
- Instituto de Cálculo, CONICET - Universidad de Buenos Aires, Buenos Aires, Argentina
| | - Michela Antonelli
- School of Biomedical Engineering and Imaging Science, King's College London, London, UK
- Centre for Medical Image Computing, University College London, London, UK
| | - Tal Arbel
- Centre for Intelligent Machines and MILA (Québec Artificial Intelligence Institute), McGill University, Montréal, Quebec, Canada
| | - Spyridon Bakas
- Division of Computational Pathology, Department of Pathology & Laboratory Medicine, Indiana University School of Medicine, IU Health Information and Translational Sciences Building, Indianapolis, IN, USA
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
| | - Arriel Benis
- Department of Digital Medical Technologies, Holon Institute of Technology, Holon, Israel
- European Federation for Medical Informatics, Le Mont-sur-Lausanne, Switzerland
| | - Matthew B Blaschko
- Center for Processing Speech and Images, Department of Electrical Engineering, KU Leuven, Leuven, Belgium
| | - M Jorge Cardoso
- School of Biomedical Engineering and Imaging Science, King's College London, London, UK
| | - Veronika Cheplygina
- Department of Computer Science, IT University of Copenhagen, Copenhagen, Denmark
| | - Beth A Cimini
- Imaging Platform, Broad Institute of MIT and Harvard, Cambridge, MA, USA
| | - Gary S Collins
- Centre for Statistics in Medicine, University of Oxford, Nuffield Orthopaedic Centre, Oxford, UK
| | - Keyvan Farahani
- Center for Biomedical Informatics and Information Technology, National Cancer Institute, Bethesda, MD, USA
| | - Luciana Ferrer
- Instituto de Investigación en Ciencias de la Computación (ICC), CONICET-UBA, Ciudad Autónoma de Buenos Aires, Buenos Aires, Argentina
| | - Adrian Galdran
- BCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain
- Australian Institute for Machine Learning AIML, University of Adelaide, Adelaide, South Australia, Australia
| | - Bram van Ginneken
- Fraunhofer MEVIS, Bremen, Germany
- Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Robert Haase
- Technische Universität (TU) Dresden, DFG Cluster of Excellence 'Physics of Life', Dresden, Germany
- Center for Systems Biology, Dresden, Germany
- Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), Leipzig University, Leipzig, Germany
| | - Daniel A Hashimoto
- Department of Surgery, Perelman School of Medicine, Philadelphia, PA, USA
- General Robotics Automation Sensing and Perception Laboratory, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA
| | - Michael M Hoffman
- Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada
- Department of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada
- Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
| | - Merel Huisman
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Pierre Jannin
- Laboratoire Traitement du Signal et de l'Image - UMR_S 1099, Université de Rennes 1, Rennes, France
- INSERM, Paris, France
| | - Charles E Kahn
- Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, PA, USA
| | - Dagmar Kainmueller
- Max-Delbrück Center for Molecular Medicine in the Helmholtz Association (MDC), Biomedical Image Analysis and HI Helmholtz Imaging, Berlin, Germany
- Digital Engineering Faculty, University of Potsdam, Potsdam, Germany
| | - Bernhard Kainz
- Department of Computing, Faculty of Engineering, Imperial College London, London, UK
- Department AIBE, Friedrich-Alexander-Universität (FAU), Erlangen-Nürnberg, Germany
| | | | | | | | - Annette Kopp-Schneider
- German Cancer Research Center (DKFZ) Heidelberg, Division of Biostatistics, Heidelberg, Germany
| | - Anna Kreshuk
- Cell Biology and Biophysics Unit, European Molecular Biology Laboratory (EMBL), Heidelberg, Germany
| | - Tahsin Kurc
- Department of Biomedical Informatics, Stony Brook University, Health Science Center, Stony Brook, NY, USA
| | | | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Amin Madani
- Department of Surgery, University Health Network, Philadelphia, PA, USA
| | - Klaus Maier-Hein
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
- Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg, Germany
| | - Anne L Martel
- Department of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- Physical Sciences, Sunnybrook Research Institute, Toronto, Ontario, Canada
| | - Peter Mattson
- Google, 1600 Amphitheatre Pkwy, Mountain View, CA, USA
| | - Erik Meijering
- School of Computer Science and Engineering, University of New South Wales, UNSW Sydney, Kensington, New South Wales, Australia
| | - Bjoern Menze
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Henning Müller
- Information Systems Institute, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland
- Medical Faculty, University of Geneva, Geneva, Switzerland
| | - Brennan Nichyporuk
- MILA (Québec Artificial Intelligence Institute), Montréal, Quebec, Canada
| | - Felix Nickel
- Department of General, Visceral and Thoracic Surgery, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Jens Petersen
- German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany
| | - Nasir Rajpoot
- Tissue Image Analytics Laboratory, Department of Computer Science, University of Warwick, Coventry, UK
| | | | - Julio Saez-Rodriguez
- Institute for Computational Biomedicine, Heidelberg University, Heidelberg, Germany
- Faculty of Medicine, Heidelberg University Hospital, Heidelberg, Germany
| | - Clara I Sánchez
- Informatics Institute, Faculty of Science, University of Amsterdam, Amsterdam, the Netherlands
| | | | - Maarten van Smeden
- Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Ronald M Summers
- National Institutes of Health Clinical Center, Bethesda, MD, USA
| | - Abdel A Taha
- Institute of Information Systems Engineering, TU Wien, Vienna, Austria
| | - Aleksei Tiulpin
- Research Unit of Health Sciences and Technology, Faculty of Medicine, University of Oulu, Oulu, Finland
- Neurocenter Oulu, Oulu University Hospital, Oulu, Finland
| | | | - Ben Van Calster
- Department of Development and Regeneration and EPI-centre, KU Leuven, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
| | - Gaël Varoquaux
- Parietal project team, INRIA Saclay-Île de France, Palaiseau, France
| | - Paul F Jäger
- German Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany.
- German Cancer Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group, Heidelberg, Germany.
| |
Collapse
|
7
|
Schouten D, van der Laak J, van Ginneken B, Litjens G. Full resolution reconstruction of whole-mount sections from digitized individual tissue fragments. Sci Rep 2024; 14:1497. [PMID: 38233535 PMCID: PMC10794243 DOI: 10.1038/s41598-024-52007-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Accepted: 01/12/2024] [Indexed: 01/19/2024] Open
Abstract
Whole-mount sectioning is a technique in histopathology where a full slice of tissue, such as a transversal cross-section of a prostate specimen, is prepared on a large microscope slide without further sectioning into smaller fragments. Although this technique can offer improved correlation with pre-operative imaging and is paramount for multimodal research, it is not commonly employed due to its technical difficulty, associated cost and cumbersome integration in (digital) pathology workflows. In this work, we present a computational tool named PythoStitcher which reconstructs artificial whole-mount sections from digitized tissue fragments, thereby bringing the benefits of whole-mount sections to pathology labs currently unable to employ this technique. Our proposed algorithm consists of a multi-step approach where it (i) automatically determines how fragments need to be reassembled, (ii) iteratively optimizes the stitch using a genetic algorithm and (iii) efficiently reconstructs the final artificial whole-mount section on full resolution (0.25 µm/pixel). PythoStitcher was validated on a total of 198 cases spanning five datasets with a varying number of tissue fragments originating from different organs from multiple centers. PythoStitcher successfully reconstructed the whole-mount section in 86-100% of cases for a given dataset with a residual registration mismatch of 0.65-2.76 mm on automatically selected landmarks. It is expected that our algorithm can aid pathology labs unable to employ whole-mount sectioning through faster clinical case evaluation and improved radiology-pathology correlation workflows.
Collapse
Affiliation(s)
- Daan Schouten
- Department of Pathology, Radboud University Medical Centre, Nijmegen, The Netherlands.
| | - Jeroen van der Laak
- Department of Pathology, Radboud University Medical Centre, Nijmegen, The Netherlands
| | - Bram van Ginneken
- Department of Radiology, Radboud University Medical Centre, Nijmegen, The Netherlands
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Centre, Nijmegen, The Netherlands
| |
Collapse
|
8
|
Linmans J, Hoogeboom E, van der Laak J, Litjens G. The Latent Doctor Model for Modeling Inter-Observer Variability. IEEE J Biomed Health Inform 2023; PP:1-12. [PMID: 37831571 DOI: 10.1109/jbhi.2023.3323582] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/15/2023]
Abstract
Many inherently ambiguous tasks in medical imaging suffer from inter-observer variability, resulting in a reference standard defined by a distribution of labels with high variance. Training only on a consensus or majority vote label, as is common in medical imaging, discards valuable information on uncertainty amongst a panel of experts. In this work, we propose to train on the full label distribution to predict the uncertainty within a panel of experts and the most likely ground-truth label. To do so, we propose a new stochastic classification framework based on the conditional variational auto-encoder, which we refer to as the Latent Doctor Model (LDM). In an extensive comparative analysis, we compare the LDM with a model trained on the majority vote label and other methods capable of learning a distribution of labels. We show that the LDM is able to reproduce the reference-standard distribution significantly better than the majority vote baseline. Compared to the other baseline methods, we demonstrate that the LDM performs best at modeling the label distribution and its corresponding uncertainty in two prostate tumor grading tasks. Furthermore, we show competitive performance of the LDM with the more computationally demanding deep ensembles on a tumor budding classification task.
Collapse
|
9
|
Laarhuis BI, Janssen MJR, Simons M, van Kalmthout LWM, van der Doelen MJ, Peters SMB, Westdorp H, van Oort IM, Litjens G, Gotthardt M, Nagarajah J, Mehra N, Privé BM. Tumoral Ki67 and PSMA Expression in Fresh Pre-PSMA-RLT Biopsies and Its Relation With PSMA-PET Imaging and Outcomes of PSMA-RLT in Patients With mCRPC. Clin Genitourin Cancer 2023; 21:e352-e361. [PMID: 37164814 DOI: 10.1016/j.clgc.2023.04.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2022] [Accepted: 04/13/2023] [Indexed: 05/12/2023]
Abstract
INTRODUCTION Prostate specific membrane antigen (PSMA) directed radioligand therapy (RLT) is a novel therapy for metastatic castration-resistant prostate cancer (mCRPC) patients. However, it is still poorly understood why approximately 40% of the patients does not respond to PSMA-RLT. The aims of this study were to evaluate the pretreatment PSMA expression on immunohistochemistry (IHC) and PSMA uptake on PET/CT imaging in mCRPC patients who underwent PSMA-RLT. We correlated these parameters and a cell proliferation marker (Ki67) to the therapeutic efficacy of PSMA-RLT. PATIENTS AND METHODS In this retrospective study, mCRPC patients who underwent PSMA-RLT were analyzed. Patients biopsies were scored for immunohistochemical Ki67 expression, PSMA staining intensity and percentage of cells with PSMA expression. Moreover, the PSMA tracer uptake of the tumor lesion(s) and healthy organs on PET/CT imaging was assessed. The primary outcome was to evaluate the association between histological PSMA protein expression of tumor in pre-PSMA-RLT biopsies and the PSMA uptake on PSMA PET/CT imaging of the biopsied lesion. Secondary outcomes were to assess the relationship between PSMA expression and Ki67 on IHC and the progression free survival (PFS) and overall survival (OS) following PSMA-RLT. RESULTS In total, 22 mCRPC patients were included in this study. Nineteen (86%) patients showed a high and homogenous PSMA expression of >80% on IHC. Three (14%) patients had low PSMA expression on IHC. Although there was limited PSMA uptake on PET/CT imaging, these 3 patients had lower PSMA uptake on PET/CT imaging compared to the patients with high PSMA expression on IHC. Yet, no correlation was found between PSMA uptake on PET/CT imaging and PSMA expression on IHC (SUVmax: R2 = 0.046 and SUVavg: R2 = 0.036). The 3 patients had a shorter PFS compared to the patients with high PSMA expression on IHC (HR: 4.76, 95% CI: 1.14-19.99; P = .033). Patients with low Ki67 expression had a longer PFS and OS compared to patients with a high Ki67 expression (HR: 0.40, 95% CI: 0.15-1.06; P = .013) CONCLUSION: The PSMA uptake on PSMA-PET/CT generally followed the PSMA expression on IHC. However, heterogeneity may be missed on PSMA-PET/CT. Immunohistochemical PSMA and Ki67 expression in fresh tumor biopsies, may contribute to predict treatment efficacy of PSMA-RLT in mCRPC patients. This needs to be further explored in prospective cohorts.
Collapse
Affiliation(s)
- Babette I Laarhuis
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands; Department of Medical Oncology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Marcel J R Janssen
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Michiel Simons
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | | | - Maarten J van der Doelen
- Department of Medical Oncology, Radboud University Medical Center, Nijmegen, The Netherlands; Department of Urology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Steffie M B Peters
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Harm Westdorp
- Department of Medical Oncology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Inge M van Oort
- Department of Urology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Martin Gotthardt
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - James Nagarajah
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Niven Mehra
- Department of Medical Oncology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Bastiaan M Privé
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands.
| |
Collapse
|
10
|
Dooper S, Pinckaers H, Aswolinskiy W, Hebeda K, Jarkman S, van der Laak J, Litjens G. Gigapixel end-to-end training using streaming and attention. Med Image Anal 2023; 88:102881. [PMID: 37437452 DOI: 10.1016/j.media.2023.102881] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Revised: 05/04/2023] [Accepted: 06/22/2023] [Indexed: 07/14/2023]
Abstract
Current hardware limitations make it impossible to train convolutional neural networks on gigapixel image inputs directly. Recent developments in weakly supervised learning, such as attention-gated multiple instance learning, have shown promising results, but often use multi-stage or patch-wise training strategies risking suboptimal feature extraction, which can negatively impact performance. In this paper, we propose to train a ResNet-34 encoder with an attention-gated classification head in an end-to-end fashion, which we call StreamingCLAM, using a streaming implementation of convolutional layers. This allows us to train end-to-end on 4-gigapixel microscopic images using only slide-level labels. We achieve a mean area under the receiver operating characteristic curve of 0.9757 for metastatic breast cancer detection (CAMELYON16), close to fully supervised approaches using pixel-level annotations. Our model can also detect MYC-gene translocation in histologic slides of diffuse large B-cell lymphoma, achieving a mean area under the ROC curve of 0.8259. Furthermore, we show that our model offers a degree of interpretability through the attention mechanism.
Collapse
Affiliation(s)
- Stephan Dooper
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands.
| | - Hans Pinckaers
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands
| | - Witali Aswolinskiy
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands
| | - Konnie Hebeda
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands
| | - Sofia Jarkman
- Department of Clinical Pathology, and Department of Biomedical and Clinical Sciences, Linköping University, Linköping 581 83, Sweden; Center for Medical Image Science and Visualization, Linköping University, Linköping 581 85, Sweden
| | - Jeroen van der Laak
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands; Center for Medical Image Science and Visualization, Linköping University, Linköping 581 85, Sweden
| | - Geert Litjens
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands
| |
Collapse
|
11
|
Bándi P, Balkenhol M, van Dijk M, Kok M, van Ginneken B, van der Laak J, Litjens G. Continual learning strategies for cancer-independent detection of lymph node metastases. Med Image Anal 2023; 85:102755. [PMID: 36724605 DOI: 10.1016/j.media.2023.102755] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2021] [Revised: 01/08/2023] [Accepted: 01/18/2023] [Indexed: 01/26/2023]
Abstract
Recently, large, high-quality public datasets have led to the development of convolutional neural networks that can detect lymph node metastases of breast cancer at the level of expert pathologists. Many cancers, regardless of the site of origin, can metastasize to lymph nodes. However, collecting and annotating high-volume, high-quality datasets for every cancer type is challenging. In this paper we investigate how to leverage existing high-quality datasets most efficiently in multi-task settings for closely related tasks. Specifically, we will explore different training and domain adaptation strategies, including prevention of catastrophic forgetting, for breast, colon and head-and-neck cancer metastasis detection in lymph nodes. Our results show state-of-the-art performance on colon and head-and-neck cancer metastasis detection tasks. We show the effectiveness of adaptation of networks from one cancer type to another to obtain multi-task metastasis detection networks. Furthermore, we show that leveraging existing high-quality datasets can significantly boost performance on new target tasks and that catastrophic forgetting can be effectively mitigated.Last, we compare different mitigation strategies.
Collapse
|
12
|
van Eekelen L, Litjens G, Hebeda KM. Artificial Intelligence in Bone Marrow Histological Diagnostics: Potential Applications and Challenges. Pathobiology 2023; 91:8-17. [PMID: 36791682 PMCID: PMC10937040 DOI: 10.1159/000529701] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2022] [Accepted: 02/13/2023] [Indexed: 02/17/2023] Open
Abstract
The expanding digitalization of routine diagnostic histological slides holds a potential to apply artificial intelligence (AI) to pathology, including bone marrow (BM) histology. In this perspective, we describe potential tasks in diagnostics that can be supported, investigations that can be guided, and questions that can be answered by the future application of AI on whole-slide images of BM biopsies. These range from characterization of cell lineages and quantification of cells and stromal structures to disease prediction. First glimpses show an exciting potential to detect subtle phenotypic changes with AI that are due to specific genotypes. The discussion is illustrated by examples of current AI research using BM biopsy slides. In addition, we briefly discuss current challenges for implementation of AI-supported diagnostics.
Collapse
Affiliation(s)
- Leander van Eekelen
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
- Computational Pathology Group, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
- Computational Pathology Group, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Konnie M. Hebeda
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
13
|
Linmans J, Elfwing S, van der Laak J, Litjens G. Predictive uncertainty estimation for out-of-distribution detection in digital pathology. Med Image Anal 2023; 83:102655. [PMID: 36306568 DOI: 10.1016/j.media.2022.102655] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2022] [Revised: 09/26/2022] [Accepted: 10/12/2022] [Indexed: 11/06/2022]
Abstract
Machine learning model deployment in clinical practice demands real-time risk assessment to identify situations in which the model is uncertain. Once deployed, models should be accurate for classes seen during training while providing informative estimates of uncertainty to flag abnormalities and unseen classes for further analysis. Although recent developments in uncertainty estimation have resulted in an increasing number of methods, a rigorous empirical evaluation of their performance on large-scale digital pathology datasets is lacking. This work provides a benchmark for evaluating prevalent methods on multiple datasets by comparing the uncertainty estimates on both in-distribution and realistic near and far out-of-distribution (OOD) data on a whole-slide level. To this end, we aggregate uncertainty values from patch-based classifiers to whole-slide level uncertainty scores. We show that results found in classical computer vision benchmarks do not always translate to the medical imaging setting. Specifically, we demonstrate that deep ensembles perform best at detecting far-OOD data but can be outperformed on a more challenging near-OOD detection task by multi-head ensembles trained for optimal ensemble diversity. Furthermore, we demonstrate the harmful impact OOD data can have on the performance of deployed machine learning models. Overall, we show that uncertainty estimates can be used to discriminate in-distribution from OOD data with high AUC scores. Still, model deployment might require careful tuning based on prior knowledge of prospective OOD data.
Collapse
Affiliation(s)
- Jasper Linmans
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands.
| | | | - Jeroen van der Laak
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands; Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | - Geert Litjens
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
14
|
Jarkman S, Karlberg M, Pocevičiūtė M, Bodén A, Bándi P, Litjens G, Lundström C, Treanor D, van der Laak J. Generalization of Deep Learning in Digital Pathology: Experience in Breast Cancer Metastasis Detection. Cancers (Basel) 2022; 14:cancers14215424. [PMID: 36358842 PMCID: PMC9659028 DOI: 10.3390/cancers14215424] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2022] [Revised: 10/13/2022] [Accepted: 10/28/2022] [Indexed: 11/06/2022] Open
Abstract
Simple Summary Pathology is a cornerstone in cancer diagnostics, and digital pathology and artificial intelligence-driven image analysis could potentially save time and enhance diagnostic accuracy. For clinical implementation of artificial intelligence, a major question is whether the computer models maintain high performance when applied to new settings. We tested the generalizability of a highly accurate deep learning model for breast cancer metastasis detection in sentinel lymph nodes from, firstly, unseen sentinel node data and, secondly, data with a small change in surgical indication, in this case lymph nodes from axillary dissections. Model performance dropped in both settings, particularly on axillary dissection nodes. Retraining of the model was needed to mitigate the performance drop. The study highlights the generalization challenge of clinical implementation of AI models, and the possibility that retraining might be necessary. Abstract Poor generalizability is a major barrier to clinical implementation of artificial intelligence in digital pathology. The aim of this study was to test the generalizability of a pretrained deep learning model to a new diagnostic setting and to a small change in surgical indication. A deep learning model for breast cancer metastases detection in sentinel lymph nodes, trained on CAMELYON multicenter data, was used as a base model, and achieved an AUC of 0.969 (95% CI 0.926–0.998) and FROC of 0.838 (95% CI 0.757–0.913) on CAMELYON16 test data. On local sentinel node data, the base model performance dropped to AUC 0.929 (95% CI 0.800–0.998) and FROC 0.744 (95% CI 0.566–0.912). On data with a change in surgical indication (axillary dissections) the base model performance indicated an even larger drop with a FROC of 0.503 (95%CI 0.201–0.911). The model was retrained with addition of local data, resulting in about a 4% increase for both AUC and FROC for sentinel nodes, and an increase of 11% in AUC and 49% in FROC for axillary nodes. Pathologist qualitative evaluation of the retrained model´s output showed no missed positive slides. False positives, false negatives and one previously undetected micro-metastasis were observed. The study highlights the generalization challenge even when using a multicenter trained model, and that a small change in indication can considerably impact the model´s performance.
Collapse
Affiliation(s)
- Sofia Jarkman
- Department of Clinical Pathology, and Department of Biomedical and Clinical Sciences, Linköping University, 581 83 Linköping, Sweden
- Center for Medical Image Science and Visualization (CMIV), Linköping University, 581 85 Linköping, Sweden
- Correspondence:
| | - Micael Karlberg
- Center for Medical Image Science and Visualization (CMIV), Linköping University, 581 85 Linköping, Sweden
- Department of Pathology, Radboud University Medical Center, P.O. Box 9101, 6500 HB Nijmegen, The Netherlands
| | - Milda Pocevičiūtė
- Center for Medical Image Science and Visualization (CMIV), Linköping University, 581 85 Linköping, Sweden
| | - Anna Bodén
- Department of Clinical Pathology, and Department of Biomedical and Clinical Sciences, Linköping University, 581 83 Linköping, Sweden
- Center for Medical Image Science and Visualization (CMIV), Linköping University, 581 85 Linköping, Sweden
| | - Péter Bándi
- Department of Pathology, Radboud University Medical Center, P.O. Box 9101, 6500 HB Nijmegen, The Netherlands
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, P.O. Box 9101, 6500 HB Nijmegen, The Netherlands
| | - Claes Lundström
- Center for Medical Image Science and Visualization (CMIV), Linköping University, 581 85 Linköping, Sweden
- Sectra AB, Teknikringen 20, 583 30 Linköping, Sweden
| | - Darren Treanor
- Department of Clinical Pathology, and Department of Biomedical and Clinical Sciences, Linköping University, 581 83 Linköping, Sweden
- Center for Medical Image Science and Visualization (CMIV), Linköping University, 581 85 Linköping, Sweden
- Leeds Teaching Hospitals NHS Trust, St James´s University Hospital, Beckett Street, Leeds LS9 7TF, UK
- Department of Pathology, University of Leeds, Woodhouse Lane, Leeds LS2 9JT, UK
| | - Jeroen van der Laak
- Center for Medical Image Science and Visualization (CMIV), Linköping University, 581 85 Linköping, Sweden
- Department of Pathology, Radboud University Medical Center, P.O. Box 9101, 6500 HB Nijmegen, The Netherlands
| |
Collapse
|
15
|
Adams LC, Makowski MR, Engel G, Rattunde M, Busch F, Asbach P, Niehues SM, Vinayahalingam S, van Ginneken B, Litjens G, Bressem KK. Dataset of prostate MRI annotated for anatomical zones and cancer. Data Brief 2022; 45:108739. [DOI: 10.1016/j.dib.2022.108739] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2022] [Revised: 11/03/2022] [Accepted: 11/04/2022] [Indexed: 11/11/2022] Open
|
16
|
Antonelli M, Reinke A, Bakas S, Farahani K, Kopp-Schneider A, Landman BA, Litjens G, Menze B, Ronneberger O, Summers RM, van Ginneken B, Bilello M, Bilic P, Christ PF, Do RKG, Gollub MJ, Heckers SH, Huisman H, Jarnagin WR, McHugo MK, Napel S, Pernicka JSG, Rhode K, Tobon-Gomez C, Vorontsov E, Meakin JA, Ourselin S, Wiesenfarth M, Arbeláez P, Bae B, Chen S, Daza L, Feng J, He B, Isensee F, Ji Y, Jia F, Kim I, Maier-Hein K, Merhof D, Pai A, Park B, Perslev M, Rezaiifar R, Rippel O, Sarasua I, Shen W, Son J, Wachinger C, Wang L, Wang Y, Xia Y, Xu D, Xu Z, Zheng Y, Simpson AL, Maier-Hein L, Cardoso MJ. The Medical Segmentation Decathlon. Nat Commun 2022; 13:4128. [PMID: 35840566 PMCID: PMC9287542 DOI: 10.1038/s41467-022-30695-9] [Citation(s) in RCA: 115] [Impact Index Per Article: 57.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2021] [Accepted: 05/13/2022] [Indexed: 02/05/2023] Open
Abstract
International challenges have become the de facto standard for comparative assessment of image analysis algorithms. Although segmentation is the most widely investigated medical image processing task, the various challenges have been organized to focus only on specific clinical tasks. We organized the Medical Segmentation Decathlon (MSD)-a biomedical image analysis challenge, in which algorithms compete in a multitude of both tasks and modalities to investigate the hypothesis that a method capable of performing well on multiple tasks will generalize well to a previously unseen task and potentially outperform a custom-designed solution. MSD results confirmed this hypothesis, moreover, MSD winner continued generalizing well to a wide range of other clinical problems for the next two years. Three main conclusions can be drawn from this study: (1) state-of-the-art image segmentation algorithms generalize well when retrained on unseen tasks; (2) consistent algorithmic performance across multiple tasks is a strong surrogate of algorithmic generalizability; (3) the training of accurate AI segmentation models is now commoditized to scientists that are not versed in AI model training.
Collapse
Affiliation(s)
- Michela Antonelli
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, UK.
| | - Annika Reinke
- Div. Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany.,HI Helmholtz Imaging, German Cancer Research Center (DKFZ), Heidelberg, Germany.,Faculty of Mathematics and Computer Science, University of Heidelberg, Heidelberg, Germany
| | - Spyridon Bakas
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA.,Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.,Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Keyvan Farahani
- Center for Biomedical Informatics and Information Technology, National Cancer Institute (NIH), Bethesda, MD, USA
| | | | - Bennett A Landman
- Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN, USA
| | - Geert Litjens
- Radboud University Medical Center, Radboud Institute for Health Sciences, Nijmegen, The Netherlands
| | - Bjoern Menze
- Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
| | | | - Ronald M Summers
- Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Department of Radiology and Imaging Sciences, National Institutes of Health Clinical Center (NIH), Bethesda, MD, USA
| | - Bram van Ginneken
- Radboud University Medical Center, Radboud Institute for Health Sciences, Nijmegen, The Netherlands
| | - Michel Bilello
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
| | - Patrick Bilic
- Department of Informatics, Technische Universität München, München, Germany
| | - Patrick F Christ
- Department of Informatics, Technische Universität München, München, Germany
| | - Richard K G Do
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Marc J Gollub
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Stephan H Heckers
- Department of Psychiatry & Behavioral Sciences, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Henkjan Huisman
- Radboud University Medical Center, Radboud Institute for Health Sciences, Nijmegen, The Netherlands
| | - William R Jarnagin
- Department of Surgery, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Maureen K McHugo
- Department of Psychiatry & Behavioral Sciences, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Sandy Napel
- Department of Radiology, Stanford University, Stanford, CA, USA
| | | | - Kawal Rhode
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, UK
| | - Catalina Tobon-Gomez
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, UK
| | - Eugene Vorontsov
- Department of Computer Science and Software Engineering, École Polytechnique de Montréal, Montréal, QC, Canada
| | - James A Meakin
- Radboud University Medical Center, Radboud Institute for Health Sciences, Nijmegen, The Netherlands
| | - Sebastien Ourselin
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, UK
| | - Manuel Wiesenfarth
- Div. Biostatistics, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | | | | | | | - Laura Daza
- Universidad de los Andes, Bogota, Colombia
| | - Jianjiang Feng
- Department of Automation, Tsinghua University, Beijing, China
| | - Baochun He
- Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Fabian Isensee
- HI Applied Computer Vision Lab, Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Yuanfeng Ji
- Department of Computer Science, Xiamen University, Xiamen, China
| | - Fucang Jia
- Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Ildoo Kim
- Kakao Brain, Seongnam-si, Republic of Korea
| | - Klaus Maier-Hein
- Cerebriu A/S, Copenhagen, Denmark.,Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg, Germany
| | - Dorit Merhof
- Institute of Imaging & Computer Vision, RWTH Aachen University, Aachen, Germany.,Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
| | - Akshay Pai
- Cerebriu A/S, Copenhagen, Denmark.,Department of Computer Science, University of Copenhagen, Copenhagen, Denmark
| | | | - Mathias Perslev
- Department of Computer Science, University of Copenhagen, Copenhagen, Denmark
| | | | - Oliver Rippel
- Institute of Imaging & Computer Vision, RWTH Aachen University, Aachen, Germany
| | - Ignacio Sarasua
- Lab for Artificial Intelligence in Medical Imaging (AI-Med), Department of Child and Adolescent Psychiatry, University Hospital, LMU München, Germany
| | - Wei Shen
- MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China
| | | | - Christian Wachinger
- Lab for Artificial Intelligence in Medical Imaging (AI-Med), Department of Child and Adolescent Psychiatry, University Hospital, LMU München, Germany
| | - Liansheng Wang
- Department of Computer Science, Xiamen University, Xiamen, China
| | - Yan Wang
- Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, China
| | - Yingda Xia
- Johns Hopkins University, Baltimore, MD, USA
| | | | - Zhanwei Xu
- Department of Automation, Tsinghua University, Beijing, China
| | | | - Amber L Simpson
- School of Computing/Department of Biomedical and Molecular Sciences, Queen's University, Kingston, ON, Canada
| | - Lena Maier-Hein
- Div. Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany.,HI Helmholtz Imaging, German Cancer Research Center (DKFZ), Heidelberg, Germany.,Faculty of Mathematics and Computer Science, University of Heidelberg, Heidelberg, Germany.,Medical Faculty, University of Heidelberg, Heidelberg, Germany
| | - M Jorge Cardoso
- School of Biomedical Engineering & Imaging Sciences, King's College London, London, UK
| |
Collapse
|
17
|
Adams LC, Makowski MR, Engel G, Rattunde M, Busch F, Asbach P, Niehues SM, Vinayahalingam S, van Ginneken B, Litjens G, Bressem KK. Prostate158 - An expert-annotated 3T MRI dataset and algorithm for prostate cancer detection. Comput Biol Med 2022; 148:105817. [PMID: 35841780 DOI: 10.1016/j.compbiomed.2022.105817] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2022] [Revised: 06/12/2022] [Accepted: 07/03/2022] [Indexed: 11/03/2022]
Abstract
BACKGROUND The development of deep learning (DL) models for prostate segmentation on magnetic resonance imaging (MRI) depends on expert-annotated data and reliable baselines, which are often not publicly available. This limits both reproducibility and comparability. METHODS Prostate158 consists of 158 expert annotated biparametric 3T prostate MRIs comprising T2w sequences and diffusion-weighted sequences with apparent diffusion coefficient maps. Two U-ResNets trained for segmentation of anatomy (central gland, peripheral zone) and suspicious lesions for prostate cancer (PCa) with a PI-RADS score of ≥4 served as baseline algorithms. Segmentation performance was evaluated using the Dice similarity coefficient (DSC), the Hausdorff distance (HD), and the average surface distance (ASD). The Wilcoxon test with Bonferroni correction was used to evaluate differences in performance. The generalizability of the baseline model was assessed using the open datasets Medical Segmentation Decathlon and PROSTATEx. RESULTS Compared to Reader 1, the models achieved a DSC/HD/ASD of 0.88/18.3/2.2 for the central gland, 0.75/22.8/1.9 for the peripheral zone, and 0.45/36.7/17.4 for PCa. Compared with Reader 2, the DSC/HD/ASD were 0.88/17.5/2.6 for the central gland, 0.73/33.2/1.9 for the peripheral zone, and 0.4/39.5/19.1 for PCa. Interrater agreement measured in DSC/HD/ASD was 0.87/11.1/1.0 for the central gland, 0.75/15.8/0.74 for the peripheral zone, and 0.6/18.8/5.5 for PCa. Segmentation performances on the Medical Segmentation Decathlon and PROSTATEx were 0.82/22.5/3.4; 0.86/18.6/2.5 for the central gland, and 0.64/29.2/4.7; 0.71/26.3/2.2 for the peripheral zone. CONCLUSIONS We provide an openly accessible, expert-annotated 3T dataset of prostate MRI and a reproducible benchmark to foster the development of prostate segmentation algorithms.
Collapse
Affiliation(s)
- Lisa C Adams
- Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute for Radiology, Luisenstraße 7, 10117, Hindenburgdamm 30, 12203, Berlin, Germany; Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany.
| | - Marcus R Makowski
- Technical University of Munich, Department of Diagnostic and Interventional Radiology, Faculty of Medicine, Ismaninger Str. 22, 81675, Munich, Germany
| | - Günther Engel
- Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute for Radiology, Luisenstraße 7, 10117, Hindenburgdamm 30, 12203, Berlin, Germany; Institute for Diagnostic and Interventional Radiology, Georg-August University, Göttingen, Germany
| | - Maximilian Rattunde
- Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute for Radiology, Luisenstraße 7, 10117, Hindenburgdamm 30, 12203, Berlin, Germany
| | - Felix Busch
- Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute for Radiology, Luisenstraße 7, 10117, Hindenburgdamm 30, 12203, Berlin, Germany
| | - Patrick Asbach
- Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute for Radiology, Luisenstraße 7, 10117, Hindenburgdamm 30, 12203, Berlin, Germany
| | - Stefan M Niehues
- Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute for Radiology, Luisenstraße 7, 10117, Hindenburgdamm 30, 12203, Berlin, Germany
| | - Shankeeth Vinayahalingam
- Department of Oral and Maxillofacial Surgery, Radboud University Medical Center, Nijmegen, GA, the Netherlands
| | | | - Geert Litjens
- Radboud University Medical Center, Nijmegen, GA, the Netherlands
| | - Keno K Bressem
- Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute for Radiology, Luisenstraße 7, 10117, Hindenburgdamm 30, 12203, Berlin, Germany; Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany
| |
Collapse
|
18
|
Litjens G, Ciompi F, van der Laak J. A Decade of GigaScience: The Challenges of Gigapixel Pathology Images. Gigascience 2022; 11:6608502. [PMID: 35701372 PMCID: PMC9197683 DOI: 10.1093/gigascience/giac056] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2022] [Accepted: 05/23/2022] [Indexed: 11/14/2022] Open
Abstract
In the last decade, the field of computational pathology has advanced at a rapid pace because of the availability of deep neural networks, which achieved their first successes in computer vision tasks in 2012. An important driver for the progress of the field were public competitions, so called 'Grand Challenges', in which increasingly large data sets were offered to the public to solve clinically relevant tasks. Going from the first Pathology challenges, which had data obtained from 23 patients, to current challenges sharing data of thousands of patients, performance of developed deep learning solutions has reached (and sometimes surpassed) the level of experienced pathologists for specific tasks. We expect future challenges to broaden the horizon, for instance by combining data from radiology, pathology and tumor genetics, and to extract prognostic and predictive information independent of currently used grading schemes.
Collapse
Affiliation(s)
- Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Francesco Ciompi
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands.,Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| |
Collapse
|
19
|
Pinckaers H, van Ipenburg J, Melamed J, De Marzo A, Platz EA, van Ginneken B, van der Laak J, Litjens G. Predicting biochemical recurrence of prostate cancer with artificial intelligence. Commun Med 2022; 2:64. [PMID: 35693032 PMCID: PMC9177591 DOI: 10.1038/s43856-022-00126-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2021] [Accepted: 05/18/2022] [Indexed: 11/23/2022] Open
Abstract
Background The first sign of metastatic prostate cancer after radical prostatectomy is rising PSA levels in the blood, termed biochemical recurrence. The prediction of recurrence relies mainly on the morphological assessment of prostate cancer using the Gleason grading system. However, in this system, within-grade morphological patterns and subtle histopathological features are currently omitted, leaving a significant amount of prognostic potential unexplored. Methods To discover additional prognostic information using artificial intelligence, we trained a deep learning system to predict biochemical recurrence from tissue in H&E-stained microarray cores directly. We developed a morphological biomarker using convolutional neural networks leveraging a nested case-control study of 685 patients and validated on an independent cohort of 204 patients. We use concept-based explainability methods to interpret the learned tissue patterns. Results The biomarker provides a strong correlation with biochemical recurrence in two sets (n = 182 and n = 204) from separate institutions. Concept-based explanations provided tissue patterns interpretable by pathologists. Conclusions These results show that the model finds predictive power in the tissue beyond the morphological ISUP grading. To determine the prognosis of patients with prostate cancer, several clinical factors are taken into account. One of these is the cancer grade, assigned by a pathologist based on the cancer’s appearance under a microscope. The grade ranges from 1 to 5, where 5 is the most aggressive tumour type. This study explored whether deep learning—a technique in which computer software learns patterns from multiple examples—can learn to predict the risk of patients’ cancers recurring from microscopic images of the tumours. We show, on two clinical datasets from different institutions, that such a system can help to better predict prognosis, beyond the information provided by grade alone. In the future, this type of method could help clinicians to predict the prognosis of individual prostate cancer patients. Pinckaers et al. develop a deep learning system to predict biochemical recurrence in prostate cancer patients treated with radical prostatectomy. The authors’ morphological biomarker provides predictive power beyond traditional Gleason grading, based on analysis of two clinical datasets from different institutions.
Collapse
|
20
|
Pinckaers H, van Ginneken B, Litjens G. Streaming Convolutional Neural Networks for End-to-End Learning With Multi-Megapixel Images. IEEE Trans Pattern Anal Mach Intell 2022; 44:1581-1590. [PMID: 32845835 DOI: 10.1109/tpami.2020.3019563] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Due to memory constraints on current hardware, most convolution neural networks (CNN) are trained on sub-megapixel images. For example, most popular datasets in computer vision contain images much less than a megapixel in size (0.09MP for ImageNet and 0.001MP for CIFAR-10). In some domains such as medical imaging, multi-megapixel images are needed to identify the presence of disease accurately. We propose a novel method to directly train convolutional neural networks using any input image size end-to-end. This method exploits the locality of most operations in modern convolutional neural networks by performing the forward and backward pass on smaller tiles of the image. In this work, we show a proof of concept using images of up to 66-megapixels (8192×8192), saving approximately 50GB of memory per image. Using two public challenge datasets, we demonstrate that CNNs can learn to extract relevant information from these large images and benefit from increasing resolution. We improved the area under the receiver-operating characteristic curve from 0.580 (4MP) to 0.706 (66MP) for metastasis detection in breast cancer (CAMELYON17). We also obtained a Spearman correlation metric approaching state-of-the-art performance on the TUPAC16 dataset, from 0.485 (1MP) to 0.570 (16MP). Code to reproduce a subset of the experiments is available at https://github.com/DIAGNijmegen/StreamingCNN.
Collapse
|
21
|
Kartasalo K, Bulten W, Chen PH, Ström P, Pinckaers H, Nagpal K, Ruusuvuori P, Litjens G, Eklund M. Crowdsourcing of artificial intelligence algorithms for diagnosis and Gleason grading of prostate cancer in biopsies. Eur Urol 2022. [DOI: 10.1016/s0302-2838(22)00693-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
22
|
de Bel T, Litjens G, Ogony J, Stallings-Mann M, Carter JM, Hilton T, Radisky DC, Vierkant RA, Broderick B, Hoskin TL, Winham SJ, Frost MH, Visscher DW, Allers T, Degnim AC, Sherman ME, van der Laak JAWM. Automated quantification of levels of breast terminal duct lobular (TDLU) involution using deep learning. NPJ Breast Cancer 2022; 8:13. [PMID: 35046392 PMCID: PMC8770616 DOI: 10.1038/s41523-021-00378-7] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2021] [Accepted: 12/06/2021] [Indexed: 02/07/2023] Open
Abstract
Convolutional neural networks (CNNs) offer the potential to generate comprehensive quantitative analysis of histologic features. Diagnostic reporting of benign breast disease (BBD) biopsies is usually limited to subjective assessment of the most severe lesion in a sample, while ignoring the vast majority of tissue features, including involution of background terminal duct lobular units (TDLUs), the structures from which breast cancers arise. Studies indicate that increased levels of age-related TDLU involution in BBD biopsies predict lower breast cancer risk, and therefore its assessment may have potential value in risk assessment and management. However, assessment of TDLU involution is time-consuming and difficult to standardize and quantitate. Accordingly, we developed a CNN to enable automated quantitative measurement of TDLU involution and tested its performance in 174 specimens selected from the pathology archives at Mayo Clinic, Rochester, MN. The CNN was trained and tested on a subset of 33 biopsies, delineating important tissue types. Nine quantitative features were extracted from delineated TDLU regions. Our CNN reached an overall dice-score of 0.871 (±0.049) for tissue classes versus reference standard annotation. Consensus of four reviewers scoring 705 images for TDLU involution demonstrated substantial agreement with the CNN method (unweighted κappa = 0.747 ± 0.01). Quantitative involution measures showed anticipated associations with BBD histology, breast cancer risk, breast density, menopausal status, and breast cancer risk prediction scores (p < 0.05). Our work demonstrates the potential to improve risk prediction for women with BBD biopsies by applying CNN approaches to generate automated quantitative evaluation of TDLU involution.
Collapse
Affiliation(s)
- Thomas de Bel
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands. .,Radboud University Medical Center, Radboud Institute for Health Sciences, Nijmegen, The Netherlands.
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands.,Radboud University Medical Center, Radboud Institute for Health Sciences, Nijmegen, The Netherlands
| | - Joshua Ogony
- Quantitative Health Sciences, Mayo Clinic, Jacksonville, FL, USA
| | | | - Jodi M Carter
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA
| | - Tracy Hilton
- Quantitative Health Sciences, Mayo Clinic, Jacksonville, FL, USA
| | - Derek C Radisky
- Department of Cancer Biology, Mayo Clinic, Jacksonville, FL, USA
| | | | | | - Tanya L Hoskin
- Quantitative Health Sciences, Mayo Clinic, Jacksonville, FL, USA
| | - Stacey J Winham
- Quantitative Health Sciences, Mayo Clinic, Jacksonville, FL, USA
| | - Marlene H Frost
- Division of Medical Oncology, Mayo Clinic, Rochester, MN, USA
| | - Daniel W Visscher
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA
| | - Teresa Allers
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA
| | - Amy C Degnim
- Department of Surgery, Mayo Clinic, Rochester, MN, USA
| | - Mark E Sherman
- Quantitative Health Sciences, Mayo Clinic, Jacksonville, FL, USA
| | - Jeroen A W M van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands.,Radboud University Medical Center, Radboud Institute for Health Sciences, Nijmegen, The Netherlands.,Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| |
Collapse
|
23
|
Bulten W, Kartasalo K, Chen PHC, Ström P, Pinckaers H, Nagpal K, Cai Y, Steiner DF, van Boven H, Vink R, Hulsbergen-van de Kaa C, van der Laak J, Amin MB, Evans AJ, van der Kwast T, Allan R, Humphrey PA, Grönberg H, Samaratunga H, Delahunt B, Tsuzuki T, Häkkinen T, Egevad L, Demkin M, Dane S, Tan F, Valkonen M, Corrado GS, Peng L, Mermel CH, Ruusuvuori P, Litjens G, Eklund M. Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge. Nat Med 2022; 28:154-163. [PMID: 35027755 PMCID: PMC8799467 DOI: 10.1038/s41591-021-01620-2] [Citation(s) in RCA: 81] [Impact Index Per Article: 40.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2021] [Accepted: 11/08/2021] [Indexed: 12/12/2022]
Abstract
Artificial intelligence (AI) has shown promise for diagnosing prostate cancer in biopsies. However, results have been limited to individual studies, lacking validation in multinational settings. Competitions have been shown to be accelerators for medical imaging innovations, but their impact is hindered by lack of reproducibility and independent validation. With this in mind, we organized the PANDA challenge-the largest histopathology competition to date, joined by 1,290 developers-to catalyze development of reproducible AI algorithms for Gleason grading using 10,616 digitized prostate biopsies. We validated that a diverse set of submitted algorithms reached pathologist-level performance on independent cross-continental cohorts, fully blinded to the algorithm developers. On United States and European external validation sets, the algorithms achieved agreements of 0.862 (quadratically weighted κ, 95% confidence interval (CI), 0.840-0.884) and 0.868 (95% CI, 0.835-0.900) with expert uropathologists. Successful generalization across different patient populations, laboratories and reference standards, achieved by a variety of algorithmic approaches, warrants evaluating AI-based Gleason grading in prospective clinical trials.
Collapse
Affiliation(s)
- Wouter Bulten
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands.
| | - Kimmo Kartasalo
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden.
- Faculty of Medicine and Health Technology, Tampere University, Tampere, Finland.
| | | | - Peter Ström
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | - Hans Pinckaers
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| | | | | | | | - Hester van Boven
- Department of Pathology, Antoni van Leeuwenhoek Hospital, The Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Robert Vink
- Laboratory of Pathology East Netherlands, Hengelo, The Netherlands
| | | | - Jeroen van der Laak
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
- Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | - Mahul B Amin
- Department of Pathology and Laboratory Medicine, University of Tennessee Health Science Center, Memphis, TN, USA
| | - Andrew J Evans
- Laboratory Medicine, Mackenzie Health, Toronto, Ontario, Canada
| | - Theodorus van der Kwast
- Department of Pathology, Laboratory Medicine and Pathology, University Health Network and University of Toronto, Toronto, Ontario, Canada
| | - Robert Allan
- Pathology and Laboratory Medicine Service, North Florida/South Georgia Veterans Health System, Department of Pathology, Immunology and Laboratory Medicine, University of Florida, Gainesville, FL, USA
| | - Peter A Humphrey
- Department of Pathology, Yale School of Medicine, New Haven, CT, USA
| | - Henrik Grönberg
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
- Department of Surgery, Capio St. Göran's Hospital, Stockholm, Sweden
| | | | - Brett Delahunt
- Department of Pathology and Molecular Medicine, Wellington School of Medicine and Health Sciences, University of Otago, Wellington, New Zealand
| | - Toyonori Tsuzuki
- Department of Surgical Pathology, School of Medicine, Aichi Medical University, Nagakute, Japan
| | - Tomi Häkkinen
- Faculty of Medicine and Health Technology, Tampere University, Tampere, Finland
| | - Lars Egevad
- Department of Oncology and Pathology, Karolinska Institutet, Stockholm, Sweden
| | | | | | | | - Masi Valkonen
- Institute of Biomedicine, Cancer Research Unit and FICAN West Cancer Centre, University of Turku and Turku University Hospital, Turku, Finland
| | | | | | | | - Pekka Ruusuvuori
- Faculty of Medicine and Health Technology, Tampere University, Tampere, Finland
- Institute of Biomedicine, Cancer Research Unit and FICAN West Cancer Centre, University of Turku and Turku University Hospital, Turku, Finland
| | - Geert Litjens
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Martin Eklund
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| |
Collapse
|
24
|
van Eekelen L, Pinckaers H, van den Brand M, Hebeda KM, Litjens G. Using deep learning for quantification of cellularity and cell lineages in bone marrow biopsies and comparison to normal age-related variation. Pathology 2021; 54:318-327. [PMID: 34772487 DOI: 10.1016/j.pathol.2021.07.011] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2020] [Revised: 07/07/2021] [Accepted: 07/14/2021] [Indexed: 01/21/2023]
Abstract
Cellularity estimation forms an important aspect of the visual examination of bone marrow biopsies. In clinical practice, cellularity is estimated by eye under a microscope, which is rapid, but subjective and subject to inter- and intraobserver variability. In addition, there is little consensus in the literature on the normal variation of cellularity with age. Digital image analysis may be used for more objective quantification of cellularity. As such, we developed a deep neural network for the segmentation of six major cell and tissue types in digitized bone marrow trephine biopsies. Using this segmentation, we calculated the overall bone marrow cellularity in a series of biopsies from 130 patients across a wide age range. Using intraclass correlation coefficients (ICC), we measured the agreement between the quantification by the neural network and visual estimation by two pathologists and compared it to baseline human performance. We also examined the age-related changes of cellularity and cell lineages in bone marrow and compared our results to those found in the literature. The network was capable of accurate segmentation (average accuracy and dice score of 0.95 and 0.76, respectively). There was good neural network-pathologist agreement on cellularity measurements (ICC=0.78, 95% CI 0.58-0.85). We found a statistically significant downward trend for cellularity, myelopoiesis and megakaryocytes with age in our cohort. The mean cellularity began at approximately 50% in the third decade of life and then decreased ±2% per decade to 40% in the seventh and eighth decade, but the normal range was very wide (30-70%).
Collapse
Affiliation(s)
- Leander van Eekelen
- Faculty of Biomedical Engineering, Technical University Eindhoven, Eindhoven, the Netherlands; Computational Pathology Group, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Hans Pinckaers
- Computational Pathology Group, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, the Netherlands; Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Michiel van den Brand
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands; Pathology-DNA, Rijnstate Hospital, Arnhem, the Netherlands
| | - Konnie M Hebeda
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands.
| | - Geert Litjens
- Computational Pathology Group, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, the Netherlands; Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| |
Collapse
|
25
|
Pinckaers H, Bulten W, van der Laak J, Litjens G. Detection of Prostate Cancer in Whole-Slide Images Through End-to-End Training With Image-Level Labels. IEEE Trans Med Imaging 2021; 40:1817-1826. [PMID: 33729928 DOI: 10.1109/tmi.2021.3066295] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Prostate cancer is the most prevalent cancer among men in Western countries, with 1.1 million new diagnoses every year. The gold standard for the diagnosis of prostate cancer is a pathologists' evaluation of prostate tissue. To potentially assist pathologists deep / learning / based cancer detection systems have been developed. Many of the state-of-the-art models are patch / based convolutional neural networks, as the use of entire scanned slides is hampered by memory limitations on accelerator cards. Patch-based systems typically require detailed, pixel-level annotations for effective training. However, such annotations are seldom readily available, in contrast to the clinical reports of pathologists, which contain slide-level labels. As such, developing algorithms which do not require manual pixel-wise annotations, but can learn using only the clinical report would be a significant advancement for the field. In this paper, we propose to use a streaming implementation of convolutional layers, to train a modern CNN (ResNet / 34) with 21 million parameters end-to-end on 4712 prostate biopsies. The method enables the use of entire biopsy images at high-resolution directly by reducing the GPU memory requirements by 2.4 TB. We show that modern CNNs, trained using our streaming approach, can extract meaningful features from high-resolution images without additional heuristics, reaching similar performance as state-of-the-art patch-based and multiple-instance learning methods. By circumventing the need for manual annotations, this approach can function as a blueprint for other tasks in histopathological diagnosis. The source code to reproduce the streaming models is available at https://github.com/DIAGNijmegen/ pathology-streaming-pipeline.
Collapse
|
26
|
Abstract
Machine learning techniques have great potential to improve medical diagnostics, offering ways to improve accuracy, reproducibility and speed, and to ease workloads for clinicians. In the field of histopathology, deep learning algorithms have been developed that perform similarly to trained pathologists for tasks such as tumor detection and grading. However, despite these promising results, very few algorithms have reached clinical implementation, challenging the balance between hope and hype for these new techniques. This Review provides an overview of the current state of the field, as well as describing the challenges that still need to be addressed before artificial intelligence in histopathology can achieve clinical value.
Collapse
Affiliation(s)
- Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands. .,Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden.
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Francesco Ciompi
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| |
Collapse
|
27
|
Balkenhol MC, Ciompi F, Świderska-Chadaj Ż, van de Loo R, Intezar M, Otte-Höller I, Geijs D, Lotz J, Weiss N, de Bel T, Litjens G, Bult P, van der Laak JA. Optimized tumour infiltrating lymphocyte assessment for triple negative breast cancer prognostics. Breast 2021; 56:78-87. [PMID: 33640523 PMCID: PMC7933536 DOI: 10.1016/j.breast.2021.02.007] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2020] [Revised: 02/05/2021] [Accepted: 02/08/2021] [Indexed: 12/29/2022] Open
Abstract
The tumour microenvironment has been shown to be a valuable source of prognostic information for different cancer types. This holds in particular for triple negative breast cancer (TNBC), a breast cancer subtype for which currently no prognostic biomarkers are established. Although different methods to assess tumour infiltrating lymphocytes (TILs) have been published, it remains unclear which method (marker, region) yields the most optimal prognostic information. In addition, to date, no objective TILs assessment methods are available. For this proof of concept study, a subset of our previously described TNBC cohort (n = 94) was stained for CD3, CD8 and FOXP3 using multiplex immunohistochemistry and subsequently imaged by a multispectral imaging system. Advanced whole-slide image analysis algorithms, including convolutional neural networks (CNN) were used to register unmixed multispectral images and corresponding H&E sections, to segment the different tissue compartments (tumour, stroma) and to detect all individual positive lymphocytes. Densities of positive lymphocytes were analysed in different regions within the tumour and its neighbouring environment and correlated to relapse free survival (RFS) and overall survival (OS). We found that for all TILs markers the presence of a high density of positive cells correlated with an improved survival. None of the TILs markers was superior to the others. The results of TILs assessment in the various regions did not show marked differences between each other. The negative correlation between TILs and survival in our cohort are in line with previous studies. Our results provide directions for optimizing TILs assessment methodology.
Collapse
Affiliation(s)
- Maschenka Ca Balkenhol
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands.
| | - Francesco Ciompi
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Żaneta Świderska-Chadaj
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands; Warsaw University of Technology, Faculty of Electrical Engineering, Warsaw, Poland
| | - Rob van de Loo
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Milad Intezar
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Irene Otte-Höller
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Daan Geijs
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Johannes Lotz
- Fraunhofer Institute for Image Computing MEVIS, Lübeck, Germany
| | - Nick Weiss
- Fraunhofer Institute for Image Computing MEVIS, Lübeck, Germany
| | - Thomas de Bel
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Geert Litjens
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Peter Bult
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Jeroen Awm van der Laak
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands; Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| |
Collapse
|
28
|
Turner OC, Knight B, Zuraw A, Litjens G, Rudmann DG. Mini Review: The Last Mile-Opportunities and Challenges for Machine Learning in Digital Toxicologic Pathology. Toxicol Pathol 2021; 49:714-719. [PMID: 33590805 DOI: 10.1177/0192623321990375] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
The 2019 manuscript by the Special Interest Group on Digital Pathology and Image Analysis of the Society of Toxicologic pathology suggested that a synergism between artificial intelligence (AI) and machine learning (ML) technologies and digital toxicologic pathology would improve the daily workflow and future impact of toxicologic pathologists globally. Now 2 years later, the authors of this review consider whether, in their opinion, there is any evidence that supports that thesis. Specifically, we consider the opportunities and challenges for applying ML (the study of computer algorithms that are able to learn from example data and extrapolate the learned information to unseen data) algorithms in toxicologic pathology and how regulatory bodies are navigating this rapidly evolving field. Although we see similarities with the "Last Mile" metaphor, the weight of evidence suggests that toxicologic pathologists should approach ML with an equal dose of skepticism and enthusiasm. There are increasing opportunities for impact in our field that leave the authors cautiously excited and optimistic. Toxicologic pathologists have the opportunity to critically evaluate ML applications with a "call-to-arms" mentality. Why should we be late adopters? There is ample evidence to encourage engagement, growth, and leadership in this field.
Collapse
Affiliation(s)
- Oliver C Turner
- Novartis, 98557Novartis Institutes for BioMedical Research, Preclinical Safety, East Hanover, NJ, USA
| | - Brian Knight
- 435339Boehringer Ingelheim Pharmaceuticals Incorporated, Nonclinical Drug Safety, Ridgefield, CT, USA
| | | | - Geert Litjens
- Diagnostic Image Analysis Group Radboud University Medical Center Nijmegen, the Netherlands
| | | |
Collapse
|
29
|
Li Z, Zhang J, Tan T, Teng X, Sun X, Zhao H, Liu L, Xiao Y, Lee B, Li Y, Zhang Q, Sun S, Zheng Y, Yan J, Li N, Hong Y, Ko J, Jung H, Liu Y, Chen YC, Wang CW, Yurovskiy V, Maevskikh P, Khanagha V, Jiang Y, Yu L, Liu Z, Li D, Schuffler PJ, Yu Q, Chen H, Tang Y, Litjens G. Deep Learning Methods for Lung Cancer Segmentation in Whole-Slide Histopathology Images-The ACDC@LungHP Challenge 2019. IEEE J Biomed Health Inform 2021; 25:429-440. [PMID: 33216724 DOI: 10.1109/jbhi.2020.3039741] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Accurate segmentation of lung cancer in pathology slides is a critical step in improving patient care. We proposed the ACDC@LungHP (Automatic Cancer Detection and Classification in Whole-slide Lung Histopathology) challenge for evaluating different computer-aided diagnosis (CADs) methods on the automatic diagnosis of lung cancer. The ACDC@LungHP 2019 focused on segmentation (pixel-wise detection) of cancer tissue in whole slide imaging (WSI), using an annotated dataset of 150 training images and 50 test images from 200 patients. This paper reviews this challenge and summarizes the top 10 submitted methods for lung cancer segmentation. All methods were evaluated using metrics using the precision, accuracy, sensitivity, specificity, and DICE coefficient (DC). The DC ranged from 0.7354 ±0.1149 to 0.8372 ±0.0858. The DC of the best method was close to the inter-observer agreement (0.8398 ±0.0890). All methods were based on deep learning and categorized into two groups: multi-model method and single model method. In general, multi-model methods were significantly better (p 0.01) than single model methods, with mean DC of 0.7966 and 0.7544, respectively. Deep learning based methods could potentially help pathologists find suspicious regions for further analysis of lung cancer in WSI.
Collapse
|
30
|
Tellez D, Litjens G, van der Laak J, Ciompi F. Neural Image Compression for Gigapixel Histopathology Image Analysis. IEEE Trans Pattern Anal Mach Intell 2021; 43:567-578. [PMID: 31442971 DOI: 10.1109/tpami.2019.2936841] [Citation(s) in RCA: 48] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
We propose Neural Image Compression (NIC), a two-step method to build convolutional neural networks for gigapixel image analysis solely using weak image-level labels. First, gigapixel images are compressed using a neural network trained in an unsupervised fashion, retaining high-level information while suppressing pixel-level noise. Second, a convolutional neural network (CNN) is trained on these compressed image representations to predict image-level labels, avoiding the need for fine-grained manual annotations. We compared several encoding strategies, namely reconstruction error minimization, contrastive training and adversarial feature learning, and evaluated NIC on a synthetic task and two public histopathology datasets. We found that NIC can exploit visual cues associated with image-level labels successfully, integrating both global and local visual information. Furthermore, we visualized the regions of the input gigapixel images where the CNN attended to, and confirmed that they overlapped with annotations from human experts.
Collapse
|
31
|
Bulten W, Balkenhol M, Belinga JJA, Brilhante A, Çakır A, Egevad L, Eklund M, Farré X, Geronatsiou K, Molinié V, Pereira G, Roy P, Saile G, Salles P, Schaafsma E, Tschui J, Vos AM, van Boven H, Vink R, van der Laak J, Hulsbergen-van der Kaa C, Litjens G. Artificial intelligence assistance significantly improves Gleason grading of prostate biopsies by pathologists. Mod Pathol 2021; 34:660-671. [PMID: 32759979 PMCID: PMC7897578 DOI: 10.1038/s41379-020-0640-y] [Citation(s) in RCA: 59] [Impact Index Per Article: 19.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2020] [Revised: 07/22/2020] [Accepted: 07/23/2020] [Indexed: 12/21/2022]
Abstract
The Gleason score is the most important prognostic marker for prostate cancer patients, but it suffers from significant observer variability. Artificial intelligence (AI) systems based on deep learning can achieve pathologist-level performance at Gleason grading. However, the performance of such systems can degrade in the presence of artifacts, foreign tissue, or other anomalies. Pathologists integrating their expertise with feedback from an AI system could result in a synergy that outperforms both the individual pathologist and the system. Despite the hype around AI assistance, existing literature on this topic within the pathology domain is limited. We investigated the value of AI assistance for grading prostate biopsies. A panel of 14 observers graded 160 biopsies with and without AI assistance. Using AI, the agreement of the panel with an expert reference standard increased significantly (quadratically weighted Cohen's kappa, 0.799 vs. 0.872; p = 0.019). On an external validation set of 87 cases, the panel showed a significant increase in agreement with a panel of international experts in prostate pathology (quadratically weighted Cohen's kappa, 0.733 vs. 0.786; p = 0.003). In both experiments, on a group-level, AI-assisted pathologists outperformed the unassisted pathologists and the standalone AI system. Our results show the potential of AI systems for Gleason grading, but more importantly, show the benefits of pathologist-AI synergy.
Collapse
Affiliation(s)
- Wouter Bulten
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands.
| | - Maschenka Balkenhol
- grid.10417.330000 0004 0444 9382Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Jean-Joël Awoumou Belinga
- grid.412661.60000 0001 2173 8504Department of Morphological Sciences and Anatomic Pathology Faculty of Medicine and Biomedical Sciences, University of Yaounde 1, Yaounde, Cameroon
| | | | - Aslı Çakır
- grid.411781.a0000 0004 0471 9346Pathology Department, School of Medicine, Istanbul Medipol University, Istanbul, Turkey
| | - Lars Egevad
- grid.4714.60000 0004 1937 0626Department of Oncology and Pathology, Karolinska Institutet, Stockholm, Sweden
| | - Martin Eklund
- grid.4714.60000 0004 1937 0626Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | - Xavier Farré
- grid.500777.2Department of Health, Public Health Agency of Catalonia, Lleida, Catalonia Spain
| | | | - Vincent Molinié
- Pathology department, Aix en Provence Hospital, Aix-en-Provence, France
| | | | - Paromita Roy
- grid.430884.30000 0004 1770 8996Department of Pathology, Tata Medical Center, Kolkata, India
| | - Günter Saile
- Iabor team w ag, Abteilung für Histopathologie und Zytologie, Goldach SG, Switzerland
| | | | - Ewout Schaafsma
- grid.10417.330000 0004 0444 9382Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| | | | - Anne-Marie Vos
- grid.10417.330000 0004 0444 9382Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| | | | - Hester van Boven
- grid.430814.aDepartment of Pathology, Antoni van Leeuwenhoek Hospital, The Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Robert Vink
- Laboratory of Pathology East Netherlands, Hengelo, The Netherlands
| | - Jeroen van der Laak
- grid.10417.330000 0004 0444 9382Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands ,grid.5640.70000 0001 2162 9922Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | | | - Geert Litjens
- grid.10417.330000 0004 0444 9382Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
32
|
Swiderska-Chadaj Z, Hebeda KM, van den Brand M, Litjens G. Artificial intelligence to detect MYC translocation in slides of diffuse large B-cell lymphoma. Virchows Arch 2020; 479:617-621. [PMID: 32979109 PMCID: PMC8448690 DOI: 10.1007/s00428-020-02931-4] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2020] [Revised: 08/24/2020] [Accepted: 09/15/2020] [Indexed: 02/02/2023]
Abstract
In patients with suspected lymphoma, the tissue biopsy provides lymphoma confirmation, classification, and prognostic factors, including genetic changes. We developed a deep learning algorithm to detect MYC rearrangement in scanned histological slides of diffuse large B-cell lymphoma. The H&E-stained slides of 287 cases from 11 hospitals were used for training and evaluation. The overall sensitivity to detect MYC rearrangement was 0.93 and the specificity 0.52, showing that prediction of MYC translocation based on morphology alone was possible in 93% of MYC-rearranged cases. This would allow a simple and fast prescreening, saving approximately 34% of genetic tests with the current algorithm.
Collapse
Affiliation(s)
- Zaneta Swiderska-Chadaj
- Department of Pathology, Radboud University Medical Center, Geert Grooteplein 10, P.O. Box 9101, 6500 HB, Nijmegen, The Netherlands. .,Faculty of Electrical Engineering, Warsaw University of Technology, Warsaw, Poland.
| | - Konnie M Hebeda
- Department of Pathology, Radboud University Medical Center, Geert Grooteplein 10, P.O. Box 9101, 6500 HB, Nijmegen, The Netherlands
| | - Michiel van den Brand
- Department of Pathology, Radboud University Medical Center, Geert Grooteplein 10, P.O. Box 9101, 6500 HB, Nijmegen, The Netherlands.,Pathology-DNA, Rijnstate Hospital, Arnhem, the Netherlands
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Geert Grooteplein 10, P.O. Box 9101, 6500 HB, Nijmegen, The Netherlands
| |
Collapse
|
33
|
Swiderska-Chadaj Z, de Bel T, Blanchet L, Baidoshvili A, Vossen D, van der Laak J, Litjens G. Impact of rescanning and normalization on convolutional neural network performance in multi-center, whole-slide classification of prostate cancer. Sci Rep 2020; 10:14398. [PMID: 32873856 PMCID: PMC7462850 DOI: 10.1038/s41598-020-71420-0] [Citation(s) in RCA: 23] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2020] [Accepted: 08/12/2020] [Indexed: 01/12/2023] Open
Abstract
Algorithms can improve the objectivity and efficiency of histopathologic slide analysis. In this paper, we investigated the impact of scanning systems (scanners) and cycle-GAN-based normalization on algorithm performance, by comparing different deep learning models to automatically detect prostate cancer in whole-slide images. Specifically, we compare U-Net, DenseNet and EfficientNet. Models were developed on a multi-center cohort with 582 WSIs and subsequently evaluated on two independent test sets including 85 and 50 WSIs, respectively, to show the robustness of the proposed method to differing staining protocols and scanner types. We also investigated the application of normalization as a pre-processing step by two techniques, the whole-slide image color standardizer (WSICS) algorithm, and a cycle-GAN based method. For the two independent datasets we obtained an AUC of 0.92 and 0.83 respectively. After rescanning the AUC improves to 0.91/0.88 and after style normalization to 0.98/0.97. In the future our algorithm could be used to automatically pre-screen prostate biopsies to alleviate the workload of pathologists.
Collapse
Affiliation(s)
| | - Thomas de Bel
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Lionel Blanchet
- Digital and Computational Pathology, Philips, Best, The Netherlands
| | - Alexi Baidoshvili
- Laboratorium Pathologie Oost-Nederland, LabPON, Hengelo, The Netherlands
| | - Dirk Vossen
- Digital and Computational Pathology, Philips, Best, The Netherlands
| | - Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands.,Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
34
|
Litjens G, Rivière DM, van Geenen EJM, Radema SA, Brosens LAA, Prokop M, van Laarhoven CJHM, Hermans JJ. Diagnostic accuracy of contrast-enhanced diffusion-weighted MRI for liver metastases of pancreatic cancer: towards adequate staging and follow-up of pancreatic cancer - DIA-PANC study: study protocol for an international, multicenter, diagnostic trial. BMC Cancer 2020; 20:744. [PMID: 32778061 PMCID: PMC7418197 DOI: 10.1186/s12885-020-07226-0] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2020] [Accepted: 07/28/2020] [Indexed: 12/19/2022] Open
Abstract
BACKGROUND At the time of surgery, approximately 10-20% of the patients with pancreatic cancer are considered unresectable because of unexpected liver metastasis, peritoneal carcinomatosis or locally advanced disease. This leads to futile surgical treatment with all the associated morbidity, mortality and costs. More than 50% of all liver metastases develop in the first six months postoperatively. These (subcentimeter) liver metastases are most likely already present at the time of diagnosis and have not been identified pre-operatively, due to the poor sensitivity of routine preoperative contrast-enhanced CT (CECT). METHODS The DIA-PANC study is a prospective, international, multicenter, diagnostic cohort study investigating diffusion-weighted, contrast-enhanced MRI for the detection of liver metastases in patients with all stages of pancreatic cancer. Indeterminate or malignant liver lesions on MRI will be further investigated histopathologically. For patients with suspected liver lesions without histopathological proof, follow up imaging with paired CT and MRI at 3-, 6- and 12-months will serve as an alternative reference standard. DISCUSSION The DIA-PANC trial is expected to report high-level evidence of the diagnostic accuracy of MRI for the detection of liver metastases, resulting in significant value for clinical decision making, guideline development and improved stratification for treatment strategies and future trials. Furthermore, DIA-PANC will contribute to our knowledge of liver metastases regarding incidence, imaging characteristics, their number and extent, and their change in time with or without treatment. It will enhance the worldwide implementation of MRI and consequently improve personalized treatment of patients with suspected pancreatic ductal adenocarcinoma. TRIAL REGISTRATION ClinicalTrials.gov Identifier: NCT03469726 . Registered on March 19th 2018 - Retrospectively registered.
Collapse
Affiliation(s)
- G. Litjens
- Department of Radiology and Nuclear Medicine, Radboudumc, Nijmegen, The Netherlands
| | - D. M. Rivière
- Department of Radiology and Nuclear Medicine, Radboudumc, Nijmegen, The Netherlands
| | - E. J. M. van Geenen
- Department of Gastroenterology and Hepatology, Radboudumc, Nijmegen, The Netherlands
| | - S. A. Radema
- Department of Medical Oncology, Radboudumc, Nijmegen, The Netherlands
| | - L. A. A. Brosens
- Department of Pathology, Radboudumc, Nijmegen, The Netherlands
- Department of Pathology, University Medical Center, Utrecht, The Netherlands
| | - M. Prokop
- Department of Radiology and Nuclear Medicine, Radboudumc, Nijmegen, The Netherlands
| | | | - J. J. Hermans
- Department of Radiology and Nuclear Medicine, Radboudumc, Nijmegen, The Netherlands
| |
Collapse
|
35
|
Bulten W, Pinckaers H, van Boven H, Vink R, de Bel T, van Ginneken B, van der Laak J, Hulsbergen-van de Kaa C, Litjens G. Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study. Lancet Oncol 2020; 21:233-241. [PMID: 31926805 DOI: 10.1016/s1470-2045(19)30739-9] [Citation(s) in RCA: 303] [Impact Index Per Article: 75.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2019] [Revised: 10/10/2019] [Accepted: 10/22/2019] [Indexed: 01/19/2023]
Abstract
BACKGROUND The Gleason score is the strongest correlating predictor of recurrence for prostate cancer, but has substantial inter-observer variability, limiting its usefulness for individual patients. Specialised urological pathologists have greater concordance; however, such expertise is not widely available. Prostate cancer diagnostics could thus benefit from robust, reproducible Gleason grading. We aimed to investigate the potential of deep learning to perform automated Gleason grading of prostate biopsies. METHODS In this retrospective study, we developed a deep-learning system to grade prostate biopsies following the Gleason grading standard. The system was developed using randomly selected biopsies, sampled by the biopsy Gleason score, from patients at the Radboud University Medical Center (pathology report dated between Jan 1, 2012, and Dec 31, 2017). A semi-automatic labelling technique was used to circumvent the need for manual annotations by pathologists, using pathologists' reports as the reference standard during training. The system was developed to delineate individual glands, assign Gleason growth patterns, and determine the biopsy-level grade. For validation of the method, a consensus reference standard was set by three expert urological pathologists on an independent test set of 550 biopsies. Of these 550, 100 were used in an observer experiment, in which the system, 13 pathologists, and two pathologists in training were compared with respect to the reference standard. The system was also compared to an external test dataset of 886 cores, which contained 245 cores from a different centre that were independently graded by two pathologists. FINDINGS We collected 5759 biopsies from 1243 patients. The developed system achieved a high agreement with the reference standard (quadratic Cohen's kappa 0·918, 95% CI 0·891-0·941) and scored highly at clinical decision thresholds: benign versus malignant (area under the curve 0·990, 95% CI 0·982-0·996), grade group of 2 or more (0·978, 0·966-0·988), and grade group of 3 or more (0·974, 0·962-0·984). In an observer experiment, the deep-learning system scored higher (kappa 0·854) than the panel (median kappa 0·819), outperforming 10 of 15 pathologist observers. On the external test dataset, the system obtained a high agreement with the reference standard set independently by two pathologists (quadratic Cohen's kappa 0·723 and 0·707) and within inter-observer variability (kappa 0·71). INTERPRETATION Our automated deep-learning system achieved a performance similar to pathologists for Gleason grading and could potentially contribute to prostate cancer diagnosis. The system could potentially assist pathologists by screening biopsies, providing second opinions on grade group, and presenting quantitative measurements of volume percentages. FUNDING Dutch Cancer Society.
Collapse
Affiliation(s)
- Wouter Bulten
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands.
| | - Hans Pinckaers
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
| | - Hester van Boven
- Department of Pathology, Antoni van Leeuwenhoek Hospital, The Netherlands Cancer Institute, Amsterdam, Netherlands
| | - Robert Vink
- Laboratory of Pathology East Netherlands, Hengelo, Netherlands
| | - Thomas de Bel
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
| | - Bram van Ginneken
- Department of Radiology & Nuclear Medicine, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
| | - Jeroen van der Laak
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
| | | | - Geert Litjens
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
| |
Collapse
|
36
|
Bándi P, Balkenhol M, van Ginneken B, van der Laak J, Litjens G. Resolution-agnostic tissue segmentation in whole-slide histopathology images with convolutional neural networks. PeerJ 2019; 7:e8242. [PMID: 31871843 PMCID: PMC6924324 DOI: 10.7717/peerj.8242] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2019] [Accepted: 11/19/2019] [Indexed: 12/20/2022] Open
Abstract
Modern pathology diagnostics is being driven toward large scale digitization of microscopic tissue sections. A prerequisite for its safe implementation is the guarantee that all tissue present on a glass slide can also be found back in the digital image. Whole-slide scanners perform a tissue segmentation in a low resolution overview image to prevent inefficient high-resolution scanning of empty background areas. However, currently applied algorithms can fail in detecting all tissue regions. In this study, we developed convolutional neural networks to distinguish tissue from background. We collected 100 whole-slide images of 10 tissue samples—staining categories from five medical centers for development and testing. Additionally, eight more images of eight unfamiliar categories were collected for testing only. We compared our fully-convolutional neural networks to three traditional methods on a range of resolution levels using Dice score and sensitivity. We also tested whether a single neural network can perform equivalently to multiple networks, each specialized in a single resolution. Overall, our solutions outperformed the traditional methods on all the tested resolutions. The resolution-agnostic network achieved average Dice scores between 0.97 and 0.98 across the tested resolution levels, only 0.0069 less than the resolution-specific networks. Finally, its excellent generalization performance was demonstrated by achieving averages of 0.98 Dice score and 0.97 sensitivity on the eight unfamiliar images. A future study should test this network prospectively.
Collapse
Affiliation(s)
- Péter Bándi
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Maschenka Balkenhol
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Bram van Ginneken
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
37
|
Tellez D, Litjens G, Bándi P, Bulten W, Bokhorst JM, Ciompi F, van der Laak J. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Med Image Anal 2019; 58:101544. [DOI: 10.1016/j.media.2019.101544] [Citation(s) in RCA: 179] [Impact Index Per Article: 35.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2019] [Revised: 06/25/2019] [Accepted: 08/19/2019] [Indexed: 11/17/2022]
|
38
|
Affiliation(s)
- Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center Nijmegen, Nijmegen, the Netherlands. .,Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden.
| | - Francesco Ciompi
- Department of Pathology, Radboud University Medical Center Nijmegen, Nijmegen, the Netherlands
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center Nijmegen, Nijmegen, the Netherlands
| |
Collapse
|
39
|
Swiderska-Chadaj Z, Pinckaers H, van Rijthoven M, Balkenhol M, Melnikova M, Geessink O, Manson Q, Sherman M, Polonia A, Parry J, Abubakar M, Litjens G, van der Laak J, Ciompi F. Learning to detect lymphocytes in immunohistochemistry with deep learning. Med Image Anal 2019; 58:101547. [PMID: 31476576 DOI: 10.1016/j.media.2019.101547] [Citation(s) in RCA: 65] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2019] [Revised: 08/12/2019] [Accepted: 08/20/2019] [Indexed: 12/17/2022]
Abstract
The immune system is of critical importance in the development of cancer. The evasion of destruction by the immune system is one of the emerging hallmarks of cancer. We have built a dataset of 171,166 manually annotated CD3+ and CD8+ cells, which we used to train deep learning algorithms for automatic detection of lymphocytes in histopathology images to better quantify immune response. Moreover, we investigate the effectiveness of four deep learning based methods when different subcompartments of the whole-slide image are considered: normal tissue areas, areas with immune cell clusters, and areas containing artifacts. We have compared the proposed methods in breast, colon and prostate cancer tissue slides collected from nine different medical centers. Finally, we report the results of an observer study on lymphocyte quantification, which involved four pathologists from different medical centers, and compare their performance with the automatic detection. The results give insights on the applicability of the proposed methods for clinical use. U-Net obtained the highest performance with an F1-score of 0.78 and the highest agreement with manual evaluation (κ=0.72), whereas the average pathologists agreement with reference standard was κ=0.64. The test set and the automatic evaluation procedure are publicly available at lyon19.grand-challenge.org.
Collapse
Affiliation(s)
| | - Hans Pinckaers
- Department of Pathology, Radboud University Medical Center, The Netherlands
| | - Mart van Rijthoven
- Department of Pathology, Radboud University Medical Center, The Netherlands
| | | | - Margarita Melnikova
- Department of Pathology, Radboud University Medical Center, The Netherlands; Department of Clinical Medicine, Aarhus University, Denmark; Institute of Pathology, Randers Regional Hospital, Denmark
| | - Oscar Geessink
- Department of Pathology, Radboud University Medical Center, The Netherlands
| | - Quirine Manson
- Department of Pathology, University Medical Center, Utrecht, The Netherlands
| | | | - Antonio Polonia
- Institute of Molecular Pathology and Immunology, University of Porto, Porto, Portugal
| | - Jeremy Parry
- Fiona Stanley Hospital, Murdoch, Perth, Western Australia
| | - Mustapha Abubakar
- Integrative Tumor Epidemiology Branch, Division of Cancer Epidemiology and Genetics, National Cancer Institute, USA
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, The Netherlands
| | - Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, The Netherlands; Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | - Francesco Ciompi
- Department of Pathology, Radboud University Medical Center, The Netherlands
| |
Collapse
|
40
|
Litjens G, Ciompi F, Wolterink JM, de Vos BD, Leiner T, Teuwen J, Išgum I. State-of-the-Art Deep Learning in Cardiovascular Image Analysis. JACC Cardiovasc Imaging 2019; 12:1549-1565. [DOI: 10.1016/j.jcmg.2019.06.009] [Citation(s) in RCA: 141] [Impact Index Per Article: 28.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/01/2018] [Revised: 05/13/2019] [Accepted: 06/13/2019] [Indexed: 02/07/2023]
|
41
|
Aprupe L, Litjens G, Brinker TJ, van der Laak J, Grabe N. Robust and accurate quantification of biomarkers of immune cells in lung cancer micro-environment using deep convolutional neural networks. PeerJ 2019; 7:e6335. [PMID: 30993030 PMCID: PMC6462181 DOI: 10.7717/peerj.6335] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2018] [Accepted: 12/23/2018] [Indexed: 01/24/2023] Open
Abstract
Recent years have seen a growing awareness of the role the immune system plays in successful cancer treatment, especially in novel therapies like immunotherapy. The characterization of the immunological composition of tumors and their micro-environment is thus becoming a necessity. In this paper we introduce a deep learning-based immune cell detection and quantification method, which is based on supervised learning, i.e., the input data for training comprises labeled images. Our approach objectively deals with staining variation and staining artifacts in immunohistochemically stained lung cancer tissue and is as precise as humans. This is evidenced by the low cell count difference to humans of 0.033 cells on average. This method, which is based on convolutional neural networks, has the potential to provide a new quantitative basis for research on immunotherapy.
Collapse
Affiliation(s)
- Lilija Aprupe
- Hamamatsu Tissue Imaging and Analysis (TIGA) Center, BioQuant, Heidelberg University, Heidelberg, Germany.,Department of Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands.,Steinbeis Center for Medical Systems Biology (STCMSB), Heidelberg, Germany
| | - Titus J Brinker
- Department of Dermatology and National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Niels Grabe
- Hamamatsu Tissue Imaging and Analysis (TIGA) Center, BioQuant, Heidelberg University, Heidelberg, Germany.,Department of Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.,Steinbeis Center for Medical Systems Biology (STCMSB), Heidelberg, Germany
| |
Collapse
|
42
|
Bandi P, Geessink O, Manson Q, Van Dijk M, Balkenhol M, Hermsen M, Ehteshami Bejnordi B, Lee B, Paeng K, Zhong A, Li Q, Zanjani FG, Zinger S, Fukuta K, Komura D, Ovtcharov V, Cheng S, Zeng S, Thagaard J, Dahl AB, Lin H, Chen H, Jacobsson L, Hedlund M, Cetin M, Halici E, Jackson H, Chen R, Both F, Franke J, Kusters-Vandevelde H, Vreuls W, Bult P, van Ginneken B, van der Laak J, Litjens G. From Detection of Individual Metastases to Classification of Lymph Node Status at the Patient Level: The CAMELYON17 Challenge. IEEE Trans Med Imaging 2019; 38:550-560. [PMID: 30716025 DOI: 10.1109/tmi.2018.2867350] [Citation(s) in RCA: 156] [Impact Index Per Article: 31.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/22/2023]
Abstract
Automated detection of cancer metastases in lymph nodes has the potential to improve the assessment of prognosis for patients. To enable fair comparison between the algorithms for this purpose, we set up the CAMELYON17 challenge in conjunction with the IEEE International Symposium on Biomedical Imaging 2017 Conference in Melbourne. Over 300 participants registered on the challenge website, of which 23 teams submitted a total of 37 algorithms before the initial deadline. Participants were provided with 899 whole-slide images (WSIs) for developing their algorithms. The developed algorithms were evaluated based on the test set encompassing 100 patients and 500 WSIs. The evaluation metric used was a quadratic weighted Cohen's kappa. We discuss the algorithmic details of the 10 best pre-conference and two post-conference submissions. All these participants used convolutional neural networks in combination with pre- and postprocessing steps. Algorithms differed mostly in neural network architecture, training strategy, and pre- and postprocessing methodology. Overall, the kappa metric ranged from 0.89 to -0.13 across all submissions. The best results were obtained with pre-trained architectures such as ResNet. Confusion matrix analysis revealed that all participants struggled with reliably identifying isolated tumor cells, the smallest type of metastasis, with detection rates below 40%. Qualitative inspection of the results of the top participants showed categories of false positives, such as nerves or contamination, which could be targeted for further optimization. Last, we show that simple combinations of the top algorithms result in higher kappa metric values than any algorithm individually, with 0.93 for the best combination.
Collapse
|
43
|
Bulten W, Bándi P, Hoven J, Loo RVD, Lotz J, Weiss N, Laak JVD, Ginneken BV, Hulsbergen-van de Kaa C, Litjens G. Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard. Sci Rep 2019; 9:864. [PMID: 30696866 PMCID: PMC6351532 DOI: 10.1038/s41598-018-37257-4] [Citation(s) in RCA: 75] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2018] [Accepted: 12/03/2018] [Indexed: 12/24/2022] Open
Abstract
Given the importance of gland morphology in grading prostate cancer (PCa), automatically differentiating between epithelium and other tissues is an important prerequisite for the development of automated methods for detecting PCa. We propose a new deep learning method to segment epithelial tissue in digitised hematoxylin and eosin (H&E) stained prostatectomy slides using immunohistochemistry (IHC) as reference standard. We used IHC to create a precise and objective ground truth compared to manual outlining on H&E slides, especially in areas with high-grade PCa. 102 tissue sections were stained with H&E and subsequently restained with P63 and CK8/18 IHC markers to highlight epithelial structures. Afterwards each pair was co-registered. First, we trained a U-Net to segment epithelial structures in IHC using a subset of the IHC slides that were preprocessed with color deconvolution. Second, this network was applied to the remaining slides to create the reference standard used to train a second U-Net on H&E. Our system accurately segmented both intact glands and individual tumour epithelial cells. The generalisation capacity of our system is shown using an independent external dataset from a different centre. We envision this segmentation as the first part of a fully automated prostate cancer grading pipeline.
Collapse
Affiliation(s)
- Wouter Bulten
- Radboud University Medical Center, Diagnostic Image Analysis Group and the Department of Pathology, 6500HB, Nijmegen, The Netherlands.
| | - Péter Bándi
- Radboud University Medical Center, Diagnostic Image Analysis Group and the Department of Pathology, 6500HB, Nijmegen, The Netherlands
| | - Jeffrey Hoven
- Radboud University Medical Center, Department of Pathology, 6500HB, Nijmegen, The Netherlands
| | - Rob van de Loo
- Radboud University Medical Center, Department of Pathology, 6500HB, Nijmegen, The Netherlands
| | | | - Nick Weiss
- Fraunhofer MEVIS, 23562, Lübeck, Germany
| | - Jeroen van der Laak
- Radboud University Medical Center, Diagnostic Image Analysis Group and the Department of Pathology, 6500HB, Nijmegen, The Netherlands
| | - Bram van Ginneken
- Radboud University Medical Center, Diagnostic Image Analysis Group and the Department of Radiology and Nuclear Medicine, 6500HB, Nijmegen, The Netherlands
| | | | - Geert Litjens
- Radboud University Medical Center, Diagnostic Image Analysis Group and the Department of Pathology, 6500HB, Nijmegen, The Netherlands
| |
Collapse
|
44
|
Litjens G, Bandi P, Ehteshami Bejnordi B, Geessink O, Balkenhol M, Bult P, Halilovic A, Hermsen M, van de Loo R, Vogels R, Manson QF, Stathonikos N, Baidoshvili A, van Diest P, Wauters C, van Dijk M, van der Laak J. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. Gigascience 2018; 7:5026175. [PMID: 29860392 PMCID: PMC6007545 DOI: 10.1093/gigascience/giy065] [Citation(s) in RCA: 130] [Impact Index Per Article: 21.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2017] [Accepted: 05/22/2018] [Indexed: 12/27/2022] Open
Abstract
Background The presence of lymph node metastases is one of the most important factors in breast cancer prognosis. The most common way to assess regional lymph node status is the sentinel lymph node procedure. The sentinel lymph node is the most likely lymph node to contain metastasized cancer cells and is excised, histopathologically processed, and examined by a pathologist. This tedious examination process is time-consuming and can lead to small metastases being missed. However, recent advances in whole-slide imaging and machine learning have opened an avenue for analysis of digitized lymph node sections with computer algorithms. For example, convolutional neural networks, a type of machine-learning algorithm, can be used to automatically detect cancer metastases in lymph nodes with high accuracy. To train machine-learning models, large, well-curated datasets are needed. Results We released a dataset of 1,399 annotated whole-slide images (WSIs) of lymph nodes, both with and without metastases, in 3 terabytes of data in the context of the CAMELYON16 and CAMELYON17 Grand Challenges. Slides were collected from five medical centers to cover a broad range of image appearance and staining variations. Each WSI has a slide-level label indicating whether it contains no metastases, macro-metastases, micro-metastases, or isolated tumor cells. Furthermore, for 209 WSIs, detailed hand-drawn contours for all metastases are provided. Last, open-source software tools to visualize and interact with the data have been made available. Conclusions A unique dataset of annotated, whole-slide digital histopathology images has been provided with high potential for re-use.
Collapse
Affiliation(s)
- Geert Litjens
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Peter Bandi
- Department of Pathology, University Medical Center Huispost H04.312, Heidelberglaan 100, 3584CX, Utrecht, The Netherlands
| | - Babak Ehteshami Bejnordi
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Oscar Geessink
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Maschenka Balkenhol
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Peter Bult
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Altuna Halilovic
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Meyke Hermsen
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Rob van de Loo
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Rob Vogels
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Quirine F Manson
- Department of Pathology, University Medical Center Huispost H04.312, Heidelberglaan 100, 3584CX, Utrecht, The Netherlands
| | - Nikolas Stathonikos
- Department of Pathology, University Medical Center Huispost H04.312, Heidelberglaan 100, 3584CX, Utrecht, The Netherlands
| | - Alexi Baidoshvili
- Laboratory for Pathology East Netherlands (LabPON), Postbus 516, 7550AM Hengelo, The Netherlands
| | - Paul van Diest
- Department of Pathology, University Medical Center Huispost H04.312, Heidelberglaan 100, 3584CX, Utrecht, The Netherlands
| | - Carla Wauters
- Department of Pathology, Canisius-Wilhelmina Hospital, Postbus 9015, 6500GS Nijmegen, The Netherlands
| | - Marcory van Dijk
- Department of Pathology, Rijnstate Hospital, Pathology-DNA, Postbus 9555, 6800TA Arnhem, The Netherlands
| | - Jeroen van der Laak
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| |
Collapse
|
45
|
Tellez D, Balkenhol M, Otte-Holler I, van de Loo R, Vogels R, Bult P, Wauters C, Vreuls W, Mol S, Karssemeijer N, Litjens G, van der Laak J, Ciompi F. Whole-Slide Mitosis Detection in H&E Breast Histology Using PHH3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks. IEEE Trans Med Imaging 2018; 37:2126-2136. [PMID: 29994086 DOI: 10.1109/tmi.2018.2820199] [Citation(s) in RCA: 117] [Impact Index Per Article: 19.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Manual counting of mitotic tumor cells in tissue sections constitutes one of the strongest prognostic markers for breast cancer. This procedure, however, is time-consuming and error-prone. We developed a method to automatically detect mitotic figures in breast cancer tissue sections based on convolutional neural networks (CNNs). Application of CNNs to hematoxylin and eosin (H&E) stained histological tissue sections is hampered by noisy and expensive reference standards established by pathologists, lack of generalization due to staining variation across laboratories, and high computational requirements needed to process gigapixel whole-slide images (WSIs). In this paper, we present a method to train and evaluate CNNs to specifically solve these issues in the context of mitosis detection in breast cancer WSIs. First, by combining image analysis of mitotic activity in phosphohistone-H3 restained slides and registration, we built a reference standard for mitosis detection in entire H&E WSIs requiring minimal manual annotation effort. Second, we designed a data augmentation strategy that creates diverse and realistic H&E stain variations by modifying H&E color channels directly. Using it during training combined with network ensembling resulted in a stain invariant mitosis detector. Third, we applied knowledge distillation to reduce the computational requirements of the mitosis detection ensemble with a negligible loss of performance. The system was trained in a single-center cohort and evaluated in an independent multicenter cohort from the cancer genome atlas on the three tasks of the tumor proliferation assessment challenge. We obtained a performance within the top three best methods for most of the tasks of the challenge.
Collapse
|
46
|
Smeets XJNM, Litjens G, da Costa DW, Kievit W, van Santvoort HC, Besselink MGH, Fockens P, Bruno MJ, Kolkman JJ, Drenth JPH, Bollen TL, van Geenen EJM. The association between portal system vein diameters and outcomes in acute pancreatitis. Pancreatology 2018; 18:494-499. [PMID: 29784597 DOI: 10.1016/j.pan.2018.05.007] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/26/2018] [Revised: 04/20/2018] [Accepted: 05/14/2018] [Indexed: 02/07/2023]
Abstract
BACKGROUND/OBJECTIVES Acute pancreatitis (AP) progresses to necrotizing pancreatitis in 15% of cases. An important pathophysiological mechanism in AP is third spacing of fluids, which leads to intravascular volume depletion. This results in a reduced splanchnic circulation and reduced venous return. Non-visualisation of the portal and splenic vein on early computed tomography (CT) scan, which might be the result of smaller vein diameter due to decreased venous flow, is associated with infected necrosis and mortality in AP. This observation led us to hypothesize that smaller diameters of portal system veins (portal, splenic and superior mesenteric) are associated with increased severity of AP. METHODS We conducted a post-hoc analysis of data from two randomized controlled trials that included patients with predicted severe and mild AP. The primary endpoint was AP-related mortality. The secondary endpoints were (infected) necrotizing pancreatitis and (persistent) organ failure. We performed additional CT measurements of portal system vein diameters and calculated their prognostic value through univariate and multivariate Poisson regression. RESULTS Multivariate regression showed a significant inverse association between splenic vein diameter and mortality (RR 0.75 (0.59-0.97)). Furthermore, there was a significant inverse association between splenic and superior mesenteric vein diameter and (infected) necrosis. Diameters of all veins were inversely associated with organ failure and persistent organ failure. CONCLUSIONS We observed an inverse relationship between portal system vein diameter and morbidity and an inverse relationship between splenic vein diameter and mortality in AP. Further research is needed to test whether these results can be implemented in predictive scoring systems.
Collapse
Affiliation(s)
- X J N M Smeets
- Department of Gastroenterology and Hepatology, Radboud University Medical Center, Nijmegen, The Netherlands.
| | - G Litjens
- Department of Radiology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - D W da Costa
- Department of Radiology, St. Antonius Hospital, Nieuwegein, The Netherlands
| | - W Kievit
- Department of Health Evidence, Radboud University Medical Center, Nijmegen, The Netherlands
| | - H C van Santvoort
- Department of Surgery, St. Antonius Hospital, Nieuwegein, The Netherlands
| | - M G H Besselink
- Department of Surgery, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands
| | - P Fockens
- Department of Gastroenterology and Hepatology, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands
| | - M J Bruno
- Department of Gastroenterology and Hepatology, Erasmus Medical Center, Rotterdam, The Netherlands
| | - J J Kolkman
- Department of Gastroenterology and Hepatology, Medisch Spectrum Twente, Enschede, The Netherlands
| | - J P H Drenth
- Department of Gastroenterology and Hepatology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - T L Bollen
- Department of Radiology, St. Antonius Hospital, Nieuwegein, The Netherlands
| | - E J M van Geenen
- Department of Gastroenterology and Hepatology, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
47
|
Affiliation(s)
- Babak Ehteshami Bejnordi
- Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| | | |
Collapse
|
48
|
Ehteshami Bejnordi B, Veta M, Johannes van Diest P, van Ginneken B, Karssemeijer N, Litjens G, van der Laak JAWM. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. JAMA 2017; 318:2199-2210. [PMID: 29234806 PMCID: PMC5820737 DOI: 10.1001/jama.2017.14585] [Citation(s) in RCA: 1308] [Impact Index Per Article: 186.9] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/22/2017] [Accepted: 10/26/2017] [Indexed: 02/06/2023]
Abstract
Importance Application of deep learning algorithms to whole-slide pathology images can potentially improve diagnostic accuracy and efficiency. Objective Assess the performance of automated deep learning algorithms at detecting metastases in hematoxylin and eosin-stained tissue sections of lymph nodes of women with breast cancer and compare it with pathologists' diagnoses in a diagnostic setting. Design, Setting, and Participants Researcher challenge competition (CAMELYON16) to develop automated solutions for detecting lymph node metastases (November 2015-November 2016). A training data set of whole-slide images from 2 centers in the Netherlands with (n = 110) and without (n = 160) nodal metastases verified by immunohistochemical staining were provided to challenge participants to build algorithms. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with time constraint (WTC) from the Netherlands to ascertain likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist without time constraint (WOTC). Exposures Deep learning algorithms submitted as part of a challenge competition or pathologist interpretation. Main Outcomes and Measures The presence of specific metastatic foci and the absence vs presence of lymph node metastasis in a slide or image using receiver operating characteristic curve analysis. The 11 pathologists participating in the simulation exercise rated their diagnostic confidence as definitely normal, probably normal, equivocal, probably tumor, or definitely tumor. Results The area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.556 to 0.994. The top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC (72.4% [95% CI, 64.3%-80.4%]) at a mean of 0.0125 false-positives per normal whole-slide image. For the whole-slide image classification task, the best algorithm (AUC, 0.994 [95% CI, 0.983-0.999]) performed significantly better than the pathologists WTC in a diagnostic simulation (mean AUC, 0.810 [range, 0.738-0.884]; P < .001). The top 5 algorithms had a mean AUC that was comparable with the pathologist interpreting the slides in the absence of time constraints (mean AUC, 0.960 [range, 0.923-0.994] for the top 5 algorithms vs 0.966 [95% CI, 0.927-0.998] for the pathologist WOTC). Conclusions and Relevance In the setting of a challenge competition, some deep learning algorithms achieved better diagnostic performance than a panel of 11 pathologists participating in a simulation exercise designed to mimic routine pathology workflow; algorithm performance was comparable with an expert pathologist interpreting whole-slide images without time constraints. Whether this approach has clinical utility will require evaluation in a clinical setting.
Collapse
Affiliation(s)
- Babak Ehteshami Bejnordi
- Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Mitko Veta
- Medical Image Analysis Group, Eindhoven University of Technology, Eindhoven, the Netherlands
| | | | - Bram van Ginneken
- Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Nico Karssemeijer
- Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| | | | | |
Collapse
|
49
|
Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez CI. A survey on deep learning in medical image analysis. Med Image Anal 2017; 42:60-88. [PMID: 28778026 DOI: 10.1016/j.media.2017.07.005] [Citation(s) in RCA: 4147] [Impact Index Per Article: 592.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2017] [Revised: 07/24/2017] [Accepted: 07/25/2017] [Indexed: 02/07/2023]
Affiliation(s)
- Geert Litjens
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands.
| | - Thijs Kooi
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands
| | | | | | - Francesco Ciompi
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Mohsen Ghafoorian
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands
| | | | - Bram van Ginneken
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Clara I Sánchez
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
50
|
Bejnordi BE, Zuidhof G, Balkenhol M, Hermsen M, Bult P, van Ginneken B, Karssemeijer N, Litjens G, van der Laak J. Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images. J Med Imaging (Bellingham) 2017; 4:044504. [PMID: 29285517 PMCID: PMC5729919 DOI: 10.1117/1.jmi.4.4.044504] [Citation(s) in RCA: 83] [Impact Index Per Article: 11.9] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2017] [Accepted: 11/14/2017] [Indexed: 12/31/2022] Open
Abstract
Currently, histopathological tissue examination by a pathologist represents the gold standard for breast lesion diagnostics. Automated classification of histopathological whole-slide images (WSIs) is challenging owing to the wide range of appearances of benign lesions and the visual similarity of ductal carcinoma in-situ (DCIS) to invasive lesions at the cellular level. Consequently, analysis of tissue at high resolutions with a large contextual area is necessary. We present context-aware stacked convolutional neural networks (CNN) for classification of breast WSIs into normal/benign, DCIS, and invasive ductal carcinoma (IDC). We first train a CNN using high pixel resolution to capture cellular level information. The feature responses generated by this model are then fed as input to a second CNN, stacked on top of the first. Training of this stacked architecture with large input patches enables learning of fine-grained (cellular) details and global tissue structures. Our system is trained and evaluated on a dataset containing 221 WSIs of hematoxylin and eosin stained breast tissue specimens. The system achieves an AUC of 0.962 for the binary classification of nonmalignant and malignant slides and obtains a three-class accuracy of 81.3% for classification of WSIs into normal/benign, DCIS, and IDC, demonstrating its potential for routine diagnostics.
Collapse
Affiliation(s)
- Babak Ehteshami Bejnordi
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands
| | - Guido Zuidhof
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| | - Maschenka Balkenhol
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| | - Meyke Hermsen
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| | - Peter Bult
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| | - Bram van Ginneken
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands
| | - Nico Karssemeijer
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands
| | - Geert Litjens
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| | - Jeroen van der Laak
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| |
Collapse
|