1
|
Linmans J, Raya G, van der Laak J, Litjens G. Diffusion models for out-of-distribution detection in digital pathology. Med Image Anal 2024; 93:103088. [PMID: 38228075 DOI: 10.1016/j.media.2024.103088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2023] [Revised: 12/29/2023] [Accepted: 01/10/2024] [Indexed: 01/18/2024]
Abstract
The ability to detect anomalies, i.e. anything not seen during training or out-of-distribution (OOD), in medical imaging applications is essential for successfully deploying machine learning systems. Filtering out OOD data using unsupervised learning is especially promising because it does not require costly annotations. A new class of models called AnoDDPMs, based on denoising diffusion probabilistic models (DDPMs), has recently achieved significant progress in unsupervised OOD detection. This work provides a benchmark for unsupervised OOD detection methods in digital pathology. By leveraging fast sampling techniques, we apply AnoDDPM on a large enough scale for whole-slide image analysis on the complete test set of the Camelyon16 challenge. Based on ROC analysis, we show that AnoDDPMs can detect OOD data with an AUC of up to 94.13 and 86.93 on two patch-level OOD detection tasks, outperforming the other unsupervised methods. We observe that AnoDDPMs alter the semantic properties of inputs, replacing anomalous data with more benign-looking tissue. Furthermore, we highlight the flexibility of AnoDDPM towards different information bottlenecks by evaluating reconstruction errors for inputs with different signal-to-noise ratios. While there is still a significant performance gap with fully supervised learning, AnoDDPMs show considerable promise in the field of OOD detection in digital pathology.
Collapse
Affiliation(s)
- Jasper Linmans
- Department of Pathology, RadboudUMC Graduate School, Radboud University Medical Center, Nijmegen, The Netherlands.
| | - Gabriel Raya
- Jheronimus Academy of Data Science, 's-Hertogenbosch, The Netherlands
| | - Jeroen van der Laak
- Department of Pathology, RadboudUMC Graduate School, Radboud University Medical Center, Nijmegen, The Netherlands; Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | - Geert Litjens
- Department of Pathology, RadboudUMC Graduate School, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
2
|
Linmans J, Hoogeboom E, van der Laak J, Litjens G. The Latent Doctor Model for Modeling Inter-Observer Variability. IEEE J Biomed Health Inform 2023; PP:1-12. [PMID: 37831571 DOI: 10.1109/jbhi.2023.3323582] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/15/2023]
Abstract
Many inherently ambiguous tasks in medical imaging suffer from inter-observer variability, resulting in a reference standard defined by a distribution of labels with high variance. Training only on a consensus or majority vote label, as is common in medical imaging, discards valuable information on uncertainty amongst a panel of experts. In this work, we propose to train on the full label distribution to predict the uncertainty within a panel of experts and the most likely ground-truth label. To do so, we propose a new stochastic classification framework based on the conditional variational auto-encoder, which we refer to as the Latent Doctor Model (LDM). In an extensive comparative analysis, we compare the LDM with a model trained on the majority vote label and other methods capable of learning a distribution of labels. We show that the LDM is able to reproduce the reference-standard distribution significantly better than the majority vote baseline. Compared to the other baseline methods, we demonstrate that the LDM performs best at modeling the label distribution and its corresponding uncertainty in two prostate tumor grading tasks. Furthermore, we show competitive performance of the LDM with the more computationally demanding deep ensembles on a tumor budding classification task.
Collapse
|
3
|
Linmans J, Elfwing S, van der Laak J, Litjens G. Predictive uncertainty estimation for out-of-distribution detection in digital pathology. Med Image Anal 2023; 83:102655. [PMID: 36306568 DOI: 10.1016/j.media.2022.102655] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2022] [Revised: 09/26/2022] [Accepted: 10/12/2022] [Indexed: 11/06/2022]
Abstract
Machine learning model deployment in clinical practice demands real-time risk assessment to identify situations in which the model is uncertain. Once deployed, models should be accurate for classes seen during training while providing informative estimates of uncertainty to flag abnormalities and unseen classes for further analysis. Although recent developments in uncertainty estimation have resulted in an increasing number of methods, a rigorous empirical evaluation of their performance on large-scale digital pathology datasets is lacking. This work provides a benchmark for evaluating prevalent methods on multiple datasets by comparing the uncertainty estimates on both in-distribution and realistic near and far out-of-distribution (OOD) data on a whole-slide level. To this end, we aggregate uncertainty values from patch-based classifiers to whole-slide level uncertainty scores. We show that results found in classical computer vision benchmarks do not always translate to the medical imaging setting. Specifically, we demonstrate that deep ensembles perform best at detecting far-OOD data but can be outperformed on a more challenging near-OOD detection task by multi-head ensembles trained for optimal ensemble diversity. Furthermore, we demonstrate the harmful impact OOD data can have on the performance of deployed machine learning models. Overall, we show that uncertainty estimates can be used to discriminate in-distribution from OOD data with high AUC scores. Still, model deployment might require careful tuning based on prior knowledge of prospective OOD data.
Collapse
Affiliation(s)
- Jasper Linmans
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands.
| | | | - Jeroen van der Laak
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands; Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | - Geert Litjens
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
4
|
Bogaerts J, Van Bommel M, Linmans J, Van den Hork N, Bulten J, De Hullu J, Simons M, Van der Laak J. 1009 Deep learning for improved detection of premalignant lesions in the fallopian tube, a proof of concept. Pathology 2021. [DOI: 10.1136/ijgc-2021-esgo.533] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/03/2022]
|
5
|
Hermsen M, Volk V, Bräsen JH, Geijs DJ, Gwinner W, Kers J, Linmans J, Schaadt NS, Schmitz J, Steenbergen EJ, Swiderska-Chadaj Z, Smeets B, Hilbrands LB, Feuerhake F, van der Laak JAWM. Quantitative assessment of inflammatory infiltrates in kidney transplant biopsies using multiplex tyramide signal amplification and deep learning. J Transl Med 2021; 101:970-982. [PMID: 34006891 PMCID: PMC8292146 DOI: 10.1038/s41374-021-00601-w] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2021] [Revised: 03/09/2021] [Accepted: 03/11/2021] [Indexed: 12/20/2022] Open
Abstract
Delayed graft function (DGF) is a strong risk factor for development of interstitial fibrosis and tubular atrophy (IFTA) in kidney transplants. Quantitative assessment of inflammatory infiltrates in kidney biopsies of DGF patients can reveal predictive markers for IFTA development. In this study, we combined multiplex tyramide signal amplification (mTSA) and convolutional neural networks (CNNs) to assess the inflammatory microenvironment in kidney biopsies of DGF patients (n = 22) taken at 6 weeks post-transplantation. Patients were stratified for IFTA development (<10% versus ≥10%) from 6 weeks to 6 months post-transplantation, based on histopathological assessment by three kidney pathologists. One mTSA panel was developed for visualization of capillaries, T- and B-lymphocytes and macrophages and a second mTSA panel for T-helper cell and macrophage subsets. The slides were multi spectrally imaged and custom-made python scripts enabled conversion to artificial brightfield whole-slide images (WSI). We used an existing CNN for the detection of lymphocytes with cytoplasmatic staining patterns in immunohistochemistry and developed two new CNNs for the detection of macrophages and nuclear-stained lymphocytes. F1-scores were 0.77 (nuclear-stained lymphocytes), 0.81 (cytoplasmatic-stained lymphocytes), and 0.82 (macrophages) on a test set of artificial brightfield WSI. The CNNs were used to detect inflammatory cells, after which we assessed the peritubular capillary extent, cell density, cell ratios, and cell distance in the two patient groups. In this cohort, distance of macrophages to other immune cells and peritubular capillary extent did not vary significantly at 6 weeks post-transplantation between patient groups. CD163+ cell density was higher in patients with ≥10% IFTA development 6 months post-transplantation (p < 0.05). CD3+CD8-/CD3+CD8+ ratios were higher in patients with <10% IFTA development (p < 0.05). We observed a high correlation between CD163+ and CD4+GATA3+ cell density (R = 0.74, p < 0.001). Our study demonstrates that CNNs can be used to leverage reliable, quantitative results from mTSA-stained, multi spectrally imaged slides of kidney transplant biopsies.
Collapse
Affiliation(s)
- Meyke Hermsen
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Valery Volk
- Institute for Pathology, Hannover Medical School, Hannover, Germany
| | | | - Daan J Geijs
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Wilfried Gwinner
- Department of Nephrology, Hannover Medical School, Hannover, Germany
| | - Jesper Kers
- Department of Pathology, Amsterdam University Medical Centers, Amsterdam, The Netherlands
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands
- Center for Analytical Sciences Amsterdam (CASA), Van 't Hoff Institute for Molecular Sciences (HIMS), University of Amsterdam, Amsterdam, The Netherlands
| | - Jasper Linmans
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Nadine S Schaadt
- Institute of Diagnostic and Interventional Neuroradiology, Hannover Medical School, Hannover, Germany
| | - Jessica Schmitz
- Institute for Pathology, Hannover Medical School, Hannover, Germany
| | - Eric J Steenbergen
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Zaneta Swiderska-Chadaj
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
- Faculty of Electrical Engineering, Warsaw University of Technology, Warsaw, Poland
| | - Bart Smeets
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Luuk B Hilbrands
- Department of Nephrology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Friedrich Feuerhake
- Institute for Pathology, Hannover Medical School, Hannover, Germany
- Institute for Neuropathology, University Clinic Freiburg, Freiburg, Germany
| | - Jeroen A W M van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands.
- Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden.
| |
Collapse
|
6
|
Abstract
OBJECTIVE To evaluate the significance of thickening of the wall of the renal collecting system by US. MATERIALS AND METHODS Wall thickening of the renal collecting system was seen during US in 62 collecting systems of 51 patients over a period of 2 years. The medical and radiological records of these patients were reviewed with special attention to the definitive diagnosis and other clinical and radiological parameters. Moreover, a control group consisting of 48 renal collecting systems was examined to establish normal values for the thickness of the wall of the collecting system. RESULTS Of the 62 collecting systems (mean wall thickness 1.6 mm, range 0.8-3.1 mm), vesicoureteric reflux (VUR) was present in 18 cases, urinary tract infection (UTI) in 11, and both VUR and UTI in 9 cases. In 10 cases, intermittent dilatation was present caused by primary obstructive megaureter (n = 2), pelvi-ureteric junction stenosis (n = 4), high-pressure bladder (n = 3), or of unknown cause (n = 1). In 11 cases, transient dilatation had been present in the recent past (usually prenatally detected hydronephrosis), but had disappeared at the time of the US examination. In 3 patients, no definite cause for the wall thickening could be established. In the control group, wall thickness ranged from 0.1 to 0.8 mm. CONCLUSIONS The upper limit for wall thickness of the normal collecting system in children is 0.8 mm. Thickening of the wall of more than 0.8 mm should be considered as pathological and is caused by urinary tract infection, intermittent dilatation (e. g., VUR), and dilatation in the recent past.
Collapse
Affiliation(s)
- S G Robben
- Department of Paediatric Radiology, University Hospital Rotterdam, Sophia Children's Hospital, Dr. Molewaterplein 60, 3015 GJ Rotterdam, The Netherlands
| | | | | | | | | |
Collapse
|