1
|
Jurgas A, Wodzinski M, D'Amato M, van der Laak J, Atzori M, Müller H. Improving quality control of whole slide images by explicit artifact augmentation. Sci Rep 2024; 14:17847. [PMID: 39090284 PMCID: PMC11294620 DOI: 10.1038/s41598-024-68667-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2024] [Accepted: 07/26/2024] [Indexed: 08/04/2024] Open
Abstract
The problem of artifacts in whole slide image acquisition, prevalent in both clinical workflows and research-oriented settings, necessitates human intervention and re-scanning. Overcoming this challenge requires developing quality control algorithms, that are hindered by the limited availability of relevant annotated data in histopathology. The manual annotation of ground-truth for artifact detection methods is expensive and time-consuming. This work addresses the issue by proposing a method dedicated to augmenting whole slide images with artifacts. The tool seamlessly generates and blends artifacts from an external library to a given histopathology dataset. The augmented datasets are then utilized to train artifact classification methods. The evaluation shows their usefulness in classification of the artifacts, where they show an improvement from 0.10 to 0.01 AUROC depending on the artifact type. The framework, model, weights, and ground-truth annotations are freely released to facilitate open science and reproducible research.
Collapse
Affiliation(s)
- Artur Jurgas
- AGH University of Krakow, Faculty of Electrical Engineering, Automatics, Computer Science and Biomedical Engineering, 30059, Krakow, Poland.
- University of Applied Sciences Western Switzerland (HES-SO), Institute of Informatics, 3960, Sierre, Switzerland.
| | - Marek Wodzinski
- AGH University of Krakow, Faculty of Electrical Engineering, Automatics, Computer Science and Biomedical Engineering, 30059, Krakow, Poland
- University of Applied Sciences Western Switzerland (HES-SO), Institute of Informatics, 3960, Sierre, Switzerland
| | - Marina D'Amato
- Radboud University Medical Center, Nijmegen, The Netherlands
| | | | - Manfredo Atzori
- University of Applied Sciences Western Switzerland (HES-SO), Institute of Informatics, 3960, Sierre, Switzerland
- Department of Neuroscience, University of Padova, Padua, Italy
| | - Henning Müller
- University of Applied Sciences Western Switzerland (HES-SO), Institute of Informatics, 3960, Sierre, Switzerland
- Medical Faculty, University of Geneva, Geneva, Switzerland
| |
Collapse
|
2
|
Irmakci I, Nateghi R, Zhou R, Vescovo M, Saft M, Ross AE, Yang XJ, Cooper LAD, Goldstein JA. Tissue Contamination Challenges the Credibility of Machine Learning Models in Real World Digital Pathology. Mod Pathol 2024; 37:100422. [PMID: 38185250 PMCID: PMC10960671 DOI: 10.1016/j.modpat.2024.100422] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Revised: 11/13/2023] [Accepted: 12/15/2023] [Indexed: 01/09/2024]
Abstract
Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. Although human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole-slide models. Three operate in placenta for the following functions: (1) detection of decidual arteriopathy, (2) estimation of gestational age, and (3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in the t-distributed stochastic neighbor embedding feature space. Every model showed performance degradation in response to one or more tissue contaminants. Decidual arteriopathy detection--balanced accuracy decreased from 0.74 to 0.69 ± 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant, raised the mean absolute error in estimating gestational age from 1.626 weeks to 2.371 ± 0.003 weeks. Blood, incorporated into placental sections, induced false-negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033 mm2, and resulted in a 97% false-positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.
Collapse
Affiliation(s)
- Ismail Irmakci
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Ramin Nateghi
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Rujoi Zhou
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Mariavittoria Vescovo
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Madeline Saft
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Ashley E Ross
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Ximing J Yang
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Lee A D Cooper
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Jeffery A Goldstein
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois.
| |
Collapse
|
3
|
Jurgas A, Wodzinski M, Celniak W, Atzori M, Muller H. Artifact Augmentation for Learning-based Quality Control of Whole Slide Images. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-4. [PMID: 38082977 DOI: 10.1109/embc40787.2023.10340997] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
The acquisition of whole slide images is prone to artifacts that can require human control and re-scanning, both in clinical workflows and in research-oriented settings. Quality control algorithms are a first step to overcome this challenge, as they limit the use of low quality images. Developing quality control systems in histopathology is not straightforward, also due to the limited availability of data related to this topic. We address the problem by proposing a tool to augment data with artifacts. The proposed method seamlessly generates and blends artifacts from an external library to a given histopathology dataset. The datasets augmented by the blended artifacts are then used to train an artifact detection network in a supervised way. We use the YOLOv5 model for the artifact detection with a slightly modified training pipeline. The proposed tool can be extended into a complete framework for the quality assessment of whole slide images.Clinical relevance- The proposed method may be useful for the initial quality screening of whole slide images. Each year, millions of whole slide images are acquired and digitized worldwide. Numerous of them contain artifacts affecting the following AI-oriented analysis. Therefore, a tool operating at the acquisition phase and improving the initial quality assessment is crucial to increase the performance of digital pathology algorithms, e.g., early cancer diagnosis.
Collapse
|
5
|
Irmakci I, Nateghi R, Zhou R, Ross AE, Yang XJ, Cooper LAD, Goldstein JA. Tissue contamination challenges the credibility of machine learning models in real world digital pathology. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.04.28.23289287. [PMID: 37205404 PMCID: PMC10187357 DOI: 10.1101/2023.04.28.23289287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. While human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole slide models. Three operate in placenta for 1) detection of decidual arteriopathy (DA), 2) estimation of gestational age (GA), and 3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in T-distributed Stochastic Neighbor Embedding (tSNE) feature space. Every model showed performance degradation in response to one or more tissue contaminants. DA detection balanced accuracy decreased from 0.74 to 0.69 +/- 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant raised the mean absolute error in estimating gestation age from 1.626 weeks to 2.371 +/ 0.003 weeks. Blood, incorporated into placental sections, induced false negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033mm2, resulted in a 97% false positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Jeffery A. Goldstein
- To whom correspondence should be addressed: Olson 2-455, 710 N. Fairbanks Ave, Chicago IL, 60611,
| |
Collapse
|
6
|
Homeyer A, Geißler C, Schwen LO, Zakrzewski F, Evans T, Strohmenger K, Westphal M, Bülow RD, Kargl M, Karjauv A, Munné-Bertran I, Retzlaff CO, Romero-López A, Sołtysiński T, Plass M, Carvalho R, Steinbach P, Lan YC, Bouteldja N, Haber D, Rojas-Carulla M, Vafaei Sadr A, Kraft M, Krüger D, Fick R, Lang T, Boor P, Müller H, Hufnagl P, Zerbe N. Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology. Mod Pathol 2022; 35:1759-1769. [PMID: 36088478 PMCID: PMC9708586 DOI: 10.1038/s41379-022-01147-y] [Citation(s) in RCA: 24] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2022] [Revised: 07/24/2022] [Accepted: 07/25/2022] [Indexed: 12/24/2022]
Abstract
Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recommendations are missing. A committee of various stakeholders, including commercial AI developers, pathologists, and researchers, discussed key aspects and conducted extensive literature reviews on test datasets in pathology. Here, we summarize the results and derive general recommendations on compiling test datasets. We address several questions: Which and how many images are needed? How to deal with low-prevalence subsets? How can potential bias be detected? How should datasets be reported? What are the regulatory requirements in different countries? The recommendations are intended to help AI developers demonstrate the utility of their products and to help pathologists and regulatory agencies verify reported performance measures. Further research is needed to formulate criteria for sufficiently representative test datasets so that AI solutions can operate with less user intervention and better support diagnostic workflows in the future.
Collapse
Affiliation(s)
- André Homeyer
- Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359, Bremen, Germany.
| | - Christian Geißler
- grid.6734.60000 0001 2292 8254Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587 Berlin, Germany
| | - Lars Ole Schwen
- grid.428590.20000 0004 0496 8246Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359 Bremen, Germany
| | - Falk Zakrzewski
- grid.412282.f0000 0001 1091 2917Institute of Pathology, Carl Gustav Carus University Hospital Dresden (UKD), TU Dresden (TUD), Fetscherstrasse 74, 01307 Dresden, Germany
| | - Theodore Evans
- grid.6734.60000 0001 2292 8254Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587 Berlin, Germany
| | - Klaus Strohmenger
- grid.6363.00000 0001 2218 4662Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117 Berlin, Germany
| | - Max Westphal
- grid.428590.20000 0004 0496 8246Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359 Bremen, Germany
| | - Roman David Bülow
- grid.412301.50000 0000 8653 1507Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Michaela Kargl
- grid.11598.340000 0000 8988 2476Medical University of Graz, Diagnostic and Research Center for Molecular BioMedicine, Diagnostic & Research Institute of Pathology, Neue Stiftingtalstrasse 6, 8010 Graz, Austria
| | - Aray Karjauv
- grid.6734.60000 0001 2292 8254Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587 Berlin, Germany
| | - Isidre Munné-Bertran
- MoticEurope, S.L.U., C. Les Corts, 12 Poligono Industrial, 08349 Barcelona, Spain
| | - Carl Orge Retzlaff
- grid.6734.60000 0001 2292 8254Technische Universität Berlin, DAI-Labor, Ernst-Reuter-Platz 7, 10587 Berlin, Germany
| | | | | | - Markus Plass
- grid.11598.340000 0000 8988 2476Medical University of Graz, Diagnostic and Research Center for Molecular BioMedicine, Diagnostic & Research Institute of Pathology, Neue Stiftingtalstrasse 6, 8010 Graz, Austria
| | - Rita Carvalho
- grid.6363.00000 0001 2218 4662Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117 Berlin, Germany
| | - Peter Steinbach
- grid.40602.300000 0001 2158 0612Helmholtz-Zentrum Dresden Rossendorf, Bautzner Landstraße 400, 01328 Dresden, Germany
| | - Yu-Chia Lan
- grid.412301.50000 0000 8653 1507Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Nassim Bouteldja
- grid.412301.50000 0000 8653 1507Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | - David Haber
- Lakera AI AG, Zelgstrasse 7, 8003 Zürich, Switzerland
| | | | - Alireza Vafaei Sadr
- grid.412301.50000 0000 8653 1507Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | | | - Daniel Krüger
- grid.474385.90000 0004 4676 7928Olympus Soft Imaging Solutions GmbH, Johann-Krane-Weg 39, 48149 Münster, Germany
| | - Rutger Fick
- Tribun Health, 2 Rue du Capitaine Scott, 75015 Paris, France
| | - Tobias Lang
- Mindpeak GmbH, Zirkusweg 2, 20359 Hamburg, Germany
| | - Peter Boor
- grid.412301.50000 0000 8653 1507Institute of Pathology, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Heimo Müller
- grid.11598.340000 0000 8988 2476Medical University of Graz, Diagnostic and Research Center for Molecular BioMedicine, Diagnostic & Research Institute of Pathology, Neue Stiftingtalstrasse 6, 8010 Graz, Austria
| | - Peter Hufnagl
- grid.6363.00000 0001 2218 4662Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117 Berlin, Germany
| | - Norman Zerbe
- grid.6363.00000 0001 2218 4662Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Institute of Pathology, Charitéplatz 1, 10117 Berlin, Germany
| |
Collapse
|