1
|
Hekler A, Maron RC, Haggenmüller S, Schmitt M, Wies C, Utikal JS, Meier F, Hobelsberger S, Gellrich FF, Sergon M, Hauschild A, French LE, Heinzerling L, Schlager JG, Ghoreschi K, Schlaak M, Hilke FJ, Poch G, Korsing S, Berking C, Heppt MV, Erdmann M, Haferkamp S, Drexler K, Schadendorf D, Sondermann W, Goebeler M, Schilling B, Kather JN, Krieghoff-Henning E, Brinker TJ. Using multiple real-world dermoscopic photographs of one lesion improves melanoma classification via deep learning. J Am Acad Dermatol 2024; 90:1028-1031. [PMID: 38199280 DOI: 10.1016/j.jaad.2023.11.065] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2023] [Revised: 10/22/2023] [Accepted: 11/27/2023] [Indexed: 01/12/2024]
Affiliation(s)
- Achim Hekler
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Roman C Maron
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sarah Haggenmüller
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Max Schmitt
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christoph Wies
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany; Medical Faculty, University Heidelberg, Heidelberg, Germany
| | - Jochen S Utikal
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karl University of Heidelberg, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany; DKFZ Hector Cancer Institute at the University Medical Center Mannheim, Mannheim, Germany
| | - Friedegund Meier
- Department of Dermatology, Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Sarah Hobelsberger
- Department of Dermatology, Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Frank F Gellrich
- Department of Dermatology, Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Mildred Sergon
- Department of Dermatology, Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital (UKSH), Kiel, Germany
| | - Lars E French
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany; Dr. Phillip Frost Department of Dermatology and Cutaneous Surgery, University of Miami, Miller School of Medicine, Miami, Florida
| | - Lucie Heinzerling
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany; Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Justin G Schlager
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Max Schlaak
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Franz J Hilke
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Gabriela Poch
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Sören Korsing
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Carola Berking
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Markus V Heppt
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Michael Erdmann
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Konstantin Drexler
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Dirk Schadendorf
- Department of Dermatology, Venereology and Allergology, University Hospital Essen, Essen, Germany
| | - Wiebke Sondermann
- Department of Dermatology, Venereology and Allergology, University Hospital Essen, Essen, Germany
| | - Matthias Goebeler
- Department of Dermatology, Venereology and Allergology, University Hospital Würzburg and National Center for Tumor Diseases (NCT) WERA Würzburg, Würzburg, Germany
| | - Bastian Schilling
- Department of Dermatology, Venereology and Allergology, University Hospital Würzburg and National Center for Tumor Diseases (NCT) WERA Würzburg, Würzburg, Germany
| | - Jakob N Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
2
|
Haggenmüller S, Maron RC, Hekler A, Krieghoff-Henning E, Utikal JS, Gaiser M, Müller V, Fabian S, Meier F, Hobelsberger S, Gellrich FF, Sergon M, Hauschild A, Weichenthal M, French LE, Heinzerling L, Schlager JG, Ghoreschi K, Schlaak M, Hilke FJ, Poch G, Korsing S, Berking C, Heppt MV, Erdmann M, Haferkamp S, Drexler K, Schadendorf D, Sondermann W, Goebeler M, Schilling B, Kather JN, Fröhling S, Kaminski K, Doppler A, Bucher T, Brinker TJ. Patients' and dermatologists' preferences in AI-driven skin cancer diagnostics: prospective multicentric survey study. J Am Acad Dermatol 2024:S0190-9622(24)00649-2. [PMID: 38670313 DOI: 10.1016/j.jaad.2024.04.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2023] [Revised: 03/12/2024] [Accepted: 04/03/2024] [Indexed: 04/28/2024]
Affiliation(s)
- Sarah Haggenmüller
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Roman C Maron
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jochen S Utikal
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karls University of Heidelberg, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany; DKFZ Hector Cancer Institute at the University Medical Center Mannheim, Mannheim, Germany
| | - Maria Gaiser
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karls University of Heidelberg, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany; DKFZ Hector Cancer Institute at the University Medical Center Mannheim, Mannheim, Germany
| | - Verena Müller
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karls University of Heidelberg, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany; DKFZ Hector Cancer Institute at the University Medical Center Mannheim, Mannheim, Germany
| | - Sascha Fabian
- Department of Economics, University of Applied Science Neu-Ulm, Neu-Ulm, Germany
| | - Friedegund Meier
- Department of Dermatology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany and Skin Cancer Center at the University Cancer Centre Dresden and National Center for Tumor Diseases, Dresden, Germany
| | - Sarah Hobelsberger
- Department of Dermatology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany and Skin Cancer Center at the University Cancer Centre Dresden and National Center for Tumor Diseases, Dresden, Germany
| | - Frank F Gellrich
- Department of Dermatology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany and Skin Cancer Center at the University Cancer Centre Dresden and National Center for Tumor Diseases, Dresden, Germany
| | - Mildred Sergon
- Department of Dermatology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany and Skin Cancer Center at the University Cancer Centre Dresden and National Center for Tumor Diseases, Dresden, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital (UKSH), Kiel, Germany
| | | | - Lars E French
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany; Dr. Phillip Frost Department of Dermatology and Cutaneous Surgery, University of Miami, Miller School of Medicine, Miami, FL, USA
| | - Lucie Heinzerling
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany; Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Justin G Schlager
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Max Schlaak
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Franz J Hilke
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Gabriela Poch
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Sören Korsing
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Carola Berking
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Markus V Heppt
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Michael Erdmann
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Konstantin Drexler
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen and German Cancer Consortium, partner site Essen and National Center for Tumor Diseases (NCT), NCT-West, Campus Essen and University Alliance Ruhr, Research Center One Health, University Duisburg-Essen, Essen, Germany
| | - Wiebke Sondermann
- Department of Dermatology, University Hospital Essen, Essen and German Cancer Consortium, partner site Essen and National Center for Tumor Diseases (NCT), NCT-West, Campus Essen and University Alliance Ruhr, Research Center One Health, University Duisburg-Essen, Essen, Germany
| | - Matthias Goebeler
- Department of Dermatology, Venereology and Allergology, University Hospital Würzburg and National Center for Tumor Diseases (NCT) WERA Würzburg, Germany
| | - Bastian Schilling
- Department of Dermatology, Venereology and Allergology, University Hospital Würzburg and National Center for Tumor Diseases (NCT) WERA Würzburg, Germany
| | - Jakob N Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Stefan Fröhling
- Department of Translational Medical Oncology, National Center for Tumor Diseases (NCT) Heidelberg and German Cancer Research Center (DKFZ), Heidelberg, Germany
| | | | | | - Tabea Bucher
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
3
|
Haggenmüller S, Schmitt M, Krieghoff-Henning E, Hekler A, Maron RC, Wies C, Utikal JS, Meier F, Hobelsberger S, Gellrich FF, Sergon M, Hauschild A, French LE, Heinzerling L, Schlager JG, Ghoreschi K, Schlaak M, Hilke FJ, Poch G, Korsing S, Berking C, Heppt MV, Erdmann M, Haferkamp S, Drexler K, Schadendorf D, Sondermann W, Goebeler M, Schilling B, Kather JN, Fröhling S, Brinker TJ. Federated Learning for Decentralized Artificial Intelligence in Melanoma Diagnostics. JAMA Dermatol 2024; 160:303-311. [PMID: 38324293 PMCID: PMC10851139 DOI: 10.1001/jamadermatol.2023.5550] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2023] [Accepted: 09/01/2023] [Indexed: 02/08/2024]
Abstract
Importance The development of artificial intelligence (AI)-based melanoma classifiers typically calls for large, centralized datasets, requiring hospitals to give away their patient data, which raises serious privacy concerns. To address this concern, decentralized federated learning has been proposed, where classifier development is distributed across hospitals. Objective To investigate whether a more privacy-preserving federated learning approach can achieve comparable diagnostic performance to a classical centralized (ie, single-model) and ensemble learning approach for AI-based melanoma diagnostics. Design, Setting, and Participants This multicentric, single-arm diagnostic study developed a federated model for melanoma-nevus classification using histopathological whole-slide images prospectively acquired at 6 German university hospitals between April 2021 and February 2023 and benchmarked it using both a holdout and an external test dataset. Data analysis was performed from February to April 2023. Exposures All whole-slide images were retrospectively analyzed by an AI-based classifier without influencing routine clinical care. Main Outcomes and Measures The area under the receiver operating characteristic curve (AUROC) served as the primary end point for evaluating the diagnostic performance. Secondary end points included balanced accuracy, sensitivity, and specificity. Results The study included 1025 whole-slide images of clinically melanoma-suspicious skin lesions from 923 patients, consisting of 388 histopathologically confirmed invasive melanomas and 637 nevi. The median (range) age at diagnosis was 58 (18-95) years for the training set, 57 (18-93) years for the holdout test dataset, and 61 (18-95) years for the external test dataset; the median (range) Breslow thickness was 0.70 (0.10-34.00) mm, 0.70 (0.20-14.40) mm, and 0.80 (0.30-20.00) mm, respectively. The federated approach (0.8579; 95% CI, 0.7693-0.9299) performed significantly worse than the classical centralized approach (0.9024; 95% CI, 0.8379-0.9565) in terms of AUROC on a holdout test dataset (pairwise Wilcoxon signed-rank, P < .001) but performed significantly better (0.9126; 95% CI, 0.8810-0.9412) than the classical centralized approach (0.9045; 95% CI, 0.8701-0.9331) on an external test dataset (pairwise Wilcoxon signed-rank, P < .001). Notably, the federated approach performed significantly worse than the ensemble approach on both the holdout (0.8867; 95% CI, 0.8103-0.9481) and external test dataset (0.9227; 95% CI, 0.8941-0.9479). Conclusions and Relevance The findings of this diagnostic study suggest that federated learning is a viable approach for the binary classification of invasive melanomas and nevi on a clinically representative distributed dataset. Federated learning can improve privacy protection in AI-based melanoma diagnostics while simultaneously promoting collaboration across institutions and countries. Moreover, it may have the potential to be extended to other image classification tasks in digital cancer histopathology and beyond.
Collapse
Affiliation(s)
- Sarah Haggenmüller
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Max Schmitt
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Roman C. Maron
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christoph Wies
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jochen S. Utikal
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karls University of Heidelberg, Mannheim, Germany
- Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
- DKFZ Hector Cancer Institute at the University Medical Center Mannheim, Mannheim, Germany
| | - Friedegund Meier
- Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Sarah Hobelsberger
- Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Frank F. Gellrich
- Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Mildred Sergon
- Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital (UKSH), Kiel, Germany
| | - Lars E. French
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
- Dr Phillip Frost Department of Dermatology and Cutaneous Surgery, Miller School of Medicine, University of Miami, Miami, Florida
| | - Lucie Heinzerling
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen–European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Justin G. Schlager
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité–Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Max Schlaak
- Department of Dermatology, Venereology and Allergology, Charité–Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Franz J. Hilke
- Department of Dermatology, Venereology and Allergology, Charité–Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Gabriela Poch
- Department of Dermatology, Venereology and Allergology, Charité–Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Sören Korsing
- Department of Dermatology, Venereology and Allergology, Charité–Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Carola Berking
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen–European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Markus V. Heppt
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen–European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Michael Erdmann
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen–European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Konstantin Drexler
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Dirk Schadendorf
- Department of Dermatology, Venereology and Allergology, University Hospital Essen, Essen, Germany
| | - Wiebke Sondermann
- Department of Dermatology, Venereology and Allergology, University Hospital Essen, Essen, Germany
| | - Matthias Goebeler
- Department of Dermatology, Venereology and Allergology, University Hospital Würzburg and National Center for Tumor Diseases (NCT) WERA, Würzburg, Germany
| | - Bastian Schilling
- Department of Dermatology, Venereology and Allergology, University Hospital Würzburg and National Center for Tumor Diseases (NCT) WERA, Würzburg, Germany
| | - Jakob N. Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Stefan Fröhling
- Department of Translational Medical Oncology, National Center for Tumor Diseases (NCT) Heidelberg and German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J. Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
4
|
Maron RC, Hekler A, Haggenmüller S, von Kalle C, Utikal JS, Müller V, Gaiser M, Meier F, Hobelsberger S, Gellrich FF, Sergon M, Hauschild A, French LE, Heinzerling L, Schlager JG, Ghoreschi K, Schlaak M, Hilke FJ, Poch G, Korsing S, Berking C, Heppt MV, Erdmann M, Haferkamp S, Schadendorf D, Sondermann W, Goebeler M, Schilling B, Kather JN, Fröhling S, Lipka DB, Krieghoff-Henning E, Brinker TJ. Model soups improve performance of dermoscopic skin cancer classifiers. Eur J Cancer 2022; 173:307-316. [PMID: 35973360 DOI: 10.1016/j.ejca.2022.07.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Accepted: 07/04/2022] [Indexed: 11/20/2022]
Abstract
BACKGROUND Image-based cancer classifiers suffer from a variety of problems which negatively affect their performance. For example, variation in image brightness or different cameras can already suffice to diminish performance. Ensemble solutions, where multiple model predictions are combined into one, can improve these problems. However, ensembles are computationally intensive and less transparent to practitioners than single model solutions. Constructing model soups, by averaging the weights of multiple models into a single model, could circumvent these limitations while still improving performance. OBJECTIVE To investigate the performance of model soups for a dermoscopic melanoma-nevus skin cancer classification task with respect to (1) generalisation to images from other clinics, (2) robustness against small image changes and (3) calibration such that the confidences correspond closely to the actual predictive uncertainties. METHODS We construct model soups by fine-tuning pre-trained models on seven different image resolutions and subsequently averaging their weights. Performance is evaluated on a multi-source dataset including holdout and external components. RESULTS We find that model soups improve generalisation and calibration on the external component while maintaining performance on the holdout component. For robustness, we observe performance improvements for pertubated test images, while the performance on corrupted test images remains on par. CONCLUSIONS Overall, souping for skin cancer classifiers has a positive effect on generalisation, robustness and calibration. It is easy for practitioners to implement and by combining multiple models into a single model, complexity is reduced. This could be an important factor in achieving clinical applicability, as less complexity generally means more transparency.
Collapse
Affiliation(s)
- Roman C Maron
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sarah Haggenmüller
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Charité University Medicine and Berlin Institute of Health (BIH), Berlin, Germany
| | - Jochen S Utikal
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karl University of Heidelberg, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany; DKFZ Hector Cancer Institute at the University Medical Center Mannheim, Mannheim, Germany
| | - Verena Müller
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karl University of Heidelberg, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany; DKFZ Hector Cancer Institute at the University Medical Center Mannheim, Mannheim, Germany
| | - Maria Gaiser
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karl University of Heidelberg, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany; DKFZ Hector Cancer Institute at the University Medical Center Mannheim, Mannheim, Germany
| | - Friedegund Meier
- Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Sarah Hobelsberger
- Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Frank F Gellrich
- Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Mildred Sergon
- Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital (UKSH), Kiel, Germany
| | - Lars E French
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany; Dr. Phillip Frost Department of Dermatology and Cutaneous Surgery, University of Miami, Miller School of Medicine, Miami, FL, USA
| | - Lucie Heinzerling
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
| | - Justin G Schlager
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Max Schlaak
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Franz J Hilke
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Gabriela Poch
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Sören Korsing
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Carola Berking
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Markus V Heppt
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Michael Erdmann
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - European Metropolitan Region Nürnberg, CCC Alliance WERA, Erlangen, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Dirk Schadendorf
- Department of Dermatology, Venereology and Allergology, University Hospital Essen, Essen, Germany
| | - Wiebke Sondermann
- Department of Dermatology, Venereology and Allergology, University Hospital Essen, Essen, Germany
| | - Matthias Goebeler
- Department of Dermatology, Venereology and Allergology, University Hospital Würzburg, Würzburg, Germany
| | - Bastian Schilling
- Department of Dermatology, Venereology and Allergology, University Hospital Würzburg, Würzburg, Germany
| | - Jakob N Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Stefan Fröhling
- Department of Translational Medical Oncology, National Center for Tumor Diseases (NCT) Heidelberg and German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Daniel B Lipka
- Department of Translational Medical Oncology, National Center for Tumor Diseases (NCT) Heidelberg and German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
5
|
Kurz A, Hauser K, Mehrtens HA, Krieghoff-Henning E, Hekler A, Kather JN, Fröhling S, von Kalle C, Brinker TJ. Uncertainty Estimation in Medical Image Classification: Systematic Review. JMIR Med Inform 2022; 10:e36427. [PMID: 35916701 PMCID: PMC9382553 DOI: 10.2196/36427] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2022] [Revised: 04/11/2022] [Accepted: 06/04/2022] [Indexed: 01/26/2023] Open
Abstract
Background Deep neural networks are showing impressive results in different medical image classification tasks. However, for real-world applications, there is a need to estimate the network’s uncertainty together with its prediction. Objective In this review, we investigate in what form uncertainty estimation has been applied to the task of medical image classification. We also investigate which metrics are used to describe the effectiveness of the applied uncertainty estimation Methods Google Scholar, PubMed, IEEE Xplore, and ScienceDirect were screened for peer-reviewed studies, published between 2016 and 2021, that deal with uncertainty estimation in medical image classification. The search terms “uncertainty,” “uncertainty estimation,” “network calibration,” and “out-of-distribution detection” were used in combination with the terms “medical images,” “medical image analysis,” and “medical image classification.” Results A total of 22 papers were chosen for detailed analysis through the systematic review process. This paper provides a table for a systematic comparison of the included works with respect to the applied method for estimating the uncertainty. Conclusions The applied methods for estimating uncertainties are diverse, but the sampling-based methods Monte-Carlo Dropout and Deep Ensembles are used most frequently. We concluded that future works can investigate the benefits of uncertainty estimation in collaborative settings of artificial intelligence systems and human experts. International Registered Report Identifier (IRRID) RR2-10.2196/11936
Collapse
Affiliation(s)
- Alexander Kurz
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Katja Hauser
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | | | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | | | - Stefan Fröhling
- Department of Translational Medical Oncology, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Berlin Institute of Health (BIH), Berlin, Germany
| | - Titus Josef Brinker
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
6
|
Schneider L, Krieghoff-Henning E, Laiouar-Pedari S, Kuntz S, Hekler A, Kather JN, Gaiser T, Fröhling S, Brinker TJ. Response to letter entitled: Re: Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. Eur J Cancer 2022; 172:403-404. [PMID: 35781181 DOI: 10.1016/j.ejca.2022.06.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2022] [Accepted: 06/01/2022] [Indexed: 11/15/2022]
Affiliation(s)
- Lucas Schneider
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sara Laiouar-Pedari
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sara Kuntz
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jakob N Kather
- Department of Medicine III, RWTH Aachen University Hospital, Aachen, Germany; Medical Oncology, National Center for Tumor Diseases, University Hospital Heidelberg, Heidelberg, Germany
| | - Timo Gaiser
- Institute of Pathology, University Medical Centre Mannheim, University of Heidelberg, Mannheim, Germany
| | - Stefan Fröhling
- Translational Medical Oncology, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
7
|
Hauser K, Kurz A, Haggenmüller S, Maron RC, von Kalle C, Utikal JS, Meier F, Hobelsberger S, Gellrich FF, Sergon M, Hauschild A, French LE, Heinzerling L, Schlager JG, Ghoreschi K, Schlaak M, Hilke FJ, Poch G, Kutzner H, Berking C, Heppt MV, Erdmann M, Haferkamp S, Schadendorf D, Sondermann W, Goebeler M, Schilling B, Kather JN, Fröhling S, Lipka DB, Hekler A, Krieghoff-Henning E, Brinker TJ. Explainable artificial intelligence in skin cancer recognition: A systematic review. Eur J Cancer 2022; 167:54-69. [PMID: 35390650 DOI: 10.1016/j.ejca.2022.02.025] [Citation(s) in RCA: 22] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2022] [Revised: 02/22/2022] [Accepted: 02/24/2022] [Indexed: 01/18/2023]
Abstract
BACKGROUND Due to their ability to solve complex problems, deep neural networks (DNNs) are becoming increasingly popular in medical applications. However, decision-making by such algorithms is essentially a black-box process that renders it difficult for physicians to judge whether the decisions are reliable. The use of explainable artificial intelligence (XAI) is often suggested as a solution to this problem. We investigate how XAI is used for skin cancer detection: how is it used during the development of new DNNs? What kinds of visualisations are commonly used? Are there systematic evaluations of XAI with dermatologists or dermatopathologists? METHODS Google Scholar, PubMed, IEEE Explore, Science Direct and Scopus were searched for peer-reviewed studies published between January 2017 and October 2021 applying XAI to dermatological images: the search terms histopathological image, whole-slide image, clinical image, dermoscopic image, skin, dermatology, explainable, interpretable and XAI were used in various combinations. Only studies concerned with skin cancer were included. RESULTS 37 publications fulfilled our inclusion criteria. Most studies (19/37) simply applied existing XAI methods to their classifier to interpret its decision-making. Some studies (4/37) proposed new XAI methods or improved upon existing techniques. 14/37 studies addressed specific questions such as bias detection and impact of XAI on man-machine-interactions. However, only three of them evaluated the performance and confidence of humans using CAD systems with XAI. CONCLUSION XAI is commonly applied during the development of DNNs for skin cancer detection. However, a systematic and rigorous evaluation of its usefulness in this scenario is lacking.
Collapse
Affiliation(s)
- Katja Hauser
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Alexander Kurz
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sarah Haggenmüller
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Roman C Maron
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Charité University Medicine and Berlin Institute of Health (BIH), Berlin, Germany
| | - Jochen S Utikal
- Department of Dermatology, Heidelberg University, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Friedegund Meier
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Sarah Hobelsberger
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Frank F Gellrich
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Mildred Sergon
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital (UKSH), Kiel, Germany
| | - Lars E French
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany; Dr. Phillip Frost Department of Dermatology and Cutaneous Surgery, University of Miami, Miller School of Medicine, Miami, FL, USA
| | - Lucie Heinzerling
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
| | - Justin G Schlager
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Max Schlaak
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Franz J Hilke
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Gabriela Poch
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Heinz Kutzner
- Dermatopathology Laboratory, Friedrichshafen, Germany
| | - Carola Berking
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - EMN, Friedrich-Alexander University Erlangen, Nuremberg, Germany
| | - Markus V Heppt
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - EMN, Friedrich-Alexander University Erlangen, Nuremberg, Germany
| | - Michael Erdmann
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - EMN, Friedrich-Alexander University Erlangen, Nuremberg, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Wiebke Sondermann
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Matthias Goebeler
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Bastian Schilling
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Jakob N Kather
- Division of Translational Medical Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Stefan Fröhling
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Daniel B Lipka
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
8
|
Schneider L, Laiouar-Pedari S, Kuntz S, Krieghoff-Henning E, Hekler A, Kather JN, Gaiser T, Fröhling S, Brinker TJ. Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review. Eur J Cancer 2021; 160:80-91. [PMID: 34810047 DOI: 10.1016/j.ejca.2021.10.007] [Citation(s) in RCA: 24] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2021] [Accepted: 10/11/2021] [Indexed: 02/07/2023]
Abstract
BACKGROUND Over the past decade, the development of molecular high-throughput methods (omics) increased rapidly and provided new insights for cancer research. In parallel, deep learning approaches revealed the enormous potential for medical image analysis, especially in digital pathology. Combining image and omics data with deep learning tools may enable the discovery of new cancer biomarkers and a more precise prediction of patient prognosis. This systematic review addresses different multimodal fusion methods of convolutional neural network-based image analyses with omics data, focussing on the impact of data combination on the classification performance. METHODS PubMed was screened for peer-reviewed articles published in English between January 2015 and June 2021 by two independent researchers. Search terms related to deep learning, digital pathology, omics, and multimodal fusion were combined. RESULTS We identified a total of 11 studies meeting the inclusion criteria, namely studies that used convolutional neural networks for haematoxylin and eosin image analysis of patients with cancer in combination with integrated omics data. Publications were categorised according to their endpoints: 7 studies focused on survival analysis and 4 studies on prediction of cancer subtypes, malignancy or microsatellite instability with spatial analysis. CONCLUSIONS Image-based classifiers already show high performances in prognostic and predictive cancer diagnostics. The integration of omics data led to improved performance in all studies described here. However, these are very early studies that still require external validation to demonstrate their generalisability and robustness. Further and more comprehensive studies with larger sample sizes are needed to evaluate performance and determine clinical benefits.
Collapse
Affiliation(s)
- Lucas Schneider
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sara Laiouar-Pedari
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sara Kuntz
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jakob N Kather
- Department of Medicine III, RWTH Aachen University Hospital, Aachen, Germany; Medical Oncology, National Center for Tumour Diseases, University Hospital Heidelberg, Heidelberg, Germany
| | - Timo Gaiser
- Institute of Pathology, University Medical Centre Mannheim, University of Heidelberg, Mannheim, Germany
| | - Stefan Fröhling
- Translational Medical Oncology, National Center for Tumour Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
9
|
Kuntz S, Krieghoff-Henning E, Kather JN, Jutzi T, Höhn J, Kiehl L, Hekler A, Alwers E, von Kalle C, Fröhling S, Utikal JS, Brenner H, Hoffmeister M, Brinker TJ. Gastrointestinal cancer classification and prognostication from histology using deep learning: Systematic review. Eur J Cancer 2021; 155:200-215. [PMID: 34391053 DOI: 10.1016/j.ejca.2021.07.012] [Citation(s) in RCA: 47] [Impact Index Per Article: 15.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2021] [Accepted: 07/06/2021] [Indexed: 02/07/2023]
Abstract
BACKGROUND Gastrointestinal cancers account for approximately 20% of all cancer diagnoses and are responsible for 22.5% of cancer deaths worldwide. Artificial intelligence-based diagnostic support systems, in particular convolutional neural network (CNN)-based image analysis tools, have shown great potential in medical computer vision. In this systematic review, we summarise recent studies reporting CNN-based approaches for digital biomarkers for characterization and prognostication of gastrointestinal cancer pathology. METHODS Pubmed and Medline were screened for peer-reviewed papers dealing with CNN-based gastrointestinal cancer analyses from histological slides, published between 2015 and 2020.Seven hundred and ninety titles and abstracts were screened, and 58 full-text articles were assessed for eligibility. RESULTS Sixteen publications fulfilled our inclusion criteria dealing with tumor or precursor lesion characterization or prognostic and predictive biomarkers: 14 studies on colorectal or rectal cancer, three studies on gastric cancer and none on esophageal cancer. These studies were categorised according to their end-points: polyp characterization, tumor characterization and patient outcome. Regarding the translation into clinical practice, we identified several studies demonstrating generalization of the classifier with external tests and comparisons with pathologists, but none presenting clinical implementation. CONCLUSIONS Results of recent studies on CNN-based image analysis in gastrointestinal cancer pathology are promising, but studies were conducted in observational and retrospective settings. Large-scale trials are needed to assess performance and predict clinical usefulness. Furthermore, large-scale trials are required for approval of CNN-based prediction models as medical devices.
Collapse
Affiliation(s)
- Sara Kuntz
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jakob N Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Tanja Jutzi
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Julia Höhn
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Lennard Kiehl
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Elizabeth Alwers
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Charité University Medicine and Berlin Institute of Health (BIH), Berlin, Germany
| | - Stefan Fröhling
- Department of Translational Medical Oncology, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jochen S Utikal
- Department of Dermatology, Heidelberg University, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Hermann Brenner
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany; Division of Preventive Oncology, German Cancer Research Center (DKFZ), National Center for Tumor Diseases (NCT), Heidelberg, Germany; German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
10
|
Brinker TJ, Kiehl L, Schmitt M, Jutzi TB, Krieghoff-Henning EI, Krahl D, Kutzner H, Gholam P, Haferkamp S, Klode J, Schadendorf D, Hekler A, Fröhling S, Kather JN, Haggenmüller S, von Kalle C, Heppt M, Hilke F, Ghoreschi K, Tiemann M, Wehkamp U, Hauschild A, Weichenthal M, Utikal JS. Deep learning approach to predict sentinel lymph node status directly from routine histology of primary melanoma tumours. Eur J Cancer 2021; 154:227-234. [PMID: 34298373 DOI: 10.1016/j.ejca.2021.05.026] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2021] [Revised: 05/18/2021] [Accepted: 05/20/2021] [Indexed: 12/28/2022]
Abstract
AIM Sentinel lymph node status is a central prognostic factor for melanomas. However, the surgical excision involves some risks for affected patients. In this study, we therefore aimed to develop a digital biomarker that can predict lymph node metastasis non-invasively from digitised H&E slides of primary melanoma tumours. METHODS A total of 415 H&E slides from primary melanoma tumours with known sentinel node (SN) status from three German university hospitals and one private pathological practice were digitised (150 SN positive/265 SN negative). Two hundred ninety-one slides were used to train artificial neural networks (ANNs). The remaining 124 slides were used to test the ability of the ANNs to predict sentinel status. ANNs were trained and/or tested on data sets that were matched or not matched between SN-positive and SN-negative cases for patient age, ulceration, and tumour thickness, factors that are known to correlate with lymph node status. RESULTS The best accuracy was achieved by an ANN that was trained and tested on unmatched cases (61.8% ± 0.2%) area under the receiver operating characteristic (AUROC). In contrast, ANNs that were trained and/or tested on matched cases achieved (55.0% ± 3.5%) AUROC or less. CONCLUSION Our results indicate that the image classifier can predict lymph node status to some, albeit so far not clinically relevant, extent. It may do so by mostly detecting equivalents of factors on histological slides that are already known to correlate with lymph node status. Our results provide a basis for future research with larger data cohorts.
Collapse
Affiliation(s)
- Titus J Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany.
| | - Lennard Kiehl
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Max Schmitt
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Tanja B Jutzi
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Eva I Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Dieter Krahl
- Private Laboratory of Dermatohistopathology, Mönchhofstraße 52, 69120, Heidelberg, Germany
| | - Heinz Kutzner
- Dermatopathology Laboratory, Friedrichshafen, Germany
| | - Patrick Gholam
- Department of Dermatology, University Hospital Heidelberg, Heidelberg. Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Joachim Klode
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Stefan Fröhling
- Translational Medical Oncology, German Cancer Research Center (DKFZ), National Center for Tumor Diseases (NCT), 69120, Heidelberg, Germany
| | - Jakob N Kather
- Translational Medical Oncology, German Cancer Research Center (DKFZ), National Center for Tumor Diseases (NCT), 69120, Heidelberg, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Sarah Haggenmüller
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Charité University Medicine and Berlin Institute of Health (BIH), Berlin, Germany
| | - Markus Heppt
- Department of Dermatology, University Hospital Erlangen, Erlangen, Germany
| | - Franz Hilke
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin, Berlin, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin, Berlin, Germany
| | | | - Ulrike Wehkamp
- Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Axel Hauschild
- Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | | | - Jochen S Utikal
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karl University of Heidelberg, Mannheim, Germany; Department of Dermatology, University Hospital (UKSH), Kiel, Germany
| |
Collapse
|
11
|
Höhn J, Hekler A, Krieghoff-Henning E, Kather JN, Utikal JS, Meier F, Gellrich FF, Hauschild A, French L, Schlager JG, Ghoreschi K, Wilhelm T, Kutzner H, Heppt M, Haferkamp S, Sondermann W, Schadendorf D, Schilling B, Maron RC, Schmitt M, Jutzi T, Fröhling S, Lipka DB, Brinker TJ. Integrating Patient Data Into Skin Cancer Classification Using Convolutional Neural Networks: Systematic Review. J Med Internet Res 2021; 23:e20708. [PMID: 34255646 PMCID: PMC8285747 DOI: 10.2196/20708] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2020] [Revised: 10/29/2020] [Accepted: 04/13/2021] [Indexed: 11/17/2022] Open
Abstract
Background Recent years have been witnessing a substantial improvement in the accuracy of skin cancer classification using convolutional neural networks (CNNs). CNNs perform on par with or better than dermatologists with respect to the classification tasks of single images. However, in clinical practice, dermatologists also use other patient data beyond the visual aspects present in a digitized image, further increasing their diagnostic accuracy. Several pilot studies have recently investigated the effects of integrating different subtypes of patient data into CNN-based skin cancer classifiers. Objective This systematic review focuses on the current research investigating the impact of merging information from image features and patient data on the performance of CNN-based skin cancer image classification. This study aims to explore the potential in this field of research by evaluating the types of patient data used, the ways in which the nonimage data are encoded and merged with the image features, and the impact of the integration on the classifier performance. Methods Google Scholar, PubMed, MEDLINE, and ScienceDirect were screened for peer-reviewed studies published in English that dealt with the integration of patient data within a CNN-based skin cancer classification. The search terms skin cancer classification, convolutional neural network(s), deep learning, lesions, melanoma, metadata, clinical information, and patient data were combined. Results A total of 11 publications fulfilled the inclusion criteria. All of them reported an overall improvement in different skin lesion classification tasks with patient data integration. The most commonly used patient data were age, sex, and lesion location. The patient data were mostly one-hot encoded. There were differences in the complexity that the encoded patient data were processed with regarding deep learning methods before and after fusing them with the image features for a combined classifier. Conclusions This study indicates the potential benefits of integrating patient data into CNN-based diagnostic algorithms. However, how exactly the individual patient data enhance classification performance, especially in the case of multiclass classification problems, is still unclear. Moreover, a substantial fraction of patient data used by dermatologists remains to be analyzed in the context of CNN-based skin cancer classification. Further exploratory analyses in this promising field may optimize patient data integration into CNN-based skin cancer diagnostics for patients’ benefits.
Collapse
Affiliation(s)
- Julia Höhn
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jakob Nikolas Kather
- Department of Medicine III, RWTH University Hospital Aachen, Aachen, Germany.,National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jochen Sven Utikal
- Department of Dermatology, University Hospital of Mannheim, Mannheim, Germany.,Skin Cancer Unit, German Cancer Research Center, Heidelberg, Germany
| | - Friedegund Meier
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Frank Friedrich Gellrich
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital of Kiel, Kiel, Germany
| | - Lars French
- Department of Dermatology and Allergology, Ludwig Maximilian University of Munich, Munich, Germany
| | - Justin Gabriel Schlager
- Department of Dermatology and Allergology, Ludwig Maximilian University of Munich, Munich, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Tabea Wilhelm
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Heinz Kutzner
- Dermatopathology Laboratory, Friedrichshafen, Germany
| | - Markus Heppt
- Department of Dermatology, University Hospital Erlangen, Erlangen, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital of Regensburg, Regensburg, Germany
| | - Wiebke Sondermann
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Bastian Schilling
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Roman C Maron
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Max Schmitt
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Tanja Jutzi
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Stefan Fröhling
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany.,Translational Cancer Epigenomics, Division of Translational Medical Oncology, German Cancer Research Center, Heidelberg, Germany
| | - Daniel B Lipka
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany.,Translational Cancer Epigenomics, Division of Translational Medical Oncology, German Cancer Research Center, Heidelberg, Germany.,Faculty of Medicine, Medical Center, Otto-von-Guericke-University, Magdeburg, Germany
| | - Titus Josef Brinker
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
12
|
Wessels F, Schmitt M, Krieghoff-Henning E, Jutzi T, Worst TS, Waldbillig F, Neuberger M, Maron RC, Steeg M, Gaiser T, Hekler A, Utikal JS, von Kalle C, Fröhling S, Michel MS, Nuhn P, Brinker TJ. Deep learning approach to predict lymph node metastasis directly from primary tumour histology in prostate cancer. BJU Int 2021; 128:352-360. [PMID: 33706408 DOI: 10.1111/bju.15386] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/31/2022]
Abstract
OBJECTIVE To develop a new digital biomarker based on the analysis of primary tumour tissue by a convolutional neural network (CNN) to predict lymph node metastasis (LNM) in a cohort matched for already established risk factors. PATIENTS AND METHODS Haematoxylin and eosin (H&E) stained primary tumour slides from 218 patients (102 N+; 116 N0), matched for Gleason score, tumour size, venous invasion, perineural invasion and age, who underwent radical prostatectomy were selected to train a CNN and evaluate its ability to predict LN status. RESULTS With 10 models trained with the same data, a mean area under the receiver operating characteristic curve (AUROC) of 0.68 (95% confidence interval [CI] 0.678-0.682) and a mean balanced accuracy of 61.37% (95% CI 60.05-62.69%) was achieved. The mean sensitivity and specificity was 53.09% (95% CI 49.77-56.41%) and 69.65% (95% CI 68.21-71.1%), respectively. These results were confirmed via cross-validation. The probability score for LNM prediction was significantly higher on image sections from N+ samples (mean [SD] N+ probability score 0.58 [0.17] vs 0.47 [0.15] N0 probability score, P = 0.002). In multivariable analysis, the probability score of the CNN (odds ratio [OR] 1.04 per percentage probability, 95% CI 1.02-1.08; P = 0.04) and lymphovascular invasion (OR 11.73, 95% CI 3.96-35.7; P < 0.001) proved to be independent predictors for LNM. CONCLUSION In our present study, CNN-based image analyses showed promising results as a potential novel low-cost method to extract relevant prognostic information directly from H&E histology to predict the LN status of patients with prostate cancer. Our ubiquitously available technique might contribute to an improved LN status prediction.
Collapse
Affiliation(s)
- Frederik Wessels
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany.,Department of Urology and Urological Surgery, Medical Faculty Mannheim of Heidelberg University, University Medical Center Mannheim, Mannheim, Germany
| | - Max Schmitt
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Tanja Jutzi
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Thomas S Worst
- Department of Urology and Urological Surgery, Medical Faculty Mannheim of Heidelberg University, University Medical Center Mannheim, Mannheim, Germany
| | - Frank Waldbillig
- Department of Urology and Urological Surgery, Medical Faculty Mannheim of Heidelberg University, University Medical Center Mannheim, Mannheim, Germany
| | - Manuel Neuberger
- Department of Urology and Urological Surgery, Medical Faculty Mannheim of Heidelberg University, University Medical Center Mannheim, Mannheim, Germany
| | - Roman C Maron
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Matthias Steeg
- Institute of Pathology, Medical Faculty Mannheim of Heidelberg University, University Medical Center Mannheim, Mannheim, Germany
| | - Timo Gaiser
- Institute of Pathology, Medical Faculty Mannheim of Heidelberg University, University Medical Center Mannheim, Mannheim, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jochen S Utikal
- Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany.,Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, University of Heidelberg, Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Berlin Institute of Health (BIH), Charité University Medicine, Berlin, Germany
| | - Stefan Fröhling
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Maurice S Michel
- Department of Urology and Urological Surgery, Medical Faculty Mannheim of Heidelberg University, University Medical Center Mannheim, Mannheim, Germany
| | - Philipp Nuhn
- Department of Urology and Urological Surgery, Medical Faculty Mannheim of Heidelberg University, University Medical Center Mannheim, Mannheim, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
13
|
Höhn J, Krieghoff-Henning E, Jutzi TB, von Kalle C, Utikal JS, Meier F, Gellrich FF, Hobelsberger S, Hauschild A, Schlager JG, French L, Heinzerling L, Schlaak M, Ghoreschi K, Hilke FJ, Poch G, Kutzner H, Heppt MV, Haferkamp S, Sondermann W, Schadendorf D, Schilling B, Goebeler M, Hekler A, Fröhling S, Lipka DB, Kather JN, Krahl D, Ferrara G, Haggenmüller S, Brinker TJ. Combining CNN-based histologic whole slide image analysis and patient data to improve skin cancer classification. Eur J Cancer 2021; 149:94-101. [PMID: 33838393 DOI: 10.1016/j.ejca.2021.02.032] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2021] [Revised: 02/12/2021] [Accepted: 02/13/2021] [Indexed: 02/07/2023]
Abstract
BACKGROUND Clinicians and pathologists traditionally use patient data in addition to clinical examination to support their diagnoses. OBJECTIVES We investigated whether a combination of histologic whole slides image (WSI) analysis based on convolutional neural networks (CNNs) and commonly available patient data (age, sex and anatomical site of the lesion) in a binary melanoma/nevus classification task could increase the performance compared with CNNs alone. METHODS We used 431 WSIs from two different laboratories and analysed the performance of classifiers that used the image or patient data individually or three common fusion techniques. Furthermore, we tested a naive combination of patient data and an image classifier: for cases interpreted as 'uncertain' (CNN output score <0.7), the decision of the CNN was replaced by the decision of the patient data classifier. RESULTS The CNN on its own achieved the best performance (mean ± standard deviation of five individual runs) with AUROC of 92.30% ± 0.23% and balanced accuracy of 83.17% ± 0.38%. While the classification performance was not significantly improved in general by any of the tested fusions, naive strategy of replacing the image classifier with the patient data classifier on slides with low output scores improved balanced accuracy to 86.72% ± 0.36%. CONCLUSION In most cases, the CNN on its own was so accurate that patient data integration did not provide any benefit. However, incorporating patient data for lesions that were classified by the CNN with low 'confidence' improved balanced accuracy.
Collapse
Affiliation(s)
- Julia Höhn
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Tanja B Jutzi
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Charité University Medicine and Berlin Institute of Health (BIH), Berlin, Germany
| | - Jochen S Utikal
- Department of Dermatology, Heidelberg University, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Friedegund Meier
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität, Dresden, Germany
| | - Frank F Gellrich
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität, Dresden, Germany
| | - Sarah Hobelsberger
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität, Dresden, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital of Kiel, Germany
| | - Justin G Schlager
- Department of Dermatology and Allergology, Ludwig Maximilian University of Munich, Munich, Germany
| | - Lars French
- Department of Dermatology and Allergology, Ludwig Maximilian University of Munich, Munich, Germany
| | - Lucie Heinzerling
- Department of Dermatology and Allergology, Ludwig Maximilian University of Munich, Munich, Germany
| | - Max Schlaak
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Franz J Hilke
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Gabriela Poch
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Heinz Kutzner
- Dermatopathology Laboratory, Friedrichshafen, Germany
| | - Markus V Heppt
- Department of Dermatology, University Hospital Erlangen, Erlangen, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Wiebke Sondermann
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Bastian Schilling
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Matthias Goebeler
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Stefan Fröhling
- Section Translational Cancer Epigenomics, Division Translational Medical Oncology, German Cancer Research Center (DKFZ) & National Center for Tumor Diseases (NCT), Heidelberg, 69120, Germany
| | - Daniel B Lipka
- Section Translational Cancer Epigenomics, Division Translational Medical Oncology, German Cancer Research Center (DKFZ) & National Center for Tumor Diseases (NCT), Heidelberg, 69120, Germany
| | - Jakob N Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Dieter Krahl
- Private Laboratory of Dermatohistopathology, Mönchhofstraße 52, Heidelberg, 69120, Germany
| | - Gerardo Ferrara
- Anatomic Pathology Unit, Macerata General Hospital, Macerata, Italy
| | - Sarah Haggenmüller
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany.
| |
Collapse
|
14
|
Maron RC, Hekler A, Krieghoff-Henning E, Schmitt M, Schlager JG, Utikal JS, Brinker TJ. Reducing the Impact of Confounding Factors on Skin Cancer Classification via Image Segmentation: Technical Model Study. J Med Internet Res 2021; 23:e21695. [PMID: 33764307 PMCID: PMC8074854 DOI: 10.2196/21695] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2020] [Revised: 11/13/2020] [Accepted: 02/08/2021] [Indexed: 01/23/2023] Open
Abstract
Background Studies have shown that artificial intelligence achieves similar or better performance than dermatologists in specific dermoscopic image classification tasks. However, artificial intelligence is susceptible to the influence of confounding factors within images (eg, skin markings), which can lead to false diagnoses of cancerous skin lesions. Image segmentation can remove lesion-adjacent confounding factors but greatly change the image representation. Objective The aim of this study was to compare the performance of 2 image classification workflows where images were either segmented or left unprocessed before the subsequent training and evaluation of a binary skin lesion classifier. Methods Separate binary skin lesion classifiers (nevus vs melanoma) were trained and evaluated on segmented and unsegmented dermoscopic images. For a more informative result, separate classifiers were trained on 2 distinct training data sets (human against machine [HAM] and International Skin Imaging Collaboration [ISIC]). Each training run was repeated 5 times. The mean performance of the 5 runs was evaluated on a multi-source test set (n=688) consisting of a holdout and an external component. Results Our findings showed that when trained on HAM, the segmented classifiers showed a higher overall balanced accuracy (75.6% [SD 1.1%]) than the unsegmented classifiers (66.7% [SD 3.2%]), which was significant in 4 out of 5 runs (P<.001). The overall balanced accuracy was numerically higher for the unsegmented ISIC classifiers (78.3% [SD 1.8%]) than for the segmented ISIC classifiers (77.4% [SD 1.5%]), which was significantly different in 1 out of 5 runs (P=.004). Conclusions Image segmentation does not result in overall performance decrease but it causes the beneficial removal of lesion-adjacent confounding factors. Thus, it is a viable option to address the negative impact that confounding factors have on deep learning models in dermatology. However, the segmentation step might introduce new pitfalls, which require further investigations.
Collapse
Affiliation(s)
- Roman C Maron
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Max Schmitt
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Justin G Schlager
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
| | - Jochen S Utikal
- Department of Dermatology, Heidelberg University, Mannheim, Germany.,Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
15
|
Schmitt M, Maron RC, Hekler A, Stenzinger A, Hauschild A, Weichenthal M, Tiemann M, Krahl D, Kutzner H, Utikal JS, Haferkamp S, Kather JN, Klauschen F, Krieghoff-Henning E, Fröhling S, von Kalle C, Brinker TJ. Hidden Variables in Deep Learning Digital Pathology and Their Potential to Cause Batch Effects: Prediction Model Study. J Med Internet Res 2021; 23:e23436. [PMID: 33528370 PMCID: PMC7886613 DOI: 10.2196/23436] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2020] [Revised: 10/14/2020] [Accepted: 12/06/2020] [Indexed: 01/21/2023] Open
Abstract
BACKGROUND An increasing number of studies within digital pathology show the potential of artificial intelligence (AI) to diagnose cancer using histological whole slide images, which requires large and diverse data sets. While diversification may result in more generalizable AI-based systems, it can also introduce hidden variables. If neural networks are able to distinguish/learn hidden variables, these variables can introduce batch effects that compromise the accuracy of classification systems. OBJECTIVE The objective of the study was to analyze the learnability of an exemplary selection of hidden variables (patient age, slide preparation date, slide origin, and scanner type) that are commonly found in whole slide image data sets in digital pathology and could create batch effects. METHODS We trained four separate convolutional neural networks (CNNs) to learn four variables using a data set of digitized whole slide melanoma images from five different institutes. For robustness, each CNN training and evaluation run was repeated multiple times, and a variable was only considered learnable if the lower bound of the 95% confidence interval of its mean balanced accuracy was above 50.0%. RESULTS A mean balanced accuracy above 50.0% was achieved for all four tasks, even when considering the lower bound of the 95% confidence interval. Performance between tasks showed wide variation, ranging from 56.1% (slide preparation date) to 100% (slide origin). CONCLUSIONS Because all of the analyzed hidden variables are learnable, they have the potential to create batch effects in dermatopathology data sets, which negatively affect AI-based classification systems. Practitioners should be aware of these and similar pitfalls when developing and evaluating such systems and address these and potentially other batch effect variables in their data sets through sufficient data set stratification.
Collapse
Affiliation(s)
- Max Schmitt
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Roman Christoph Maron
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Albrecht Stenzinger
- Institute of Pathology, University Hospital Heidelberg, University of Heidelberg, Heidelberg, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital Kiel, University of Kiel, Kiel, Germany
| | - Michael Weichenthal
- Department of Dermatology, University Hospital Kiel, University of Kiel, Kiel, Germany
| | | | - Dieter Krahl
- Private Institute of Dermatopathology, Heidelberg, Germany
| | - Heinz Kutzner
- Private Institute of Dermatopathology, Friedrichshafen, Germany
| | - Jochen Sven Utikal
- Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany.,Department of Dermatology, University Medical Center Mannheim, University of Heidelberg, Mannheim, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital of Regensburg, Regensburg, Germany
| | | | | | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Stefan Fröhling
- National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Charité and Berlin Institute of Health, Berlin, Germany
| | - Titus Josef Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
16
|
Kutzner H, Jutzi TB, Krahl D, Krieghoff‐Henning EI, Heppt MV, Hekler A, Schmitt M, Maron RCR, Fröhling S, Kalle C, Brinker TJ. Überdiagnose von Melanomen – Ursachen, Konsequenzen und Lösungsansätze. J Dtsch Dermatol Ges 2020; 18:1236-1244. [DOI: 10.1111/ddg.14233_g] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2019] [Accepted: 04/13/2020] [Indexed: 11/28/2022]
Affiliation(s)
| | - Tanja B. Jutzi
- Nachwuchsgruppe Digitale Biomarker für die Onkologie (DBO), Nationales Centrum für Tumorerkrankungen (NCT) Deutsches Krebsforschungszentrum (DKFZ) Heidelberg
| | - Dieter Krahl
- Privates Labor für Dermatohistopathologie Mönchhofstraße 52 Heidelberg
| | - Eva I. Krieghoff‐Henning
- Nachwuchsgruppe Digitale Biomarker für die Onkologie (DBO), Nationales Centrum für Tumorerkrankungen (NCT) Deutsches Krebsforschungszentrum (DKFZ) Heidelberg
| | | | - Achim Hekler
- Nachwuchsgruppe Digitale Biomarker für die Onkologie (DBO), Nationales Centrum für Tumorerkrankungen (NCT) Deutsches Krebsforschungszentrum (DKFZ) Heidelberg
| | - Max Schmitt
- Nachwuchsgruppe Digitale Biomarker für die Onkologie (DBO), Nationales Centrum für Tumorerkrankungen (NCT) Deutsches Krebsforschungszentrum (DKFZ) Heidelberg
| | - Roman C. R. Maron
- Nachwuchsgruppe Digitale Biomarker für die Onkologie (DBO), Nationales Centrum für Tumorerkrankungen (NCT) Deutsches Krebsforschungszentrum (DKFZ) Heidelberg
| | - Stefan Fröhling
- Nachwuchsgruppe Digitale Biomarker für die Onkologie (DBO), Nationales Centrum für Tumorerkrankungen (NCT) Deutsches Krebsforschungszentrum (DKFZ) Heidelberg
| | - Christof Kalle
- Berlin Institute of Health (BIH) und Charité – Universitätsmedizin Berlin
| | - Titus J. Brinker
- Nachwuchsgruppe Digitale Biomarker für die Onkologie (DBO), Nationales Centrum für Tumorerkrankungen (NCT) Deutsches Krebsforschungszentrum (DKFZ) Heidelberg
| |
Collapse
|
17
|
Maron RC, Utikal JS, Hekler A, Hauschild A, Sattler E, Sondermann W, Haferkamp S, Schilling B, Heppt MV, Jansen P, Reinholz M, Franklin C, Schmitt L, Hartmann D, Krieghoff-Henning E, Schmitt M, Weichenthal M, von Kalle C, Fröhling S, Brinker TJ. Artificial Intelligence and Its Effect on Dermatologists' Accuracy in Dermoscopic Melanoma Image Classification: Web-Based Survey Study. J Med Internet Res 2020; 22:e18091. [PMID: 32915161 PMCID: PMC7519424 DOI: 10.2196/18091] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2020] [Revised: 03/28/2020] [Accepted: 05/14/2020] [Indexed: 01/17/2023] Open
Abstract
BACKGROUND Early detection of melanoma can be lifesaving but this remains a challenge. Recent diagnostic studies have revealed the superiority of artificial intelligence (AI) in classifying dermoscopic images of melanoma and nevi, concluding that these algorithms should assist a dermatologist's diagnoses. OBJECTIVE The aim of this study was to investigate whether AI support improves the accuracy and overall diagnostic performance of dermatologists in the dichotomous image-based discrimination between melanoma and nevus. METHODS Twelve board-certified dermatologists were presented disjoint sets of 100 unique dermoscopic images of melanomas and nevi (total of 1200 unique images), and they had to classify the images based on personal experience alone (part I) and with the support of a trained convolutional neural network (CNN, part II). Additionally, dermatologists were asked to rate their confidence in their final decision for each image. RESULTS While the mean specificity of the dermatologists based on personal experience alone remained almost unchanged (70.6% vs 72.4%; P=.54) with AI support, the mean sensitivity and mean accuracy increased significantly (59.4% vs 74.6%; P=.003 and 65.0% vs 73.6%; P=.002, respectively) with AI support. Out of the 10% (10/94; 95% CI 8.4%-11.8%) of cases where dermatologists were correct and AI was incorrect, dermatologists on average changed to the incorrect answer for 39% (4/10; 95% CI 23.2%-55.6%) of cases. When dermatologists were incorrect and AI was correct (25/94, 27%; 95% CI 24.0%-30.1%), dermatologists changed their answers to the correct answer for 46% (11/25; 95% CI 33.1%-58.4%) of cases. Additionally, the dermatologists' average confidence in their decisions increased when the CNN confirmed their decision and decreased when the CNN disagreed, even when the dermatologists were correct. Reported values are based on the mean of all participants. Whenever absolute values are shown, the denominator and numerator are approximations as every dermatologist ended up rating a varying number of images due to a quality control step. CONCLUSIONS The findings of our study show that AI support can improve the overall accuracy of the dermatologists in the dichotomous image-based discrimination between melanoma and nevus. This supports the argument for AI-based tools to aid clinicians in skin lesion classification and provides a rationale for studies of such classifiers in real-life settings, wherein clinicians can integrate additional information such as patient age and medical history into their decisions.
Collapse
Affiliation(s)
- Roman C Maron
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jochen S Utikal
- Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany.,Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, University of Heidelberg, Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital Kiel, University of Kiel, Kiel, Germany
| | - Elke Sattler
- Department of Dermatology, University Hospital Munich (LMU), Munich, Germany
| | - Wiebke Sondermann
- Department of Dermatology, University Hospital Essen, University of Duisburg-Essen, Essen, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Bastian Schilling
- Department of Dermatology, University Hospital Würzburg, University of Würzburg, Würzburg, Germany
| | - Markus V Heppt
- Department of Dermatology, University Hospital Erlangen, University of Erlangen, Erlangen, Germany
| | - Philipp Jansen
- Department of Dermatology, University Hospital Essen, University of Duisburg-Essen, Essen, Germany
| | - Markus Reinholz
- Department of Dermatology, University Hospital Munich (LMU), Munich, Germany
| | - Cindy Franklin
- Department of Dermatology, University Hospital Cologne, Cologne, Germany
| | - Laurenz Schmitt
- Department of Dermatology, University Hospital Aachen, Aachen, Germany
| | - Daniela Hartmann
- Department of Dermatology, University Hospital Munich (LMU), Munich, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Max Schmitt
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Michael Weichenthal
- Department of Dermatology, University Hospital Kiel, University of Kiel, Kiel, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Charité University Medicine and Berlin Institute of Health (BIH), Berlin, Germany
| | - Stefan Fröhling
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
18
|
Kutzner H, Jutzi TB, Krahl D, Krieghoff-Henning EI, Heppt MV, Hekler A, Schmitt M, Maron RCR, Fröhling S, von Kalle C, Brinker TJ. Overdiagnosis of melanoma - causes, consequences and solutions. J Dtsch Dermatol Ges 2020; 18:1236-1243. [PMID: 32841508 DOI: 10.1111/ddg.14233] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2019] [Accepted: 04/13/2020] [Indexed: 12/14/2022]
Abstract
Malignant melanoma is the skin tumor that causes most deaths in Germany. At an early stage, melanoma is well treatable, so early detection is essential. However, the skin cancer screening program in Germany has been criticized because although melanomas have been diagnosed more frequently since introduction of the program, the mortality from malignant melanoma has not decreased. This indicates that the observed increase in melanoma diagnoses be due to overdiagnosis, i.e. to the detection of lesions that would never have created serious health problems for the patients. One of the reasons is the challenging distinction between some benign and malignant lesions. In addition, there may be lesions that are biologically equivocal, and other lesions that are classified as malignant according to current criteria, but that grow so slowly that they would never have posed a threat to patient's life. So far, these "indolent" melanomas cannot be identified reliably due to a lack of biomarkers. Moreover, the likelihood that an in-situ melanoma will progress to an invasive tumor still cannot be determined with any certainty. When benign lesions are diagnosed as melanoma, the consequences are unnecessary psychological and physical stress for the affected patients and incurred therapy costs. Vice versa, underdiagnoses in the sense of overlooked melanomas can adversely affect patients' prognoses and may necessitate more intense therapies. Novel diagnostic options could reduce the number of over- and underdiagnoses and contribute to more objective diagnoses in borderline cases. One strategy that has yielded promising results in pilot studies is the use of artificial intelligence-based diagnostic tools. However, these applications still await translation into clinical and pathological routine.
Collapse
Affiliation(s)
| | - Tanja B Jutzi
- Digital Biomarkers for Oncology group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Dieter Krahl
- Private Laboratory for Dermatohistopathology, Mönchhofstraße 52, Heidelberg, Germany
| | - Eva I Krieghoff-Henning
- Digital Biomarkers for Oncology group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Markus V Heppt
- Department of Dermatology, University Hospital Erlangen, Erlangen, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Max Schmitt
- Digital Biomarkers for Oncology group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Roman C R Maron
- Digital Biomarkers for Oncology group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Stefan Fröhling
- Digital Biomarkers for Oncology group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christof von Kalle
- Berlin Institute of Health (BIH) and Charité-University Medical Center Berlin, Berlin, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology group (DBO), National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
19
|
Jutzi TB, Krieghoff-Henning EI, Holland-Letz T, Utikal JS, Hauschild A, Schadendorf D, Sondermann W, Fröhling S, Hekler A, Schmitt M, Maron RC, Brinker TJ. Artificial Intelligence in Skin Cancer Diagnostics: The Patients' Perspective. Front Med (Lausanne) 2020; 7:233. [PMID: 32671078 PMCID: PMC7326111 DOI: 10.3389/fmed.2020.00233] [Citation(s) in RCA: 57] [Impact Index Per Article: 14.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2020] [Accepted: 05/05/2020] [Indexed: 12/18/2022] Open
Abstract
Background: Artificial intelligence (AI) has shown promise in numerous experimental studies, particularly in skin cancer diagnostics. Translation of these findings into the clinic is the logical next step. This translation can only be successful if patients' concerns and questions are addressed suitably. We therefore conducted a survey to evaluate the patients' view of artificial intelligence in melanoma diagnostics in Germany, with a particular focus on patients with a history of melanoma. Participants and Methods: A web-based questionnaire was designed using LimeSurvey, sent by e-mail to university hospitals and melanoma support groups and advertised on social media. The anonymous questionnaire evaluated patients' expectations and concerns toward artificial intelligence in general as well as their attitudes toward different application scenarios. Descriptive analysis was performed with expression of categorical variables as percentages and 95% confidence intervals. Statistical tests were performed to investigate associations between sociodemographic data and selected items of the questionnaire. Results: 298 individuals (154 with a melanoma diagnosis, 143 without) responded to the questionnaire. About 94% [95% CI = 0.91–0.97] of respondents supported the use of artificial intelligence in medical approaches. 88% [95% CI = 0.85–0.92] would even make their own health data anonymously available for the further development of AI-based applications in medicine. Only 41% [95% CI = 0.35–0.46] of respondents were amenable to the use of artificial intelligence as stand-alone system, 94% [95% CI = 0.92–0.97] to its use as assistance system for physicians. In sub-group analyses, only minor differences were detectable. Respondents with a previous history of melanoma were more amenable to the use of AI applications for early detection even at home. They would prefer an application scenario where physician and AI classify the lesions independently. With respect to AI-based applications in medicine, patients were concerned about insufficient data protection, impersonality and susceptibility to errors, but expected faster, more precise and unbiased diagnostics, less diagnostic errors and support for physicians. Conclusions: The vast majority of participants exhibited a positive attitude toward the use of artificial intelligence in melanoma diagnostics, especially as an assistance system.
Collapse
Affiliation(s)
- Tanja B Jutzi
- Division of Translational Oncology, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva I Krieghoff-Henning
- Division of Translational Oncology, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Tim Holland-Letz
- Division of Biostatistics, German Cancer Research Center, Heidelberg, Germany
| | - Jochen Sven Utikal
- Department of Dermatology, Heidelberg University, Mannheim, Germany.,Skin Cancer Unit, German Cancer Research Center, Heidelberg, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital Schleswig-Holstein, Kiel, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Wiebke Sondermann
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Stefan Fröhling
- Division of Translational Oncology, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Division of Translational Oncology, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Max Schmitt
- Division of Translational Oncology, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Roman C Maron
- Division of Translational Oncology, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Division of Translational Oncology, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
20
|
Hekler A, Kather JN, Krieghoff-Henning E, Utikal JS, Meier F, Gellrich FF, Upmeier Zu Belzen J, French L, Schlager JG, Ghoreschi K, Wilhelm T, Kutzner H, Berking C, Heppt MV, Haferkamp S, Sondermann W, Schadendorf D, Schilling B, Izar B, Maron R, Schmitt M, Fröhling S, Lipka DB, Brinker TJ. Effects of Label Noise on Deep Learning-Based Skin Cancer Classification. Front Med (Lausanne) 2020; 7:177. [PMID: 32435646 PMCID: PMC7218064 DOI: 10.3389/fmed.2020.00177] [Citation(s) in RCA: 19] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2020] [Accepted: 04/16/2020] [Indexed: 11/19/2022] Open
Abstract
Recent studies have shown that deep learning is capable of classifying dermatoscopic images at least as well as dermatologists. However, many studies in skin cancer classification utilize non-biopsy-verified training images. This imperfect ground truth introduces a systematic error, but the effects on classifier performance are currently unknown. Here, we systematically examine the effects of label noise by training and evaluating convolutional neural networks (CNN) with 804 images of melanoma and nevi labeled either by dermatologists or by biopsy. The CNNs are evaluated on a test set of 384 images by means of 4-fold cross validation comparing the outputs with either the corresponding dermatological or the biopsy-verified diagnosis. With identical ground truths of training and test labels, high accuracies with 75.03% (95% CI: 74.39–75.66%) for dermatological and 73.80% (95% CI: 73.10–74.51%) for biopsy-verified labels can be achieved. However, if the CNN is trained and tested with different ground truths, accuracy drops significantly to 64.53% (95% CI: 63.12–65.94%, p < 0.01) on a non-biopsy-verified and to 64.24% (95% CI: 62.66–65.83%, p < 0.01) on a biopsy-verified test set. In conclusion, deep learning methods for skin cancer classification are highly sensitive to label noise and future work should use biopsy-verified training images to mitigate this problem.
Collapse
Affiliation(s)
- Achim Hekler
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Jakob N Kather
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany.,Department of Medicine III, RWTH University Hospital Aachen, Aachen, Germany
| | - Eva Krieghoff-Henning
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Jochen S Utikal
- Department of Dermatology, Heidelberg University, Mannheim, Germany.,Skin Cancer Unit, German Cancer Research Center, Heidelberg, Germany
| | - Friedegund Meier
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Dresden, Germany.,Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Frank F Gellrich
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Dresden, Germany.,Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | | | - Lars French
- Department of Dermatology and Allergology, Ludwig Maximilian University of Munich, Munich, Germany
| | - Justin G Schlager
- Department of Dermatology and Allergology, Ludwig Maximilian University of Munich, Munich, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Tabea Wilhelm
- Department of Dermatology, Venereology and Allergology, Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Heinz Kutzner
- Dermatopathology Laboratory, Friedrichshafen, Germany
| | - Carola Berking
- Department of Dermatology, University Hospital Erlangen, Erlangen, Germany
| | - Markus V Heppt
- Department of Dermatology, University Hospital Erlangen, Erlangen, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Wiebke Sondermann
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Bastian Schilling
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Benjamin Izar
- Department of Medical Oncology, Dana-Farber Cancer Institute, Boston, MA, United States
| | - Roman Maron
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Max Schmitt
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Stefan Fröhling
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany.,Translational Cancer Epigenomics, Division of Translational Medical Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Daniel B Lipka
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany.,Translational Cancer Epigenomics, Division of Translational Medical Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany.,Faculty of Medicine, Medical Center, Otto-von-Guericke-University, Magdeburg, Germany
| | - Titus J Brinker
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| |
Collapse
|
21
|
Hekler A, Utikal JS, Enk AH, Hauschild A, Weichenthal M, Maron RC, Berking C, Haferkamp S, Klode J, Schadendorf D, Schilling B, Holland-Letz T, Izar B, von Kalle C, Fröhling S, Brinker TJ, Schmitt L, Peitsch WK, Hoffmann F, Becker JC, Drusio C, Jansen P, Klode J, Lodde G, Sammet S, Schadendorf D, Sondermann W, Ugurel S, Zader J, Enk A, Salzmann M, Schäfer S, Schäkel K, Winkler J, Wölbing P, Asper H, Bohne AS, Brown V, Burba B, Deffaa S, Dietrich C, Dietrich M, Drerup KA, Egberts F, Erkens AS, Greven S, Harde V, Jost M, Kaeding M, Kosova K, Lischner S, Maagk M, Messinger AL, Metzner M, Motamedi R, Rosenthal AC, Seidl U, Stemmermann J, Torz K, Velez JG, Haiduk J, Alter M, Bär C, Bergenthal P, Gerlach A, Holtorf C, Karoglan A, Kindermann S, Kraas L, Felcht M, Gaiser MR, Klemke CD, Kurzen H, Leibing T, Müller V, Reinhard RR, Utikal J, Winter F, Berking C, Eicher L, Hartmann D, Heppt M, Kilian K, Krammer S, Lill D, Niesert AC, Oppel E, Sattler E, Senner S, Wallmichrath J, Wolff H, Gesierich A, Giner T, Glutsch V, Kerstan A, Presser D, Schrüfer P, Schummer P, Stolze I, Weber J, Drexler K, Haferkamp S, Mickler M, Stauner CT, Thiem A. Superior skin cancer classification by the combination of human and artificial intelligence. Eur J Cancer 2019; 120:114-121. [DOI: 10.1016/j.ejca.2019.07.019] [Citation(s) in RCA: 80] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2019] [Accepted: 07/18/2019] [Indexed: 10/26/2022]
|
22
|
Hekler A, Utikal JS, Enk AH, Solass W, Schmitt M, Klode J, Schadendorf D, Sondermann W, Franklin C, Bestvater F, Flaig MJ, Krahl D, von Kalle C, Fröhling S, Brinker TJ. Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images. Eur J Cancer 2019; 118:91-96. [DOI: 10.1016/j.ejca.2019.06.012] [Citation(s) in RCA: 74] [Impact Index Per Article: 14.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2019] [Revised: 06/18/2019] [Accepted: 06/21/2019] [Indexed: 12/29/2022]
|
23
|
Brinker TJ, Hekler A, Enk AH, Berking C, Haferkamp S, Hauschild A, Weichenthal M, Klode J, Schadendorf D, Holland-Letz T, von Kalle C, Fröhling S, Schilling B, Utikal JS. Deep neural networks are superior to dermatologists in melanoma image classification. Eur J Cancer 2019; 119:11-17. [DOI: 10.1016/j.ejca.2019.05.023] [Citation(s) in RCA: 79] [Impact Index Per Article: 15.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2019] [Revised: 05/24/2019] [Accepted: 05/28/2019] [Indexed: 10/26/2022]
|
24
|
Maron RC, Weichenthal M, Utikal JS, Hekler A, Berking C, Hauschild A, Enk AH, Haferkamp S, Klode J, Schadendorf D, Jansen P, Holland-Letz T, Schilling B, von Kalle C, Fröhling S, Gaiser MR, Hartmann D, Gesierich A, Kähler KC, Wehkamp U, Karoglan A, Bär C, Brinker TJ, Schmitt L, Peitsch WK, Hoffmann F, Becker JC, Drusio C, Jansen P, Klode J, Lodde G, Sammet S, Schadendorf D, Sondermann W, Ugurel S, Zader J, Enk A, Salzmann M, Schäfer S, Schäkel K, Winkler J, Wölbing P, Asper H, Bohne AS, Brown V, Burba B, Deffaa S, Dietrich C, Dietrich M, Drerup KA, Egberts F, Erkens AS, Greven S, Harde V, Jost M, Kaeding M, Kosova K, Lischner S, Maagk M, Messinger AL, Metzner M, Motamedi R, Rosenthal AC, Seidl U, Stemmermann J, Torz K, Velez JG, Haiduk J, Alter M, Bär C, Bergenthal P, Gerlach A, Holtorf C, Karoglan A, Kindermann S, Kraas L, Felcht M, Gaiser MR, Klemke CD, Kurzen H, Leibing T, Müller V, Reinhard RR, Utikal J, Winter F, Berking C, Eicher L, Hartmann D, Heppt M, Kilian K, Krammer S, Lill D, Niesert AC, Oppel E, Sattler E, Senner S, Wallmichrath J, Wolff H, Giner T, Glutsch V, Kerstan A, Presser D, Schrüfer P, Schummer P, Stolze I, Weber J, Drexler K, Haferkamp S, Mickler M, Stauner CT, Thiem A. Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks. Eur J Cancer 2019; 119:57-65. [DOI: 10.1016/j.ejca.2019.06.013] [Citation(s) in RCA: 92] [Impact Index Per Article: 18.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2019] [Revised: 06/19/2019] [Accepted: 06/21/2019] [Indexed: 01/07/2023]
|
25
|
Brinker TJ, Hekler A, Enk AH, von Kalle C. Enhanced classifier training to improve precision of a convolutional neural network to identify images of skin lesions. PLoS One 2019; 14:e0218713. [PMID: 31233565 PMCID: PMC6590821 DOI: 10.1371/journal.pone.0218713] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2018] [Accepted: 06/09/2019] [Indexed: 01/30/2023] Open
Abstract
BACKGROUND In recent months, multiple publications have demonstrated the use of convolutional neural networks (CNN) to classify images of skin cancer as precisely as dermatologists. However, these CNNs failed to outperform the International Symposium on Biomedical Imaging (ISBI) 2016 challenge which ranked the average precision for classification of dermoscopic melanoma images. Accordingly, the technical progress represented by these studies is limited. In addition, the available reports are impossible to reproduce, due to incomplete descriptions of training procedures and the use of proprietary image databases or non-disclosure of used images. These factors prevent the comparison of various CNN classifiers in equal terms. OBJECTIVE To demonstrate the training of an image-classifier CNN that outperforms the winner of the ISBI 2016 CNNs challenge by using open source images exclusively. METHODS A detailed description of the training procedure is reported while the used images and test sets are disclosed fully, to insure the reproducibility of our work. RESULTS Our CNN classifier outperforms all recent attempts to classify the original ISBI 2016 challenge test data (full set of 379 test images), with an average precision of 0.709 (vs. 0.637 of the ISBI winner) and with an area under the receiver operating curve of 0.85. CONCLUSION This work illustrates the potential for improving skin cancer classification with enhanced training procedures for CNNs, while avoiding the use of costly equipment or proprietary image data.
Collapse
Affiliation(s)
- Titus J. Brinker
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
- Department of Dermatology, University Hospital Heidelberg, Heidelberg, Germany
- * E-mail:
| | - Achim Hekler
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Alexander H. Enk
- Department of Dermatology, University Hospital Heidelberg, Heidelberg, Germany
| | - Christof von Kalle
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
26
|
Hekler A, Utikal JS, Enk AH, Berking C, Klode J, Schadendorf D, Jansen P, Franklin C, Holland-Letz T, Krahl D, von Kalle C, Fröhling S, Brinker TJ. Pathologist-level classification of histopathological melanoma images with deep neural networks. Eur J Cancer 2019; 115:79-83. [PMID: 31129383 DOI: 10.1016/j.ejca.2019.04.021] [Citation(s) in RCA: 102] [Impact Index Per Article: 20.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2019] [Revised: 04/04/2019] [Accepted: 04/10/2019] [Indexed: 01/19/2023]
Abstract
BACKGROUND The diagnosis of most cancers is made by a board-certified pathologist based on a tissue biopsy under the microscope. Recent research reveals a high discordance between individual pathologists. For melanoma, the literature reports 25-26% of discordance for classifying a benign nevus versus malignant melanoma. Deep learning was successfully implemented to enhance the precision of lung and breast cancer diagnoses. The aim of this study is to illustrate the potential of deep learning to assist human assessment for a histopathologic melanoma diagnosis. METHODS Six hundred ninety-five lesions were classified by an expert histopathologist in accordance with current guidelines (350 nevi and 345 melanomas). Only the haematoxylin and eosin stained (H&E) slides of these lesions were digitalised using a slide scanner and then randomly cropped. Five hundred ninety-five of the resulting images were used for the training of a convolutional neural network (CNN). The additional 100 H&E image sections were used to test the results of the CNN in comparison with the original class labels. FINDINGS The total discordance with the histopathologist was 18% for melanoma (95% confidence interval [CI]: 7.4-28.6%), 20% for nevi (95% CI: 8.9-31.1%) and 19% for the full set of images (95% CI: 11.3-26.7%). INTERPRETATION Even in the worst case, the discordance of the CNN was about the same compared with the discordance between human pathologists as reported in the literature. Despite the vastly reduced amount of data, time necessary for diagnosis and cost compared with the pathologist, our CNN archived on-par performance. Conclusively, CNNs indicate to be a valuable tool to assist human melanoma diagnoses.
Collapse
Affiliation(s)
- Achim Hekler
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Jochen Sven Utikal
- Department of Dermatology, Heidelberg University, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center, Heidelberg, Germany
| | - Alexander H Enk
- Department of Dermatology, University Hospital Heidelberg, Heidelberg, Germany
| | - Carola Berking
- Department of Dermatology, University Hospital Munich (LMU), Munich, Germany
| | - Joachim Klode
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Philipp Jansen
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Cindy Franklin
- Department of Dermatology, University Hospital Cologne, Cologne, Germany
| | - Tim Holland-Letz
- Department of Biostatistics, German Cancer Research Center (DKFZ), Heidelberg
| | - Dieter Krahl
- Private Laboratory of Dermatohistopathology, Mönchhofstraße 52, 69120 Heidelberg
| | - Christof von Kalle
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Stefan Fröhling
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Titus Josef Brinker
- National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany; Department of Dermatology, University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
27
|
Brinker TJ, Hekler A, Enk AH, Klode J, Hauschild A, Berking C, Schilling B, Haferkamp S, Schadendorf D, Holland-Letz T, Utikal JS, von Kalle C. Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. Eur J Cancer 2019; 113:47-54. [PMID: 30981091 DOI: 10.1016/j.ejca.2019.04.001] [Citation(s) in RCA: 176] [Impact Index Per Article: 35.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2019] [Revised: 03/19/2019] [Accepted: 04/02/2019] [Indexed: 10/27/2022]
Abstract
BACKGROUND Recent studies have successfully demonstrated the use of deep-learning algorithms for dermatologist-level classification of suspicious lesions by the use of excessive proprietary image databases and limited numbers of dermatologists. For the first time, the performance of a deep-learning algorithm trained by open-source images exclusively is compared to a large number of dermatologists covering all levels within the clinical hierarchy. METHODS We used methods from enhanced deep learning to train a convolutional neural network (CNN) with 12,378 open-source dermoscopic images. We used 100 images to compare the performance of the CNN to that of the 157 dermatologists from 12 university hospitals in Germany. Outperformance of dermatologists by the deep neural network was measured in terms of sensitivity, specificity and receiver operating characteristics. FINDINGS The mean sensitivity and specificity achieved by the dermatologists with dermoscopic images was 74.1% (range 40.0%-100%) and 60% (range 21.3%-91.3%), respectively. At a mean sensitivity of 74.1%, the CNN exhibited a mean specificity of 86.5% (range 70.8%-91.3%). At a mean specificity of 60%, a mean sensitivity of 87.5% (range 80%-95%) was achieved by our algorithm. Among the dermatologists, the chief physicians showed the highest mean specificity of 69.2% at a mean sensitivity of 73.3%. With the same high specificity of 69.2%, the CNN had a mean sensitivity of 84.5%. INTERPRETATION A CNN trained by open-source images exclusively outperformed 136 of the 157 dermatologists and all the different levels of experience (from junior to chief physicians) in terms of average specificity and sensitivity.
Collapse
Affiliation(s)
- Titus J Brinker
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 460, 69120 Heidelberg, Germany; Department of Dermatology, University Hospital Heidelberg, Heidelberg, Germany.
| | - Achim Hekler
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 460, 69120 Heidelberg, Germany
| | - Alexander H Enk
- Department of Dermatology, University Hospital Heidelberg, Heidelberg, Germany
| | - Joachim Klode
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital Kiel, Kiel, Germany
| | - Carola Berking
- Department of Dermatology, University Hospital Munich (LMU), Munich, Germany
| | - Bastian Schilling
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Tim Holland-Letz
- Department of Biostatistics, German Cancer Research Center, Heidelberg, Germany
| | - Jochen S Utikal
- Department of Dermatology, Heidelberg University, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christof von Kalle
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 460, 69120 Heidelberg, Germany
| | | |
Collapse
|
28
|
Brinker TJ, Hekler A, von Kalle C, Schadendorf D, Esser S, Berking C, Zacher MT, Sondermann W, Grabe N, Steeb T, Utikal JS, French LE, Enk AH. Teledermatology: Comparison of Store-and-Forward Versus Live Interactive Video Conferencing. J Med Internet Res 2018; 20:e11871. [PMID: 30355564 PMCID: PMC6231765 DOI: 10.2196/11871] [Citation(s) in RCA: 24] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2018] [Revised: 08/30/2018] [Accepted: 08/30/2018] [Indexed: 01/30/2023] Open
Abstract
A decreasing number of dermatologists and an increasing number of patients in Western countries have led to a relative lack of clinicians providing expert dermatologic care. This, in turn, has prolonged wait times for patients to be examined, putting them at risk. Store-and-forward teledermatology improves patient access to dermatologists through asynchronous consultations, reducing wait times to obtain a consultation. However, live video conferencing as a synchronous service is also frequently used by practitioners because it allows immediate interaction between patient and physician. This raises the question of which of the two approaches is superior in terms of quality of care and convenience. There are pros and cons for each in terms of technical requirements and features. This viewpoint compares the two techniques based on a literature review and a clinical perspective to help dermatologists assess the value of teledermatology and determine which techniques would be valuable in their practice.
Collapse
Affiliation(s)
- Titus Josef Brinker
- Department of Translational Oncology, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany.,Department of Dermatology, University Hospital Heidelberg, University of Heidelberg, Heidelberg, Germany.,German Cancer Consortium, University of Heidelberg, Heidelberg, Germany
| | - Achim Hekler
- Department of Translational Oncology, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Christof von Kalle
- Department of Translational Oncology, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, University of Duisburg-Essen, Essen, Germany
| | - Stefan Esser
- Department of Dermatology, University Hospital Essen, University of Duisburg-Essen, Essen, Germany.,University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Carola Berking
- Department of Dermatology and Allergology, University Medical Center Munich, Ludwig Maximilian University of Munich, Munich, Germany
| | - Martina T Zacher
- Department of Dermatology and Allergology, University Medical Center Munich, Ludwig Maximilian University of Munich, Munich, Germany
| | - Wiebke Sondermann
- Department of Dermatology, University Hospital Essen, University of Duisburg-Essen, Essen, Germany
| | - Niels Grabe
- BioQuant, Hamamatsu Tissue Imaging and Analysis Center (TIGA), University of Heidelberg, Heidelberg, Germany
| | - Theresa Steeb
- Department of Dermatology and Allergology, University Medical Center Munich, Ludwig Maximilian University of Munich, Munich, Germany
| | - Jochen Sven Utikal
- German Cancer Research Center, Skin Cancer Unit, University of Heidelberg, Mannheim, Germany
| | - Lars E French
- Department of Dermatology, University Hospital of Zurich, University of Zurich, Zurich, Switzerland
| | - Alexander H Enk
- Department of Dermatology, University Hospital Heidelberg, University of Heidelberg, Heidelberg, Germany
| |
Collapse
|
29
|
Brinker TJ, Hekler A, Utikal JS, Grabe N, Schadendorf D, Klode J, Berking C, Steeb T, Enk AH, von Kalle C. Skin Cancer Classification Using Convolutional Neural Networks: Systematic Review. J Med Internet Res 2018; 20:e11936. [PMID: 30333097 PMCID: PMC6231861 DOI: 10.2196/11936] [Citation(s) in RCA: 119] [Impact Index Per Article: 19.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2018] [Revised: 09/05/2018] [Accepted: 09/08/2018] [Indexed: 11/24/2022] Open
Abstract
Background State-of-the-art classifiers based on convolutional neural networks (CNNs) were shown to classify images of skin cancer on par with dermatologists and could enable lifesaving and fast diagnoses, even outside the hospital via installation of apps on mobile devices. To our knowledge, at present there is no review of the current work in this research area. Objective This study presents the first systematic review of the state-of-the-art research on classifying skin lesions with CNNs. We limit our review to skin lesion classifiers. In particular, methods that apply a CNN only for segmentation or for the classification of dermoscopic patterns are not considered here. Furthermore, this study discusses why the comparability of the presented procedures is very difficult and which challenges must be addressed in the future. Methods We searched the Google Scholar, PubMed, Medline, ScienceDirect, and Web of Science databases for systematic reviews and original research articles published in English. Only papers that reported sufficient scientific proceedings are included in this review. Results We found 13 papers that classified skin lesions using CNNs. In principle, classification methods can be differentiated according to three principles. Approaches that use a CNN already trained by means of another large dataset and then optimize its parameters to the classification of skin lesions are the most common ones used and they display the best performance with the currently available limited datasets. Conclusions CNNs display a high performance as state-of-the-art skin lesion classifiers. Unfortunately, it is difficult to compare different classification methods because some approaches use nonpublic datasets for training and/or testing, thereby making reproducibility difficult. Future publications should use publicly available benchmarks and fully disclose methods used for training to allow comparability.
Collapse
Affiliation(s)
- Titus Josef Brinker
- National Center for Tumor Diseases, Department of Translational Oncology, German Cancer Research Center, Heidelberg, Germany.,Department of Dermatology, University Hospital Heidelberg, University of Heidelberg, Heidelberg, Germany
| | - Achim Hekler
- National Center for Tumor Diseases, Department of Translational Oncology, German Cancer Research Center, Heidelberg, Germany
| | - Jochen Sven Utikal
- Skin Cancer Unit, German Cancer Research Center, Heidelberg, Germany.,Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karl University of Heidelberg, Heidelberg, Germany
| | - Niels Grabe
- Bioquant, Hamamatsu Tissue Imaging and Analysis Center, University of Heidelberg, Heidelberg, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital of Essen, University of Duisburg-Essen, Essen, Germany
| | - Joachim Klode
- Department of Dermatology, University Hospital of Essen, University of Duisburg-Essen, Essen, Germany
| | - Carola Berking
- Department of Dermatology, University Hospital Munich, Ludwig Maximilian University of Munich, Munich, Germany
| | - Theresa Steeb
- Department of Dermatology, University Hospital Munich, Ludwig Maximilian University of Munich, Munich, Germany
| | - Alexander H Enk
- Department of Dermatology, University Hospital Heidelberg, University of Heidelberg, Heidelberg, Germany
| | - Christof von Kalle
- National Center for Tumor Diseases, Department of Translational Oncology, German Cancer Research Center, Heidelberg, Germany
| |
Collapse
|