1
|
Chanda T, Hauser K, Hobelsberger S, Bucher TC, Garcia CN, Wies C, Kittler H, Tschandl P, Navarrete-Dechent C, Podlipnik S, Chousakos E, Crnaric I, Majstorovic J, Alhajwan L, Foreman T, Peternel S, Sarap S, Özdemir İ, Barnhill RL, Llamas-Velasco M, Poch G, Korsing S, Sondermann W, Gellrich FF, Heppt MV, Erdmann M, Haferkamp S, Drexler K, Goebeler M, Schilling B, Utikal JS, Ghoreschi K, Fröhling S, Krieghoff-Henning E, Brinker TJ. Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanoma. Nat Commun 2024; 15:524. [PMID: 38225244 PMCID: PMC10789736 DOI: 10.1038/s41467-023-43095-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2023] [Accepted: 10/31/2023] [Indexed: 01/17/2024] Open
Abstract
Artificial intelligence (AI) systems have been shown to help dermatologists diagnose melanoma more accurately, however they lack transparency, hindering user acceptance. Explainable AI (XAI) methods can help to increase transparency, yet often lack precise, domain-specific explanations. Moreover, the impact of XAI methods on dermatologists' decisions has not yet been evaluated. Building upon previous research, we introduce an XAI system that provides precise and domain-specific explanations alongside its differential diagnoses of melanomas and nevi. Through a three-phase study, we assess its impact on dermatologists' diagnostic accuracy, diagnostic confidence, and trust in the XAI-support. Our results show strong alignment between XAI and dermatologist explanations. We also show that dermatologists' confidence in their diagnoses, and their trust in the support system significantly increase with XAI compared to conventional AI. This study highlights dermatologists' willingness to adopt such XAI systems, promoting future use in the clinic.
Collapse
Affiliation(s)
- Tirtha Chanda
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Katja Hauser
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sarah Hobelsberger
- Department of Dermatology, University Hospital, Technical University Dresden, Dresden, Germany
| | - Tabea-Clara Bucher
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Carina Nogueira Garcia
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christoph Wies
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Medical Faculty of University Heidelberg, Heidelberg, Germany
| | - Harald Kittler
- Department of Dermatology, Medical University of Vienna, Vienna, Austria
| | - Philipp Tschandl
- Department of Dermatology, Medical University of Vienna, Vienna, Austria
| | - Cristian Navarrete-Dechent
- Department of Dermatology, Escuela de Medicina, Pontificia Universidad Católica de Chile, Santiago, Chile
| | - Sebastian Podlipnik
- Dermatology Department, Hospital Clínic of Barcelona, University of Barcelona, IDIBAPS, Barcelona, Spain
| | - Emmanouil Chousakos
- 1st Department of Pathology, Medical School, National & Kapodistrian University of Athens, Athens, Greece
| | - Iva Crnaric
- Department of Dermatovenereology, Sestre milosrdnice University Hospital Center, Zagreb, Croatia
| | | | - Linda Alhajwan
- Department of Dermatology, Dubai London Clinic, Dubai, United Arab Emirates
| | | | - Sandra Peternel
- Department of Dermatovenereology, Clinical Hospital Center Rijeka, Faculty of Medicine, University of Rijeka, Rijeka, Croatia
| | | | - İrem Özdemir
- Department of Dermatology, Faculty of Medicine, Gazi University, Ankara, Turkey
| | - Raymond L Barnhill
- Department of Translational Research, Institut Curie, Unit of Formation and Research of Medicine University of Paris, Paris, France
| | | | - Gabriela Poch
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Department of Dermatology, Venereology and Allergology, Berlin, Germany
| | - Sören Korsing
- Department of Dermatology, University Hospital Essen, University Duisburg-Essen, Essen, Germany
| | - Wiebke Sondermann
- Department of Dermatology, Uniklinikum Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | | | - Markus V Heppt
- Department of Dermatology, Uniklinikum Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Michael Erdmann
- Department of Dermatology, Uniklinikum Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Konstantin Drexler
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Matthias Goebeler
- Department of Dermatology, Venereology and Allergology, University Hospital Würzburg, Würzburg, Germany
| | - Bastian Schilling
- Department of Dermatology, Venereology and Allergology, University Hospital Würzburg, Würzburg, Germany
| | - Jochen S Utikal
- Department of Dermatology, Venereology and Allergology, University Medical Center Mannheim, Ruprecht-Karl University of Heidelberg, Mannheim, Germany
| | - Kamran Ghoreschi
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Department of Dermatology, Venereology and Allergology, Berlin, Germany
| | - Stefan Fröhling
- Division of Translational Medical Oncology, National Center for Tumor Diseases (NCT) Heidelberg and German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
2
|
Gassner M, Barranco Garcia J, Tanadini-Lang S, Bertoldo F, Fröhlich F, Guckenberger M, Haueis S, Pelzer C, Reyes M, Schmithausen P, Simic D, Staeger R, Verardi F, Andratschke N, Adelmann A, Braun RP. Saliency-Enhanced Content-Based Image Retrieval for Diagnosis Support in Dermatology Consultation: Reader Study. JMIR DERMATOLOGY 2023; 6:e42129. [PMID: 37616039 PMCID: PMC10485719 DOI: 10.2196/42129] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Revised: 04/07/2023] [Accepted: 06/16/2023] [Indexed: 08/25/2023] Open
Abstract
BACKGROUND Previous research studies have demonstrated that medical content image retrieval can play an important role by assisting dermatologists in skin lesion diagnosis. However, current state-of-the-art approaches have not been adopted in routine consultation, partly due to the lack of interpretability limiting trust by clinical users. OBJECTIVE This study developed a new image retrieval architecture for polarized or dermoscopic imaging guided by interpretable saliency maps. This approach provides better feature extraction, leading to better quantitative retrieval performance as well as providing interpretability for an eventual real-world implementation. METHODS Content-based image retrieval (CBIR) algorithms rely on the comparison of image features embedded by convolutional neural network (CNN) against a labeled data set. Saliency maps are computer vision-interpretable methods that highlight the most relevant regions for the prediction made by a neural network. By introducing a fine-tuning stage that includes saliency maps to guide feature extraction, the accuracy of image retrieval is optimized. We refer to this approach as saliency-enhanced CBIR (SE-CBIR). A reader study was designed at the University Hospital Zurich Dermatology Clinic to evaluate SE-CBIR's retrieval accuracy as well as the impact of the participant's confidence on the diagnosis. RESULTS SE-CBIR improved the retrieval accuracy by 7% (77% vs 84%) when doing single-lesion retrieval against traditional CBIR. The reader study showed an overall increase in classification accuracy of 22% (62% vs 84%) when the participant is provided with SE-CBIR retrieved images. In addition, the overall confidence in the lesion's diagnosis increased by 24%. Finally, the use of SE-CBIR as a support tool helped the participants reduce the number of nonmelanoma lesions previously diagnosed as melanoma (overdiagnosis) by 53%. CONCLUSIONS SE-CBIR presents better retrieval accuracy compared to traditional CBIR CNN-based approaches. Furthermore, we have shown how these support tools can help dermatologists and residents improve diagnosis accuracy and confidence. Additionally, by introducing interpretable methods, we should expect increased acceptance and use of these tools in routine consultation.
Collapse
Affiliation(s)
- Mathias Gassner
- Department of Radio Oncology, University Hospital Zurich, Zurich, Switzerland
- Physics Department, Swiss Federal Institute of Technology Zurich, Zurich, Switzerland
| | - Javier Barranco Garcia
- Department of Radio Oncology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Stephanie Tanadini-Lang
- Department of Radio Oncology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Fabio Bertoldo
- Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
| | - Fabienne Fröhlich
- Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
| | - Matthias Guckenberger
- Department of Radio Oncology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Silvia Haueis
- Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
| | - Christin Pelzer
- Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
| | - Mauricio Reyes
- ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
- Department of Radiation Oncology, Inselspital, Bern University Hospital, Bern, Switzerland
| | | | - Dario Simic
- Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
| | - Ramon Staeger
- Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
| | - Fabio Verardi
- Department of Dermatology, University Hospital Zurich, Zurich, Switzerland
| | - Nicolaus Andratschke
- Department of Radio Oncology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Andreas Adelmann
- Laboratory for Scientific Computing and Modelling, Paul Scherrer Institut, Villigen, Switzerland
| | - Ralph P Braun
- Department of Dermatology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| |
Collapse
|
3
|
Rashad M, Afifi I, Abdelfatah M. RbQE: An Efficient Method for Content-Based Medical Image Retrieval Based on Query Expansion. J Digit Imaging 2023; 36:1248-1261. [PMID: 36702987 PMCID: PMC10287886 DOI: 10.1007/s10278-022-00769-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2022] [Revised: 12/18/2022] [Accepted: 12/19/2022] [Indexed: 01/27/2023] Open
Abstract
Systems for retrieving and managing content-based medical images are becoming more important, especially as medical imaging technology advances and the medical image database grows. In addition, these systems can also use medical images to better grasp and gain a deeper understanding of the causes and treatments of different diseases, not just for diagnostic purposes. For achieving all these purposes, there is a critical need for an efficient and accurate content-based medical image retrieval (CBMIR) method. This paper proposes an efficient method (RbQE) for the retrieval of computed tomography (CT) and magnetic resonance (MR) images. RbQE is based on expanding the features of querying and exploiting the pre-trained learning models AlexNet and VGG-19 to extract compact, deep, and high-level features from medical images. There are two searching procedures in RbQE: a rapid search and a final search. In the rapid search, the original query is expanded by retrieving the top-ranked images from each class and is used to reformulate the query by calculating the mean values for deep features of the top-ranked images, resulting in a new query for each class. In the final search, the new query that is most similar to the original query will be used for retrieval from the database. The performance of the proposed method has been compared to state-of-the-art methods on four publicly available standard databases, namely, TCIA-CT, EXACT09-CT, NEMA-CT, and OASIS-MRI. Experimental results show that the proposed method exceeds the compared methods by 0.84%, 4.86%, 1.24%, and 14.34% in average retrieval precision (ARP) for the TCIA-CT, EXACT09-CT, NEMA-CT, and OASIS-MRI databases, respectively.
Collapse
Affiliation(s)
- Metwally Rashad
- Department of Computer Science, Faculty of Computers & Artificial Intelligence, Benha University, Benha, Egypt
- Faculty of Artificial Intelligence, Delta University for Science and Technology, Gamasa, Egypt
| | - Ibrahem Afifi
- Department of Information System, Faculty of Computers & Artificial Intelligence, Benha University, Benha, Egypt
| | - Mohammed Abdelfatah
- Department of Information System, Faculty of Computers & Artificial Intelligence, Benha University, Benha, Egypt
| |
Collapse
|
4
|
Ahmedt-Aristizabal D, Nguyen C, Tychsen-Smith L, Stacey A, Li S, Pathikulangara J, Petersson L, Wang D. Monitoring of Pigmented Skin Lesions Using 3D Whole Body Imaging. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 232:107451. [PMID: 36893580 DOI: 10.1016/j.cmpb.2023.107451] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/17/2022] [Revised: 02/23/2023] [Accepted: 02/26/2023] [Indexed: 06/18/2023]
Abstract
BACKGROUND AND OBJECTIVES Advanced artificial intelligence and machine learning have great potential to redefine how skin lesions are detected, mapped, tracked and documented. Here, we propose a 3D whole-body imaging system known as 3DSkin-mapper to enable automated detection, evaluation and mapping of skin lesions. METHODS A modular camera rig arranged in a cylindrical configuration was designed to automatically capture images of the entire skin surface of a subject synchronously from multiple angles. Based on the images, we developed algorithms for 3D model reconstruction, data processing and skin lesion detection and tracking based on deep convolutional neural networks. We also introduced a customised, user-friendly, and adaptable interface that enables individuals to interactively visualise, manipulate, and annotate the images. The interface includes built-in features such as mapping 2D skin lesions onto the corresponding 3D model. RESULTS The proposed system is developed for skin lesion screening, the focus of this paper is to introduce the system instead of clinical study. Using synthetic and real images we demonstrate the effectiveness of the proposed system by providing multiple views of a target skin lesion, enabling further 3D geometry analysis and longitudinal tracking. Skin lesions are identified as outliers which deserve more attention from a skin cancer physician. Our detector leverages expert annotated labels to learn representations of skin lesions, while capturing the effects of anatomical variability. It takes only a few seconds to capture the entire skin surface, and about half an hour to process and analyse the images. CONCLUSIONS Our experiments show that the proposed system allows fast and easy whole body 3D imaging. It can be used by dermatological clinics to conduct skin screening, detect and track skin lesions over time, identify suspicious lesions, and document pigmented lesions. The system can potentially save clinicians time and effort significantly. The 3D imaging and analysis has the potential to change the paradigm of whole body photography with many applications in skin diseases, including inflammatory and pigmentary disorders. With reduced time requirements for recording and documenting high-quality skin information, doctors could spend more time providing better-quality treatment based on more detailed and accurate information.
Collapse
Affiliation(s)
| | - Chuong Nguyen
- Imaging and Computer Vision group, CSIRO Data61, Australia.
| | | | | | - Shenghong Li
- Imaging and Computer Vision group, CSIRO Data61, Australia.
| | | | - Lars Petersson
- Imaging and Computer Vision group, CSIRO Data61, Australia.
| | - Dadong Wang
- Imaging and Computer Vision group, CSIRO Data61, Australia.
| |
Collapse
|
5
|
Hasan MK, Ahamad MA, Yap CH, Yang G. A survey, review, and future trends of skin lesion segmentation and classification. Comput Biol Med 2023; 155:106624. [PMID: 36774890 DOI: 10.1016/j.compbiomed.2023.106624] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2022] [Revised: 01/04/2023] [Accepted: 01/28/2023] [Indexed: 02/03/2023]
Abstract
The Computer-aided Diagnosis or Detection (CAD) approach for skin lesion analysis is an emerging field of research that has the potential to alleviate the burden and cost of skin cancer screening. Researchers have recently indicated increasing interest in developing such CAD systems, with the intention of providing a user-friendly tool to dermatologists to reduce the challenges encountered or associated with manual inspection. This article aims to provide a comprehensive literature survey and review of a total of 594 publications (356 for skin lesion segmentation and 238 for skin lesion classification) published between 2011 and 2022. These articles are analyzed and summarized in a number of different ways to contribute vital information regarding the methods for the development of CAD systems. These ways include: relevant and essential definitions and theories, input data (dataset utilization, preprocessing, augmentations, and fixing imbalance problems), method configuration (techniques, architectures, module frameworks, and losses), training tactics (hyperparameter settings), and evaluation criteria. We intend to investigate a variety of performance-enhancing approaches, including ensemble and post-processing. We also discuss these dimensions to reveal their current trends based on utilization frequencies. In addition, we highlight the primary difficulties associated with evaluating skin lesion segmentation and classification systems using minimal datasets, as well as the potential solutions to these difficulties. Findings, recommendations, and trends are disclosed to inform future research on developing an automated and robust CAD system for skin lesion analysis.
Collapse
Affiliation(s)
- Md Kamrul Hasan
- Department of Bioengineering, Imperial College London, UK; Department of Electrical and Electronic Engineering (EEE), Khulna University of Engineering & Technology (KUET), Khulna 9203, Bangladesh.
| | - Md Asif Ahamad
- Department of Electrical and Electronic Engineering (EEE), Khulna University of Engineering & Technology (KUET), Khulna 9203, Bangladesh.
| | - Choon Hwai Yap
- Department of Bioengineering, Imperial College London, UK.
| | - Guang Yang
- National Heart and Lung Institute, Imperial College London, UK; Cardiovascular Research Centre, Royal Brompton Hospital, UK.
| |
Collapse
|
6
|
Birkner M, Schalk J, von den Driesch P, Schultz ES. Computer-Assisted Differential Diagnosis of Pyoderma Gangrenosum and Venous Ulcers with Deep Neural Networks. J Clin Med 2022; 11:jcm11237103. [PMID: 36498674 PMCID: PMC9740900 DOI: 10.3390/jcm11237103] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Revised: 11/21/2022] [Accepted: 11/24/2022] [Indexed: 12/03/2022] Open
Abstract
(1) Background: Pyoderma gangrenosum (PG) is often situated on the lower legs, and the differentiation from conventional leg ulcers (LU) is a challenging task due to the lack of clear clinical diagnostic criteria. Because of the different therapy concepts, misdiagnosis or delayed diagnosis bears a great risk for patients. (2) Objective: to develop a deep convolutional neural network (CNN) capable of analysing wound photographs to facilitate the PG diagnosis for health professionals. (3) Methods: A CNN was trained with 422 expert-selected pictures of PG and LU. In a man vs. machine contest, 33 pictures of PG and 36 pictures of LU were presented for diagnosis to 18 dermatologists at two maximum care hospitals and to the CNN. The results were statistically evaluated in terms of sensitivity, specificity and accuracy for the CNN and for dermatologists with different experience levels. (4) Results: The CNN achieved a sensitivity of 97% (95% confidence interval (CI) 84.2−99.9%) and outperformed dermatologists, with a sensitivity of 72.7% (CI 54.4−86.7%) significantly (p < 0.03). However, dermatologists achieved a slightly higher specificity (88.9% vs. 83.3%). (5) Conclusions: For the first time, a deep neural network was demonstrated to be capable of diagnosing PG, solely on the basis of photographs, and with a greater sensitivity compared to that of dermatologists.
Collapse
Affiliation(s)
- Mattias Birkner
- Institute of Medical Physics, Paracelsus Medical University Nuremberg, City Hospital of Nuremberg, 90419 Nürnberg, Germany
- Correspondence:
| | - Julia Schalk
- Department of Dermatology, Paracelsus Medical University Nuremberg, City Hospital of Nuremberg, 90419 Nürnberg, Germany
| | - Peter von den Driesch
- Department of Dermatology, Klinikum Stuttgart, Bad Cannstatt, 70174 Stuttgart, Germany
| | - Erwin S. Schultz
- Department of Dermatology, Paracelsus Medical University Nuremberg, City Hospital of Nuremberg, 90419 Nürnberg, Germany
| |
Collapse
|
7
|
Foahom Gouabou AC, Collenne J, Monnier J, Iguernaissi R, Damoiseaux JL, Moudafi A, Merad D. Computer Aided Diagnosis of Melanoma Using Deep Neural Networks and Game Theory: Application on Dermoscopic Images of Skin Lesions. Int J Mol Sci 2022; 23:ijms232213838. [PMID: 36430315 PMCID: PMC9696950 DOI: 10.3390/ijms232213838] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2022] [Revised: 10/31/2022] [Accepted: 11/07/2022] [Indexed: 11/12/2022] Open
Abstract
Early detection of melanoma remains a daily challenge due to the increasing number of cases and the lack of dermatologists. Thus, AI-assisted diagnosis is considered as a possible solution for this issue. Despite the great advances brought by deep learning and especially convolutional neural networks (CNNs), computer-aided diagnosis (CAD) systems are still not used in clinical practice. This may be explained by the dermatologist's fear of being misled by a false negative and the assimilation of CNNs to a "black box", making their decision process difficult to understand by a non-expert. Decision theory, especially game theory, is a potential solution as it focuses on identifying the best decision option that maximizes the decision-maker's expected utility. This study presents a new framework for automated melanoma diagnosis. Pursuing the goal of improving the performance of existing systems, our approach also attempts to bring more transparency in the decision process. The proposed framework includes a multi-class CNN and six binary CNNs assimilated to players. The players' strategies is to first cluster the pigmented lesions (melanoma, nevus, and benign keratosis), using the introduced method of evaluating the confidence of the predictions, into confidence level (confident, medium, uncertain). Then, a subset of players has the strategy to refine the diagnosis for difficult lesions with medium and uncertain prediction. We used EfficientNetB5 as the backbone of our networks and evaluated our approach on the public ISIC dataset consisting of 8917 lesions: melanoma (1113), nevi (6705) and benign keratosis (1099). The proposed framework achieved an area under the receiver operating curve (AUROC) of 0.93 for melanoma, 0.96 for nevus and 0.97 for benign keratosis. Furthermore, our approach outperformed existing methods in this task, improving the balanced accuracy (BACC) of the best compared method from 77% to 86%. These results suggest that our framework provides an effective and explainable decision-making strategy. This approach could help dermatologists in their clinical practice for patients with atypical and difficult-to-diagnose pigmented lesions. We also believe that our system could serve as a didactic tool for less experienced dermatologists.
Collapse
Affiliation(s)
| | - Jules Collenne
- LIS, CNRS, Aix Marseille University, 13288 Marseille, France
| | - Jilliana Monnier
- LIS, CNRS, Aix Marseille University, 13288 Marseille, France
- Research Cancer Centre of Marseille, Inserm, CNRS, Aix-Marseille University, 13273 Marseille, France
- Dermatology and Skin Cancer Department, La Timone Hospital, AP-HM, Aix-Marseille University, 13385 Marseille, France
| | | | | | | | - Djamal Merad
- LIS, CNRS, Aix Marseille University, 13288 Marseille, France
- Correspondence: (A.C.F.G.); (D.M.)
| |
Collapse
|
8
|
Wang Y, Fariah Haq N, Cai J, Kalia S, Lui H, Jane Wang Z, Lee TK. Multi-channel content based image retrieval method for skin diseases using similarity network fusion and deep community analysis. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103893] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
9
|
Garcia JB, Tanadini-Lang S, Andratschke N, Gassner M, Braun R. Suspicious Skin Lesion Detection in Wide-Field Body Images using Deep Learning Outlier Detection. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:2928-2932. [PMID: 36085609 DOI: 10.1109/embc48229.2022.9871655] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
During consultation dermatologists have to address hundreds of lesions in a limited amount of time. They will not only evaluate the single lesion of interest but more importantly the context of it. Visually comparing the similarity of the majority of lesions within the same patient provides a strong indication for lesions with significantly differing aspects. Deep learning algorithms are capable to identify such outliers, i.e. images that differ considerably from the expected appearance on a larger cohort, and highlight the main differences in those cases. In the present study we evaluate the use of autoencoders as unsupervised tools to detect suspicious skin lesions based on evaluation of real world data acquired during consultation at the USZ Dermatology Clinic. Clinical Relevance- Deep learning algorithms are showing many promising results in dermatology lesion classification. However the context of the lesion is normally not considered in the analysis which prevents these tools to transition into routine practice. An outlier detector based on real world data would allow a dermatologist or general practitioner to detect the suspicious lesions for further examination. The algorithm would additionally provide useful insights by highlighting the feature differences between the original outlier (malignant lesion) and the lesion reconstructed by the autoencoder.
Collapse
|
10
|
Eapen BR, Kaliyadan F, Ashique KT. DICODerma: A Practical Approach for Metadata Management of Images in Dermatology. J Digit Imaging 2022; 35:1231-1237. [PMID: 35488074 PMCID: PMC9054111 DOI: 10.1007/s10278-022-00636-5] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2021] [Revised: 03/07/2022] [Accepted: 04/10/2022] [Indexed: 11/29/2022] Open
Abstract
Clinical images are vital for diagnosing and monitoring skin diseases, and their importance has increased with the growing popularity of machine learning. Lack of standards has stifled innovation in dermatological imaging, unlike other image-intensive specialties such as radiology. We investigate the meta-requirements for utilizing the popular DICOM standard for metadata management of images in dermatology. We propose practical design solutions and provide open-source tools to integrate dermatologists’ workflow with enterprise imaging systems. Using the tool, dermatologists can tag, search, organize and convert clinical images to the DICOM format. We believe that our less disruptive approach will improve the adoption of standards in the specialty.
Collapse
Affiliation(s)
- Bell Raj Eapen
- McMaster University, 1280 Main Street West, Hamilton, Ontario, L8S 4L8, Canada.
| | - Feroze Kaliyadan
- Department of Dermatology, Sree Narayana Institute of Medical Sciences, Kunnukara, Kerala, India
| | | |
Collapse
|
11
|
Hauser K, Kurz A, Haggenmüller S, Maron RC, von Kalle C, Utikal JS, Meier F, Hobelsberger S, Gellrich FF, Sergon M, Hauschild A, French LE, Heinzerling L, Schlager JG, Ghoreschi K, Schlaak M, Hilke FJ, Poch G, Kutzner H, Berking C, Heppt MV, Erdmann M, Haferkamp S, Schadendorf D, Sondermann W, Goebeler M, Schilling B, Kather JN, Fröhling S, Lipka DB, Hekler A, Krieghoff-Henning E, Brinker TJ. Explainable artificial intelligence in skin cancer recognition: A systematic review. Eur J Cancer 2022; 167:54-69. [PMID: 35390650 DOI: 10.1016/j.ejca.2022.02.025] [Citation(s) in RCA: 29] [Impact Index Per Article: 14.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2022] [Revised: 02/22/2022] [Accepted: 02/24/2022] [Indexed: 01/18/2023]
Abstract
BACKGROUND Due to their ability to solve complex problems, deep neural networks (DNNs) are becoming increasingly popular in medical applications. However, decision-making by such algorithms is essentially a black-box process that renders it difficult for physicians to judge whether the decisions are reliable. The use of explainable artificial intelligence (XAI) is often suggested as a solution to this problem. We investigate how XAI is used for skin cancer detection: how is it used during the development of new DNNs? What kinds of visualisations are commonly used? Are there systematic evaluations of XAI with dermatologists or dermatopathologists? METHODS Google Scholar, PubMed, IEEE Explore, Science Direct and Scopus were searched for peer-reviewed studies published between January 2017 and October 2021 applying XAI to dermatological images: the search terms histopathological image, whole-slide image, clinical image, dermoscopic image, skin, dermatology, explainable, interpretable and XAI were used in various combinations. Only studies concerned with skin cancer were included. RESULTS 37 publications fulfilled our inclusion criteria. Most studies (19/37) simply applied existing XAI methods to their classifier to interpret its decision-making. Some studies (4/37) proposed new XAI methods or improved upon existing techniques. 14/37 studies addressed specific questions such as bias detection and impact of XAI on man-machine-interactions. However, only three of them evaluated the performance and confidence of humans using CAD systems with XAI. CONCLUSION XAI is commonly applied during the development of DNNs for skin cancer detection. However, a systematic and rigorous evaluation of its usefulness in this scenario is lacking.
Collapse
Affiliation(s)
- Katja Hauser
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Alexander Kurz
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sarah Haggenmüller
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Roman C Maron
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Charité University Medicine and Berlin Institute of Health (BIH), Berlin, Germany
| | - Jochen S Utikal
- Department of Dermatology, Heidelberg University, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Friedegund Meier
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Sarah Hobelsberger
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Frank F Gellrich
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Mildred Sergon
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital (UKSH), Kiel, Germany
| | - Lars E French
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany; Dr. Phillip Frost Department of Dermatology and Cutaneous Surgery, University of Miami, Miller School of Medicine, Miami, FL, USA
| | - Lucie Heinzerling
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
| | - Justin G Schlager
- Department of Dermatology and Allergy, University Hospital, LMU Munich, Munich, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Max Schlaak
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Franz J Hilke
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Gabriela Poch
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Heinz Kutzner
- Dermatopathology Laboratory, Friedrichshafen, Germany
| | - Carola Berking
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - EMN, Friedrich-Alexander University Erlangen, Nuremberg, Germany
| | - Markus V Heppt
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - EMN, Friedrich-Alexander University Erlangen, Nuremberg, Germany
| | - Michael Erdmann
- Department of Dermatology, University Hospital Erlangen, Comprehensive Cancer Center Erlangen - EMN, Friedrich-Alexander University Erlangen, Nuremberg, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Wiebke Sondermann
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Matthias Goebeler
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Bastian Schilling
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Jakob N Kather
- Division of Translational Medical Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Stefan Fröhling
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Daniel B Lipka
- National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
12
|
Mahesh DB, Madhuri B, Lakshmi D R. Integration of optimized local directional weber pattern with faster region convolutional neural network for enhanced medical image retrieval and classification. Comput Intell 2022. [DOI: 10.1111/coin.12506] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
13
|
Zhang Y, Xie F, Song X, Zheng Y, Liu J, Wang J. Dermoscopic image retrieval based on rotation-invariance deep hashing. Med Image Anal 2021; 77:102301. [PMID: 34836790 DOI: 10.1016/j.media.2021.102301] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2021] [Revised: 09/20/2021] [Accepted: 11/04/2021] [Indexed: 11/19/2022]
Abstract
Dermoscopic image retrieval technology can provide dermatologists with valuable information such as similar confirmed skin disease cases and diagnosis reports to assist doctors in their diagnosis. In this study, we design a dermoscopic image retrieval algorithm using convolutional neural networks (CNNs) and hash coding. A hybrid dilated convolution spatial attention module is proposed, which can focus on important information and suppress irrelevant information based on the complex morphological characteristics of dermoscopic images. Furthermore, we also propose a Cauchy rotation invariance loss function in view of the skin lesion target without the main direction. This function constrains CNNs to learn output differences in samples from different angles and to make CNNs obtain a certain rotation invariance. Extensive experiments are conducted on dermoscopic image datasets to verify the effectiveness and versatility of the proposed module, algorithm, and loss function. Experiment results show that the rotation-invariance deep hashing network with the proposed spatial attention module obtains better performance on the task of dermoscopic image retrieval.
Collapse
Affiliation(s)
- Yilan Zhang
- Image Processing Center, School of Astronautics, Beihang University, Beijing 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China.
| | - Fengying Xie
- Image Processing Center, School of Astronautics, Beihang University, Beijing 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China.
| | - Xuedong Song
- Image Processing Center, School of Astronautics, Beihang University, Beijing 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China; Shanghai Aerospace Control Technology Institute, Shanghai 201109, China
| | - Yushan Zheng
- Image Processing Center, School of Astronautics, Beihang University, Beijing 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing 100191, China
| | - Jie Liu
- Department of Dermatology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100730, China
| | - Juncheng Wang
- Department of Dermatology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100730, China
| |
Collapse
|
14
|
|
15
|
Yang Y, Wang J, Xie F, Liu J, Shu C, Wang Y, Zheng Y, Zhang H. A convolutional neural network trained with dermoscopic images of psoriasis performed on par with 230 dermatologists. Comput Biol Med 2021; 139:104924. [PMID: 34688173 DOI: 10.1016/j.compbiomed.2021.104924] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2021] [Revised: 09/29/2021] [Accepted: 10/04/2021] [Indexed: 01/25/2023]
Abstract
BACKGROUND Psoriasis is a common chronic inflammatory skin disease that causes physical and psychological burden to patients. A Convolutional Neural Network (CNN) focused on dermoscopic images would substantially aid the classification and increase the accuracy of diagnosis of psoriasis. OBJECTIVES This study aimed to train an efficient deep-learning network to recognize dermoscopic images of psoriasis (and other papulosquamous diseases), improving the accuracy of the diagnosis of psoriasis. METHODS EfficientNet-B4 architecture was trained with 7033 dermoscopic images from 1166 patients collected from the Department of Dermatology, Peking Union Medical College Hospital (China). We performed a five-fold cross-validation on the training set to compare the classification performance of EfficientNet-B4 over different networks commonly used in previous studies. From the test set, 90 images were used to compare the performance between our four-class model and that of board-certified dermatologists, whose diagnoses and information (e.g., age, titles) were obtained through an online questionnaire. RESULTS The mean sensitivity and specificity of EfficientNet-B4 on the training set was 0.927± 0.028 and 0.827 ± 0.043 for the two-class task, and 0.889 ± 0.014 and 0.968 ± 0.004 four-class task. The diagnostic sensitivity and specificity of the 230 dermatologists were 0.688 and 0.903 for psoriasis, 0.677 and 0.838 for eczema, 0.669 and 0.953 for lichen planus, and 0.832 and 0.932 for the "others" group, respectively; the diagnostic sensitivity and specificity of our four-class CNN was 0.929 and 0.952 for psoriasis, 0.773 and 0.926 for eczema, 0.933 and 0.960 for lichen planus, and 0.840 and 0.985 for the "others" group, respectively. Both the 230 dermatologists and CNN achieved at least moderate consistency with the reference standard, and there was no significant difference between them (P > 0.05). CONCLUSIONS The two-classification and four-classification models of psoriasis established in our study could accurately classify papulosquamous skin diseases. They showed generally comparable performances to the average level of dermatologists and would provide a strong support for the diagnosis of psoriasis.
Collapse
Affiliation(s)
- Yiguang Yang
- Image Processing Center, School of Astronautics, Beihang University, Beijing, 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, 100191, China
| | - Juncheng Wang
- Department of Dermatology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases, Beijing, 100730, China
| | - Fengying Xie
- Image Processing Center, School of Astronautics, Beihang University, Beijing, 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, 100191, China.
| | - Jie Liu
- Department of Dermatology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases, Beijing, 100730, China.
| | - Chang Shu
- Department of Dermatology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases, Beijing, 100730, China
| | - Yukun Wang
- Department of Dermatology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Science and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases, Beijing, 100730, China
| | - Yushan Zheng
- Image Processing Center, School of Astronautics, Beihang University, Beijing, 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, 100191, China
| | - Haopeng Zhang
- Image Processing Center, School of Astronautics, Beihang University, Beijing, 100191, China; Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, 100191, China
| |
Collapse
|
16
|
|
17
|
Deep learning-based, computer-aided classifier developed with dermoscopic images shows comparable performance to 164 dermatologists in cutaneous disease diagnosis in the Chinese population. Chin Med J (Engl) 2021; 133:2027-2036. [PMID: 32826613 PMCID: PMC7478660 DOI: 10.1097/cm9.0000000000001023] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023] Open
Abstract
Background Diagnoses of Skin diseases are frequently delayed in China due to lack of dermatologists. A deep learning-based diagnosis supporting system can facilitate pre-screening patients to prioritize dermatologists’ efforts. We aimed to evaluate the classification sensitivity and specificity of deep learning models to classify skin tumors and psoriasis for Chinese population with a modest number of dermoscopic images. Methods We developed a convolutional neural network (CNN) based on two datasets from a consecutive series of patients who underwent the dermoscopy in the clinic of the Department of Dermatology, Peking Union Medical College Hospital, between 2016 and 2018, prospectively. In order to evaluate the feasibility of the algorithm, we used two datasets. Dataset I consisted of 7192 dermoscopic images for a multi-class model to differentiate three most common skin tumors and other diseases. Dataset II consisted of 3115 dermoscopic images for a two-class model to classify psoriasis from other inflammatory diseases. We compared the performance of CNN with 164 dermatologists in a reader study with 130 dermoscopic images. The experts’ consensus was used as the reference standard except for the cases of basal cell carcinoma (BCC), which were all confirmed by histopathology. Results The accuracies of multi-class and two-class models were 81.49% ± 0.88% and 77.02% ± 1.81%, respectively. In the reader study, for the multi-class tasks, the diagnosis sensitivity and specificity of 164 dermatologists were 0.770 and 0.962 for BCC, 0.807 and 0.897 for melanocytic nevus, 0.624 and 0.976 for seborrheic keratosis, 0.939 and 0.875 for the “others” group, respectively; the diagnosis sensitivity and specificity of multi-class CNN were 0.800 and 1.000 for BCC, 0.800 and 0.840 for melanocytic nevus, 0.850 and 0.940 for seborrheic keratosis, 0.750 and 0.940 for the “others” group, respectively. For the two-class tasks, the sensitivity and specificity of dermatologists and CNN for classifying psoriasis were 0.872 and 0.838, 1.000 and 0.605, respectively. Both the dermatologists and CNN achieved at least moderate consistency with the reference standard, and there was no significant difference in Kappa coefficients between them (P > 0.05). Conclusions The performance of CNN developed with relatively modest number of dermoscopic images of skin tumors and psoriasis for Chinese population is comparable with 164 dermatologists. These two models could be used for screening in patients suspected with skin tumors and psoriasis respectively in primary care hospital.
Collapse
|
18
|
Lee EY, Maloney NJ, Cheng K, Bach DQ. Machine learning for precision dermatology: Advances, opportunities, and outlook. J Am Acad Dermatol 2021; 84:1458-1459. [PMID: 32645400 PMCID: PMC8023050 DOI: 10.1016/j.jaad.2020.06.1019] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2019] [Revised: 06/08/2020] [Accepted: 06/26/2020] [Indexed: 11/16/2022]
Affiliation(s)
- Ernest Y Lee
- Department of Bioengineering, University of California-Los Angeles; Division of Dermatology, Department of Medicine, University of California-Los Angeles; University of California-Los Angeles-Caltech Medical Scientist Training Program, David Geffen School of Medicine at University of California-Los Angeles.
| | - Nolan J Maloney
- Division of Dermatology, Department of Medicine, University of California-Los Angeles
| | - Kyle Cheng
- Division of Dermatology, Department of Medicine, University of California-Los Angeles
| | - Daniel Q Bach
- Division of Dermatology, Department of Medicine, University of California-Los Angeles
| |
Collapse
|
19
|
Rotemberg V, Kurtansky N, Betz-Stablein B, Caffery L, Chousakos E, Codella N, Combalia M, Dusza S, Guitera P, Gutman D, Halpern A, Helba B, Kittler H, Kose K, Langer S, Lioprys K, Malvehy J, Musthaq S, Nanda J, Reiter O, Shih G, Stratigos A, Tschandl P, Weber J, Soyer HP. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Sci Data 2021; 8:34. [PMID: 33510154 PMCID: PMC7843971 DOI: 10.1038/s41597-021-00815-z] [Citation(s) in RCA: 84] [Impact Index Per Article: 28.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2020] [Accepted: 12/18/2020] [Indexed: 11/09/2022] Open
Abstract
Prior skin image datasets have not addressed patient-level information obtained from multiple skin lesions from the same patient. Though artificial intelligence classification algorithms have achieved expert-level performance in controlled studies examining single images, in practice dermatologists base their judgment holistically from multiple lesions on the same patient. The 2020 SIIM-ISIC Melanoma Classification challenge dataset described herein was constructed to address this discrepancy between prior challenges and clinical practice, providing for each image in the dataset an identifier allowing lesions from the same patient to be mapped to one another. This patient-level contextual information is frequently used by clinicians to diagnose melanoma and is especially useful in ruling out false positives in patients with many atypical nevi. The dataset represents 2,056 patients (20.8% with at least one melanoma, 79.2% with zero melanomas) from three continents with an average of 16 lesions per patient, consisting of 33,126 dermoscopic images and 584 (1.8%) histopathologically confirmed melanomas compared with benign melanoma mimickers.
Collapse
Affiliation(s)
- Veronica Rotemberg
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA.
| | - Nicholas Kurtansky
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Brigid Betz-Stablein
- The University of Queensland Diamantina Institute, The University of Queensland, Dermatology Research Centre, Brisbane, Australia
| | - Liam Caffery
- The University of Queensland Diamantina Institute, The University of Queensland, Dermatology Research Centre, Brisbane, Australia
| | - Emmanouil Chousakos
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA.,University of Athens Medical School, Athens, Greece
| | | | - Marc Combalia
- Melanoma Unit, Dermatology Department, Hospital Cĺınic Barcelona, Universitat de Barcelona, IDIBAPS, Barcelona, Spain
| | - Stephen Dusza
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Pascale Guitera
- Melanoma Institute Australia and Sydney Melanoma Diagnostic Center, Sydney, Australia
| | - David Gutman
- Emory University School of Medicine, Department of Biomedical Informatics, Atlanta, GA, USA
| | - Allan Halpern
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | | | - Harald Kittler
- Medical University of Vienna, Department of Dermatology, Vienna, Austria
| | - Kivanc Kose
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Steve Langer
- Division of Radiology Informatics, Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | | | - Josep Malvehy
- Melanoma Unit, Dermatology Department, Hospital Cĺınic Barcelona, Universitat de Barcelona, IDIBAPS, Barcelona, Spain
| | - Shenara Musthaq
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA.,SUNY Downstate Medical School, New York, NY, USA
| | - Jabpani Nanda
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA.,Stony Brook Medical School, Stony Brook, NY, USA
| | - Ofer Reiter
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA.,Rabin Medical Center, Tel Aviv, Israel
| | - George Shih
- Department of Radiology, Weill Cornell Medical College, New York, NY, USA
| | | | - Philipp Tschandl
- Medical University of Vienna, Department of Dermatology, Vienna, Austria
| | - Jochen Weber
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - H Peter Soyer
- The University of Queensland Diamantina Institute, The University of Queensland, Dermatology Research Centre, Brisbane, Australia
| |
Collapse
|
20
|
Castro-Medina F, Rodríguez-Mazahua L, López-Chau A, Cervantes J, Alor-Hernández G, Machorro-Cano I. Application of Dynamic Fragmentation Methods in Multimedia Databases: A Review. ENTROPY 2020; 22:e22121352. [PMID: 33266019 PMCID: PMC7760714 DOI: 10.3390/e22121352] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/19/2020] [Revised: 11/20/2020] [Accepted: 11/24/2020] [Indexed: 11/16/2022]
Abstract
Fragmentation is a design technique widely used in multimedia databases, because it produces substantial benefits in reducing response times, causing lower execution costs in each operation performed. Multimedia databases include data whose main characteristic is their large size, therefore, database administrators face a challenge of great importance, since they must contemplate the different qualities of non-trivial data. These databases over time undergo changes in their access patterns. Different fragmentation techniques presented in related studies show adequate workflows, however, some do not contemplate changes in access patterns. This paper aims to provide an in-depth review of the literature related to dynamic fragmentation of multimedia databases, to identify the main challenges, technologies employed, types of fragmentation used, and characteristics of the cost model. This review provides valuable information for database administrators by showing essential characteristics to perform proper fragmentation and to improve the performance of fragmentation schemes. The reduction of costs in fragmentation methods is one of the most desired main properties. To fulfill this objective, the works include cost models, covering different qualities. In this analysis, a set of characteristics used in the cost models of each work is presented to facilitate the creation of a new cost model including the most used qualities. In addition, different data sets or reference points used in the testing stage of each work analyzed are presented.
Collapse
Affiliation(s)
- Felipe Castro-Medina
- Tecnológico Nacional de México/I. T. Orizaba, Division of Research and Postgraduate Studies, Av. Oriente 9 852. Col. Emiliano Zapata, C.P. 94320 Orizaba, Mexico; (F.C.-M.); (G.A.-H.)
| | - Lisbeth Rodríguez-Mazahua
- Tecnológico Nacional de México/I. T. Orizaba, Division of Research and Postgraduate Studies, Av. Oriente 9 852. Col. Emiliano Zapata, C.P. 94320 Orizaba, Mexico; (F.C.-M.); (G.A.-H.)
- Correspondence:
| | - Asdrúbal López-Chau
- Universidad Autónoma del Estado de México, Centro Universitario UAEM Zumpango, Camino viejo a Jilotzingo continuación Calle Rayón, Valle Hermoso, C.P. 55600 Zumpango, Estado de México, Mexico;
| | - Jair Cervantes
- Universidad Autónoma del Estado de México, Centro Universitario UAEM Texcoco, Av. Jardín Zumpango, s/n, Fraccionamiento El Tejocote, C.P. 56259 Texcoco, Estado de México, Mexico;
| | - Giner Alor-Hernández
- Tecnológico Nacional de México/I. T. Orizaba, Division of Research and Postgraduate Studies, Av. Oriente 9 852. Col. Emiliano Zapata, C.P. 94320 Orizaba, Mexico; (F.C.-M.); (G.A.-H.)
| | - Isaac Machorro-Cano
- Universidad del Papaloapan, Circuito Central #200, colonia Parque Industrial, C.P. 68301 Tuxtepec, Oaxaca, Mexico;
| |
Collapse
|
21
|
Lucius M, De All J, De All JA, Belvisi M, Radizza L, Lanfranconi M, Lorenzatti V, Galmarini CM. Deep Neural Frameworks Improve the Accuracy of General Practitioners in the Classification of Pigmented Skin Lesions. Diagnostics (Basel) 2020; 10:E969. [PMID: 33218060 PMCID: PMC7698907 DOI: 10.3390/diagnostics10110969] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2020] [Revised: 11/16/2020] [Accepted: 11/17/2020] [Indexed: 02/07/2023] Open
Abstract
This study evaluated whether deep learning frameworks trained in large datasets can help non-dermatologist physicians improve their accuracy in categorizing the seven most common pigmented skin lesions. Open-source skin images were downloaded from the International Skin Imaging Collaboration (ISIC) archive. Different deep neural networks (DNNs) (n = 8) were trained based on a random dataset constituted of 8015 images. A test set of 2003 images was used to assess the classifiers' performance at low (300 × 224 RGB) and high (600 × 450 RGB) image resolution and aggregated data (age, sex and lesion localization). We also organized two different contests to compare the DNN performance to that of general practitioners by means of unassisted image observation. Both at low and high image resolution, the DNN framework differentiated dermatological images with appreciable performance. In all cases, the accuracy was improved when adding clinical data to the framework. Finally, the least accurate DNN outperformed general practitioners. The physician's accuracy was statistically improved when allowed to use the output of this algorithmic framework as guidance. DNNs are proven to be high performers as skin lesion classifiers and can improve general practitioner diagnosis accuracy in a routine clinical scenario.
Collapse
Affiliation(s)
- Maximiliano Lucius
- Topazium Artificial Intelligence, Paseo de la Castellana 40 Pl 8, 28046 Madrid, Spain; (M.L.); (M.B.)
| | - Jorge De All
- Sanatorio Otamendi, C1115AAB Buenos Aires, Argentina; (J.D.A.); (J.A.D.A.); (M.L.); (V.L.)
| | - José Antonio De All
- Sanatorio Otamendi, C1115AAB Buenos Aires, Argentina; (J.D.A.); (J.A.D.A.); (M.L.); (V.L.)
| | - Martín Belvisi
- Topazium Artificial Intelligence, Paseo de la Castellana 40 Pl 8, 28046 Madrid, Spain; (M.L.); (M.B.)
| | - Luciana Radizza
- Instituto de Obra Social de las Fuerzas Armadas, C1115AAB Buenos Aires, Argentina;
| | - Marisa Lanfranconi
- Sanatorio Otamendi, C1115AAB Buenos Aires, Argentina; (J.D.A.); (J.A.D.A.); (M.L.); (V.L.)
| | - Victoria Lorenzatti
- Sanatorio Otamendi, C1115AAB Buenos Aires, Argentina; (J.D.A.); (J.A.D.A.); (M.L.); (V.L.)
| | - Carlos M. Galmarini
- Topazium Artificial Intelligence, Paseo de la Castellana 40 Pl 8, 28046 Madrid, Spain; (M.L.); (M.B.)
| |
Collapse
|
22
|
Blum A, Bosch S, Haenssle HA, Fink C, Hofmann-Wellenhof R, Zalaudek I, Kittler H, Tschandl P. [Artificial intelligence and smartphone program applications (Apps) : Relevance for dermatological practice]. Hautarzt 2020; 71:691-698. [PMID: 32720165 DOI: 10.1007/s00105-020-04658-4] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
Abstract
ADVANTAGES OF ARTIFICIAL INTELLIGENCE (AI) With responsible, safe and successful use of artificial intelligence (AI), possible advantages in the field of dermato-oncology include the following: (1) medical work can focus on skin cancer patients, (2) patients can be more quickly and effectively treated despite the increasing incidence of skin cancer and the decreasing number of actively working dermatologists and (3) users can learn from the AI results. POTENTIAL DISADVANTAGES AND RISKS OF AI USE: (1) Lack of mutual trust can develop due to the decreased patient-physician contact, (2) additional time effort will be necessary to promptly evaluate the AI-classified benign lesions, (3) lack of adequate medical experience to recognize misclassified AI decisions and (4) recontacting a patient in due time in the case of incorrect AI classifications. Still problematic in the use of AI are the medicolegal situation and remuneration. Apps using AI currently cannot provide sufficient assistance based on clinical images of skin cancer. REQUIREMENTS AND POSSIBLE USE OF SMARTPHONE PROGRAM APPLICATIONS Smartphone program applications (apps) can be implemented responsibly when the image quality is good, the patient's history can be entered easily, transmission of the image and results are assured and medicolegal aspects as well as remuneration are clarified. Apps can be used for disease-specific information material and can optimize patient care by using teledermatology.
Collapse
Affiliation(s)
- A Blum
- Hautarzt- und Lehrpraxis, Augustinerplatz 7, 78462, Konstanz, Deutschland.
| | - S Bosch
- Hautarztpraxis, Ludwigsburg, Deutschland
| | - H A Haenssle
- Universitäts-Hautklinik Heidelberg, Heidelberg, Deutschland
| | - C Fink
- Universitäts-Hautklinik Heidelberg, Heidelberg, Deutschland
| | - R Hofmann-Wellenhof
- Universitätsklinik für Dermatologie, Medizinische Universität Graz, Graz, Österreich
| | - I Zalaudek
- Dermatology Clinic, University Hospital of Trieste, Hospital Maggiore, Trieste, Italien
| | - H Kittler
- Universitätsklinik für Dermatologie, Medizinische Universität Wien, Wien, Österreich
| | - P Tschandl
- Universitätsklinik für Dermatologie, Medizinische Universität Wien, Wien, Österreich
| |
Collapse
|
23
|
Tschandl P, Rinner C, Apalla Z, Argenziano G, Codella N, Halpern A, Janda M, Lallas A, Longo C, Malvehy J, Paoli J, Puig S, Rosendahl C, Soyer HP, Zalaudek I, Kittler H. Human-computer collaboration for skin cancer recognition. Nat Med 2020; 26:1229-1234. [PMID: 32572267 DOI: 10.1038/s41591-020-0942-0] [Citation(s) in RCA: 286] [Impact Index Per Article: 71.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2019] [Accepted: 05/15/2020] [Indexed: 01/13/2023]
Abstract
The rapid increase in telemedicine coupled with recent advances in diagnostic artificial intelligence (AI) create the imperative to consider the opportunities and risks of inserting AI-based support into new paradigms of care. Here we build on recent achievements in the accuracy of image-based AI for skin cancer diagnosis to address the effects of varied representations of AI-based support across different levels of clinical expertise and multiple clinical workflows. We find that good quality AI-based support of clinical decision-making improves diagnostic accuracy over that of either AI or physicians alone, and that the least experienced clinicians gain the most from AI-based support. We further find that AI-based multiclass probabilities outperformed content-based image retrieval (CBIR) representations of AI in the mobile technology environment, and AI-based support had utility in simulations of second opinions and of telemedicine triage. In addition to demonstrating the potential benefits associated with good quality AI in the hands of non-expert clinicians, we find that faulty AI can mislead the entire spectrum of clinicians, including experts. Lastly, we show that insights derived from AI class-activation maps can inform improvements in human diagnosis. Together, our approach and findings offer a framework for future studies across the spectrum of image-based diagnostics to improve human-computer collaboration in clinical practice.
Collapse
Affiliation(s)
- Philipp Tschandl
- ViDIR Group, Department of Dermatology, Medical University of Vienna, Vienna, Austria
| | - Christoph Rinner
- Center for Medical Statistics, Informatics and Intelligent Systems (CeMSIIS), Medical University of Vienna, Vienna, Austria
| | - Zoe Apalla
- Department of Dermatology, Aristotle University of Thessaloniki, Thessaloniki, Greece
| | | | - Noel Codella
- IBM T. J. Watson Research Center, New York, NY, USA
| | - Allan Halpern
- Dermatology Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Monika Janda
- Centre for Health Services Research, Faculty of Medicine, The University of Queensland, Brisbane, Queensland, Australia
| | - Aimilios Lallas
- Department of Dermatology, Aristotle University of Thessaloniki, Thessaloniki, Greece
| | - Caterina Longo
- Dermatology Unit, University of Modena and Reggio Emilia, Modena, Italy.,Centro Oncologico ad Alta Tecnologia Diagnostica-Dermatologia, Azienda Unità Sanitaria Locale-IRCCS di Reggio Emilia, Reggio Emilia, Italy
| | - Josep Malvehy
- Dermatology Department, Melanoma Unit, Hospital Clínic de Barcelona, IDIBAPS, Universitat de Barcelona, Barcelona, Spain.,Centro de Investigación Biomédica en Red de Enfermedades Raras (CIBER ER), Instituto de Salud Carlos III, Barcelona, Spain
| | - John Paoli
- Department of Dermatology and Venereology, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden.,Department of Dermatology and Venereology, Region Västra Götaland, Sahlgrenska University Hospital, Gothenburg, Sweden
| | - Susana Puig
- Dermatology Department, Melanoma Unit, Hospital Clínic de Barcelona, IDIBAPS, Universitat de Barcelona, Barcelona, Spain.,Centro de Investigación Biomédica en Red de Enfermedades Raras (CIBER ER), Instituto de Salud Carlos III, Barcelona, Spain
| | - Cliff Rosendahl
- Faculty of Medicine, The University of Queensland, Brisbane, Queensland, Australia
| | - H Peter Soyer
- Dermatology Research Centre, The University of Queensland Diamantina Institute, The University of Queensland, Brisbane, Queensland, Australia
| | - Iris Zalaudek
- Department of Dermatology, Medical University of Trieste, Trieste, Italy
| | - Harald Kittler
- ViDIR Group, Department of Dermatology, Medical University of Vienna, Vienna, Austria.
| |
Collapse
|
24
|
Ganascia JG. [The future of AI in dermatology and of dermatology with AI]. Ann Dermatol Venereol 2020; 147:331-333. [PMID: 32241556 DOI: 10.1016/j.annder.2020.03.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/01/2022]
Affiliation(s)
- J-G Ganascia
- Sorbonne Université - LIP6, B.C. 169, 4, place Jussieu, 75252 Paris cedex 05, France.
| |
Collapse
|
25
|
Artificial Intelligence in Dermatology: A Primer. J Invest Dermatol 2020; 140:1504-1512. [PMID: 32229141 DOI: 10.1016/j.jid.2020.02.026] [Citation(s) in RCA: 84] [Impact Index Per Article: 21.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2019] [Revised: 02/22/2020] [Accepted: 02/25/2020] [Indexed: 01/17/2023]
Abstract
Artificial intelligence is becoming increasingly important in dermatology, with studies reporting accuracy matching or exceeding dermatologists for the diagnosis of skin lesions from clinical and dermoscopic images. However, real-world clinical validation is currently lacking. We review dermatological applications of deep learning, the leading artificial intelligence technology for image analysis, and discuss its current capabilities, potential failure modes, and challenges surrounding performance assessment and interpretability. We address the following three primary applications: (i) teledermatology, including triage for referral to dermatologists; (ii) augmenting clinical assessment during face-to-face visits; and (iii) dermatopathology. We discuss equity and ethical issues related to future clinical adoption and recommend specific standardization of metrics for reporting model performance.
Collapse
|
26
|
Sadeghi M, Chilana P, Yap J, Tschandl P, Atkins MS. Using content-based image retrieval of dermoscopic images for interpretation and education: A pilot study. Skin Res Technol 2019; 26:503-512. [PMID: 31845429 DOI: 10.1111/srt.12822] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2019] [Accepted: 11/09/2019] [Indexed: 12/26/2022]
Abstract
BACKGROUND Dermoscopic content-based image retrieval (CBIR) systems provide a set of visually similar dermoscopic (magnified and illuminated) skin images with a pathology-confirmed diagnosis for a given dermoscopic query image of a skin lesion. Although recent advances in machine learning have spurred novel CBIR algorithms, we have few insights into how end users interact with CBIRs and to what extent CBIRs can be useful for education and image interpretation. MATERIALS AND METHODS We developed an interactive user interface for a CBIR system with dermoscopic images as a decision support tool and investigated users' interactions and decisions with the system. We performed a pilot experiment with 14 non-medically trained users for a given set of annotated dermoscopic images. RESULTS Our pilot showed that the number of correct classifications and users' confidence levels significantly increased with the CBIR interface compared with a non-CBIR interface, although the timing also increased significantly. The users found the CBIR interface of high educational value, engaging and easy to use. CONCLUSION Overall, users became more accurate, found the CBIR approach provided a useful decision aid, and had educational value for learning about skin conditions.
Collapse
Affiliation(s)
- Mahya Sadeghi
- School of Computing Science, Simon Fraser University, Burnaby, BC, Canada
| | - Parmit Chilana
- School of Computing Science, Simon Fraser University, Burnaby, BC, Canada
| | - Jordan Yap
- MetaOptima Technology Inc., Vancouver, BC, Canada
| | - Philipp Tschandl
- School of Computing Science, Simon Fraser University, Burnaby, BC, Canada.,Department of Dermatology, Medical University of Vienna, Vienna, Austria
| | - M Stella Atkins
- School of Computing Science, Simon Fraser University, Burnaby, BC, Canada.,School of Dermatology and Skin Science, University of British Columbia, Vancouver, BC, Canada
| |
Collapse
|
27
|
|
28
|
Rotemberg V, Halpern A. Towards 'interpretable' artificial intelligence for dermatology. Br J Dermatol 2019; 181:5-6. [PMID: 31259397 DOI: 10.1111/bjd.18038] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Affiliation(s)
- V Rotemberg
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, U.S.A
| | - A Halpern
- Dermatology Service, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, U.S.A
| |
Collapse
|
29
|
Sadeghi M, Chilana PK, Atkins MS. How Users Perceive Content-Based Image Retrieval for Identifying Skin Images. UNDERSTANDING AND INTERPRETING MACHINE LEARNING IN MEDICAL IMAGE COMPUTING APPLICATIONS 2018. [DOI: 10.1007/978-3-030-02628-8_16] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
|