1
|
Ghezloo F, Chang OH, Knezevich SR, Shaw KC, Thigpen KG, Reisch LM, Shapiro LG, Elmore JG. Robust ROI Detection in Whole Slide Images Guided by Pathologists' Viewing Patterns. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024:10.1007/s10278-024-01202-x. [PMID: 39122892 DOI: 10.1007/s10278-024-01202-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/01/2024] [Revised: 06/24/2024] [Accepted: 07/05/2024] [Indexed: 08/12/2024]
Abstract
Deep learning techniques offer improvements in computer-aided diagnosis systems. However, acquiring image domain annotations is challenging due to the knowledge and commitment required of expert pathologists. Pathologists often identify regions in whole slide images with diagnostic relevance rather than examining the entire slide, with a positive correlation between the time spent on these critical image regions and diagnostic accuracy. In this paper, a heatmap is generated to represent pathologists' viewing patterns during diagnosis and used to guide a deep learning architecture during training. The proposed system outperforms traditional approaches based on color and texture image characteristics, integrating pathologists' domain expertise to enhance region of interest detection without needing individual case annotations. Evaluating our best model, a U-Net model with a pre-trained ResNet-18 encoder, on a skin biopsy whole slide image dataset for melanoma diagnosis, shows its potential in detecting regions of interest, surpassing conventional methods with an increase of 20%, 11%, 22%, and 12% in precision, recall, F1-score, and Intersection over Union, respectively. In a clinical evaluation, three dermatopathologists agreed on the model's effectiveness in replicating pathologists' diagnostic viewing behavior and accurately identifying critical regions. Finally, our study demonstrates that incorporating heatmaps as supplementary signals can enhance the performance of computer-aided diagnosis systems. Without the availability of eye tracking data, identifying precise focus areas is challenging, but our approach shows promise in assisting pathologists in improving diagnostic accuracy and efficiency, streamlining annotation processes, and aiding the training of new pathologists.
Collapse
Affiliation(s)
- Fatemeh Ghezloo
- Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA.
| | - Oliver H Chang
- Department of Laboratory Medicine and Pathology, University of Washington, Seattle, WA, USA
| | | | | | | | - Lisa M Reisch
- Department of Biostatistics, University of Washington, Seattle, WA, USA
| | - Linda G Shapiro
- Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA
| | - Joann G Elmore
- Department of Medicine, David Geffen School of Medicine, University of California, Los AngelesLos Angeles, CA, USA
| |
Collapse
|
2
|
McGenity C, Clarke EL, Jennings C, Matthews G, Cartlidge C, Freduah-Agyemang H, Stocken DD, Treanor D. Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy. NPJ Digit Med 2024; 7:114. [PMID: 38704465 PMCID: PMC11069583 DOI: 10.1038/s41746-024-01106-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2023] [Accepted: 04/12/2024] [Indexed: 05/06/2024] Open
Abstract
Ensuring diagnostic performance of artificial intelligence (AI) before introduction into clinical practice is essential. Growing numbers of studies using AI for digital pathology have been reported over recent years. The aim of this work is to examine the diagnostic accuracy of AI in digital pathology images for any disease. This systematic review and meta-analysis included diagnostic accuracy studies using any type of AI applied to whole slide images (WSIs) for any disease. The reference standard was diagnosis by histopathological assessment and/or immunohistochemistry. Searches were conducted in PubMed, EMBASE and CENTRAL in June 2022. Risk of bias and concerns of applicability were assessed using the QUADAS-2 tool. Data extraction was conducted by two investigators and meta-analysis was performed using a bivariate random effects model, with additional subgroup analyses also performed. Of 2976 identified studies, 100 were included in the review and 48 in the meta-analysis. Studies were from a range of countries, including over 152,000 whole slide images (WSIs), representing many diseases. These studies reported a mean sensitivity of 96.3% (CI 94.1-97.7) and mean specificity of 93.3% (CI 90.5-95.4). There was heterogeneity in study design and 99% of studies identified for inclusion had at least one area at high or unclear risk of bias or applicability concerns. Details on selection of cases, division of model development and validation data and raw performance data were frequently ambiguous or missing. AI is reported as having high diagnostic accuracy in the reported areas but requires more rigorous evaluation of its performance.
Collapse
Affiliation(s)
- Clare McGenity
- University of Leeds, Leeds, UK.
- Leeds Teaching Hospitals NHS Trust, Leeds, UK.
| | - Emily L Clarke
- University of Leeds, Leeds, UK
- Leeds Teaching Hospitals NHS Trust, Leeds, UK
| | - Charlotte Jennings
- University of Leeds, Leeds, UK
- Leeds Teaching Hospitals NHS Trust, Leeds, UK
| | | | | | | | | | - Darren Treanor
- University of Leeds, Leeds, UK
- Leeds Teaching Hospitals NHS Trust, Leeds, UK
- Department of Clinical Pathology and Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden
- Centre for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden
| |
Collapse
|
3
|
Yee J, Rosendahl C, Aoude LG. The role of artificial intelligence and convolutional neural networks in the management of melanoma: a clinical, pathological, and radiological perspective. Melanoma Res 2024; 34:96-104. [PMID: 38141179 PMCID: PMC10906187 DOI: 10.1097/cmr.0000000000000951] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2023] [Accepted: 11/29/2023] [Indexed: 12/25/2023]
Abstract
Clinical dermatoscopy and pathological slide assessment are essential in the diagnosis and management of patients with cutaneous melanoma. For those presenting with stage IIC disease and beyond, radiological investigations are often considered. The dermatoscopic, whole slide and radiological images used during clinical care are often stored digitally, enabling artificial intelligence (AI) and convolutional neural networks (CNN) to learn, analyse and contribute to the clinical decision-making. A keyword search of the Medline database was performed to assess the progression, capabilities and limitations of AI and CNN and its use in diagnosis and management of cutaneous melanoma. Full-text articles were reviewed if they related to dermatoscopy, pathological slide assessment or radiology. Through analysis of 95 studies, we demonstrate that diagnostic accuracy of AI/CNN can be superior (or at least equal) to clinicians. However, variability in image acquisition, pre-processing, segmentation, and feature extraction remains challenging. With current technological abilities, AI/CNN and clinicians synergistically working together are better than one another in all subspecialty domains relating to cutaneous melanoma. AI has the potential to enhance the diagnostic capabilities of junior dermatology trainees, primary care skin cancer clinicians and general practitioners. For experienced clinicians, AI provides a cost-efficient second opinion. From a pathological and radiological perspective, CNN has the potential to improve workflow efficiency, allowing clinicians to achieve more in a finite amount of time. Until the challenges of AI/CNN are reliably met, however, they can only remain an adjunct to clinical decision-making.
Collapse
Affiliation(s)
- Joshua Yee
- Faculty of Medicine, University of Queensland, St Lucia
| | - Cliff Rosendahl
- Primary Care Clinical Unit, Medical School, The University of Queensland, Herston
| | - Lauren G. Aoude
- Frazer Institute, The University of Queensland, Woolloongabba, QLD, Australia
| |
Collapse
|
4
|
Neimy H, Helmy JE, Snyder A, Valdebran M. Artificial Intelligence in Melanoma Dermatopathology: A Review of Literature. Am J Dermatopathol 2024; 46:83-94. [PMID: 37982502 DOI: 10.1097/dad.0000000000002593] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2023]
Abstract
ABSTRACT Pathology serves as a promising field to integrate artificial intelligence into clinical practice as a powerful screening tool. Melanoma is a common skin cancer with high mortality and morbidity, requiring timely and accurate histopathologic diagnosis. This review explores applications of artificial intelligence in melanoma dermatopathology, including differential diagnostics, prognosis prediction, and personalized medicine decision-making.
Collapse
Affiliation(s)
- Hannah Neimy
- College of Medicine, Medical University of South Carolina, Charleston, SC; and
| | - John Elia Helmy
- College of Medicine, Medical University of South Carolina, Charleston, SC; and
| | - Alan Snyder
- Department of Dermatology & Dermatologic Surgery, Medical University of South Carolina, Charleston, SC
| | - Manuel Valdebran
- Department of Dermatology & Dermatologic Surgery, Medical University of South Carolina, Charleston, SC
| |
Collapse
|
5
|
Lin Z, Shen H, Liu X, Ma W, Wang M, Ruan J, Yu H, Ma S, Sun X. Recent advances of artificial intelligence in melanoma clinical practice. Melanoma Res 2023; 33:454-461. [PMID: 37696256 DOI: 10.1097/cmr.0000000000000922] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/13/2023]
Abstract
Skin melanoma is a lethal cancer. The incidence of melanoma is increasing rapidly in all regions of the world. Despite significant breakthroughs in melanoma treatment in recent years, precise diagnosis of melanoma is still a challenge in some cases. Even specialized physicians may need time and effort to make accurate judgments. As artificial intelligence (AI) technology advances into medical practice, it may bring new solutions to this problem based on its efficiency, accuracy, and speed. This paper summarizes the recent progress of AI in melanoma-related applications, including melanoma diagnosis and classification, the discovery of new medication, guiding treatment, and prognostic assessment. The paper also compares the effectiveness of various algorithms in melanoma application and suggests future research directions for AI in melanoma clinical practice.
Collapse
Affiliation(s)
- Zijun Lin
- Guangdong Provincial Key Laboratory of Medical Molecular Diagnostics, The First Dongguan Affiliated Hospital, Guangdong Medical University
- Institute of Aging Research, School of Medical Technology, Guangdong Medical University
| | - Haoyan Shen
- School of Biomedical Engineering, Guangdong Medical University
| | - Xinguang Liu
- Guangdong Provincial Key Laboratory of Medical Molecular Diagnostics, The First Dongguan Affiliated Hospital, Guangdong Medical University
- Institute of Aging Research, School of Medical Technology, Guangdong Medical University
| | - Wanrui Ma
- Department of General Medicine, The First Dongguan Affiliated Hospital, Guangdong Medical University, Dongguan
| | - Mingfa Wang
- Department of Pathology, The Second Affiliated Hospital of Hainan Medical University, Haikou
| | - Jie Ruan
- Institute of Aging Research, School of Medical Technology, Guangdong Medical University
| | - Hongbin Yu
- Guangdong Provincial Key Laboratory of Medical Molecular Diagnostics, Chinese American Tumor Institute, Guangdong Medical University, Dongguan, China
| | - Sha Ma
- School of Biomedical Engineering, Guangdong Medical University
| | - Xuerong Sun
- Guangdong Provincial Key Laboratory of Medical Molecular Diagnostics, The First Dongguan Affiliated Hospital, Guangdong Medical University
- Institute of Aging Research, School of Medical Technology, Guangdong Medical University
| |
Collapse
|
6
|
Ugolini F, De Logu F, Iannone LF, Brutti F, Simi S, Maio V, de Giorgi V, Maria di Giacomo A, Miracco C, Federico F, Peris K, Palmieri G, Cossu A, Mandalà M, Massi D, Laurino M. Tumor-Infiltrating Lymphocyte Recognition in Primary Melanoma by Deep Learning Convolutional Neural Network. THE AMERICAN JOURNAL OF PATHOLOGY 2023; 193:2099-2110. [PMID: 37734590 DOI: 10.1016/j.ajpath.2023.08.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/18/2023] [Revised: 07/25/2023] [Accepted: 08/02/2023] [Indexed: 09/23/2023]
Abstract
The presence of tumor-infiltrating lymphocytes (TILs) is associated with a favorable prognosis of primary melanoma (PM). Recently, artificial intelligence (AI)-based approach in digital pathology was proposed for the standardized assessment of TILs on hematoxylin and eosin-stained whole slide images (WSIs). Herein, the study applied a new convolution neural network (CNN) analysis of PM WSIs to automatically assess the infiltration of TILs and extract a TIL score. A CNN was trained and validated in a retrospective cohort of 307 PMs including a training set (237 WSIs, 57,758 patches) and an independent testing set (70 WSIs, 29,533 patches). An AI-based TIL density index (AI-TIL) was identified after the classification of tumor patches by the presence or absence of TILs. The proposed CNN showed high performance in recognizing TILs in PM WSIs, showing 100% specificity and sensitivity on the testing set. The AI-based TIL index correlated with conventional TIL evaluation and clinical outcome. The AI-TIL index was an independent prognostic marker associated directly with a favorable prognosis. A fully automated and standardized AI-TIL appeared to be superior to conventional methods at differentiating the PM clinical outcome. Further studies are required to develop an easy-to-use tool to assist pathologists to assess TILs in the clinical evaluation of solid tumors.
Collapse
Affiliation(s)
- Filippo Ugolini
- Section of Pathological Anatomy, Department of Health Sciences, University of Florence, Florence, Italy
| | - Francesco De Logu
- Section of Clinical Pharmacology and Oncology, Department of Health Sciences, University of Florence, Florence, Italy
| | - Luigi Francesco Iannone
- Section of Clinical Pharmacology and Oncology, Department of Health Sciences, University of Florence, Florence, Italy
| | - Francesca Brutti
- Institute of Clinical Physiology, National Research Council, Pisa, Italy
| | - Sara Simi
- Section of Pathological Anatomy, Department of Health Sciences, University of Florence, Florence, Italy
| | - Vincenza Maio
- Section of Pathological Anatomy, Department of Health Sciences, University of Florence, Florence, Italy
| | - Vincenzo de Giorgi
- Section of Dermatology, Department of Health Sciences, University of Florence, Florence, Italy
| | - Anna Maria di Giacomo
- Center for Immuno-Oncology, Medical Oncology and Immunotherapy, University of Siena, Siena, Italy
| | - Clelia Miracco
- Center for Immuno-Oncology, Medical Oncology and Immunotherapy, University of Siena, Siena, Italy
| | | | - Ketty Peris
- Institute of Dermatology, Sacred Heart Catholic University, Rome, Italy
| | - Giuseppe Palmieri
- Unit of Cancer Genetics, Institute of Genetic and Biomedical Research, National Research Council, Sassari, Italy
| | - Antonio Cossu
- Department of Medical, Surgical and Experimental Sciences, University of Sassari, Sassari, Italy
| | - Mario Mandalà
- Oncology Unit, Department of Medicine and Surgery, University of Perugia, Perugia, Italy
| | - Daniela Massi
- Section of Pathological Anatomy, Department of Health Sciences, University of Florence, Florence, Italy
| | - Marco Laurino
- Institute of Clinical Physiology, National Research Council, Pisa, Italy.
| |
Collapse
|
7
|
Brutti F, La Rosa F, Lazzeri L, Benvenuti C, Bagnoni G, Massi D, Laurino M. Artificial Intelligence Algorithms for Benign vs. Malignant Dermoscopic Skin Lesion Image Classification. Bioengineering (Basel) 2023; 10:1322. [PMID: 38002446 PMCID: PMC10669580 DOI: 10.3390/bioengineering10111322] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2023] [Revised: 11/13/2023] [Accepted: 11/14/2023] [Indexed: 11/26/2023] Open
Abstract
In recent decades, the incidence of melanoma has grown rapidly. Hence, early diagnosis is crucial to improving clinical outcomes. Here, we propose and compare a classical image analysis-based machine learning method with a deep learning one to automatically classify benign vs. malignant dermoscopic skin lesion images. The same dataset of 25,122 publicly available dermoscopic images was used to train both models, while a disjointed test set of 200 images was used for the evaluation phase. The training dataset was randomly divided into 10 datasets of 19,932 images to obtain an equal distribution between the two classes. By testing both models on the disjoint set, the deep learning-based method returned accuracy of 85.4 ± 3.2% and specificity of 75.5 ± 7.6%, while the machine learning one showed accuracy and specificity of 73.8 ± 1.1% and 44.5 ± 4.7%, respectively. Although both approaches performed well in the validation phase, the convolutional neural network outperformed the ensemble boosted tree classifier on the disjoint test set, showing better generalization ability. The integration of new melanoma detection algorithms with digital dermoscopic devices could enable a faster screening of the population, improve patient management, and achieve better survival rates.
Collapse
Affiliation(s)
- Francesca Brutti
- Institute of Clinical Physiology, National Research Council, 56124 Pisa, Italy; (F.B.); (F.L.R.); (C.B.)
| | - Federica La Rosa
- Institute of Clinical Physiology, National Research Council, 56124 Pisa, Italy; (F.B.); (F.L.R.); (C.B.)
| | - Linda Lazzeri
- Uniti of Dermatologia, Specialist Surgery Area, Department of General Surgery, Livorno Hospital, Azienda Usl Toscana Nord Ovest, 57124 Livorno, Italy; (L.L.); (G.B.)
| | - Chiara Benvenuti
- Institute of Clinical Physiology, National Research Council, 56124 Pisa, Italy; (F.B.); (F.L.R.); (C.B.)
| | - Giovanni Bagnoni
- Uniti of Dermatologia, Specialist Surgery Area, Department of General Surgery, Livorno Hospital, Azienda Usl Toscana Nord Ovest, 57124 Livorno, Italy; (L.L.); (G.B.)
| | - Daniela Massi
- Department of Health Sciences, Section of Pathological Anatomy, University of Florence, 50139 Florence, Italy;
| | - Marco Laurino
- Institute of Clinical Physiology, National Research Council, 56124 Pisa, Italy; (F.B.); (F.L.R.); (C.B.)
| |
Collapse
|
8
|
Del Amor R, Pérez-Cano J, López-Pérez M, Terradez L, Aneiros-Fernandez J, Morales S, Mateos J, Molina R, Naranjo V. Annotation protocol and crowdsourcing multiple instance learning classification of skin histological images: The CR-AI4SkIN dataset. Artif Intell Med 2023; 145:102686. [PMID: 37925214 DOI: 10.1016/j.artmed.2023.102686] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Revised: 10/11/2023] [Accepted: 10/13/2023] [Indexed: 11/06/2023]
Abstract
Digital Pathology (DP) has experienced a significant growth in recent years and has become an essential tool for diagnosing and prognosis of tumors. The availability of Whole Slide Images (WSIs) and the implementation of Deep Learning (DL) algorithms have paved the way for the appearance of Artificial Intelligence (AI) systems that support the diagnosis process. These systems require extensive and varied data for their training to be successful. However, creating labeled datasets in histopathology is laborious and time-consuming. We have developed a crowdsourcing-multiple instance labeling/learning protocol that is applied to the creation and use of the CR-AI4SkIN dataset.2 CR-AI4SkIN contains 271 WSIs of 7 Cutaneous Spindle Cell (CSC) neoplasms with expert and non-expert labels at region and WSI levels. It is the first dataset of these types of neoplasms made available. The regions selected by the experts are used to learn an automatic extractor of Regions of Interest (ROIs) from WSIs. To produce the embedding of each WSI, the representations of patches within the ROIs are obtained using a contrastive learning method, and then combined. Finally, they are fed to a Gaussian process-based crowdsourcing classifier, which utilizes the noisy non-expert WSI labels. We validate our crowdsourcing-multiple instance learning method in the CR-AI4SkIN dataset, addressing a binary classification problem (malign vs. benign). The proposed method obtains an F1 score of 0.7911 on the test set, outperforming three widely used aggregation methods for crowdsourcing tasks. Furthermore, our crowdsourcing method also outperforms the supervised model with expert labels on the test set (F1-score = 0.6035). The promising results support the proposed crowdsourcing multiple instance learning annotation protocol. It also validates the automatic extraction of interest regions and the use of contrastive embedding and Gaussian process classification to perform crowdsourcing classification tasks.
Collapse
Affiliation(s)
- Rocío Del Amor
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València, Valencia, Spain
| | - Jose Pérez-Cano
- Department of Computer Science and Artificial Intelligence, University of Granada, 18010 Granada, Spain
| | - Miguel López-Pérez
- Department of Computer Science and Artificial Intelligence, University of Granada, 18010 Granada, Spain.
| | - Liria Terradez
- Pathology Department. Hospital Clínico Universitario de Valencia, Universidad de Valencia, Spain
| | | | - Sandra Morales
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València, Valencia, Spain
| | - Javier Mateos
- Department of Computer Science and Artificial Intelligence, University of Granada, 18010 Granada, Spain
| | - Rafael Molina
- Department of Computer Science and Artificial Intelligence, University of Granada, 18010 Granada, Spain
| | - Valery Naranjo
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València, Valencia, Spain
| |
Collapse
|
9
|
Sauter D, Lodde G, Nensa F, Schadendorf D, Livingstone E, Kukuk M. Deep learning in computational dermatopathology of melanoma: A technical systematic literature review. Comput Biol Med 2023; 163:107083. [PMID: 37315382 DOI: 10.1016/j.compbiomed.2023.107083] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2022] [Revised: 05/10/2023] [Accepted: 05/27/2023] [Indexed: 06/16/2023]
Abstract
Deep learning (DL) has become one of the major approaches in computational dermatopathology, evidenced by a significant increase in this topic in the current literature. We aim to provide a structured and comprehensive overview of peer-reviewed publications on DL applied to dermatopathology focused on melanoma. In comparison to well-published DL methods on non-medical images (e.g., classification on ImageNet), this field of application comprises a specific set of challenges, such as staining artifacts, large gigapixel images, and various magnification levels. Thus, we are particularly interested in the pathology-specific technical state-of-the-art. We also aim to summarize the best performances achieved thus far with respect to accuracy, along with an overview of self-reported limitations. Accordingly, we conducted a systematic literature review of peer-reviewed journal and conference articles published between 2012 and 2022 in the databases ACM Digital Library, Embase, IEEE Xplore, PubMed, and Scopus, expanded by forward and backward searches to identify 495 potentially eligible studies. After screening for relevance and quality, a total of 54 studies were included. We qualitatively summarized and analyzed these studies from technical, problem-oriented, and task-oriented perspectives. Our findings suggest that the technical aspects of DL for histopathology in melanoma can be further improved. The DL methodology was adopted later in this field, and still lacks the wider adoption of DL methods already shown to be effective for other applications. We also discuss upcoming trends toward ImageNet-based feature extraction and larger models. While DL has achieved human-competitive accuracy in routine pathological tasks, its performance on advanced tasks is still inferior to wet-lab testing (for example). Finally, we discuss the challenges impeding the translation of DL methods to clinical practice and provide insight into future research directions.
Collapse
Affiliation(s)
- Daniel Sauter
- Department of Computer Science, Fachhochschule Dortmund, 44227 Dortmund, Germany.
| | - Georg Lodde
- Department of Dermatology, University Hospital Essen, 45147 Essen, Germany
| | - Felix Nensa
- Institute for AI in Medicine (IKIM), University Hospital Essen, 45131 Essen, Germany; Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, 45147 Essen, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, 45147 Essen, Germany
| | | | - Markus Kukuk
- Department of Computer Science, Fachhochschule Dortmund, 44227 Dortmund, Germany
| |
Collapse
|
10
|
Park HS, Chong Y, Lee Y, Yim K, Seo KJ, Hwang G, Kim D, Gong G, Cho NH, Yoo CW, Choi HJ. Deep Learning-Based Computational Cytopathologic Diagnosis of Metastatic Breast Carcinoma in Pleural Fluid. Cells 2023; 12:1847. [PMID: 37508511 PMCID: PMC10377793 DOI: 10.3390/cells12141847] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Revised: 07/07/2023] [Accepted: 07/11/2023] [Indexed: 07/30/2023] Open
Abstract
A Pleural effusion cytology is vital for treating metastatic breast cancer; however, concerns have arisen regarding the low accuracy and inter-observer variability in cytologic diagnosis. Although artificial intelligence-based image analysis has shown promise in cytopathology research, its application in diagnosing breast cancer in pleural fluid remains unexplored. To overcome these limitations, we evaluate the diagnostic accuracy of an artificial intelligence-based model using a large collection of cytopathological slides, to detect the malignant pleural effusion cytology associated with breast cancer. This study includes a total of 569 cytological slides of malignant pleural effusion of metastatic breast cancer from various institutions. We extracted 34,221 augmented image patches from whole-slide images and trained and validated a deep convolutional neural network model (DCNN) (Inception-ResNet-V2) with the images. Using this model, we classified 845 randomly selected patches, which were reviewed by three pathologists to compare their accuracy. The DCNN model outperforms the pathologists by demonstrating higher accuracy, sensitivity, and specificity compared to the pathologists (81.1% vs. 68.7%, 95.0% vs. 72.5%, and 98.6% vs. 88.9%, respectively). The pathologists reviewed the discordant cases of DCNN. After re-examination, the average accuracy, sensitivity, and specificity of the pathologists improved to 87.9, 80.2, and 95.7%, respectively. This study shows that DCNN can accurately diagnose malignant pleural effusion cytology in breast cancer and has the potential to support pathologists.
Collapse
Affiliation(s)
- Hong Sik Park
- Department of Hospital Pathology, The Catholic University of Korea College of Medicine, Seoul 06591, Republic of Korea
| | - Yosep Chong
- Department of Hospital Pathology, The Catholic University of Korea College of Medicine, Seoul 06591, Republic of Korea
| | - Yujin Lee
- Department of Hospital Pathology, The Catholic University of Korea College of Medicine, Seoul 06591, Republic of Korea
| | - Kwangil Yim
- Department of Hospital Pathology, The Catholic University of Korea College of Medicine, Seoul 06591, Republic of Korea
| | - Kyung Jin Seo
- Department of Hospital Pathology, The Catholic University of Korea College of Medicine, Seoul 06591, Republic of Korea
| | - Gisu Hwang
- AI Team, DeepNoid Inc., Seoul 08376, Republic of Korea
| | - Dahyeon Kim
- AI Team, DeepNoid Inc., Seoul 08376, Republic of Korea
| | - Gyungyub Gong
- Department of Pathology, Asan Medical Center, Seoul 05505, Republic of Korea
| | - Nam Hoon Cho
- Department of Pathology, Yonsei University College of Medicine, Seoul 03722, Republic of Korea
| | - Chong Woo Yoo
- Department of Pathology, National Cancer Center, Ilsan, Goyang-si 10408, Gyeonggi-do, Republic of Korea
| | - Hyun Joo Choi
- Department of Hospital Pathology, The Catholic University of Korea College of Medicine, Seoul 06591, Republic of Korea
| |
Collapse
|
11
|
Chatziioannou E, Roßner J, Aung TN, Rimm DL, Niessner H, Keim U, Serna-Higuita LM, Bonzheim I, Kuhn Cuellar L, Westphal D, Steininger J, Meier F, Pop OT, Forchhammer S, Flatz L, Eigentler T, Garbe C, Röcken M, Amaral T, Sinnberg T. Deep learning-based scoring of tumour-infiltrating lymphocytes is prognostic in primary melanoma and predictive to PD-1 checkpoint inhibition in melanoma metastases. EBioMedicine 2023; 93:104644. [PMID: 37295047 PMCID: PMC10363450 DOI: 10.1016/j.ebiom.2023.104644] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2023] [Revised: 05/15/2023] [Accepted: 05/24/2023] [Indexed: 06/12/2023] Open
Abstract
BACKGROUND Recent advances in digital pathology have enabled accurate and standardised enumeration of tumour-infiltrating lymphocytes (TILs). Here, we aim to evaluate TILs as a percentage electronic TIL score (eTILs) and investigate its prognostic and predictive relevance in cutaneous melanoma. METHODS We included stage I to IV cutaneous melanoma patients and used hematoxylin-eosin-stained slides for TIL analysis. We assessed eTILs as a continuous and categorical variable using the published cut-off of 16.6% and applied Cox regression models to evaluate associations of eTILs with relapse-free, distant metastasis-free, and overall survival. We compared eTILs of the primaries with matched metastasis. Moreover, we assessed the predictive relevance of eTILs in therapy-naïve metastases according to the first-line therapy. FINDINGS We analysed 321 primary cutaneous melanomas and 191 metastatic samples. In simple Cox regression, tumour thickness (p < 0.0001), presence of ulceration (p = 0.0001) and eTILs ≤16.6% (p = 0.0012) were found to be significant unfavourable prognostic factors for RFS. In multiple Cox regression, eTILs ≤16.6% (p = 0.0161) remained significant and downgraded the current staging. Lower eTILs in the primary tissue was associated with unfavourable relapse-free (p = 0.0014) and distant metastasis-free survival (p = 0.0056). In multiple Cox regression adjusted for tumour thickness and ulceration, eTILs as continuous remained significant (p = 0.019). When comparing TILs in primary tissue and corresponding metastasis of the same patient, eTILs in metastases was lower than in primary melanomas (p < 0.0001). In therapy-naïve metastases, an eTILs >12.2% was associated with longer progression-free survival (p = 0.037) and melanoma-specific survival (p = 0.0038) in patients treated with anti-PD-1-based immunotherapy. In multiple Cox regression, lactate dehydrogenase (p < 0.0001) and eTILs ≤12.2% (p = 0.0130) were significantly associated with unfavourable melanoma-specific survival. INTERPRETATION Assessment of TILs is prognostic in primary melanoma samples, and the eTILs complements staging. In therapy-naïve metastases, eTILs ≤12.2% is predictive of unfavourable survival outcomes in patients receiving anti-PD-1-based therapy. FUNDING See a detailed list of funding bodies in the Acknowledgements section at the end of the manuscript.
Collapse
Affiliation(s)
- Eftychia Chatziioannou
- Department of Dermatology, University of Tübingen, Liebermeisterstr. 25, 72076 Tübingen, Germany; Cluster of Excellence iFIT (EXC 2180) "Image-Guided and Functionally Instructed Tumor Therapies", Tübingen, Germany
| | - Jana Roßner
- Department of Dermatology, University of Heidelberg, Im Neuenheimer Feld 440, 69120 Heidelberg, Germany
| | - Thazin New Aung
- Department of Pathology, Yale University School of Medicine, New Haven, CT, USA
| | - David L Rimm
- Department of Pathology, Yale University School of Medicine, New Haven, CT, USA
| | - Heike Niessner
- Department of Dermatology, University of Tübingen, Liebermeisterstr. 25, 72076 Tübingen, Germany; Cluster of Excellence iFIT (EXC 2180) "Image-Guided and Functionally Instructed Tumor Therapies", Tübingen, Germany
| | - Ulrike Keim
- Department of Dermatology, University of Tübingen, Liebermeisterstr. 25, 72076 Tübingen, Germany
| | - Lina Maria Serna-Higuita
- Department of Clinical Epidemiology and Applied Biostatistics, Eberhard Karls University of Tübingen, 72076 Tübingen, Germany
| | - Irina Bonzheim
- Institute of Pathology and Neuropathology, Eberhard Karls University of Tübingen, 72076 Tübingen, Germany
| | - Luis Kuhn Cuellar
- Quantitative Biology Center (QBiC), University of Tübingen, Tübingen, Germany
| | - Dana Westphal
- Department of Dermatology, Faculty of Medicine and University Hospital Carl Gustav Carus, Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases, Technical University Dresden, 01307 Dresden, Germany
| | - Julian Steininger
- Department of Dermatology, Faculty of Medicine and University Hospital Carl Gustav Carus, Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases, Technical University Dresden, 01307 Dresden, Germany
| | - Friedegund Meier
- Department of Dermatology, Faculty of Medicine and University Hospital Carl Gustav Carus, Skin Cancer Center at the University Cancer Center and National Center for Tumor Diseases, Technical University Dresden, 01307 Dresden, Germany
| | - Oltin Tiberiu Pop
- Institute of Immunobiology, Kantonsspital St. Gallen, St. Gallen, Switzerland
| | - Stephan Forchhammer
- Department of Dermatology, University of Tübingen, Liebermeisterstr. 25, 72076 Tübingen, Germany
| | - Lukas Flatz
- Department of Dermatology, University of Tübingen, Liebermeisterstr. 25, 72076 Tübingen, Germany; Institute of Immunobiology, Kantonsspital St. Gallen, St. Gallen, Switzerland
| | - Thomas Eigentler
- Department of Dermatology, Venereology and Allergology, Charité-Universitätsmedizin Berlin, Charitéplatz 1, 10117 Berlin, Germany
| | - Claus Garbe
- Department of Dermatology, University of Tübingen, Liebermeisterstr. 25, 72076 Tübingen, Germany
| | - Martin Röcken
- Department of Dermatology, University of Tübingen, Liebermeisterstr. 25, 72076 Tübingen, Germany; Cluster of Excellence iFIT (EXC 2180) "Image-Guided and Functionally Instructed Tumor Therapies", Tübingen, Germany
| | - Teresa Amaral
- Department of Dermatology, University of Tübingen, Liebermeisterstr. 25, 72076 Tübingen, Germany; Cluster of Excellence iFIT (EXC 2180) "Image-Guided and Functionally Instructed Tumor Therapies", Tübingen, Germany
| | - Tobias Sinnberg
- Department of Dermatology, University of Tübingen, Liebermeisterstr. 25, 72076 Tübingen, Germany; Cluster of Excellence iFIT (EXC 2180) "Image-Guided and Functionally Instructed Tumor Therapies", Tübingen, Germany; Department of Dermatology, Venereology and Allergology, Charité-Universitätsmedizin Berlin, Charitéplatz 1, 10117 Berlin, Germany.
| |
Collapse
|
12
|
Grossarth S, Mosley D, Madden C, Ike J, Smith I, Huo Y, Wheless L. Recent Advances in Melanoma Diagnosis and Prognosis Using Machine Learning Methods. Curr Oncol Rep 2023; 25:635-645. [PMID: 37000340 PMCID: PMC10339689 DOI: 10.1007/s11912-023-01407-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/13/2023] [Indexed: 04/01/2023]
Abstract
PURPOSE OF REVIEW The purpose was to summarize the current role and state of artificial intelligence and machine learning in the diagnosis and management of melanoma. RECENT FINDINGS Deep learning algorithms can identify melanoma from clinical, dermoscopic, and whole slide pathology images with increasing accuracy. Efforts to provide more granular annotation to datasets and to identify new predictors are ongoing. There have been many incremental advances in both melanoma diagnostics and prognostic tools using artificial intelligence and machine learning. Higher quality input data will further improve these models' capabilities.
Collapse
Affiliation(s)
- Sarah Grossarth
- Quillen College of Medicine, East Tennessee State University, Johnson City, TN, USA
| | | | - Christopher Madden
- Department of Dermatology, Vanderbilt University Medicine Center, Nashville, TN, USA
- State University of New York Downstate College of Medicine, Brooklyn, NY, USA
| | - Jacqueline Ike
- Department of Dermatology, Vanderbilt University Medicine Center, Nashville, TN, USA
- Meharry Medical College, Nashville, TN, USA
| | - Isabelle Smith
- Department of Dermatology, Vanderbilt University Medicine Center, Nashville, TN, USA
- Vanderbilt University, Nashville, TN, USA
| | - Yuankai Huo
- Department of Computer Science and Electrical Engineering, Vanderbilt University, Nashville, TN, 37235, USA
| | - Lee Wheless
- Department of Dermatology, Vanderbilt University Medicine Center, Nashville, TN, USA.
- Department of Medicine, Division of Epidemiology, Vanderbilt University Medical Center, Nashville, TN, USA.
- Tennessee Valley Healthcare System VA Medical Center, Nashville, TN, USA.
| |
Collapse
|
13
|
Allaume P, Rabilloud N, Turlin B, Bardou-Jacquet E, Loréal O, Calderaro J, Khene ZE, Acosta O, De Crevoisier R, Rioux-Leclercq N, Pecot T, Kammerer-Jacquet SF. Artificial Intelligence-Based Opportunities in Liver Pathology-A Systematic Review. Diagnostics (Basel) 2023; 13:diagnostics13101799. [PMID: 37238283 DOI: 10.3390/diagnostics13101799] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Revised: 05/04/2023] [Accepted: 05/12/2023] [Indexed: 05/28/2023] Open
Abstract
BACKGROUND Artificial Intelligence (AI)-based Deep Neural Networks (DNNs) can handle a wide range of applications in image analysis, ranging from automated segmentation to diagnostic and prediction. As such, they have revolutionized healthcare, including in the liver pathology field. OBJECTIVE The present study aims to provide a systematic review of applications and performances provided by DNN algorithms in liver pathology throughout the Pubmed and Embase databases up to December 2022, for tumoral, metabolic and inflammatory fields. RESULTS 42 articles were selected and fully reviewed. Each article was evaluated through the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool, highlighting their risks of bias. CONCLUSIONS DNN-based models are well represented in the field of liver pathology, and their applications are diverse. Most studies, however, presented at least one domain with a high risk of bias according to the QUADAS-2 tool. Hence, DNN models in liver pathology present future opportunities and persistent limitations. To our knowledge, this review is the first one solely focused on DNN-based applications in liver pathology, and to evaluate their bias through the lens of the QUADAS2 tool.
Collapse
Affiliation(s)
- Pierre Allaume
- Department of Pathology CHU de Rennes, Rennes 1 University, Pontchaillou Hospital, 2 rue Henri Le Guilloux, CEDEX 09, 35033 Rennes, France
| | - Noémie Rabilloud
- Impact TEAM, Laboratoire Traitement du Signal et de l'Image (LTSI) INSERM, Rennes 1 University, Pontchaillou Hospital, 35033 Rennes, France
| | - Bruno Turlin
- Department of Pathology CHU de Rennes, Rennes 1 University, Pontchaillou Hospital, 2 rue Henri Le Guilloux, CEDEX 09, 35033 Rennes, France
- Research Unit n°UMR1341 NuMeCan-Nutrition, Métabolismes et Cancer, Rennes 1 University, Pontchaillou Hospital, 2 rue Henri Le Guilloux, CEDEX 09, 35033 Rennes, France
| | - Edouard Bardou-Jacquet
- Research Unit n°UMR1341 NuMeCan-Nutrition, Métabolismes et Cancer, Rennes 1 University, Pontchaillou Hospital, 2 rue Henri Le Guilloux, CEDEX 09, 35033 Rennes, France
- Department of Liver Diseases CHU de Rennes, Rennes 1 University, Pontchaillou Hospital, 35033 Rennes, France
| | - Olivier Loréal
- Research Unit n°UMR1341 NuMeCan-Nutrition, Métabolismes et Cancer, Rennes 1 University, Pontchaillou Hospital, 2 rue Henri Le Guilloux, CEDEX 09, 35033 Rennes, France
| | - Julien Calderaro
- Assistance Publique-Hôpitaux de Paris, Department of Pathology Henri Mondor, 94000 Créteil, France
- INSERM U955, Team Pathophysiology and Therapy of Chronic Viral Hepatitis and Related Cancers, 94000 Créteil, France
| | - Zine-Eddine Khene
- Impact TEAM, Laboratoire Traitement du Signal et de l'Image (LTSI) INSERM, Rennes 1 University, Pontchaillou Hospital, 35033 Rennes, France
- Department of Urology, CHU de Rennes, Rennes 1 University, Pontchaillou Hospital, 2 rue Henri Le Guilloux, CEDEX 09, 35033 Rennes, France
| | - Oscar Acosta
- Impact TEAM, Laboratoire Traitement du Signal et de l'Image (LTSI) INSERM, Rennes 1 University, Pontchaillou Hospital, 35033 Rennes, France
| | - Renaud De Crevoisier
- Impact TEAM, Laboratoire Traitement du Signal et de l'Image (LTSI) INSERM, Rennes 1 University, Pontchaillou Hospital, 35033 Rennes, France
- Department of Radiotherapy, Centre Eugène Marquis, 35033 Rennes, France
| | - Nathalie Rioux-Leclercq
- Department of Pathology CHU de Rennes, Rennes 1 University, Pontchaillou Hospital, 2 rue Henri Le Guilloux, CEDEX 09, 35033 Rennes, France
| | - Thierry Pecot
- Biosit Platform UAR 3480 CNRS US18 INSERM U955, Rennes 1 University, Pontchaillou Hospital, 35033 Rennes, France
| | - Solène-Florence Kammerer-Jacquet
- Department of Pathology CHU de Rennes, Rennes 1 University, Pontchaillou Hospital, 2 rue Henri Le Guilloux, CEDEX 09, 35033 Rennes, France
- Impact TEAM, Laboratoire Traitement du Signal et de l'Image (LTSI) INSERM, Rennes 1 University, Pontchaillou Hospital, 35033 Rennes, France
| |
Collapse
|
14
|
Del Amor R, Silva-Rodríguez J, Naranjo V. Labeling confidence for uncertainty-aware histology image classification. Comput Med Imaging Graph 2023; 107:102231. [PMID: 37087899 DOI: 10.1016/j.compmedimag.2023.102231] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2022] [Revised: 02/23/2023] [Accepted: 03/27/2023] [Indexed: 04/25/2023]
Abstract
Deep learning-based models applied to digital pathology require large, curated datasets with high-quality (HQ) annotations to perform correctly. In many cases, recruiting expert pathologists to annotate large databases is not feasible, and it is necessary to collect additional labeled data with varying label qualities, e.g., pathologists-in-training (henceforth, non-expert annotators). Learning from datasets with noisy labels is more challenging in medical applications since medical imaging datasets tend to have instance-dependent noise and suffer from high inter/intra-observer variability. In this paper, we design an uncertainty-driven labeling strategy with which we generate soft labels from 10 non-expert annotators for multi-class skin cancer classification. Based on this soft annotation, we propose an uncertainty estimation-based framework to handle these noisy labels. This framework is based on a novel formulation using a dual-branch min-max entropy calibration to penalize inexact labels during the training. Comprehensive experiments demonstrate the promising performance of our labeling strategy. Results show a consistent improvement by using soft labels with standard cross-entropy loss during training (∼4.0% F1-score) and increases when calibrating the model with the proposed min-max entropy calibration (∼6.6% F1-score). These improvements are produced at negligible cost, both in terms of annotation and calculation.
Collapse
Affiliation(s)
- Rocío Del Amor
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València, Valencia, Spain.
| | | | - Valery Naranjo
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València, Valencia, Spain.
| |
Collapse
|
15
|
Clarke EL, Wade RG, Magee D, Newton-Bishop J, Treanor D. Image analysis of cutaneous melanoma histology: a systematic review and meta-analysis. Sci Rep 2023; 13:4774. [PMID: 36959221 PMCID: PMC10036523 DOI: 10.1038/s41598-023-31526-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Accepted: 03/14/2023] [Indexed: 03/25/2023] Open
Abstract
The current subjective histopathological assessment of cutaneous melanoma is challenging. The application of image analysis algorithms to histological images may facilitate improvements in workflow and prognostication. To date, several individual algorithms applied to melanoma histological images have been reported with variations in approach and reported accuracies. Histological digital images can be created using a camera mounted on a light microscope, or through whole slide image (WSI) generation using a whole slide scanner. Before any such tool could be integrated into clinical workflow, the accuracy of the technology should be carefully evaluated and summarised. Therefore, the objective of this review was to evaluate the accuracy of existing image analysis algorithms applied to digital histological images of cutaneous melanoma. Database searching of PubMed and Embase from inception to 11th March 2022 was conducted alongside citation checking and examining reports from organisations. All studies reporting accuracy of any image analysis applied to histological images of cutaneous melanoma, were included. The reference standard was any histological assessment of haematoxylin and eosin-stained slides and/or immunohistochemical staining. Citations were independently deduplicated and screened by two review authors and disagreements were resolved through discussion. The data was extracted concerning study demographics; type of image analysis; type of reference standard; conditions included and test statistics to construct 2 × 2 tables. Data was extracted in accordance with our protocol and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses-Diagnostic Test Accuracy (PRISMA-DTA) Statement. A bivariate random-effects meta-analysis was used to estimate summary sensitivities and specificities with 95% confidence intervals (CI). Assessment of methodological quality was conducted using a tailored version of the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. The primary outcome was the pooled sensitivity and specificity of image analysis applied to cutaneous melanoma histological images. Sixteen studies were included in the systematic review, representing 4,888 specimens. Six studies were included in the meta-analysis. The mean sensitivity and specificity of automated image analysis algorithms applied to melanoma histological images was 90% (CI 82%, 95%) and 92% (CI 79%, 97%), respectively. Based on limited and heterogeneous data, image analysis appears to offer high accuracy when applied to histological images of cutaneous melanoma. However, given the early exploratory nature of these studies, further development work is necessary to improve their performance.
Collapse
Affiliation(s)
- Emily L Clarke
- Department of Histopathology, Leeds Teaching Hospitals NHS Trust, Leeds, UK.
- Division of Pathology and Data Analytics, Leeds Institute of Cancer and Pathology, University of Leeds, Beckett Street, Leeds, LS9 7TF, UK.
| | - Ryckie G Wade
- Leeds Institute for Medical Research, University of Leeds, Leeds, UK
| | - Derek Magee
- School of Computing, University of Leeds, Leeds, UK
| | - Julia Newton-Bishop
- Division of Pathology and Data Analytics, Leeds Institute of Cancer and Pathology, University of Leeds, Beckett Street, Leeds, LS9 7TF, UK
| | - Darren Treanor
- Department of Histopathology, Leeds Teaching Hospitals NHS Trust, Leeds, UK
- Division of Pathology and Data Analytics, Leeds Institute of Cancer and Pathology, University of Leeds, Beckett Street, Leeds, LS9 7TF, UK
- Department of Clinical Pathology, and Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden
- Center for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden
| |
Collapse
|
16
|
Deep Learning for Skin Melanocytic Tumors in Whole-Slide Images: A Systematic Review. Cancers (Basel) 2022; 15:cancers15010042. [PMID: 36612037 PMCID: PMC9817526 DOI: 10.3390/cancers15010042] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 12/05/2022] [Accepted: 12/16/2022] [Indexed: 12/24/2022] Open
Abstract
The rise of Artificial Intelligence (AI) has shown promising performance as a support tool in clinical pathology workflows. In addition to the well-known interobserver variability between dermatopathologists, melanomas present a significant challenge in their histological interpretation. This study aims to analyze all previously published studies on whole-slide images of melanocytic tumors that rely on deep learning techniques for automatic image analysis. Embase, Pubmed, Web of Science, and Virtual Health Library were used to search for relevant studies for the systematic review, in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. Articles from 2015 to July 2022 were included, with an emphasis placed on the used artificial intelligence methods. Twenty-eight studies that fulfilled the inclusion criteria were grouped into four groups based on their clinical objectives, including pathologists versus deep learning models (n = 10), diagnostic prediction (n = 7); prognosis (n = 5), and histological features (n = 6). These were then analyzed to draw conclusions on the general parameters and conditions of AI in pathology, as well as the necessary factors for better performance in real scenarios.
Collapse
|
17
|
Diagnostic and Prognostic Deep Learning Applications for Histological Assessment of Cutaneous Melanoma. Cancers (Basel) 2022; 14:cancers14246231. [PMID: 36551716 PMCID: PMC9776963 DOI: 10.3390/cancers14246231] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2022] [Revised: 12/08/2022] [Accepted: 12/14/2022] [Indexed: 12/23/2022] Open
Abstract
Melanoma is among the most devastating human malignancies. Accurate diagnosis and prognosis are essential to offer optimal treatment. Histopathology is the gold standard for establishing melanoma diagnosis and prognostic features. However, discrepancies often exist between pathologists, and analysis is costly and time-consuming. Deep-learning algorithms are deployed to improve melanoma diagnosis and prognostication from histological images of melanoma. In recent years, the development of these machine-learning tools has accelerated, and machine learning is poised to become a clinical tool to aid melanoma histology. Nevertheless, a review of the advances in machine learning in melanoma histology was lacking. We performed a comprehensive literature search to provide a complete overview of the recent advances in machine learning in the assessment of melanoma based on hematoxylin eosin digital pathology images. In our work, we review 37 recent publications, compare the methods and performance of the reviewed studies, and highlight the variety of promising machine-learning applications in melanoma histology.
Collapse
|
18
|
Jartarkar SR. Artificial intelligence: Its role in dermatopathology. Indian J Dermatol Venereol Leprol 2022:1-4. [PMID: 36688886 DOI: 10.25259/ijdvl_725_2021] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2021] [Accepted: 08/01/2022] [Indexed: 12/15/2022]
Abstract
Artificial intelligence (AI), a major frontier in the field of medical research, can potentially lead to a paradigm shift in clinical practice. A type of artificial intelligence system known as convolutional neural network points to the possible utility of deep learning in dermatopathology. Though pathology has been traditionally restricted to microscopes and glass slides, recent advancement in digital pathological imaging has led to a transition making it a potential branch for the implementation of artificial intelligence. The current application of artificial intelligence in dermatopathology is to complement the diagnosis and requires a well-trained dermatopathologist's guidance for better designing and development of deep learning algorithms. Here we review the recent advances of artificial intelligence in dermatopathology, its applications in disease diagnosis and in research, along with its limitations and future potential.
Collapse
Affiliation(s)
- Shishira R Jartarkar
- Department of Dermatology, Venereology and Leprosy, Vydehi Institute of Medical Sciences and Research Centre, Whitefield, Bengaluru, Karnataka, India
| |
Collapse
|
19
|
Requa J, Godard T, Mandal R, Balzer B, Whittemore D, George E, Barcelona F, Lambert C, Lee J, Lambert A, Larson A, Osmond G. High-fidelity detection, subtyping, and localization of five skin neoplasms using supervised and semi-supervised learning. J Pathol Inform 2022; 14:100159. [PMID: 36506813 PMCID: PMC9731861 DOI: 10.1016/j.jpi.2022.100159] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2022] [Revised: 11/16/2022] [Accepted: 11/22/2022] [Indexed: 11/27/2022] Open
Abstract
Background Skin cancers are the most common malignancies diagnosed worldwide. While the early detection and treatment of pre-cancerous and cancerous skin lesions can dramatically improve outcomes, factors such as a global shortage of pathologists, increased workloads, and high rates of diagnostic discordance underscore the need for techniques that improve pathology workflows. Although AI models are now being used to classify lesions from whole slide images (WSIs), diagnostic performance rarely surpasses that of expert pathologists. Objectives The objective of the present study was to create an AI model to detect and classify skin lesions with a higher degree of sensitivity than previously demonstrated, with potential to match and eventually surpass expert pathologists to improve clinical workflows. Methods We combined supervised learning (SL) with semi-supervised learning (SSL) to produce an end-to-end multi-level skin detection system that not only detects 5 main types of skin lesions with high sensitivity and specificity, but also subtypes, localizes, and provides margin status to evaluate the proximity of the lesion to non-epidermal margins. The Supervised Training Subset consisted of 2188 random WSIs collected by the PathologyWatch (PW) laboratory between 2013 and 2018, while the Weakly Supervised Subset consisted of 5161 WSIs from daily case specimens. The Validation Set consisted of 250 curated daily case WSIs obtained from the PW tissue archives and included 50 "mimickers". The Testing Set (3821 WSIs) was composed of non-curated daily case specimens collected from July 20, 2021 to August 20, 2021 from PW laboratories. Results The performance characteristics of our AI model (i.e., Mihm) were assessed retrospectively by running the Testing Set through the Mihm Evaluation Pipeline. Our results show that the sensitivity of Mihm in classifying melanocytic lesions, basal cell carcinoma, and atypical squamous lesions, verruca vulgaris, and seborrheic keratosis was 98.91% (95% CI: 98.27%, 99.55%), 97.24% (95% CI: 96.15%, 98.33%), 95.26% (95% CI: 93.79%, 96.73%), 93.50% (95% CI: 89.14%, 97.86%), and 86.91% (95% CI: 82.13%, 91.69%), respectively. Additionally, our multi-level (i.e., patch-level, ROI-level, and WSI-level) detection algorithm includes a qualitative feature that subtypes lesions, an AI overlay in the front-end digital display that localizes diagnostic ROIs, and reports on margin status by detecting overlap between lesions and non-epidermal tissue margins. Conclusions Our AI model, developed in collaboration with dermatopathologists, detects 5 skin lesion types with higher sensitivity than previously published AI models, and provides end users with information such as subtyping, localization, and margin status in a front-end digital display. Our end-to-end system has the potential to improve pathology workflows by increasing diagnostic accuracy, expediting the course of patient care, and ultimately improving patient outcomes.
Collapse
Affiliation(s)
- James Requa
- Pathology Watch, 497 West 4800 South, Suite 201, Murray, UT 84123, USA
| | - Tuatini Godard
- Pathology Watch, 497 West 4800 South, Suite 201, Murray, UT 84123, USA
| | - Rajni Mandal
- Pathology Watch, 497 West 4800 South, Suite 201, Murray, UT 84123, USA
| | - Bonnie Balzer
- Cedars-Sinai Medical Center, 8700 Beverly Blvd, Los Angeles, CA 90048, USA
| | - Darren Whittemore
- Pathology Watch, 497 West 4800 South, Suite 201, Murray, UT 84123, USA
| | - Eva George
- Pathology Watch, 497 West 4800 South, Suite 201, Murray, UT 84123, USA
| | | | - Chalette Lambert
- Kirk Kerkorian School of Medicine at UNLV, University of Nevada, Las Vegas, Mail Stop: 3070, 2040 W Charleston Blvd., Las Vegas, NV 89102-2244, USA
| | - Jonathan Lee
- Bethesda Dermatopathology Laboratory, 1730 Elton Road, Silver Spring, MD 20903, USA
| | - Allison Lambert
- Pathology Watch, 497 West 4800 South, Suite 201, Murray, UT 84123, USA
| | - April Larson
- Pathology Watch, 497 West 4800 South, Suite 201, Murray, UT 84123, USA
| | - Gregory Osmond
- Intermountain Healthcare, Saint George Regional Hospital, Department of Pathology, 1380 East Medical Center Drive, Saint George, Utah 84790, USA,Corresponding author.
| |
Collapse
|
20
|
Kriegsmann K, Lobers F, Zgorzelski C, Kriegsmann J, Janßen C, Meliß RR, Muley T, Sack U, Steinbuss G, Kriegsmann M. Deep learning for the detection of anatomical tissue structures and neoplasms of the skin on scanned histopathological tissue sections. Front Oncol 2022; 12:1022967. [PMID: 36483044 PMCID: PMC9723465 DOI: 10.3389/fonc.2022.1022967] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2022] [Accepted: 10/24/2022] [Indexed: 01/25/2023] Open
Abstract
Basal cell carcinoma (BCC), squamous cell carcinoma (SqCC) and melanoma are among the most common cancer types. Correct diagnosis based on histological evaluation after biopsy or excision is paramount for adequate therapy stratification. Deep learning on histological slides has been suggested to complement and improve routine diagnostics, but publicly available curated and annotated data and usable models trained to distinguish common skin tumors are rare and often lack heterogeneous non-tumor categories. A total of 16 classes from 386 cases were manually annotated on scanned histological slides, 129,364 100 x 100 µm (~395 x 395 px) image tiles were extracted and split into a training, validation and test set. An EfficientV2 neuronal network was trained and optimized to classify image categories. Cross entropy loss, balanced accuracy and Matthews correlation coefficient were used for model evaluation. Image and patient data were assessed with confusion matrices. Application of the model to an external set of whole slides facilitated localization of melanoma and non-tumor tissue. Automated differentiation of BCC, SqCC, melanoma, naevi and non-tumor tissue structures was possible, and a high diagnostic accuracy was achieved in the validation (98%) and test (97%) set. In summary, we provide a curated dataset including the most common neoplasms of the skin and various anatomical compartments to enable researchers to train, validate and improve deep learning models. Automated classification of skin tumors by deep learning techniques is possible with high accuracy, facilitates tumor localization and has the potential to support and improve routine diagnostics.
Collapse
Affiliation(s)
- Katharina Kriegsmann
- Department of Hematology, Oncology and Rheumatology, Heidelberg University, Heidelberg, Germany
| | - Frithjof Lobers
- Department of Clinical Immunology, Medical Faculty, University of Leipzig, Leipzig, Germany
| | | | - Jörg Kriegsmann
- MVZ Histology, Cytology and Molecular Diagnostics Trier, Trier, Germany,Proteopath Trier, Trier, Germany
| | - Charlotte Janßen
- Center for Industrial Mathematics (ZeTeM), University of Bremen, Bremen, Germany
| | | | - Thomas Muley
- Translational Lung Research Centre (TLRC) Heidelberg, Member of the German Centre for Lung Research (DZL), Heidelberg, Germany
| | - Ulrich Sack
- Department of Clinical Immunology, Medical Faculty, University of Leipzig, Leipzig, Germany
| | - Georg Steinbuss
- Department of Hematology, Oncology and Rheumatology, Heidelberg University, Heidelberg, Germany
| | - Mark Kriegsmann
- Institute of Pathology, Heidelberg University, Heidelberg, Germany,*Correspondence: Mark Kriegsmann,
| |
Collapse
|
21
|
Histologic Screening of Malignant Melanoma, Spitz, Dermal and Junctional Melanocytic Nevi Using a Deep Learning Model. Am J Dermatopathol 2022; 44:650-657. [PMID: 35925282 DOI: 10.1097/dad.0000000000002232] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
OBJECTIVE The integration of an artificial intelligence tool into pathologists' workflow may lead to a more accurate and timely diagnosis of melanocytic lesions, directly patient care. The objective of this study was to create and evaluate the performance of such a model in achieving clinical-grade diagnoses of Spitz nevi, dermal and junctional melanocytic nevi, and melanomas. METHODS We created a beginner-level training environment by teaching our algorithm to perform cytologic inferences on 136,216 manually annotated tiles of hematoxylin and eosin-stained slides consisting of unequivocal melanocytic nevi, Spitz nevi, and invasive melanoma cases. We sequentially trained and tested our network to provide a final diagnosis-classification on 39 cases in total. Positive predictive value (precision) and sensitivity (recall) were used to measure our performance. RESULTS The tile-classification algorithm predicted the 136,216 irrelevant, melanoma, melanocytic nevi, and Spitz nevi tiles at sensitivities of 96%, 93%, 94% and 73%, respectively. The final trained model was able to correctly classify and predict the correct diagnosis in 85.7% of unseen cases (n = 28), reporting at or near screening-level performances for precision and recall of melanoma (76.2%, 100.0%), melanocytic nevi (100.0%, 75.0%), and Spitz nevi (100.0%, 75.0%). CONCLUSIONS Our pilot study proves that convolutional networks trained on cellular morphology to classify melanocytic proliferations can be used as a powerful tool to assist pathologists in screening for melanoma versus other benign lesions.
Collapse
|
22
|
Deep Learning Assisted Diagnosis of Onychomycosis on Whole-Slide Images. J Fungi (Basel) 2022; 8:jof8090912. [PMID: 36135637 PMCID: PMC9504700 DOI: 10.3390/jof8090912] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2022] [Revised: 08/18/2022] [Accepted: 08/25/2022] [Indexed: 11/20/2022] Open
Abstract
Background: Onychomycosis numbers among the most common fungal infections in humans affecting finger- or toenails. Histology remains a frequently applied screening technique to diagnose onychomycosis. Screening slides for fungal elements can be time-consuming for pathologists, and sensitivity in cases with low amounts of fungi remains a concern. Convolutional neural networks (CNNs) have revolutionized image classification in recent years. The goal of our project was to evaluate if a U-NET-based segmentation approach as a subcategory of CNNs can be applied to detect fungal elements on digitized histologic sections of human nail specimens and to compare it with the performance of 11 board-certified dermatopathologists. Methods: In total, 664 corresponding H&E- and PAS-stained histologic whole-slide images (WSIs) of human nail plates from four different laboratories were digitized. Histologic structures were manually annotated. A U-NET image segmentation model was trained for binary segmentation on the dataset generated by annotated slides. Results: The U-NET algorithm detected 90.5% of WSIs with fungi, demonstrating a comparable sensitivity with that of the 11 board-certified dermatopathologists (sensitivity of 89.2%). Conclusions: Our results demonstrate that machine-learning-based algorithms applied to real-world clinical cases can produce comparable sensitivities to human pathologists. Our established U-NET may be used as a supportive diagnostic tool to preselect possible slides with fungal elements. Slides where fungal elements are indicated by our U-NET should be reevaluated by the pathologist to confirm or refute the diagnosis of onychomycosis.
Collapse
|
23
|
Advantages of manual and automatic computer-aided compared to traditional histopathological diagnosis of melanoma: A pilot study. Pathol Res Pract 2022; 237:154014. [PMID: 35870238 DOI: 10.1016/j.prp.2022.154014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/04/2022] [Revised: 07/04/2022] [Accepted: 07/07/2022] [Indexed: 11/20/2022]
Abstract
BACKGROUND Cutaneous malignant melanoma (CMM) accounts for the highest mortality rate among all skin cancers. Traditional histopathologic diagnosis may be limited by the pathologists' subjectivity. Second-opinion strategies and multidisciplinary consultations are usually performed to overcome this issue. An available solution in the future could be the use of automated solutions based on a computational algorithm that could help the pathologist in everyday practice. The aim of this pilot study was to investigate the potential diagnostic aid of a machine-based algorithm in the histopathologic diagnosis of CMM. METHODS We retrospectively examined excisional biopsies of 50 CMM and 20 benign congenital compound nevi. Hematoxylin and eosin (H&E) stained WSI were reviewed independently by two expert dermatopathologists. A fully automated pipeline for WSI processing to support the estimation and prioritization of the melanoma areas was developed. RESULTS The spatial distribution of the nuclei in the sample provided a multi-scale overview of the tumor. A global overview of the lesion's silhouette was achieved and, by increasing the magnification, the topological distribution of the nuclei and the most informative areas of interest for the CMM diagnosis were identified and highlighted. These silhouettes allow the histopathologist to discriminate between nevus and CMM with an accuracy of 96% without any extra information. CONCLUSION In this study we proposed an easy-to-use model that produces segmentations of CMM silhouettes at fine detail level.
Collapse
|
24
|
Malciu AM, Lupu M, Voiculescu VM. Artificial Intelligence-Based Approaches to Reflectance Confocal Microscopy Image Analysis in Dermatology. J Clin Med 2022; 11:jcm11020429. [PMID: 35054123 PMCID: PMC8780225 DOI: 10.3390/jcm11020429] [Citation(s) in RCA: 15] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2021] [Revised: 01/11/2022] [Accepted: 01/12/2022] [Indexed: 12/22/2022] Open
Abstract
Reflectance confocal microscopy (RCM) is a non-invasive imaging method designed to identify various skin diseases. Confocal based diagnosis may be subjective due to the learning curve of the method, the scarcity of training programs available for RCM, and the lack of clearly defined diagnostic criteria for all skin conditions. Given that in vivo RCM is becoming more widely used in dermatology, numerous deep learning technologies have been developed in recent years to provide a more objective approach to RCM image analysis. Machine learning-based algorithms are used in RCM image quality assessment to reduce the number of artifacts the operator has to view, shorten evaluation times, and decrease the number of patient visits to the clinic. However, the current visual method for identifying the dermal-epidermal junction (DEJ) in RCM images is subjective, and there is a lot of variation. The delineation of DEJ on RCM images could be automated through artificial intelligence, saving time and assisting novice RCM users in studying the key DEJ morphological structure. The purpose of this paper is to supply a current summary of machine learning and artificial intelligence’s impact on the quality control of RCM images, key morphological structures identification, and detection of different skin lesion types on static RCM images.
Collapse
Affiliation(s)
- Ana Maria Malciu
- Department of Dermatology, Elias University Emergency Hospital, 011461 Bucharest, Romania;
| | - Mihai Lupu
- Department of Dermatology, Carol Davila University of Medicine and Pharmacy, 050474 Bucharest, Romania
- Correspondence: (M.L.); (V.M.V.)
| | - Vlad Mihai Voiculescu
- Department of Dermatology, Elias University Emergency Hospital, 011461 Bucharest, Romania;
- Department of Dermatology, Carol Davila University of Medicine and Pharmacy, 050474 Bucharest, Romania
- Correspondence: (M.L.); (V.M.V.)
| |
Collapse
|
25
|
Del Amor R, Launet L, Colomer A, Moscardó A, Mosquera-Zamudio A, Monteagudo C, Naranjo V. An attention-based weakly supervised framework for spitzoid melanocytic lesion diagnosis in whole slide images. Artif Intell Med 2021; 121:102197. [PMID: 34763799 DOI: 10.1016/j.artmed.2021.102197] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2021] [Revised: 10/08/2021] [Accepted: 10/12/2021] [Indexed: 02/07/2023]
Abstract
Melanoma is an aggressive neoplasm responsible for the majority of deaths from skin cancer. Specifically, spitzoid melanocytic tumors are one of the most challenging melanocytic lesions due to their ambiguous morphological features. The gold standard for its diagnosis and prognosis is the analysis of skin biopsies. In this process, dermatopathologists visualize skin histology slides under a microscope, in a highly time-consuming and subjective task. In the last years, computer-aided diagnosis (CAD) systems have emerged as a promising tool that could support pathologists in daily clinical practice. Nevertheless, no automatic CAD systems have yet been proposed for the analysis of spitzoid lesions. Regarding common melanoma, no system allows both the selection of the tumor region and the prediction of the benign or malignant form in the diagnosis. Motivated by this, we propose a novel end-to-end weakly supervised deep learning model, based on inductive transfer learning with an improved convolutional neural network (CNN) to refine the embedding features of the latent space. The framework is composed of a source model in charge of finding the tumor patch-level patterns, and a target model focuses on the specific diagnosis of a biopsy. The latter retrains the backbone of the source model through a multiple instance learning workflow to obtain the biopsy-level scoring. To evaluate the performance of the proposed methods, we performed extensive experiments on a private skin database with spitzoid lesions. Test results achieved an accuracy of 0.9231 and 0.80 for the source and the target models, respectively. In addition, the heat map findings are directly in line with the clinicians' medical decision and even highlight, in some cases, patterns of interest that were overlooked by the pathologist.
Collapse
Affiliation(s)
- Rocío Del Amor
- Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, 46022 Valencia, Spain.
| | - Laëtitia Launet
- Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, 46022 Valencia, Spain
| | - Adrián Colomer
- Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, 46022 Valencia, Spain
| | - Anaïs Moscardó
- Pathology Department, Hospital Clínico Universitario de Valencia, Universidad de Valencia, Valencia, Spain
| | - Andrés Mosquera-Zamudio
- Pathology Department, Hospital Clínico Universitario de Valencia, Universidad de Valencia, Valencia, Spain
| | - Carlos Monteagudo
- Pathology Department, Hospital Clínico Universitario de Valencia, Universidad de Valencia, Valencia, Spain
| | - Valery Naranjo
- Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, 46022 Valencia, Spain
| |
Collapse
|
26
|
Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts. Eur J Cancer 2021; 156:202-216. [PMID: 34509059 DOI: 10.1016/j.ejca.2021.06.049] [Citation(s) in RCA: 79] [Impact Index Per Article: 26.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2021] [Revised: 06/18/2021] [Accepted: 06/28/2021] [Indexed: 12/23/2022]
Abstract
BACKGROUND Multiple studies have compared the performance of artificial intelligence (AI)-based models for automated skin cancer classification to human experts, thus setting the cornerstone for a successful translation of AI-based tools into clinicopathological practice. OBJECTIVE The objective of the study was to systematically analyse the current state of research on reader studies involving melanoma and to assess their potential clinical relevance by evaluating three main aspects: test set characteristics (holdout/out-of-distribution data set, composition), test setting (experimental/clinical, inclusion of metadata) and representativeness of participating clinicians. METHODS PubMed, Medline and ScienceDirect were screened for peer-reviewed studies published between 2017 and 2021 and dealing with AI-based skin cancer classification involving melanoma. The search terms skin cancer classification, deep learning, convolutional neural network (CNN), melanoma (detection), digital biomarkers, histopathology and whole slide imaging were combined. Based on the search results, only studies that considered direct comparison of AI results with clinicians and had a diagnostic classification as their main objective were included. RESULTS A total of 19 reader studies fulfilled the inclusion criteria. Of these, 11 CNN-based approaches addressed the classification of dermoscopic images; 6 concentrated on the classification of clinical images, whereas 2 dermatopathological studies utilised digitised histopathological whole slide images. CONCLUSIONS All 19 included studies demonstrated superior or at least equivalent performance of CNN-based classifiers compared with clinicians. However, almost all studies were conducted in highly artificial settings based exclusively on single images of the suspicious lesions. Moreover, test sets mainly consisted of holdout images and did not represent the full range of patient populations and melanoma subtypes encountered in clinical practice.
Collapse
|
27
|
Fu T, Dai LJ, Wu SY, Xiao Y, Ma D, Jiang YZ, Shao ZM. Spatial architecture of the immune microenvironment orchestrates tumor immunity and therapeutic response. J Hematol Oncol 2021; 14:98. [PMID: 34172088 PMCID: PMC8234625 DOI: 10.1186/s13045-021-01103-4] [Citation(s) in RCA: 172] [Impact Index Per Article: 57.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2021] [Accepted: 06/03/2021] [Indexed: 02/08/2023] Open
Abstract
Tumors are not only aggregates of malignant cells but also well-organized complex ecosystems. The immunological components within tumors, termed the tumor immune microenvironment (TIME), have long been shown to be strongly related to tumor development, recurrence and metastasis. However, conventional studies that underestimate the potential value of the spatial architecture of the TIME are unable to completely elucidate its complexity. As innovative high-flux and high-dimensional technologies emerge, researchers can more feasibly and accurately detect and depict the spatial architecture of the TIME. These findings have improved our understanding of the complexity and role of the TIME in tumor biology. In this review, we first epitomized some representative emerging technologies in the study of the spatial architecture of the TIME and categorized the description methods used to characterize these structures. Then, we determined the functions of the spatial architecture of the TIME in tumor biology and the effects of the gradient of extracellular nonspecific chemicals (ENSCs) on the TIME. We also discussed the potential clinical value of our understanding of the spatial architectures of the TIME, as well as current limitations and future prospects in this novel field. This review will bring spatial architectures of the TIME, an emerging dimension of tumor ecosystem research, to the attention of more researchers and promote its application in tumor research and clinical practice.
Collapse
Affiliation(s)
- Tong Fu
- Department of Breast Surgery, Fudan University Shanghai Cancer Center, Shanghai, 200032, China
- Key Laboratory of Breast Cancer in Shanghai, Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China
| | - Lei-Jie Dai
- Department of Breast Surgery, Fudan University Shanghai Cancer Center, Shanghai, 200032, China
- Key Laboratory of Breast Cancer in Shanghai, Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China
| | - Song-Yang Wu
- Department of Breast Surgery, Fudan University Shanghai Cancer Center, Shanghai, 200032, China
- Key Laboratory of Breast Cancer in Shanghai, Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China
| | - Yi Xiao
- Department of Breast Surgery, Fudan University Shanghai Cancer Center, Shanghai, 200032, China
- Key Laboratory of Breast Cancer in Shanghai, Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China
| | - Ding Ma
- Department of Breast Surgery, Fudan University Shanghai Cancer Center, Shanghai, 200032, China.
- Key Laboratory of Breast Cancer in Shanghai, Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China.
| | - Yi-Zhou Jiang
- Department of Breast Surgery, Fudan University Shanghai Cancer Center, Shanghai, 200032, China.
- Key Laboratory of Breast Cancer in Shanghai, Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China.
| | - Zhi-Ming Shao
- Department of Breast Surgery, Fudan University Shanghai Cancer Center, Shanghai, 200032, China.
- Key Laboratory of Breast Cancer in Shanghai, Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China.
| |
Collapse
|
28
|
Höhn J, Krieghoff-Henning E, Jutzi TB, von Kalle C, Utikal JS, Meier F, Gellrich FF, Hobelsberger S, Hauschild A, Schlager JG, French L, Heinzerling L, Schlaak M, Ghoreschi K, Hilke FJ, Poch G, Kutzner H, Heppt MV, Haferkamp S, Sondermann W, Schadendorf D, Schilling B, Goebeler M, Hekler A, Fröhling S, Lipka DB, Kather JN, Krahl D, Ferrara G, Haggenmüller S, Brinker TJ. Combining CNN-based histologic whole slide image analysis and patient data to improve skin cancer classification. Eur J Cancer 2021; 149:94-101. [PMID: 33838393 DOI: 10.1016/j.ejca.2021.02.032] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2021] [Revised: 02/12/2021] [Accepted: 02/13/2021] [Indexed: 02/07/2023]
Abstract
BACKGROUND Clinicians and pathologists traditionally use patient data in addition to clinical examination to support their diagnoses. OBJECTIVES We investigated whether a combination of histologic whole slides image (WSI) analysis based on convolutional neural networks (CNNs) and commonly available patient data (age, sex and anatomical site of the lesion) in a binary melanoma/nevus classification task could increase the performance compared with CNNs alone. METHODS We used 431 WSIs from two different laboratories and analysed the performance of classifiers that used the image or patient data individually or three common fusion techniques. Furthermore, we tested a naive combination of patient data and an image classifier: for cases interpreted as 'uncertain' (CNN output score <0.7), the decision of the CNN was replaced by the decision of the patient data classifier. RESULTS The CNN on its own achieved the best performance (mean ± standard deviation of five individual runs) with AUROC of 92.30% ± 0.23% and balanced accuracy of 83.17% ± 0.38%. While the classification performance was not significantly improved in general by any of the tested fusions, naive strategy of replacing the image classifier with the patient data classifier on slides with low output scores improved balanced accuracy to 86.72% ± 0.36%. CONCLUSION In most cases, the CNN on its own was so accurate that patient data integration did not provide any benefit. However, incorporating patient data for lesions that were classified by the CNN with low 'confidence' improved balanced accuracy.
Collapse
Affiliation(s)
- Julia Höhn
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Tanja B Jutzi
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Charité University Medicine and Berlin Institute of Health (BIH), Berlin, Germany
| | - Jochen S Utikal
- Department of Dermatology, Heidelberg University, Mannheim, Germany; Skin Cancer Unit, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Friedegund Meier
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität, Dresden, Germany
| | - Frank F Gellrich
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität, Dresden, Germany
| | - Sarah Hobelsberger
- Skin Cancer Center at the University Cancer Centre and National Center for Tumor Diseases Dresden, Department of Dermatology, University Hospital Carl Gustav Carus, Technische Universität, Dresden, Germany
| | - Axel Hauschild
- Department of Dermatology, University Hospital of Kiel, Germany
| | - Justin G Schlager
- Department of Dermatology and Allergology, Ludwig Maximilian University of Munich, Munich, Germany
| | - Lars French
- Department of Dermatology and Allergology, Ludwig Maximilian University of Munich, Munich, Germany
| | - Lucie Heinzerling
- Department of Dermatology and Allergology, Ludwig Maximilian University of Munich, Munich, Germany
| | - Max Schlaak
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Kamran Ghoreschi
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Franz J Hilke
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Gabriela Poch
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Heinz Kutzner
- Dermatopathology Laboratory, Friedrichshafen, Germany
| | - Markus V Heppt
- Department of Dermatology, University Hospital Erlangen, Erlangen, Germany
| | - Sebastian Haferkamp
- Department of Dermatology, University Hospital Regensburg, Regensburg, Germany
| | - Wiebke Sondermann
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, Essen, Germany
| | - Bastian Schilling
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Matthias Goebeler
- Department of Dermatology, University Hospital Würzburg, Würzburg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Stefan Fröhling
- Section Translational Cancer Epigenomics, Division Translational Medical Oncology, German Cancer Research Center (DKFZ) & National Center for Tumor Diseases (NCT), Heidelberg, 69120, Germany
| | - Daniel B Lipka
- Section Translational Cancer Epigenomics, Division Translational Medical Oncology, German Cancer Research Center (DKFZ) & National Center for Tumor Diseases (NCT), Heidelberg, 69120, Germany
| | - Jakob N Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Dieter Krahl
- Private Laboratory of Dermatohistopathology, Mönchhofstraße 52, Heidelberg, 69120, Germany
| | - Gerardo Ferrara
- Anatomic Pathology Unit, Macerata General Hospital, Macerata, Italy
| | - Sarah Haggenmüller
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, National Center for Tumor Diseases, German Cancer Research Center, Heidelberg, Germany.
| |
Collapse
|