Recognition of Cutaneous Melanoma on Digitized Histopathological Slides via Artificial Intelligence Algorithm

Robust ROI Detection in Whole Slide Images Guided by Pathologists' Viewing Patterns

Deep learning techniques offer improvements in computer-aided diagnosis systems. However, acquiring image domain annotations is challenging due to the knowledge and commitment required of expert pathologists. Pathologists often identify regions in whole slide images with diagnostic relevance rather than examining the entire slide, with a positive correlation between the time spent on these critical image regions and diagnostic accuracy. In this paper, a heatmap is generated to represent pathologists' viewing patterns during diagnosis and used to guide a deep learning architecture during training. The proposed system outperforms traditional approaches based on color and texture image characteristics, integrating pathologists' domain expertise to enhance region of interest detection without needing individual case annotations. Evaluating our best model, a U-Net model with a pre-trained ResNet-18 encoder, on a skin biopsy whole slide image dataset for melanoma diagnosis, shows its potential in detecting regions of interest, surpassing conventional methods with an increase of 20%, 11%, 22%, and 12% in precision, recall, F1-score, and Intersection over Union, respectively. In a clinical evaluation, three dermatopathologists agreed on the model's effectiveness in replicating pathologists' diagnostic viewing behavior and accurately identifying critical regions. Finally, our study demonstrates that incorporating heatmaps as supplementary signals can enhance the performance of computer-aided diagnosis systems. Without the availability of eye tracking data, identifying precise focus areas is challenging, but our approach shows promise in assisting pathologists in improving diagnostic accuracy and efficiency, streamlining annotation processes, and aiding the training of new pathologists.

Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy

Ensuring diagnostic performance of artificial intelligence (AI) before introduction into clinical practice is essential. Growing numbers of studies using AI for digital pathology have been reported over recent years. The aim of this work is to examine the diagnostic accuracy of AI in digital pathology images for any disease. This systematic review and meta-analysis included diagnostic accuracy studies using any type of AI applied to whole slide images (WSIs) for any disease. The reference standard was diagnosis by histopathological assessment and/or immunohistochemistry. Searches were conducted in PubMed, EMBASE and CENTRAL in June 2022. Risk of bias and concerns of applicability were assessed using the QUADAS-2 tool. Data extraction was conducted by two investigators and meta-analysis was performed using a bivariate random effects model, with additional subgroup analyses also performed. Of 2976 identified studies, 100 were included in the review and 48 in the meta-analysis. Studies were from a range of countries, including over 152,000 whole slide images (WSIs), representing many diseases. These studies reported a mean sensitivity of 96.3% (CI 94.1-97.7) and mean specificity of 93.3% (CI 90.5-95.4). There was heterogeneity in study design and 99% of studies identified for inclusion had at least one area at high or unclear risk of bias or applicability concerns. Details on selection of cases, division of model development and validation data and raw performance data were frequently ambiguous or missing. AI is reported as having high diagnostic accuracy in the reported areas but requires more rigorous evaluation of its performance.

The role of artificial intelligence and convolutional neural networks in the management of melanoma: a clinical, pathological, and radiological perspective

Clinical dermatoscopy and pathological slide assessment are essential in the diagnosis and management of patients with cutaneous melanoma. For those presenting with stage IIC disease and beyond, radiological investigations are often considered. The dermatoscopic, whole slide and radiological images used during clinical care are often stored digitally, enabling artificial intelligence (AI) and convolutional neural networks (CNN) to learn, analyse and contribute to the clinical decision-making. A keyword search of the Medline database was performed to assess the progression, capabilities and limitations of AI and CNN and its use in diagnosis and management of cutaneous melanoma. Full-text articles were reviewed if they related to dermatoscopy, pathological slide assessment or radiology. Through analysis of 95 studies, we demonstrate that diagnostic accuracy of AI/CNN can be superior (or at least equal) to clinicians. However, variability in image acquisition, pre-processing, segmentation, and feature extraction remains challenging. With current technological abilities, AI/CNN and clinicians synergistically working together are better than one another in all subspecialty domains relating to cutaneous melanoma. AI has the potential to enhance the diagnostic capabilities of junior dermatology trainees, primary care skin cancer clinicians and general practitioners. For experienced clinicians, AI provides a cost-efficient second opinion. From a pathological and radiological perspective, CNN has the potential to improve workflow efficiency, allowing clinicians to achieve more in a finite amount of time. Until the challenges of AI/CNN are reliably met, however, they can only remain an adjunct to clinical decision-making.

Artificial Intelligence in Melanoma Dermatopathology: A Review of Literature

ABSTRACT

Pathology serves as a promising field to integrate artificial intelligence into clinical practice as a powerful screening tool. Melanoma is a common skin cancer with high mortality and morbidity, requiring timely and accurate histopathologic diagnosis. This review explores applications of artificial intelligence in melanoma dermatopathology, including differential diagnostics, prognosis prediction, and personalized medicine decision-making.

Recent advances of artificial intelligence in melanoma clinical practice

Skin melanoma is a lethal cancer. The incidence of melanoma is increasing rapidly in all regions of the world. Despite significant breakthroughs in melanoma treatment in recent years, precise diagnosis of melanoma is still a challenge in some cases. Even specialized physicians may need time and effort to make accurate judgments. As artificial intelligence (AI) technology advances into medical practice, it may bring new solutions to this problem based on its efficiency, accuracy, and speed. This paper summarizes the recent progress of AI in melanoma-related applications, including melanoma diagnosis and classification, the discovery of new medication, guiding treatment, and prognostic assessment. The paper also compares the effectiveness of various algorithms in melanoma application and suggests future research directions for AI in melanoma clinical practice.

Tumor-Infiltrating Lymphocyte Recognition in Primary Melanoma by Deep Learning Convolutional Neural Network

The presence of tumor-infiltrating lymphocytes (TILs) is associated with a favorable prognosis of primary melanoma (PM). Recently, artificial intelligence (AI)-based approach in digital pathology was proposed for the standardized assessment of TILs on hematoxylin and eosin-stained whole slide images (WSIs). Herein, the study applied a new convolution neural network (CNN) analysis of PM WSIs to automatically assess the infiltration of TILs and extract a TIL score. A CNN was trained and validated in a retrospective cohort of 307 PMs including a training set (237 WSIs, 57,758 patches) and an independent testing set (70 WSIs, 29,533 patches). An AI-based TIL density index (AI-TIL) was identified after the classification of tumor patches by the presence or absence of TILs. The proposed CNN showed high performance in recognizing TILs in PM WSIs, showing 100% specificity and sensitivity on the testing set. The AI-based TIL index correlated with conventional TIL evaluation and clinical outcome. The AI-TIL index was an independent prognostic marker associated directly with a favorable prognosis. A fully automated and standardized AI-TIL appeared to be superior to conventional methods at differentiating the PM clinical outcome. Further studies are required to develop an easy-to-use tool to assist pathologists to assess TILs in the clinical evaluation of solid tumors.

Artificial Intelligence Algorithms for Benign vs. Malignant Dermoscopic Skin Lesion Image Classification

In recent decades, the incidence of melanoma has grown rapidly. Hence, early diagnosis is crucial to improving clinical outcomes. Here, we propose and compare a classical image analysis-based machine learning method with a deep learning one to automatically classify benign vs. malignant dermoscopic skin lesion images. The same dataset of 25,122 publicly available dermoscopic images was used to train both models, while a disjointed test set of 200 images was used for the evaluation phase. The training dataset was randomly divided into 10 datasets of 19,932 images to obtain an equal distribution between the two classes. By testing both models on the disjoint set, the deep learning-based method returned accuracy of 85.4 ± 3.2% and specificity of 75.5 ± 7.6%, while the machine learning one showed accuracy and specificity of 73.8 ± 1.1% and 44.5 ± 4.7%, respectively. Although both approaches performed well in the validation phase, the convolutional neural network outperformed the ensemble boosted tree classifier on the disjoint test set, showing better generalization ability. The integration of new melanoma detection algorithms with digital dermoscopic devices could enable a faster screening of the population, improve patient management, and achieve better survival rates.

Annotation protocol and crowdsourcing multiple instance learning classification of skin histological images: The CR-AI4SkIN dataset

Digital Pathology (DP) has experienced a significant growth in recent years and has become an essential tool for diagnosing and prognosis of tumors. The availability of Whole Slide Images (WSIs) and the implementation of Deep Learning (DL) algorithms have paved the way for the appearance of Artificial Intelligence (AI) systems that support the diagnosis process. These systems require extensive and varied data for their training to be successful. However, creating labeled datasets in histopathology is laborious and time-consuming. We have developed a crowdsourcing-multiple instance labeling/learning protocol that is applied to the creation and use of the CR-AI4SkIN dataset.2 CR-AI4SkIN contains 271 WSIs of 7 Cutaneous Spindle Cell (CSC) neoplasms with expert and non-expert labels at region and WSI levels. It is the first dataset of these types of neoplasms made available. The regions selected by the experts are used to learn an automatic extractor of Regions of Interest (ROIs) from WSIs. To produce the embedding of each WSI, the representations of patches within the ROIs are obtained using a contrastive learning method, and then combined. Finally, they are fed to a Gaussian process-based crowdsourcing classifier, which utilizes the noisy non-expert WSI labels. We validate our crowdsourcing-multiple instance learning method in the CR-AI4SkIN dataset, addressing a binary classification problem (malign vs. benign). The proposed method obtains an F1 score of 0.7911 on the test set, outperforming three widely used aggregation methods for crowdsourcing tasks. Furthermore, our crowdsourcing method also outperforms the supervised model with expert labels on the test set (F1-score = 0.6035). The promising results support the proposed crowdsourcing multiple instance learning annotation protocol. It also validates the automatic extraction of interest regions and the use of contrastive embedding and Gaussian process classification to perform crowdsourcing classification tasks.

Deep learning in computational dermatopathology of melanoma: A technical systematic literature review

Deep learning (DL) has become one of the major approaches in computational dermatopathology, evidenced by a significant increase in this topic in the current literature. We aim to provide a structured and comprehensive overview of peer-reviewed publications on DL applied to dermatopathology focused on melanoma. In comparison to well-published DL methods on non-medical images (e.g., classification on ImageNet), this field of application comprises a specific set of challenges, such as staining artifacts, large gigapixel images, and various magnification levels. Thus, we are particularly interested in the pathology-specific technical state-of-the-art. We also aim to summarize the best performances achieved thus far with respect to accuracy, along with an overview of self-reported limitations. Accordingly, we conducted a systematic literature review of peer-reviewed journal and conference articles published between 2012 and 2022 in the databases ACM Digital Library, Embase, IEEE Xplore, PubMed, and Scopus, expanded by forward and backward searches to identify 495 potentially eligible studies. After screening for relevance and quality, a total of 54 studies were included. We qualitatively summarized and analyzed these studies from technical, problem-oriented, and task-oriented perspectives. Our findings suggest that the technical aspects of DL for histopathology in melanoma can be further improved. The DL methodology was adopted later in this field, and still lacks the wider adoption of DL methods already shown to be effective for other applications. We also discuss upcoming trends toward ImageNet-based feature extraction and larger models. While DL has achieved human-competitive accuracy in routine pathological tasks, its performance on advanced tasks is still inferior to wet-lab testing (for example). Finally, we discuss the challenges impeding the translation of DL methods to clinical practice and provide insight into future research directions.

Deep Learning-Based Computational Cytopathologic Diagnosis of Metastatic Breast Carcinoma in Pleural Fluid

A Pleural effusion cytology is vital for treating metastatic breast cancer; however, concerns have arisen regarding the low accuracy and inter-observer variability in cytologic diagnosis. Although artificial intelligence-based image analysis has shown promise in cytopathology research, its application in diagnosing breast cancer in pleural fluid remains unexplored. To overcome these limitations, we evaluate the diagnostic accuracy of an artificial intelligence-based model using a large collection of cytopathological slides, to detect the malignant pleural effusion cytology associated with breast cancer. This study includes a total of 569 cytological slides of malignant pleural effusion of metastatic breast cancer from various institutions. We extracted 34,221 augmented image patches from whole-slide images and trained and validated a deep convolutional neural network model (DCNN) (Inception-ResNet-V2) with the images. Using this model, we classified 845 randomly selected patches, which were reviewed by three pathologists to compare their accuracy. The DCNN model outperforms the pathologists by demonstrating higher accuracy, sensitivity, and specificity compared to the pathologists (81.1% vs. 68.7%, 95.0% vs. 72.5%, and 98.6% vs. 88.9%, respectively). The pathologists reviewed the discordant cases of DCNN. After re-examination, the average accuracy, sensitivity, and specificity of the pathologists improved to 87.9, 80.2, and 95.7%, respectively. This study shows that DCNN can accurately diagnose malignant pleural effusion cytology in breast cancer and has the potential to support pathologists.

Deep learning-based scoring of tumour-infiltrating lymphocytes is prognostic in primary melanoma and predictive to PD-1 checkpoint inhibition in melanoma metastases

BACKGROUND

Recent advances in digital pathology have enabled accurate and standardised enumeration of tumour-infiltrating lymphocytes (TILs). Here, we aim to evaluate TILs as a percentage electronic TIL score (eTILs) and investigate its prognostic and predictive relevance in cutaneous melanoma.

METHODS

We included stage I to IV cutaneous melanoma patients and used hematoxylin-eosin-stained slides for TIL analysis. We assessed eTILs as a continuous and categorical variable using the published cut-off of 16.6% and applied Cox regression models to evaluate associations of eTILs with relapse-free, distant metastasis-free, and overall survival. We compared eTILs of the primaries with matched metastasis. Moreover, we assessed the predictive relevance of eTILs in therapy-naïve metastases according to the first-line therapy.

FINDINGS

We analysed 321 primary cutaneous melanomas and 191 metastatic samples. In simple Cox regression, tumour thickness (p < 0.0001), presence of ulceration (p = 0.0001) and eTILs ≤16.6% (p = 0.0012) were found to be significant unfavourable prognostic factors for RFS. In multiple Cox regression, eTILs ≤16.6% (p = 0.0161) remained significant and downgraded the current staging. Lower eTILs in the primary tissue was associated with unfavourable relapse-free (p = 0.0014) and distant metastasis-free survival (p = 0.0056). In multiple Cox regression adjusted for tumour thickness and ulceration, eTILs as continuous remained significant (p = 0.019). When comparing TILs in primary tissue and corresponding metastasis of the same patient, eTILs in metastases was lower than in primary melanomas (p < 0.0001). In therapy-naïve metastases, an eTILs >12.2% was associated with longer progression-free survival (p = 0.037) and melanoma-specific survival (p = 0.0038) in patients treated with anti-PD-1-based immunotherapy. In multiple Cox regression, lactate dehydrogenase (p < 0.0001) and eTILs ≤12.2% (p = 0.0130) were significantly associated with unfavourable melanoma-specific survival.

INTERPRETATION

Assessment of TILs is prognostic in primary melanoma samples, and the eTILs complements staging. In therapy-naïve metastases, eTILs ≤12.2% is predictive of unfavourable survival outcomes in patients receiving anti-PD-1-based therapy.

FUNDING

See a detailed list of funding bodies in the Acknowledgements section at the end of the manuscript.

Recent Advances in Melanoma Diagnosis and Prognosis Using Machine Learning Methods

PURPOSE OF REVIEW

The purpose was to summarize the current role and state of artificial intelligence and machine learning in the diagnosis and management of melanoma.

RECENT FINDINGS

Deep learning algorithms can identify melanoma from clinical, dermoscopic, and whole slide pathology images with increasing accuracy. Efforts to provide more granular annotation to datasets and to identify new predictors are ongoing. There have been many incremental advances in both melanoma diagnostics and prognostic tools using artificial intelligence and machine learning. Higher quality input data will further improve these models' capabilities.

Artificial Intelligence-Based Opportunities in Liver Pathology-A Systematic Review

BACKGROUND

Artificial Intelligence (AI)-based Deep Neural Networks (DNNs) can handle a wide range of applications in image analysis, ranging from automated segmentation to diagnostic and prediction. As such, they have revolutionized healthcare, including in the liver pathology field.

OBJECTIVE

The present study aims to provide a systematic review of applications and performances provided by DNN algorithms in liver pathology throughout the Pubmed and Embase databases up to December 2022, for tumoral, metabolic and inflammatory fields.

RESULTS

42 articles were selected and fully reviewed. Each article was evaluated through the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool, highlighting their risks of bias.

CONCLUSIONS

DNN-based models are well represented in the field of liver pathology, and their applications are diverse. Most studies, however, presented at least one domain with a high risk of bias according to the QUADAS-2 tool. Hence, DNN models in liver pathology present future opportunities and persistent limitations. To our knowledge, this review is the first one solely focused on DNN-based applications in liver pathology, and to evaluate their bias through the lens of the QUADAS2 tool.

Labeling confidence for uncertainty-aware histology image classification

Deep learning-based models applied to digital pathology require large, curated datasets with high-quality (HQ) annotations to perform correctly. In many cases, recruiting expert pathologists to annotate large databases is not feasible, and it is necessary to collect additional labeled data with varying label qualities, e.g., pathologists-in-training (henceforth, non-expert annotators). Learning from datasets with noisy labels is more challenging in medical applications since medical imaging datasets tend to have instance-dependent noise and suffer from high inter/intra-observer variability. In this paper, we design an uncertainty-driven labeling strategy with which we generate soft labels from 10 non-expert annotators for multi-class skin cancer classification. Based on this soft annotation, we propose an uncertainty estimation-based framework to handle these noisy labels. This framework is based on a novel formulation using a dual-branch min-max entropy calibration to penalize inexact labels during the training. Comprehensive experiments demonstrate the promising performance of our labeling strategy. Results show a consistent improvement by using soft labels with standard cross-entropy loss during training (∼4.0% F1-score) and increases when calibrating the model with the proposed min-max entropy calibration (∼6.6% F1-score). These improvements are produced at negligible cost, both in terms of annotation and calculation.

Image analysis of cutaneous melanoma histology: a systematic review and meta-analysis

The current subjective histopathological assessment of cutaneous melanoma is challenging. The application of image analysis algorithms to histological images may facilitate improvements in workflow and prognostication. To date, several individual algorithms applied to melanoma histological images have been reported with variations in approach and reported accuracies. Histological digital images can be created using a camera mounted on a light microscope, or through whole slide image (WSI) generation using a whole slide scanner. Before any such tool could be integrated into clinical workflow, the accuracy of the technology should be carefully evaluated and summarised. Therefore, the objective of this review was to evaluate the accuracy of existing image analysis algorithms applied to digital histological images of cutaneous melanoma. Database searching of PubMed and Embase from inception to 11th March 2022 was conducted alongside citation checking and examining reports from organisations. All studies reporting accuracy of any image analysis applied to histological images of cutaneous melanoma, were included. The reference standard was any histological assessment of haematoxylin and eosin-stained slides and/or immunohistochemical staining. Citations were independently deduplicated and screened by two review authors and disagreements were resolved through discussion. The data was extracted concerning study demographics; type of image analysis; type of reference standard; conditions included and test statistics to construct 2 × 2 tables. Data was extracted in accordance with our protocol and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses-Diagnostic Test Accuracy (PRISMA-DTA) Statement. A bivariate random-effects meta-analysis was used to estimate summary sensitivities and specificities with 95% confidence intervals (CI). Assessment of methodological quality was conducted using a tailored version of the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. The primary outcome was the pooled sensitivity and specificity of image analysis applied to cutaneous melanoma histological images. Sixteen studies were included in the systematic review, representing 4,888 specimens. Six studies were included in the meta-analysis. The mean sensitivity and specificity of automated image analysis algorithms applied to melanoma histological images was 90% (CI 82%, 95%) and 92% (CI 79%, 97%), respectively. Based on limited and heterogeneous data, image analysis appears to offer high accuracy when applied to histological images of cutaneous melanoma. However, given the early exploratory nature of these studies, further development work is necessary to improve their performance.

Deep Learning for Skin Melanocytic Tumors in Whole-Slide Images: A Systematic Review

The rise of Artificial Intelligence (AI) has shown promising performance as a support tool in clinical pathology workflows. In addition to the well-known interobserver variability between dermatopathologists, melanomas present a significant challenge in their histological interpretation. This study aims to analyze all previously published studies on whole-slide images of melanocytic tumors that rely on deep learning techniques for automatic image analysis. Embase, Pubmed, Web of Science, and Virtual Health Library were used to search for relevant studies for the systematic review, in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. Articles from 2015 to July 2022 were included, with an emphasis placed on the used artificial intelligence methods. Twenty-eight studies that fulfilled the inclusion criteria were grouped into four groups based on their clinical objectives, including pathologists versus deep learning models (n = 10), diagnostic prediction (n = 7); prognosis (n = 5), and histological features (n = 6). These were then analyzed to draw conclusions on the general parameters and conditions of AI in pathology, as well as the necessary factors for better performance in real scenarios.

Diagnostic and Prognostic Deep Learning Applications for Histological Assessment of Cutaneous Melanoma

Melanoma is among the most devastating human malignancies. Accurate diagnosis and prognosis are essential to offer optimal treatment. Histopathology is the gold standard for establishing melanoma diagnosis and prognostic features. However, discrepancies often exist between pathologists, and analysis is costly and time-consuming. Deep-learning algorithms are deployed to improve melanoma diagnosis and prognostication from histological images of melanoma. In recent years, the development of these machine-learning tools has accelerated, and machine learning is poised to become a clinical tool to aid melanoma histology. Nevertheless, a review of the advances in machine learning in melanoma histology was lacking. We performed a comprehensive literature search to provide a complete overview of the recent advances in machine learning in the assessment of melanoma based on hematoxylin eosin digital pathology images. In our work, we review 37 recent publications, compare the methods and performance of the reviewed studies, and highlight the variety of promising machine-learning applications in melanoma histology.

Artificial intelligence: Its role in dermatopathology

Artificial intelligence (AI), a major frontier in the field of medical research, can potentially lead to a paradigm shift in clinical practice. A type of artificial intelligence system known as convolutional neural network points to the possible utility of deep learning in dermatopathology. Though pathology has been traditionally restricted to microscopes and glass slides, recent advancement in digital pathological imaging has led to a transition making it a potential branch for the implementation of artificial intelligence. The current application of artificial intelligence in dermatopathology is to complement the diagnosis and requires a well-trained dermatopathologist's guidance for better designing and development of deep learning algorithms. Here we review the recent advances of artificial intelligence in dermatopathology, its applications in disease diagnosis and in research, along with its limitations and future potential.

High-fidelity detection, subtyping, and localization of five skin neoplasms using supervised and semi-supervised learning

Background

Skin cancers are the most common malignancies diagnosed worldwide. While the early detection and treatment of pre-cancerous and cancerous skin lesions can dramatically improve outcomes, factors such as a global shortage of pathologists, increased workloads, and high rates of diagnostic discordance underscore the need for techniques that improve pathology workflows. Although AI models are now being used to classify lesions from whole slide images (WSIs), diagnostic performance rarely surpasses that of expert pathologists.

Objectives

The objective of the present study was to create an AI model to detect and classify skin lesions with a higher degree of sensitivity than previously demonstrated, with potential to match and eventually surpass expert pathologists to improve clinical workflows.

Methods

We combined supervised learning (SL) with semi-supervised learning (SSL) to produce an end-to-end multi-level skin detection system that not only detects 5 main types of skin lesions with high sensitivity and specificity, but also subtypes, localizes, and provides margin status to evaluate the proximity of the lesion to non-epidermal margins. The Supervised Training Subset consisted of 2188 random WSIs collected by the PathologyWatch (PW) laboratory between 2013 and 2018, while the Weakly Supervised Subset consisted of 5161 WSIs from daily case specimens. The Validation Set consisted of 250 curated daily case WSIs obtained from the PW tissue archives and included 50 "mimickers". The Testing Set (3821 WSIs) was composed of non-curated daily case specimens collected from July 20, 2021 to August 20, 2021 from PW laboratories.

Results

The performance characteristics of our AI model (i.e., Mihm) were assessed retrospectively by running the Testing Set through the Mihm Evaluation Pipeline. Our results show that the sensitivity of Mihm in classifying melanocytic lesions, basal cell carcinoma, and atypical squamous lesions, verruca vulgaris, and seborrheic keratosis was 98.91% (95% CI: 98.27%, 99.55%), 97.24% (95% CI: 96.15%, 98.33%), 95.26% (95% CI: 93.79%, 96.73%), 93.50% (95% CI: 89.14%, 97.86%), and 86.91% (95% CI: 82.13%, 91.69%), respectively. Additionally, our multi-level (i.e., patch-level, ROI-level, and WSI-level) detection algorithm includes a qualitative feature that subtypes lesions, an AI overlay in the front-end digital display that localizes diagnostic ROIs, and reports on margin status by detecting overlap between lesions and non-epidermal tissue margins.

Conclusions

Our AI model, developed in collaboration with dermatopathologists, detects 5 skin lesion types with higher sensitivity than previously published AI models, and provides end users with information such as subtyping, localization, and margin status in a front-end digital display. Our end-to-end system has the potential to improve pathology workflows by increasing diagnostic accuracy, expediting the course of patient care, and ultimately improving patient outcomes.

Deep learning for the detection of anatomical tissue structures and neoplasms of the skin on scanned histopathological tissue sections

Basal cell carcinoma (BCC), squamous cell carcinoma (SqCC) and melanoma are among the most common cancer types. Correct diagnosis based on histological evaluation after biopsy or excision is paramount for adequate therapy stratification. Deep learning on histological slides has been suggested to complement and improve routine diagnostics, but publicly available curated and annotated data and usable models trained to distinguish common skin tumors are rare and often lack heterogeneous non-tumor categories. A total of 16 classes from 386 cases were manually annotated on scanned histological slides, 129,364 100 x 100 µm (~395 x 395 px) image tiles were extracted and split into a training, validation and test set. An EfficientV2 neuronal network was trained and optimized to classify image categories. Cross entropy loss, balanced accuracy and Matthews correlation coefficient were used for model evaluation. Image and patient data were assessed with confusion matrices. Application of the model to an external set of whole slides facilitated localization of melanoma and non-tumor tissue. Automated differentiation of BCC, SqCC, melanoma, naevi and non-tumor tissue structures was possible, and a high diagnostic accuracy was achieved in the validation (98%) and test (97%) set. In summary, we provide a curated dataset including the most common neoplasms of the skin and various anatomical compartments to enable researchers to train, validate and improve deep learning models. Automated classification of skin tumors by deep learning techniques is possible with high accuracy, facilitates tumor localization and has the potential to support and improve routine diagnostics.

Histologic Screening of Malignant Melanoma, Spitz, Dermal and Junctional Melanocytic Nevi Using a Deep Learning Model

OBJECTIVE

The integration of an artificial intelligence tool into pathologists' workflow may lead to a more accurate and timely diagnosis of melanocytic lesions, directly patient care. The objective of this study was to create and evaluate the performance of such a model in achieving clinical-grade diagnoses of Spitz nevi, dermal and junctional melanocytic nevi, and melanomas.

METHODS

We created a beginner-level training environment by teaching our algorithm to perform cytologic inferences on 136,216 manually annotated tiles of hematoxylin and eosin-stained slides consisting of unequivocal melanocytic nevi, Spitz nevi, and invasive melanoma cases. We sequentially trained and tested our network to provide a final diagnosis-classification on 39 cases in total. Positive predictive value (precision) and sensitivity (recall) were used to measure our performance.

RESULTS

The tile-classification algorithm predicted the 136,216 irrelevant, melanoma, melanocytic nevi, and Spitz nevi tiles at sensitivities of 96%, 93%, 94% and 73%, respectively. The final trained model was able to correctly classify and predict the correct diagnosis in 85.7% of unseen cases (n = 28), reporting at or near screening-level performances for precision and recall of melanoma (76.2%, 100.0%), melanocytic nevi (100.0%, 75.0%), and Spitz nevi (100.0%, 75.0%).

CONCLUSIONS

Our pilot study proves that convolutional networks trained on cellular morphology to classify melanocytic proliferations can be used as a powerful tool to assist pathologists in screening for melanoma versus other benign lesions.

Deep Learning Assisted Diagnosis of Onychomycosis on Whole-Slide Images

Background: Onychomycosis numbers among the most common fungal infections in humans affecting finger- or toenails. Histology remains a frequently applied screening technique to diagnose onychomycosis. Screening slides for fungal elements can be time-consuming for pathologists, and sensitivity in cases with low amounts of fungi remains a concern. Convolutional neural networks (CNNs) have revolutionized image classification in recent years. The goal of our project was to evaluate if a U-NET-based segmentation approach as a subcategory of CNNs can be applied to detect fungal elements on digitized histologic sections of human nail specimens and to compare it with the performance of 11 board-certified dermatopathologists. Methods: In total, 664 corresponding H&E- and PAS-stained histologic whole-slide images (WSIs) of human nail plates from four different laboratories were digitized. Histologic structures were manually annotated. A U-NET image segmentation model was trained for binary segmentation on the dataset generated by annotated slides. Results: The U-NET algorithm detected 90.5% of WSIs with fungi, demonstrating a comparable sensitivity with that of the 11 board-certified dermatopathologists (sensitivity of 89.2%). Conclusions: Our results demonstrate that machine-learning-based algorithms applied to real-world clinical cases can produce comparable sensitivities to human pathologists. Our established U-NET may be used as a supportive diagnostic tool to preselect possible slides with fungal elements. Slides where fungal elements are indicated by our U-NET should be reevaluated by the pathologist to confirm or refute the diagnosis of onychomycosis.

Advantages of manual and automatic computer-aided compared to traditional histopathological diagnosis of melanoma: A pilot study

BACKGROUND

Cutaneous malignant melanoma (CMM) accounts for the highest mortality rate among all skin cancers. Traditional histopathologic diagnosis may be limited by the pathologists' subjectivity. Second-opinion strategies and multidisciplinary consultations are usually performed to overcome this issue. An available solution in the future could be the use of automated solutions based on a computational algorithm that could help the pathologist in everyday practice. The aim of this pilot study was to investigate the potential diagnostic aid of a machine-based algorithm in the histopathologic diagnosis of CMM.

METHODS

We retrospectively examined excisional biopsies of 50 CMM and 20 benign congenital compound nevi. Hematoxylin and eosin (H&E) stained WSI were reviewed independently by two expert dermatopathologists. A fully automated pipeline for WSI processing to support the estimation and prioritization of the melanoma areas was developed.

RESULTS

The spatial distribution of the nuclei in the sample provided a multi-scale overview of the tumor. A global overview of the lesion's silhouette was achieved and, by increasing the magnification, the topological distribution of the nuclei and the most informative areas of interest for the CMM diagnosis were identified and highlighted. These silhouettes allow the histopathologist to discriminate between nevus and CMM with an accuracy of 96% without any extra information.

CONCLUSION

In this study we proposed an easy-to-use model that produces segmentations of CMM silhouettes at fine detail level.

Artificial Intelligence-Based Approaches to Reflectance Confocal Microscopy Image Analysis in Dermatology

Reflectance confocal microscopy (RCM) is a non-invasive imaging method designed to identify various skin diseases. Confocal based diagnosis may be subjective due to the learning curve of the method, the scarcity of training programs available for RCM, and the lack of clearly defined diagnostic criteria for all skin conditions. Given that in vivo RCM is becoming more widely used in dermatology, numerous deep learning technologies have been developed in recent years to provide a more objective approach to RCM image analysis. Machine learning-based algorithms are used in RCM image quality assessment to reduce the number of artifacts the operator has to view, shorten evaluation times, and decrease the number of patient visits to the clinic. However, the current visual method for identifying the dermal-epidermal junction (DEJ) in RCM images is subjective, and there is a lot of variation. The delineation of DEJ on RCM images could be automated through artificial intelligence, saving time and assisting novice RCM users in studying the key DEJ morphological structure. The purpose of this paper is to supply a current summary of machine learning and artificial intelligence’s impact on the quality control of RCM images, key morphological structures identification, and detection of different skin lesion types on static RCM images.

An attention-based weakly supervised framework for spitzoid melanocytic lesion diagnosis in whole slide images

Melanoma is an aggressive neoplasm responsible for the majority of deaths from skin cancer. Specifically, spitzoid melanocytic tumors are one of the most challenging melanocytic lesions due to their ambiguous morphological features. The gold standard for its diagnosis and prognosis is the analysis of skin biopsies. In this process, dermatopathologists visualize skin histology slides under a microscope, in a highly time-consuming and subjective task. In the last years, computer-aided diagnosis (CAD) systems have emerged as a promising tool that could support pathologists in daily clinical practice. Nevertheless, no automatic CAD systems have yet been proposed for the analysis of spitzoid lesions. Regarding common melanoma, no system allows both the selection of the tumor region and the prediction of the benign or malignant form in the diagnosis. Motivated by this, we propose a novel end-to-end weakly supervised deep learning model, based on inductive transfer learning with an improved convolutional neural network (CNN) to refine the embedding features of the latent space. The framework is composed of a source model in charge of finding the tumor patch-level patterns, and a target model focuses on the specific diagnosis of a biopsy. The latter retrains the backbone of the source model through a multiple instance learning workflow to obtain the biopsy-level scoring. To evaluate the performance of the proposed methods, we performed extensive experiments on a private skin database with spitzoid lesions. Test results achieved an accuracy of 0.9231 and 0.80 for the source and the target models, respectively. In addition, the heat map findings are directly in line with the clinicians' medical decision and even highlight, in some cases, patterns of interest that were overlooked by the pathologist.

Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts

BACKGROUND

Multiple studies have compared the performance of artificial intelligence (AI)-based models for automated skin cancer classification to human experts, thus setting the cornerstone for a successful translation of AI-based tools into clinicopathological practice.

OBJECTIVE

The objective of the study was to systematically analyse the current state of research on reader studies involving melanoma and to assess their potential clinical relevance by evaluating three main aspects: test set characteristics (holdout/out-of-distribution data set, composition), test setting (experimental/clinical, inclusion of metadata) and representativeness of participating clinicians.

METHODS

PubMed, Medline and ScienceDirect were screened for peer-reviewed studies published between 2017 and 2021 and dealing with AI-based skin cancer classification involving melanoma. The search terms skin cancer classification, deep learning, convolutional neural network (CNN), melanoma (detection), digital biomarkers, histopathology and whole slide imaging were combined. Based on the search results, only studies that considered direct comparison of AI results with clinicians and had a diagnostic classification as their main objective were included.

RESULTS

A total of 19 reader studies fulfilled the inclusion criteria. Of these, 11 CNN-based approaches addressed the classification of dermoscopic images; 6 concentrated on the classification of clinical images, whereas 2 dermatopathological studies utilised digitised histopathological whole slide images.

CONCLUSIONS

All 19 included studies demonstrated superior or at least equivalent performance of CNN-based classifiers compared with clinicians. However, almost all studies were conducted in highly artificial settings based exclusively on single images of the suspicious lesions. Moreover, test sets mainly consisted of holdout images and did not represent the full range of patient populations and melanoma subtypes encountered in clinical practice.

Spatial architecture of the immune microenvironment orchestrates tumor immunity and therapeutic response

Tumors are not only aggregates of malignant cells but also well-organized complex ecosystems. The immunological components within tumors, termed the tumor immune microenvironment (TIME), have long been shown to be strongly related to tumor development, recurrence and metastasis. However, conventional studies that underestimate the potential value of the spatial architecture of the TIME are unable to completely elucidate its complexity. As innovative high-flux and high-dimensional technologies emerge, researchers can more feasibly and accurately detect and depict the spatial architecture of the TIME. These findings have improved our understanding of the complexity and role of the TIME in tumor biology. In this review, we first epitomized some representative emerging technologies in the study of the spatial architecture of the TIME and categorized the description methods used to characterize these structures. Then, we determined the functions of the spatial architecture of the TIME in tumor biology and the effects of the gradient of extracellular nonspecific chemicals (ENSCs) on the TIME. We also discussed the potential clinical value of our understanding of the spatial architectures of the TIME, as well as current limitations and future prospects in this novel field. This review will bring spatial architectures of the TIME, an emerging dimension of tumor ecosystem research, to the attention of more researchers and promote its application in tumor research and clinical practice.

Combining CNN-based histologic whole slide image analysis and patient data to improve skin cancer classification

BACKGROUND

Clinicians and pathologists traditionally use patient data in addition to clinical examination to support their diagnoses.

OBJECTIVES

We investigated whether a combination of histologic whole slides image (WSI) analysis based on convolutional neural networks (CNNs) and commonly available patient data (age, sex and anatomical site of the lesion) in a binary melanoma/nevus classification task could increase the performance compared with CNNs alone.

METHODS

We used 431 WSIs from two different laboratories and analysed the performance of classifiers that used the image or patient data individually or three common fusion techniques. Furthermore, we tested a naive combination of patient data and an image classifier: for cases interpreted as 'uncertain' (CNN output score <0.7), the decision of the CNN was replaced by the decision of the patient data classifier.

RESULTS

The CNN on its own achieved the best performance (mean ± standard deviation of five individual runs) with AUROC of 92.30% ± 0.23% and balanced accuracy of 83.17% ± 0.38%. While the classification performance was not significantly improved in general by any of the tested fusions, naive strategy of replacing the image classifier with the patient data classifier on slides with low output scores improved balanced accuracy to 86.72% ± 0.36%.

CONCLUSION

In most cases, the CNN on its own was so accurate that patient data integration did not provide any benefit. However, incorporating patient data for lesions that were classified by the CNN with low 'confidence' improved balanced accuracy.