Automated curation of large-scale cancer histopathology image datasets using deep learning

BACKGROUND

Artificial intelligence (AI) has numerous applications in pathology, supporting diagnosis and prognostication in cancer. However, most AI models are trained on highly selected data, typically one tissue slide per patient. In reality, especially for large surgical resection specimens, dozens of slides can be available for each patient. Manually sorting and labelling whole-slide images (WSIs) is a very time-consuming process, hindering the direct application of AI on the collected tissue samples from large cohorts. In this study we addressed this issue by developing a deep-learning (DL)-based method for automatic curation of large pathology datasets with several slides per patient.

METHODS

We collected multiple large multicentric datasets of colorectal cancer histopathological slides from the United Kingdom (FOXTROT, N = 21,384 slides; CR07, N = 7985 slides) and Germany (DACHS, N = 3606 slides). These datasets contained multiple types of tissue slides, including bowel resection specimens, endoscopic biopsies, lymph node resections, immunohistochemistry-stained slides, and tissue microarrays. We developed, trained, and tested a deep convolutional neural network model to predict the type of slide from the slide overview (thumbnail) image. The primary statistical endpoint was the macro-averaged area under the receiver operating curve (AUROCs) for detection of the type of slide.

RESULTS

In the primary dataset (FOXTROT), with an AUROC of 0.995 [95% confidence interval [CI]: 0.994-0.996] the algorithm achieved a high classification performance and was able to accurately predict the type of slide from the thumbnail image alone. In the two external test cohorts (CR07, DACHS) AUROCs of 0.982 [95% CI: 0.979-0.985] and 0.875 [95% CI: 0.864-0.887] were observed, which indicates the generalizability of the trained model on unseen datasets. With a confidence threshold of 0.95, the model reached an accuracy of 94.6% (7331 classified cases) in CR07 and 85.1% (2752 classified cases) for the DACHS cohort.

CONCLUSION

Our findings show that using the low-resolution thumbnail image is sufficient to accurately classify the type of slide in digital pathology. This can support researchers to make the vast resource of existing pathology archives accessible to modern AI models with only minimal manual annotations.

Multi-domain stain normalization for digital pathology: A cycle-consistent adversarial network for whole slide images

The variation in histologic staining between different medical centers is one of the most profound challenges in the field of computer-aided diagnosis. The appearance disparity of pathological whole slide images causes algorithms to become less reliable, which in turn impedes the wide-spread applicability of downstream tasks like cancer diagnosis. Furthermore, different stainings lead to biases in the training which in case of domain shifts negatively affect the test performance. Therefore, in this paper we propose MultiStain-CycleGAN, a multi-domain approach to stain normalization based on CycleGAN. Our modifications to CycleGAN allow us to normalize images of different origins without retraining or using different models. We perform an extensive evaluation of our method using various metrics and compare it to commonly used methods that are multi-domain capable. First, we evaluate how well our method fools a domain classifier that tries to assign a medical center to an image. Then, we test our normalization on the tumor classification performance of a downstream classifier. Furthermore, we evaluate the image quality of the normalized images using the Structural similarity index and the ability to reduce the domain shift using the Fréchet inception distance. We show that our method proves to be multi-domain capable, provides a very high image quality among the compared methods, and can most reliably fool the domain classifier while keeping the tumor classifier performance high. By reducing the domain influence, biases in the data can be removed on the one hand and the origin of the whole slide image can be disguised on the other, thus enhancing patient data privacy.

3-Dimensional Reconstruction From Histopathological Sections: A Systematic Review

Although pathological tissue analysis is typically performed on single 2-dimensional (2D) histologic reference slides, 3-dimensional (3D) reconstruction from a sequence of histologic sections could provide novel opportunities for spatial analysis of the extracted tissue. In this review, we analyze recent works published after 2018 and report information on the extracted tissue types, the section thickness, and the number of sections used for reconstruction. By analyzing the technological requirements for 3D reconstruction, we observe that software tools exist, both free and commercial, which include the functionality to perform 3D reconstruction from a sequence of histologic images. Through the analysis of the most recent works, we provide an overview of the workflows and tools that are currently used for 3D reconstruction from histologic sections and address points for future work, such as a missing common file format or computer-aided analysis of the reconstructed model.

Federated Learning for Decentralized Artificial Intelligence in Melanoma Diagnostics

Importance

The development of artificial intelligence (AI)-based melanoma classifiers typically calls for large, centralized datasets, requiring hospitals to give away their patient data, which raises serious privacy concerns. To address this concern, decentralized federated learning has been proposed, where classifier development is distributed across hospitals.

Objective

To investigate whether a more privacy-preserving federated learning approach can achieve comparable diagnostic performance to a classical centralized (ie, single-model) and ensemble learning approach for AI-based melanoma diagnostics.

Design, Setting, and Participants

This multicentric, single-arm diagnostic study developed a federated model for melanoma-nevus classification using histopathological whole-slide images prospectively acquired at 6 German university hospitals between April 2021 and February 2023 and benchmarked it using both a holdout and an external test dataset. Data analysis was performed from February to April 2023.

Exposures

All whole-slide images were retrospectively analyzed by an AI-based classifier without influencing routine clinical care.

Main Outcomes and Measures

The area under the receiver operating characteristic curve (AUROC) served as the primary end point for evaluating the diagnostic performance. Secondary end points included balanced accuracy, sensitivity, and specificity.

Results

The study included 1025 whole-slide images of clinically melanoma-suspicious skin lesions from 923 patients, consisting of 388 histopathologically confirmed invasive melanomas and 637 nevi. The median (range) age at diagnosis was 58 (18-95) years for the training set, 57 (18-93) years for the holdout test dataset, and 61 (18-95) years for the external test dataset; the median (range) Breslow thickness was 0.70 (0.10-34.00) mm, 0.70 (0.20-14.40) mm, and 0.80 (0.30-20.00) mm, respectively. The federated approach (0.8579; 95% CI, 0.7693-0.9299) performed significantly worse than the classical centralized approach (0.9024; 95% CI, 0.8379-0.9565) in terms of AUROC on a holdout test dataset (pairwise Wilcoxon signed-rank, P < .001) but performed significantly better (0.9126; 95% CI, 0.8810-0.9412) than the classical centralized approach (0.9045; 95% CI, 0.8701-0.9331) on an external test dataset (pairwise Wilcoxon signed-rank, P < .001). Notably, the federated approach performed significantly worse than the ensemble approach on both the holdout (0.8867; 95% CI, 0.8103-0.9481) and external test dataset (0.9227; 95% CI, 0.8941-0.9479).

Conclusions and Relevance

The findings of this diagnostic study suggest that federated learning is a viable approach for the binary classification of invasive melanomas and nevi on a clinically representative distributed dataset. Federated learning can improve privacy protection in AI-based melanoma diagnostics while simultaneously promoting collaboration across institutions and countries. Moreover, it may have the potential to be extended to other image classification tasks in digital cancer histopathology and beyond.

Evaluating deep learning-based melanoma classification using immunohistochemistry and routine histology: A three center study

Pathologists routinely use immunohistochemical (IHC)-stained tissue slides against MelanA in addition to hematoxylin and eosin (H&E)-stained slides to improve their accuracy in diagnosing melanomas. The use of diagnostic Deep Learning (DL)-based support systems for automated examination of tissue morphology and cellular composition has been well studied in standard H&E-stained tissue slides. In contrast, there are few studies that analyze IHC slides using DL. Therefore, we investigated the separate and joint performance of ResNets trained on MelanA and corresponding H&E-stained slides. The MelanA classifier achieved an area under receiver operating characteristics curve (AUROC) of 0.82 and 0.74 on out of distribution (OOD)-datasets, similar to the H&E-based benchmark classification of 0.81 and 0.75, respectively. A combined classifier using MelanA and H&E achieved AUROCs of 0.85 and 0.81 on the OOD datasets. DL MelanA-based assistance systems show the same performance as the benchmark H&E classification and may be improved by multi stain classification to assist pathologists in their clinical routine.

Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanoma

Artificial intelligence (AI) systems have been shown to help dermatologists diagnose melanoma more accurately, however they lack transparency, hindering user acceptance. Explainable AI (XAI) methods can help to increase transparency, yet often lack precise, domain-specific explanations. Moreover, the impact of XAI methods on dermatologists' decisions has not yet been evaluated. Building upon previous research, we introduce an XAI system that provides precise and domain-specific explanations alongside its differential diagnoses of melanomas and nevi. Through a three-phase study, we assess its impact on dermatologists' diagnostic accuracy, diagnostic confidence, and trust in the XAI-support. Our results show strong alignment between XAI and dermatologist explanations. We also show that dermatologists' confidence in their diagnoses, and their trust in the support system significantly increase with XAI compared to conventional AI. This study highlights dermatologists' willingness to adopt such XAI systems, promoting future use in the clinic.

End-to-end prognostication in colorectal cancer by deep learning: a retrospective, multicentre study

BACKGROUND

Precise prognosis prediction in patients with colorectal cancer (ie, forecasting survival) is pivotal for individualised treatment and care. Histopathological tissue slides of colorectal cancer specimens contain rich prognostically relevant information. However, existing studies do not have multicentre external validation with real-world sample processing protocols, and algorithms are not yet widely used in clinical routine.

METHODS

In this retrospective, multicentre study, we collected tissue samples from four groups of patients with resected colorectal cancer from Australia, Germany, and the USA. We developed and externally validated a deep learning-based prognostic-stratification system for automatic prediction of overall and cancer-specific survival in patients with resected colorectal cancer. We used the model-predicted risk scores to stratify patients into different risk groups and compared survival outcomes between these groups. Additionally, we evaluated the prognostic value of these risk groups after adjusting for established prognostic variables.

FINDINGS

We trained and validated our model on a total of 4428 patients. We found that patients could be divided into high-risk and low-risk groups on the basis of the deep learning-based risk score. On the internal test set, the group with a high-risk score had a worse prognosis than the group with a low-risk score, as reflected by a hazard ratio (HR) of 4·50 (95% CI 3·33-6·09) for overall survival and 8·35 (5·06-13·78) for disease-specific survival (DSS). We found consistent performance across three large external test sets. In a test set of 1395 patients, the high-risk group had a lower DSS than the low-risk group, with an HR of 3·08 (2·44-3·89). In two additional test sets, the HRs for DSS were 2·23 (1·23-4·04) and 3·07 (1·78-5·3). We showed that the prognostic value of the deep learning-based risk score is independent of established clinical risk factors.

INTERPRETATION

Our findings indicate that attention-based self-supervised deep learning can robustly offer a prognosis on clinical outcomes in patients with colorectal cancer, generalising across different populations and serving as a potentially new prognostic tool in clinical decision making for colorectal cancer management. We release all source codes and trained models under an open-source licence, allowing other researchers to reuse and build upon our work.

FUNDING

The German Federal Ministry of Health, the Max-Eder-Programme of German Cancer Aid, the German Federal Ministry of Education and Research, the German Academic Exchange Service, and the EU.

Deep learning to predict breast cancer sentinel lymph node status on INSEMA histological images

BACKGROUND

Sentinel lymph node (SLN) status is a clinically important prognostic biomarker in breast cancer and is used to guide therapy, especially for hormone receptor-positive, HER2-negative cases. However, invasive lymph node staging is increasingly omitted before therapy, and studies such as the randomised Intergroup Sentinel Mamma (INSEMA) trial address the potential for further de-escalation of axillary surgery. Therefore, it would be helpful to accurately predict the pretherapeutic sentinel status using medical images.

METHODS

Using a ResNet 50 architecture pretrained on ImageNet and a previously successful strategy, we trained deep learning (DL)-based image analysis algorithms to predict sentinel status on hematoxylin/eosin-stained images of predominantly luminal, primary breast tumours from the INSEMA trial and three additional, independent cohorts (The Cancer Genome Atlas (TCGA) and cohorts from the University hospitals of Mannheim and Regensburg), and compared their performance with that of a logistic regression using clinical data only. Performance on an INSEMA hold-out set was investigated in a blinded manner.

RESULTS

None of the generated image analysis algorithms yielded significantly better than random areas under the receiver operating characteristic curves on the test sets, including the hold-out test set from INSEMA. In contrast, the logistic regression fitted on the Mannheim cohort retained a better than random performance on INSEMA and Regensburg. Including the image analysis model output in the logistic regression did not improve performance further on INSEMA.

CONCLUSIONS

Employing DL-based image analysis on histological slides, we could not predict SLN status for unseen cases in the INSEMA trial and other predominantly luminal cohorts.

A 3-dimensional histology computer model of malignant melanoma and its implications for digital pathology

BACKGROUND

Historically, cancer diagnoses have been made by pathologists using two-dimensional histological slides. However, with the advent of digital pathology and artificial intelligence, slides are being digitised, providing new opportunities to integrate their information. Since nature is 3-dimensional (3D), it seems intuitive to digitally reassemble the 3D structure for diagnosis.

OBJECTIVE

To develop the first human-3D-melanoma-histology-model with full data and code availability. Further, to evaluate the 3D-simulation together with experienced pathologists in the field and discuss the implications of digital 3D-models for the future of digital pathology.

METHODS

A malignant melanoma of the skin was digitised via 3 µm cuts by a slide scanner; an open-source software was then leveraged to construct the 3D model. A total of nine pathologists from four different countries with at least 10 years of experience in the histologic diagnosis of melanoma tested the model and discussed their experiences as well as implications for future pathology.

RESULTS

We successfully constructed and tested the first 3D-model of human melanoma. Based on testing, 88.9% of pathologists believe that the technology is likely to enter routine pathology within the next 10 years; advantages include a better reflectance of anatomy, 3D assessment of symmetry and the opportunity to simultaneously evaluate different tissue levels at the same time; limitations include the high consumption of tissue and a yet inferior resolution due to computational limitations.

CONCLUSIONS

3D-histology-models are promising for digital pathology of cancer and melanoma specifically, however, there are yet limitations which need to be carefully addressed.

Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise

In the past years, deep learning has seen an increase in usage in the domain of histopathological applications. However, while these approaches have shown great potential, in high-risk environments deep learning models need to be able to judge their uncertainty and be able to reject inputs when there is a significant chance of misclassification. In this work, we conduct a rigorous evaluation of the most commonly used uncertainty and robustness methods for the classification of Whole Slide Images, with a focus on the task of selective classification, where the model should reject the classification in situations in which it is uncertain. We conduct our experiments on tile-level under the aspects of domain shift and label noise, as well as on slide-level. In our experiments, we compare Deep Ensembles, Monte-Carlo Dropout, Stochastic Variational Inference, Test-Time Data Augmentation as well as ensembles of the latter approaches. We observe that ensembles of methods generally lead to better uncertainty estimates as well as an increased robustness towards domain shifts and label noise, while contrary to results from classical computer vision benchmarks no systematic gain of the other methods can be shown. Across methods, a rejection of the most uncertain samples reliably leads to a significant increase in classification accuracy on both in-distribution as well as out-of-distribution data. Furthermore, we conduct experiments comparing these methods under varying conditions of label noise. Lastly, we publish our code framework to facilitate further research on uncertainty estimation on histopathological data.

Colorectal cancer risk stratification on histological slides based on survival curves predicted by deep learning

Studies have shown that colorectal cancer prognosis can be predicted by deep learning-based analysis of histological tissue sections of the primary tumor. So far, this has been achieved using a binary prediction. Survival curves might contain more detailed information and thus enable a more fine-grained risk prediction. Therefore, we established survival curve-based CRC survival predictors and benchmarked them against standard binary survival predictors, comparing their performance extensively on the clinical high and low risk subsets of one internal and three external cohorts. Survival curve-based risk prediction achieved a very similar risk stratification to binary risk prediction for this task. Exchanging other components of the pipeline, namely input tissue and feature extractor, had largely identical effects on model performance independently of the type of risk prediction. An ensemble of all survival curve-based models exhibited a more robust performance, as did a similar ensemble based on binary risk prediction. Patients could be further stratified within clinical risk groups. However, performance still varied across cohorts, indicating limited generalization of all investigated image analysis pipelines, whereas models using clinical data performed robustly on all cohorts.

A self-supervised vision transformer to predict survival from histopathology in renal cell carcinoma

PURPOSE

To develop and validate an interpretable deep learning model to predict overall and disease-specific survival (OS/DSS) in clear cell renal cell carcinoma (ccRCC).

METHODS

Digitised haematoxylin and eosin-stained slides from The Cancer Genome Atlas were used as a training set for a vision transformer (ViT) to extract image features with a self-supervised model called DINO (self-distillation with no labels). Extracted features were used in Cox regression models to prognosticate OS and DSS. Kaplan-Meier for univariable evaluation and Cox regression analyses for multivariable evaluation of the DINO-ViT risk groups were performed for prediction of OS and DSS. For validation, a cohort from a tertiary care centre was used.

RESULTS

A significant risk stratification was achieved in univariable analysis for OS and DSS in the training (n = 443, log rank test, p < 0.01) and validation set (n = 266, p < 0.01). In multivariable analysis, including age, metastatic status, tumour size and grading, the DINO-ViT risk stratification was a significant predictor for OS (hazard ratio [HR] 3.03; 95%-confidence interval [95%-CI] 2.11-4.35; p < 0.01) and DSS (HR 4.90; 95%-CI 2.78-8.64; p < 0.01) in the training set but only for DSS in the validation set (HR 2.31; 95%-CI 1.15-4.65; p = 0.02). DINO-ViT visualisation showed that features were mainly extracted from nuclei, cytoplasm, and peritumoural stroma, demonstrating good interpretability.

CONCLUSION

The DINO-ViT can identify high-risk patients using histological images of ccRCC. This model might improve individual risk-adapted renal cancer therapy in the future.

Domain shifts in dermoscopic skin cancer datasets: Evaluation of essential limitations for clinical translation

The limited ability of Convolutional Neural Networks to generalize to images from previously unseen domains is a major limitation, in particular, for safety-critical clinical tasks such as dermoscopic skin cancer classification. In order to translate CNN-based applications into the clinic, it is essential that they are able to adapt to domain shifts. Such new conditions can arise through the use of different image acquisition systems or varying lighting conditions. In dermoscopy, shifts can also occur as a change in patient age or occurence of rare lesion localizations (e.g. palms). These are not prominently represented in most training datasets and can therefore lead to a decrease in performance. In order to verify the generalizability of classification models in real world clinical settings it is crucial to have access to data which mimics such domain shifts. To our knowledge no dermoscopic image dataset exists where such domain shifts are properly described and quantified. We therefore grouped publicly available images from ISIC archive based on their metadata (e.g. acquisition location, lesion localization, patient age) to generate meaningful domains. To verify that these domains are in fact distinct, we used multiple quantification measures to estimate the presence and intensity of domain shifts. Additionally, we analyzed the performance on these domains with and without an unsupervised domain adaptation technique. We observed that in most of our grouped domains, domain shifts in fact exist. Based on our results, we believe these datasets to be helpful for testing the generalization capabilities of dermoscopic skin cancer classifiers.

Multimodal integration of image, epigenetic and clinical data to predict BRAF mutation status in melanoma

BACKGROUND

In machine learning, multimodal classifiers can provide more generalised performance than unimodal classifiers. In clinical practice, physicians usually also rely on a range of information from different examinations for diagnosis. In this study, we used BRAF mutation status prediction in melanoma as a model system to analyse the contribution of different data types in a combined classifier because BRAF status can be determined accurately by sequencing as the current gold standard, thus nearly eliminating label noise.

METHODS

We trained a deep learning-based classifier by combining individually trained random forests of image, clinical and methylation data to predict BRAF-V600 mutation status in primary and metastatic melanomas of The Cancer Genome Atlas cohort.

RESULTS

With our multimodal approach, we achieved an area under the receiver operating characteristic curve of 0.80, whereas the individual classifiers yielded areas under the receiver operating characteristic curve of 0.63 (histopathologic image data), 0.66 (clinical data) and 0.66 (methylation data) on an independent data set.

CONCLUSIONS

Our combined approach can predict BRAF status to some extent by identifying BRAF-V600 specific patterns at the histologic, clinical and epigenetic levels. The multimodal classifiers have improved generalisability in predicting BRAF mutation status.

Generalizable biomarker prediction from cancer pathology slides with self-supervised deep learning: A retrospective multi-centric study

Deep learning (DL) can predict microsatellite instability (MSI) from routine histopathology slides of colorectal cancer (CRC). However, it is unclear whether DL can also predict other biomarkers with high performance and whether DL predictions generalize to external patient populations. Here, we acquire CRC tissue samples from two large multi-centric studies. We systematically compare six different state-of-the-art DL architectures to predict biomarkers from pathology slides, including MSI and mutations in BRAF, KRAS, NRAS, and PIK3CA. Using a large external validation cohort to provide a realistic evaluation setting, we show that models using self-supervised, attention-based multiple-instance learning consistently outperform previous approaches while offering explainable visualizations of the indicative regions and morphologies. While the prediction of MSI and BRAF mutations reaches a clinical-grade performance, mutation prediction of PIK3CA, KRAS, and NRAS was clinically insufficient.

[The Rise of Artificial Intelligence - High Prediction Accuracy in Early Detection of Pigmented Melanoma]

The incidence of malignant melanoma is increasing worldwide. If detected early, melanoma is highly treatable, so early detection is vital.Skin cancer early detection has improved significantly in recent decades, for example by the introduction of screening in 2008 and dermoscopy. Nevertheless, in particular visual detection of early melanomas remains challenging because they show many morphological overlaps with nevi. Hence, there continues to be a high medical need to further develop methods for early skin cancer detection in order to be able to reliably diagnosemelanomas at a very early stage.Routine diagnostics for melanoma detection include visual whole body inspection, often supplemented by dermoscopy, which can significantly increase the diagnostic accuracy of experienced dermatologists. A procedure that is additionally offered in some practices and clinics is wholebody photography combined with digital dermoscopy for the early detection of malignant melanoma, especially for monitoring high-risk patients.In recent decades, numerous noninvasive adjunctive diagnostic techniques were developed for the examination of suspicious pigmented moles, that may have the potential to allow improved and, in some cases, automated evaluation of these lesions. First, confocal laser microscopy should be mentioned here, as well as electrical impedance spectroscopy, multiphoton laser tomography, multispectral analysis, Raman spectroscopy or optical coherence tomography. These diagnostic techniques usually focus on high sensitivity to avoid malignant melanoma being overlooked. However, this usually implies lower specificity, which may lead to unnecessary excision of benign lesions in screening. Also, some of the procedures are time-consuming and costly, which also limits their applicability in skin cancer screening. In the near future, the use of artificial intelligence might change skin cancer diagnostics in many ways. The most promising approach may be the analysis of routine macroscopic and dermoscopic images by artificial intelligence.For the classification of pigmented skin lesions based on macroscopic and dermoscopic images, artificial intelligence, especially in form of neural networks, has achieved comparable diagnostic accuracies to dermatologists under experimental conditions in numerous studies. In particular, it achieved high accuracies in the binary melanoma/nevus classification task, but it also performed comparably well to dermatologists in multiclass differentiation of various skin diseases. However, proof of the basic applicability and utility of such systems in clinical practice is still pending. Prerequisites that remain to be established to enable translation of such diagnostic systems into dermatological routine are means that allow users to comprehend the system's decisions as well as a uniformly high performance of the algorithms on image data from other hospitals and practices.At present, hints are accumulating that computer-aided diagnosis systems could provide their greatest benefit as assistance systems, since studies indicate that a combination of human and machine achieves the best results. Diagnostic systems based on artificial intelligence are capable of detecting morphological characteristics quickly, quantitatively, objectively and reproducibly, and could thus provide a more objective analytical basis - in addition to medical experience.

Artificial intelligence to predict oncological outcome directly from hematoxylin and eosin-stained slides in urology

INTRODUCTION

Artificial intelligence (AI) has been successfully applied for automatic tumor detection and grading in histopathological image analysis in urologic oncology. The aim of this review was to assess the applicability of these approaches in image-based oncological outcome prediction.

EVIDENCE ACQUISITION

A systematic literature search was conducted using the databases MEDLINE through PubMed and Web of Science up to April 20, 2021. Studies investigating AI approaches to determine the risk of recurrence, metastasis, or survival directly from H&E-stained tissue sections in prostate, renal cell or urothelial carcinoma were included. Characteristics of the AI approach and performance metrics were extracted and summarized. Risk of bias (RoB) was assessed using the PROBAST tool.

EVIDENCE SYNTHESIS

16 studies yielding a total of 6658 patients and reporting on 17 outcome predictions were included. Six studies focused on renal cell, six on prostate and three on urothelial carcinoma while one study investigated renal cell and urothelial carcinoma. Handcrafted feature extraction was used in five, a convolutional neural network (CNN) in six and a deep feature extraction in four studies. One study compared a CNN with handcrafted feature extraction. In seven outcome predictions, a multivariable comparison with clinicopathological parameters was reported. Five of them showed statistically significant hazard ratios for the AI's model's-prediction. However, RoB was high in 15 outcome predictions and unclear in two.

CONCLUSIONS

The included studies are promising but predominantly early pilot studies, therefore primarily highlighting the potential of AI approaches. Additional well-designed studies are needed to assess the actual clinical applicability.

Model soups improve performance of dermoscopic skin cancer classifiers

BACKGROUND

Image-based cancer classifiers suffer from a variety of problems which negatively affect their performance. For example, variation in image brightness or different cameras can already suffice to diminish performance. Ensemble solutions, where multiple model predictions are combined into one, can improve these problems. However, ensembles are computationally intensive and less transparent to practitioners than single model solutions. Constructing model soups, by averaging the weights of multiple models into a single model, could circumvent these limitations while still improving performance.

OBJECTIVE

To investigate the performance of model soups for a dermoscopic melanoma-nevus skin cancer classification task with respect to (1) generalisation to images from other clinics, (2) robustness against small image changes and (3) calibration such that the confidences correspond closely to the actual predictive uncertainties.

METHODS

We construct model soups by fine-tuning pre-trained models on seven different image resolutions and subsequently averaging their weights. Performance is evaluated on a multi-source dataset including holdout and external components.

RESULTS

We find that model soups improve generalisation and calibration on the external component while maintaining performance on the holdout component. For robustness, we observe performance improvements for pertubated test images, while the performance on corrupted test images remains on par.

CONCLUSIONS

Overall, souping for skin cancer classifiers has a positive effect on generalisation, robustness and calibration. It is easy for practitioners to implement and by combining multiple models into a single model, complexity is reduced. This could be an important factor in achieving clinical applicability, as less complexity generally means more transparency.

Explainable artificial intelligence in skin cancer recognition: A systematic review

BACKGROUND

Due to their ability to solve complex problems, deep neural networks (DNNs) are becoming increasingly popular in medical applications. However, decision-making by such algorithms is essentially a black-box process that renders it difficult for physicians to judge whether the decisions are reliable. The use of explainable artificial intelligence (XAI) is often suggested as a solution to this problem. We investigate how XAI is used for skin cancer detection: how is it used during the development of new DNNs? What kinds of visualisations are commonly used? Are there systematic evaluations of XAI with dermatologists or dermatopathologists?

METHODS

Google Scholar, PubMed, IEEE Explore, Science Direct and Scopus were searched for peer-reviewed studies published between January 2017 and October 2021 applying XAI to dermatological images: the search terms histopathological image, whole-slide image, clinical image, dermoscopic image, skin, dermatology, explainable, interpretable and XAI were used in various combinations. Only studies concerned with skin cancer were included.

RESULTS

37 publications fulfilled our inclusion criteria. Most studies (19/37) simply applied existing XAI methods to their classifier to interpret its decision-making. Some studies (4/37) proposed new XAI methods or improved upon existing techniques. 14/37 studies addressed specific questions such as bias detection and impact of XAI on man-machine-interactions. However, only three of them evaluated the performance and confidence of humans using CAD systems with XAI.

CONCLUSION

XAI is commonly applied during the development of DNNs for skin cancer detection. However, a systematic and rigorous evaluation of its usefulness in this scenario is lacking.

Artificial intelligence for detection of microsatellite instability in colorectal cancer-a multicentric analysis of a pre-screening tool for clinical application

BACKGROUND

Microsatellite instability (MSI)/mismatch repair deficiency (dMMR) is a key genetic feature which should be tested in every patient with colorectal cancer (CRC) according to medical guidelines. Artificial intelligence (AI) methods can detect MSI/dMMR directly in routine pathology slides, but the test performance has not been systematically investigated with predefined test thresholds.

METHOD

We trained and validated AI-based MSI/dMMR detectors and evaluated predefined performance metrics using nine patient cohorts of 8343 patients across different countries and ethnicities.

RESULTS

Classifiers achieved clinical-grade performance, yielding an area under the receiver operating curve (AUROC) of up to 0.96 without using any manual annotations. Subsequently, we show that the AI system can be applied as a rule-out test: by using cohort-specific thresholds, on average 52.73% of tumors in each surgical cohort [total number of MSI/dMMR = 1020, microsatellite stable (MSS)/ proficient mismatch repair (pMMR) = 7323 patients] could be identified as MSS/pMMR with a fixed sensitivity at 95%. In an additional cohort of N = 1530 (MSI/dMMR = 211, MSS/pMMR = 1319) endoscopy biopsy samples, the system achieved an AUROC of 0.89, and the cohort-specific threshold ruled out 44.12% of tumors with a fixed sensitivity at 95%. As a more robust alternative to cohort-specific thresholds, we showed that with a fixed threshold of 0.25 for all the cohorts, we can rule-out 25.51% in surgical specimens and 6.10% in biopsies.

INTERPRETATION

When applied in a clinical setting, this means that the AI system can rule out MSI/dMMR in a quarter (with global thresholds) or half of all CRC patients (with local fine-tuning), thereby reducing cost and turnaround time for molecular profiling.

Künstliche Intelligenz auf dem Vormarsch – Hohe Vorhersage-Genauigkeit bei der Früherkennung pigmentierter Melanome

Weltweit steigt die Inzidenz des malignen Melanoms an. Bei frühzeitiger Erkennung ist das Melanom gut behandelbar, eine Früherkennung ist also lebenswichtig.

Die Hautkrebs-Früherkennung hat sich in den letzten Jahrzehnten bspw. durch die Einführung des Screenings im Jahr 2008 und die Dermatoskopie deutlich verbessert. Dennoch bleibt die visuelle Erkennung insbesondere von frühen Melanomen eine Herausforderung, weil diese viele morphologische Überlappungen mit Nävi zeigen. Daher ist der medizinische Bedarf weiterhin hoch, die Methoden zur Hautkrebsfrüherkennung gezielt weiterzuentwickeln, um Melanome bereits in einem sehr frühen Stadium sicher diagnostizieren zu können.

Die Routinediagnostik zur Hautkrebs-Früherkennung umfasst die visuelle Ganzkörperinspektion, oft ergänzt durch die Dermatoskopie, durch die sich die diagnostische Treffsicherheit erfahrener Hautärzte deutlich erhöhen lässt. Ein Verfahren, was in einigen Praxen und Kliniken zusätzlich angeboten wird, ist die kombinierte Ganzkörperfotografie mit der digitalen Dermatoskopie für die Früherkennung maligner Melanome, insbesondere für das Monitoring von Hochrisiko-Patienten.

In den letzten Jahrzenten wurden zahlreiche nicht invasive zusatzdiagnostische Verfahren zur Beurteilung verdächtiger Pigmentmale entwickelt, die das Potenzial haben könnten, eine verbesserte und z. T. automatisierte Bewertung dieser Läsionen zu ermöglichen. In erster Linie ist hier die konfokale Lasermikroskopie zu nennen, ebenso die elektrische Impedanzspektroskopie, die Multiphotonen-Lasertomografie, die Multispektralanalyse, die Raman-Spektroskopie oder die optische Kohärenztomografie. Diese diagnostischen Verfahren fokussieren i. d. R. auf hohe Sensitivität, um zu vermeiden, ein malignes Melanom zu übersehen. Dies bedingt allerdings üblicherweise eine geringere Spezifität, was im Screening zu unnötigen Exzisionen vieler gutartiger Läsionen führen kann. Auch sind einige der Verfahren zeitaufwendig und kostenintensiv, was die Anwendbarkeit im Screening ebenfalls einschränkt.

In naher Zukunft wird insbesondere die Nutzung von künstlicher Intelligenz die Diagnosefindung in vielfältiger Weise verändern. Vielversprechend ist v. a. die Analyse der makroskopischen und dermatoskopischen Routine-Bilder durch künstliche Intelligenz. Für die Klassifizierung von pigmentierten Hautläsionen anhand makroskopischer und dermatoskopischer Bilder erzielte die künstliche Intelligenz v. a. in Form neuronaler Netze unter experimentellen Bedingungen in zahlreichen Studien bereits eine vergleichbare diagnostische Genauigkeit wie Dermatologen. Insbesondere bei der binären Klassifikationsaufgabe Melanom/Nävus erreichte sie hohe Genauigkeiten, doch auch in der Multiklassen-Differenzierung von verschiedenen Hauterkrankungen zeigt sie sich vergleichbar gut wie Dermatologen. Der Nachweis der grundsätzlichen Anwendbarkeit und des Nutzens solcher Systeme in der klinischen Praxis steht jedoch noch aus. Noch zu schaffende Grundvoraussetzungen für die Translation solcher Diagnosesysteme in die dermatologischen Routine sind Möglichkeiten für die Nutzer, die Entscheidungen des Systems nachzuvollziehen, sowie eine gleichbleibend gute Leistung der Algorithmen auf Bilddaten aus fremden Kliniken und Praxen.

Derzeit zeichnet sich ab, dass computergestützte Diagnosesysteme als Assistenzsysteme den größten Nutzen bringen könnten, denn Studien deuten darauf hin, dass eine Kombination von Mensch und Maschine die besten Ergebnisse erzielt. Diagnosesysteme basierend auf künstlicher Intelligenz sind in der Lage, Merkmale schnell, quantitativ, objektiv und reproduzierbar zu erfassen, und könnten somit die Medizin auf eine mathematische Grundlage stellen – zusätzlich zur ärztlichen Erfahrung.

Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review

BACKGROUND

Over the past decade, the development of molecular high-throughput methods (omics) increased rapidly and provided new insights for cancer research. In parallel, deep learning approaches revealed the enormous potential for medical image analysis, especially in digital pathology. Combining image and omics data with deep learning tools may enable the discovery of new cancer biomarkers and a more precise prediction of patient prognosis. This systematic review addresses different multimodal fusion methods of convolutional neural network-based image analyses with omics data, focussing on the impact of data combination on the classification performance.

METHODS

PubMed was screened for peer-reviewed articles published in English between January 2015 and June 2021 by two independent researchers. Search terms related to deep learning, digital pathology, omics, and multimodal fusion were combined.

RESULTS

We identified a total of 11 studies meeting the inclusion criteria, namely studies that used convolutional neural networks for haematoxylin and eosin image analysis of patients with cancer in combination with integrated omics data. Publications were categorised according to their endpoints: 7 studies focused on survival analysis and 4 studies on prediction of cancer subtypes, malignancy or microsatellite instability with spatial analysis.

CONCLUSIONS

Image-based classifiers already show high performances in prognostic and predictive cancer diagnostics. The integration of omics data led to improved performance in all studies described here. However, these are very early studies that still require external validation to demonstrate their generalisability and robustness. Further and more comprehensive studies with larger sample sizes are needed to evaluate performance and determine clinical benefits.

Deep Learning identifies inflamed fat as a risk factor for lymph node metastasis in early colorectal cancer

The spread of early-stage (T1 and T2) adenocarcinomas to loco-regional lymph nodes is a key event in disease progression of colorectal cancer (CRC). The cellular mechanisms behind this event are not completely understood and existing predictive biomarkers are imperfect. Here, we used an end-to-end Deep Learning algorithm to identify risk factors for lymph node metastasis (LNM) status in digitized histopathology slides of the primary CRC and its surrounding tissue. In two large population-based cohorts, we show that this system can predict the presence of more than one LNM in pT2 CRC patients with an area under the receiver operating curve (AUROC) of 0.733 (0.67-0.758) and patients with any LNM with an AUROC of 0.711 (0.597-0.797). Similarly, in pT1 CRC patients, the presence of more than one LNM or any LNM was predictable with an AUROC of 0.733 (0.644-0.778) and 0.567 (0.542-0.597), respectively. Based on these findings, we used the Deep Learning system to guide human pathology experts towards highly predictive regions for LNM in the whole slide images. This hybrid human observer and Deep Learning approach identified inflamed adipose tissue as the highest predictive feature for LNM presence. Our study is a first proof of concept that artificial intelligence (AI) systems may be able to discover potentially new biological mechanisms in cancer progression. Our Deep Learning algorithm is publicly available and can be used for biomarker discovery in any disease setting. This article is protected by copyright. All rights reserved.

Deep learning can predict lymph node status directly from histology in colorectal cancer

BACKGROUND

Lymph node status is a prognostic marker and strongly influences therapeutic decisions in colorectal cancer (CRC).

OBJECTIVES

The objective of the study is to investigate whether image features extracted by a deep learning model from routine histological slides and/or clinical data can be used to predict CRC lymph node metastasis (LNM).

METHODS

Using histological whole slide images (WSIs) of primary tumours of 2431 patients in the DACHS cohort, we trained a convolutional neural network to predict LNM. In parallel, we used clinical data derived from the same cases in logistic regression analyses. Subsequently, the slide-based artificial intelligence predictor (SBAIP) score was included in the regression. WSIs and data from 582 patients of the TCGA cohort were used as the external test set.

RESULTS

On the internal test set, the SBAIP achieved an area under receiver operating characteristic (AUROC) of 71.0%, the clinical classifier achieved an AUROC of 67.0% and a combination of the two classifiers yielded an improvement to 74.1%. Whereas the clinical classifier's performance remained stable on the TCGA set, performance of the SBAIP dropped to an AUROC of 61.2%. Performance of the clinical classifier depended strongly on the T stage.

CONCLUSION

Deep learning-based image analysis may help predict LNM of patients with CRC using routine histological slides. Combination with clinical data such as T stage might be useful. Strategies to increase performance of the SBAIP on external images should be investigated.

Weakly supervised annotation-free cancer detection and prediction of genotype in routine histopathology

Deep learning is a powerful tool in computational pathology: it can be used for tumor detection and for predicting genetic alterations based on histopathology images alone. Conventionally, tumor detection and prediction of genetic alterations are two separate workflows. Newer methods have combined them, but require complex, manually engineered computational pipelines, restricting reproducibility and robustness. To address these issues, we present a new method for simultaneous tumor detection and prediction of genetic alterations: The Slide-Level Assessment Model (SLAM) uses a single off-the-shelf neural network to predict molecular alterations directly from routine pathology slides without any manual annotations, improving upon previous methods by automatically excluding normal and non-informative tissue regions. SLAM requires only standard programming libraries and is conceptually simpler than previous approaches. We have extensively validated SLAM for clinically relevant tasks using two large multicentric cohorts of colorectal cancer patients, Darmkrebs: Chancen der Verhütung durch Screening (DACHS) from Germany and Yorkshire Cancer Research Bowel Cancer Improvement Programme (YCR-BCIP) from the UK. We show that SLAM yields reliable slide-level classification of tumor presence with an area under the receiver operating curve (AUROC) of 0.980 (confidence interval 0.975, 0.984; n = 2,297 tumor and n = 1,281 normal slides). In addition, SLAM can detect microsatellite instability (MSI)/mismatch repair deficiency (dMMR) or microsatellite stability/mismatch repair proficiency with an AUROC of 0.909 (0.888, 0.929; n = 2,039 patients) and BRAF mutational status with an AUROC of 0.821 (0.786, 0.852; n = 2,075 patients). The improvement with respect to previous methods was validated in a large external testing cohort in which MSI/dMMR status was detected with an AUROC of 0.900 (0.864, 0.931; n = 805 patients). In addition, SLAM provides human-interpretable visualization maps, enabling the analysis of multiplexed network predictions by human experts. In summary, SLAM is a new simple and powerful method for computational pathology that could be applied to multiple disease contexts. © 2021 The Authors. The Journal of Pathology published by John Wiley & Sons, Ltd. on behalf of The Pathological Society of Great Britain and Ireland.

Gastrointestinal cancer classification and prognostication from histology using deep learning: Systematic review

BACKGROUND

Gastrointestinal cancers account for approximately 20% of all cancer diagnoses and are responsible for 22.5% of cancer deaths worldwide. Artificial intelligence-based diagnostic support systems, in particular convolutional neural network (CNN)-based image analysis tools, have shown great potential in medical computer vision. In this systematic review, we summarise recent studies reporting CNN-based approaches for digital biomarkers for characterization and prognostication of gastrointestinal cancer pathology.

METHODS

Pubmed and Medline were screened for peer-reviewed papers dealing with CNN-based gastrointestinal cancer analyses from histological slides, published between 2015 and 2020.Seven hundred and ninety titles and abstracts were screened, and 58 full-text articles were assessed for eligibility.

RESULTS

Sixteen publications fulfilled our inclusion criteria dealing with tumor or precursor lesion characterization or prognostic and predictive biomarkers: 14 studies on colorectal or rectal cancer, three studies on gastric cancer and none on esophageal cancer. These studies were categorised according to their end-points: polyp characterization, tumor characterization and patient outcome. Regarding the translation into clinical practice, we identified several studies demonstrating generalization of the classifier with external tests and comparisons with pathologists, but none presenting clinical implementation.

CONCLUSIONS

Results of recent studies on CNN-based image analysis in gastrointestinal cancer pathology are promising, but studies were conducted in observational and retrospective settings. Large-scale trials are needed to assess performance and predict clinical usefulness. Furthermore, large-scale trials are required for approval of CNN-based prediction models as medical devices.

Evaluation of Long-term Clearance Rates of Interventions for Actinic Keratosis: A Systematic Review and Network Meta-analysis

Importance

Multiple interventions are available for the treatment of actinic keratosis (AK). However, most randomized clinical trials and meta-analyses focus on short-term efficacy outcomes.

Objective

To investigate and synthesize the long-term efficacy (≥12 months) of interventions for AK from parallel-arm randomized clinical trials.

Data Sources

Searches in MEDLINE, Embase, and Central were conducted from inception until April 6, 2020. The reference lists of the included studies and pertinent trial registers were hand searched. The study was completed February 27, 2021.

Study Selection

Two reviewers screened the titles and abstracts of 2741 records. Finally, 17 published reports (original studies and follow-up reports) referring to 15 independent randomized clinical trials with an overall sample size of 4252 patients were included.

Data Extraction and Synthesis

Two reviewers independently extracted data on study, patient, and intervention characteristics. Network meta-analysis (NMA) of each outcome was conducted with a frequentist approach. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guidance for NMA was used to assess the certainty of evidence. The revised Cochrane risk-of-bias tool for randomized clinical trials was used to evaluate the methodologic quality.

Main Outcomes and Measures

Participant complete clearance, participant partial clearance, and lesion-specific clearance were the outcomes, with each assessed at least 12 months after the end of treatment.

Results

Data from 15 independent randomized clinical trials including 4252 patients were extracted and synthesized. Ten studies were included in an NMA for the outcome of participant complete clearance, with photodynamic therapy with aminolevulinate (ALA-PDT) showing the most favorable risk ratio (RR) compared with placebo (RR, 8.06; 95% CI, 2.07-31.37; GRADE, moderate), followed by imiquimod, 5% (RR, 5.98; 95% CI, 2.26-15.84; GRADE, very low), photodynamic therapy with methyl aminolevulinate (MAL-PDT) (RR, 5.95; 95% CI, 1.21-29.41; GRADE, low), and cryosurgery (RR, 4.67; 95% CI, 1.36-16.66; GRADE, very low). Similarly, ALA-PDT had the highest RR in the NMA for lesion-specific clearance (RR, 5.08; 95% CI, 2.49-10.33; GRADE, moderate). No NMA was possible for participant partial clearance owing to poor reporting of this outcome.

Conclusions and Relevance

This systematic review and network meta-analysis found that therapy including ALA-PDT, imiquimod, 5%, MAL-PDT, and cryosurgery was associated with significant long-term efficacy in the NMA. This study provides data for a possible use in an evidence-based framework for selecting interventions with sustained lesion clearance.

Deep learning approach to predict sentinel lymph node status directly from routine histology of primary melanoma tumours

AIM

Sentinel lymph node status is a central prognostic factor for melanomas. However, the surgical excision involves some risks for affected patients. In this study, we therefore aimed to develop a digital biomarker that can predict lymph node metastasis non-invasively from digitised H&E slides of primary melanoma tumours.

METHODS

A total of 415 H&E slides from primary melanoma tumours with known sentinel node (SN) status from three German university hospitals and one private pathological practice were digitised (150 SN positive/265 SN negative). Two hundred ninety-one slides were used to train artificial neural networks (ANNs). The remaining 124 slides were used to test the ability of the ANNs to predict sentinel status. ANNs were trained and/or tested on data sets that were matched or not matched between SN-positive and SN-negative cases for patient age, ulceration, and tumour thickness, factors that are known to correlate with lymph node status.

RESULTS

The best accuracy was achieved by an ANN that was trained and tested on unmatched cases (61.8% ± 0.2%) area under the receiver operating characteristic (AUROC). In contrast, ANNs that were trained and/or tested on matched cases achieved (55.0% ± 3.5%) AUROC or less.

CONCLUSION

Our results indicate that the image classifier can predict lymph node status to some, albeit so far not clinically relevant, extent. It may do so by mostly detecting equivalents of factors on histological slides that are already known to correlate with lymph node status. Our results provide a basis for future research with larger data cohorts.

Deep learning approach to predict lymph node metastasis directly from primary tumour histology in prostate cancer

OBJECTIVE

To develop a new digital biomarker based on the analysis of primary tumour tissue by a convolutional neural network (CNN) to predict lymph node metastasis (LNM) in a cohort matched for already established risk factors.

PATIENTS AND METHODS

Haematoxylin and eosin (H&E) stained primary tumour slides from 218 patients (102 N+; 116 N0), matched for Gleason score, tumour size, venous invasion, perineural invasion and age, who underwent radical prostatectomy were selected to train a CNN and evaluate its ability to predict LN status.

RESULTS

With 10 models trained with the same data, a mean area under the receiver operating characteristic curve (AUROC) of 0.68 (95% confidence interval [CI] 0.678-0.682) and a mean balanced accuracy of 61.37% (95% CI 60.05-62.69%) was achieved. The mean sensitivity and specificity was 53.09% (95% CI 49.77-56.41%) and 69.65% (95% CI 68.21-71.1%), respectively. These results were confirmed via cross-validation. The probability score for LNM prediction was significantly higher on image sections from N+ samples (mean [SD] N+ probability score 0.58 [0.17] vs 0.47 [0.15] N0 probability score, P = 0.002). In multivariable analysis, the probability score of the CNN (odds ratio [OR] 1.04 per percentage probability, 95% CI 1.02-1.08; P = 0.04) and lymphovascular invasion (OR 11.73, 95% CI 3.96-35.7; P < 0.001) proved to be independent predictors for LNM.

CONCLUSION

In our present study, CNN-based image analyses showed promising results as a potential novel low-cost method to extract relevant prognostic information directly from H&E histology to predict the LN status of patients with prostate cancer. Our ubiquitously available technique might contribute to an improved LN status prediction.

Artificial Intelligence-based Detection of FGFR3 Mutational Status Directly from Routine Histology in Bladder Cancer: A Possible Preselection for Molecular Testing?

BACKGROUND

Fibroblast growth factor receptor (FGFR) inhibitor treatment has become the first clinically approved targeted therapy in bladder cancer. However, it requires previous molecular testing of each patient, which is costly and not ubiquitously available.

OBJECTIVE

To determine whether an artificial intelligence system is able to predict mutations of the FGFR3 gene directly from routine histology slides of bladder cancer.

DESIGN, SETTING, AND PARTICIPANTS

We trained a deep learning network to detect FGFR3 mutations on digitized slides of muscle-invasive bladder cancers stained with hematoxylin and eosin from the Cancer Genome Atlas (TCGA) cohort (n = 327) and validated the algorithm on the "Aachen" cohort (n = 182; n = 121 pT2-4, n = 34 stroma-invasive pT1, and n = 27 noninvasive pTa tumors).

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

The primary endpoint was the area under the receiver operating curve (AUROC) for mutation detection. Performance of the deep learning system was compared with visual scoring by an uropathologist.

RESULTS AND LIMITATIONS

In the TCGA cohort, FGFR3 mutations were detected with an AUROC of 0.701 (p < 0.0001). In the Aachen cohort, FGFR3 mutants were found with an AUROC of 0.725 (p < 0.0001). When trained on TCGA, the network generalized to the Aachen cohort, and detected FGFR3 mutants with an AUROC of 0.625 (p = 0.0112). A subgroup analysis and histological evaluation found highest accuracy in papillary growth, luminal gene expression subtypes, females, and American Joint Committee on Cancer (AJCC) stage II tumors. In a head-to-head comparison, the deep learning system outperformed the uropathologist in detecting FGFR3 mutants.

CONCLUSIONS

Our computer-based artificial intelligence system was able to detect genetic alterations of the FGFR3 gene of bladder cancer patients directly from histological slides. In the future, this system could be used to preselect patients for further molecular testing. However, analyses of larger, multicenter, muscle-invasive bladder cancer cohorts are now needed in order to validate and extend our findings.

PATIENT SUMMARY

In this report, a computer-based artificial intelligence (AI) system was applied to histological slides to predict genetic alterations of the FGFR3 gene in bladder cancer. We found that the AI system was able to find the alteration with high accuracy. In the future, this system could be used to preselect patients for further molecular testing.

Combining CNN-based histologic whole slide image analysis and patient data to improve skin cancer classification

BACKGROUND

Clinicians and pathologists traditionally use patient data in addition to clinical examination to support their diagnoses.

OBJECTIVES

We investigated whether a combination of histologic whole slides image (WSI) analysis based on convolutional neural networks (CNNs) and commonly available patient data (age, sex and anatomical site of the lesion) in a binary melanoma/nevus classification task could increase the performance compared with CNNs alone.

METHODS

We used 431 WSIs from two different laboratories and analysed the performance of classifiers that used the image or patient data individually or three common fusion techniques. Furthermore, we tested a naive combination of patient data and an image classifier: for cases interpreted as 'uncertain' (CNN output score <0.7), the decision of the CNN was replaced by the decision of the patient data classifier.

RESULTS

The CNN on its own achieved the best performance (mean ± standard deviation of five individual runs) with AUROC of 92.30% ± 0.23% and balanced accuracy of 83.17% ± 0.38%. While the classification performance was not significantly improved in general by any of the tested fusions, naive strategy of replacing the image classifier with the patient data classifier on slides with low output scores improved balanced accuracy to 86.72% ± 0.36%.

CONCLUSION

In most cases, the CNN on its own was so accurate that patient data integration did not provide any benefit. However, incorporating patient data for lesions that were classified by the CNN with low 'confidence' improved balanced accuracy.

Reducing the Impact of Confounding Factors on Skin Cancer Classification via Image Segmentation: Technical Model Study

Background

Studies have shown that artificial intelligence achieves similar or better performance than dermatologists in specific dermoscopic image classification tasks. However, artificial intelligence is susceptible to the influence of confounding factors within images (eg, skin markings), which can lead to false diagnoses of cancerous skin lesions. Image segmentation can remove lesion-adjacent confounding factors but greatly change the image representation.

Objective

The aim of this study was to compare the performance of 2 image classification workflows where images were either segmented or left unprocessed before the subsequent training and evaluation of a binary skin lesion classifier.

Methods

Separate binary skin lesion classifiers (nevus vs melanoma) were trained and evaluated on segmented and unsegmented dermoscopic images. For a more informative result, separate classifiers were trained on 2 distinct training data sets (human against machine [HAM] and International Skin Imaging Collaboration [ISIC]). Each training run was repeated 5 times. The mean performance of the 5 runs was evaluated on a multi-source test set (n=688) consisting of a holdout and an external component.

Results

Our findings showed that when trained on HAM, the segmented classifiers showed a higher overall balanced accuracy (75.6% [SD 1.1%]) than the unsegmented classifiers (66.7% [SD 3.2%]), which was significant in 4 out of 5 runs (P<.001). The overall balanced accuracy was numerically higher for the unsegmented ISIC classifiers (78.3% [SD 1.8%]) than for the segmented ISIC classifiers (77.4% [SD 1.5%]), which was significantly different in 1 out of 5 runs (P=.004).

Conclusions

Image segmentation does not result in overall performance decrease but it causes the beneficial removal of lesion-adjacent confounding factors. Thus, it is a viable option to address the negative impact that confounding factors have on deep learning models in dermatology. However, the segmentation step might introduce new pitfalls, which require further investigations.

Deep learning detects genetic alterations in cancer histology generated by adversarial networks

Deep learning can detect microsatellite instability (MSI) from routine histology images in colorectal cancer (CRC). However, ethical and legal barriers impede sharing of images and genetic data, hampering development of new algorithms for detection of MSI and other biomarkers. We hypothesized that histology images synthesized by conditional generative adversarial networks (CGANs) retain information about genetic alterations. To test this, we developed a 'histology CGAN' which was trained on 256 patients (training cohort 1) and 1457 patients (training cohort 2). The CGAN synthesized 10 000 synthetic MSI and non-MSI images which contained a range of tissue types and were deemed realistic by trained observers in a blinded study. Subsequently, we trained a deep learning detector of MSI on real or synthetic images and evaluated the performance of MSI detection in a held-out set of 142 patients. When trained on real images from training cohort 1, this system achieved an area under the receiver operating curve (AUROC) of 0.742 [0.681, 0.854]. Training on the larger cohort 2 only marginally improved the AUROC to 0.757 [0.707, 0.869]. Training on purely synthetic data resulted in an AUROC of 0.743 [0.658, 0.801]. Training on both real and synthetic data further increased AUROC to 0.777 [0.715, 0.821]. We conclude that synthetic histology images retain information reflecting underlying genetic alterations in colorectal cancer. Using synthetic instead of real images to train deep learning systems yields non-inferior classifiers. This approach can be used to create large shareable data sets or to augment small data sets with rare molecular features. © 2021 The Authors. The Journal of Pathology published by John Wiley & Sons, Ltd. on behalf of The Pathological Society of Great Britain and Ireland.

Effect of a Face-Aging Mobile App-Based Intervention on Skin Cancer Protection Behavior in Secondary Schools in Brazil: A Cluster-Randomized Clinical Trial

Question

Can a face-aging mobile app improve the skin cancer protection behavior of secondary school students?

Findings

In this cluster-randomized clinical trial of 52 school classes with 1573 Brazilian pupils, meaningful improvements were observed in sunscreen use, tanning behavior, and skin self-examinations 3 to 6 months after an intervention using a face-aging app compared with the nonintervention group.

Meaning

Face-aging apps may be useful tools to increase skin cancer protection in adolescents and thereby decrease skin cancer risk.

Importance

Because exposure to UV radiation early in life is an important risk factor for melanoma development, reducing UV exposure in children and adolescents is of paramount importance. New interventions are urgently required.

Objective

To determine the effect of the free face-aging mobile app Sunface on the skin cancer protection behavior of adolescents.

Design, Setting, and Participants

This cluster-randomized clinical trial included a single intervention and a 6-month follow-up from February 1 to November 30, 2018. Randomization was performed on the class level in 52 school classes within 8 public secondary schools (grades 9-12) in Itauna, Southeast Brazil. Data were analyzed from May 1 to October 10, 2019.

Interventions

In a classroom seminar delivered by medical students, adolescents’ selfies were altered by the app to show UV effects on their future faces and were shown in front of their class, accompanied by information about UV protection. Information about relevant parameters was collected via anonymous questionnaires before and 3 and 6 months after the intervention.

Main Outcomes and Measures

The primary end point of the study was the difference in daily sunscreen use at 6 months of follow-up. Secondary end points included the difference in daily sunscreen use at 3 months of follow-up, at least 1 skin self-examination within 6 months, and at least 1 tanning session in the preceding 30 days. All analyses were predefined and based on intention to treat. Cluster effects were taken into account.

Results

Participants included 1573 pupils (812 girls [51.6%] and 761 boys [48.4%]; mean [SD] age, 15.9 [1.3] years) from 52 school classes. Daily sunscreen use increased from 110 of 734 pupils (15.0%) to 139 of 607 (22.9%; P < .001) at the 6-month follow-up in the intervention group. The proportion of pupils performing at least 1 skin self-examination in the intervention group rose from 184 of 734 (25.1%) to 300 of 607 (49.4%; P < .001). Use of tanning decreased from 138 of 734 pupils (18.8%) to 92 of 607 (15.2%; P = .04). No significant changes were observed in the control group. The intervention was more effective for female students (number needed to treat for the primary end point: 8 for girls and 31 for boys).

Conclusions and Relevance

These findings suggest that interventions based on face-aging apps may increase skin cancer protection behavior in Brazilian adolescents. Further studies are required to maximize the effect and to investigate the generalizability of the effects.

Trial Registration

ClinicalTrials.gov Identifier: NCT03178240

The Value of Total Body Photography for the Early Detection of Melanoma: A Systematic Review

Early detection of melanoma is critical to reduce the mortality and morbidity rates of this tumor. Total body photography (TBP) may aid in the early detection of melanoma. To summarize the current evidence on TBP for the early detection of melanoma, we performed a systematic literature search in Medline, Embase, and the Cochrane Central Register of Controlled Trials (CENTRAL) for eligible records up to 6th August 2020. Outcomes of interest included melanoma incidence, incisional and excisional biopsy rates, as well as the Breslow's index of detected tumors. Results from individual studies were described qualitatively. The risks of bias and applicability of the included studies was assessed using the QUADAS-2 checklist. In total, 14 studies published between 1997 and 2020 with an overall sample size of n = 12082 (range 100-4692) were included in the qualitative analysis. Individuals undergoing TBP showed a trend towards a lower Breslow's thickness and a higher proportion of in situ melanomas compared to those without TBP. The number needed to excise one melanoma varied from 3:1 to 14.3:1 and was better for lesions that arose de novo than for tracked ones. The included studies were judged to be of unclear methodological concern with specific deficiencies in the domains "flow and timing" and "reference standard". The use of TBP can improve the early detection of melanoma in high-risk populations. Future studies are warranted to reduce the heterogeneity of phenotypic risk factor definition and the technical implementation of TBP. Artificial intelligence-assisted analysis of images derived from 3-D TBP systems and digital dermoscopy may further improve the early detection of melanoma.

Artificial Intelligence and Its Effect on Dermatologists' Accuracy in Dermoscopic Melanoma Image Classification: Web-Based Survey Study

BACKGROUND

Early detection of melanoma can be lifesaving but this remains a challenge. Recent diagnostic studies have revealed the superiority of artificial intelligence (AI) in classifying dermoscopic images of melanoma and nevi, concluding that these algorithms should assist a dermatologist's diagnoses.

OBJECTIVE

The aim of this study was to investigate whether AI support improves the accuracy and overall diagnostic performance of dermatologists in the dichotomous image-based discrimination between melanoma and nevus.

METHODS

Twelve board-certified dermatologists were presented disjoint sets of 100 unique dermoscopic images of melanomas and nevi (total of 1200 unique images), and they had to classify the images based on personal experience alone (part I) and with the support of a trained convolutional neural network (CNN, part II). Additionally, dermatologists were asked to rate their confidence in their final decision for each image.

RESULTS

While the mean specificity of the dermatologists based on personal experience alone remained almost unchanged (70.6% vs 72.4%; P=.54) with AI support, the mean sensitivity and mean accuracy increased significantly (59.4% vs 74.6%; P=.003 and 65.0% vs 73.6%; P=.002, respectively) with AI support. Out of the 10% (10/94; 95% CI 8.4%-11.8%) of cases where dermatologists were correct and AI was incorrect, dermatologists on average changed to the incorrect answer for 39% (4/10; 95% CI 23.2%-55.6%) of cases. When dermatologists were incorrect and AI was correct (25/94, 27%; 95% CI 24.0%-30.1%), dermatologists changed their answers to the correct answer for 46% (11/25; 95% CI 33.1%-58.4%) of cases. Additionally, the dermatologists' average confidence in their decisions increased when the CNN confirmed their decision and decreased when the CNN disagreed, even when the dermatologists were correct. Reported values are based on the mean of all participants. Whenever absolute values are shown, the denominator and numerator are approximations as every dermatologist ended up rating a varying number of images due to a quality control step.

CONCLUSIONS

The findings of our study show that AI support can improve the overall accuracy of the dermatologists in the dichotomous image-based discrimination between melanoma and nevus. This supports the argument for AI-based tools to aid clinicians in skin lesion classification and provides a rationale for studies of such classifiers in real-life settings, wherein clinicians can integrate additional information such as patient age and medical history into their decisions.

Overdiagnosis of melanoma - causes, consequences and solutions

Malignant melanoma is the skin tumor that causes most deaths in Germany. At an early stage, melanoma is well treatable, so early detection is essential. However, the skin cancer screening program in Germany has been criticized because although melanomas have been diagnosed more frequently since introduction of the program, the mortality from malignant melanoma has not decreased. This indicates that the observed increase in melanoma diagnoses be due to overdiagnosis, i.e. to the detection of lesions that would never have created serious health problems for the patients. One of the reasons is the challenging distinction between some benign and malignant lesions. In addition, there may be lesions that are biologically equivocal, and other lesions that are classified as malignant according to current criteria, but that grow so slowly that they would never have posed a threat to patient's life. So far, these "indolent" melanomas cannot be identified reliably due to a lack of biomarkers. Moreover, the likelihood that an in-situ melanoma will progress to an invasive tumor still cannot be determined with any certainty. When benign lesions are diagnosed as melanoma, the consequences are unnecessary psychological and physical stress for the affected patients and incurred therapy costs. Vice versa, underdiagnoses in the sense of overlooked melanomas can adversely affect patients' prognoses and may necessitate more intense therapies. Novel diagnostic options could reduce the number of over- and underdiagnoses and contribute to more objective diagnoses in borderline cases. One strategy that has yielded promising results in pilot studies is the use of artificial intelligence-based diagnostic tools. However, these applications still await translation into clinical and pathological routine.

A single-cell landscape of high-grade serous ovarian cancer

Malignant abdominal fluid (ascites) frequently develops in women with advanced high-grade serous ovarian cancer (HGSOC) and is associated with drug resistance and a poor prognosis1. To comprehensively characterize the HGSOC ascites ecosystem, we used single-cell RNA sequencing to profile ~11,000 cells from 22 ascites specimens from 11 patients with HGSOC. We found significant inter-patient variability in the composition and functional programs of ascites cells, including immunomodulatory fibroblast sub-populations and dichotomous macrophage populations. We found that the previously described immunoreactive and mesenchymal subtypes of HGSOC, which have prognostic implications, reflect the abundance of immune infiltrates and fibroblasts rather than distinct subsets of malignant cells2. Malignant cell variability was partly explained by heterogeneous copy number alteration patterns or expression of a stemness program. Malignant cells shared expression of inflammatory programs that were largely recapitulated in single-cell RNA sequencing of ~35,000 cells from additionally collected samples, including three ascites, two primary HGSOC tumors and three patient ascites-derived xenograft models. Inhibition of the JAK/STAT pathway, which was expressed in both malignant cells and cancer-associated fibroblasts, had potent anti-tumor activity in primary short-term cultures and patient-derived xenograft models. Our work contributes to resolving the HSGOC landscape3-5 and provides a resource for the development of novel therapeutic approaches.

Artificial Intelligence in Skin Cancer Diagnostics: The Patients' Perspective

Background: Artificial intelligence (AI) has shown promise in numerous experimental studies, particularly in skin cancer diagnostics. Translation of these findings into the clinic is the logical next step. This translation can only be successful if patients' concerns and questions are addressed suitably. We therefore conducted a survey to evaluate the patients' view of artificial intelligence in melanoma diagnostics in Germany, with a particular focus on patients with a history of melanoma.

Participants and Methods: A web-based questionnaire was designed using LimeSurvey, sent by e-mail to university hospitals and melanoma support groups and advertised on social media. The anonymous questionnaire evaluated patients' expectations and concerns toward artificial intelligence in general as well as their attitudes toward different application scenarios. Descriptive analysis was performed with expression of categorical variables as percentages and 95% confidence intervals. Statistical tests were performed to investigate associations between sociodemographic data and selected items of the questionnaire.

Results: 298 individuals (154 with a melanoma diagnosis, 143 without) responded to the questionnaire. About 94% [95% CI = 0.91–0.97] of respondents supported the use of artificial intelligence in medical approaches. 88% [95% CI = 0.85–0.92] would even make their own health data anonymously available for the further development of AI-based applications in medicine. Only 41% [95% CI = 0.35–0.46] of respondents were amenable to the use of artificial intelligence as stand-alone system, 94% [95% CI = 0.92–0.97] to its use as assistance system for physicians. In sub-group analyses, only minor differences were detectable. Respondents with a previous history of melanoma were more amenable to the use of AI applications for early detection even at home. They would prefer an application scenario where physician and AI classify the lesions independently. With respect to AI-based applications in medicine, patients were concerned about insufficient data protection, impersonality and susceptibility to errors, but expected faster, more precise and unbiased diagnostics, less diagnostic errors and support for physicians.

Conclusions: The vast majority of participants exhibited a positive attitude toward the use of artificial intelligence in melanoma diagnostics, especially as an assistance system.

Effects of Label Noise on Deep Learning-Based Skin Cancer Classification

Recent studies have shown that deep learning is capable of classifying dermatoscopic images at least as well as dermatologists. However, many studies in skin cancer classification utilize non-biopsy-verified training images. This imperfect ground truth introduces a systematic error, but the effects on classifier performance are currently unknown. Here, we systematically examine the effects of label noise by training and evaluating convolutional neural networks (CNN) with 804 images of melanoma and nevi labeled either by dermatologists or by biopsy. The CNNs are evaluated on a test set of 384 images by means of 4-fold cross validation comparing the outputs with either the corresponding dermatological or the biopsy-verified diagnosis. With identical ground truths of training and test labels, high accuracies with 75.03% (95% CI: 74.39–75.66%) for dermatological and 73.80% (95% CI: 73.10–74.51%) for biopsy-verified labels can be achieved. However, if the CNN is trained and tested with different ground truths, accuracy drops significantly to 64.53% (95% CI: 63.12–65.94%, p < 0.01) on a non-biopsy-verified and to 64.24% (95% CI: 62.66–65.83%, p < 0.01) on a biopsy-verified test set. In conclusion, deep learning methods for skin cancer classification are highly sensitive to label noise and future work should use biopsy-verified training images to mitigate this problem.

Salivary cortisol levels and anxiety in melanoma patients undergoing sentinel lymph node excision under local anesthesia versus general anesthesia: a prospective study

BACKGROUND

Sentinel lymph node excision (SLNE) can be performed in tumescent local anesthesia (TLA) or general anesthesia (GA). Perioperative cortisol level changes and anxiety are common in surgical interventions and might be influenced by the type of anesthesia. In this study, we intended to determine whether the type of anesthesia impacts the patients' perioperative levels of salivary cortisol (primary outcome) and the feeling of anxiety evaluated by psychological questionnaires (secondary outcome).

METHODS

All melanoma patients of age undergoing SLNE at the University Hospital Essen, Germany, could be included in the study. Exclusion criteria were patients' intake of glucocorticoids or psychotropic medication during the former 6 months, pregnancy, age under 18 years, and BMI ≥ 30 as salivary cortisol levels were reported to be significantly impacted by obesity and might confound results.

RESULTS

In total, 111 melanoma patients undergoing SLNE were included in our prospective study between May 2011 and April 2017 and could choose between TLA or GA. Salivary cortisol levels were measured three times intraoperatively, twice on the third and second preoperative day and twice on the second postoperative day. To assess anxiety, patients completed questionnaires (Hospital Anxiety and Depression Scale (HADS), State-Trait Anxiety Inventory (STAI)) perioperatively. Patients of both groups exhibited comparable baseline levels of cortisol and perioperative anxiety levels. Independent of the type of anesthesia, all patients showed significantly increasing salivary cortisol level from baseline to 30 min before surgery (T3) (TLA: t = 5.07, p < 0.001; GA: t = 3.09, p = 0.006). Post hoc independent t tests showed that the TLA group exhibited significantly higher cortisol concentrations at the beginning of surgery (T4; t = 3.29, p = 0.002) as well as 20 min after incision (T5; t = 277, p = 0.008) compared to the GA group.

CONCLUSIONS

The type of anesthesia chosen for SLNE surgery significantly affects intraoperative cortisol levels in melanoma patients. Further studies are mandatory to evaluate the relevance of endogenous perioperative cortisol levels on the postoperative clinical course.

TRIAL REGISTRATION

German Clinical Trials Register DRKS00003076, registered 1 May 2011.