Diagnostic performance of augmented intelligence with 2D and 3D total body photography and convolutional neural networks in a high-risk population for melanoma under real-world conditions: A new era of skin cancer screening?

Effect of patient-contextual skin images in human- and artificial intelligence-based diagnosis of melanoma: Results from the 2020 SIIM-ISIC melanoma classification challenge

BACKGROUND

While the high accuracy of reported AI tools for melanoma detection is promising, the lack of holistic consideration of the patient is often criticized. Along with medical history, a dermatologist would also consider intra-patient nevi patterns, such that nevi that are different from others on a given patient are treated with suspicion.

OBJECTIVE

To evaluate whether patient-contextual lesion-images improves diagnostic accuracy for melanoma in a dermoscopic image-based AI competition and a human reader study.

METHODS

An international online AI competition was held in 2020. The task was to classify dermoscopy images as melanoma or benign lesions. A multi-source dataset of dermoscopy images grouped by patient were provided, and additional use of public datasets was permitted. Competitors were judged on area under the receiver operating characteristic (AUROC) on a private leaderboard. Concurrently, a human reader study was hosted using a subset of the test data. Participants gave their initial diagnosis of an index case (melanoma vs. benign) and were then presented with seven additional lesion-images of that patient before giving a second prediction of the index case. Outcome measures were sensitivity and specificity.

RESULTS

The top 50 of 3308 AI competition entries achieved AUROC scores ranging from 0.943 to 0.949. Few algorithms considered intra-patient lesion patterns and instead most evaluated images independently. The median sensitivity and specificity of human readers before receiving contextual images were 60.0% and 86.7%, and after were 60.0% and 85.7%. Human and AI algorithm performance varied by image source.

CONCLUSION

This study provided an open-source state-of-the-art algorithm for melanoma detection that has been evaluated at multiple centres. Patient-contextual images did not positively impact performance of AI algorithms or human readers. Providing seven contextual images and no total body image may have been insufficient to test the applicability of the intra-patient lesion patterns.

Patient and dermatologists' perspectives on augmented intelligence for melanoma screening: A prospective study

BACKGROUND

Artificial intelligence (AI) shows promising potential to enhance human decision-making as synergistic augmented intelligence (AuI), but requires critical evaluation for skin cancer screening in a real-world setting.

OBJECTIVES

To investigate the perspectives of patients and dermatologists after skin cancer screening by human, artificial and augmented intelligence.

METHODS

A prospective comparative cohort study conducted at the University Hospital Basel included 205 patients (at high-risk of developing melanoma, with resected or advanced disease) and 8 dermatologists. Patients underwent skin cancer screening by a dermatologist with subsequent 2D and 3D total-body photography (TBP). Any suspicious and all melanocytic skin lesions ≥3 mm were imaged with digital dermoscopes and classified by corresponding convolutional neural networks (CNNs). Excisions were performed based on dermatologist's melanoma suspicion, study-defined elevated CNN risk-scores and/or melanoma suspicion by AuI. Subsequently, all patients and dermatologists were surveyed about their experience using questionnaires, including quantification of patient's safety sense following different examinations (subjective safety score (SSS): 0-10).

RESULTS

Most patients believed AI could improve diagnostic performance (95.5%, n = 192/201). In total, 83.4% preferred AuI-based skin cancer screening compared to examination by AI or dermatologist alone (3D-TBP: 61.3%; 2D-TBP: 22.1%, n = 199). Regarding SSS, AuI induced a significantly higher feeling of safety than AI (mean-SSS (mSSS): 9.5 vs. 7.7, p < 0.0001) or dermatologist screening alone (mSSS: 9.5 vs. 9.1, p = 0.001). Most dermatologists expressed high trust in AI examination results (3D-TBP: 90.2%; 2D-TBP: 96.1%, n = 205). In 68.3% of the examinations, dermatologists felt that diagnostic accuracy improved through additional AI-assessment (n = 140/205). Especially beginners (<2 years' dermoscopic experience; 61.8%, n = 94/152) felt AI facilitated their clinical work compared to experts (>5 years' dermoscopic experience; 20.9%, n = 9/43). Contrarily, in divergent risk assessments, only 1.5% of dermatologists trusted a benign CNN-classification more than personal malignancy suspicion (n = 3/205).

CONCLUSIONS

While patients already prefer AuI with 3D-TBP for melanoma recognition, dermatologists continue to rely largely on their own decision-making despite high confidence in AI-results.

TRIAL REGISTRATION

ClinicalTrials.gov (NCT04605822).

Three-Dimensional Total Body Photography, Digital Dermoscopy, and in vivo Reflectance Confocal Microscopy for Follow-Up Assessments of High-Risk Patients for Melanoma: A Prospective, Controlled Study

INTRODUCTION

The combination of total body photography (TBP) and digital dermoscopy (DD) for monitoring patients with a high risk for melanoma can allow early detection of melanoma. This study aimed to examine if the use of three-dimensional (3D)-TBP, DD, and reflectance confocal microscopy (RCM) for regular monitoring of patients at high risk for melanoma was beneficial in comparison to monitoring using dermoscopy alone.

METHODS

The intervention group (IG) underwent 3D-TBP examinations at every visit, along with DD and/or RCM for diagnosis and/or monitoring of pigmented lesions if necessary. The control group (CG) underwent dermoscopy examinations alone.

RESULTS

A total of 600 patients (324 male and 276 female) were followed up over a median period of 23 months (mean, 2.85 visits) in the IG and 22 months (mean, 2.74 visits) in the CG (p = 0.009). DD and RCM monitoring were performed for 166 and 105 lesions, respectively. The number needed to treat (NNT) to diagnose melanoma with RCM was 2.83. The IG included more second primary melanomas (22 vs. 1, p = 0.022) and more excised nevi (186 vs. 10, p < 0.001), which consisted of more dysplastic nevi (137 vs. 2, p < 0.001). Among the melanomas diagnosed in the IG, three were diagnosed directly with RCM, nine with a combination of 3D-TBP and RCM, and 10 with dermoscopy alone.

CONCLUSION

Follow-up assessments with a combination of 3D-TBP, DD, and RCM led to the detection of more melanomas in comparison to the CG. The use of RCM reduced the NNT for melanocytic lesions.

Cutaneous Imaging Techniques

Cutaneous imaging is a central tenant to the practice of dermatology. In this article, the authors explore various noninvasive and invasive skin imaging techniques, as well as the latest deployment of these technologies in conjunction with the use artificial intelligence and machine learning. The authors also provide insight into the benefits, limitations, and challenges around integrating these technologies into dermatologic practice.

Preoperative perineural invasion in rectal cancer based on deep learning radiomics stacking nomogram: A retrospective study

BACKGROUND

The presence of perineural invasion (PNI) in patients with rectal cancer (RC) is associated with significantly poorer outcomes. However, traditional diagnostic modalities have many limitations.

AIM

To develop a deep learning radiomics stacking nomogram model to predict preoperative PNI status in patients with RC.

METHODS

We recruited 303 RC patients and separated them into the training (n = 242) and test (n = 61) datasets on an 8: 2 scale. A substantial number of deep learning and hand-crafted radiomics features of primary tumors were extracted from the arterial and venous phases of computed tomography (CT) images. Four machine learning models were used to predict PNI status in RC patients: support vector machine, k-nearest neighbor, logistic regression, and multilayer perceptron. The stacking nomogram was created by combining optimal machine learning models for the arterial and venous phases with predicting clinical variables.

RESULTS

With an area under the curve (AUC) of 0.964 [95% confidence interval (CI): 0.944-0.983] in the training dataset and an AUC of 0.955 (95%CI: 0.900-0.999) in the test dataset, the stacking nomogram demonstrated strong performance in predicting PNI status. In the training dataset, the AUC of the stacking nomogram was greater than that of the arterial support vector machine (ASVM), venous SVM, and CT-T stage models (P < 0.05). Although the AUC of the stacking nomogram was greater than that of the ASVM in the test dataset, the difference was not particularly noticeable (P = 0.05137).

CONCLUSION

The developed deep learning radiomics stacking nomogram was effective in predicting preoperative PNI status in RC patients.

The Role of Digital Dermoscopy and Follow-Up in the Detection of Amelanotic/Hypomelanotic Melanoma in a Group of High-Risk Patients-Is It Useful?

The prognosis, outcome, and overall survival of melanoma patients improve with early diagnosis which has been facilitated in the past few decades with the introduction of dermoscopy. Further advancements in dermoscopic research, coupled with skilled, educated dermatologists in dermoscopy, have contributed to timely diagnoses. However, detecting amelanotic and hypomelanotic melanoma remains a challenge even to the most skilled experts because these melanomas can mimic inflammatory diseases, numerous benign lesions, and non-melanoma skin cancers. The list of the possible differential diagnoses can be long. Melanoma prediction without the pigment relies only on vascular criteria, and all classic dermoscopic algorithms have failed to fulfill our expectations. In fact, the diagnosis of amelanotic and hypomelanotic melanomas is very challenging, which is why every tool in detecting these lesions is of significance. This review aims to explore the current knowledge and the literature on the possibility of detecting amelanotic/hypomelanotic melanomas using sequential monitoring with digital dermoscopy and total body skin photography.

Classification of melanocytic lesions using direct illumination multispectral imaging

With rising melanoma incidence and mortality, early detection and surgical removal of primary lesions is essential. Multispectral imaging is a new, non-invasive technique that can facilitate skin cancer detection by measuring the reflectance spectra of biological tissues. Currently, incident illumination allows little light to be reflected from deeper skin layers due to high surface reflectance. A pilot study was conducted at the University Hospital Basel to evaluate, whether multispectral imaging with direct light coupling could extract more information from deeper skin layers for more accurate dignity classification of melanocytic lesions. 27 suspicious pigmented lesions from 23 patients were included (6 melanomas, 6 dysplastic nevi, 12 melanocytic nevi, 3 other). Lesions were imaged before excision using a prototype snapshot mosaic multispectral camera with incident and direct illumination with subsequent dignity classification by a pre-trained multispectral image analysis model. Using incident light, a sensitivity of 83.3% and a specificity of 58.8% were achieved compared to dignity as determined by histopathological examination. Direct light coupling resulted in a superior sensitivity of 100% and specificity of 82.4%. Convolutional neural network classification of corresponding red, green, and blue lesion images resulted in 16.7% lower sensitivity (83.3%, 5/6 malignant lesions detected) and 20.9% lower specificity (61.5%) compared to direct light coupling with multispectral image classification. Our results show that incorporating direct light multispectral imaging into the melanoma detection process could potentially increase the accuracy of dignity classification. This newly evaluated illumination method could improve multispectral applications in skin cancer detection. Further larger studies are needed to validate the camera prototype.

The SLICE-3D dataset: 400,000 skin lesion image crops extracted from 3D TBP for skin cancer detection

AI image classification algorithms have shown promising results when applied to skin cancer detection. Most public skin cancer image datasets are comprised of dermoscopic photos and are limited by selection bias, lack of standardization, and lend themselves to development of algorithms that can only be used by skilled clinicians. The SLICE-3D ("Skin Lesion Image Crops Extracted from 3D TBP") dataset described here addresses those concerns and contains images of over 400,000 distinct skin lesions from seven dermatologic centers from around the world. De-identified images were systematically extracted from sensitive 3D Total Body Photographs and are comparable in optical resolution to smartphone images. Algorithms trained on lower quality images could improve clinical workflows and detect skin cancers earlier if deployed in primary care or non-clinical settings, where photos are captured by non-expert physicians or patients. Such a tool could prompt individuals to visit a specialized dermatologist. This dataset circumvents many inherent limitations of prior datasets and may be used to build upon previous applications of skin imaging for cancer detection.

A survey of skin tone assessment in prospective research

Increasing evidence supports reduced accuracy of noninvasive assessment tools, such as pulse oximetry, temperature probes, and AI skin diagnosis benchmarks, in patients with darker skin tones. The FDA is exploring potential strategies for device regulation to improve performance across diverse skin tones by including skin tone criteria. However, there is no consensus about how prospective studies should perform skin tone assessment in order to take this bias into account. There are several tools available to conduct skin tone assessments including administered visual scales (e.g., Fitzpatrick Skin Type, Pantone, Monk Skin Tone) and color measurement tools (e.g., reflectance colorimeters, reflectance spectrophotometers, cameras), although none are consistently used or validated across multiple medical domains. Accurate and consistent skin tone measurement depends on many factors including standardized environments, lighting, body parts assessed, patient conditions, and choice of skin tone assessment tool(s). As race and ethnicity are inadequate proxies for skin tone, these considerations can be helpful in standardizing the effect of skin tone on studies such as AI dermatology diagnoses, pulse oximetry, and temporal thermometers. Skin tone bias in medical devices is likely due to systemic factors that lead to inadequate validation across diverse skin tones. There is an opportunity for researchers to use skin tone assessment methods with standardized considerations in prospective studies of noninvasive tools that may be affected by skin tone. We propose considerations that researchers must take in order to improve device robustness to skin tone bias.

A Narrative Review: Opportunities and Challenges in Artificial Intelligence Skin Image Analyses Using Total Body Photography

Artificial intelligence (AI) algorithms for skin lesion classification have reported accuracy at par with and even outperformance of expert dermatologists in experimental settings. However, the majority of algorithms do not represent real-world clinical approach where skin phenotype and clinical background information are considered. We review the current state of AI for skin lesion classification and present opportunities and challenges when applied to total body photography (TBP). AI in TBP analysis presents opportunities for intrapatient assessment of skin phenotype and holistic risk assessment by incorporating patient-level metadata, although challenges exist for protecting patient privacy in algorithm development and improving explainable AI methods.

Performance of an automated total body mapping algorithm to detect melanocytic lesions of clinical relevance

IMPORTANCE

Total body photography for skin cancer screening is a well-established tool allowing documentation and follow-up of the entire skin surface. Artificial intelligence-based systems are increasingly applied for automated lesion detection and diagnosis.

DESIGN AND PATIENTS

In this prospective observational international multicentre study experienced dermatologists performed skin cancer screenings and identified clinically relevant melanocytic lesions (CRML, requiring biopsy or observation). Additionally, patients received 2D automated total body mapping (ATBM) with automated lesion detection (ATBM master, Fotofinder Systems GmbH). Primary endpoint was the percentage of CRML detected by the bodyscan software. Secondary endpoints included the percentage of correctly identified "new" and "changed" lesions during follow-up examinations.

RESULTS

At baseline, dermatologists identified 1075 CRML in 236 patients and 999 CRML (92.9%) were also detected by the automated software. During follow-up examinations dermatologists identified 334 CRMLs in 55 patients, with 323 (96.7%) also being detected by ATBM with automated lesions detection. Moreover, all new (n = 13) or changed CRML (n = 24) during follow-up were detected by the software. Average time requirements per baseline examination was 14.1 min (95% CI [12.8-15.5]). Subgroup analysis of undetected lesions revealed either technical (e.g. covering by clothing, hair) or lesion-specific reasons (e.g. hypopigmentation, palmoplantar sites).

CONCLUSIONS

ATBM with lesion detection software correctly detected the vast majority of CRML and new or changed CRML during follow-up examinations in a favourable amount of time. Our prospective international study underlines that automated lesion detection in TBP images is feasible, which is of relevance for developing AI-based skin cancer screenings.

A protocol for annotation of total body photography for machine learning to analyze skin phenotype and lesion classification

Introduction

Artificial Intelligence (AI) has proven effective in classifying skin cancers using dermoscopy images. In experimental settings, algorithms have outperformed expert dermatologists in classifying melanoma and keratinocyte cancers. However, clinical application is limited when algorithms are presented with 'untrained' or out-of-distribution lesion categories, often misclassifying benign lesions as malignant, or misclassifying malignant lesions as benign. Another limitation often raised is the lack of clinical context (e.g., medical history) used as input for the AI decision process. The increasing use of Total Body Photography (TBP) in clinical examinations presents new opportunities for AI to perform holistic analysis of the whole patient, rather than a single lesion. Currently there is a lack of existing literature or standards for image annotation of TBP, or on preserving patient privacy during the machine learning process.

Methods

This protocol describes the methods for the acquisition of patient data, including TBP, medical history, and genetic risk factors, to create a comprehensive dataset for machine learning. 500 patients of various risk profiles will be recruited from two clinical sites (Australia and Spain), to undergo temporal total body imaging, complete surveys on sun behaviors and medical history, and provide a DNA sample. This patient-level metadata is applied to image datasets using DICOM labels. Anonymization and masking methods are applied to preserve patient privacy. A two-step annotation process is followed to label skin images for lesion detection and classification using deep learning models. Skin phenotype characteristics are extracted from images, including innate and facultative skin color, nevi distribution, and UV damage. Several algorithms will be developed relating to skin lesion detection, segmentation and classification, 3D mapping, change detection, and risk profiling. Simultaneously, explainable AI (XAI) methods will be incorporated to foster clinician and patient trust. Additionally, a publicly released dataset of anonymized annotated TBP images will be released for an international challenge to advance the development of new algorithms using this type of data.

Conclusion

The anticipated results from this protocol are validated AI-based tools to provide holistic risk assessment for individual lesions, and risk stratification of patients to assist clinicians in monitoring for skin cancer.

Performance of Commercial Dermatoscopic Systems That Incorporate Artificial Intelligence for the Identification of Melanoma in General Practice: A Systematic Review

BACKGROUND

Cutaneous melanoma remains an increasing global public health burden, particularly in fair-skinned populations. Advancing technologies, particularly artificial intelligence (AI), may provide an additional tool for clinicians to help detect malignancies with a more accurate success rate. This systematic review aimed to report the performance metrics of commercially available convolutional neural networks (CNNs) tasked with detecting MM.

METHODS

A systematic literature search was performed using CINAHL, Medline, Scopus, ScienceDirect and Web of Science databases.

RESULTS

A total of 16 articles reporting MM were included in this review. The combined number of melanomas detected was 1160, and non-melanoma lesions were 33,010. The performance of market-approved technology and clinician performance for classifying melanoma was highly heterogeneous, with sensitivity ranging from 16.4 to 100.0%, specificity between 40.0 and 98.3% and accuracy between 44.0 and 92.0%. Less heterogeneity was observed when clinicians worked in unison with AI, with sensitivity ranging between 83.3 and 100.0%, specificity between 83.7 and 87.3%, and accuracy between 86.4 and 86.9%.

CONCLUSION

Instead of focusing on the performance of AI versus clinicians for classifying melanoma, more consistent performance has been obtained when clinicians' work is supported by AI, facilitating management decisions and improving health outcomes.

Artificial Intelligence in Skin Cancer Diagnosis: A Reality Check

The field of skin cancer detection offers a compelling use case for the application of artificial intelligence (AI) within the realm of image-based diagnostic medicine. Through the analysis of large datasets, AI algorithms have the capacity to classify clinical or dermoscopic images with remarkable accuracy. Although these AI-based applications can operate both autonomously and under human supervision, the best results are achieved through a collaborative approach that leverages the expertise of both AI and human experts. However, it is important to note that most studies focus on assessing the diagnostic accuracy of AI in artificial settings rather than in real-world scenarios. Consequently, the practical utility of AI-assisted diagnosis in a clinical environment is still largely unknown. Furthermore, there exists a knowledge gap concerning the optimal use cases and deployment settings for these AI systems as well as the practical challenges that may arise from widespread implementation. This review explores the advantages and limitations of AI in a variety of real-world contexts, with a specific focus on its value to consumers, general practitioners, and dermatologists.