Accuracy of commercially available smartphone applications for the detection of melanoma

Multidisciplinary patient-centered approach to the management of skin cancer

In recent years, new approaches for optimal patient management of cancer have focused on patient-centered care, with integration of tumour-directed treatment and patient-directed supportive and palliative care throughout the disease journey from prevention through screening, diagnosis, treatment, and follow-up. In 2022, at the International Forum of Dermatology (IFD), a scientific session was entirely dedicated to highlight recent developments on patient-centered approaches in skin cancer. An international panel of different groups of participants involved in a patient's journey on the management of skin cancer presented and discussed challenges and barriers that persist in the field of skin cancer prevention and care pathways. Although primary prevention remains a crucial step in the prevention of melanoma, the different surveys performed during the last 20 years demonstrate that the use of sunscreen increases very slowly. Secondary prevention that includes skin screening and diagnostic measures may benefit from the development of digital tools. To improve adherence, patients need accurate, reliable information about their disease and the treatment options, and this type of content that can also be made available on digital tools. Shared decision-making is a hallmark of a patient-centered approach and requires health care providers who can communicate well to patients and their families, underscoring the pivotal role of health care professionals all through the patient journey. Health care providers have a crucial role in supporting patients through their journey in skin cancer. They will benefit from mobile apps and technologies that have been developed recently to address challenges in skin cancer prevention, detection and care, including those that are primarily directed to the patient. However, more peer-reviewed studies are needed as well as regulations to ensure that apps are accurate, reliable, and up to date.

Multicenter prospective blinded melanoma detection study with a handheld elastic scattering spectroscopy device

Background

The elastic scattering spectroscopy (ESS) device (DermaSensor Inc., Miami, FL) is a noninvasive, painless, adjunctive tool for skin cancer detection.

Objectives

To investigate the performance of the ESS device in the detection of melanoma.

Methods

A prospective, investigator-blinded, multicenter study was conducted at 8 United States (US) and 2 Australian sites. All eligible skin lesions were clinically concerning for melanoma, examined with the ESS device, subsequently biopsied according to dermatologists' standard of care, and evaluated with histopathology. A total of 311 participants with 440 lesions were enrolled, including 44 melanomas (63.6% in situ and 36.4% invasive) and 44 severely dysplastic nevi.

Results

The observed sensitivity of the ESS device for melanoma detection was 95.5% (95% CI, 84.5% to 98.8%, 42 of 44 melanomas), and the observed specificity was 32.5% (95% CI, 27.2% to 38.3%). The positive and negative predictive values were 16.0% and 98.1%, respectively.

Limitations

The device was tested in a high-risk population with lesions selected for biopsy based on clinical and dermoscopic assessments of board-certified dermatologists. Most enrolled lesions were pigmented.

Conclusion

The ESS device's high sensitivity and NPV for the detection of melanoma suggest the device may be a useful adjunctive, point-of-care tool for melanoma detection.

Examining labelling guidelines for AI-based software as a medical device: A review and analysis of dermatology mobile applications in Australia

In recent years, there has been a surge in the development of AI-based Software as a Medical Device (SaMD), particularly in visual specialties such as dermatology. In Australia, the Therapeutic Goods Administration (TGA) regulates AI-based SaMD to ensure its safe use. Proper labelling of these devices is crucial to ensure that healthcare professionals and the general public understand how to use them and interpret results accurately. However, guidelines for labelling AI-based SaMD in dermatology are lacking, which may result in products failing to provide essential information about algorithm development and performance metrics. This review examines existing labelling guidelines for AI-based SaMD across visual medical specialties, with a specific focus on dermatology. Common recommendations for labelling are identified and applied to currently available dermatology AI-based SaMD mobile applications to determine usage of these labels. Of the 21 AI-based SaMD mobile applications identified, none fully comply with common labelling recommendations. Results highlight the need for standardized labelling guidelines. Ensuring transparency and accessibility of information is essential for the safe integration of AI into health care and preventing potential risks associated with inaccurate clinical decisions.

Accuracy of an artificial intelligence as a medical device as part of a UK-based skin cancer teledermatology service

Introduction

An artificial intelligence as a medical device (AIaMD), built on convolutional neural networks, has demonstrated high sensitivity for melanoma. To be of clinical value, it needs to safely reduce referral rates. The primary objective of this study was to demonstrate that the AIaMD had a higher rate of correctly classifying lesions that did not need to be referred for biopsy or urgent face-to-face dermatologist review, compared to teledermatology standard of care (SoC), while achieving the same sensitivity to detect malignancy. Secondary endpoints included the sensitivity, specificity, positive and negative predictive values, and number needed to biopsy to identify one case of melanoma or squamous cell carcinoma (SCC) by both the AIaMD and SoC.

Methods

This prospective, single-centre, single-arm, masked, non-inferiority, adaptive, group sequential design trial recruited patients referred to a teledermatology cancer pathway (clinicaltrials.gov NCT04123678). Additional dermoscopic images of each suspicious lesion were taken using a smartphone with a dermoscopic lens attachment. The images were assessed independently by a consultant dermatologist and the AIaMD. The outputs were compared with the final histological or clinical diagnosis.

Results

A total of 700 patients with 867 lesions were recruited, of which 622 participants with 789 lesions were included in the per-protocol (PP) population. In total, 63.3% of PP participants were female; 89.0% identified as white, and the median age was 51 (range 18-95); and all Fitzpatrick skin types were represented including 25/622 (4.0%) type IV-VI skin. A total of 67 malignant lesions were identified, including 8 diagnosed as melanoma. The AIaMD sensitivity was set at 91 and 92.5%, to match the literature-defined clinician sensitivity (91.46%) as closely as possible. In both settings, the AIaMD identified had a significantly higher rate of identifying lesions that did not need a biopsy or urgent referral compared to SoC (p-value = 0.001) with comparable sensitivity for skin cancer.

Discussion

The AIaMD identified significantly more lesions that did not need to be referred for biopsy or urgent face-to-face dermatologist review, compared to teledermatologists. This has the potential to reduce the burden of unnecessary referrals when used as part of a teledermatology service.

Artificial Intelligence in Skin Cancer Diagnosis: A Reality Check

The field of skin cancer detection offers a compelling use case for the application of artificial intelligence (AI) within the realm of image-based diagnostic medicine. Through the analysis of large datasets, AI algorithms have the capacity to classify clinical or dermoscopic images with remarkable accuracy. Although these AI-based applications can operate both autonomously and under human supervision, the best results are achieved through a collaborative approach that leverages the expertise of both AI and human experts. However, it is important to note that most studies focus on assessing the diagnostic accuracy of AI in artificial settings rather than in real-world scenarios. Consequently, the practical utility of AI-assisted diagnosis in a clinical environment is still largely unknown. Furthermore, there exists a knowledge gap concerning the optimal use cases and deployment settings for these AI systems as well as the practical challenges that may arise from widespread implementation. This review explores the advantages and limitations of AI in a variety of real-world contexts, with a specific focus on its value to consumers, general practitioners, and dermatologists.

Position statement of the EADV Artificial Intelligence (AI) Task Force on AI-assisted smartphone apps and web-based services for skin disease

BACKGROUND

As the use of smartphones continues to surge globally, mobile applications (apps) have become a powerful tool for healthcare engagement. Prominent among these are dermatology apps powered by Artificial Intelligence (AI), which provide immediate diagnostic guidance and educational resources for skin diseases, including skin cancer.

OBJECTIVE

This article, authored by the EADV AI Task Force, seeks to offer insights and recommendations for the present and future deployment of AI-assisted smartphone applications (apps) and web-based services for skin diseases with emphasis on skin cancer detection.

METHODS

An initial position statement was drafted on a comprehensive literature review, which was subsequently refined through two rounds of digital discussions and meticulous feedback by the EADV AI Task Force, ensuring its accuracy, clarity and relevance.

RESULTS

Eight key considerations were identified, including risks associated with inaccuracy and improper user education, a decline in professional skills, the influence of non-medical commercial interests, data security, direct and indirect costs, regulatory approval and the necessity of multidisciplinary implementation. Following these considerations, three main recommendations were formulated: (1) to ensure user trust, app developers should prioritize transparency in data quality, accuracy, intended use, privacy and costs; (2) Apps and web-based services should ensure a uniform user experience for diverse groups of patients; (3) European authorities should adopt a rigorous and consistent regulatory framework for dermatology apps to ensure their safety and accuracy for users.

CONCLUSIONS

The utilisation of AI-assisted smartphone apps and web-based services in diagnosing and treating skin diseases has the potential to greatly benefit patients in their dermatology journeys. By prioritising innovation, fostering collaboration and implementing effective regulations, we can ensure the successful integration of these apps into clinical practice.

Auditing the inference processes of medical-image classifiers by leveraging generative AI and the expertise of physicians

The inferences of most machine-learning models powering medical artificial intelligence are difficult to interpret. Here we report a general framework for model auditing that combines insights from medical experts with a highly expressive form of explainable artificial intelligence. Specifically, we leveraged the expertise of dermatologists for the clinical task of differentiating melanomas from melanoma 'lookalikes' on the basis of dermoscopic and clinical images of the skin, and the power of generative models to render 'counterfactual' images to understand the 'reasoning' processes of five medical-image classifiers. By altering image attributes to produce analogous images that elicit a different prediction by the classifiers, and by asking physicians to identify medically meaningful features in the images, the counterfactual images revealed that the classifiers rely both on features used by human dermatologists, such as lesional pigmentation patterns, and on undesirable features, such as background skin texture and colour balance. The framework can be applied to any specialized medical domain to make the powerful inference processes of machine-learning models medically understandable.

Advances in melanoma: epidemiology, diagnosis, and prognosis

Unraveling the multidimensional complexities of melanoma has required concerted efforts by dedicated community of researchers and clinicians battling against this deadly form of skin cancer. Remarkable advances have been made in the realm of epidemiology, classification, diagnosis, and therapy of melanoma. The treatment of advanced melanomas has entered the golden era as targeted personalized therapies have emerged that have significantly altered the mortality rate. A paradigm shift in the approach to melanoma classification, diagnosis, prognosis, and staging is underway, fueled by discoveries of genetic alterations in melanocytic neoplasms. A morphologic clinicopathologic classification of melanoma is expected to be replaced by a more precise molecular based one. As validated, convenient, and cost-effective molecular-based tests emerge, molecular diagnostics will play a greater role in the clinical and histologic diagnosis of melanoma. Artificial intelligence augmented clinical and histologic diagnosis of melanoma is expected to make the process more streamlined and efficient. A more accurate model of prognosis and staging of melanoma is emerging based on molecular understanding melanoma. This contribution summarizes the recent advances in melanoma epidemiology, classification, diagnosis, and prognosis.

Real-world post-deployment performance of a novel machine learning-based digital health technology for skin lesion assessment and suggestions for post-market surveillance

Introduction

Deep Ensemble for Recognition of Malignancy (DERM) is an artificial intelligence as a medical device (AIaMD) tool for skin lesion assessment.

Methods

We report prospective real-world performance from its deployment within skin cancer pathways at two National Health Service hospitals (UK) between July 2021 and October 2022.

Results

A total of 14,500 cases were seen, including patients 18-100 years old with Fitzpatrick skin types I-VI represented. Based on 8,571 lesions assessed by DERM with confirmed outcomes, versions A and B demonstrated very high sensitivity for detecting melanoma (95.0-100.0%) or malignancy (96.0-100.0%). Benign lesion specificity was 40.7-49.4% (DERM-vA) and 70.1-73.4% (DERM-vB). DERM identified 15.0-31.0% of cases as eligible for discharge.

Discussion

We show DERM performance in-line with sensitivity targets and pre-marketing authorisation research, and it reduced the caseload for hospital specialists in two pathways. Based on our experience we offer suggestions on key elements of post-market surveillance for AIaMDs.

Recent Advances in Melanoma Diagnosis and Prognosis Using Machine Learning Methods

PURPOSE OF REVIEW

The purpose was to summarize the current role and state of artificial intelligence and machine learning in the diagnosis and management of melanoma.

RECENT FINDINGS

Deep learning algorithms can identify melanoma from clinical, dermoscopic, and whole slide pathology images with increasing accuracy. Efforts to provide more granular annotation to datasets and to identify new predictors are ongoing. There have been many incremental advances in both melanoma diagnostics and prognostic tools using artificial intelligence and machine learning. Higher quality input data will further improve these models' capabilities.

Artificial intelligence in mobile health for skin cancer diagnostics at home (AIM HIGH): a pilot feasibility study

Background

Artificial intelligence (AI)-based mobile phone apps (mHealth) have the potential to streamline care for suspicious skin lesions in primary care. This study aims to investigate the conditions and feasibility of a study that incorporates an AI-based app in primary care and evaluates its potential impact.

Methods

We conducted a pilot feasibility study from November 22nd, 2021 to June 9th, 2022 with a mixed-methods design on implementation of an AI-based mHealth app for skin cancer detection in three primary care practices in the Netherlands (Rotterdam, Leiden and Katwijk). The primary outcome was the inclusion and successful participation rate of patients and general practitioners (GPs). Secondary outcomes were the reasons, facilitators and barriers for successful participation and the potential impact in both pathways for future sample size calculations. Patients were offered use of an AI-based mHealth app before consulting their GP. GPs assessed the patients blinded and then unblinded to the app. Qualitative data included observations and audio-diaries from patients and GPs and focus-groups and interviews with GPs and GP assistants.

Findings

Fifty patients were included with a median age of 52 years (IQR 33.5-60.3), 64% were female, and 90% had a light skin type. The average patient inclusion rate was 4-6 per GP practice per month and 84% (n = 42) successfully participated. Similarly, in 90% (n = 45 patients) the GPs also successfully completed the study. GPs never changed their working diagnosis, but did change their treatment plan (n = 5) based on the app's assessments. Notably, 54% of patients with a benign skin lesion and low risk rating, indicated that they would be reassured and cancel their GP visit with these results (p < 0.001).

Interpretation

Our findings suggest that studying implementation of an AI-based mHealth app for detection of skin cancer in the hands of patients or as a diagnostic tool used by GPs in primary care appears feasible. Preliminary results indicate potential to further investigate both intended use settings.

Funding

SkinVision B.V.

Dissection of medical AI reasoning processes via physician and generative-AI collaboration

Despite the proliferation and clinical deployment of artificial intelligence (AI)-based medical software devices, most remain black boxes that are uninterpretable to key stakeholders including patients, physicians, and even the developers of the devices. Here, we present a general model auditing framework that combines insights from medical experts with a highly expressive form of explainable AI that leverages generative models, to understand the reasoning processes of AI devices. We then apply this framework to generate the first thorough, medically interpretable picture of the reasoning processes of machine-learning-based medical image AI. In our synergistic framework, a generative model first renders "counterfactual" medical images, which in essence visually represent the reasoning process of a medical AI device, and then physicians translate these counterfactual images to medically meaningful features. As our use case, we audit five high-profile AI devices in dermatology, an area of particular interest since dermatology AI devices are beginning to achieve deployment globally. We reveal how dermatology AI devices rely both on features used by human dermatologists, such as lesional pigmentation patterns, as well as multiple, previously unreported, potentially undesirable features, such as background skin texture and image color balance. Our study also sets a precedent for the rigorous application of explainable AI to understand AI in any specialized domain and provides a means for practitioners, clinicians, and regulators to uncloak AI's powerful but previously enigmatic reasoning processes in a medically understandable way.

Diagnostics Using Non-Invasive Technologies in Dermatological Oncology

The growing incidence of skin cancer, with its associated mortality and morbidity, has in recent years led to the developing of new non-invasive technologies, which allow an earlier and more accurate diagnosis. Some of these, such as digital photography, 2D and 3D total-body photography and dermoscopy are now widely used and others, such as reflectance confocal microscopy and optical coherence tomography, are limited to a few academic and referral skin cancer centers because of their cost or the long training period required. Health care professionals involved in the treatment of patients with skin cancer need to know the implications and benefits of new non-invasive technologies for dermatological oncology. In this article we review the characteristics and usability of the main diagnostic imaging methods available today.

Over-Detection of Melanoma-Suspect Lesions by a CE-Certified Smartphone App: Performance in Comparison to Dermatologists, 2D and 3D Convolutional Neural Networks in a Prospective Data Set of 1204 Pigmented Skin Lesions Involving Patients' Perception

Simple Summary

Early detection and resection of cutaneous melanoma are crucial for a good prognosis. However, visual distinction of early melanomas from benign nevi remains challenging. New artificial intelligence-based approaches for risk stratification of pigmented skin lesions provide screening methods for laypersons with increasing use of smartphone applications (apps). Our study aims to prospectively investigate the diagnostic accuracy of a CE-certified smartphone app, SkinVision^®, in melanoma recognition. Based on classification into three different risk scores, the app provides a recommendation to consult a dermatologist. In addition, both patients’ and dermatologists’ perspectives towards AI-based mobile health apps were evaluated. We observed that the app classified a significantly higher number of lesions as high-risk than dermatologists, which would have led to a clinically harmful number of unnecessary excisions. The diagnostic performance of the app in dichotomous classification of 1204 pigmented skin lesions (risk classification for nevus vs. melanoma) remained below advertised rates with low sensitivity (41.3–83.3%) and specificity (60.0–82.9%). The confidence in the app was low among both patients and dermatologists, and no patient favored an assessment by the app alone. Although smartphone apps are a potential medium for increasing awareness of melanoma screening in the lay population, they should be evaluated for certification with prospective real-world evidence.

Abstract

The exponential increase in algorithm-based mobile health (mHealth) applications (apps) for melanoma screening is a reaction to a growing market. However, the performance of available apps remains to be investigated. In this prospective study, we investigated the diagnostic accuracy of a class 1 CE-certified smartphone app in melanoma risk stratification and its patient and dermatologist satisfaction. Pigmented skin lesions ≥ 3 mm and any suspicious smaller lesions were assessed by the smartphone app SkinVision^® (SkinVision^® B.V., Amsterdam, the Netherlands, App-Version 6.8.1), 2D FotoFinder ATBM^® master (FotoFinder ATBM^® Systems GmbH, Bad Birnbach, Germany, Version 3.3.1.0), 3D Vectra^® WB360 (Canfield Scientific, Parsippany, NJ, USA, Version 4.7.1) total body photography (TBP) devices, and dermatologists. The high-risk score of the smartphone app was compared with the two gold standards: histological diagnosis, or if not available, the combination of dermatologists’, 2D and 3D risk assessments. A total of 1204 lesions among 114 patients (mean age 59 years; 51% females (55 patients at high-risk for developing a melanoma, 59 melanoma patients)) were included. The smartphone app’s sensitivity, specificity, and area under the receiver operating characteristics (AUROC) varied between 41.3–83.3%, 60.0–82.9%, and 0.62–0.72% according to two study-defined reference standards. Additionally, all patients and dermatologists completed a newly created questionnaire for preference and trust of screening type. The smartphone app was rated as trustworthy by 36% (20/55) of patients at high-risk for melanoma, 49% (29/59) of melanoma patients, and 8.8% (10/114) of dermatologists. Most of the patients rated the 2D TBP imaging (93% (51/55) resp. 88% (52/59)) and the 3D TBP imaging (91% (50/55) resp. 90% (53/59)) as trustworthy. A skin cancer screening by combination of dermatologist and smartphone app was favored by only 1.8% (1/55) resp. 3.4% (2/59) of the patients; no patient preferred an assessment by a smartphone app alone. The diagnostic accuracy in clinical practice was not as reliable as previously advertised and the satisfaction with smartphone apps for melanoma risk stratification was scarce. MHealth apps might be a potential medium to increase awareness for melanoma screening in the lay population, but healthcare professionals and users should be alerted to the potential harm of over-detection and poor performance. In conclusion, we suggest further robust evidence-based evaluation before including market-approved apps in self-examination for public health benefits.