Interpretability-Based Multimodal Convolutional Neural Networks for Skin Lesion Diagnosis

Differentiable self-supervised clustering with intrinsic interpretability

Self-supervised clustering has garnered widespread attention due to its ability to discover latent clustering structures without the need for external labels. However, most existing approaches on self-supervised clustering lack of inherent interpretability in the data clustering process. In this paper, we propose a differentiable self-supervised clustering method with intrinsic interpretability (DSC2I), which provides an interpretable data clustering mechanism by reformulating clustering process based on differentiable programming. To be specific, we first design a differentiable mutual information measurement to explicitly train a neural network with analytical gradients, which avoids variational inference and learns a discriminative and compact representation. Then, an interpretable clustering mechanism based on differentiable programming is devised to transform fundamental clustering process (i.e., minimum intra-cluster distance, maximum inter-cluster distance) into neural networks and convert cluster centers to learnable neural parameters, which allows us to obtain a transparent and interpretable clustering layer. Finally, a unified optimization method is designed, in which the differentiable representation learning and interpretable clustering can be optimized simultaneously in a self-supervised manner. Extensive experiments demonstrate the effectiveness of the proposed DSC2I method compared with 16 clustering approaches.

Systematic review of approaches to detection and classification of skin cancer using artificial intelligence: Development and prospects

In recent years, there has been a significant improvement in the accuracy of the classification of pigmented skin lesions using artificial intelligence algorithms. Intelligent analysis and classification systems are significantly superior to visual diagnostic methods used by dermatologists and oncologists. However, the application of such systems in clinical practice is severely limited due to a lack of generalizability and risks of potential misclassification. Successful implementation of artificial intelligence-based tools into clinicopathological practice requires a comprehensive study of the effectiveness and performance of existing models, as well as further promising areas for potential research development. The purpose of this systematic review is to investigate and evaluate the accuracy of artificial intelligence technologies for detecting malignant forms of pigmented skin lesions. For the study, 10,589 scientific research and review articles were selected from electronic scientific publishers, of which 171 articles were included in the presented systematic review. All selected scientific articles are distributed according to the proposed neural network algorithms from machine learning to multimodal intelligent architectures and are described in the corresponding sections of the manuscript. This research aims to explore automated skin cancer recognition systems, from simple machine learning algorithms to multimodal ensemble systems based on advanced encoder-decoder models, visual transformers (ViT), and generative and spiking neural networks. In addition, as a result of the analysis, future directions of research, prospects, and potential for further development of automated neural network systems for classifying pigmented skin lesions are discussed.

Analysis of dermoscopy images of multi-class for early detection of skin lesions by hybrid systems based on integrating features of CNN models

Skin cancer is one of the most fatal skin lesions, capable of leading to fatality if not detected in its early stages. The characteristics of skin lesions are similar in many of the early stages of skin lesions. The AI in categorizing diverse types of skin lesions significantly contributes to and helps dermatologists to preserve patients' lives. This study introduces a novel approach that capitalizes on the strengths of hybrid systems of Convolutional Neural Network (CNN) models to extract intricate features from dermoscopy images with Random Forest (Rf) and Feed Forward Neural Networks (FFNN) networks, leading to the development of hybrid systems that have superior capabilities early detection of all types of skin lesions. By integrating multiple CNN features, the proposed methods aim to improve the robustness and discriminatory capabilities of the AI system. The dermoscopy images were optimized for the ISIC2019 dataset. Then, the area of the lesions was segmented and isolated from the rest of the image by a Gradient Vector Flow (GVF) algorithm. The first strategy for dermoscopy image analysis for early diagnosis of skin lesions is by the CNN-RF and CNN-FFNN hybrid models. CNN models (DenseNet121, MobileNet, and VGG19) receive a region of interest (skin lesions) and produce highly representative feature maps for each lesion. The second strategy to analyze the area of skin lesions and diagnose their type by means of CNN-RF and CNN-FFNN hybrid models based on the features of the combined CNN models. Hybrid models based on combined CNN features have achieved promising results for diagnosing dermoscopy images of the ISIC 2019 dataset and distinguishing skin cancers from other skin lesions. The Dense-Net121-MobileNet-RF hybrid model achieved an AUC of 95.7%, an accuracy of 97.7%, a precision of 93.65%, a sensitivity of 91.93%, and a specificity of 99.49%.

Novel research and future prospects of artificial intelligence in cancer diagnosis and treatment

Research into the potential benefits of artificial intelligence for comprehending the intricate biology of cancer has grown as a result of the widespread use of deep learning and machine learning in the healthcare sector and the availability of highly specialized cancer datasets. Here, we review new artificial intelligence approaches and how they are being used in oncology. We describe how artificial intelligence might be used in the detection, prognosis, and administration of cancer treatments and introduce the use of the latest large language models such as ChatGPT in oncology clinics. We highlight artificial intelligence applications for omics data types, and we offer perspectives on how the various data types might be combined to create decision-support tools. We also evaluate the present constraints and challenges to applying artificial intelligence in precision oncology. Finally, we discuss how current challenges may be surmounted to make artificial intelligence useful in clinical settings in the future.

A transfer learning-based multimodal neural network combining metadata and multiple medical images for glaucoma type diagnosis

Glaucoma is an acquired optic neuropathy, which can lead to irreversible vision loss. Deep learning(DL), especially convolutional neural networks(CNN), has achieved considerable success in the field of medical image recognition due to the availability of large-scale annotated datasets and CNNs. However, obtaining fully annotated datasets like ImageNet in the medical field is still a challenge. Meanwhile, single-modal approaches remain both unreliable and inaccurate due to the diversity of glaucoma disease types and the complexity of symptoms. In this paper, a new multimodal dataset for glaucoma is constructed and a new multimodal neural network for glaucoma diagnosis and classification (GMNNnet) is proposed aiming to address both of these issues. Specifically, the dataset includes the five most important types of glaucoma labels, electronic medical records and four kinds of high-resolution medical images. The structure of GMNNnet consists of three branches. Branch 1 consisting of convolutional, cyclic and transposition layers processes patient metadata, branch 2 uses Unet to extract features from glaucoma segmentation based on domain knowledge, and branch 3 uses ResFormer to directly process glaucoma medical images.Branch one and branch two are mixed together and then processed by the Catboost classifier. We introduce a gradient-weighted class activation mapping (Grad-GAM) method to increase the interpretability of the model and a transfer learning method for the case of insufficient training data,i.e.,fine-tuning CNN models pre-trained from natural image dataset to medical image tasks. The results show that GMNNnet can better present the high-dimensional information of glaucoma and achieves excellent performance under multimodal data.

Deep learning for pancreatic diseases based on endoscopic ultrasound: A systematic review

BACKGROUND AND AIMS

Endoscopic ultrasonography (EUS) is one of the main examinations in pancreatic diseases. A series of the studies reported the application of deep learning (DL)-assisted EUS in the diagnosis of pancreatic diseases. This systematic review is to evaluate the role of DL algorithms in assisting EUS diagnosis of pancreatic diseases.

METHODS

Literature search were conducted in PubMed and Semantic Scholar databases. Studies that developed DL models for pancreatic diseases based on EUS were eligible for inclusion. This review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines and quality assessment of the included studies was performed according to the IJMEDI checklist.

RESULTS

A total of 23 studies were enrolled into this systematic review, which could be categorized into three groups according to computer vision tasks: classification, detection and segmentation. Seventeen studies focused on the classification task, among which five studies developed simple neural network (NN) models while twelve studies constructed convolutional NN (CNN) models. Three studies were concerned the detection task and five studies were the segmentation task, all based on CNN architectures. All models presented in the studies performed well based on EUS images, videos or voice. According to the IJMEDI checklist, six studies were recognized as high-grade quality, with scores beyond 35 points.

CONCLUSIONS

DL algorithms show great potential in EUS images/videos/voice for pancreatic diseases. However, there is room for improvement such as sample sizes, multi-center cooperation, data preprocessing, model interpretability, and code sharing.

Computer Aided Diagnosis of Melanoma Using Deep Neural Networks and Game Theory: Application on Dermoscopic Images of Skin Lesions

Early detection of melanoma remains a daily challenge due to the increasing number of cases and the lack of dermatologists. Thus, AI-assisted diagnosis is considered as a possible solution for this issue. Despite the great advances brought by deep learning and especially convolutional neural networks (CNNs), computer-aided diagnosis (CAD) systems are still not used in clinical practice. This may be explained by the dermatologist's fear of being misled by a false negative and the assimilation of CNNs to a "black box", making their decision process difficult to understand by a non-expert. Decision theory, especially game theory, is a potential solution as it focuses on identifying the best decision option that maximizes the decision-maker's expected utility. This study presents a new framework for automated melanoma diagnosis. Pursuing the goal of improving the performance of existing systems, our approach also attempts to bring more transparency in the decision process. The proposed framework includes a multi-class CNN and six binary CNNs assimilated to players. The players' strategies is to first cluster the pigmented lesions (melanoma, nevus, and benign keratosis), using the introduced method of evaluating the confidence of the predictions, into confidence level (confident, medium, uncertain). Then, a subset of players has the strategy to refine the diagnosis for difficult lesions with medium and uncertain prediction. We used EfficientNetB5 as the backbone of our networks and evaluated our approach on the public ISIC dataset consisting of 8917 lesions: melanoma (1113), nevi (6705) and benign keratosis (1099). The proposed framework achieved an area under the receiver operating curve (AUROC) of 0.93 for melanoma, 0.96 for nevus and 0.97 for benign keratosis. Furthermore, our approach outperformed existing methods in this task, improving the balanced accuracy (BACC) of the best compared method from 77% to 86%. These results suggest that our framework provides an effective and explainable decision-making strategy. This approach could help dermatologists in their clinical practice for patients with atypical and difficult-to-diagnose pigmented lesions. We also believe that our system could serve as a didactic tool for less experienced dermatologists.

Deep Clustering via Center-Oriented Margin Free-Triplet Loss for Skin Lesion Detection in Highly Imbalanced Datasets

Melanoma is a fatal skin cancer that is curable and has dramatically increasing survival rate when diagnosed at early stages. Learning-based methods hold significant promise for the detection of melanoma from dermoscopic images. However, since melanoma is a rare disease, existing databases of skin lesions predominantly contain highly imbalanced numbers of benign versus malignant samples. In turn, this imbalance introduces substantial bias in classification models due to the statistical dominance of the majority class. To address this issue, we introduce a deep clustering approach based on the latent-space embedding of dermoscopic images. Clustering is achieved using a novel center-oriented margin-free triplet loss (COM-Triplet) enforced on image embeddings from a convolutional neural network backbone. The proposed method aims to form maximally-separated cluster centers as opposed to minimizing classification error, so it is less sensitive to class imbalance. To avoid the need for labeled data, we further propose to implement COM-Triplet based on pseudo-labels generated by a Gaussian mixture model (GMM). Comprehensive experiments show that deep clustering with COM-Triplet loss outperforms clustering with triplet loss, and competing classifiers in both supervised and unsupervised settings.

System for the Recognizing of Pigmented Skin Lesions with Fusion and Analysis of Heterogeneous Data Based on a Multimodal Neural Network

Simple Summary

Skin cancer is one of the most common cancers in humans. This study aims to create a system for recognizing pigmented skin lesions by analyzing heterogeneous data based on a multimodal neural network. Fusing patient statistics and multidimensional visual data allows for finding additional links between dermoscopic images and medical diagnostic results, significantly improving neural network classification accuracy. The use by specialists of the proposed system of neural network recognition of pigmented skin lesions will enhance the efficiency of diagnosis compared to visual diagnostic methods.

Abstract

Today, skin cancer is one of the most common malignant neoplasms in the human body. Diagnosis of pigmented lesions is challenging even for experienced dermatologists due to the wide range of morphological manifestations. Artificial intelligence technologies are capable of equaling and even surpassing the capabilities of a dermatologist in terms of efficiency. The main problem of implementing intellectual analysis systems is low accuracy. One of the possible ways to increase this indicator is using stages of preliminary processing of visual data and the use of heterogeneous data. The article proposes a multimodal neural network system for identifying pigmented skin lesions with a preliminary identification, and removing hair from dermatoscopic images. The novelty of the proposed system lies in the joint use of the stage of preliminary cleaning of hair structures and a multimodal neural network system for the analysis of heterogeneous data. The accuracy of pigmented skin lesions recognition in 10 diagnostically significant categories in the proposed system was 83.6%. The use of the proposed system by dermatologists as an auxiliary diagnostic method will minimize the impact of the human factor, assist in making medical decisions, and expand the possibilities of early detection of skin cancer.