Multi-modality fusion learning for the automatic diagnosis of optic neuropathy

Multimodality Fusion Strategies in Eye Disease Diagnosis

Multimodality fusion has gained significance in medical applications, particularly in diagnosing challenging diseases like eye diseases, notably diabetic eye diseases that pose risks of vision loss and blindness. Mono-modality eye disease diagnosis proves difficult, often missing crucial disease indicators. In response, researchers advocate multimodality-based approaches to enhance diagnostics. This study is a unique exploration, evaluating three multimodality fusion strategies-early, joint, and late-in conjunction with state-of-the-art convolutional neural network models for automated eye disease binary detection across three datasets: fundus fluorescein angiography, macula, and combination of digital retinal images for vessel extraction, structured analysis of the retina, and high-resolution fundus. Findings reveal the efficacy of each fusion strategy: type 0 early fusion with DenseNet121 achieves an impressive 99.45% average accuracy. InceptionResNetV2 emerges as the top-performing joint fusion architecture with an average accuracy of 99.58%. Late fusion ResNet50V2 achieves a perfect score of 100% across all metrics, surpassing both early and joint fusion. Comparative analysis demonstrates that late fusion ResNet50V2 matches the accuracy of state-of-the-art feature-level fusion model for multiview learning. In conclusion, this study substantiates late fusion as the optimal strategy for eye disease diagnosis compared to early and joint fusion, showcasing its superiority in leveraging multimodal information.

Semi-Supervised 3D Medical Image Segmentation Based on Dual-Task Consistent Joint Learning and Task-Level Regularization

Semi-supervised learning has attracted wide attention from many researchers since its ability to utilize a few data with labels and relatively more data without labels to learn information. Some existing semi-supervised methods for medical image segmentation enforce the regularization of training by implicitly perturbing data or networks to perform the consistency. Most consistency regularization methods focus on data level or network structure level, and rarely of them focus on the task level. It may not directly lead to an improvement in task accuracy. To overcome the problem, this work proposes a semi-supervised dual-task consistent joint learning framework with task-level regularization for 3D medical image segmentation. Two branches are utilized to simultaneously predict the segmented and signed distance maps, and they can learn useful information from each other by constructing a consistency loss function between the two tasks. The segmentation branch learns rich information from both labeled and unlabeled data to strengthen the constraints on the geometric structure of the target. Experimental results on two benchmark datasets show that the proposed method can achieve better performance compared with other state-of-the-art works. It illustrates our method improves segmentation performance by utilizing unlabeled data and consistent regularization.

A Deep Learning Approach for Detecting Colorectal Cancer via Raman Spectra

Objective and Impact Statement. Distinguishing tumors from normal tissues is vital in the intraoperative diagnosis and pathological examination. In this work, we propose to utilize Raman spectroscopy as a novel modality in surgery to detect colorectal cancer tissues. Introduction. Raman spectra can reflect the substance components of the target tissues. However, the feature peak is slight and hard to detect due to environmental noise. Collecting a high-quality Raman spectroscopy dataset and developing effective deep learning detection methods are possibly viable approaches. Methods. First, we collect a large Raman spectroscopy dataset from 26 colorectal cancer patients with the Raman shift ranging from 385 to 1545 cm  - 1 . Second, a one-dimensional residual convolutional neural network (1D-ResNet) architecture is designed to classify the tumor tissues of colorectal cancer. Third, we visualize and interpret the fingerprint peaks found by our deep learning model. Results. Experimental results show that our deep learning method achieves 98.5% accuracy in the detection of colorectal cancer and outperforms traditional methods. Conclusion. Overall, Raman spectra are a novel modality for clinical detection of colorectal cancer. Our proposed ensemble 1D-ResNet could effectively classify the Raman spectra obtained from colorectal tumor tissues or normal tissues.

Classification of Diabetic Foot Ulcers Using Class Knowledge Banks

Diabetic foot ulcers (DFUs) are one of the most common complications of diabetes. Identifying the presence of infection and ischemia in DFU is important for ulcer examination and treatment planning. Recently, the computerized classification of infection and ischaemia of DFU based on deep learning methods has shown promising performance. Most state-of-the-art DFU image classification methods employ deep neural networks, especially convolutional neural networks, to extract discriminative features, and predict class probabilities from the extracted features by fully connected neural networks. In the testing, the prediction depends on an individual input image and trained parameters, where knowledge in the training data is not explicitly utilized. To better utilize the knowledge in the training data, we propose class knowledge banks (CKBs) consisting of trainable units that can effectively extract and represent class knowledge. Each unit in a CKB is used to compute similarity with a representation extracted from an input image. The averaged similarity between units in the CKB and the representation can be regarded as the logit of the considered input. In this way, the prediction depends not only on input images and trained parameters in networks but the class knowledge extracted from the training data and stored in the CKBs. Experimental results show that the proposed method can effectively improve the performance of DFU infection and ischaemia classifications.

Two-Stage Deep Neural Network via Ensemble Learning for Melanoma Classification

Melanoma is a skin disease with a high fatality rate. Early diagnosis of melanoma can effectively increase the survival rate of patients. There are three types of dermoscopy images, malignant melanoma, benign nevis, and seborrheic keratosis, so using dermoscopy images to classify melanoma is an indispensable task in diagnosis. However, early melanoma classification works can only use the low-level information of images, so the melanoma cannot be classified efficiently; the recent deep learning methods mainly depend on a single network, although it can extract high-level features, the poor scale and type of the features limited the results of the classification. Therefore, we need an automatic classification method for melanoma, which can make full use of the rich and deep feature information of images for classification. In this study, we propose an ensemble method that can integrate different types of classification networks for melanoma classification. Specifically, we first use U-net to segment the lesion area of images to generate a lesion mask, thus resize images to focus on the lesion; then, we use five excellent classification models to classify dermoscopy images, and adding squeeze-excitation block (SE block) to models to emphasize the more informative features; finally, we use our proposed new ensemble network to integrate five different classification results. The experimental results prove the validity of our results. We test our method on the ISIC 2017 challenge dataset and obtain excellent results on multiple metrics; especially, we get 0.909 on accuracy. Our classification framework can provide an efficient and accurate way for melanoma classification using dermoscopy images, laying the foundation for early diagnosis and later treatment of melanoma.

CariesNet: a deep learning approach for segmentation of multi-stage caries lesion from oral panoramic X-ray image

Dental caries has been a common health issue throughout the world, which can even lead to dental pulp and root apical inflammation eventually. Timely and effective treatment of dental caries is vital for patients to reduce pain. Traditional caries disease diagnosis methods like naked-eye detection and panoramic radiograph examinations rely on experienced doctors, which may cause misdiagnosis and high time-consuming. To this end, we propose a novel deep learning architecture called CariesNet to delineate different caries degrees from panoramic radiographs. We firstly collect a high-quality panoramic radiograph dataset with 3127 well-delineated caries lesions, including shallow caries, moderate caries, and deep caries. Then we construct CariesNet as a U-shape network with the additional full-scale axial attention module to segment these three caries types from the oral panoramic images. Moreover, we test the segmentation performance between CariesNet and other baseline methods. Experiments show that our method can achieve a mean 93.64% Dice coefficient and 93.61% accuracy in the segmentation of three different levels of caries.