Novel Transfer Learning Approach for Medical Imaging with Limited Labeled Data

DeepAlienorNet: A deep learning model to extract clinical features from colour fundus photography in age-related macular degeneration

OBJECTIVE

This study aimed to develop a deep learning (DL) model, named 'DeepAlienorNet', to automatically extract clinical signs of age-related macular degeneration (AMD) from colour fundus photography (CFP).

METHODS AND ANALYSIS

The ALIENOR Study is a cohort of French individuals 77 years of age or older. A multi-label DL model was developed to grade the presence of 7 clinical signs: large soft drusen (>125 μm), intermediate soft (63-125 μm), large area of soft drusen (total area >500 μm), presence of central soft drusen (large or intermediate), hyperpigmentation, hypopigmentation, and advanced AMD (defined as neovascular or atrophic AMD). Prediction performances were evaluated using cross-validation and the expert human interpretation of the clinical signs as the ground truth.

RESULTS

A total of 1178 images were included in the study. Averaging the 7 clinical signs' detection performances, DeepAlienorNet achieved an overall sensitivity, specificity, and AUROC of 0.77, 0.83, and 0.87, respectively. The model demonstrated particularly strong performance in predicting advanced AMD and large areas of soft drusen. It can also generate heatmaps, highlighting the relevant image areas for interpretation.

CONCLUSION

DeepAlienorNet demonstrates promising performance in automatically identifying clinical signs of AMD from CFP, offering several notable advantages. Its high interpretability reduces the black box effect, addressing ethical concerns. Additionally, the model can be easily integrated to automate well-established and validated AMD progression scores, and the user-friendly interface further enhances its usability. The main value of DeepAlienorNet lies in its ability to assist in precise severity scoring for further adapted AMD management, all while preserving interpretability.

Comparison of deep learning architectures for predicting amyloid positivity in Alzheimer's disease, mild cognitive impairment, and healthy aging, from T1-weighted brain structural MRI

Abnormal β-amyloid (Aβ) accumulation in the brain is an early indicator of Alzheimer's disease (AD) and is typically assessed through invasive procedures such as PET (positron emission tomography) or CSF (cerebrospinal fluid) assays. As new anti-Alzheimer's treatments can now successfully target amyloid pathology, there is a growing interest in predicting Aβ positivity (Aβ+) from less invasive, more widely available types of brain scans, such as T1-weighted (T1w) MRI. Here we compare multiple approaches to infer Aβ + from standard anatomical MRI: (1) classical machine learning algorithms, including logistic regression, XGBoost, and shallow artificial neural networks, (2) deep learning models based on 2D and 3D convolutional neural networks (CNNs), (3) a hybrid ANN-CNN, combining the strengths of shallow and deep neural networks, (4) transfer learning models based on CNNs, and (5) 3D Vision Transformers. All models were trained on paired MRI/PET data from 1,847 elderly participants (mean age: 75.1 yrs. ± 7.6SD; 863 females/984 males; 661 healthy controls, 889 with mild cognitive impairment (MCI), and 297 with Dementia), scanned as part of the Alzheimer's Disease Neuroimaging Initiative. We evaluated each model's balanced accuracy and F1 scores. While further tests on more diverse data are warranted, deep learning models trained on standard MRI showed promise for estimating Aβ + status, at least in people with MCI. This may offer a potential screening option before resorting to more invasive procedures.

Improving the Generalizability and Performance of an Ultrasound Deep Learning Model Using Limited Multicenter Data for Lung Sliding Artifact Identification

Deep learning (DL) models for medical image classification frequently struggle to generalize to data from outside institutions. Additional clinical data are also rarely collected to comprehensively assess and understand model performance amongst subgroups. Following the development of a single-center model to identify the lung sliding artifact on lung ultrasound (LUS), we pursued a validation strategy using external LUS data. As annotated LUS data are relatively scarce-compared to other medical imaging data-we adopted a novel technique to optimize the use of limited external data to improve model generalizability. Externally acquired LUS data from three tertiary care centers, totaling 641 clips from 238 patients, were used to assess the baseline generalizability of our lung sliding model. We then employed our novel Threshold-Aware Accumulative Fine-Tuning (TAAFT) method to fine-tune the baseline model and determine the minimum amount of data required to achieve predefined performance goals. A subgroup analysis was also performed and Grad-CAM++ explanations were examined. The final model was fine-tuned on one-third of the external dataset to achieve 0.917 sensitivity, 0.817 specificity, and 0.920 area under the receiver operator characteristic curve (AUC) on the external validation dataset, exceeding our predefined performance goals. Subgroup analyses identified LUS characteristics that most greatly challenged the model's performance. Grad-CAM++ saliency maps highlighted clinically relevant regions on M-mode images. We report a multicenter study that exploits limited available external data to improve the generalizability and performance of our lung sliding model while identifying poorly performing subgroups to inform future iterative improvements. This approach may contribute to efficiencies for DL researchers working with smaller quantities of external validation data.

Application of convolutional neural networks in medical images: a bibliometric analysis

Background

In the field of medical imaging, the rapid rise of convolutional neural networks (CNNs) has presented significant opportunities for conserving healthcare resources. However, with the wide spread application of CNNs, several challenges have emerged, such as enormous data annotation costs, difficulties in ensuring user privacy and security, weak model interpretability, and the consumption of substantial computational resources. The fundamental challenge lies in optimizing and seamlessly integrating CNN technology to enhance the precision and efficiency of medical diagnosis.

Methods

This study sought to provide a comprehensive bibliometric overview of current research on the application of CNNs in medical imaging. Initially, bibliometric methods were used to calculate the frequency statistics, and perform the cluster analysis and the co-citation analysis of countries, institutions, authors, keywords, and references. Subsequently, the latent Dirichlet allocation (LDA) method was employed for the topic modeling of the literature. Next, an in-depth analysis of the topics was conducted, and the topics in the medical field, technical aspects, and trends in topic evolution were summarized. Finally, by integrating the bibliometrics and LDA results, the developmental trajectory, milestones, and future directions in this field were outlined.

Results

A data set containing 6,310 articles in this field published from January 2013 to December 2023 was complied. With a total of 55,538 articles, the United States led in terms of the citation count, while in terms of the publication volume, China led with 2,385 articles. Harvard University emerged as the most influential institution, boasting an average of 69.92 citations per article. Within the realm of CNNs, residual neural network (ResNet) and U-Net stood out, receiving 1,602 and 1,419 citations, respectively, which highlights the significant attention these models have received. The impact of coronavirus disease 2019 (COVID-19) was unmistakable, as reflected by the publication of 597 articles, making it a focal point of research. Additionally, among various disease topics, with 290 articles, brain-related research was the most prevalent. Computed tomography (CT) imaging dominated the research landscape, representing 73% of the 30 different topics.

Conclusions

Over the past 11 years, CNN-related research in medical imaging has grown exponentially. The findings of the present study provide insights into the field's status and research hotspots. In addition, this article meticulously chronicled the development of CNNs and highlighted key milestones, starting with LeNet in 1989, followed by a challenging 20-year exploration period, and culminating in the breakthrough moment with AlexNet in 2012. Finally, this article explored recent advancements in CNN technology, including semi-supervised learning, efficient learning, trustworthy artificial intelligence (AI), and federated learning methods, and also addressed challenges related to data annotation costs, diagnostic efficiency, model performance, and data privacy.

TDASD: Generating medically significant fine-grained lung adenocarcinoma nodule CT images based on stable diffusion models with limited sample size

BACKGROUND AND OBJECTIVES

Spread through air spaces (STAS) is an emerging lung cancer infiltration pattern. Predicting its spread through CT scans is crucial. However, limited STAS data makes this prediction task highly challenging. Stable diffusion is capable of generating more diverse and higher-quality images compared to traditional GAN models, surpassing the dominating GAN family models in image synthesis over the past few years. To alleviate the issue of limited STAS data, we propose a method TDASD based on stable diffusion, which is able to generate high-resolution CT images of pulmonary nodules corresponding to specific nodular signs according to the medical professionals.

METHODS

First, we apply the stable diffusion method for fine-tuning training on publicly available lung datasets. Subsequently, we extract nodules from our hospital's lung adenocarcinoma data and apply slight rotations to the original nodule CT slices within a reasonable range before undergoing another round of fine-tuning through stable diffusion. Finally, employing DDIM and Ksample sampling methods, we generate lung adenocarcinoma nodule CT images with signs based on prompts provided by doctors. The method we propose not only safeguards patient privacy but also enhances the diversity of medical images under limited data conditions. Furthermore, our approach to generating medical images incorporates medical knowledge, resulting in images that exhibit pertinent medical features, thus holding significant value in tumor discrimination diagnostics.

RESULTS

Our TDASD method has the capability to generate medically meaningful images by optimizing input prompts based on medical descriptions provided by experts. The images generated by our method can improve the model's classification accuracy. Furthermore, Utilizing solely the data generated by our method for model training, the test results on the original real dataset reveal an accuracy rate that closely aligns with the testing accuracy achieved through training on real data.

CONCLUSIONS

The method we propose not only safeguards patient privacy but also enhances the diversity of medical images under limited data conditions. Furthermore, our approach to generating medical images incorporates medical knowledge, resulting in images that exhibit pertinent medical features, thus holding significant value in tumor discrimination diagnostics.

Deep Transfer Learning Using Real-World Image Features for Medical Image Classification, with a Case Study on Pneumonia X-ray Images

Deep learning has profoundly influenced various domains, particularly medical image analysis. Traditional transfer learning approaches in this field rely on models pretrained on domain-specific medical datasets, which limits their generalizability and accessibility. In this study, we propose a novel framework called real-world feature transfer learning, which utilizes backbone models initially trained on large-scale general-purpose datasets such as ImageNet. We evaluate the effectiveness and robustness of this approach compared to models trained from scratch, focusing on the task of classifying pneumonia in X-ray images. Our experiments, which included converting grayscale images to RGB format, demonstrate that real-world-feature transfer learning consistently outperforms conventional training approaches across various performance metrics. This advancement has the potential to accelerate deep learning applications in medical imaging by leveraging the rich feature representations learned from general-purpose pretrained models. The proposed methodology overcomes the limitations of domain-specific pretrained models, thereby enabling accelerated innovation in medical diagnostics and healthcare. From a mathematical perspective, we formalize the concept of real-world feature transfer learning and provide a rigorous mathematical formulation of the problem. Our experimental results provide empirical evidence supporting the effectiveness of this approach, laying the foundation for further theoretical analysis and exploration. This work contributes to the broader understanding of feature transferability across domains and has significant implications for the development of accurate and efficient models for medical image analysis, even in resource-constrained settings.

Trustworthy deep learning framework for the detection of abnormalities in X-ray shoulder images

Musculoskeletal conditions affect an estimated 1.7 billion people worldwide, causing intense pain and disability. These conditions lead to 30 million emergency room visits yearly, and the numbers are only increasing. However, diagnosing musculoskeletal issues can be challenging, especially in emergencies where quick decisions are necessary. Deep learning (DL) has shown promise in various medical applications. However, previous methods had poor performance and a lack of transparency in detecting shoulder abnormalities on X-ray images due to a lack of training data and better representation of features. This often resulted in overfitting, poor generalisation, and potential bias in decision-making. To address these issues, a new trustworthy DL framework has been proposed to detect shoulder abnormalities (such as fractures, deformities, and arthritis) using X-ray images. The framework consists of two parts: same-domain transfer learning (TL) to mitigate imageNet mismatch and feature fusion to reduce error rates and improve trust in the final result. Same-domain TL involves training pre-trained models on a large number of labelled X-ray images from various body parts and fine-tuning them on the target dataset of shoulder X-ray images. Feature fusion combines the extracted features with seven DL models to train several ML classifiers. The proposed framework achieved an excellent accuracy rate of 99.2%, F1Score of 99.2%, and Cohen's kappa of 98.5%. Furthermore, the accuracy of the results was validated using three visualisation tools, including gradient-based class activation heat map (Grad CAM), activation visualisation, and locally interpretable model-independent explanations (LIME). The proposed framework outperformed previous DL methods and three orthopaedic surgeons invited to classify the test set, who obtained an average accuracy of 79.1%. The proposed framework has proven effective and robust, improving generalisation and increasing trust in the final results.

Improving Tumor-Infiltrating Lymphocytes Score Prediction in Breast Cancer with Self-Supervised Learning

Tumor microenvironment (TME) plays a pivotal role in immuno-oncology, which investigates the intricate interactions between tumors and the human immune system. Specifically, tumor-infiltrating lymphocytes (TILs) are crucial biomarkers for evaluating the prognosis of breast cancer patients and have the potential to refine immunotherapy precision and accurately identify tumor cells in specific cancer types. In this study, we conducted tissue segmentation and lymphocyte detection tasks to predict TIL scores by employing self-supervised learning (SSL) model-based approaches capable of addressing limited labeling data issues. Our experiments showed a 1.9% improvement in tissue segmentation and a 2% improvement in lymphocyte detection over the ImageNet pre-training model. Using these SSL-based models, we achieved a TIL score of 0.718 with a 4.4% improvement. In particular, when trained with only 10% of the entire dataset, the SwAV pre-trained model exhibited a superior performance over other models. Our work highlights improved tissue segmentation and lymphocyte detection using the SSL model with less labeled data for TIL score prediction.

Advancing diabetes prediction with a progressive self-transfer learning framework for discrete time series data

Although diabetes mellitus is a complex and pervasive disease, most studies to date have focused on individual features, rather than considering the complexities of multivariate, multi-instance, and time-series data. In this study, we developed a novel diabetes prediction model that incorporates these complex data types. We applied advanced techniques of data imputation (bidirectional recurrent imputation for time series; BRITS) and feature selection (the least absolute shrinkage and selection operator; LASSO). Additionally, we utilized self-supervised algorithms and transfer learning to address the common issues with medical datasets, such as irregular data collection and sparsity. We also proposed a novel approach for discrete time-series data preprocessing, utilizing both shifting and rolling time windows and modifying time resolution. Our study evaluated the performance of a progressive self-transfer network for predicting diabetes, which demonstrated a significant improvement in metrics compared to non-progressive and single self-transfer prediction tasks, particularly in AUC, recall, and F1 score. These findings suggest that the proposed approach can mitigate accumulated errors and reflect temporal information, making it an effective tool for accurate diagnosis and disease management. In summary, our study highlights the importance of considering the complexities of multivariate, multi-instance, and time-series data in diabetes prediction.

Diagnosis of Salivary Gland Tumors Using Transfer Learning with Fine-Tuning and Gradual Unfreezing

Ultrasound is the primary tool for evaluating salivary gland tumors (SGTs); however, tumor diagnosis currently relies on subjective features. This study aimed to establish an objective ultrasound diagnostic method using deep learning. We collected 446 benign and 223 malignant SGT ultrasound images in the training/validation set and 119 benign and 44 malignant SGT ultrasound images in the testing set. We trained convolutional neural network (CNN) models from scratch and employed transfer learning (TL) with fine-tuning and gradual unfreezing to classify malignant and benign SGTs. The diagnostic performances of these models were compared. By utilizing the pretrained ResNet50V2 with fine-tuning and gradual unfreezing, we achieved a 5-fold average validation accuracy of 0.920. The diagnostic performance on the testing set demonstrated an accuracy of 89.0%, a sensitivity of 81.8%, a specificity of 91.6%, a positive predictive value of 78.3%, and a negative predictive value of 93.2%. This performance surpasses that of other models in our study. The corresponding Grad-CAM visualizations were also presented to provide explanations for the diagnosis. This study presents an effective and objective ultrasound method for distinguishing between malignant and benign SGTs, which could assist in preoperative evaluation.

CervixFormer: A Multi-scale swin transformer-Based cervical pap-Smear WSI classification framework

BACKGROUND AND OBJECTIVES

Cervical cancer affects around 0.5 million women per year, resulting in over 0.3 million fatalities. Therefore, repetitive screening for cervical cancer is of utmost importance. Computer-assisted diagnosis is key for scaling up cervical cancer screening. Current recognition algorithms, however, perform poorly on the whole-slide image (WSI) analysis, fail to generalize for different staining methods and on uneven distribution for subtype imaging, and provide sub-optimal clinical-level interpretations. Herein, we developed CervixFormer-an end-to-end, multi-scale swin transformer-based adversarial ensemble learning framework to assess pre-cancerous and cancer-specific cervical malignant lesions on WSIs.

METHODS

The proposed framework consists of (1) a self-attention generative adversarial network (SAGAN) for generating synthetic images during patch-level training to address the class imbalanced problems; (2) a multi-scale transformer-based ensemble learning method for cell identification at various stages, including atypical squamous cells (ASC) and atypical squamous cells of undetermined significance (ASCUS), which have not been demonstrated in previous studies; and (3) a fusion model for concatenating ensemble-based results and producing final outcomes.

RESULTS

In the evaluation, the proposed method is first evaluated on a private dataset of 717 annotated samples from six classes, obtaining a high recall and precision of 0.940 and 0.934, respectively, in roughly 1.2 minutes. To further examine the generalizability of CervixFormer, we evaluated it on four independent, publicly available datasets, namely, the CRIC cervix, Mendeley LBC, SIPaKMeD Pap Smear, and Cervix93 Extended Depth of Field image datasets. CervixFormer obtained a fairly better performance on two-, three-, four-, and six-class classification of smear- and cell-level datasets. For clinical interpretation, we used GradCAM to visualize a coarse localization map, highlighting important regions in the WSI. Notably, CervixFormer extracts feature mostly from the cell nucleus and partially from the cytoplasm.

CONCLUSIONS

In comparison with the existing state-of-the-art benchmark methods, the CervixFormer outperforms them in terms of recall, accuracy, and computing time.

Making the Improbable Possible: Generalizing Models Designed for a Syndrome-Based, Heterogeneous Patient Landscape

Syndromic conditions, such as sepsis, are commonly encountered in the intensive care unit. Although these conditions are easy for clinicians to grasp, these conditions may limit the performance of machine-learning algorithms. Individual hospital practice patterns may limit external generalizability. Data missingness is another barrier to optimal algorithm performance and various strategies exist to mitigate this. Recent advances in data science, such as transfer learning, conformal prediction, and continual learning, may improve generalizability of machine-learning algorithms in critically ill patients. Randomized trials with these approaches are indicated to demonstrate improvements in patient-centered outcomes at this point.

Deep learning in computational dermatopathology of melanoma: A technical systematic literature review

Deep learning (DL) has become one of the major approaches in computational dermatopathology, evidenced by a significant increase in this topic in the current literature. We aim to provide a structured and comprehensive overview of peer-reviewed publications on DL applied to dermatopathology focused on melanoma. In comparison to well-published DL methods on non-medical images (e.g., classification on ImageNet), this field of application comprises a specific set of challenges, such as staining artifacts, large gigapixel images, and various magnification levels. Thus, we are particularly interested in the pathology-specific technical state-of-the-art. We also aim to summarize the best performances achieved thus far with respect to accuracy, along with an overview of self-reported limitations. Accordingly, we conducted a systematic literature review of peer-reviewed journal and conference articles published between 2012 and 2022 in the databases ACM Digital Library, Embase, IEEE Xplore, PubMed, and Scopus, expanded by forward and backward searches to identify 495 potentially eligible studies. After screening for relevance and quality, a total of 54 studies were included. We qualitatively summarized and analyzed these studies from technical, problem-oriented, and task-oriented perspectives. Our findings suggest that the technical aspects of DL for histopathology in melanoma can be further improved. The DL methodology was adopted later in this field, and still lacks the wider adoption of DL methods already shown to be effective for other applications. We also discuss upcoming trends toward ImageNet-based feature extraction and larger models. While DL has achieved human-competitive accuracy in routine pathological tasks, its performance on advanced tasks is still inferior to wet-lab testing (for example). Finally, we discuss the challenges impeding the translation of DL methods to clinical practice and provide insight into future research directions.

Deep Transfer Learning with Enhanced Feature Fusion for Detection of Abnormalities in X-ray Images

Medical image classification poses significant challenges in real-world scenarios. One major obstacle is the scarcity of labelled training data, which hampers the performance of image-classification algorithms and generalisation. Gathering sufficient labelled data is often difficult and time-consuming in the medical domain, but deep learning (DL) has shown remarkable performance, although it typically requires a large amount of labelled data to achieve optimal results. Transfer learning (TL) has played a pivotal role in reducing the time, cost, and need for a large number of labelled images. This paper presents a novel TL approach that aims to overcome the limitations and disadvantages of TL that are characteristic of an ImageNet dataset, which belongs to a different domain. Our proposed TL approach involves training DL models on numerous medical images that are similar to the target dataset. These models were then fine-tuned using a small set of annotated medical images to leverage the knowledge gained from the pre-training phase. We specifically focused on medical X-ray imaging scenarios that involve the humerus and wrist from the musculoskeletal radiographs (MURA) dataset. Both of these tasks face significant challenges regarding accurate classification. The models trained with the proposed TL were used to extract features and were subsequently fused to train several machine learning (ML) classifiers. We combined these diverse features to represent various relevant characteristics in a comprehensive way. Through extensive evaluation, our proposed TL and feature-fusion approach using ML classifiers achieved remarkable results. For the classification of the humerus, we achieved an accuracy of 87.85%, an F1-score of 87.63%, and a Cohen's Kappa coefficient of 75.69%. For wrist classification, our approach achieved an accuracy of 85.58%, an F1-score of 82.70%, and a Cohen's Kappa coefficient of 70.46%. The results demonstrated that the models trained using our proposed TL approach outperformed those trained with ImageNet TL. We employed visualisation techniques to further validate these findings, including a gradient-based class activation heat map (Grad-CAM) and locally interpretable model-independent explanations (LIME). These visualisation tools provided additional evidence to support the superior accuracy of models trained with our proposed TL approach compared to those trained with ImageNet TL. Furthermore, our proposed TL approach exhibited greater robustness in various experiments compared to ImageNet TL. Importantly, the proposed TL approach and the feature-fusion technique are not limited to specific tasks. They can be applied to various medical image applications, thus extending their utility and potential impact. To demonstrate the concept of reusability, a computed tomography (CT) case was adopted. The results obtained from the proposed method showed improvements.

Spatially localized sparse approximations of deep features for breast mass characterization

We propose a deep feature-based sparse approximation classification technique for classification of breast masses into benign and malignant categories in film screen mammographs. This is a significant application as breast cancer is a leading cause of death in the modern world and improvements in diagnosis may help to decrease rates of mortality for large populations. While deep learning techniques have produced remarkable results in the field of computer-aided diagnosis of breast cancer, there are several aspects of this field that remain under-studied. In this work, we investigate the applicability of deep-feature-generated dictionaries to sparse approximation-based classification. To this end we construct dictionaries from deep features and compute sparse approximations of Regions Of Interest (ROIs) of breast masses for classification. Furthermore, we propose block and patch decomposition methods to construct overcomplete dictionaries suitable for sparse coding. The effectiveness of our deep feature spatially localized ensemble sparse analysis (DF-SLESA) technique is evaluated on a merged dataset of mass ROIs from the CBIS-DDSM and MIAS datasets. Experimental results indicate that dictionaries of deep features yield more discriminative sparse approximations of mass characteristics than dictionaries of imaging patterns and dictionaries learned by unsupervised machine learning techniques such as K-SVD. Of note is that the proposed block and patch decomposition strategies may help to simplify the sparse coding problem and to find tractable solutions. The proposed technique achieves competitive performances with state-of-the-art techniques for benign/malignant breast mass classification, using 10-fold cross-validation in merged datasets of film screen mammograms.

A Deep Learning Framework for the Characterization of Thyroid Nodules from Ultrasound Images Using Improved Inception Network and Multi-Level Transfer Learning

In the past few years, deep learning has gained increasingly widespread attention and has been applied to diagnosing benign and malignant thyroid nodules. It is difficult to acquire sufficient medical images, resulting in insufficient data, which hinders the development of an efficient deep-learning model. In this paper, we developed a deep-learning-based characterization framework to differentiate malignant and benign nodules from the thyroid ultrasound images. This approach improves the recognition accuracy of the inception network by combining squeeze and excitation networks with the inception modules. We have also integrated the concept of multi-level transfer learning using breast ultrasound images as a bridge dataset. This transfer learning approach addresses the issues regarding domain differences between natural images and ultrasound images during transfer learning. This paper aimed to investigate how the entire framework could help radiologists improve diagnostic performance and avoid unnecessary fine-needle aspiration. The proposed approach based on multi-level transfer learning and improved inception blocks achieved higher precision (0.9057 for the benign class and 0.9667 for the malignant class), recall (0.9796 for the benign class and 0.8529 for malignant), and F1-score (0.9412 for benign class and 0.9062 for malignant class). It also obtained an AUC value of 0.9537, which is higher than that of the single-level transfer learning method. The experimental results show that this model can achieve satisfactory classification accuracy comparable to experienced radiologists. Using this model, we can save time and effort as well as deliver potential clinical application value.

Domain Adaptation and Feature Fusion for the Detection of Abnormalities in X-Ray Forearm Images

The main challenge in adopting deep learning models is limited data for training, which can lead to poor generalization and a high risk of overfitting, particularly when detecting forearm abnormalities in X-ray images. Transfer learning from ImageNet is commonly used to address these issues. However, this technique is ineffective for grayscale medical imaging because of a mismatch between the learned features. To mitigate this issue, we propose a domain adaptation deep TL approach that involves training six pre-trained ImageNet models on a large number of X-ray images from various body parts, then fine-tuning the models on a target dataset of forearm X-ray images. Furthermore, the feature fusion technique combines the extracted features with deep neural models to train machine learning classifiers. Gradient-based class activation heat map (Grad CAM) was used to verify the accuracy of our results. This method allows us to see which parts of an image the model uses to make its classification decisions. The statically results and Grad CAM have shown that the proposed TL approach is able to alleviate the domain mismatch problem and is more accurate in their decision-making compared to models that were trained using the ImageNet TL technique, achieving an accuracy of 90.7%, an F1-score of 90.6%, and a Cohen's kappa of 81.3%. These results indicate that the proposed approach effectively improved the performance of the employed models individually and with the fusion technique. It helped to reduce the domain mismatch between the source of TL and the target task.

Linear fine-tuning: a linear transformation based transfer strategy for deep MRI reconstruction

Introduction

Fine-tuning (FT) is a generally adopted transfer learning method for deep learning-based magnetic resonance imaging (MRI) reconstruction. In this approach, the reconstruction model is initialized with pre-trained weights derived from a source domain with ample data and subsequently updated with limited data from the target domain. However, the direct full-weight update strategy can pose the risk of "catastrophic forgetting" and overfitting, hindering its effectiveness. The goal of this study is to develop a zero-weight update transfer strategy to preserve pre-trained generic knowledge and reduce overfitting.

Methods

Based on the commonality between the source and target domains, we assume a linear transformation relationship of the optimal model weights from the source domain to the target domain. Accordingly, we propose a novel transfer strategy, linear fine-tuning (LFT), which introduces scaling and shifting (SS) factors into the pre-trained model. In contrast to FT, LFT only updates SS factors in the transfer phase, while the pre-trained weights remain fixed.

Results

To evaluate the proposed LFT, we designed three different transfer scenarios and conducted a comparative analysis of FT, LFT, and other methods at various sampling rates and data volumes. In the transfer scenario between different contrasts, LFT outperforms typical transfer strategies at various sampling rates and considerably reduces artifacts on reconstructed images. In transfer scenarios between different slice directions or anatomical structures, LFT surpasses the FT method, particularly when the target domain contains a decreasing number of training images, with a maximum improvement of up to 2.06 dB (5.89%) in peak signal-to-noise ratio.

Discussion

The LFT strategy shows great potential to address the issues of "catastrophic forgetting" and overfitting in transfer scenarios for MRI reconstruction, while reducing the reliance on the amount of data in the target domain. Linear fine-tuning is expected to shorten the development cycle of reconstruction models for adapting complicated clinical scenarios, thereby enhancing the clinical applicability of deep MRI reconstruction.

Evaluating Synthetic Medical Images Using Artificial Intelligence with the GAN Algorithm

In recent years, considerable work has been conducted on the development of synthetic medical images, but there are no satisfactory methods for evaluating their medical suitability. Existing methods mainly evaluate the quality of noise in the images, and the similarity of the images to the real images used to generate them. For this purpose, they use feature maps of images extracted in different ways or distribution of images set. Then, the proximity of synthetic images to the real set is evaluated using different distance metrics. However, it is not possible to determine whether only one synthetic image was generated repeatedly, or whether the synthetic set exactly repeats the training set. In addition, most evolution metrics take a lot of time to calculate. Taking these issues into account, we have proposed a method that can quantitatively and qualitatively evaluate synthetic images. This method is a combination of two methods, namely, FMD and CNN-based evaluation methods. The estimation methods were compared with the FID method, and it was found that the FMD method has a great advantage in terms of speed, while the CNN method has the ability to estimate more accurately. To evaluate the reliability of the methods, a dataset of different real images was checked.

Systematic Review of Tumor Segmentation Strategies for Bone Metastases

Purpose: To investigate the segmentation approaches for bone metastases in differentiating benign from malignant bone lesions and characterizing malignant bone lesions. Method: The literature search was conducted in Scopus, PubMed, IEEE and MedLine, and Web of Science electronic databases following the guidelines of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). A total of 77 original articles, 24 review articles, and 1 comparison paper published between January 2010 and March 2022 were included in the review. Results: The results showed that most studies used neural network-based approaches (58.44%) and CT-based imaging (50.65%) out of 77 original articles. However, the review highlights the lack of a gold standard for tumor boundaries and the need for manual correction of the segmentation output, which largely explains the absence of clinical translation studies. Moreover, only 19 studies (24.67%) specifically mentioned the feasibility of their proposed methods for use in clinical practice. Conclusion: Development of tumor segmentation techniques that combine anatomical information and metabolic activities is encouraging despite not having an optimal tumor segmentation method for all applications or can compensate for all the difficulties built into data limitations.

Predicting Brain Amyloid Positivity from T1 weighted brain MRI and MRI-derived Gray Matter, White Matter and CSF maps using Transfer Learning on 3D CNNs

Abnormal β-amyloid (Aβ) accumulation in the brain is an early indicator of Alzheimer's disease and practical tests could help identify patients who could respond to treatment, now that promising anti-amyloid drugs are available. Even so, Aβ positivity (Aβ+) is assessed using PET or CSF assays, both highly invasive procedures. Here, we investigate how well Aβ+ can be predicted from T1 weighted brain MRI and gray matter, white matter and cerebrospinal fluid segmentations from T1-weighted brain MRI (T1w), a less invasive alternative. We used 3D convolutional neural networks to predict Aβ+ based on 3D brain MRI data, from 762 elderly subjects (mean age: 75.1 yrs. ± 7.6SD; 394F/368M; 459 healthy controls, 67 with MCI and 236 with dementia) scanned as part of the Alzheimer's Disease Neuroimaging Initiative. We also tested whether the accuracy increases when using transfer learning from the larger UK Biobank dataset. Overall, the 3D CNN predicted Aβ+ with 76% balanced accuracy from T1w scans. The closest performance to this was using white matter maps alone when the model was pre-trained on an age prediction in the UK Biobank. The performance of individual tissue maps was less than the T1w, but transfer learning helped increase the accuracy. Although tests on more diverse data are warranted, deep learned models from standard MRI show initial promise for Aβ+ estimation, before considering more invasive procedures.

A comprehensive survey of deep learning research on medical image analysis with focus on transfer learning

This survey aims to identify commonly used methods, datasets, future trends, knowledge gaps, constraints, and limitations in the field to provide an overview of current solutions used in medical image analysis in parallel with the rapid developments in transfer learning (TL). Unlike previous studies, this survey grouped the last five years of current studies for the period between January 2017 and February 2021 according to different anatomical regions and detailed the modality, medical task, TL method, source data, target data, and public or private datasets used in medical imaging. Also, it provides readers with detailed information on technical challenges, opportunities, and future research trends. In this way, an overview of recent developments is provided to help researchers to select the most effective and efficient methods and access widely used and publicly available medical datasets, research gaps, and limitations of the available literature.

Diseased thyroid tissue classification in OCT images using deep learning: Towards surgical decision support

Intraoperative guidance tools for thyroid surgery based on optical coherence tomography (OCT) could aid distinguish between normal and diseased tissue. However, OCT images are difficult to interpret, thus, real-time automatic analysis could support the clinical decision-making. In this study, several deep learning models were investigated for thyroid disease classification on 2D and 3D OCT data obtained from ex vivo specimens of 22 patients undergoing surgery and diagnosed with several thyroid pathologies. Additionally, two open-access datasets were used to evaluate the custom models. On the thyroid dataset, the best performance was achieved by the 3D vision transformer model with a Matthew's correlation coefficient (MCC) of 0.79 (accuracy = 0.90) for the normal-versus-abnormal classification. On the open-access datasets, the custom models achieved the best performance (MCC > 0.88, accuracy > 0.96). Results obtained for the normal-versus-abnormal classification suggest OCT, complemented with deep learning-based analysis, as a tool for real-time automatic diseased tissue identification in thyroid surgery.

Incorporating a Novel Dual Transfer Learning Approach for Medical Images

Recently, transfer learning approaches appeared to reduce the need for many classified medical images. However, these approaches still contain some limitations due to the mismatch of the domain between the source domain and the target domain. Therefore, this study aims to propose a novel approach, called Dual Transfer Learning (DTL), based on the convergence of patterns between the source and target domains. The proposed approach is applied to four pre-trained models (VGG16, Xception, ResNet50, MobileNetV2) using two datasets: ISIC2020 skin cancer images and ICIAR2018 breast cancer images, by fine-tuning the last layers on a sufficient number of unclassified images of the same disease and on a small number of classified images of the target task, in addition to using data augmentation techniques to balance classes and to increase the number of samples. According to the obtained results, it has been experimentally proven that the proposed approach has improved the performance of all models, where without data augmentation, the performance of the VGG16 model, Xception model, ResNet50 model, and MobileNetV2 model are improved by 0.28%, 10.96%, 15.73%, and 10.4%, respectively, while, with data augmentation, the VGG16 model, Xception model, ResNet50 model, and MobileNetV2 model are improved by 19.66%, 34.76%, 31.76%, and 33.03%, respectively. The Xception model obtained the highest performance compared to the rest of the models when classifying skin cancer images in the ISIC2020 dataset, as it obtained 96.83%, 96.919%, 96.826%, 96.825%, 99.07%, and 94.58% for accuracy, precision, recall, F1-score, sensitivity, and specificity respectively. To classify the images of the ICIAR 2018 dataset for breast cancer, the Xception model obtained 99%, 99.003%, 98.995%, 99%, 98.55%, and 99.14% for accuracy, precision, recall, F1-score, sensitivity, and specificity, respectively. Through these results, the proposed approach improved the models' performance when fine-tuning was performed on unclassified images of the same disease.

A radiographic, deep transfer learning framework, adapted to estimate lung opacities from chest x-rays

Chest radiographs (CXRs) are the most widely available radiographic imaging modality used to detect respiratory diseases that result in lung opacities. CXR reports often use non-standardized language that result in subjective, qualitative, and non-reproducible opacity estimates. Our goal was to develop a robust deep transfer learning framework and adapt it to estimate the degree of lung opacity from CXRs. Following CXR data selection based on exclusion criteria, segmentation schemes were used for ROI (Region Of Interest) extraction, and all combinations of segmentation, data balancing, and classification methods were tested to pick the top performing models. Multifold cross validation was used to determine the best model from the initial selected top models, based on appropriate performance metrics, as well as a novel Macro-Averaged Heatmap Concordance Score (MA HCS). Performance of the best model is compared against that of expert physician annotators, and heatmaps were produced. Finally, model performance sensitivity analysis across patient populations of interest was performed. The proposed framework was adapted to the specific use case of estimation of degree of CXR lung opacity using ordinal multiclass classification. Acquired between March 24, 2020, and May 22, 2020, 38,365 prospectively annotated CXRs from 17,418 patients were used. We tested three neural network architectures (ResNet-50, VGG-16, and ChexNet), three segmentation schemes (no segmentation, lung segmentation, and lateral segmentation based on spine detection), and three data balancing strategies (undersampling, double-stage sampling, and synthetic minority oversampling) using 38,079 CXR images for training, and validation with 286 images as the out-of-the-box dataset that underwent expert radiologist adjudication. Based on the results of these experiments, the ResNet-50 model with undersampling and no ROI segmentation is recommended for lung opacity classification, based on optimal values for the MAE metric and HCS (Heatmap Concordance Score). The degree of agreement between the opacity scores predicted by this model with respect to the two sets of radiologist scores (OR or Original Reader and OOBTR or Out Of Box Reader) in terms of performance metrics is superior to the inter-radiologist opacity score agreement.

XDecompo: Explainable Decomposition Approach in Convolutional Neural Networks for Tumour Image Classification

Of the various tumour types, colorectal cancer and brain tumours are still considered among the most serious and deadly diseases in the world. Therefore, many researchers are interested in improving the accuracy and reliability of diagnostic medical machine learning models. In computer-aided diagnosis, self-supervised learning has been proven to be an effective solution when dealing with datasets with insufficient data annotations. However, medical image datasets often suffer from data irregularities, making the recognition task even more challenging. The class decomposition approach has provided a robust solution to such a challenging problem by simplifying the learning of class boundaries of a dataset. In this paper, we propose a robust self-supervised model, called XDecompo, to improve the transferability of features from the pretext task to the downstream task. XDecompo has been designed based on an affinity propagation-based class decomposition to effectively encourage learning of the class boundaries in the downstream task. XDecompo has an explainable component to highlight important pixels that contribute to classification and explain the effect of class decomposition on improving the speciality of extracted features. We also explore the generalisability of XDecompo in handling different medical datasets, such as histopathology for colorectal cancer and brain tumour images. The quantitative results demonstrate the robustness of XDecompo with high accuracy of 96.16% and 94.30% for CRC and brain tumour images, respectively. XDecompo has demonstrated its generalization capability and achieved high classification accuracy (both quantitatively and qualitatively) in different medical image datasets, compared with other models. Moreover, a post hoc explainable method has been used to validate the feature transferability, demonstrating highly accurate feature representations.

Inhalation Injury Grading Using Transfer Learning Based on Bronchoscopy Images and Mechanical Ventilation Period

The abbreviated injury score (AIS) is commonly used as a grading system for inhalation injuries. While inhalation injury grades have inconsistently been shown to correlate positively with the time mechanical ventilation is needed, grading is subjective and relies heavily on the clinicians' experience and expertise. Additionally, no correlation has been shown between these patients' inhalation injury grades and outcomes. In this paper, we propose a novel inhalation injury grading method which uses deep learning algorithms in bronchoscopy images to determine the injury grade from the carbonaceous deposits, blistering, and fibrin casts in the bronchoscopy images. The proposed method adopts transfer learning and data augmentation concepts to enhance the accuracy performance to avoid overfitting. We tested our proposed model on the bronchoscopy images acquired from eighteen patients who had suffered inhalation injuries, with the degree of severity 1, 2, 3, 4, 5, or 6. As performance metrics, we consider accuracy, sensitivity, specificity, F-1 score, and precision. Experimental results show that our proposed method, with both transfer learning and data augmentation components, provides an overall 86.11% accuracy. Moreover, the experimental results also show that the performance of the proposed method outperforms the method without transfer learning or data augmentation.

Neurosurgery inpatient outcome prediction for discharge planning with deep learning and transfer learning

INTRODUCTION

Deep learning may be able to assist with the prediction of neurosurgical inpatient outcomes. The aims of this study were to investigate deep learning and transfer learning in the prediction of several inpatient outcomes including timing of discharge and discharge destination.

METHOD

Data were collected on consecutive neurosurgical admissions from existing databases over a 15-month period. Following pre-processing artificial neural networks were applied to admission notes and ward round notes to predict four inpatient outcomes. Models were developed on the training dataset, before being tested on a hold-out test dataset and a validation dataset.

RESULTS

1341 individual admissions were included in the study. Using transfer learning and an artificial neural network an area under the receiver operator curve (AUC) of 0.81 and 0.80 on the derivation and validation datasets was able to be achieved for the prediction of discharge within the next 48 hours using daily ward round notes. This result is in comparison to an AUC of 0.71 and 0.68 using an artificial neural network without transfer learning for the same outcome. When the artificial neural network with transfer learning was applied to the other outcomes AUC of 0.72, 0.93 and 0.83 was achieved on the validation datasets for predicting discharge within the next 7 days, survival to discharge and discharge to home as a destination.

CONCLUSIONS

Deep learning may predict inpatient neurosurgery outcomes from free-text medical data. Recurrent predictions with ward round notes enable the use of information obtained throughout hospital admissions in these estimates.

Artificial Intelligence in Dermatology Image Analysis: Current Developments and Future Trends

BACKGROUND

Thanks to the rapid development of computer-based systems and deep-learning-based algorithms, artificial intelligence (AI) has long been integrated into the healthcare field. AI is also particularly helpful in image recognition, surgical assistance and basic research. Due to the unique nature of dermatology, AI-aided dermatological diagnosis based on image recognition has become a modern focus and future trend. Key scientific concepts of review: The use of 3D imaging systems allows clinicians to screen and label skin pigmented lesions and distributed disorders, which can provide an objective assessment and image documentation of lesion sites. Dermatoscopes combined with intelligent software help the dermatologist to easily correlate each close-up image with the corresponding marked lesion in the 3D body map. In addition, AI in the field of prosthetics can assist in the rehabilitation of patients and help to restore limb function after amputation in patients with skin tumors.

THE AIM OF THE STUDY

For the benefit of patients, dermatologists have an obligation to explore the opportunities, risks and limitations of AI applications. This study focuses on the application of emerging AI in dermatology to aid clinical diagnosis and treatment, analyzes the current state of the field and summarizes its future trends and prospects so as to help dermatologists realize the impact of new technological innovations on traditional practices so that they can embrace and use AI-based medical approaches more quickly.

Classification of Skin Lesion through Active Learning Strategies

BACKGROUND AND OBJECTIVE

According to the National Cancer Institute, among all malignant tumors, non-melanoma skin cancer, and melanoma are the most frequent in Brazil. Despite having a lower incidence, the melanoma type has accelerated growth and greater lethality. Several studies have been performed in recent years in the computer vision area to assist in the early diagnosis of skin cancer. Despite being widely used and presenting good results, deep learning approaches require a large amount of annotated data and considerable computational cost for training the model. Therefore, the present work explores active learning approaches to select a small set of more informative data for training the classifier. For that, different selection criteria are considered to obtain more effective and efficient classifiers for skin lesions.

METHODS

We perform an extensive experimental evaluation considering three datasets and different learning strategies and scenarios for validation. In addition to data augmentation, we evaluated two segmentation strategies considering the U-net CNN model and the Fully Convolutional Networks (FCN) with a manual expert review. We also analyzed the best (handcrafted and deep) features that describe each skin lesion and the most suitable classifiers and combinations (extractor-classifier) for this context. The active learning approach evaluated different criteria based on uncertainty, diversity, and representativeness to select the most informative samples. The strategies used were Decreasing Boundary Edges, Entropy, Least Confidence, Margin Sampling, Minimum-Spanning Tree Boundary Edges, and Root-Distance based Sampling.

RESULTS

It can be observed that the segmentation with FCN and manual correction by the specialist, the Border-Interior Classification (BIC) extractor, and the Random Forest (RF) classifier showed a better performance. Regarding the active learning approach, the Margin Sampling strategy presented the best classification accuracies (about 93%) with only 35% of the training set compared to the traditional learning approach (which requires the entire set).

CONCLUSIONS

According to the results, it is possible to observe that the selection strategies allow for achieving high accuracies faster (fewer learning iterations) and with a smaller amount of labeled samples compared to the traditional learning approach. Hence, active learning can contribute significantly to the diagnosis of skin lesions, beneficially reducing specialists' annotation costs.

A review of non-invasive sensors and artificial intelligence models for diabetic foot monitoring

Diabetic foot complications have multiple adverse effects in a person's quality of life. Yet, efficient monitoring schemes can mitigate or postpone any disorders, mainly by early detecting regions of interest. Nowadays, optical sensors and artificial intelligence (AI) tools can contribute efficiently to such monitoring processes. In this work, we provide information on the adopted imaging schemes and related optical sensors on this topic. The analysis considers both the physiology of the patients and the characteristics of the sensors. Currently, there are multiple approaches considering both visible and infrared bands (multiple ranges), most of them coupled with various AI tools. The source of the data (sensor type) can support different monitoring strategies and imposes restrictions on the AI tools that should be used with. This review provides a comprehensive literature review of AI-assisted DFU monitoring methods. The paper presents the outcomes of a large number of recently published scholarly articles. Furthermore, the paper discusses the highlights of these methods and the challenges for transferring these methods into a practical and trustworthy framework for sufficient remote management of the patients.

A conditional GAN-based approach for enhancing transfer learning performance in few-shot HCR tasks

Supervised learning with the restriction of a few existing training samples is called Few-Shot Learning. FSL is a subarea that puts deep learning performance in a gap, as building robust deep networks requires big training data. Using transfer learning in FSL tasks is an acceptable way to avoid the challenge of building new deep models from scratch. Transfer learning methodology considers borrowing the architecture and parameters of a previously trained model on a large-scale dataset and fine-tuning it for low-data target tasks. But practically, fine-tuning pretrained models in target FSL tasks suffers from overfitting. The few existing samples are not enough to correctly adjust the pretrained model's parameters to provide the best fit for the target task. In this study, we consider mitigating the overfitting problem when applying transfer learning in few-shot Handwritten Character Recognition (HCR) tasks. A data augmentation approach based on Conditional Generative Adversarial Networks is introduced. CGAN is a generative model that can create artificial instances that appear more real and indistinguishable from the original samples. CGAN helps generate extra samples that hold the possible variations of human handwriting instead of applying traditional image transformations. These transformations are low-level, data-independent operations, and only produce augmented samples with limited diversity. The introduced approach was evaluated in fine-tuning the three pretrained models: AlexNet, VGG-16, and GoogleNet. The results show that the samples generated by CGAN can enhance transfer learning performance in few-shot HCR tasks. This is by achieving model fine-tuning with fewer epochs and by increasing the model's [Formula: see text] and decreasing the Generalization Error [Formula: see text].

Detection of COVID-19 and its pulmonary stage using Bayesian hyperparameter optimization and deep feature selection methods

Since the first case of COVID-19 was reported in December 2019, many studies have been carried out on artificial intelligence for the rapid diagnosis of the disease to support health services. Therefore, in this study, we present a powerful approach to detect COVID-19 and COVID-19 findings from computed tomography images using pre-trained models using two different datasets. COVID-19, influenza A (H1N1) pneumonia, bacterial pneumonia and healthy lung image classes were used in the first dataset. Consolidation, crazy-paving pattern, ground-glass opacity, ground-glass opacity and consolidation, ground-glass opacity and nodule classes were used in the second dataset. The study consists of four steps. In the first two steps, distinctive features were extracted from the final layers of the pre-trained ShuffleNet, GoogLeNet and MobileNetV2 models trained with the datasets. In the next steps, the most relevant features were selected from the models using the Sine-Cosine optimization algorithm. Then, the hyperparameters of the Support Vector Machines were optimized with the Bayesian optimization algorithm and used to reclassify the feature subset that achieved the highest accuracy in the third step. The overall accuracy obtained for the first and second datasets is 99.46% and 99.82%, respectively. Finally, the performance of the results visualized with Occlusion Sensitivity Maps was compared with Gradient-weighted class activation mapping. The approach proposed in this paper outperformed other methods in detecting COVID-19 from multiclass viral pneumonia. Moreover, detecting the stages of COVID-19 in the lungs was an innovative and successful approach.

An extensive review of state-of-the-art transfer learning techniques used in medical imaging: Open issues and challenges

Deep learning techniques, which use a massive technology known as convolutional neural networks, have shown excellent results in a variety of areas, including image processing and interpretation. However, as the depth of these networks grows, so does the demand for a large amount of labeled data required to train these networks. In particular, the medical field suffers from a lack of images because the procedure for obtaining labeled medical images in the healthcare field is difficult, expensive, and requires specialized expertise to add labels to images. Moreover, the process may be prone to errors and time-consuming. Current research has revealed transfer learning as a viable solution to this problem. Transfer learning allows us to transfer knowledge gained from a previous process to improve and tackle a new problem. This study aims to conduct a comprehensive survey of recent studies that dealt with solving this problem and the most important metrics used to evaluate these methods. In addition, this study identifies problems in transfer learning techniques and highlights the problems of the medical dataset and potential problems that can be addressed in future research. According to our review, many researchers use pre-trained models on the Imagenet dataset (VGG16, ResNet, Inception v3) in many applications such as skin cancer, breast cancer, and diabetic retinopathy classification tasks. These techniques require further investigation of these models, due to training them on natural, non-medical images. In addition, many researchers use data augmentation techniques to expand their dataset and avoid overfitting. However, not enough studies have shown the effect of performance with or without data augmentation. Accuracy, recall, precision, F1 score, receiver operator characteristic curve, and area under the curve (AUC) were the most widely used measures in these studies. Furthermore, we identified problems in the datasets for melanoma and breast cancer and suggested corresponding solutions.

Is my food safe? - AI-based classification of lentil flour samples with trace levels of gluten or nuts

An artificial intelligence-based method to rapidly detect adulterated lentil flour in real time is presented. Mathematical models based on convolutional neural networks and transfer learning (viz., ResNet34) have been trained to identify lentil flour samples that contain trace levels of wheat (gluten) or pistachios (nuts), aiding two relevant populations (people with celiac disease and with nut allergies, respectively). The technique is based on the analysis of photographs taken by a simple reflex camera and further classification into groups assigned to adulterant type and amount (up to 50 ppm). Two different algorithms were trained, one per adulterant, using a total of 2200 images for each neural network. Using blind sets of data (10% of the collected images; initially and randomly separated) to evaluate the performance of the models led to strong performances, as 99.1% of lentil flour samples containing ground pistachio were correctly classified, while 96.4% accuracy was reached to classify the samples containing wheat flour.

Classification of breast cancer using a manta-ray foraging optimized transfer learning framework

Due to its high prevalence and wide dissemination, breast cancer is a particularly dangerous disease. Breast cancer survival chances can be improved by early detection and diagnosis. For medical image analyzers, diagnosing is tough, time-consuming, routine, and repetitive. Medical image analysis could be a useful method for detecting such a disease. Recently, artificial intelligence technology has been utilized to help radiologists identify breast cancer more rapidly and reliably. Convolutional neural networks, among other technologies, are promising medical image recognition and classification tools. This study proposes a framework for automatic and reliable breast cancer classification based on histological and ultrasound data. The system is built on CNN and employs transfer learning technology and metaheuristic optimization. The Manta Ray Foraging Optimization (MRFO) approach is deployed to improve the framework's adaptability. Using the Breast Cancer Dataset (two classes) and the Breast Ultrasound Dataset (three-classes), eight modern pre-trained CNN architectures are examined to apply the transfer learning technique. The framework uses MRFO to improve the performance of CNN architectures by optimizing their hyperparameters. Extensive experiments have recorded performance parameters, including accuracy, AUC, precision, F1-score, sensitivity, dice, recall, IoU, and cosine similarity. The proposed framework scored 97.73% on histopathological data and 99.01% on ultrasound data in terms of accuracy. The experimental results show that the proposed framework is superior to other state-of-the-art approaches in the literature review.

Superior temporal gyrus functional connectivity predicts transcranial direct current stimulation response in Schizophrenia: A machine learning study

Transcranial direct current stimulation (tDCS) is a promising adjuvant treatment for persistent auditory verbal hallucinations (AVH) in Schizophrenia (SZ). Nonetheless, there is considerable inter-patient variability in the treatment response of AVH to tDCS in SZ. Machine-learned models have the potential to predict clinical response to tDCS in SZ. This study aims to examine the feasibility of identifying SZ patients with persistent AVH (SZ-AVH) who will respond to tDCS based on resting-state functional connectivity (rs-FC). Thirty-four SZ-AVH patients underwent resting-state functional MRI at baseline followed by add-on, twice-daily, 20-min sessions with tDCS (conventional/high-definition) for 5 days. A machine learning model was developed to identify tDCS treatment responders based on the rs-FC pattern, using the left superior temporal gyrus (LSTG) as the seed region. Functional connectivity between LSTG and brain regions involved in auditory and sensorimotor processing emerged as the important predictors of the tDCS treatment response. L1-regularized logistic regression model had an overall accuracy of 72.5% in classifying responders vs. non-responders. This model outperformed the state-of-the-art convolutional neural networks (CNN) model-both without (59.41%) and with pre-training (68.82%). It also outperformed the L1-logistic regression model trained with baseline demographic features and clinical scores of SZ patients. This study reports the first evidence that rs-fMRI-derived brain connectivity pattern can predict the clinical response of persistent AVH to add-on tDCS in SZ patients with 72.5% accuracy.

Skin Cancer Classification With Deep Learning: A Systematic Review

Skin cancer is one of the most dangerous diseases in the world. Correctly classifying skin lesions at an early stage could aid clinical decision-making by providing an accurate disease diagnosis, potentially increasing the chances of cure before cancer spreads. However, achieving automatic skin cancer classification is difficult because the majority of skin disease images used for training are imbalanced and in short supply; meanwhile, the model’s cross-domain adaptability and robustness are also critical challenges. Recently, many deep learning-based methods have been widely used in skin cancer classification to solve the above issues and achieve satisfactory results. Nonetheless, reviews that include the abovementioned frontier problems in skin cancer classification are still scarce. Therefore, in this article, we provide a comprehensive overview of the latest deep learning-based algorithms for skin cancer classification. We begin with an overview of three types of dermatological images, followed by a list of publicly available datasets relating to skin cancers. After that, we review the successful applications of typical convolutional neural networks for skin cancer classification. As a highlight of this paper, we next summarize several frontier problems, including data imbalance, data limitation, domain adaptation, model robustness, and model efficiency, followed by corresponding solutions in the skin cancer classification task. Finally, by summarizing different deep learning-based methods to solve the frontier challenges in skin cancer classification, we can conclude that the general development direction of these approaches is structured, lightweight, and multimodal. Besides, for readers’ convenience, we have summarized our findings in figures and tables. Considering the growing popularity of deep learning, there are still many issues to overcome as well as chances to pursue in the future.

Federated Learning-Based Detection of Invasive Carcinoma of No Special Type with Histopathological Images

Invasive carcinoma of no special type (IC-NST) is known to be one of the most prevalent kinds of breast cancer, hence the growing research interest in studying automated systems that can detect the presence of breast tumors and appropriately classify them into subtypes. Machine learning (ML) and, more specifically, deep learning (DL) techniques have been used to approach this problem. However, such techniques usually require massive amounts of data to obtain competitive results. This requirement makes their application in specific areas such as health problematic as privacy concerns regarding the release of patients’ data publicly result in a limited number of publicly available datasets for the research community. This paper proposes an approach that leverages federated learning (FL) to securely train mathematical models over multiple clients with local IC-NST images partitioned from the breast histopathology image (BHI) dataset to obtain a global model. First, we used residual neural networks for automatic feature extraction. Then, we proposed a second network consisting of Gabor kernels to extract another set of features from the IC-NST dataset. After that, we performed a late fusion of the two sets of features and passed the output through a custom classifier. Experiments were conducted for the federated learning (FL) and centralized learning (CL) scenarios, and the results were compared. Competitive results were obtained, indicating the positive prospects of adopting FL for IC-NST detection. Additionally, fusing the Gabor features with the residual neural network features resulted in the best performance in terms of accuracy, F1 score, and area under the receiver operation curve (AUC-ROC). The models show good generalization by performing well on another domain dataset, the breast cancer histopathological (BreakHis) image dataset. Our method also outperformed other methods from the literature.

Topology optimization search of deep convolution neural networks for CT and X-ray image classification

Covid-19 is a disease that can lead to pneumonia, respiratory syndrome, septic shock, multiple organ failure, and death. This pandemic is viewed as a critical component of the fight against an enormous threat to the human population. Deep convolutional neural networks have recently proved their ability to perform well in classification and dimension reduction tasks. Selecting hyper-parameters is critical for these networks. This is because the search space expands exponentially in size as the number of layers increases. All existing approaches utilize a pre-trained or designed architecture as an input. None of them takes design and pruning into account throughout the process. In fact, there exists a convolutional topology for any architecture, and each block of a CNN corresponds to an optimization problem with a large search space. However, there are no guidelines for designing a specific architecture for a specific purpose; thus, such design is highly subjective and heavily reliant on data scientists’ knowledge and expertise. Motivated by this observation, we propose a topology optimization method for designing a convolutional neural network capable of classifying radiography images and detecting probable chest anomalies and infections, including COVID-19. Our method has been validated in a number of comparative studies against relevant state-of-the-art architectures.

Deep Clustering via Center-Oriented Margin Free-Triplet Loss for Skin Lesion Detection in Highly Imbalanced Datasets

Melanoma is a fatal skin cancer that is curable and has dramatically increasing survival rate when diagnosed at early stages. Learning-based methods hold significant promise for the detection of melanoma from dermoscopic images. However, since melanoma is a rare disease, existing databases of skin lesions predominantly contain highly imbalanced numbers of benign versus malignant samples. In turn, this imbalance introduces substantial bias in classification models due to the statistical dominance of the majority class. To address this issue, we introduce a deep clustering approach based on the latent-space embedding of dermoscopic images. Clustering is achieved using a novel center-oriented margin-free triplet loss (COM-Triplet) enforced on image embeddings from a convolutional neural network backbone. The proposed method aims to form maximally-separated cluster centers as opposed to minimizing classification error, so it is less sensitive to class imbalance. To avoid the need for labeled data, we further propose to implement COM-Triplet based on pseudo-labels generated by a Gaussian mixture model (GMM). Comprehensive experiments show that deep clustering with COM-Triplet loss outperforms clustering with triplet loss, and competing classifiers in both supervised and unsupervised settings.

Evaluation of cancer outcome assessment using MRI: A review of deep-learning methods

Accurate evaluation of tumor response to treatment is critical to allow personalized treatment regimens according to the predicted response and to support clinical trials investigating new therapeutic agents by providing them with an accurate response indicator. Recent advances in medical imaging, computer hardware, and machine-learning algorithms have resulted in the increased use of these tools in the field of medicine as a whole and specifically in cancer imaging for detection and characterization of malignant lesions, prognosis, and assessment of treatment response. Among the currently available imaging techniques, magnetic resonance imaging (MRI) plays an important role in the evaluation of treatment assessment of many cancers, given its superior soft-tissue contrast and its ability to allow multiplanar imaging and functional evaluation. In recent years, deep learning (DL) has become an active area of research, paving the way for computer-assisted clinical and radiological decision support. DL can uncover associations between imaging features that cannot be visually identified by the naked eye and pertinent clinical outcomes. The aim of this review is to highlight the use of DL in the evaluation of tumor response assessed on MRI. In this review, we will first provide an overview of common DL architectures used in medical imaging research in general. Then, we will review the studies to date that have applied DL to magnetic resonance imaging for the task of treatment response assessment. Finally, we will discuss the challenges and opportunities of using DL within the clinical workflow.

Low-Cost Probabilistic 3D Denoising with Applications for Ultra-Low-Radiation Computed Tomography

We propose a pipeline for synthetic generation of personalized Computer Tomography (CT) images, with a radiation exposure evaluation and a lifetime attributable risk (LAR) assessment. We perform a patient-specific performance evaluation for a broad range of denoising algorithms (including the most popular deep learning denoising approaches, wavelets-based methods, methods based on Mumford−Shah denoising, etc.), focusing both on accessing the capability to reduce the patient-specific CT-induced LAR and on computational cost scalability. We introduce a parallel Probabilistic Mumford−Shah denoising model (PMS) and show that it markedly-outperforms the compared common denoising methods in denoising quality and cost scaling. In particular, we show that it allows an approximately 22-fold robust patient-specific LAR reduction for infants and a 10-fold LAR reduction for adults. Using a normal laptop, the proposed algorithm for PMS allows cheap and robust (with a multiscale structural similarity index >90%) denoising of very large 2D videos and 3D images (with over 107 voxels) that are subject to ultra-strong noise (Gaussian and non-Gaussian) for signal-to-noise ratios far below 1.0. The code is provided for open access.

Deep learning model for tongue cancer diagnosis using endoscopic images

In this study, we developed a deep learning model to identify patients with tongue cancer based on a validated dataset comprising oral endoscopic images. We retrospectively constructed a dataset of 12,400 verified endoscopic images from five university hospitals in South Korea, collected between 2010 and 2020 with the participation of otolaryngologists. To calculate the probability of malignancy using various convolutional neural network (CNN) architectures, several deep learning models were developed. Of the 12,400 total images, 5576 images related to the tongue were extracted. The CNN models showed a mean area under the receiver operating characteristic curve (AUROC) of 0.845 and a mean area under the precision-recall curve (AUPRC) of 0.892. The results indicate that the best model was DenseNet169 (AUROC 0.895 and AUPRC 0.918). The deep learning model, general physicians, and oncology specialists had sensitivities of 81.1%, 77.3%, and 91.7%; specificities of 86.8%, 75.0%, and 90.9%; and accuracies of 84.7%, 75.9%, and 91.2%, respectively. Meanwhile, fair agreement between the oncologist and the developed model was shown for cancer diagnosis (kappa value = 0.685). The deep learning model developed based on the verified endoscopic image dataset showed acceptable performance in tongue cancer diagnosis.

Fast Segmentation of Metastatic Foci in H&E Whole-Slide Images for Breast Cancer Diagnosis

Breast cancer is the leading cause of death for women globally. In clinical practice, pathologists visually scan over enormous amounts of gigapixel microscopic tissue slide images, which is a tedious and challenging task. In breast cancer diagnosis, micro-metastases and especially isolated tumor cells are extremely difficult to detect and are easily neglected because tiny metastatic foci might be missed in visual examinations by medical doctors. However, the literature poorly explores the detection of isolated tumor cells, which could be recognized as a viable marker to determine the prognosis for T1NoMo breast cancer patients. To address these issues, we present a deep learning-based framework for efficient and robust lymph node metastasis segmentation in routinely used histopathological hematoxylin−eosin-stained (H−E) whole-slide images (WSI) in minutes, and a quantitative evaluation is conducted using 188 WSIs, containing 94 pairs of H−E-stained WSIs and immunohistochemical CK(AE1/AE3)-stained WSIs, which are used to produce a reliable and objective reference standard. The quantitative results demonstrate that the proposed method achieves 89.6% precision, 83.8% recall, 84.4% F1-score, and 74.9% mIoU, and that it performs significantly better than eight deep learning approaches, including two recently published models (v3_DCNN and Xception-65), and three variants of Deeplabv3+ with three different backbones, namely, U-Net, SegNet, and FCN, in precision, recall, F1-score, and mIoU (p<0.001). Importantly, the proposed system is shown to be capable of identifying tiny metastatic foci in challenging cases, for which there are high probabilities of misdiagnosis in visual inspection, while the baseline approaches tend to fail in detecting tiny metastatic foci. For computational time comparison, the proposed method takes 2.4 min for processing a WSI utilizing four NVIDIA Geforce GTX 1080Ti GPU cards and 9.6 min using a single NVIDIA Geforce GTX 1080Ti GPU card, and is notably faster than the baseline methods (4-times faster than U-Net and SegNet, 5-times faster than FCN, 2-times faster than the 3 different variants of Deeplabv3+, 1.4-times faster than v3_DCNN, and 41-times faster than Xception-65).

Transfer learning for medical image classification: a literature review

BACKGROUND

Transfer learning (TL) with convolutional neural networks aims to improve performances on a new task by leveraging the knowledge of similar tasks learned in advance. It has made a major contribution to medical image analysis as it overcomes the data scarcity problem as well as it saves time and hardware resources. However, transfer learning has been arbitrarily configured in the majority of studies. This review paper attempts to provide guidance for selecting a model and TL approaches for the medical image classification task.

METHODS

425 peer-reviewed articles were retrieved from two databases, PubMed and Web of Science, published in English, up until December 31, 2020. Articles were assessed by two independent reviewers, with the aid of a third reviewer in the case of discrepancies. We followed the PRISMA guidelines for the paper selection and 121 studies were regarded as eligible for the scope of this review. We investigated articles focused on selecting backbone models and TL approaches including feature extractor, feature extractor hybrid, fine-tuning and fine-tuning from scratch.

RESULTS

The majority of studies (n = 57) empirically evaluated multiple models followed by deep models (n = 33) and shallow (n = 24) models. Inception, one of the deep models, was the most employed in literature (n = 26). With respect to the TL, the majority of studies (n = 46) empirically benchmarked multiple approaches to identify the optimal configuration. The rest of the studies applied only a single approach for which feature extractor (n = 38) and fine-tuning from scratch (n = 27) were the two most favored approaches. Only a few studies applied feature extractor hybrid (n = 7) and fine-tuning (n = 3) with pretrained models.

CONCLUSION

The investigated studies demonstrated the efficacy of transfer learning despite the data scarcity. We encourage data scientists and practitioners to use deep models (e.g. ResNet or Inception) as feature extractors, which can save computational costs and time without degrading the predictive power.