DDK: Dynamic structure pruning based on differentiable search and recursive knowledge distillation for BERT

Large-scale pre-trained models, such as BERT, have demonstrated outstanding performance in Natural Language Processing (NLP). Nevertheless, the high number of parameters in these models has increased the demand for hardware storage and computational resources while posing a challenge for their practical deployment. In this article, we propose a combined method of model pruning and knowledge distillation to compress and accelerate large-scale pre-trained language models. Specifically, we introduce a dynamic structure pruning method based on differentiable search and recursive knowledge distillation to automatically prune the BERT model, named DDK. We define the search space for network pruning as all feed-forward layer channels and self-attention heads at each layer of the network, and utilize differentiable methods to determine their optimal number. Additionally, we design a recursive knowledge distillation method that employs adaptive weighting to extract the most important features from multiple intermediate layers of the teacher model and fuse them to supervise the student network learning. Our experimental results on the GLUE benchmark dataset and ablation analysis demonstrate that our proposed method outperforms other advanced methods in terms of average performance.

Generalizable disease detection using model ensemble on chest X-ray images

In the realm of healthcare, the demand for swift and precise diagnostic tools has been steadily increasing. This study delves into a comprehensive performance analysis of three pre-trained convolutional neural network (CNN) architectures: ResNet50, DenseNet121, and Inception-ResNet-v2. To ensure the broad applicability of our approach, we curated a large-scale dataset comprising a diverse collection of chest X-ray images, that included both positive and negative cases of COVID-19. The models' performance was evaluated using separate datasets for internal validation (from the same source as the training images) and external validation (from different sources). Our examination uncovered a significant drop in network efficacy, registering a 10.66% reduction for ResNet50, a 36.33% decline for DenseNet121, and a 19.55% decrease for Inception-ResNet-v2 in terms of accuracy. Best results were obtained with DenseNet121 achieving the highest accuracy at 96.71% in internal validation and Inception-ResNet-v2 attaining 76.70% accuracy in external validation. Furthermore, we introduced a model ensemble approach aimed at improving network performance when making inferences on images from diverse sources beyond their training data. The proposed method uses uncertainty-based weighting by calculating the entropy in order to assign appropriate weights to the outputs of each network. Our results showcase the effectiveness of the ensemble method in enhancing accuracy up to 97.38% for internal validation and 81.18% for external validation, while maintaining a balanced ability to detect both positive and negative cases.

Classification of H. pylori Infection from Histopathological Images Using Deep Learning

Helicobacter pylori (H. pylori) is a widespread pathogenic bacterium, impacting over 4 billion individuals globally. It is primarily linked to gastric diseases, including gastritis, peptic ulcers, and cancer. The current histopathological method for diagnosing H. pylori involves labour-intensive examination of endoscopic biopsies by trained pathologists. However, this process can be time-consuming and may occasionally result in the oversight of small bacterial quantities. Our study explored the potential of five pre-trained models for binary classification of 204 histopathological images, distinguishing between H. pylori-positive and H. pylori-negative cases. These models include EfficientNet-b0, DenseNet-201, ResNet-101, MobileNet-v2, and Xception. To evaluate the models' performance, we conducted a five-fold cross-validation, ensuring the models' reliability across different subsets of the dataset. After extensive evaluation and comparison of the models, ResNet101 emerged as the most promising. It achieved an average accuracy of 0.920, with impressive scores for sensitivity, specificity, positive predictive value, negative predictive value, F1 score, Matthews's correlation coefficient, and Cohen's kappa coefficient. Our study achieved these robust results using a smaller dataset compared to previous studies, highlighting the efficacy of deep learning models even with limited data. These findings underscore the potential of deep learning models, particularly ResNet101, to support pathologists in achieving precise and dependable diagnostic procedures for H. pylori. This is particularly valuable in scenarios where swift and accurate diagnoses are essential.

Deep learning innovations in diagnosing diabetic retinopathy: The potential of transfer learning and the DiaCNN model

Diabetic retinopathy (DR) is a significant cause of vision impairment, emphasizing the critical need for early detection and timely intervention to avert visual deterioration. Diagnosing DR is inherently complex, as it necessitates the meticulous examination of intricate retinal images by experienced specialists. This makes the early diagnosis of DR essential for effective treatment and prevention of eventual blindness. Traditional diagnostic methods, relying on human interpretation of medical images, face challenges in terms of accuracy and efficiency. In the present research, we introduce a novel method that offers superior precision in DR diagnosis, compared to traditional methods, by employing advanced deep learning techniques. Central to this approach is the concept of transfer learning. This entails the utilization of pre-existing, well-established models, specifically InceptionResNetv2 and Inceptionv3, to extract features and fine-tune selected layers to cater to the unique requirements of this specific diagnostic task. Concurrently, we also present a newly devised model, DiaCNN, which is tailored for the classification of eye diseases. To prove the efficacy of the proposed methodology, we leveraged the Ocular Disease Intelligent Recognition (ODIR) dataset, which comprises eight different eye disease categories. The results are promising. The InceptionResNetv2 model, incorporating transfer learning, registered an impressive 97.5% accuracy in both the training and testing phases. Its counterpart, the Inceptionv3 model, achieved an even more commendable 99.7% accuracy during training, and 97.5% during testing. Remarkably, the DiaCNN model showcased unparalleled precision, achieving 100% accuracy in training and 98.3% in testing. These figures represent a significant leap in classification accuracy when juxtaposed with existing state-of-the-art diagnostic methods. Such advancements hold immense promise for the future, emphasizing the potential of our proposed technique to revolutionize the accuracy of DR and other eye disease diagnoses. By facilitating earlier detection and more timely interventions, this approach stands poised to significantly reduce the incidence of blindness associated with DR, thus heralding a new era of improved patient outcomes. Therefore, this work, through its novel approach and stellar results, not only pushes the boundaries of DR diagnostic accuracy but also promises a transformative impact in early detection and intervention, aiming to substantially diminish DR-induced blindness and champion enhanced patient care.

Improved transfer learning using textural features conflation and dynamically fine-tuned layers

Transfer learning involves using previously learnt knowledge of a model task in addressing another task. However, this process works well when the tasks are closely related. It is, therefore, important to select data points that are closely relevant to the previous task and fine-tune the suitable pre-trained model's layers for effective transfer. This work utilises the least divergent textural features of the target datasets and pre-trained model's layers, minimising the lost knowledge during the transfer learning process. This study extends previous works on selecting data points with good textural features and dynamically selected layers using divergence measures by combining them into one model pipeline. Five pre-trained models are used: ResNet50, DenseNet169, InceptionV3, VGG16 and MobileNetV2 on nine datasets: CIFAR-10, CIFAR-100, MNIST, Fashion-MNIST, Stanford Dogs, Caltech 256, ISIC 2016, ChestX-ray8 and MIT Indoor Scenes. Experimental results show that data points with lower textural feature divergence and layers with more positive weights give better accuracy than other data points and layers. The data points with lower divergence give an average improvement of 3.54% to 6.75%, while the layers improve by 2.42% to 13.04% for the CIFAR-100 dataset. Combining the two methods gives an extra accuracy improvement of 1.56%. This combined approach shows that data points with lower divergence from the source dataset samples can lead to a better adaptation for the target task. The results also demonstrate that selecting layers with more positive weights reduces instances of trial and error in selecting fine-tuning layers for pre-trained models.

Deep neural network for food image classification and nutrient identification: A systematic review

Technology impacts human life in both the aspects such as positive and negative, which helps in better communication and eliminating geographical boundaries. However, social media and mobile devices may lead to severe health conditions such as sleep problems, depression, obesity, etc. A systematic review is conducted to analyze health issues by tracking food intake by considering positive aspects using Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) Guidelines. The major scientific databases (such as Web of Science, Scopus, and IEEE explore) are explored to search the image recognition and analysis articles. The search query is applied to the databases using keywords like "Food Image," "Food Image Classification," "Nutrient Identification," "Nutrient Estimation," and using "Machine Learning," etc. 771 articles are extracted from these databases, and 56 are identified for final consideration after rigorous screening. A few investigations are extracted based on available food image datasets, hyperparameters tuning, a technique used, performance metrics, and challenges of Food Image Classification (FIC). This study discusses different investigations with their proposed FIC and nutrient estimation solution. Finally, this intensive research presents a case study using FIC and object detection techniques to estimate nutrition with food image analysis.

The accuracy of artificial intelligence used for non-melanoma skin cancer diagnoses: a meta-analysis

BACKGROUND

With rising incidence of skin cancer and relatively increased mortality rates, an improved diagnosis of such a potentially fatal disease is of vital importance. Although frequently curable, it nevertheless places a considerable burden upon healthcare systems. Among the various types of skin cancers, non-melanoma skin cancer is most prevalent. Despite such prevalence and its associated cost, scant proof concerning the diagnostic accuracy via Artificial Intelligence (AI) for non-melanoma skin cancer exists. This study meta-analyzes the diagnostic test accuracy of AI used to diagnose non-melanoma forms of skin cancer, and it identifies potential covariates that account for heterogeneity between extant studies.

METHODS

Various electronic databases (Scopus, PubMed, ScienceDirect, SpringerLink, and Dimensions) were examined to discern eligible studies beginning from March 2022. Those AI studies predictive of non-melanoma skin cancer were included. Summary estimates of sensitivity, specificity, and area under receiver operating characteristic curves were used to evaluate diagnostic accuracy. The revised Quality Assessment of Diagnostic Studies served to assess any risk of bias.

RESULTS

A literature search produced 39 eligible articles for meta-analysis. The summary sensitivity, specificity, and area under receiver operating characteristic curve of AI for diagnosing non-melanoma skin cancer was 0.78, 0.98, & 0.97, respectively. Skin cancer typology, data sources, cross validation, ensemble models, types of techniques, pre-trained models, and image augmentation became significant covariates accounting for heterogeneity in terms of both sensitivity and/or specificity.

CONCLUSIONS

Meta-analysis results revealed that AI is predictive of non-melanoma with an acceptable performance, but sensitivity may become improved. Further, ensemble models and pre-trained models are employable to improve true positive rating.

Using deep learning for ultrasound images to diagnose carpal tunnel syndrome with high accuracy

Recently, deep learning (DL) algorithms have been adapted for the diagnosis of medical images. The purpose of this study was to detect image features using DL without measuring median nerve cross-sectional area (CSA) in ultrasonography (US) images of carpal tunnel syndrome (CTS) and calculate the diagnostic accuracy from the confusion matrix obtained. US images of 50 hands without CTS and 50 hands diagnosed with CTS were used in this study. The short-axis image of the median nerve was visualized, and 5000 images of both groups were prepared. Forty hands in each group were used as training data for the DL algorithm, while the remainder were used as test data. Transfer learning was performed using three pre-trained models. The confusion matrix and receiver operating characteristic curves were used to evaluate diagnostic accuracy. Furthermore, regions where DL was determined to be important were visualized. The highest score had an accuracy of 0.96, precision of 0.99 and recall of 0.94. Visualization of the important features revealed that the DL models focused on the epineurium of the median nerve and the surrounding soft tissue. The proposed technique enables the accurate prediction of CTS without measurement of the CSA.

Transfer learning based novel ensemble classifier for COVID-19 detection from chest CT-scans

Coronavirus Disease 2019 (COVID-19) is a deadly infection that affects the respiratory organs in humans as well as animals. By 2020, this disease turned out to be a pandemic affecting millions of individuals across the globe. Conducting rapid tests for a large number of suspects preventing the spread of the virus has become a challenge. In the recent past, several deep learning based approaches have been developed for automating the process of detecting COVID-19 infection from Lung Computerized Tomography (CT) scan images. However, most of them rely on a single model prediction for the final decision which may or may not be accurate. In this paper, we propose a novel ensemble approach that aggregates the strength of multiple deep neural network architectures before arriving at the final decision. We use various pre-trained models such as VGG16, VGG19, InceptionV3, ResNet50, ResNet50V2, InceptionResNetV2, Xception, and MobileNet and fine-tune them using Lung CT Scan images. All these trained models are further used to create a strong ensemble classifier that makes the final prediction. Our experiments exhibit that the proposed ensemble approach is superior to existing ensemble approaches and set state-of-the-art results for detecting COVID-19 infection from lung CT scan images.

A new composite approach for COVID-19 detection in X-ray images using deep features

The new type of coronavirus, COVID 19, appeared in China at the end of 2019. It has become a pandemic that is spreading all over the world in a very short time. The detection of this disease, which has serious health and socio-economic damages, is of vital importance. COVID-19 detection is performed by applying PCR and serological tests. Additionally, COVID detection is possible using X-ray and computed tomography images. Disease detection has an important position in scientific researches that includes artificial intelligence methods.

The combined models, which consist of different phases, are frequently used for classification problems. In this paper, a new combined approach is proposed to detect COVID-19 cases using deep features obtained from X-ray images. Two main variances of the approach can be presented as single layer-based (SLB) and feature fusion-based (FFB). SLB model consists of pre-processing, deep feature extraction, post-processing, and classification phases. On the other side, the FFB model consists of pre-processing, deep feature extraction, feature fusion, post-processing, and classification phases. Four different SLB and six different FFB models were developed according to the number and binary combination of layers used in the feature extraction phase. Each model is employed for binary and multi-class classification experiments.

According to experimental results, the accuracy performance for COVID-19 and no-findings classification of the proposed FFB3 model is 99.52%, which is better than the best performance accuracy (of 98.08%) in the literature. Concurrently, for multi-class classification, the proposed FFB3 model has an accuracy performance of 87.64% outperforming the best existing work (which reported an 87.02% classification performance). Various metrics, including sensitivity, specificity, precision, and F1-score metrics are used for performance analysis. For all performance metrics, the FFB3 model recorded a higher success rate than existing work in the literature. To the best of our knowledge, these accuracy rates are the best in the literature for the dataset and data split type (five-fold cross-validation).

Composite models (SLBs and FFBs), which are generated in this paper, are successful ways to detect COVID-19. Experimental results show that feature extraction, pre-processing, post-processing, and hyperparameter tuning are the steps are necessary to obtain a higher success. For prospective works, different types of pre-trained models and other hyperparameter tuning methods can be implemented.

Pre-trained convolutional neural networks as feature extractors toward improved malaria parasite detection in thin blood smear images

Malaria is a blood disease caused by the Plasmodium parasites transmitted through the bite of female Anopheles mosquito. Microscopists commonly examine thick and thin blood smears to diagnose disease and compute parasitemia. However, their accuracy depends on smear quality and expertise in classifying and counting parasitized and uninfected cells. Such an examination could be arduous for large-scale diagnoses resulting in poor quality. State-of-the-art image-analysis based computer-aided diagnosis (CADx) methods using machine learning (ML) techniques, applied to microscopic images of the smears using hand-engineered features demand expertise in analyzing morphological, textural, and positional variations of the region of interest (ROI). In contrast, Convolutional Neural Networks (CNN), a class of deep learning (DL) models promise highly scalable and superior results with end-to-end feature extraction and classification. Automated malaria screening using DL techniques could, therefore, serve as an effective diagnostic aid. In this study, we evaluate the performance of pre-trained CNN based DL models as feature extractors toward classifying parasitized and uninfected cells to aid in improved disease screening. We experimentally determine the optimal model layers for feature extraction from the underlying data. Statistical validation of the results demonstrates the use of pre-trained CNNs as a promising tool for feature extraction for this purpose.