Medical image retrieval using deep convolutional neural network

On image search in histopathology

Pathology images of histopathology can be acquired from camera-mounted microscopes or whole-slide scanners. Utilizing similarity calculations to match patients based on these images holds significant potential in research and clinical contexts. Recent advancements in search technologies allow for implicit quantification of tissue morphology across diverse primary sites, facilitating comparisons, and enabling inferences about diagnosis, and potentially prognosis, and predictions for new patients when compared against a curated database of diagnosed and treated cases. In this article, we comprehensively review the latest developments in image search technologies for histopathology, offering a concise overview tailored for computational pathology researchers seeking effective, fast, and efficient image search methods in their work.

Cross-modality sub-image retrieval using contrastive multimodal image representations

In tissue characterization and cancer diagnostics, multimodal imaging has emerged as a powerful technique. Thanks to computational advances, large datasets can be exploited to discover patterns in pathologies and improve diagnosis. However, this requires efficient and scalable image retrieval methods. Cross-modality image retrieval is particularly challenging, since images of similar (or even the same) content captured by different modalities might share few common structures. We propose a new application-independent content-based image retrieval (CBIR) system for reverse (sub-)image search across modalities, which combines deep learning to generate representations (embedding the different modalities in a common space) with robust feature extraction and bag-of-words models for efficient and reliable retrieval. We illustrate its advantages through a replacement study, exploring a number of feature extractors and learned representations, as well as through comparison to recent (cross-modality) CBIR methods. For the task of (sub-)image retrieval on a (publicly available) dataset of brightfield and second harmonic generation microscopy images, the results show that our approach is superior to all tested alternatives. We discuss the shortcomings of the compared methods and observe the importance of equivariance and invariance properties of the learned representations and feature extractors in the CBIR pipeline. Code is available at: https://github.com/MIDA-group/CrossModal_ImgRetrieval .

Multiple semantic X-ray medical image retrieval using efficient feature vector extracted by FPN

OBJECTIVE

Content-based medical image retrieval (CBMIR) has become an important part of computer-aided diagnostics (CAD) systems. The complex medical semantic information inherent in medical images is the most difficult part to improve the accuracy of image retrieval. Highly expressive feature vectors play a crucial role in the search process. In this paper, we propose an effective deep convolutional neural network (CNN) model to extract concise feature vectors for multiple semantic X-ray medical image retrieval.

METHODS

We build a feature pyramid based CNN model with ResNet50V2 backbone to extract multi-level semantic information. And we use the well-known public multiple semantic annotated X-ray medical image data set IRMA to train and test the proposed model.

RESULTS

Our method achieves an IRMA error of 32.2, which is the best score compared to the existing literature on this dataset.

CONCLUSIONS

The proposed CNN model can effectively extract multi-level semantic information from X-ray medical images. The concise feature vectors can improve the retrieval accuracy of multi-semantic and unevenly distributed X-ray medical images.

Deep Attention Fusion Hashing (DAFH) Model for Medical Image Retrieval

In medical image retrieval, accurately retrieving relevant images significantly impacts clinical decision making and diagnostics. Traditional image-retrieval systems primarily rely on single-dimensional image data, while current deep-hashing methods are capable of learning complex feature representations. However, retrieval accuracy and efficiency are hindered by diverse modalities and limited sample sizes.

OBJECTIVE

To address this, we propose a novel deep learning-based hashing model, the Deep Attention Fusion Hashing (DAFH) model, which integrates advanced attention mechanisms with medical imaging data.

METHODS

The DAFH model enhances retrieval performance by integrating multi-modality medical imaging data and employing attention mechanisms to optimize the feature extraction process. Utilizing multimodal medical image data from the Cancer Imaging Archive (TCIA), this study constructed and trained a deep hashing network that achieves high-precision classification of various cancer types.

RESULTS

At hash code lengths of 16, 32, and 48 bits, the model respectively attained Mean Average Precision (MAP@10) values of 0.711, 0.754, and 0.762, highlighting the potential and advantage of the DAFH model in medical image retrieval.

CONCLUSIONS

The DAFH model demonstrates significant improvements in the efficiency and accuracy of medical image retrieval, proving to be a valuable tool in clinical settings.

Medical Image Retrieval via Nearest Neighbor Search on Pre-trained Image Features

Nearest neighbor search, also known as NNS, is a technique used to locate the points in a high-dimensional space closest to a given query point. This technique has multiple applications in medicine, such as searching large medical imaging databases, disease classification, and diagnosis. However, when the number of points is significantly large, the brute-force approach for finding the nearest neighbor becomes computationally infeasible. Therefore, various approaches have been developed to make the search faster and more efficient to support the applications. With a focus on medical imaging, this paper proposes DenseLinkSearch (DLS), an effective and efficient algorithm that searches and retrieves the relevant images from heterogeneous sources of medical images. Towards this, given a medical database, the proposed algorithm builds an index that consists of pre-computed links of each point in the database. The search algorithm utilizes the index to efficiently traverse the database in search of the nearest neighbor. We also explore the role of medical image feature representation in content-based medical image retrieval tasks. We propose a Transformer-based feature representation technique that outperformed the existing pre-trained Transformer-based approaches on benchmark medical image retrieval datasets. We extensively tested the proposed NNS approach and compared the performance with state-of-the-art NNS approaches on benchmark datasets and our created medical image datasets. The proposed approach outperformed the existing approaches in terms of retrieving accurate neighbors and retrieval speed. In comparison to the existing approximate NNS approaches, our proposed DLS approach outperformed them in terms of lower average time per query and ≥ 99% R@10 on 11 out of 13 benchmark datasets. We also found that the proposed medical feature representation approach is better for representing medical images compared to the existing pre-trained image models. The proposed feature extraction strategy obtained an improvement of 9.37%, 7.0%, and 13.33% in terms of P@5, P@10, and P@20, respectively, in comparison to the best-performing pre-trained image model. The source code and datasets of our experiments are available at https://github.com/deepaknlp/DLS.

OBIA: An Open Biomedical Imaging Archive

With the development of artificial intelligence (AI) technologies, biomedical imaging data play an important role in scientific research and clinical application, but the available resources are limited. Here we present Open Biomedical Imaging Archive (OBIA), a repository for archiving biomedical imaging and related clinical data. OBIA adopts five data objects (Collection, Individual, Study, Series, and Image) for data organization, and accepts the submission of biomedical images of multiple modalities, organs, and diseases. In order to protect personal privacy, OBIA has formulated a unified de-identification and quality control process. In addition, OBIA provides friendly and intuitive web interfaces for data submission, browsing, and retrieval, as well as image retrieval. As of September 2023, OBIA has housed data for a total of 937 individuals, 4136 studies, 24,701 series, and 1,938,309 images covering 9 modalities and 30 anatomical sites. Collectively, OBIA provides a reliable platform for biomedical imaging data management and offers free open access to all publicly available data to support research activities throughout the world. OBIA can be accessed at https://ngdc.cncb.ac.cn/obia.

Multi-modal medical image classification using deep residual network and genetic algorithm

Artificial intelligence (AI) development across the health sector has recently been the most crucial. Early medical information, identification, diagnosis, classification, then analysis, along with viable remedies, are always beneficial developments. Precise and consistent image classification has critical in diagnosing and tactical decisions for healthcare. The core issue with image classification has become the semantic gap. Conventional machine learning algorithms for classification rely mainly on low-level but rather high-level characteristics, employ some handmade features to close the gap, but force intense feature extraction as well as classification approaches. Deep learning is a powerful tool with considerable advances in recent years, with deep convolution neural networks (CNNs) succeeding in image classification. The main goal is to bridge the semantic gap and enhance the classification performance of multi-modal medical images based on the deep learning-based model ResNet50. The data set included 28378 multi-modal medical images to train and validate the model. Overall accuracy, precision, recall, and F1-score evaluation parameters have been calculated. The proposed model classifies medical images more accurately than other state-of-the-art methods. The intended research experiment attained an accuracy level of 98.61%. The suggested study directly benefits the health service.

An interpretable and interactive deep learning algorithm for a clinically applicable retinal fundus diagnosis system by modelling finding-disease relationship

The identification of abnormal findings manifested in retinal fundus images and diagnosis of ophthalmic diseases are essential to the management of potentially vision-threatening eye conditions. Recently, deep learning-based computer-aided diagnosis systems (CADs) have demonstrated their potential to reduce reading time and discrepancy amongst readers. However, the obscure reasoning of deep neural networks (DNNs) has been the leading cause to reluctance in its clinical use as CAD systems. Here, we present a novel architectural and algorithmic design of DNNs to comprehensively identify 15 abnormal retinal findings and diagnose 8 major ophthalmic diseases from macula-centered fundus images with the accuracy comparable to experts. We then define a notion of counterfactual attribution ratio (CAR) which luminates the system's diagnostic reasoning, representing how each abnormal finding contributed to its diagnostic prediction. By using CAR, we show that both quantitative and qualitative interpretation and interactive adjustment of the CAD result can be achieved. A comparison of the model's CAR with experts' finding-disease diagnosis correlation confirms that the proposed model identifies the relationship between findings and diseases similarly as ophthalmologists do.

MR-Class: A Python Tool for Brain MR Image Classification Utilizing One-vs-All DCNNs to Deal with the Open-Set Recognition Problem

Background: MR image classification in datasets collected from multiple sources is complicated by inconsistent and missing DICOM metadata. Therefore, we aimed to establish a method for the efficient automatic classification of MR brain sequences. Methods: Deep convolutional neural networks (DCNN) were trained as one-vs-all classifiers to differentiate between six classes: T1 weighted (w), contrast-enhanced T1w, T2w, T2w-FLAIR, ADC, and SWI. Each classifier yields a probability, allowing threshold-based and relative probability assignment while excluding images with low probability (label: unknown, open-set recognition problem). Data from three high-grade glioma (HGG) cohorts was assessed; C1 (320 patients, 20,101 MRI images) was used for training, while C2 (197, 11,333) and C3 (256, 3522) were for testing. Two raters manually checked images through an interactive labeling tool. Finally, MR-Class’ added value was evaluated via radiomics model performance for progression-free survival (PFS) prediction in C2, utilizing the concordance index (C-I). Results: Approximately 10% of annotation errors were observed in each cohort between the DICOM series descriptions and the derived labels. MR-Class accuracy was 96.7% [95% Cl: 95.8, 97.3] for C2 and 94.4% [93.6, 96.1] for C3. A total of 620 images were misclassified; manual assessment of those frequently showed motion artifacts or alterations of anatomy by large tumors. Implementation of MR-Class increased the PFS model C-I by 14.6% on average, compared to a model trained without MR-Class. Conclusions: We provide a DCNN-based method for the sequence classification of brain MR images and demonstrate its usability in two independent HGG datasets.

Steel Strip Defect Sample Generation Method Based on Fusible Feature GAN Model under Few Samples

Due to the shortage of defect samples and the high cost of labelling during the process of hot-rolled strip production in the metallurgical industry, it is difficult to obtain a large quantity of defect data with diversity, which seriously affects the identification accuracy of different types of defects on the steel surface. To address the problem of insufficient defect sample data in the task of strip steel defect identification and classification, this paper proposes the Strip Steel Surface Defect-ConSinGAN (SDE-ConSinGAN) model for strip steel defect identification which is based on a single-image model trained by the generative adversarial network (GAN) and which builds a framework of image-feature cutting and splicing. The model aims to reduce training time by dynamically adjusting the number of iterations for different training stages. The detailed defect features of training samples are highlighted by introducing a new size-adjustment function and increasing the channel attention mechanism. In addition, real image features will be cut and synthesized to obtain new images with multiple defect features for training. The emergence of new images is able to richen generated samples. Eventually, the generated simulated samples can be directly used in deep-learning-based automatic classification of surface defects in cold-rolled thin strips. The experimental results show that, when SDE-ConSinGAN is used to enrich the image dataset, the generated defect images have higher quality and more diversity than the current methods do.

Activation Functions for Convolutional Neural Networks: Proposals and Experimental Study

Activation functions lie at the core of every neural network model from shallow to deep convolutional neural networks. Their properties and characteristics shape the output range of each layer and, thus, their capabilities. Modern approaches rely mostly on a single function choice for the whole network, usually ReLU or other similar alternatives. In this work, we propose two new activation functions and analyze their properties and compare them with 17 different function proposals from recent literature on six distinct problems with different characteristics. The objective is to shed some light on their comparative performance. The results show that the proposed functions achieved better performance than the most commonly used ones.

Efficient U-Net Architecture with Multiple Encoders and Attention Mechanism Decoders for Brain Tumor Segmentation

The brain is the center of human control and communication. Hence, it is very important to protect it and provide ideal conditions for it to function. Brain cancer remains one of the leading causes of death in the world, and the detection of malignant brain tumors is a priority in medical image segmentation. The brain tumor segmentation task aims to identify the pixels that belong to the abnormal areas when compared to normal tissue. Deep learning has shown in recent years its power to solve this problem, especially the U-Net-like architectures. In this paper, we proposed an efficient U-Net architecture with three different encoders: VGG-19, ResNet50, and MobileNetV2. This is based on transfer learning followed by a bidirectional features pyramid network applied to each encoder to obtain more spatial pertinent features. Then, we fused the feature maps extracted from the output of each network and merged them into our decoder with an attention mechanism. The method was evaluated on the BraTS 2020 dataset to segment the different types of tumors and the results show a good performance in terms of dice similarity, with coefficients of 0.8741, 0.8069, and 0.7033 for the whole tumor, core tumor, and enhancing tumor, respectively.

A novel denoising method for CT images based on U-net and multi-attention

Reducing the radiation dose may lead to increased noise in medical computed tomography (CT), which can adversely affect the radiologists' judgment. Many efforts have been devoted to the denoising of low-dose CT (LDCT) images. However, it is often observed that denoised medical images usually lose some important clinical lesion edge information and may affect doctors' clinical diagnosis. For a denoising neural network, it is expected that the neural network can well retain the detailed features and make the network more anthropomorphic, and to simulate the attention mechanism of observation, being a valuable feature of the thinking process of human brain. Based on U-network (U-Net) and multi-attention mechanism, a novel denoising method for medical CT images is proposed in this study. To obtain different feature information in CT images, three attention modules are proposed in our method. The local attention module is developed to localize the surrounding information of the feature map and calculate each pixel from the context extracted from the feature map. The multi-feature channel attention module can automatically learn and extract features, suppress some invalid information and add different weights to each channel in the feature map according to different tasks. The hierarchical attention module allows the deep neural network to extract a large amount of feature information. This study also introduces an enhanced learning module to learn and retain the detail information in the image by stacking multi-layer convolution layer, batch normalization (BN) layer and activation function layer to increase the network depth. Experimental studies are conducted, and comparisons with the state-of-the-art networks are made, and the results demonstrate that the developed method can effectively remove the noise in CT images and improve the image quality in the evaluation metrics of peak signal to noise ratio (PSNR) and structural similarity (SSIM). Our method achieved 34.7329 of PSNR and 0.9293 of SSIM for σ = 10 on the QIN_LUNG_CT dataset, and achieved 28.9163 of PSNR and 0.8602 of SSIM on the Mayo Clinic LDCT Grand Challenge dataset.

Computer-aided diagnosis through medical image retrieval in radiology

Currently, radiologists face an excessive workload, which leads to high levels of fatigue, and consequently, to undesired diagnosis mistakes. Decision support systems can be used to prioritize and help radiologists making quicker decisions. In this sense, medical content-based image retrieval systems can be of extreme utility by providing well-curated similar examples. Nonetheless, most medical content-based image retrieval systems work by finding the most similar image, which is not equivalent to finding the most similar image in terms of disease and its severity. Here, we propose an interpretability-driven and an attention-driven medical image retrieval system. We conducted experiments in a large and publicly available dataset of chest radiographs with structured labels derived from free-text radiology reports (MIMIC-CXR-JPG). We evaluated the methods on two common conditions: pleural effusion and (potential) pneumonia. As ground-truth to perform the evaluation, query/test and catalogue images were classified and ordered by an experienced board-certified radiologist. For a profound and complete evaluation, additional radiologists also provided their rankings, which allowed us to infer inter-rater variability, and yield qualitative performance levels. Based on our ground-truth ranking, we also quantitatively evaluated the proposed approaches by computing the normalized Discounted Cumulative Gain (nDCG). We found that the Interpretability-guided approach outperforms the other state-of-the-art approaches and shows the best agreement with the most experienced radiologist. Furthermore, its performance lies within the observed inter-rater variability.

Explainable, trustworthy, and ethical machine learning for healthcare: A survey

With the advent of machine learning (ML) and deep learning (DL) empowered applications for critical applications like healthcare, the questions about liability, trust, and interpretability of their outputs are raising. The black-box nature of various DL models is a roadblock to clinical utilization. Therefore, to gain the trust of clinicians and patients, we need to provide explanations about the decisions of models. With the promise of enhancing the trust and transparency of black-box models, researchers are in the phase of maturing the field of eXplainable ML (XML). In this paper, we provided a comprehensive review of explainable and interpretable ML techniques for various healthcare applications. Along with highlighting security, safety, and robustness challenges that hinder the trustworthiness of ML, we also discussed the ethical issues arising because of the use of ML/DL for healthcare. We also describe how explainable and trustworthy ML can resolve all these ethical problems. Finally, we elaborate on the limitations of existing approaches and highlight various open research problems that require further development.

A Parallel Classification Model for Marine Mammal Sounds Based on Multi-Dimensional Feature Extraction and Data Augmentation

Due to the poor visibility of the deep-sea environment, acoustic signals are often collected and analyzed to explore the behavior of marine species. With the progress of underwater signal-acquisition technology, the amount of acoustic data obtained from the ocean has exceeded the limit that human can process manually, so designing efficient marine-mammal classification algorithms has become a research hotspot. In this paper, we design a classification model based on a multi-channel parallel structure, which can process multi-dimensional acoustic features extracted from audio samples, and fuse the prediction results of different channels through a trainable full connection layer. It uses transfer learning to obtain faster convergence speed, and introduces data augmentation to improve the classification accuracy. The k-fold cross-validation method was used to segment the data set to comprehensively evaluate the prediction accuracy and robustness of the model. The evaluation results showed that the model can achieve a mean accuracy of 95.21% while maintaining a standard deviation of 0.65%. There was excellent consistency in performance over multiple tests.

ExpertosLF: dynamic late fusion of CBIR systems using online learning with relevance feedback

One of the main challenges in CBIR systems is to choose discriminative and compact features, among dozens, to represent the images under comparison. Over the years, a great effort has been made to combine multiple features, mainly using early, late, and hierarchical fusion techniques. Unveiling the perfect combination of features is highly domain-specific and dependent on the type of image. Thus, the process of designing a CBIR system for new datasets or domains involves a huge experimentation overhead, leading to multiple fine-tuned CBIR systems. It would be desirable to dynamically find the best combination of CBIR systems without needing to go through such extensive experimentation and without requiring previous domain knowledge. In this paper, we propose ExpertosLF, a model-agnostic interpretable late fusion technique based on online learning with expert advice, which dynamically combines CBIR systems without knowing a priori which ones are the best for a given domain. At each query, ExpertosLF takes advantage of user's feedback to determine each CBIR contribution in the ensemble for the following queries. ExpertosLF produces an interpretable ensemble that is independent of the dataset and domain. Moreover, ExpertosLF is designed to be modular, and scalable. Experiments on 13 benchmark datasets from the Biomedical, Real, and Sketch domains revealed that: (i) ExpertosLF surpasses the performance of state of the art late-fusion techniques; (ii) it successfully and quickly converges to the performance of the best CBIR sets across domains without any previous domain knowledge (in most cases, fewer than 25 queries need to receive human feedback).

Low-Light Image Enhancement via Retinex-Style Decomposition of Denoised Deep Image Prior

Low-light images are a common phenomenon when taking photos in low-light environments with inappropriate camera equipment, leading to shortcomings such as low contrast, color distortion, uneven brightness, and high loss of detail. These shortcomings are not only subjectively annoying but also affect the performance of many computer vision systems. Enhanced low-light images can be better applied to image recognition, object detection and image segmentation. This paper proposes a novel RetinexDIP method to enhance images. Noise is considered as a factor in image decomposition using deep learning generative strategies. The involvement of noise makes the image more real, weakens the coupling relationship between the three components, avoids overfitting, and improves generalization. Extensive experiments demonstrate that our method outperforms existing methods qualitatively and quantitatively.

An active learning method for diabetic retinopathy classification with uncertainty quantification

In recent years, deep learning (DL) techniques have provided state-of-the-art performance in medical imaging. However, good quality (annotated) medical data is in general hard to find due to the usually high cost of medical images, limited availability of expert annotators (e.g., radiologists), and the amount of time required for annotation. In addition, DL is data-hungry and its training requires extensive computational resources. Furthermore, DL being a black-box method lacks transparency on its inner working and lacks fundamental understanding behind decisions made by the model and consequently, this notion enhances the uncertainty on its predictions. To this end, we address these challenges by proposing a hybrid model, which uses a Bayesian convolutional neural network (BCNN) for uncertainty quantification, and an active learning approach for annotating the unlabeled data. The BCNN is used as a feature descriptor and these features are then used for training a model, in an active learning setting. We evaluate the proposed framework for diabetic retinopathy classification problem and demonstrate state-of-the-art performance in terms of different metrics.

On the use of multi–objective evolutionary classifiers for breast cancer detection

Purpose

Breast cancer is one of the most common tumours in women, nevertheless, it is also one of the cancers that is most usually treated. As a result, early detection is critical, which can be accomplished by routine mammograms. This paper aims to describe, analyze, compare and evaluate three image descriptors involved in classifying breast cancer images from four databases.

Approach

Multi–Objective Evolutionary Algorithms (MOEAs) prove themselves as being efficient methods for selection and classification problems. This paper aims to study combinations of well–known classification objectives in order to compare the results of their application in solving very specific learning problems. The experimental results undergo empirical analysis which is supported by a statistical approach. The results are illustrated on a collection of medical image databases, but with a focus on the MOEAs’ performance in terms of several well–known measures. The databases were chosen specifically to feature reliable human annotations, so as to measure the correlation between the gold standard classifications and the various MOEA classifications.

Results

We have seen how different statistical tests rank one algorithm over the others in our set as being better. These findings are unsurprising, revealing that there is no single gold standard for comparing diverse techniques or evolutionary algorithms. Furthermore, building meta-classifiers and evaluating them using a single, favorable metric is both extremely unwise and unsatisfactory, as the impact is to skew the results.

Conclusions

The best method to address these flaws is to select the right set of objectives and criteria. Using accuracy-related objectives, for example, is directly linked to maximizing the number of true positives. If, on the other hand, accuracy is chosen as the generic metric, the primary classification goal is shifted to increasing the positively categorized data points.

S. Content-based medical image retrieval system for lung diseases using deep CNNs

Content-based image retrieval (CBIR) systems are designed to retrieve images that are relevant, based on detailed analysis of latent image characteristics, thus eliminating the dependency of natural language tags, text descriptions, or keywords associated with the images. A CBIR system maintains high-level image visuals in the form of feature vectors, which the retrieval engine leverages for similarity-based matching and ranking for a given query image. In this paper, a CBIR system is proposed for the retrieval of medical images (CBMIR) for enabling the early detection and classification of lung diseases based on lung X-ray images. The proposed CBMIR system is built on the predictive power of deep neural models for the identification and classification of disease-specific features using transfer learning based models trained on standard COVID-19 Chest X-ray image datasets. Experimental evaluation on the standard dataset revealed that the proposed approach achieved an improvement of 49.71% in terms of precision, averaging across various distance metrics. Also, an improvement of 26.55% was observed in the area under precision-recall curve (AUPRC) values across all subclasses.

Knowledge Graph Applications in Medical Imaging Analysis: A Scoping Review

Background

There is an increasing trend to represent domain knowledge in structured graphs, which provide efficient knowledge representations for many downstream tasks. Knowledge graphs are widely used to model prior knowledge in the form of nodes and edges to represent semantically connected knowledge entities, which several works have adopted into different medical imaging applications.

Methods

We systematically searched over five databases to find relevant articles that applied knowledge graphs to medical imaging analysis. After screening, evaluating, and reviewing the selected articles, we performed a systematic analysis.

Results

We looked at four applications in medical imaging analysis, including disease classification, disease localization and segmentation, report generation, and image retrieval. We also identified limitations of current work, such as the limited amount of available annotated data and weak generalizability to other tasks. We further identified the potential future directions according to the identified limitations, including employing semisupervised frameworks to alleviate the need for annotated data and exploring task-agnostic models to provide better generalizability.

Conclusions

We hope that our article will provide the readers with aggregated documentation of the state-of-the-art knowledge graph applications for medical imaging to encourage future research.

A fully-convolutional residual encoder-decoder neural network to localize breast cancer on histopathology images

Cancer detection in its early stages may allow patients to receive the proper treatment and save lives along with recovering the routine lifestyles. Breast cancer is of the top leading causes of mortality among women all around the globe. A source to find these cancerous nuclei is through analyzing histopathology images. These images, however, are very complex and large. Thus, locating the cancerous nuclei in them is very challenging. Hence, if an expert fails to diagnose their patients via these images, the situation may be exacerbated. Therefore, this study aims to introduce a method to mask as many cancer nuclei on histopathology images as possible with a high visual aesthetic to make them distinguishable by experts easily. A tailored residual fully convolutional encoder-decoder neural network based on end-to-end learning is proposed to issue the matter. The proposed method is evaluated quantitatively and qualitatively on ER + BCa H&E-stained dataset. The average detection accuracy achieved by the method is 98.61%, which is much better than that of competitors.

Automated Facial Expression Recognition Framework Using Deep Learning

Facial expression is one of the most significant elements which can tell us about the mental state of any person. A human can convey approximately 55% of information nonverbally and the remaining almost 45% through verbal communication. Automatic facial expression recognition is presently one of the most difficult tasks in the computer science field. Applications of facial expression recognition (FER) are not just limited to understanding human behavior and monitoring person's mood and the mental state of humans. It is also penetrating into other fields such as criminology, holographic, smart healthcare systems, security systems, education, robotics, entertainment, and stress detection. Currently, facial expressions are playing an important role in medical sciences, particularly helping the patients with bipolar disease, whose mood changes very frequently. In this study, an algorithm, automated framework for facial detection using a convolutional neural network (FD-CNN) is proposed with four convolution layers and two hidden layers to improve accuracy. An extended Cohn-Kanade (CK+) dataset is used that includes facial images of different males and females with expressions such as anger, fear, disgust, contempt, neutral, happy, sad, and surprise. In this study, FD-CNN is performed in three major steps that include preprocessing, feature extraction, and classification. By using this proposed method, an accuracy of 94% is obtained in FER. In order to validate the proposed algorithm, K-fold cross-validation is performed. After validation, sensitivity and specificity are calculated which are 94.02% and 99.14%, respectively. Furthermore, the f1 score, recall, and precision are calculated to validate the quality of the model which is 84.07%, 78.22%, and 94.09%, respectively.

AutoRet: A Self-Supervised Spatial Recurrent Network for Content-Based Image Retrieval

Image retrieval techniques are becoming famous due to the vast availability of multimedia data. The present image retrieval system performs excellently on labeled data. However, often, data labeling becomes costly and sometimes impossible. Therefore, self-supervised and unsupervised learning strategies are currently becoming illustrious. Most of the self/unsupervised strategies are sensitive to the number of classes and can not mix labeled data on availability. In this paper, we introduce AutoRet, a deep convolutional neural network (DCNN) based self-supervised image retrieval system. The system is trained on pairwise constraints. Therefore, it can work in self-supervision and can also be trained on a partially labeled dataset. The overall strategy includes a DCNN that extracts embeddings from multiple patches of images. Further, the embeddings are fused for quality information used for the image retrieval process. The method is benchmarked with three different datasets. From the overall benchmark, it is evident that the proposed method works better in a self-supervised manner. In addition, the evaluation exhibits the proposed method’s performance to be highly convincing while a small portion of labeled data are mixed on availability.

Constructing a mobile visual search framework for Dunhuang murals based on fine-tuned CNN and ontology semantic distance

Purpose

Dunhuang murals are rich in cultural and artistic value. The purpose of this paper is to construct a novel mobile visual search (MVS) framework for Dunhuang murals, enabling users to efficiently search for similar, relevant and diversified images.

Design/methodology/approach

The convolutional neural network (CNN) model is fine-tuned in the data set of Dunhuang murals. Image features are extracted through the fine-tuned CNN model, and the similarities between different candidate images and the query image are calculated by the dot product. Then, the candidate images are sorted by similarity, and semantic labels are extracted from the most similar image. Ontology semantic distance (OSD) is proposed to match relevant images using semantic labels. Furthermore, the improved DivScore is introduced to diversify search results.

Findings

The results illustrate that the fine-tuned ResNet152 is the best choice to search for similar images at the visual feature level, and OSD is the effective method to search for the relevant images at the semantic level. After re-ranking based on DivScore, the diversification of search results is improved.

Originality/value

This study collects and builds the Dunhuang mural data set and proposes an effective MVS framework for Dunhuang murals to protect and inherit Dunhuang cultural heritage. Similar, relevant and diversified Dunhuang murals are searched to meet different demands.

Efficient Quality Control with Mixed CT and CTA Datasets

Deep learning promises the extraction of valuable information from traumatic brain injury (TBI) datasets and depends on efficient navigation when using large-scale mixed computed tomography (CT) datasets from clinical systems. To ensure a cleaner signal while training deep learning models, removal of computed tomography angiography (CTA) and scans with streaking artifacts is sensible. On massive datasets of heterogeneously sized scans, time-consuming manual quality assurance (QA) by visual inspection is still often necessary, despite the expectation of CTA annotation (artifact annotation is not expected). We propose an automatic QA approach for retrieving CT scans without artifacts by representing 3D scans as 2D axial slice montages and using a multi-headed convolutional neural network to detect CT vs CTA and artifact vs no artifact. We sampled 848 scans from a mixed CT dataset of TBI patients and performed 4-fold stratified cross-validation on 698 montages followed by an ablation experiment-150 stratified montages were withheld for external validation evaluation. Aggregate AUC for our main model was 0.978 for CT detection, 0.675 for artifact detection during cross-validation and 0.965 for CT detection, 0.698 for artifact detection on the external validation set, while the ablated model showed 0.946 for CT detection, 0.735 for artifact detection during cross-validation and 0.937 for CT detection, 0.708 for artifact detection on the external validation set. While our approach is successful for CT detection, artifact detection performance is potentially depressed due to the heterogeneity of present streaking artifacts and a suboptimal number of artifact scans in our training data.

Content Based Medical Image Retrieval for Accurate Disease Diagnosis

Introduction:

Content Based Image Retrieval (CBIR) system is an innovative technology to retrieve images from various media types. One of the CBIR applications is Content Based Medical Image Retrieval (CBMIR). The image retrieval system retrieves the most similar images from the historical cases, and such systems can only support the physician's decision to diagnose a disease. To extract the useful features from the query image for linking similar types of images is the major challenge in the CBIR domain. The Convolution Neural Network (CNN) can overcome the drawbacks of traditional algorithms, dependent on the low-level feature extraction technique.

Objective:

The objective of the study is to develop a CNN model with a minimum number of convolution layers and to get the maximum possible accuracy for the CBMIR system. The minimum number of convolution layers reduces the number of mathematical operations and the time for the model's training. It also reduces the number of training parameters, like weights and bias. Thus, it reduces the memory requirement for the model storage. This work mainly focused on developing an optimized CNN model for the CBMIR system. Such systems can only support the physicians' decision to diagnose a disease from the images and retrieve the relevant cases to help the doctor decide the precise treatment.

Methods:

The deep learning-based model is proposed in this paper. The experiment is done with several convolution layers and various optimizers to get the maximum accuracy with a minimum number of convolution layers. Thus, the ten-layer CNN model is developed from scratch and used to derive the training and testing images' features and classify the test image. Once the image class is identified, the most relevant images are determined based on the Euclidean distance between the query features and database features of the identified class. Based on this distance, the most relevant images are displayed from the respective class of images. The general dataset CIFAR10, which has 60,000 images of 10 different classes, and the medical dataset IRMA, which has 2508 images of 9 various classes, have been used to analyze the proposed method. The proposed model is also applied for the medical x-ray image dataset of chest disease and compared with the other pre-trained models.

Results:

The accuracy and the average precision rate are the measurement parameters utilized to compare the proposed model with different machine learning techniques. The accuracy of the proposed model for the CIFAR10 dataset is 93.9%, which is better than the state-of-the-art methods. After the success for the general dataset, the model is also tested for the medical dataset. For the x-ray images of the IRMA dataset, it is 86.53%, which is better than the different pre-trained model results. The model is also tested for the other x-ray dataset, which is utilized to identify chest-related disease. The average precision rate for such a dataset is 97.25%. Also, the proposed model fulfills the major challenge of the semantic gap. The semantic gap of the proposed model for the chest disease dataset is 2.75%, and for the IRMA dataset, it is 13.47%. Also, only ten convolution layers are utilized in the proposed model, which is very small in number compared to the other pre-trained models.

Conclusion:

The proposed technique shows remarkable improvement in performance metrics over CNN-based state-of-the-art methods. It also offers a significant improvement in performance metrics over different pre-trained models for the two different medical x-ray image datasets.

Content Based Medical Image Retrieval for Accurate Disease Diagnosis

Introduction:

Content Based Image Retrieval (CBIR) system is an innovative technology to retrieve images from various media types. One of the CBIR applications is Content Based Medical Image Retrieval (CBMIR). The image retrieval system retrieves the most similar images from the historical cases, and such systems can only support the physician's decision to diagnose a disease. To extract the useful features from the query image for linking similar types of images is the major challenge in the CBIR domain. The Convolution Neural Network (CNN) can overcome the drawbacks of traditional algorithms, dependent on the low-level feature extraction technique.

Objective:

The objective of the study is to develop a CNN model with a minimum number of convolution layers and to get the maximum possible accuracy for the CBMIR system. The minimum number of convolution layers reduces the number of mathematical operations and the time for the model's training. It also reduces the number of training parameters, like weights and bias. Thus, it reduces the memory requirement for the model storage. This work mainly focused on developing an optimized CNN model for the CBMIR system. Such systems can only support the physicians' decision to diagnose a disease from the images and retrieve the relevant cases to help the doctor decide the precise treatment.

Methods:

The deep learning-based model is proposed in this paper. The experiment is done with several convolution layers and various optimizers to get the maximum accuracy with a minimum number of convolution layers. Thus, the ten-layer CNN model is developed from scratch and used to derive the training and testing images' features and classify the test image. Once the image class is identified, the most relevant images are determined based on the Euclidean distance between the query features and database features of the identified class. Based on this distance, the most relevant images are displayed from the respective class of images. The general dataset CIFAR10, which has 60,000 images of 10 different classes, and the medical dataset IRMA, which has 2508 images of 9 various classes, have been used to analyze the proposed method. The proposed model is also applied for the medical x-ray image dataset of chest disease and compared with the other pre-trained models.

Results:

The accuracy and the average precision rate are the measurement parameters utilized to compare the proposed model with different machine learning techniques. The accuracy of the proposed model for the CIFAR10 dataset is 93.9%, which is better than the state-of-the-art methods. After the success for the general dataset, the model is also tested for the medical dataset. For the x-ray images of the IRMA dataset, it is 86.53%, which is better than the different pre-trained model results. The model is also tested for the other x-ray dataset, which is utilized to identify chest-related disease. The average precision rate for such a dataset is 97.25%. Also, the proposed model fulfills the major challenge of the semantic gap. The semantic gap of the proposed model for the chest disease dataset is 2.75%, and for the IRMA dataset, it is 13.47%. Also, only ten convolution layers are utilized in the proposed model, which is very small in number compared to the other pre-trained models.

Conclusion:

The proposed technique shows remarkable improvement in performance metrics over CNN-based state-of-the-art methods. It also offers a significant improvement in performance metrics over different pre-trained models for the two different medical x-ray image datasets.

Brain Tumor Segmentation Based on Deep Learning's Feature Representation

Brain tumor is considered as one of the most serious causes of death in the world. Thus, it is very important to detect it as early as possible. In order to predict and segment the tumor, many approaches have been proposed. However, they suffer from different problems such as the necessity of the intervention of a specialist, the long required run-time and the choice of the appropriate feature extractor. To address these issues, we proposed an approach based on convolution neural network architecture aiming at predicting and segmenting simultaneously a cerebral tumor. The proposal was divided into two phases. Firstly, aiming at avoiding the use of the labeled image that implies a subject intervention of the specialist, we used a simple binary annotation that reflects the existence of the tumor or not. Secondly, the prepared image data were fed into our deep learning model in which the final classification was obtained; if the classification indicated the existence of the tumor, the brain tumor was segmented based on the feature representations generated by the convolutional neural network architectures. The proposed method was trained on the BraTS 2017 dataset with different types of gliomas. The achieved results show the performance of the proposed approach in terms of accuracy, precision, recall and Dice similarity coefficient. Our model showed an accuracy of 91% in tumor classification and a Dice similarity coefficient of 82.35% in tumor segmentation.

A Similarity Measure of Histopathology Images by Deep Embeddings

Histopathology digital scans are large-size images that contain valuable information at the pixel level. Contentbased comparison of these images is a challenging task. This study proposes a content-based similarity measure for highresolution gigapixel histopathology images. The proposed similarity measure is an expansion of cosine vector similarity to a matrix. Each image is divided into same-size patches with a meaningful amount of information (i.e., contained enough tissue). The similarity is measured by the extraction of patchlevel deep embeddings of the last pooling layer of a pre-trained deep model at four different magnification levels, namely, 1x, 2.5x, 5x, and 10x magnifications. In addition, for faster measurement, embedding reduction is investigated. Finally, to assess the proposed method, an image search method is implemented. Results show that the similarity measure represents the slide labels with a maximum accuracy of 93.18% for top-5 search at 5x magnification.

Epileptic Spike Detection Using Neural Networks with Linear-Phase Convolutions

To cope with the lack of highly skilled professionals, machine learning with proper signal processing is key for establishing automated diagnostic-aid technologies with which to conduct epileptic electroencephalogram (EEG) testing. In particular, frequency filtering with the appropriate passbands is essential for enhancing the biomarkerssuch as epileptic spike wavesthat are noted in the EEG. This paper introduces a novel class of neural networks (NNs) that have a bank of linear-phase finite impulse response filters at the first layer as a preprocessor that can behave as bandpass filters that extract biomarkers without destroying waveforms because of a linear-phase condition. Besides, the parameters of the filters are also data-driven. The proposed NNs were trained with a large amount of clinical EEG data, including 15,833 epileptic spike waveforms recorded from 50 patients, and their labels were annotated by specialists. In the experiments, we compared three scenarios for the first layer: no preprocessing, discrete wavelet transform, and the proposed data-driven filters. The experimental results show that the trained data-driven filter bank with supervised learning behaves like multiple bandpass filters. In particular, the trained filter passed a frequency band of approximately 1030 Hz. Moreover, the proposed method detected epileptic spikes, with the area under the receiver operating characteristic curve of 0.967 in the mean of 50 intersubject validations.

Estimation of Stroke Volume Variance from Arterial Blood Pressure: Using a 1-D Convolutional Neural Network

BACKGROUND

We aimed to create a novel model using a deep learning method to estimate stroke volume variation (SVV), a widely used predictor of fluid responsiveness, from arterial blood pressure waveform (ABPW).

METHODS

In total, 557 patients and 8,512,564 SVV datasets were collected and were divided into three groups: training, validation, and test. Data was composed of 10 s of ABPW and corresponding SVV data recorded every 2 s. We built a convolutional neural network (CNN) model to estimate SVV from the ABPW with pre-existing commercialized model (EV1000) as a reference. We applied pre-processing, multichannel, and dimension reduction to improve the CNN model with diversified inputs.

RESULTS

Our CNN model showed an acceptable performance with sample data (r = 0.91, MSE = 6.92). Diversification of inputs, such as normalization, frequency, and slope of ABPW significantly improved the model correlation (r = 0.95), lowered mean squared error (MSE = 2.13), and resulted in a high concordance rate (96.26%) with the SVV from the commercialized model.

CONCLUSIONS

We developed a new CNN deep-learning model to estimate SVV. Our CNN model seems to be a viable alternative when the necessary medical device is not available, thereby allowing a wider range of application and resulting in optimal patient management.

Searching for pneumothorax in x-ray images using autoencoded deep features

Fast diagnosis and treatment of pneumothorax, a collapsed or dropped lung, is crucial to avoid fatalities. Pneumothorax is typically detected on a chest X-ray image through visual inspection by experienced radiologists. However, the detection rate is quite low due to the complexity of visual inspection for small lung collapses. Therefore, there is an urgent need for automated detection systems to assist radiologists. Although deep learning classifiers generally deliver high accuracy levels in many applications, they may not be useful in clinical practice due to the lack of high-quality and representative labeled image sets. Alternatively, searching in the archive of past cases to find matching images may serve as a "virtual second opinion" through accessing the metadata of matched evidently diagnosed cases. To use image search as a triaging or diagnosis assistant, we must first tag all chest X-ray images with expressive identifiers, i.e., deep features. Then, given a query chest X-ray image, the majority vote among the top k retrieved images can provide a more explainable output. In this study, we searched in a repository with more than 550,000 chest X-ray images. We developed the Autoencoding Thorax Net (short AutoThorax -Net) for image search in chest radiographs. Experimental results show that image search based on AutoThorax -Net features can achieve high identification performance providing a path towards real-world deployment. We achieved 92% AUC accuracy for a semi-automated search in 194,608 images (pneumothorax and normal) and 82% AUC accuracy for fully automated search in 551,383 images (normal, pneumothorax and many other chest diseases).