A new deep learning method for image deblurring in optical microscopic systems

Deep multi-task learning for nephropathy diagnosis on immunofluorescence images

BACKGROUND AND OBJECTIVE

As an advanced technique, immunofluorescence (IF) is one of the most widely-used medical image for nephropathy diagnosis, due to its ease of acquisition with low cost. In practice, the clinically collected IF images are commonly corrupted by blurs at different degrees, mainly because of the inaccurate focus at the acquisition stage. Although deep neural network (DNN) methods achieve the great success in nephropathy diagnosis, their performance dramatically drops over the blurred IF images. This significantly limits the potential of leveraging the advanced DNN techniques in real-world nephropathy diagnosis scenarios.

METHODS

This paper first establishes two IF databases with synthetic blurs (IFVB) and real-world blurs (Real-IF) for nephropathy diagnosis, respectively, including 1,659 patients and 6,521 IF images with various degrees of blurs. According to the analysis on these two databases, we propose a deep hierarchical multi-task learning based nephropathy diagnosis (DeepMT-ND) method to bridge the gap between the low-level vision and high-level medical tasks. Specifically, DeepMT-ND simultaneously handles the main task of automatic nephropathy diagnosis, as well as the auxiliary tasks of image quality assessment (IQA) and de-blurring.

RESULTS

Extensive experiments show the superiority of our DeepMT-ND in terms of diagnosis accuracy and generalization ability. For instance, our method performs better than nephrologists with at least 15.4% and 6.5% accuracy improvements in IFVB and Real-IF, respectively. Meanwhile, our method also achieves comparable performance in two auxiliary tasks of IQA and de-blurring on blurred IF images.

CONCLUSIONS

In this paper, we propose a new DeepMT-ND method for nephropathy diagnosis on blurred IF images. The proposed hierarchical multi-task learning framework provides the new scope to narrow the gap between the low-level vision and high-level medical tasks, and will contribute to nephropathy diagnosis in clinical scenarios. The diagnosis accuracy and generalization ability of DeepMT-ND are experimentally verified to be effective over both synthetic and real-world databases.

Imaging inside highly scattering media using hybrid deep learning and analytical algorithm

Imaging through highly scattering media is a challenging problem with numerous applications in biomedical and remote-sensing fields. Existing methods that use analytical or deep learning tools are limited by simplified forward models or a requirement for prior physical knowledge, resulting in blurry images or a need for large training databases. To address these limitations, we propose a hybrid scheme called Hybrid-DOT that combines analytically derived image estimates with a deep learning network. Our analysis demonstrates that Hybrid-DOT outperforms a state-of-the-art ToF-DOT algorithm by improving the PSNR ratio by 4.6 dB and reducing the resolution by a factor of 2.5. Furthermore, when compared to a deep learning stand-alone model, Hybrid-DOT achieves a 0.8 dB increase in PSNR, 1.5 times the resolution, and a significant reduction in the required dataset size (factor of 1.6-3). The proposed model remains effective at higher depths, providing similar improvements for up to 160 mean-free paths.

Enhanced resolution and sensitivity acoustic-resolution photoacoustic microscopy with semi/unsupervised GANs

Acoustic-resolution photoacoustic microscopy (AR-PAM) enables visualization of biological tissues at depths of several millimeters with superior optical absorption contrast. However, the lateral resolution and sensitivity of AR-PAM are generally lower than those of optical-resolution PAM (OR-PAM) owing to the intrinsic physical acoustic focusing mechanism. Here, we demonstrate a computational strategy with two generative adversarial networks (GANs) to perform semi/unsupervised reconstruction with high resolution and sensitivity in AR-PAM by maintaining its imaging capability at enhanced depths. The b-scan PAM images were prepared as paired (for semi-supervised conditional GAN) and unpaired (for unsupervised CycleGAN) groups for label-free reconstructed AR-PAM b-scan image generation and training. The semi/unsupervised GANs successfully improved resolution and sensitivity in a phantom and in vivo mouse ear test with ground truth. We also confirmed that GANs could enhance resolution and sensitivity of deep tissues without the ground truth.

Joint Spine Segmentation and Noise Removal From Ultrasound Volume Projection Images With Selective Feature Sharing

Volume Projection Imaging from ultrasound data is a promising technique to visualize spine features and diagnose Adolescent Idiopathic Scoliosis. In this paper, we present a novel multi-task framework to reduce the scan noise in volume projection images and to segment different spine features simultaneously, which provides an appealing alternative for intelligent scoliosis assessment in clinical applications. Our proposed framework consists of two streams: i) A noise removal stream based on generative adversarial networks, which aims to achieve effective scan noise removal in a weakly-supervised manner, i.e., without paired noisy-clean samples for learning; ii) A spine segmentation stream, which aims to predict accurate bone masks. To establish the interaction between these two tasks, we propose a selective feature-sharing strategy to transfer only the beneficial features, while filtering out the useless or harmful information. We evaluate our proposed framework on both scan noise removal and spine segmentation tasks. The experimental results demonstrate that our proposed method achieves promising performance on both tasks, which provides an appealing approach to facilitating clinical diagnosis.

Reconstructing high fidelity digital rock images using deep convolutional neural networks

Imaging methods have broad applications in geosciences. Scanning electron microscopy (SEM) and micro-CT scanning have been applied for studying various geological problems. Despite significant advances in imaging capabilities, and image processing algorithms, acquiring high-quality data from images is still challenging and time-consuming. Obtaining a 3D representative volume for a tight rock sample takes days to weeks. Image artifacts such as noise further complicate the use of imaging methods for the determination of rock properties. In this study, we present applications of several convolutional neural networks (CNN) for rapid image denoising, deblurring and super-resolving digital rock images. Such an approach enables rapid imaging of larger samples, which in turn improves the statistical relevance of the subsequent analysis. We demonstrate the application of several CNNs for image restoration applicable to scientific imaging. The results show that images can be denoised without a priori knowledge of the noise with great confidence. Furthermore, we show how attaching several CNNs in an end-to-end fashion can improve the final quality of reconstruction. Our experiments with SEM and CT scan images of several rock types show image denoising, deblurring and super-resolution can be performed simultaneously.

High-resolution photoacoustic microscopy with deep penetration through learning

Optical-resolution photoacoustic microscopy (OR-PAM) enjoys superior spatial resolution and has received intense attention in recent years. The application, however, has been limited to shallow depths because of strong scattering of light in biological tissues. In this work, we propose to achieve deep-penetrating OR-PAM performance by using deep learning enabled image transformation on blurry living mouse vascular images that were acquired with an acoustic-resolution photoacoustic microscopy (AR-PAM) setup. A generative adversarial network (GAN) was trained in this study and improved the imaging lateral resolution of AR-PAM from 54.0 µm to 5.1 µm, comparable to that of a typical OR-PAM (4.7 µm). The feasibility of the network was evaluated with living mouse ear data, producing superior microvasculature images that outperforms blind deconvolution. The generalization of the network was validated with in vivo mouse brain data. Moreover, it was shown experimentally that the deep-learning method can retain high resolution at tissue depths beyond one optical transport mean free path. Whilst it can be further improved, the proposed method provides new horizons to expand the scope of OR-PAM towards deep-tissue imaging and wide applications in biomedicine.

State-of-the-Art Approaches for Image Deconvolution Problems, including Modern Deep Learning Architectures

In modern digital microscopy, deconvolution methods are widely used to eliminate a number of image defects and increase resolution. In this review, we have divided these methods into classical, deep learning-based, and optimization-based methods. The review describes the major architectures of neural networks, such as convolutional and generative adversarial networks, autoencoders, various forms of recurrent networks, and the attention mechanism used for the deconvolution problem. Special attention is paid to deep learning as the most powerful and flexible modern approach. The review describes the major architectures of neural networks used for the deconvolution problem. We describe the difficulties in their application, such as the discrepancy between the standard loss functions and the visual content and the heterogeneity of the images. Next, we examine how to deal with this by introducing new loss functions, multiscale learning, and prior knowledge of visual content. In conclusion, a review of promising directions and further development of deconvolution methods in microscopy is given.

Medical Image Recognition Technology in the Effect of Substituting Soybean Meal for Fish Meal on the Diversity of Intestinal Microflora in Channa argus

Purpose

To study the application of medical image recognition technology based on backpropagation neural network (BPNN) in the effect of soybean meal replacing fish meal on intestinal microbial diversity of Channa argus and to evaluate the application value of this intelligent algorithm, Channa argus was fed with different contents of soybean meal instead of fish meal.

Methods

After intestinal samples were collected and bacteria were isolated, microscopic imaging was performed, and the images were classified and identified. BPNN was constructed to perform denoising, smoothing, and segmentation.

Results

After BPNN processing, the bacteria were completely separated from the original image background, and the bacteria was in the closed state, which was beneficial to feature extraction and species recognition. If there were 2 hidden layer nodes, the segmentation accuracy of bacterial microscopic images was the highest, up to 97.3%. With the replacement ratio of fish meal increased, the species of intestinal microbiome gradually enriched, and the relative abundance of intestinal microbiome was higher after fish meal was completely replaced by soybean meal (replacement). The intestinal microbial enzyme activities were affected by different fish meal and soybean meal contents in the diet. The glutamate transaminase and adenosine deaminase activities were increased after the replacement and were higher than those before the replacement, with statistically significant differences (P < 0.05).

Conclusion

Replacement of fish meal with soybean meal has a significant effect on the intestinal flora diversity of Channa argus, and there is a close relationship between them. The image recognition technology based on BPNN has high recognition rate and segmentation accuracy for microbiological microscopic images.

Blur-Robust Nuclei Segmentation for Immunofluorescence Images

Automated nuclei segmentation from immunofluorescence (IF) microscopic image is a crucial first step in digital pathology. A lot of research has been devoted to develop novel nuclei segmentation algorithms to give high performance on good quality images. However, fewer methods were developed for poor-quality images like out-of-focus (blurry) data. In this work, we take a principled approach to study the performance of nuclei segmentation algorithms on out-of-focus images for different levels of blur. A deep learning encoder-decoder framework with a novel Y forked decoder is proposed here. The two fork ends are tied to segmentation and deblur output. The addition of a separate deblurring task in the training paradigm helps to regularize the network on blurry images. Our proposed method accurately predicts the instance nuclei segmentation on sharp as well as out-of-focus images. Additionally, predicted deblurred image provides interpretable insights to experts. Experimental analysis on the Human U2OS cells (out-of-focus) dataset shows that our algorithm is robust and outperforms the state-of-the-art methods.

Multi-Task Learning-Based Immunofluorescence Classification of Kidney Disease

Chronic kidney disease is one of the most important causes of mortality worldwide, but a shortage of nephrology pathologists has led to delays or errors in its diagnosis and treatment. Immunofluorescence (IF) images of patients with IgA nephropathy (IgAN), membranous nephropathy (MN), diabetic nephropathy (DN), and lupus nephritis (LN) were obtained from the General Hospital of Chinese PLA. The data were divided into training and test data. To simulate the inaccurate focus of the fluorescence microscope, the Gaussian method was employed to blur the IF images. We proposed a novel multi-task learning (MTL) method for image quality assessment, de-blurring, and disease classification tasks. A total of 1608 patients’ IF images were included—1289 in the training set and 319 in the test set. For non-blurred IF images, the classification accuracy of the test set was 0.97, with an AUC of 1.000. For blurred IF images, the proposed MTL method had a higher accuracy (0.94 vs. 0.93, p < 0.01) and higher AUC (0.993 vs. 0.986) than the common MTL method. The novel MTL method not only diagnosed four types of kidney diseases through blurred IF images but also showed good performance in two auxiliary tasks: image quality assessment and de-blurring.

Deep learning for biomedical photoacoustic imaging: A review

Photoacoustic imaging (PAI) is a promising emerging imaging modality that enables spatially resolved imaging of optical tissue properties up to several centimeters deep in tissue, creating the potential for numerous exciting clinical applications. However, extraction of relevant tissue parameters from the raw data requires the solving of inverse image reconstruction problems, which have proven extremely difficult to solve. The application of deep learning methods has recently exploded in popularity, leading to impressive successes in the context of medical imaging and also finding first use in the field of PAI. Deep learning methods possess unique advantages that can facilitate the clinical translation of PAI, such as extremely fast computation times and the fact that they can be adapted to any given problem. In this review, we examine the current state of the art regarding deep learning in PAI and identify potential directions of research that will help to reach the goal of clinical applicability.

Another decade of photoacoustic imaging

Photoacoustic imaging - a hybrid biomedical imaging modality finding its way to clinical practices. Although the photoacoustic phenomenon was known more than a century back, only in the last two decades it has been widely researched and used for biomedical imaging applications. In this review we focus on the development and progress of the technology in the last decade (2010-2020). From becoming more and more user friendly, cheaper in cost, portable in size, photoacoustic imaging promises a wide range of applications, if translated to clinic. The growth of photoacoustic community is steady, and with several new directions researchers are exploring, it is inevitable that photoacoustic imaging will one day establish itself as a regular imaging system in the clinical practices.

Convolutional neural network for resolution enhancement and noise reduction in acoustic resolution photoacoustic microscopy

In acoustic resolution photoacoustic microscopy (AR-PAM), a high numerical aperture focused ultrasound transducer (UST) is used for deep tissue high resolution photoacoustic imaging. There is a significant degradation of lateral resolution in the out-of-focus region. Improvement in out-of-focus resolution without degrading the image quality remains a challenge. In this work, we propose a deep learning-based method to improve the resolution of AR-PAM images, especially at the out of focus plane. A modified fully dense U-Net based architecture was trained on simulated AR-PAM images. Applying the trained model on experimental images showed that the variation in resolution is ∼10% across the entire imaging depth (∼4 mm) in the deep learning-based method, compared to ∼180% variation in the original PAM images. Performance of the trained network on in vivo rat vasculature imaging further validated that noise-free, high resolution images can be obtained using this method.

Hybrid deep learning network for vascular segmentation in photoacoustic imaging

Photoacoustic (PA) technology has been used extensively on vessel imaging due to its capability of identifying molecular specificities and achieving high optical-diffraction-limited lateral resolution down to the cellular level. Vessel images carry essential medical information that provides guidelines for a professional diagnosis. Modern image processing techniques provide a decent contribution to vessel segmentation. However, these methods suffer from under or over-segmentation. Thus, we demonstrate both the results of adopting a fully convolutional network and U-net, and propose a hybrid network consisting of both applied on PA vessel images. Comparison results indicate that the hybrid network can significantly increase the segmentation accuracy and robustness.