Non-Local Means Inpainting of MS Lesions in Longitudinal Image Processing

Improving the Segmentation Accuracy of Ovarian-Tumor Ultrasound Images Using Image Inpainting

Diagnostic results can be radically influenced by the quality of 2D ovarian-tumor ultrasound images. However, clinically processed 2D ovarian-tumor ultrasound images contain many artificially recognized symbols, such as fingers, crosses, dashed lines, and letters which assist artificial intelligence (AI) in image recognition. These symbols are widely distributed within the lesion's boundary, which can also affect the useful feature-extraction-utilizing networks and thus decrease the accuracy of lesion classification and segmentation. Image inpainting techniques are used for noise and object elimination from images. To solve this problem, we observed the MMOTU dataset and built a 2D ovarian-tumor ultrasound image inpainting dataset by finely annotating the various symbols in the images. A novel framework called mask-guided generative adversarial network (MGGAN) is presented in this paper for 2D ovarian-tumor ultrasound images to remove various symbols from the images. The MGGAN performs to a high standard in corrupted regions by using an attention mechanism in the generator to pay more attention to valid information and ignore symbol information, making lesion boundaries more realistic. Moreover, fast Fourier convolutions (FFCs) and residual networks are used to increase the global field of perception; thus, our model can be applied to high-resolution ultrasound images. The greatest benefit of this algorithm is that it achieves pixel-level inpainting of distorted regions without clean images. Compared with other models, our model achieveed better results with only one stage in terms of objective and subjective evaluations. Our model obtained the best results for 256 × 256 and 512 × 512 resolutions. At a resolution of 256 × 256, our model achieved 0.9246 for SSIM, 22.66 for FID, and 0.07806 for LPIPS. At a resolution of 512 × 512, our model achieved 0.9208 for SSIM, 25.52 for FID, and 0.08300 for LPIPS. Our method can considerably improve the accuracy of computerized ovarian tumor diagnosis. The segmentation accuracy was improved from 71.51% to 76.06% for the Unet model and from 61.13% to 66.65% for the PSPnet model in clean images.

Inpainting Missing Tissue in Multiplexed Immunofluorescence Imaging

Multiplex immunofluorescence (MxIF) is an emerging technique that allows for staining multiple cellular and histological markers to stain simultaneously on a single tissue section. However, with multiple rounds of staining and bleaching, it is inevitable that the scarce tissue may be physically depleted. Thus, a digital way of synthesizing such missing tissue would be appealing since it would increase the useable areas for the downstream single-cell analysis. In this work, we investigate the feasibility of employing generative adversarial network (GAN) approaches to synthesize missing tissues using 11 MxIF structural molecular markers (i.e., epithelial and stromal). Briefly, we integrate a multi-channel high-resolution image synthesis approach to synthesize the missing tissue from the remaining markers. The performance of different methods is quantitatively evaluated via the downstream cell membrane segmentation task. Our contribution is that we, for the first time, assess the feasibility of synthesizing missing tissues in MxIF via quantitative segmentation. The proposed synthesis method has comparable reproducibility with the baseline method on performance for the missing tissue region reconstruction only, but it improves 40% on whole tissue synthesis that is crucial for practical application. We conclude that GANs are a promising direction of advancing MxIF imaging with deep image synthesis.

Lesion synthesis to improve intracranial hemorrhage detection and classification for CT images

Computer-aided diagnosis (CAD) for intracranial hemorrhage (ICH) is needed due to its high mortality rate and time sensitivity. Training a stable and robust deep learning-based model usually requires enough training examples, which may be impractical in many real-world scenarios. Lesion synthesis offers a possible solution to solve this problem, especially for the issue of the lack of micro bleedings. In this paper, we propose a novel strategy to generate artificial lesions on non-lesion CT images so as to produce additional labeled training examples. Artificial masks in any location, size, or shape can be generated through Artificial Mask Generator (AMG) and then be converted into hemorrhage lesions through Lesion Synthesis Network (LSN). Images with and without artificial lesions are combined for training an ICH detection with a novel Residual Score. We evaluate our method by the auxiliary diagnosis task of ICH. Our experiments demonstrate that the proposed approach can improve the AUC value from 84% to 91% in the ICH detection task and from 89% to 96% in the classification task. Moreover, by adding artificial lesions of small size, the sensitivity of micro bleeding is remarkably improved from 49% to 70%. Besides, the proposed method overcomes the other three synthetic approaches by a large margin.

Robust Multiple Sclerosis Lesion Inpainting with Edge Prior

Inpainting lesions is an important preprocessing task for algorithms analyzing brain MRIs of multiple sclerosis (MS) patients, such as tissue segmentation and cortical surface reconstruction. We propose a new deep learning approach for this task. Unlike existing inpainting approaches which ignore the lesion areas of the input image, we leverage the edge information around the lesions as a prior to help the inpainting process. Thus, the input of this network includes the T1-w image, lesion mask and the edge map computed from the T1-w image, and the output is the lesion-free image. The introduction of the edge prior is based on our observation that the edge detection results of the MRI scans will usually contain the contour of white matter (WM) and grey matter (GM), even though some undesired edges appear near the lesions. Instead of losing all the information around the neighborhood of lesions, our approach preserves the local tissue shape (brain/WM/GM) with the guidance of the input edges. The qualitative results show that our pipeline inpaints the lesion areas in a realistic and shape-consistent way. Our quantitative evaluation shows that our approach outperforms the existing state-of-the-art inpainting methods in both image-based metrics and in FreeSurfer segmentation accuracy. Furthermore, our approach demonstrates robustness to inaccurate lesion mask inputs. This is important for practical usability, because it allows for a generous over-segmentation of lesions instead of requiring precise boundaries, while still yielding accurate results.

Inpainting as a Technique for Estimation of Missing Voxels in Brain Imaging

Issues with model fitting (i.e. suboptimal standard deviation, linewidth/full-width-at-half-maximum, and/or signal-to-noise ratio) in multi-voxel MRI spectroscopy, or chemical shift imaging (CSI) can result in the significant loss of usable voxels. A potential solution to minimize this problem is to estimate the value of unusable voxels by utilizing information from reliable voxels in the same image. We assessed an image restoration method called inpainting as a tool to restore unusable voxels, and compared it with traditional interpolation methods (nearest neighbor, trilinear interpolation and tricubic interpolation). In order to evaluate the performance across varying image contrasts and spatial resolutions, we applied the same techniques to a T1-weighted MRI brain dataset, and N-acetylaspartate (NAA) spectroscopy maps from a CSI dataset. For all image types, inpainting exhibited superior performance (lower normalized root-mean-square errors, NRMSE) compared to all other methods considered (p's < 0.001). Inpainting maintained an average NRMSE of less than 5% even with 50% missing voxels, whereas the other techniques demonstrated up to three times that value, depending on the nature of the image. For CSI maps, inpainting maintained its superiority whether the previously unusable voxels were randomly distributed, or located in regions most commonly affected by voxel loss in real-world data. Inpainting is a promising approach for recovering unusable or missing voxels in voxel-wise analyses, particularly in imaging modalities characterized by low SNR such as CSI. We hypothesize that this technique may also be applicable for datasets from other imaging modalities, such as positron emission tomography, or dynamic susceptibility contrast MRI.

A multi-time-point modality-agnostic patch-based method for lesion filling in multiple sclerosis

Multiple sclerosis lesions influence the process of image analysis, leading to tissue segmentation problems and biased morphometric estimates. Existing techniques try to reduce this bias by filling all lesions as normal-appearing white matter on T1-weighted images, considering each time-point separately. However, due to lesion segmentation errors and the presence of structures adjacent to the lesions, such as the ventricles and deep grey matter nuclei, filling all lesions with white matter-like intensities introduces errors and artefacts. In this paper, we present a novel lesion filling strategy inspired by in-painting techniques used in computer graphics applications for image completion. The proposed technique uses a five-dimensional (5D), patch-based (multi-modality and multi-time-point), Non-Local Means algorithm that fills lesions with the most plausible texture. We demonstrate that this strategy introduces less bias, fewer artefacts and spurious edges than the current, publicly available techniques. The proposed method is modality-agnostic and can be applied to multiple time-points simultaneously. In addition, it preserves anatomical structures and signal-to-noise characteristics even when the lesions are neighbouring grey matter or cerebrospinal fluid, and avoids excess of blurring or rasterisation due to the choice of the segmentation plane, shape of the lesions, and their size and/or location.

Clinical relevance of brain atrophy assessment in multiple sclerosis. Implications for its use in a clinical routine

INTRODUCTION

Brain atrophy measurement in multiple sclerosis (MS) has become an important outcome for determining patients at risk for developing physical and cognitive disability.

AREAS COVERED

In this article, we discuss the methodological issues related to using this MRI metric routinely, in a clinical setting. Understanding trajectories of annualized whole brain, gray and white matter, thalamic volume loss, and enlargement of ventricular space in specific MS phenotypes is becoming increasingly important. Evidence is mounting that disease-modifying treatments exert a positive effect on slowing brain atrophy progression in MS. Expert Commentary: While there is a need to translate measurement of brain atrophy to clinical routine at the individual patient level, there are still a number of challenges to be met before this can actually happen, including how to account for biological confounding factors and pseudoatrophy, standardize acquisition and analyses parameters, which can influence the accuracy of the assessments.