1
|
Improving the Segmentation Accuracy of Ovarian-Tumor Ultrasound Images Using Image Inpainting. Bioengineering (Basel) 2023; 10:bioengineering10020184. [PMID: 36829679 PMCID: PMC9952248 DOI: 10.3390/bioengineering10020184] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Revised: 01/05/2023] [Accepted: 01/28/2023] [Indexed: 02/04/2023] Open
Abstract
Diagnostic results can be radically influenced by the quality of 2D ovarian-tumor ultrasound images. However, clinically processed 2D ovarian-tumor ultrasound images contain many artificially recognized symbols, such as fingers, crosses, dashed lines, and letters which assist artificial intelligence (AI) in image recognition. These symbols are widely distributed within the lesion's boundary, which can also affect the useful feature-extraction-utilizing networks and thus decrease the accuracy of lesion classification and segmentation. Image inpainting techniques are used for noise and object elimination from images. To solve this problem, we observed the MMOTU dataset and built a 2D ovarian-tumor ultrasound image inpainting dataset by finely annotating the various symbols in the images. A novel framework called mask-guided generative adversarial network (MGGAN) is presented in this paper for 2D ovarian-tumor ultrasound images to remove various symbols from the images. The MGGAN performs to a high standard in corrupted regions by using an attention mechanism in the generator to pay more attention to valid information and ignore symbol information, making lesion boundaries more realistic. Moreover, fast Fourier convolutions (FFCs) and residual networks are used to increase the global field of perception; thus, our model can be applied to high-resolution ultrasound images. The greatest benefit of this algorithm is that it achieves pixel-level inpainting of distorted regions without clean images. Compared with other models, our model achieveed better results with only one stage in terms of objective and subjective evaluations. Our model obtained the best results for 256 × 256 and 512 × 512 resolutions. At a resolution of 256 × 256, our model achieved 0.9246 for SSIM, 22.66 for FID, and 0.07806 for LPIPS. At a resolution of 512 × 512, our model achieved 0.9208 for SSIM, 25.52 for FID, and 0.08300 for LPIPS. Our method can considerably improve the accuracy of computerized ovarian tumor diagnosis. The segmentation accuracy was improved from 71.51% to 76.06% for the Unet model and from 61.13% to 66.65% for the PSPnet model in clean images.
Collapse
|
2
|
Bao S, Tang Y, Lee HH, Gao R, Yang Q, Yu X, Chiron S, Coburn LA, Wilson KT, Roland JT, Landman BA, Huo Y. Inpainting Missing Tissue in Multiplexed Immunofluorescence Imaging. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2022; 12039:120390K. [PMID: 35531320 PMCID: PMC9070577 DOI: 10.1117/12.2611827] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Multiplex immunofluorescence (MxIF) is an emerging technique that allows for staining multiple cellular and histological markers to stain simultaneously on a single tissue section. However, with multiple rounds of staining and bleaching, it is inevitable that the scarce tissue may be physically depleted. Thus, a digital way of synthesizing such missing tissue would be appealing since it would increase the useable areas for the downstream single-cell analysis. In this work, we investigate the feasibility of employing generative adversarial network (GAN) approaches to synthesize missing tissues using 11 MxIF structural molecular markers (i.e., epithelial and stromal). Briefly, we integrate a multi-channel high-resolution image synthesis approach to synthesize the missing tissue from the remaining markers. The performance of different methods is quantitatively evaluated via the downstream cell membrane segmentation task. Our contribution is that we, for the first time, assess the feasibility of synthesizing missing tissues in MxIF via quantitative segmentation. The proposed synthesis method has comparable reproducibility with the baseline method on performance for the missing tissue region reconstruction only, but it improves 40% on whole tissue synthesis that is crucial for practical application. We conclude that GANs are a promising direction of advancing MxIF imaging with deep image synthesis.
Collapse
Affiliation(s)
- Shunxing Bao
- Dept. of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
- Dept. of Computer Science, Vanderbilt University, Nashville, TN, USA
| | - Yucheng Tang
- Dept. of Computer Science, Vanderbilt University, Nashville, TN, USA
| | - Ho Hin Lee
- Dept. of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
| | - Riqiang Gao
- Dept. of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
| | - Qi Yang
- Dept. of Computer Science, Vanderbilt University, Nashville, TN, USA
| | - Xin Yu
- Dept. of Computer Science, Vanderbilt University, Nashville, TN, USA
| | - Sophie Chiron
- Division of Gastroenterology, Hepatology, and Nutrition, Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Lori A Coburn
- Division of Gastroenterology, Hepatology, and Nutrition, Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, USA
- Vanderbilt Center for Mucosal Inflammation and Cancer, Nashville, TN, USA
- Dept. of Pathology, Microbiology, and Immunology, Vanderbilt University Medical Center, Nashville, TN, USA
- Veterans Affairs Tennessee Valley Healthcare System, Nashville, TN, USA
| | - Keith T Wilson
- Division of Gastroenterology, Hepatology, and Nutrition, Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, USA
- Vanderbilt Center for Mucosal Inflammation and Cancer, Nashville, TN, USA
- Program in Cancer Biology, Vanderbilt University School of Medicine, Nashville, TN, USA
- Dept. of Pathology, Microbiology, and Immunology, Vanderbilt University Medical Center, Nashville, TN, USA
- Veterans Affairs Tennessee Valley Healthcare System, Nashville, TN, USA
| | - Joseph T Roland
- Epithelial Biology Center, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Bennett A Landman
- Dept. of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
- Dept. of Computer Science, Vanderbilt University, Nashville, TN, USA
| | - Yuankai Huo
- Dept. of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA
- Dept. of Computer Science, Vanderbilt University, Nashville, TN, USA
| |
Collapse
|
3
|
Zhang G, Chen K, Xu S, Cho PC, Nan Y, Zhou X, Lv C, Li C, Xie G. Lesion synthesis to improve intracranial hemorrhage detection and classification for CT images. Comput Med Imaging Graph 2021; 90:101929. [PMID: 33984782 DOI: 10.1016/j.compmedimag.2021.101929] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2020] [Revised: 02/02/2021] [Accepted: 04/21/2021] [Indexed: 11/17/2022]
Abstract
Computer-aided diagnosis (CAD) for intracranial hemorrhage (ICH) is needed due to its high mortality rate and time sensitivity. Training a stable and robust deep learning-based model usually requires enough training examples, which may be impractical in many real-world scenarios. Lesion synthesis offers a possible solution to solve this problem, especially for the issue of the lack of micro bleedings. In this paper, we propose a novel strategy to generate artificial lesions on non-lesion CT images so as to produce additional labeled training examples. Artificial masks in any location, size, or shape can be generated through Artificial Mask Generator (AMG) and then be converted into hemorrhage lesions through Lesion Synthesis Network (LSN). Images with and without artificial lesions are combined for training an ICH detection with a novel Residual Score. We evaluate our method by the auxiliary diagnosis task of ICH. Our experiments demonstrate that the proposed approach can improve the AUC value from 84% to 91% in the ICH detection task and from 89% to 96% in the classification task. Moreover, by adding artificial lesions of small size, the sensitivity of micro bleeding is remarkably improved from 49% to 70%. Besides, the proposed method overcomes the other three synthetic approaches by a large margin.
Collapse
Affiliation(s)
- Guyue Zhang
- Ping An Technology (Shenzhen) Co., Ltd., Shanghai 200000, China; Zhejiang Institute of Standardization, Hangzhou, Zhejiang Province 310007, China.
| | - Kaixing Chen
- Ping An Technology (Shenzhen) Co., Ltd., Shanghai 200000, China.
| | - Shangliang Xu
- Ping An Technology (Shenzhen) Co., Ltd., Shanghai 200000, China.
| | - Po Chuan Cho
- Ping An Technology (Shenzhen) Co., Ltd., Shanghai 200000, China.
| | - Yang Nan
- Ping An Technology (Shenzhen) Co., Ltd., Shanghai 200000, China; National Heart and Lung Institute, Imperial College London, London, UK.
| | - Xin Zhou
- Ping An Technology (Shenzhen) Co., Ltd., Shanghai 200000, China.
| | - Chuanfeng Lv
- Ping An Technology (Shenzhen) Co., Ltd., Shanghai 200000, China.
| | - Changsheng Li
- School of Computer Science & Technology, Beijing Institute of Technology, Beijing 210023, China.
| | - Guotong Xie
- Ping An Technology (Shenzhen) Co., Ltd., Shanghai 200000, China.
| |
Collapse
|
4
|
Zhang H, Bakshi R, Bagnato F, Oguz I. Robust Multiple Sclerosis Lesion Inpainting with Edge Prior. MACHINE LEARNING IN MEDICAL IMAGING. MLMI (WORKSHOP) 2020; 12436:120-129. [PMID: 34950933 PMCID: PMC8692168 DOI: 10.1007/978-3-030-59861-7_13] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Inpainting lesions is an important preprocessing task for algorithms analyzing brain MRIs of multiple sclerosis (MS) patients, such as tissue segmentation and cortical surface reconstruction. We propose a new deep learning approach for this task. Unlike existing inpainting approaches which ignore the lesion areas of the input image, we leverage the edge information around the lesions as a prior to help the inpainting process. Thus, the input of this network includes the T1-w image, lesion mask and the edge map computed from the T1-w image, and the output is the lesion-free image. The introduction of the edge prior is based on our observation that the edge detection results of the MRI scans will usually contain the contour of white matter (WM) and grey matter (GM), even though some undesired edges appear near the lesions. Instead of losing all the information around the neighborhood of lesions, our approach preserves the local tissue shape (brain/WM/GM) with the guidance of the input edges. The qualitative results show that our pipeline inpaints the lesion areas in a realistic and shape-consistent way. Our quantitative evaluation shows that our approach outperforms the existing state-of-the-art inpainting methods in both image-based metrics and in FreeSurfer segmentation accuracy. Furthermore, our approach demonstrates robustness to inaccurate lesion mask inputs. This is important for practical usability, because it allows for a generous over-segmentation of lesions instead of requiring precise boundaries, while still yielding accurate results.
Collapse
Affiliation(s)
| | - Rohit Bakshi
- Brigham and Women's Hospital, Boston, MA 02115, USA
| | - Francesca Bagnato
- Vanderbilt University Medical Center and Nashville VA Medical Center, Nashville, TN 37212, USA
| | - Ipek Oguz
- Vanderbilt University, Nashville, TN 37235, USA
| |
Collapse
|
5
|
Torrado-Carvajal A, Albrecht DS, Lee J, Andronesi OC, Ratai EM, Napadow V, Loggia ML. Inpainting as a Technique for Estimation of Missing Voxels in Brain Imaging. Ann Biomed Eng 2020; 49:345-353. [PMID: 32632531 DOI: 10.1007/s10439-020-02556-3] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2020] [Accepted: 06/19/2020] [Indexed: 10/23/2022]
Abstract
Issues with model fitting (i.e. suboptimal standard deviation, linewidth/full-width-at-half-maximum, and/or signal-to-noise ratio) in multi-voxel MRI spectroscopy, or chemical shift imaging (CSI) can result in the significant loss of usable voxels. A potential solution to minimize this problem is to estimate the value of unusable voxels by utilizing information from reliable voxels in the same image. We assessed an image restoration method called inpainting as a tool to restore unusable voxels, and compared it with traditional interpolation methods (nearest neighbor, trilinear interpolation and tricubic interpolation). In order to evaluate the performance across varying image contrasts and spatial resolutions, we applied the same techniques to a T1-weighted MRI brain dataset, and N-acetylaspartate (NAA) spectroscopy maps from a CSI dataset. For all image types, inpainting exhibited superior performance (lower normalized root-mean-square errors, NRMSE) compared to all other methods considered (p's < 0.001). Inpainting maintained an average NRMSE of less than 5% even with 50% missing voxels, whereas the other techniques demonstrated up to three times that value, depending on the nature of the image. For CSI maps, inpainting maintained its superiority whether the previously unusable voxels were randomly distributed, or located in regions most commonly affected by voxel loss in real-world data. Inpainting is a promising approach for recovering unusable or missing voxels in voxel-wise analyses, particularly in imaging modalities characterized by low SNR such as CSI. We hypothesize that this technique may also be applicable for datasets from other imaging modalities, such as positron emission tomography, or dynamic susceptibility contrast MRI.
Collapse
Affiliation(s)
- Angel Torrado-Carvajal
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA, USA. .,Medical Image Analysis and Biometry Laboratory, Universidad Rey Juan Carlos, Madrid, Spain.
| | - Daniel S Albrecht
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA, USA
| | - Jeungchan Lee
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA, USA
| | - Ovidiu C Andronesi
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA, USA
| | - Eva-Maria Ratai
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA, USA
| | - Vitaly Napadow
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA, USA
| | - Marco L Loggia
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Charlestown, MA, USA
| |
Collapse
|
6
|
Prados F, Cardoso MJ, Kanber B, Ciccarelli O, Kapoor R, Gandini Wheeler-Kingshott CAM, Ourselin S. A multi-time-point modality-agnostic patch-based method for lesion filling in multiple sclerosis. Neuroimage 2016; 139:376-384. [PMID: 27377222 PMCID: PMC4988790 DOI: 10.1016/j.neuroimage.2016.06.053] [Citation(s) in RCA: 58] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2016] [Revised: 06/10/2016] [Accepted: 06/27/2016] [Indexed: 01/12/2023] Open
Abstract
Multiple sclerosis lesions influence the process of image analysis, leading to tissue segmentation problems and biased morphometric estimates. Existing techniques try to reduce this bias by filling all lesions as normal-appearing white matter on T1-weighted images, considering each time-point separately. However, due to lesion segmentation errors and the presence of structures adjacent to the lesions, such as the ventricles and deep grey matter nuclei, filling all lesions with white matter-like intensities introduces errors and artefacts. In this paper, we present a novel lesion filling strategy inspired by in-painting techniques used in computer graphics applications for image completion. The proposed technique uses a five-dimensional (5D), patch-based (multi-modality and multi-time-point), Non-Local Means algorithm that fills lesions with the most plausible texture. We demonstrate that this strategy introduces less bias, fewer artefacts and spurious edges than the current, publicly available techniques. The proposed method is modality-agnostic and can be applied to multiple time-points simultaneously. In addition, it preserves anatomical structures and signal-to-noise characteristics even when the lesions are neighbouring grey matter or cerebrospinal fluid, and avoids excess of blurring or rasterisation due to the choice of the segmentation plane, shape of the lesions, and their size and/or location.
Collapse
Affiliation(s)
- Ferran Prados
- Translational Imaging Group, Centre for Medical Image Computing (CMIC), Department of Medical Physics and Bioengineering, University College London, Malet Place Engineering Building, Gower Street, London, WC1E 6BT, UK; NMR Research Unit, Queen Square MS Centre, Department of Neuroinflammation, UCL Institute of Neurology, 1st Floor, Russell Square House, 10-12 Russell Square, London WC1B 5EH, UK.
| | - Manuel Jorge Cardoso
- Translational Imaging Group, Centre for Medical Image Computing (CMIC), Department of Medical Physics and Bioengineering, University College London, Malet Place Engineering Building, Gower Street, London, WC1E 6BT, UK
| | - Baris Kanber
- Translational Imaging Group, Centre for Medical Image Computing (CMIC), Department of Medical Physics and Bioengineering, University College London, Malet Place Engineering Building, Gower Street, London, WC1E 6BT, UK
| | - Olga Ciccarelli
- NMR Research Unit, Queen Square MS Centre, Department of Neuroinflammation, UCL Institute of Neurology, 1st Floor, Russell Square House, 10-12 Russell Square, London WC1B 5EH, UK
| | - Raju Kapoor
- Department of Neuroinflammation, National Hospital for Neurology and Neurosurgery, Queen Square, London WC1N 3BG, UK
| | - Claudia A M Gandini Wheeler-Kingshott
- NMR Research Unit, Queen Square MS Centre, Department of Neuroinflammation, UCL Institute of Neurology, 1st Floor, Russell Square House, 10-12 Russell Square, London WC1B 5EH, UK; Brain MRI 3T Center, C. Mondino National Neurological Institute, Pavia, Italy
| | - Sebastien Ourselin
- Translational Imaging Group, Centre for Medical Image Computing (CMIC), Department of Medical Physics and Bioengineering, University College London, Malet Place Engineering Building, Gower Street, London, WC1E 6BT, UK; Dementia Research Centre, Department of Neurodegenerative Disease, UCL Institute of Neurology, Queen Square, London WC1N 3BG, UK
| |
Collapse
|
7
|
Zivadinov R, Jakimovski D, Gandhi S, Ahmed R, Dwyer MG, Horakova D, Weinstock-Guttman B, Benedict RRH, Vaneckova M, Barnett M, Bergsland N. Clinical relevance of brain atrophy assessment in multiple sclerosis. Implications for its use in a clinical routine. Expert Rev Neurother 2016; 16:777-93. [PMID: 27105209 DOI: 10.1080/14737175.2016.1181543] [Citation(s) in RCA: 112] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Abstract
INTRODUCTION Brain atrophy measurement in multiple sclerosis (MS) has become an important outcome for determining patients at risk for developing physical and cognitive disability. AREAS COVERED In this article, we discuss the methodological issues related to using this MRI metric routinely, in a clinical setting. Understanding trajectories of annualized whole brain, gray and white matter, thalamic volume loss, and enlargement of ventricular space in specific MS phenotypes is becoming increasingly important. Evidence is mounting that disease-modifying treatments exert a positive effect on slowing brain atrophy progression in MS. Expert Commentary: While there is a need to translate measurement of brain atrophy to clinical routine at the individual patient level, there are still a number of challenges to be met before this can actually happen, including how to account for biological confounding factors and pseudoatrophy, standardize acquisition and analyses parameters, which can influence the accuracy of the assessments.
Collapse
Affiliation(s)
- Robert Zivadinov
- a Buffalo Neuroimaging Analysis Center, Department of Neurology, Jacobs School of Medicine and Biomedical Sciences , University at Buffalo, State University of New York , Buffalo , NY , USA.,b MR Imaging Clinical Translational Research Center, Jacobs School of Medicine and Biomedical Sciences , University at Buffalo, State University of New York , Buffalo , NY , USA
| | - Dejan Jakimovski
- a Buffalo Neuroimaging Analysis Center, Department of Neurology, Jacobs School of Medicine and Biomedical Sciences , University at Buffalo, State University of New York , Buffalo , NY , USA
| | - Sirin Gandhi
- a Buffalo Neuroimaging Analysis Center, Department of Neurology, Jacobs School of Medicine and Biomedical Sciences , University at Buffalo, State University of New York , Buffalo , NY , USA
| | - Rahil Ahmed
- a Buffalo Neuroimaging Analysis Center, Department of Neurology, Jacobs School of Medicine and Biomedical Sciences , University at Buffalo, State University of New York , Buffalo , NY , USA
| | - Michael G Dwyer
- a Buffalo Neuroimaging Analysis Center, Department of Neurology, Jacobs School of Medicine and Biomedical Sciences , University at Buffalo, State University of New York , Buffalo , NY , USA
| | - Dana Horakova
- c Department of Neurology and Center of Clinical Neuroscience , Charles University in Prague, First Faculty of Medicine and General University Hospital , Prague , Czech Republic
| | - Bianca Weinstock-Guttman
- d Jacobs Multiple Sclerosis Center, Department of Neurology, School of Medicine and Biomedical Sciences , University at Buffalo, State University of New York , Buffalo , NY , USA
| | - Ralph R H Benedict
- d Jacobs Multiple Sclerosis Center, Department of Neurology, School of Medicine and Biomedical Sciences , University at Buffalo, State University of New York , Buffalo , NY , USA
| | - Manuela Vaneckova
- e Department of Radiology, First Faculty of Medicine and General University Hospital , Charles University , Prague , Czech Republic
| | - Michael Barnett
- f Sydney Neuroimaging Analysis Centre; Brain & Mind Centre , University of Sydney , Sydney , Australia
| | - Niels Bergsland
- a Buffalo Neuroimaging Analysis Center, Department of Neurology, Jacobs School of Medicine and Biomedical Sciences , University at Buffalo, State University of New York , Buffalo , NY , USA.,g IRCCS 'S.Maria Nascente' , Don Gnocchi Foundation , Milan , Italy
| |
Collapse
|