1
|
Chen Y, Zhang X, Li D, Park H, Li X, Liu P, Jin J, Shen Y. Automatic segmentation of thyroid with the assistance of the devised boundary improvement based on multicomponent small dataset. APPL INTELL 2023; 53:1-16. [PMID: 37363389 PMCID: PMC10015528 DOI: 10.1007/s10489-023-04540-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/23/2023] [Indexed: 03/17/2023]
Abstract
Deep learning has been widely considered in medical image segmentation. However, the difficulty of acquiring medical images and labels can affect the accuracy of the segmentation results for deep learning methods. In this paper, an automatic segmentation method is proposed by devising a multicomponent neighborhood extreme learning machine to improve the boundary attention region of the preliminary segmentation results. The neighborhood features are acquired by training U-Nets with the multicomponent small dataset, which consists of original thyroid ultrasound images, Sobel edge images and superpixel images. Afterward, the neighborhood features are selected by min-redundancy and max-relevance filter in the designed extreme learning machine, and the selected features are used to train the extreme learning machine to obtain supplementary segmentation results. Finally, the accuracy of the segmentation results is improved by adjusting the boundary attention region of the preliminary segmentation results with the supplementary segmentation results. This method combines the advantages of deep learning and traditional machine learning, boosting the accuracy of thyroid segmentation accuracy with a small dataset in a multigroup test.
Collapse
Affiliation(s)
- Yifei Chen
- Control Science and Engineering, Harbin Institute of Technology, Harbin, 150001 China
- Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, 34141 Korea
| | - Xin Zhang
- Control Science and Engineering, Harbin Institute of Technology, Harbin, 150001 China
| | - Dandan Li
- Control Science and Engineering, Harbin Institute of Technology, Harbin, 150001 China
| | - HyunWook Park
- Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, 34141 Korea
| | - Xinran Li
- Mathematics, Harbin Institute of Technology, Harbin, 150001 China
| | - Peng Liu
- Heilongjiang Provincial Key Laboratory of Trace Elements and Human Health, Harbin Medical University, Harbin, 150081 China
| | - Jing Jin
- Control Science and Engineering, Harbin Institute of Technology, Harbin, 150001 China
| | - Yi Shen
- Control Science and Engineering, Harbin Institute of Technology, Harbin, 150001 China
| |
Collapse
|
2
|
Zhang Q, Liang Y, Zhang Y, Tao Z, Li R, Bi H. A comparative study of attention mechanism based deep learning methods for bladder tumor segmentation. Int J Med Inform 2023; 171:104984. [PMID: 36634475 DOI: 10.1016/j.ijmedinf.2023.104984] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2022] [Revised: 12/17/2022] [Accepted: 01/01/2023] [Indexed: 01/07/2023]
Abstract
BACKGROUND Artificial intelligence aided tumor segmentation has been applied in various medical scenarios and showed effectiveness in helping physicians observe the potential malignant tissues. However, little research has been conducted for the cystoscopic image segmentation problem. METHODS This paper provided a comprehensive comparison of various attention modules for improving the bladder tumor segmentation performance by utilizing the cystoscopic images from Peking University Third Hospital within 2017-2022. Furthermore, this paper presented an attention mechanism based cystoscopic images segmentation (ACS) model, which was featured by the following points: (1) A mixed attention module including both the channel and spatial attention modules was integrated in the encoder-decoder path, which helped to exploit the global information of the tumor area more effectively. (2) A guidance and fusion attention module was introduced in the skip connection part, facilitating the integration of the high-level semantic features with low-level fine-grained features and the discarding of irrelevant features. (3) An inception attention module was added to enhance the feature expression in the scale of pixel level, so as to better discriminate multi-scale targets. RESULTS The proposed ACS model showed obviously better tumor segmentation performance than the compared models, with Dice of 82.7% and MIoU of 69% achieved. CONCLUSIONS The proposed ACS model achieved significantly better diagnostic performance than the previous bladder tumor segmentation method based on U-Net. Our ACS model is expected to be a useful support tool to assist the tumor segmentation under cystoscopy.
Collapse
Affiliation(s)
- Qi Zhang
- School of Information Technology & Management, University of International Business & Economics, Beijing 100029, China
| | - Yinglu Liang
- School of Information Technology & Management, University of International Business & Economics, Beijing 100029, China
| | - Yi Zhang
- School of Information Technology & Management, University of International Business & Economics, Beijing 100029, China
| | - Zihao Tao
- Department of Urology, Peking University Third Hospital, Beijing 100191, China
| | - Rui Li
- School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, China.
| | - Hai Bi
- Department of Urology, Peking University Third Hospital, Beijing 100191, China.
| |
Collapse
|
3
|
Zhang J, Tao X, Jiang Y, Wu X, Yan D, Xue W, Zhuang S, Chen L, Luo L, Ni D. Application of Convolution Neural Network Algorithm Based on Multicenter ABUS Images in Breast Lesion Detection. Front Oncol 2022; 12:938413. [PMID: 35898876 PMCID: PMC9310547 DOI: 10.3389/fonc.2022.938413] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2022] [Accepted: 05/30/2022] [Indexed: 11/24/2022] Open
Abstract
Objective This study aimed to evaluate a convolution neural network algorithm for breast lesion detection with multi-center ABUS image data developed based on ABUS image and Yolo v5. Methods A total of 741 cases with 2,538 volume data of ABUS examinations were analyzed, which were recruited from 7 hospitals between October 2016 and December 2020. A total of 452 volume data of 413 cases were used as internal validation data, and 2,086 volume data from 328 cases were used as external validation data. There were 1,178 breast lesions in 413 patients (161 malignant and 1,017 benign) and 1,936 lesions in 328 patients (57 malignant and 1,879 benign). The efficiency and accuracy of the algorithm were analyzed in detecting lesions with different allowable false positive values and lesion sizes, and the differences were compared and analyzed, which included the various indicators in internal validation and external validation data. Results The study found that the algorithm had high sensitivity for all categories of lesions, even when using internal or external validation data. The overall detection rate of the algorithm was as high as 78.1 and 71.2% in the internal and external validation sets, respectively. The algorithm could detect more lesions with increasing nodule size (87.4% in ≥10 mm lesions but less than 50% in <10 mm). The detection rate of BI-RADS 4/5 lesions was higher than that of BI-RADS 3 or 2 (96.5% vs 79.7% vs 74.7% internal, 95.8% vs 74.7% vs 88.4% external). Furthermore, the detection performance was better for malignant nodules than benign (98.1% vs 74.9% internal, 98.2% vs 70.4% external). Conclusions This algorithm showed good detection efficiency in the internal and external validation sets, especially for category 4/5 lesions and malignant lesions. However, there are still some deficiencies in detecting category 2 and 3 lesions and lesions smaller than 10 mm.
Collapse
Affiliation(s)
- Jianxing Zhang
- Department of Medical Imaging Center, The First Affiliated Hospital, Jinan University, Guangzhou, China
- Department of Ultrasound, Remote Consultation Center of ABUS, The Second Affiliated Hospital, Guangzhou University of Chinese Medicine, Guangzhou, China
- *Correspondence: Jianxing Zhang, ; Liangping Luo, ; Dong Ni,
| | - Xing Tao
- Medical Ultrasound Image Computing Lab, Shenzhen University, Shenzhen, China
| | - Yanhui Jiang
- Medical Ultrasound Image Computing Lab, Shenzhen University, Shenzhen, China
| | - Xiaoxi Wu
- Department of Ultrasound, Remote Consultation Center of ABUS, The Second Affiliated Hospital, Guangzhou University of Chinese Medicine, Guangzhou, China
| | - Dan Yan
- Department of Ultrasound, Remote Consultation Center of ABUS, The Second Affiliated Hospital, Guangzhou University of Chinese Medicine, Guangzhou, China
| | - Wen Xue
- Department of Ultrasound, Remote Consultation Center of ABUS, The Second Affiliated Hospital, Guangzhou University of Chinese Medicine, Guangzhou, China
| | - Shulian Zhuang
- Department of Ultrasound, Remote Consultation Center of ABUS, The Second Affiliated Hospital, Guangzhou University of Chinese Medicine, Guangzhou, China
| | - Ling Chen
- Department of Ultrasound, Remote Consultation Center of ABUS, The Second Affiliated Hospital, Guangzhou University of Chinese Medicine, Guangzhou, China
| | - Liangping Luo
- Department of Medical Imaging Center, The First Affiliated Hospital, Jinan University, Guangzhou, China
- *Correspondence: Jianxing Zhang, ; Liangping Luo, ; Dong Ni,
| | - Dong Ni
- Medical Ultrasound Image Computing Lab, Shenzhen University, Shenzhen, China
- *Correspondence: Jianxing Zhang, ; Liangping Luo, ; Dong Ni,
| |
Collapse
|
4
|
Jeong JJ, Tariq A, Adejumo T, Trivedi H, Gichoya JW, Banerjee I. Systematic Review of Generative Adversarial Networks (GANs) for Medical Image Classification and Segmentation. J Digit Imaging 2022; 35:137-152. [PMID: 35022924 PMCID: PMC8921387 DOI: 10.1007/s10278-021-00556-w] [Citation(s) in RCA: 31] [Impact Index Per Article: 15.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2021] [Revised: 11/23/2021] [Accepted: 11/26/2021] [Indexed: 11/28/2022] Open
Abstract
In recent years, generative adversarial networks (GANs) have gained tremendous popularity for various imaging related tasks such as artificial image generation to support AI training. GANs are especially useful for medical imaging-related tasks where training datasets are usually limited in size and heavily imbalanced against the diseased class. We present a systematic review, following the PRISMA guidelines, of recent GAN architectures used for medical image analysis to help the readers in making an informed decision before employing GANs in developing medical image classification and segmentation models. We have extracted 54 papers that highlight the capabilities and application of GANs in medical imaging from January 2015 to August 2020 and inclusion criteria for meta-analysis. Our results show four main architectures of GAN that are used for segmentation or classification in medical imaging. We provide a comprehensive overview of recent trends in the application of GANs in clinical diagnosis through medical image segmentation and classification and ultimately share experiences for task-based GAN implementations.
Collapse
Affiliation(s)
- Jiwoong J Jeong
- Department of Biomedical Informatics, Emory School of Medicine, Atlanta, USA.
| | - Amara Tariq
- Department of Biomedical Informatics, Emory School of Medicine, Atlanta, USA
| | | | - Hari Trivedi
- Department of Radiology, Emory School of Medicine, Atlanta, USA
| | - Judy W Gichoya
- Department of Radiology, Emory School of Medicine, Atlanta, USA
| | - Imon Banerjee
- Department of Biomedical Informatics, Emory School of Medicine, Atlanta, USA.,Department of Radiology, Emory School of Medicine, Atlanta, USA
| |
Collapse
|
5
|
Deep convolutional neural networks for computer-aided breast cancer diagnostic: a survey. Neural Comput Appl 2022. [DOI: 10.1007/s00521-021-06804-y] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
|
6
|
Xun S, Li D, Zhu H, Chen M, Wang J, Li J, Chen M, Wu B, Zhang H, Chai X, Jiang Z, Zhang Y, Huang P. Generative adversarial networks in medical image segmentation: A review. Comput Biol Med 2022; 140:105063. [PMID: 34864584 DOI: 10.1016/j.compbiomed.2021.105063] [Citation(s) in RCA: 24] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2021] [Revised: 11/14/2021] [Accepted: 11/20/2021] [Indexed: 12/13/2022]
Abstract
PURPOSE Since Generative Adversarial Network (GAN) was introduced into the field of deep learning in 2014, it has received extensive attention from academia and industry, and a lot of high-quality papers have been published. GAN effectively improves the accuracy of medical image segmentation because of its good generating ability and capability to capture data distribution. This paper introduces the origin, working principle, and extended variant of GAN, and it reviews the latest development of GAN-based medical image segmentation methods. METHOD To find the papers, we searched on Google Scholar and PubMed with the keywords like "segmentation", "medical image", and "GAN (or generative adversarial network)". Also, additional searches were performed on Semantic Scholar, Springer, arXiv, and the top conferences in computer science with the above keywords related to GAN. RESULTS We reviewed more than 120 GAN-based architectures for medical image segmentation that were published before September 2021. We categorized and summarized these papers according to the segmentation regions, imaging modality, and classification methods. Besides, we discussed the advantages, challenges, and future research directions of GAN in medical image segmentation. CONCLUSIONS We discussed in detail the recent papers on medical image segmentation using GAN. The application of GAN and its extended variants has effectively improved the accuracy of medical image segmentation. Obtaining the recognition of clinicians and patients and overcoming the instability, low repeatability, and uninterpretability of GAN will be an important research direction in the future.
Collapse
Affiliation(s)
- Siyi Xun
- Shandong Key Laboratory of Medical Physics and Image Processing, Shandong Institute of Industrial Technology for Health Sciences and Precision Medicine, School of Physics and Electronics, Shandong Normal University, Jinan, Shandong, 250358, China
| | - Dengwang Li
- Shandong Key Laboratory of Medical Physics and Image Processing, Shandong Institute of Industrial Technology for Health Sciences and Precision Medicine, School of Physics and Electronics, Shandong Normal University, Jinan, Shandong, 250358, China.
| | - Hui Zhu
- Department of Radiation Oncology, Shandong Cancer Hospital and Institute, Shandong First Medical University and Shandong Academy of Medical Sciences, Jinan, Shandong, China
| | - Min Chen
- The Second Hospital of Shandong University, Shandong University, The Department of Medicine, The Second Hospital of Shandong University, Jinan, China
| | - Jianbo Wang
- Department of Radiation Oncology, Qilu Hospital, Cheeloo College of Medicine, Shandong University, Jinan, Shandong, 250012, China
| | - Jie Li
- Department of Infectious Disease, Shandong Provincial Hospital Affiliated to Shandong University, Cheeloo College of Medicine, Shandong University, Jinan, Shandong, China
| | - Meirong Chen
- Affiliated Hospital of Shandong University of Traditional Chinese Medicine, Jinan, Shandong, China
| | - Bing Wu
- Laibo Biotechnology Co., Ltd., Jinan, Shandong, China
| | - Hua Zhang
- LinkingMed Technology Co., Ltd., Beijing, China
| | - Xiangfei Chai
- Huiying Medical Technology (Beijing) Co., Ltd., Beijing, China
| | - Zekun Jiang
- Shandong Key Laboratory of Medical Physics and Image Processing, Shandong Institute of Industrial Technology for Health Sciences and Precision Medicine, School of Physics and Electronics, Shandong Normal University, Jinan, Shandong, 250358, China
| | - Yan Zhang
- Shandong Key Laboratory of Medical Physics and Image Processing, Shandong Institute of Industrial Technology for Health Sciences and Precision Medicine, School of Physics and Electronics, Shandong Normal University, Jinan, Shandong, 250358, China
| | - Pu Huang
- Shandong Key Laboratory of Medical Physics and Image Processing, Shandong Institute of Industrial Technology for Health Sciences and Precision Medicine, School of Physics and Electronics, Shandong Normal University, Jinan, Shandong, 250358, China.
| |
Collapse
|
7
|
Liu F, Li G, Lin L. A novel method for selecting the set optimal wavelength combination in multi-spectral transmission image. SPECTROCHIMICA ACTA. PART A, MOLECULAR AND BIOMOLECULAR SPECTROSCOPY 2021; 261:120080. [PMID: 34147734 DOI: 10.1016/j.saa.2021.120080] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/23/2021] [Revised: 05/24/2021] [Accepted: 06/09/2021] [Indexed: 06/12/2023]
Abstract
In the process of detecting heterogeneity in breast tissue based on multi-spectral transmission imaging, the detection accuracy will be affected due to the high redundancy degree of information between bands. In order to select the reasonable wavelength combination, this paper uses various nonlinear transformations to convert the multi-spectral images into spectral data for the first time, so as to select the set optimal wavelength combination based on the successive projections algorithm (SPA). Firstly, we design the collection experiment of 4-wavelength multi-spectral image. And then, K-SVD dictionary learning method, texture extraction method and gray correlation analysis method are used to obtain the feature spectral information. Finally, the set optimal wavelength combination is selected based on SPA. The experimental results show that random forest (RF) classification model and Faster-RCNN recognition models effectively verify that the combination of wavelengths 1,2,4 selected has the highest accuracy in the heterogeneous detection. In conclusion, this paper uses modulation-frame accumulation technique to improve the quality of multi-spectral transmission images. And based on the RF and Faster-RCNN models, the effectiveness of SPA-based optimal wavelength combination method proposed is verified, which will provide a new idea of feature wavelength selection for screening early breast masses through multi-spectral transmission imaging.
Collapse
Affiliation(s)
- Fulong Liu
- State Key Laboratory of Precision Measuring Technology and Instruments, Tianjin University, Tianjin 300072, China; Tianjin Key Laboratory of Biomedical Detecting Techniques and Instruments, Tianjin University, Tianjin 300072, China
| | - Gang Li
- State Key Laboratory of Precision Measuring Technology and Instruments, Tianjin University, Tianjin 300072, China; Tianjin Key Laboratory of Biomedical Detecting Techniques and Instruments, Tianjin University, Tianjin 300072, China
| | - Ling Lin
- State Key Laboratory of Precision Measuring Technology and Instruments, Tianjin University, Tianjin 300072, China; Tianjin Key Laboratory of Biomedical Detecting Techniques and Instruments, Tianjin University, Tianjin 300072, China.
| |
Collapse
|
8
|
Tian Y, Li E, Liang Z, Tan M, He X. Diagnosis of Typical Apple Diseases: A Deep Learning Method Based on Multi-Scale Dense Classification Network. FRONTIERS IN PLANT SCIENCE 2021; 12:698474. [PMID: 34659279 PMCID: PMC8517256 DOI: 10.3389/fpls.2021.698474] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/22/2021] [Accepted: 09/06/2020] [Indexed: 06/13/2023]
Abstract
Disease has always been one of the main reasons for the decline of apple quality and yield, which directly harms the development of agricultural economy. Therefore, precise diagnosis of apple diseases and correct decision making are important measures to reduce agricultural losses and promote economic growth. In this paper, a novel Multi-scale Dense classification network is adopted to realize the diagnosis of 11 types of images, including healthy and diseased apple fruits and leaves. The diagnosis of different kinds of diseases and the same disease with different grades was accomplished. First of all, to solve the problem of insufficient images of anthracnose and ring rot, Cycle-GAN algorithm was applied to achieve dataset expansion on the basis of traditional image augmentation methods. Cycle-GAN learned the image characteristics of healthy apples and diseased apples to generate anthracnose and ring rot lesions on the surface of healthy apple fruits. The diseased apple images generated by Cycle-GAN were added to the training set, which improved the diagnosis performance compared with other traditional image augmentation methods. Subsequently, DenseNet and Multi-scale connection were adopted to establish two kinds of models, Multi-scale Dense Inception-V4 and Multi-scale Dense Inception-Resnet-V2, which facilitated the reuse of image features of the bottom layers in the classification neural networks. Both models accomplished the diagnosis of 11 different types of images. The classification accuracy was 94.31 and 94.74%, respectively, which exceeded DenseNet-121 network and reached the state-of-the-art level.
Collapse
Affiliation(s)
- Yunong Tian
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - En Li
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Zize Liang
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Min Tan
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Xiongkui He
- College of Science, China Agricultural University, Beijing, China
| |
Collapse
|
9
|
Cao X, Chen H, Li Y, Peng Y, Wang S, Cheng L. Dilated densely connected U-Net with uncertainty focus loss for 3D ABUS mass segmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 209:106313. [PMID: 34364182 DOI: 10.1016/j.cmpb.2021.106313] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/25/2020] [Accepted: 07/21/2021] [Indexed: 06/13/2023]
Abstract
BACKGROUND AND OBJECTIVE Accurate segmentation of breast mass in 3D automated breast ultrasound (ABUS) images plays an important role in qualitative and quantitative ABUS image analysis. Yet this task is challenging due to the low signal to noise ratio and serious artifacts in ABUS images, the large shape and size variation of breast masses, as well as the small training dataset compared with natural images. The purpose of this study is to address these difficulties by designing a dilated densely connected U-Net (D2U-Net) together with an uncertainty focus loss. METHODS A lightweight yet effective densely connected segmentation network is constructed to extensively explore feature representations in the small ABUS dataset. In order to deal with the high variation in shape and size of breast masses, a set of hybrid dilated convolutions is integrated into the dense blocks of the D2U-Net. We further suggest an uncertainty focus loss to put more attention on unreliable network predictions, especially the ambiguous mass boundaries caused by low signal to noise ratio and artifacts. Our segmentation algorithm is evaluated on an ABUS dataset of 170 volumes from 107 patients. Ablation analysis and comparison with existing methods are conduct to verify the effectiveness of the proposed method. RESULTS Experiment results demonstrate that the proposed algorithm outperforms existing methods on 3D ABUS mass segmentation tasks, with Dice similarity coefficient, Jaccard index and 95% Hausdorff distance of 69.02%, 56.61% and 4.92 mm, respectively. CONCLUSIONS The proposed method is effective in segmenting breast masses on our small ABUS dataset, especially breast masses with large shape and size variations.
Collapse
Affiliation(s)
- Xuyang Cao
- School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China
| | - Houjin Chen
- School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China.
| | - Yanfeng Li
- School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China
| | - Yahui Peng
- School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing 100044, China
| | - Shu Wang
- Peking University People's Hospital, Beijing 100044, China
| | - Lin Cheng
- Peking University People's Hospital, Beijing 100044, China
| |
Collapse
|
10
|
|
11
|
Shen T, Hao K, Gou C, Wang FY. Mass Image Synthesis in Mammogram with Contextual Information Based on GANs. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 202:106019. [PMID: 33640650 DOI: 10.1016/j.cmpb.2021.106019] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/15/2020] [Accepted: 02/16/2021] [Indexed: 06/12/2023]
Abstract
BACKGROUND AND OBJECTIVE In medical imaging, the scarcity of labeled lesion data has hindered the application of many deep learning algorithms. To overcome this problem, the simulation of diverse lesions in medical images is proposed. However, synthesizing labeled mass images in mammograms is still challenging due to the lack of consistent patterns in shape, margin, and contextual information. Therefore, we aim to generate various labeled medical images based on contextual information in mammograms. METHODS In this paper, we propose a novel approach based on GANs to generate various mass images and then perform contextual infilling by inserting the synthetic lesions into healthy screening mammograms. Through incorporating features of both realistic mass images and corresponding masks into the adversarial learning scheme, the generator can not only learn the distribution of the real mass images but also capture the matching shape, margin and context information. RESULTS To demonstrate the effectiveness of our proposed method, we conduct experiments on publicly available mammogram database of DDSM and a private database provided by Nanfang Hospital in China. Qualitative and quantitative evaluations validate the effectiveness of our approach. Additionally, through the data augmentation by image generation of the proposed method, an improvement of 5.03% in detection rate can be achieved over the same model trained on original real lesion images. CONCLUSIONS The results show that the data augmentation based on our method increases the diversity of dataset. Our method can be viewed as one of the first steps toward generating labeled breast mass images for precise detection and can be extended in other medical imaging domains to solve similar problems.
Collapse
Affiliation(s)
- Tianyu Shen
- Institute of Automation, Chinese Academy of Sciences, Beijing, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
| | - Kunkun Hao
- School of Software Engineering, Xi'an Jiaotong University, Xi'an, China
| | - Chao Gou
- School of Intelligent Systems Engineering, Sun Yat-Sen University, Guangzhou, China.
| | - Fei-Yue Wang
- Institute of Automation, Chinese Academy of Sciences, Beijing, China; Qingdao Academy of Intelligent Industries, Qingdao, China; Institute of Systems Engineering, Macau University of Science and Technology, Macau, China
| |
Collapse
|
12
|
Marzullo A, Moccia S, Catellani M, Calimeri F, Momi ED. Towards realistic laparoscopic image generation using image-domain translation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 200:105834. [PMID: 33229016 DOI: 10.1016/j.cmpb.2020.105834] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/28/2020] [Accepted: 11/05/2020] [Indexed: 06/11/2023]
Abstract
Background and ObjectivesOver the last decade, Deep Learning (DL) has revolutionized data analysis in many areas, including medical imaging. However, there is a bottleneck in the advancement of DL in the surgery field, which can be seen in a shortage of large-scale data, which in turn may be attributed to the lack of a structured and standardized methodology for storing and analyzing surgical images in clinical centres. Furthermore, accurate annotations manually added are expensive and time consuming. A great help can come from the synthesis of artificial images; in this context, in the latest years, the use of Generative Adversarial Neural Networks (GANs) achieved promising results in obtaining photo-realistic images. MethodsIn this study, a method for Minimally Invasive Surgery (MIS) image synthesis is proposed. To this aim, the generative adversarial network pix2pix is trained to generate paired annotated MIS images by transforming rough segmentation of surgical instruments and tissues into realistic images. An additional regularization term was added to the original optimization problem, in order to enhance realism of surgical tools with respect to the background. Results Quantitative and qualitative (i.e., human-based) evaluations of generated images have been carried out in order to assess the effectiveness of the method. ConclusionsExperimental results show that the proposed method is actually able to translate MIS segmentations to realistic MIS images, which can in turn be used to augment existing data sets and help at overcoming the lack of useful images; this allows physicians and algorithms to take advantage from new annotated instances for their training.
Collapse
Affiliation(s)
- Aldo Marzullo
- Department of Mathematics and Computer Science, University of Calabria, Rende, Italy.
| | - Sara Moccia
- Department of Information Engineering, Unviersitá Politecnica delle Marche, Ancona, Italy; Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Michele Catellani
- Department of urology, European Institute of Oncology, IRCCS, Milan, Italy
| | - Francesco Calimeri
- Department of Mathematics and Computer Science, University of Calabria, Rende, Italy
| | - Elena De Momi
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy
| |
Collapse
|
13
|
Li Y, Zhao G, Zhang Q, Lin Y, Wang M. SAP-cGAN: Adversarial learning for breast mass segmentation in digital mammogram based on superpixel average pooling. Med Phys 2021; 48:1157-1167. [PMID: 33340125 DOI: 10.1002/mp.14671] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2020] [Revised: 12/11/2020] [Accepted: 12/11/2020] [Indexed: 01/23/2023] Open
Abstract
PURPOSE Breast mass segmentation is a prerequisite step in the use of computer-aided tools designed for breast cancer diagnosis and treatment planning. However, mass segmentation remains challenging due to the low contrast, irregular shapes, and fuzzy boundaries of masses. In this work, we propose a mammography mass segmentation model for improving segmentation performance. METHODS We propose a mammography mass segmentation model called SAP-cGAN, which is based on an improved conditional generative adversarial network (cGAN). We introduce a superpixel average pooling layer into the cGAN decoder, which utilizes superpixels as a pooling layout to improve boundary segmentation. In addition, we adopt a multiscale input strategy to enable the network to learn scale-invariant features with increased robustness. The performance of the model is evaluated with two public datasets: CBIS-DDSM and INbreast. Moreover, ablation analysis is conducted to evaluate further the individual contribution of each block to the performance of the network. RESULTS Dice and Jaccard scores of 93.37% and 87.57%, respectively, are obtained for the CBIS-DDSM dataset. The Dice and Jaccard scores for the INbreast dataset are 91.54% and 84.40%, respectively. These results indicate that our proposed model outperforms current state-of-the-art breast mass segmentation methods. The superpixel average pooling layer and multiscale input strategy has improved the Dice and Jaccard scores of the original cGAN by 7.8% and 12.79%, respectively. CONCLUSIONS Adversarial learning with the addition of a superpixel average pooling layer and multiscale input strategy can encourage the Generator network to generate masks with increased realism and improve breast mass segmentation performance through the minimax game between the Generator network and Discriminator network.
Collapse
Affiliation(s)
- Yamei Li
- School of Information Engineering, Zhengzhou University, Zhengzhou, 450001, China.,Collaborative Innovation Center for Internet Healthcare, Zhengzhou University, Zhengzhou, 450052, China
| | - Guohua Zhao
- School of Information Engineering, Zhengzhou University, Zhengzhou, 450001, China.,Collaborative Innovation Center for Internet Healthcare, Zhengzhou University, Zhengzhou, 450052, China
| | - Qian Zhang
- School of Computer Science, Zhongyuan University of Technology, Zhengzhou, 450007, China
| | - Yusong Lin
- Collaborative Innovation Center for Internet Healthcare, Zhengzhou University, Zhengzhou, 450052, China.,School of Software, Zhengzhou University, Zhengzhou, 450002, China.,Hanwei IoT Institute, Zhengzhou University, Zhengzhou, 450002, China
| | - Meiyun Wang
- Collaborative Innovation Center for Internet Healthcare, Zhengzhou University, Zhengzhou, 450052, China.,Department of Radiology, People's Hospital of Zhengzhou University, Zhengzhou, 450003, China
| |
Collapse
|
14
|
Zhang Z, Gao S, Huang Z. An Automatic Glioma Segmentation System Using a Multilevel Attention Pyramid Scene Parsing Network. Curr Med Imaging 2020; 17:751-761. [PMID: 33390119 DOI: 10.2174/1573405616666201231100623] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2020] [Revised: 09/15/2020] [Accepted: 10/15/2020] [Indexed: 11/22/2022]
Abstract
BACKGROUND Due to the significant variances in their shape and size, it is a challenging task to automatically segment gliomas. To improve the performance of glioma segmentation tasks, this paper proposed a multilevel attention pyramid scene parsing network (MLAPSPNet) that aggregates the multiscale context and multilevel features. METHODS First, T1 pre-contrast, T2-weighted fluid-attenuated inversion recovery (FLAIR) and T1 post-contrast sequences of each slice are combined to form the input. Afterwards, image normalization and augmentation techniques are applied to accelerate the training process and avoid overfitting, respectively. Furthermore, the proposed MLAPSPNet that introduces multilevel pyramid pooling modules (PPMs) and attention gates is constructed. Eventually, the proposed network is compared with some existing networks. RESULTS The dice similarity coefficient (DSC), sensitivity and Jaccard score of the proposed system can reach 0.885, 0.933 and 0.8, respectively. The introduction of multilevel pyramid pooling modules and attention gates can improve the DSC by 0.029 and 0.022, respectively. Moreover, compared with Res-UNet, Dense-UNet, residual channel attention UNet (RCA-UNet), DeepLab V3+ and UNet++, the DSC is improved by 0.032, 0.026, 0.014, 0.041 and 0.011, respectively. CONCLUSION The proposed multilevel attention pyramid scene parsing network can achieve stateof- the-art performance, and the introduction of multilevel pyramid pooling modules and attention gates can improve the performance of glioma segmentation tasks.
Collapse
Affiliation(s)
- Zhenyu Zhang
- School of Mechatronic Engineering and Automation, Shanghai University, Shanghai, China
| | - Shouwei Gao
- School of Mechatronic Engineering and Automation, Shanghai University, Shanghai, China
| | - Zheng Huang
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China
| |
Collapse
|
15
|
Yu D, Zhang K, Huang L, Zhao B, Zhang X, Guo X, Li M, Gu Z, Fu G, Hu M, Ping Y, Sheng Y, Liu Z, Hu X, Zhao R. Detection of peripherally inserted central catheter (PICC) in chest X-ray images: A multi-task deep learning model. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2020; 197:105674. [PMID: 32738678 DOI: 10.1016/j.cmpb.2020.105674] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/19/2020] [Accepted: 07/17/2020] [Indexed: 06/11/2023]
Abstract
BACKGROUND AND OBJECTIVE Peripherally inserted central catheter (PICC) is a novel drug delivery mode which has been widely used in clinical practice. However, long-term retention and some improper actions of patients may cause some severe complications of PICC, such as the drift and prolapse of its catheter. Clinically, the postoperative care of PICC is mainly completed by nurses. However, they cannot recognize the correct position of PICC from X-ray chest images as soon as the complications happen, which may lead to improper treatment. Therefore, it is necessary to identify the position of the PICC catheter as soon as these complications occur. Here we proposed a novel multi-task deep learning framework to detect PICC automatically through X-ray images, which could help nurses to solve this problem. METHODS We collected 348 X-ray chest images from 326 patients with visible PICC. Then we proposed a multi-task deep learning framework for line segmentation and tip detection of PICC catheters simultaneously. The proposed deep learning model is composed of an extraction structure and three routes, an up-sampling route for segmentation, an RPNs route, and an RoI Pooling route for detection. We further compared the effectiveness of our model with the models previously proposed. RESULTS In the catheter segmentation task, 300 X-ray images were utilized for training the model, then 48 images were tested. In the tip detection task, 154 X-ray images were used for retraining and 20 images were used in the test. Our model achieved generally better results among several popular deep learning models previously proposed. CONCLUSIONS We proposed a multi-task deep learning model that could segment the catheter and detect the tip of PICC simultaneously from X-ray chest images. This model could help nurses to recognize the correct position of PICC, and therefore, to handle the potential complications properly.
Collapse
Affiliation(s)
- Dingding Yu
- School of Mathematical Sciences, Zhejiang University. Hangzhou, Zhejiang Province, China, 310027
| | - Kaijie Zhang
- Department of Vascular Surgery, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang Province, China, 310009; Key Laboratory of Cardiovascular Intervention and Regenerative Medicine of Zhejiang Province. Sir Run Shaw Hospital, School of Medicine, Zhejiang University. Hangzhou, Zhejiang Province, China, 310016
| | - Lingyan Huang
- Department of Radiation Oncology, Zhejiang Quhua Hospital, Quzhou, Zhejiang Province, China, 324000
| | - Bonan Zhao
- School of Mathematical Sciences, Zhejiang University. Hangzhou, Zhejiang Province, China, 310027
| | - Xiaoshan Zhang
- School of Mathematical Sciences, Zhejiang University. Hangzhou, Zhejiang Province, China, 310027
| | - Xin Guo
- Department of Vascular Surgery, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang Province, China, 310009; Bone Marrow Transplantation Center, The First Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang Province, China, 310000
| | - Miaomiao Li
- Department of Vascular Surgery, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang Province, China, 310009; Department of Reproductive Endocrinology, Women's Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang Province, China, 310019
| | - Zheng Gu
- Department of Vascular Surgery, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang Province, China, 310009
| | - Guosheng Fu
- Key Laboratory of Cardiovascular Intervention and Regenerative Medicine of Zhejiang Province. Sir Run Shaw Hospital, School of Medicine, Zhejiang University. Hangzhou, Zhejiang Province, China, 310016
| | - Minchun Hu
- Department of Radiation Oncology, Zhejiang Quhua Hospital, Quzhou, Zhejiang Province, China, 324000
| | - Yan Ping
- Department of Radiation Oncology, Zhejiang Quhua Hospital, Quzhou, Zhejiang Province, China, 324000
| | - Ye Sheng
- Department of Nursing, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang Province, China, 310009
| | - Zhenjie Liu
- Department of Vascular Surgery, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang Province, China, 310009.
| | - Xianliang Hu
- School of Mathematical Sciences, Zhejiang University. Hangzhou, Zhejiang Province, China, 310027.
| | - Ruiyi Zhao
- Department of Nursing, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang Province, China, 310009.
| |
Collapse
|
16
|
Wang Y, Wang S, Chen J, Wu C. Whole mammographic mass segmentation using attention mechanism and multiscale pooling adversarial network. J Med Imaging (Bellingham) 2020; 7:054503. [PMID: 33102621 DOI: 10.1117/1.jmi.7.5.054503] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2020] [Accepted: 09/28/2020] [Indexed: 12/24/2022] Open
Abstract
Purpose: Since breast mass is a clear sign of breast cancer, its precise segmentation is of great significance for the diagnosis of breast cancer. However, the current diagnosis relies mainly on radiologists who spend time extracting features manually, which inevitably reduces the efficiency of diagnosis. Therefore, designing an automatic segmentation method is urgently necessary for the accurate segmentation of breast masses. Approach: We propose an effective attention mechanism and multiscale pooling conditional generative adversarial network (AM-MSP-cGAN), which accurately achieves mass automatic segmentation in whole mammograms. In AM-MSP-cGAN, U-Net is utilized as a generator network by incorporating attention mechanism (AM) into it, which allows U-Net to pay more attention to the target mass regions without additional cost. As a discriminator network, a convolutional neural network with multiscale pooling module is used to learn more meticulous features from the masses with rough and fuzzy boundaries. The proposed model is trained and tested on two public datasets: CBIS-DDSM and INbreast. Results: Compared with other state-of-the-art methods, the AM-MSP-cGAN can achieve better segmentation results in terms of the dice similarity coefficient (Dice) and Hausdorff distance metrics, achieving top scores of 84.49% and 5.01 on CBIS-DDSM, and 83.92% and 5.81 on INbreast, respectively. Therefore, qualitative and quantitative experiments illustrate that the proposed model is effective and robust for the mass segmentation in whole mammograms. Conclusions: The proposed deep learning model is suitable for the automatic segmentation of breast masses, which provides technical assistance for subsequent pathological structure analysis.
Collapse
Affiliation(s)
- Yuehang Wang
- Jilin University, College of Software, Changchun, China
| | - Shengsheng Wang
- Jilin University, College of Computer Science and Technology, Changchun, China
| | - Juan Chen
- Jilin University, College of Computer Science and Technology, Changchun, China
| | - Chun Wu
- Jilin University, College of Software, Changchun, China
| |
Collapse
|