1
|
Shao Y, Zhou K, Zhang L. CSSNet: Cascaded spatial shift network for multi- organ segmentation. Comput Biol Med 2024; 170:107955. [PMID: 38215618 DOI: 10.1016/j.compbiomed.2024.107955] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 12/26/2023] [Accepted: 01/01/2024] [Indexed: 01/14/2024]
Abstract
Multi-organ segmentation is vital for clinical diagnosis and treatment. Although CNN and its extensions are popular in organ segmentation, they suffer from the local receptive field. In contrast, MultiLayer-Perceptron-based models (e.g., MLP-Mixer) have a global receptive field. However, these MLP-based models employ fully connected layers with many parameters and tend to overfit on sample-deficient medical image datasets. Therefore, we propose a Cascaded Spatial Shift Network, CSSNet, for multi-organ segmentation. Specifically, we design a novel cascaded spatial shift block to reduce the number of model parameters and aggregate feature segments in a cascaded way for efficient and effective feature extraction. Then, we propose a feature refinement network to aggregate multi-scale features with location information, and enhance the multi-scale features along the channel and spatial axis to obtain a high-quality feature map. Finally, we employ a self-attention-based fusion strategy to focus on the discriminative feature information for better multi-organ segmentation performance. Experimental results on the Synapse (multiply organs) and LiTS (liver & tumor) datasets demonstrate that our CSSNet achieves promising segmentation performance compared with CNN, MLP, and Transformer models. The source code will be available at https://github.com/zkyseu/CSSNet.
Collapse
Affiliation(s)
- Yeqin Shao
- School of Transportation, Nantong University, Jiangsu, 226019, China.
| | - Kunyang Zhou
- School of Zhangjian, Nantong University, Jiangsu, 226019, China
| | - Lichi Zhang
- School of Biomedical Engineering, Shanghai Jiaotong University, Shanghai, 200240, China
| |
Collapse
|
2
|
Zhuang M, Chen Z, Yang Y, Kettunen L, Wang H. Annotation-efficient training of medical image segmentation network based on scribble guidance in difficult areas. Int J Comput Assist Radiol Surg 2024; 19:87-96. [PMID: 37233894 DOI: 10.1007/s11548-023-02931-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/01/2023] [Accepted: 04/19/2023] [Indexed: 05/27/2023]
Abstract
PURPOSE The training of deep medical image segmentation networks usually requires a large amount of human-annotated data. To alleviate the burden of human labor, many semi- or non-supervised methods have been developed. However, due to the complexity of clinical scenario, insufficient training labels still causes inaccurate segmentation in some difficult local areas such as heterogeneous tumors and fuzzy boundaries. METHODS We propose an annotation-efficient training approach, which only requires scribble guidance in the difficult areas. A segmentation network is initially trained with a small amount of fully annotated data and then used to produce pseudo labels for more training data. Human supervisors draw scribbles in the areas of incorrect pseudo labels (i.e., difficult areas), and the scribbles are converted into pseudo label maps using a probability-modulated geodesic transform. To reduce the influence of the potential errors in the pseudo labels, a confidence map of the pseudo labels is generated by jointly considering the pixel-to-scribble geodesic distance and the network output probability. The pseudo labels and confidence maps are iteratively optimized with the update of the network, and the network training is promoted by the pseudo labels and the confidence maps in turn. RESULTS Cross-validation based on two data sets (brain tumor MRI and liver tumor CT) showed that our method significantly reduces the annotation time while maintains the segmentation accuracy of difficult areas (e.g., tumors). Using 90 scribble-annotated training images (annotated time: ~ 9 h), our method achieved the same performance as using 45 fully annotated images (annotation time: > 100 h) but required much shorter annotation time. CONCLUSION Compared to the conventional full annotation approaches, the proposed method significantly saves the annotation efforts by focusing the human supervisions on the most difficult regions. It provides an annotation-efficient way for training medical image segmentation networks in complex clinical scenario.
Collapse
Affiliation(s)
- Mingrui Zhuang
- School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, 116024, China
| | - Zhonghua Chen
- School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, 116024, China
- Faculty of Information Technology, University of Jyväskylä, 40100, Jyvaskyla, Finland
| | - Yuxin Yang
- School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, 116024, China
| | - Lauri Kettunen
- Faculty of Information Technology, University of Jyväskylä, 40100, Jyvaskyla, Finland
| | - Hongkai Wang
- School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, 116024, China.
- Liaoning Key Laboratory of Integrated Circuit and Biomedical Electronic System, Dalian, China.
| |
Collapse
|
3
|
Zhang W, Wu S, Wen W, Lu X, Wang C, Gou W, Li Y, Guo X, Zhao C. Three-dimensional branch segmentation and phenotype extraction of maize tassel based on deep learning. Plant Methods 2023; 19:76. [PMID: 37528454 PMCID: PMC10394845 DOI: 10.1186/s13007-023-01051-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Accepted: 07/10/2023] [Indexed: 08/03/2023]
Abstract
BACKGROUND The morphological structure phenotype of maize tassel plays an important role in plant growth, reproduction, and yield formation. It is an important step in the distinctness, uniformity, and stability (DUS) testing to obtain maize tassel phenotype traits. Plant organ segmentation can be achieved with high-precision and automated acquisition of maize tassel phenotype traits because of the advances in the point cloud deep learning method. However, this method requires a large number of data sets and is not robust to automatic segmentation of highly adherent organ components; thus, it should be combined with point cloud processing technology. RESULTS An innovative method of incomplete annotation of point cloud data was proposed for easy development of the dataset of maize tassels,and an automatic maize tassel phenotype analysis system: MaizeTasselSeg was developed. The tip feature of point cloud is trained and learned based on PointNet + + network, and the tip point cloud of tassel branch was automatically segmented. Complete branch segmentation was realized based on the shortest path algorithm. The Intersection over Union (IoU), precision, and recall of the segmentation results were 96.29, 96.36, and 93.01, respectively. Six phenotypic traits related to morphological structure (branch count, branch length, branch angle, branch curvature, tassel volume, and dispersion) were automatically extracted from the segmentation point cloud. The squared correlation coefficients (R2) for branch length, branch angle, and branch count were 0.9897, 0.9317, and 0.9587, respectively. The root mean squared error (RMSE) for branch length, branch angle, and branch count were 0.529 cm, 4.516, and 0.875, respectively. CONCLUSION The proposed method provides an efficient scheme for high-throughput organ segmentation of maize tassels and can be used for the automatic extraction of phenotypic traits of maize tassel. In addition, the incomplete annotation approach provides a new idea for morphology-based plant segmentation.
Collapse
Affiliation(s)
- Wenqi Zhang
- Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing, 100097, China
- Beijing Key Lab of Digital Plant, National Engineering Research Center for Information Technology in Agriculture, Beijing, 100097, China
- College of Information Technology, Shanghai Ocean University, Shanghai, 201306, China
| | - Sheng Wu
- Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing, 100097, China
- Beijing Key Lab of Digital Plant, National Engineering Research Center for Information Technology in Agriculture, Beijing, 100097, China
| | - Weiliang Wen
- Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing, 100097, China
- Beijing Key Lab of Digital Plant, National Engineering Research Center for Information Technology in Agriculture, Beijing, 100097, China
| | - Xianju Lu
- Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing, 100097, China
- Beijing Key Lab of Digital Plant, National Engineering Research Center for Information Technology in Agriculture, Beijing, 100097, China
| | - Chuanyu Wang
- Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing, 100097, China
- Beijing Key Lab of Digital Plant, National Engineering Research Center for Information Technology in Agriculture, Beijing, 100097, China
| | - Wenbo Gou
- Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing, 100097, China
- Beijing Key Lab of Digital Plant, National Engineering Research Center for Information Technology in Agriculture, Beijing, 100097, China
| | - Yuankun Li
- Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing, 100097, China
- Beijing Key Lab of Digital Plant, National Engineering Research Center for Information Technology in Agriculture, Beijing, 100097, China
| | - Xinyu Guo
- Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing, 100097, China.
- Beijing Key Lab of Digital Plant, National Engineering Research Center for Information Technology in Agriculture, Beijing, 100097, China.
| | - Chunjiang Zhao
- Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing, 100097, China.
- Beijing Key Lab of Digital Plant, National Engineering Research Center for Information Technology in Agriculture, Beijing, 100097, China.
| |
Collapse
|
4
|
Feng Y, Wang Y, Li H, Qu M, Yang J. Learning what and where to segment: A new perspective on medical image few-shot segmentation. Med Image Anal 2023; 87:102834. [PMID: 37207524 DOI: 10.1016/j.media.2023.102834] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Revised: 02/24/2023] [Accepted: 04/20/2023] [Indexed: 05/21/2023]
Abstract
Traditional medical image segmentation methods based on deep learning require experts to provide extensive manual delineations for model training. Few-shot learning aims to reduce the dependence on the scale of training data but usually shows poor generalizability to the new target. The trained model tends to favor the training classes rather than being absolutely class-agnostic. In this work, we propose a novel two-branch segmentation network based on unique medical prior knowledge to alleviate the above problem. Specifically, we explicitly introduce a spatial branch to provide the spatial information of the target. In addition, we build a segmentation branch based on the classical encoder-decoder structure in supervised learning and integrate prototype similarity and spatial information as prior knowledge. To achieve effective information integration, we propose an attention-based fusion module (AF) that enables the content interaction of decoder features and prior knowledge. Experiments on an echocardiography dataset and an abdominal MRI dataset show that the proposed model achieves substantial improvements over state-of-the-art methods. Moreover, some results are comparable to those of the fully supervised model. The source code is available at github.com/warmestwind/RAPNet.
Collapse
Affiliation(s)
- Yong Feng
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, China; School of Computer Science and Engineering, Northeastern University, Shenyang, China
| | - Yonghuai Wang
- Department of Cardiovascular Ultrasound, The First Hospital of China Medical University, Shenyang, China
| | - Honghe Li
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, China; School of Computer Science and Engineering, Northeastern University, Shenyang, China
| | - Mingjun Qu
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, China; School of Computer Science and Engineering, Northeastern University, Shenyang, China
| | - Jinzhu Yang
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, China; School of Computer Science and Engineering, Northeastern University, Shenyang, China.
| |
Collapse
|
5
|
Zhuang M, Chen Z, Wang H, Tang H, He J, Qin B, Yang Y, Jin X, Yu M, Jin B, Li T, Kettunen L. Efficient contour-based annotation by iterative deep learning for organ segmentation from volumetric medical images. Int J Comput Assist Radiol Surg 2023; 18:379-94. [PMID: 36048319 DOI: 10.1007/s11548-022-02730-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2022] [Accepted: 07/29/2022] [Indexed: 02/04/2023]
Abstract
PURPOSE Training deep neural networks usually require a large number of human-annotated data. For organ segmentation from volumetric medical images, human annotation is tedious and inefficient. To save human labour and to accelerate the training process, the strategy of annotation by iterative deep learning recently becomes popular in the research community. However, due to the lack of domain knowledge or efficient human-interaction tools, the current AID methods still suffer from long training time and high annotation burden. METHODS We develop a contour-based annotation by iterative deep learning (AID) algorithm which uses boundary representation instead of voxel labels to incorporate high-level organ shape knowledge. We propose a contour segmentation network with a multi-scale feature extraction backbone to improve the boundary detection accuracy. We also developed a contour-based human-intervention method to facilitate easy adjustments of organ boundaries. By combining the contour-based segmentation network and the contour-adjustment intervention method, our algorithm achieves fast few-shot learning and efficient human proofreading. RESULTS For validation, two human operators independently annotated four abdominal organs in computed tomography (CT) images using our method and two compared methods, i.e. a traditional contour-interpolation method and a state-of-the-art (SOTA) convolutional network (CNN) method based on voxel label representation. Compared to these methods, our approach considerably saved annotation time and reduced inter-rater variabilities. Our contour detection network also outperforms the SOTA nnU-Net in producing anatomically plausible organ shape with only a small training set. CONCLUSION Taking advantage of the boundary shape prior and the contour representation, our method is more efficient, more accurate and less prone to inter-operator variability than the SOTA AID methods for organ segmentation from volumetric medical images. The good shape learning ability and flexible boundary adjustment function make it suitable for fast annotation of organ structures with regular shape.
Collapse
|
6
|
Liu Y, Qin C, Yu Z, Yang R, Suqing T, Liu X, Ma X. Double-branch U-Net for multi-scale organ segmentation. Methods 2022; 205:220-225. [PMID: 35809769 DOI: 10.1016/j.ymeth.2022.07.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2021] [Revised: 06/03/2022] [Accepted: 07/01/2022] [Indexed: 11/30/2022] Open
Abstract
U-Net has achieved great success in the task of medical image segmentation. It encodes and extracts information from several convolution blocks, and then decodes the feature maps to get the segmentation results. Our experiments show that in a multi-scale medical segmentation task, excessive downsampling will cause the model to ignore the small segmentation objects and thus fail to complete the segmentation task. In this work, we propose a more complete method Double-branch U-Net (2BUNet) to solve the multi-scale organ segmentation challenge. Our model is divided into four parts: main branch, tributary branch, information exchange module and classification module. The main advantages of the new model consist of: (1) Extracting information to improve model decoding capabilities using the complete encoding structure. (2) The information exchange module is added to the main branch and tributaries to provide regularization for the model, so as to avoid the large gap between the two paths. (3) Main branch structure for extracting major features of large organ. (4) The tributary structure is used to enlarge the image to extract the microscopic characteristics of small organ. (5) A classification assistant module is proposed to increase the class constraint for the output tensor. The comparative experiments show that our method achieves state-of-the-art performances in real scenes.
Collapse
Affiliation(s)
- Yuhao Liu
- Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Harbin University of Science and Technology, Harbin 150080, China.
| | - Caijie Qin
- Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; Institute of Information Engineering, Sanming University, Sanming 365004, China
| | - Zhiqian Yu
- Information Science and Technology, Northwest University, Xian, China
| | - Ruijie Yang
- Department of Radiation Oncology, Peking University Third Hospital, Beijing 100190, China
| | - Tian Suqing
- Department of Radiation Oncology, Peking University Third Hospital, Beijing 100190, China
| | - Xia Liu
- Harbin University of Science and Technology, Harbin 150080, China.
| | - Xibo Ma
- Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China.
| |
Collapse
|
7
|
Liu X, Guo S, Yang B, Ma S, Zhang H, Li J, Sun C, Jin L, Li X, Yang Q, Fu Y. Automatic Organ Segmentation for CT Scans Based on Super-Pixel and Convolutional Neural Networks. J Digit Imaging 2019; 31:748-760. [PMID: 29679242 DOI: 10.1007/s10278-018-0052-4] [Citation(s) in RCA: 31] [Impact Index Per Article: 6.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023] Open
Abstract
Accurate segmentation of specific organ from computed tomography (CT) scans is a basic and crucial task for accurate diagnosis and treatment. To avoid time-consuming manual optimization and to help physicians distinguish diseases, an automatic organ segmentation framework is presented. The framework utilized convolution neural networks (CNN) to classify pixels. To reduce the redundant inputs, the simple linear iterative clustering (SLIC) of super-pixels and the support vector machine (SVM) classifier are introduced. To establish the perfect boundary of organs in one-pixel-level, the pixels need to be classified step-by-step. First, the SLIC is used to cut an image into grids and extract respective digital signatures. Next, the signature is classified by the SVM, and the rough edges are acquired. Finally, a precise boundary is obtained by the CNN, which is based on patches around each pixel-point. The framework is applied to abdominal CT scans of livers and high-resolution computed tomography (HRCT) scans of lungs. The experimental CT scans are derived from two public datasets (Sliver 07 and a Chinese local dataset). Experimental results show that the proposed method can precisely and efficiently detect the organs. This method consumes 38 s/slice for liver segmentation. The Dice coefficient of the liver segmentation results reaches to 97.43%. For lung segmentation, the Dice coefficient is 97.93%. This finding demonstrates that the proposed framework is a favorable method for lung segmentation of HRCT scans.
Collapse
Affiliation(s)
- Xiaoming Liu
- College of Electronic Science & Engineering, Jilin University, D451 Room of Tangaoqing Building, No. 2699 of Qianjin Street, Changchun, Jilin, China
| | - Shuxu Guo
- College of Electronic Science & Engineering, Jilin University, D451 Room of Tangaoqing Building, No. 2699 of Qianjin Street, Changchun, Jilin, China
| | - Bingtao Yang
- College of Communication Engineering, Jilin University, Changchun, 130012, China
| | - Shuzhi Ma
- LUSTER LightTech Group, Beijing, China
| | - Huimao Zhang
- Department of Radiology, The First Hospital of Jilin University, Changchun, China
| | - Jing Li
- Department of Radiology, The First Hospital of Jilin University, Changchun, China
| | - Changjian Sun
- College of Electronic Science & Engineering, Jilin University, D451 Room of Tangaoqing Building, No. 2699 of Qianjin Street, Changchun, Jilin, China
| | - Lanyi Jin
- College of Electronic Science & Engineering, Jilin University, D451 Room of Tangaoqing Building, No. 2699 of Qianjin Street, Changchun, Jilin, China
| | - Xueyan Li
- College of Electronic Science & Engineering, Jilin University, D451 Room of Tangaoqing Building, No. 2699 of Qianjin Street, Changchun, Jilin, China.
| | - Qi Yang
- Department of Radiology, The First Hospital of Jilin University, Changchun, China
| | - Yu Fu
- Department of Radiology, The First Hospital of Jilin University, Changchun, China
| |
Collapse
|
8
|
Guha Roy A, Siddiqui S, Pölsterl S, Navab N, Wachinger C. 'Squeeze & excite' guided few-shot segmentation of volumetric images. Med Image Anal 2019; 59:101587. [PMID: 31630012 DOI: 10.1016/j.media.2019.101587] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2019] [Revised: 10/02/2019] [Accepted: 10/10/2019] [Indexed: 10/25/2022]
Abstract
Deep neural networks enable highly accurate image segmentation, but require large amounts of manually annotated data for supervised training. Few-shot learning aims to address this shortcoming by learning a new class from a few annotated support examples. We introduce, a novel few-shot framework, for the segmentation of volumetric medical images with only a few annotated slices. Compared to other related works in computer vision, the major challenges are the absence of pre-trained networks and the volumetric nature of medical scans. We address these challenges by proposing a new architecture for few-shot segmentation that incorporates 'squeeze & excite' blocks. Our two-armed architecture consists of a conditioner arm, which processes the annotated support input and generates a task-specific representation. This representation is passed on to the segmenter arm that uses this information to segment the new query image. To facilitate efficient interaction between the conditioner and the segmenter arm, we propose to use 'channel squeeze & spatial excitation' blocks - a light-weight computational module - that enables heavy interaction between both the arms with negligible increase in model complexity. This contribution allows us to perform image segmentation without relying on a pre-trained model, which generally is unavailable for medical scans. Furthermore, we propose an efficient strategy for volumetric segmentation by optimally pairing a few slices of the support volume to all the slices of the query volume. We perform experiments for organ segmentation on whole-body contrast-enhanced CT scans from the Visceral Dataset. Our proposed model outperforms multiple baselines and existing approaches with respect to the segmentation accuracy by a significant margin. The source code is available at https://github.com/abhi4ssj/few-shot-segmentation.
Collapse
Affiliation(s)
- Abhijit Guha Roy
- Artificial Intelligence in Medical Imaging (AI-Med), Department of Child and Adolescent Psychiatry, LMU München, Germany; Computer Aided Medical Procedures, Department of Informatics, Technical University of Munich, Germany.
| | - Shayan Siddiqui
- Artificial Intelligence in Medical Imaging (AI-Med), Department of Child and Adolescent Psychiatry, LMU München, Germany; Computer Aided Medical Procedures, Department of Informatics, Technical University of Munich, Germany
| | - Sebastian Pölsterl
- Artificial Intelligence in Medical Imaging (AI-Med), Department of Child and Adolescent Psychiatry, LMU München, Germany
| | - Nassir Navab
- Computer Aided Medical Procedures, Department of Informatics, Technical University of Munich, Germany; Computer Aided Medical Procedures, Johns Hopkins University, Baltimore, USA
| | - Christian Wachinger
- Artificial Intelligence in Medical Imaging (AI-Med), Department of Child and Adolescent Psychiatry, LMU München, Germany
| |
Collapse
|
9
|
Abstract
Deep learning-based image segmentation is by now firmly established as a robust tool in image segmentation. It has been widely used to separate homogeneous areas as the first and critical component of diagnosis and treatment pipeline. In this article, we present a critical appraisal of popular methods that have employed deep-learning techniques for medical image segmentation. Moreover, we summarize the most common challenges incurred and suggest possible solutions.
Collapse
Affiliation(s)
- Mohammad Hesam Hesamian
- School of Electrical and Data Engineering (SEDE), University of Technology Sydney, 2007, Sydney, Australia.
- CB11.09, University of Technology Sydney, 81 Broadway, Ultimo NSW, 2007, Sydney, Australia.
| | - Wenjing Jia
- School of Electrical and Data Engineering (SEDE), University of Technology Sydney, 2007, Sydney, Australia
| | - Xiangjian He
- School of Electrical and Data Engineering (SEDE), University of Technology Sydney, 2007, Sydney, Australia
| | - Paul Kennedy
- School of Software, University of Technology Sydney, 2007, Sydney, Australia
| |
Collapse
|
10
|
Cai J, Zhang Z, Cui L, Zheng Y, Yang L. Towards cross-modal organ translation and segmentation: A cycle- and shape-consistent generative adversarial network. Med Image Anal 2018; 52:174-184. [PMID: 30594770 DOI: 10.1016/j.media.2018.12.002] [Citation(s) in RCA: 45] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2018] [Revised: 12/12/2018] [Accepted: 12/14/2018] [Indexed: 11/25/2022]
Abstract
Synthesized medical images have several important applications. For instance, they can be used as an intermedium in cross-modality image registration or used as augmented training samples to boost the generalization capability of a classifier. In this work, we propose a generic cross-modality synthesis approach with the following targets: 1) synthesizing realistic looking 2D/3D images without needing paired training data, 2) ensuring consistent anatomical structures, which could be changed by geometric distortion in cross-modality synthesis and 3) more importantly, improving volume segmentation by using synthetic data for modalities with limited training samples. We show that these goals can be achieved with an end-to-end 2D/3D convolutional neural network (CNN) composed of mutually-beneficial generators and segmentors for image synthesis and segmentation tasks. The generators are trained with an adversarial loss, a cycle-consistency loss, and also a shape-consistency loss (supervised by segmentors) to reduce the geometric distortion. From the segmentation view, the segmentors are boosted by synthetic data from generators in an online manner. Generators and segmentors prompt each other alternatively in an end-to-end training fashion. We validate our proposed method on three datasets, including cardiovascular CT and magnetic resonance imaging (MRI), abdominal CT and MRI, and mammography X-rays from different data domains, showing both tasks are beneficial to each other and coupling these two tasks results in better performance than solving them exclusively.
Collapse
Affiliation(s)
- Jinzheng Cai
- J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida, Gainesville, FL, 32611, USA.
| | - Zizhao Zhang
- Department of Computer Information and Science Engineering, University of Florida, Gainesville, FL, 32611, USA
| | - Lei Cui
- Department of Information Science and Technology, Northwest University, Xi'an 710127, China
| | - Yefeng Zheng
- Medical Imaging Technologies, Siemens Healthcare, Princeton, NJ, 08540, USA
| | - Lin Yang
- J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida, Gainesville, FL, 32611, USA.
| |
Collapse
|