1
|
Long L, Xue X, Xiao H. CCMNet: Cross-scale correlation-aware mapping network for 3D lung CT image registration. Comput Biol Med 2024; 182:109103. [PMID: 39244962 DOI: 10.1016/j.compbiomed.2024.109103] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2024] [Revised: 08/04/2024] [Accepted: 09/01/2024] [Indexed: 09/10/2024]
Abstract
The lung is characterized by high elasticity and complex structure, which implies that the lung is capable of undergoing complex deformation and the shape variable is substantial. Large deformation estimation poses significant challenges to lung image registration. The traditional U-Net architecture is difficult to cover complex deformation due to its limited receptive field. Moreover, the relationship between voxels weakens as the number of downsampling times increases, that is, the long-range dependence issue. In this paper, we propose a novel multilevel registration framework which enhances the correspondence between voxels to improve the ability of estimating large deformations. Our approach consists of a convolutional neural network (CNN) with a two-stream registration structure and a cross-scale mapping attention (CSMA) mechanism. The former extracts the robust features of image pairs within layers, while the latter establishes frequent connections between layers to maintain the correlation of image pairs. This method fully utilizes the context information of different scales to establish the mapping relationship between low-resolution and high-resolution feature maps. We have achieved remarkable results on DIRLAB (TRE 1.56 ± 1.60) and POPI (NCC 99.72% SSIM 91.42%) dataset, demonstrating that this strategy can effectively address the large deformation issues, mitigate long-range dependence, and ultimately achieve more robust lung CT image registration.
Collapse
Affiliation(s)
- Li Long
- School of Artificial Intelligence, Chongqing University of Technology, Chongqing 401135, China
| | - Xufeng Xue
- School of Artificial Intelligence, Chongqing University of Technology, Chongqing 401135, China
| | - Hanguang Xiao
- School of Artificial Intelligence, Chongqing University of Technology, Chongqing 401135, China.
| |
Collapse
|
2
|
Pham XL, Luu MH, van Walsum T, Mai HS, Klein S, Le NH, Chu DT. CMAN: Cascaded Multi-scale Spatial Channel Attention-guided Network for large 3D deformable registration of liver CT images. Med Image Anal 2024; 96:103212. [PMID: 38830326 DOI: 10.1016/j.media.2024.103212] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2023] [Revised: 03/27/2024] [Accepted: 05/17/2024] [Indexed: 06/05/2024]
Abstract
Deformable image registration is an essential component of medical image analysis and plays an irreplaceable role in clinical practice. In recent years, deep learning-based registration methods have demonstrated significant improvements in convenience, robustness and execution time compared to traditional algorithms. However, registering images with large displacements, such as those of the liver organ, remains underexplored and challenging. In this study, we present a novel convolutional neural network (CNN)-based unsupervised learning registration method, Cascaded Multi-scale Spatial-Channel Attention-guided Network (CMAN), which addresses the challenge of large deformation fields using a double coarse-to-fine registration approach. The main contributions of CMAN include: (i) local coarse-to-fine registration in the base network, which generates the displacement field for each resolution and progressively propagates these local deformations as auxiliary information for the final deformation field; (ii) global coarse-to-fine registration, which stacks multiple base networks for sequential warping, thereby incorporating richer multi-layer contextual details into the final deformation field; (iii) integration of the spatial-channel attention module in the decoder stage, which better highlights important features and improves the quality of feature maps. The proposed network was trained using two public datasets and evaluated on another public dataset as well as a private dataset across several experimental scenarios. We compared CMAN with four state-of-the-art CNN-based registration methods and two well-known traditional algorithms. The results show that the proposed double coarse-to-fine registration strategy outperforms other methods in most registration evaluation metrics. In conclusion, CMAN can effectively handle the large-deformation registration problem and show potential for application in clinical practice. The source code is made publicly available at https://github.com/LocPham263/CMAN.git.
Collapse
Affiliation(s)
- Xuan Loc Pham
- FET, VNU University of Engineering and Technology, Hanoi, Viet Nam; Diagnostic Image Analysis Group, Radboud UMC, Nijmegen, The Netherlands
| | - Manh Ha Luu
- FET, VNU University of Engineering and Technology, Hanoi, Viet Nam; AVITECH, VNU University of Engineering and Technology, Hanoi, Viet Nam; Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, The Netherlands.
| | - Theo van Walsum
- Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, The Netherlands
| | - Hong Son Mai
- Department of Nuclear Medicine, Hospital 108, Hanoi, Viet Nam
| | - Stefan Klein
- Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, The Netherlands
| | - Ngoc Ha Le
- Department of Nuclear Medicine, Hospital 108, Hanoi, Viet Nam
| | - Duc Trinh Chu
- FET, VNU University of Engineering and Technology, Hanoi, Viet Nam
| |
Collapse
|
3
|
Li Y, Huang XT, Feng YB, Fan QR, Wang DW, Lv FJ, He XQ, Li Q. Value of CT-Based Deep Learning Model in Differentiating Benign and Malignant Solid Pulmonary Nodules ≤ 8 mm. Acad Radiol 2024:S1076-6332(24)00305-2. [PMID: 38806374 DOI: 10.1016/j.acra.2024.05.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2024] [Revised: 04/27/2024] [Accepted: 05/12/2024] [Indexed: 05/30/2024]
Abstract
RATIONALE AND OBJECTIVES We examined the effectiveness of computed tomography (CT)-based deep learning (DL) models in differentiating benign and malignant solid pulmonary nodules (SPNs) ≤ 8 mm. MATERIALS AND METHODS The study patients (n = 719) were divided into internal training, internal validation, and external validation cohorts; all had small SPNs and had undergone preoperative chest CTs and surgical resection. We developed five DL models incorporating features of the nodule and five different peri-nodular regions with the Multiscale Dual Attention Network (MDANet) to differentiate benign and malignant SPNs. We selected the best-performing model, which was then compared to four conventional algorithms (VGG19, ResNet50, ResNeXt50, and DenseNet121). Furthermore, another five DL models were constructed using MDANet to distinguish benign tumors from inflammatory nodules and the one performed best was selected out. RESULTS Model 4, which incorporated the nodule and 15 mm peri-nodular region, best differentiated benign and malignant SPNs. The model had an area under the curve (AUC), accuracy, recall, precision, and F1-score of 0.730, 0.724, 0.711, 0.705, and 0.707 in the external validation cohort. Model 4 also performed better than the other four conventional algorithms. Model 8, which incorporated the nodule and 10 mm peri-nodular region, was the best model for distinguishing benign tumors from inflammatory nodules. The model had an AUC, accuracy, recall, precision, and F1-score of 0.871, 0.938, 0.863, 0.904, and 0.882 in the external validation cohort. CONCLUSION The study concludes that CT-based DL models built with MDANet can accurately discriminate among small benign and malignant SPNs, benign tumors and inflammatory nodules.
Collapse
Affiliation(s)
- Yuan Li
- Department of Thoracic Surgery, the First Affiliated Hospital of Chongqing Medical University, No.1 Youyi Road, Yuzhong District, Chongqing, China (Y.L.); Department of Thoracic Surgery, National Cancer Center/ National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China (Y.L.)
| | - Xing-Tao Huang
- Department of Radiology, the Fifth People's Hospital of Chongqing, No. 24 Renji Road, Nan'an District, Chongqing, China (X.T.H.)
| | - Yi-Bo Feng
- Institute of Research, Infervision Medical Technology Co., Ltd, 25F Building E, Yuanyang International Center, Chaoyang District. Beijing, China (B.Y.F., R.Q.F., W.D.W.)
| | - Qian-Rui Fan
- Institute of Research, Infervision Medical Technology Co., Ltd, 25F Building E, Yuanyang International Center, Chaoyang District. Beijing, China (B.Y.F., R.Q.F., W.D.W.)
| | - Da-Wei Wang
- Institute of Research, Infervision Medical Technology Co., Ltd, 25F Building E, Yuanyang International Center, Chaoyang District. Beijing, China (B.Y.F., R.Q.F., W.D.W.)
| | - Fa-Jin Lv
- Department of Radiology, the First Affiliated Hospital of Chongqing Medical University, No.1 Youyi Road, Yuzhong District, Chongqing, China (F.J.L., X.Q.H., Q.L.)
| | - Xiao-Qun He
- Department of Radiology, the First Affiliated Hospital of Chongqing Medical University, No.1 Youyi Road, Yuzhong District, Chongqing, China (F.J.L., X.Q.H., Q.L.)
| | - Qi Li
- Department of Radiology, the First Affiliated Hospital of Chongqing Medical University, No.1 Youyi Road, Yuzhong District, Chongqing, China (F.J.L., X.Q.H., Q.L.).
| |
Collapse
|
4
|
Xiao H, Xue X, Zhu M, Jiang X, Xia Q, Chen K, Li H, Long L, Peng K. Deep learning-based lung image registration: A review. Comput Biol Med 2023; 165:107434. [PMID: 37696177 DOI: 10.1016/j.compbiomed.2023.107434] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Revised: 08/13/2023] [Accepted: 08/28/2023] [Indexed: 09/13/2023]
Abstract
Lung image registration can effectively describe the relative motion of lung tissues, thereby helping to solve series problems in clinical applications. Since the lungs are soft and fairly passive organs, they are influenced by respiration and heartbeat, resulting in discontinuity of lung motion and large deformation of anatomic features. This poses great challenges for accurate registration of lung image and its applications. The recent application of deep learning (DL) methods in the field of medical image registration has brought promising results. However, a versatile registration framework has not yet emerged due to diverse challenges of registration for different regions of interest (ROI). DL-based image registration methods used for other ROI cannot achieve satisfactory results in lungs. In addition, there are few review articles available on DL-based lung image registration. In this review, the development of conventional methods for lung image registration is briefly described and a more comprehensive survey of DL-based methods for lung image registration is illustrated. The DL-based methods are classified according to different supervision types, including fully-supervised, weakly-supervised and unsupervised. The contributions of researchers in addressing various challenges are described, as well as the limitations of these approaches. This review also presents a comprehensive statistical analysis of the cited papers in terms of evaluation metrics and loss functions. In addition, publicly available datasets for lung image registration are also summarized. Finally, the remaining challenges and potential trends in DL-based lung image registration are discussed.
Collapse
Affiliation(s)
- Hanguang Xiao
- College of Artificial Intelligent, Chongqing University of Technology, Chongqing 401135, China
| | - Xufeng Xue
- College of Artificial Intelligent, Chongqing University of Technology, Chongqing 401135, China
| | - Mi Zhu
- College of Artificial Intelligent, Chongqing University of Technology, Chongqing 401135, China.
| | - Xin Jiang
- College of Artificial Intelligent, Chongqing University of Technology, Chongqing 401135, China
| | - Qingling Xia
- College of Artificial Intelligent, Chongqing University of Technology, Chongqing 401135, China
| | - Kai Chen
- College of Artificial Intelligent, Chongqing University of Technology, Chongqing 401135, China
| | - Huanqi Li
- College of Artificial Intelligent, Chongqing University of Technology, Chongqing 401135, China
| | - Li Long
- College of Artificial Intelligent, Chongqing University of Technology, Chongqing 401135, China
| | - Ke Peng
- College of Artificial Intelligent, Chongqing University of Technology, Chongqing 401135, China.
| |
Collapse
|
5
|
Tian D, Sun G, Zheng H, Yu S, Jiang J. CT-CBCT deformable registration using weakly-supervised artifact-suppression transfer learning network. Phys Med Biol 2023; 68:165011. [PMID: 37433303 DOI: 10.1088/1361-6560/ace675] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2023] [Accepted: 07/11/2023] [Indexed: 07/13/2023]
Abstract
Objective.Computed tomography-cone-beam computed tomography (CT-CBCT) deformable registration has great potential in adaptive radiotherapy. It plays an important role in tumor tracking, secondary planning, accurate irradiation, and the protection of at-risk organs. Neural networks have been improving CT-CBCT deformable registration, and almost all registration algorithms based on neural networks rely on the gray values of both CT and CBCT. The gray value is a key factor in the loss function, parameter training, and final efficacy of the registration. Unfortunately, the scattering artifacts in CBCT affect the gray values of different pixels inconsistently. Therefore, the direct registration of the original CT-CBCT introduces artifact superposition loss.Approach. In this study, a histogram analysis method for the gray values was used. Based on an analysis of the gray value distribution characteristics of different regions in CT and CBCT, the degree of superposition of the artifact in the region of disinterest was found to be much higher than that in the region of interest. Moreover, the former was the main reason for artifact superposition loss. Consequently, a new weakly supervised two-stage transfer-learning network based on artifact suppression was proposed. The first stage was a pre-training network designed to suppress artifacts contained in the region of disinterest. The second stage was a convolutional neural network that registered the suppressed CBCT and CT.Main Results. Through a comparative test of the thoracic CT-CBCT deformable registration, whose data were collected from the Elekta XVI system, the rationality and accuracy after artifact suppression were confirmed to be significantly improved compared with the other algorithms without artifact suppression.Significance. This study proposed and verified a new deformable registration method with multi-stage neural networks, which can effectively suppress artifacts and further improve registration by incorporating a pre-training technique and an attention mechanism.
Collapse
Affiliation(s)
- Dingshu Tian
- University of Science and Technology of China, Hefei 230026, People's Republic of China
- Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, People's Republic of China
| | - Guangyao Sun
- SuperSafety Science and Technology Co., Ltd, Hefei 230088, People's Republic of China
- International Academy of Neutron Science, Qingdao 266199, People's Republic of China
| | - Huaqing Zheng
- International Academy of Neutron Science, Qingdao 266199, People's Republic of China
- Super Accuracy Science and Technology Co., Ltd, Nanjing 210044, People's Republic of China
| | - Shengpeng Yu
- SuperSafety Science and Technology Co., Ltd, Hefei 230088, People's Republic of China
- International Academy of Neutron Science, Qingdao 266199, People's Republic of China
| | - Jieqiong Jiang
- Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, People's Republic of China
| |
Collapse
|
6
|
Gao X, Su T, Zhang Y, Zhu J, Tan Y, Cui H, Long X, Zheng H, Liang D, Ge Y. Attention-based dual-branch deep network for sparse-view computed tomography image reconstruction. Quant Imaging Med Surg 2023; 13:1360-1374. [PMID: 36915341 PMCID: PMC10006128 DOI: 10.21037/qims-22-609] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2022] [Accepted: 12/01/2022] [Indexed: 02/25/2023]
Abstract
Background The widespread application of X-ray computed tomography (CT) imaging in medical screening makes radiation safety a major concern for public health. Sparse-view CT is a promising solution to reduce the radiation dose. However, the reconstructed CT images obtained using sparse-view CT may suffer severe streaking artifacts and structural information loss. Methods In this study, a novel attention-based dual-branch network (ADB-Net) is proposed to solve the ill-posed problem of sparse-view CT image reconstruction. In this network, downsampled sinogram input is processed through 2 parallel branches (CT branch and signogram branch) of the ADB-Net to independently extract the distinct, high-level feature maps. These feature maps are fused in a specified attention module from 3 perspectives (channel, plane, and spatial) to allow complementary optimizations that can mitigate the streaking artifacts and the structure loss in sparse-view CT imaging. Results Numerical simulations, an anthropomorphic thorax phantom, and in vivo preclinical experiments were conducted to verify the sparse-view CT imaging performance of the ADB-Net. The proposed network achieved a root-mean-square error (RMSE) of 20.6160, a structural similarity (SSIM) of 0.9257, and a peak signal-to-noise ratio (PSNR) of 38.8246 on numerical data. The visualization results demonstrate that this newly developed network can consistently remove the streaking artifacts while maintaining the fine structures. Conclusions The proposed attention-based dual-branch deep network, ADB-Net, provides a promising alternative to reconstruct high-quality sparse-view CT images for low-dose CT imaging.
Collapse
Affiliation(s)
- Xiang Gao
- Research Center for Medical Artificial Intelligence, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.,University of Chinese Academy of Sciences, Beijing, China
| | - Ting Su
- Research Center for Medical Artificial Intelligence, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Yunxin Zhang
- Department of Vascular Surgery, Beijing Jishuitan Hospital, Beijing, China
| | - Jiongtao Zhu
- Research Center for Medical Artificial Intelligence, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.,College of Physics and Optoelectronic Engineering, Shenzhen University, Shenzhen, China
| | - Yuhang Tan
- Research Center for Medical Artificial Intelligence, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Han Cui
- Research Center for Medical Artificial Intelligence, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Xiaojing Long
- Research Center for Medical Artificial Intelligence, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Hairong Zheng
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Dong Liang
- Research Center for Medical Artificial Intelligence, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.,Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Yongshuai Ge
- Research Center for Medical Artificial Intelligence, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.,Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| |
Collapse
|
7
|
Liu H, Gong G, Zou W, Hu N, Wang J. Topologically preserved registration of 3D CT images with deep networks. Phys Med Biol 2023; 68. [PMID: 36623316 DOI: 10.1088/1361-6560/acb197] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Accepted: 01/09/2023] [Indexed: 01/11/2023]
Abstract
Objective. Computed Tomography (CT) image registration makes fast and accurate imaging-based disease diagnosis possible. We aim to develop a framework which can perform accurate local registration of organs in 3D CT images while preserving the topology of transformation.Approach. In this framework, the Faster R-CNN method is first used to detect local areas containing organs from fixed and moving images whose results are then registered with a weakly supervised deep neural network. In this network, a novel 3D channel coordinate attention (CA) module is introduced to reduce the loss of position information. The image edge loss and the organ labelling loss are used to weakly supervise the training process of our deep network, which enables the network learning to focus on registering organs and image structures. An intuitive inverse module is also used to reduce the folding of deformation field. More specifically, the folding is suppressed directly by simultaneously maximizing forward and backward registration accuracy in the image domain rather than indirectly by measuring the consistency of forward and inverse deformation fields as usual.Main results. Our method achieves an average dice similarity coefficient (DSC) of 0.954 and an average Similarity (Sim) of 0.914 on publicly available liver datasets (LiTS for training and Sliver07 for testing) and achieves an average DSC of 0.914 and an average Sim of 0.947 on our home-built left ventricular myocardium (LVM) dataset.Significance. Experimental results show that our proposed method can significantly improve the registration accuracy of organs such as the liver and LVM. Moreover, our inverse module can intuitively improve the inherent topological preservation of transformations.
Collapse
Affiliation(s)
- Huaying Liu
- School of Electronic and Information Engineering, Soochow University, Suzhou 215006, People's Republic of China
| | - Guanzhong Gong
- Department of Radiation Oncology Physics and Technology, Shandong Cancer Hospital and Institute, Shandong First Medical University and Shandong Academy of Medical Sciences, Jinan 250000, People's Republic of China
| | - Wei Zou
- School of Electronic and Information Engineering, Soochow University, Suzhou 215006, People's Republic of China
| | - Nan Hu
- School of Electronic and Information Engineering, Soochow University, Suzhou 215006, People's Republic of China
| | - Jiajun Wang
- School of Electronic and Information Engineering, Soochow University, Suzhou 215006, People's Republic of China
| |
Collapse
|