1
|
Rao F, Lyu T, Feng Z, Wu Y, Ni Y, Zhu W. A landmark-supervised registration framework for multi-phase CT images with cross-distillation. Phys Med Biol 2024; 69:115059. [PMID: 38768601 DOI: 10.1088/1361-6560/ad4e01] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Accepted: 05/20/2024] [Indexed: 05/22/2024]
Abstract
Objective.Multi-phase computed tomography (CT) has become a leading modality for identifying hepatic tumors. Nevertheless, the presence of misalignment in the images of different phases poses a challenge in accurately identifying and analyzing the patient's anatomy. Conventional registration methods typically concentrate on either intensity-based features or landmark-based features in isolation, so imposing limitations on the accuracy of the registration process.Method.We establish a nonrigid cycle-registration network that leverages semi-supervised learning techniques, wherein a point distance term based on Euclidean distance between registered landmark points is introduced into the loss function. Additionally, a cross-distillation strategy is proposed in network training to further improve registration performance which incorporates response-based knowledge concerning the distances between feature points.Results.We conducted experiments using multi-centered liver CT datasets to evaluate the performance of the proposed method. The results demonstrate that our method outperforms baseline methods in terms of target registration error. Additionally, Dice scores of the warped tumor masks were calculated. Our method consistently achieved the highest scores among all the comparing methods. Specifically, it achieved scores of 82.9% and 82.5% in the hepatocellular carcinoma and the intrahepatic cholangiocarcinoma dataset, respectively.Significance.The superior registration performance indicates its potential to serve as an important tool in hepatic tumor identification and analysis.
Collapse
Affiliation(s)
- Fan Rao
- Research Center for Augmented Intelligence, Zhejiang Lab, Hangzhou 310000, People's Republic of China
| | - Tianling Lyu
- Research Center for Augmented Intelligence, Zhejiang Lab, Hangzhou 310000, People's Republic of China
| | - Zhan Feng
- Department of Radiology, College of Medicine, The First Affiliated Hospital, Zhejiang University, Hangzhou 311100, People's Republic of China
| | - Yuanfeng Wu
- Research Center for Augmented Intelligence, Zhejiang Lab, Hangzhou 310000, People's Republic of China
| | - Yangfan Ni
- Research Center for Augmented Intelligence, Zhejiang Lab, Hangzhou 310000, People's Republic of China
| | - Wentao Zhu
- Research Center for Augmented Intelligence, Zhejiang Lab, Hangzhou 310000, People's Republic of China
| |
Collapse
|
2
|
Osman AFI, Al-Mugren KS, Tamam NM, Shahine B. Deformable registration of magnetic resonance images using unsupervised deep learning in neuro-/radiation oncology. Radiat Oncol 2024; 19:61. [PMID: 38773620 PMCID: PMC11110381 DOI: 10.1186/s13014-024-02452-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2023] [Accepted: 05/13/2024] [Indexed: 05/24/2024] Open
Abstract
PURPOSE Accurate deformable registration of magnetic resonance imaging (MRI) scans containing pathologies is challenging due to changes in tissue appearance. In this paper, we developed a novel automated three-dimensional (3D) convolutional U-Net based deformable image registration (ConvUNet-DIR) method using unsupervised learning to establish correspondence between baseline pre-operative and follow-up MRI scans of patients with brain glioma. METHODS This study involved multi-parametric brain MRI scans (T1, T1-contrast enhanced, T2, FLAIR) acquired at pre-operative and follow-up time for 160 patients diagnosed with glioma, representing the BraTS-Reg 2022 challenge dataset. ConvUNet-DIR, a deep learning-based deformable registration workflow using 3D U-Net style architecture as a core, was developed to establish correspondence between the MRI scans. The workflow consists of three components: (1) the U-Net learns features from pairs of MRI scans and estimates a mapping between them, (2) the grid generator computes the sampling grid based on the derived transformation parameters, and (3) the spatial transformation layer generates a warped image by applying the sampling operation using interpolation. A similarity measure was used as a loss function for the network with a regularization parameter limiting the deformation. The model was trained via unsupervised learning using pairs of MRI scans on a training data set (n = 102) and validated on a validation data set (n = 26) to assess its generalizability. Its performance was evaluated on a test set (n = 32) by computing the Dice score and structural similarity index (SSIM) quantitative metrics. The model's performance also was compared with the baseline state-of-the-art VoxelMorph (VM1 and VM2) learning-based algorithms. RESULTS The ConvUNet-DIR model showed promising competency in performing accurate 3D deformable registration. It achieved a mean Dice score of 0.975 ± 0.003 and SSIM of 0.908 ± 0.011 on the test set (n = 32). Experimental results also demonstrated that ConvUNet-DIR outperformed the VoxelMorph algorithms concerning Dice (VM1: 0.969 ± 0.006 and VM2: 0.957 ± 0.008) and SSIM (VM1: 0.893 ± 0.012 and VM2: 0.857 ± 0.017) metrics. The time required to perform a registration for a pair of MRI scans is about 1 s on the CPU. CONCLUSIONS The developed deep learning-based model can perform an end-to-end deformable registration of a pair of 3D MRI scans for glioma patients without human intervention. The model could provide accurate, efficient, and robust deformable registration without needing pre-alignment and labeling. It outperformed the state-of-the-art VoxelMorph learning-based deformable registration algorithms and other supervised/unsupervised deep learning-based methods reported in the literature.
Collapse
Affiliation(s)
- Alexander F I Osman
- Department of Medical Physics, Al-Neelain University, Khartoum, 11121, Sudan.
| | - Kholoud S Al-Mugren
- Department of Physics, College of Science, Princess Nourah bint Abdulrahman University, P.O. Box 84428, Riyadh, 11671, Saudi Arabia
| | - Nissren M Tamam
- Department of Physics, College of Science, Princess Nourah bint Abdulrahman University, P.O. Box 84428, Riyadh, 11671, Saudi Arabia
| | - Bilal Shahine
- Department of Radiation Oncology, American University of Beirut Medical Center, Beirut, Lebanon
| |
Collapse
|
3
|
Azad R, Kazerouni A, Heidari M, Aghdam EK, Molaei A, Jia Y, Jose A, Roy R, Merhof D. Advances in medical image analysis with vision Transformers: A comprehensive review. Med Image Anal 2024; 91:103000. [PMID: 37883822 DOI: 10.1016/j.media.2023.103000] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2023] [Revised: 09/30/2023] [Accepted: 10/11/2023] [Indexed: 10/28/2023]
Abstract
The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in https://github.com/mindflow-institue/Awesome-Transformer.
Collapse
Affiliation(s)
- Reza Azad
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Amirhossein Kazerouni
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Moein Heidari
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | | | - Amirali Molaei
- School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Yiwei Jia
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Abin Jose
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Rijo Roy
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Dorit Merhof
- Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany; Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
| |
Collapse
|
4
|
Xu L, Jiang P, Tsui T, Liu J, Zhang X, Yu L, Niu T. 4D-CT deformable image registration using unsupervised recursive cascaded full-resolution residual networks. Bioeng Transl Med 2023; 8:e10587. [PMID: 38023695 PMCID: PMC10658570 DOI: 10.1002/btm2.10587] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2023] [Revised: 06/30/2023] [Accepted: 07/30/2023] [Indexed: 12/01/2023] Open
Abstract
A novel recursive cascaded full-resolution residual network (RCFRR-Net) for abdominal four-dimensional computed tomography (4D-CT) image registration was proposed. The entire network was end-to-end and trained in the unsupervised approach, which meant that the deformation vector field, which presented the ground truth, was not needed during training. The network was designed by cascading three full-resolution residual subnetworks with different architectures. The training loss consisted of the image similarity loss and the deformation vector field regularization loss, which were calculated based on the final warped image and the fixed image, allowing all cascades to be trained jointly and perform the progressive registration cooperatively. Extensive network testing was conducted using diverse datasets, including an internal 4D-CT dataset, a public DIRLAB 4D-CT dataset, and a 4D cone-beam CT (4D-CBCT) dataset. Compared with the iteration-based demon method and two deep learning-based methods (VoxelMorph and recursive cascaded network), the RCFRR-Net achieved consistent and significant gains, which demonstrated that the proposed method had superior performance and generalization capability in medical image registration. The proposed RCFRR-Net was a promising tool for various clinical applications.
Collapse
Affiliation(s)
- Lei Xu
- Department of Radiation Oncologythe First Affiliated Hospital of Xi'an Jiaotong UniversityXi'anShaanxiChina
- Institute of Biomedical EngineeringShenzhen Bay LaboratoryShenzhenGuangdongChina
| | - Ping Jiang
- Department of Radiation OncologyPeking University 3rd HospitalBeijingChina
| | - Tiffany Tsui
- Loyola University Medical CenterMaywoodIllinoisUSA
| | - Junyan Liu
- Department of Radiation OncologyStanford University School of MedicineStanfordCaliforniaUSA
| | - Xiping Zhang
- Department of Radiation OncologyOzarks HealthcareWest PlainsMissouriUSA
| | - Lequan Yu
- Department of Statistics and Actuarial ScienceThe University of Hong Kong, HKSARHong KongChina
| | - Tianye Niu
- Institute of Biomedical EngineeringShenzhen Bay LaboratoryShenzhenGuangdongChina
- Peking University Aerospace School of Clinical Medicine, Aerospace Center HospitalBeijingChina
| |
Collapse
|
5
|
Deng L, Zhang Y, Wang J, Huang S, Yang X. Improving performance of medical image alignment through super-resolution. Biomed Eng Lett 2023; 13:397-406. [PMID: 37519883 PMCID: PMC10382383 DOI: 10.1007/s13534-023-00268-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2022] [Revised: 01/29/2023] [Accepted: 02/01/2023] [Indexed: 02/21/2023] Open
Abstract
Medical image alignment is an important tool for tracking patient conditions, but the quality of alignment is influenced by the effectiveness of low-dose Cone-beam CT (CBCT) imaging and patient characteristics. To address these two issues, we propose an unsupervised alignment method that incorporates a preprocessing super-resolution process. We constructed the model based on a private clinical dataset and validated the enhancement of the super-resolution on alignment using clinical and public data. Through all three experiments, we demonstrate that higher resolution data yields better results in the alignment process. To fully constrain similarity and structure, a new loss function is proposed; Pearson correlation coefficient combined with regional mutual information. In all test samples, the newly proposed loss function obtains higher results than the common loss function and improve alignment accuracy. Subsequent experiments verified that, combined with the newly proposed loss function, the super-resolution processed data boosts alignment, can reaching up to 9.58%. Moreover, this boost is not limited to a single model, but is effective in different alignment models. These experiments demonstrate that the unsupervised alignment method with super-resolution preprocessing proposed in this study effectively improved alignment and plays an important role in tracking different patient conditions over time.
Collapse
Affiliation(s)
- Liwei Deng
- Heilongjiang Provincial Key Laboratory of Complex Intelligent System and Integration, School of Automation, Harbin University of Science and Technology, Harbin, 150080 Heilongjiang China
| | - Yuanzhi Zhang
- Heilongjiang Provincial Key Laboratory of Complex Intelligent System and Integration, School of Automation, Harbin University of Science and Technology, Harbin, 150080 Heilongjiang China
| | - Jing Wang
- Faculty of Rehabilitation Medicine, Biofeedback Laboratory, Guangzhou Xinhua University, Guangzhou, 510520 Guangdong China
| | - Sijuan Huang
- Department of Radiation Oncology State Key Laboratory of Oncology in South China Collaborative Innovation Center for Cancer Medicine Guangdong Key Laboratory of Nasopharyngeal Carcinoma Diagnosis and Therapy, Sun Yat-sen University Cancer Center, Guangzhou, 510060 Guangdong China
| | - Xin Yang
- Department of Radiation Oncology State Key Laboratory of Oncology in South China Collaborative Innovation Center for Cancer Medicine Guangdong Key Laboratory of Nasopharyngeal Carcinoma Diagnosis and Therapy, Sun Yat-sen University Cancer Center, Guangzhou, 510060 Guangdong China
| |
Collapse
|
6
|
Liang X, Chun J, Morgan H, Bai T, Nguyen D, Park JC, Jiang S. Segmentation by test-time optimization for CBCT-based adaptive radiation therapy. Med Phys 2023; 50:1947-1961. [PMID: 36310403 PMCID: PMC10121749 DOI: 10.1002/mp.15960] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2022] [Revised: 08/02/2022] [Accepted: 08/21/2022] [Indexed: 11/13/2022] Open
Abstract
PURPOSE Online adaptive radiotherapy (ART) requires accurate and efficient auto-segmentation of target volumes and organs-at-risk (OARs) in mostly cone-beam computed tomography (CBCT) images, which often have severe artifacts and lack soft-tissue contrast, making direct segmentation very challenging. Propagating expert-drawn contours from the pretreatment planning CT through traditional or deep learning (DL)-based deformable image registration (DIR) can achieve improved results in many situations. Typical DL-based DIR models are population based, that is, trained with a dataset for a population of patients, and so they may be affected by the generalizability problem. METHODS In this paper, we propose a method called test-time optimization (TTO) to refine a pretrained DL-based DIR population model, first for each individual test patient, and then progressively for each fraction of online ART treatment. Our proposed method is less susceptible to the generalizability problem and thus can improve overall performance of different DL-based DIR models by improving model accuracy, especially for outliers. Our experiments used data from 239 patients with head-and-neck squamous cell carcinoma to test the proposed method. First, we trained a population model with 200 patients and then applied TTO to the remaining 39 test patients by refining the trained population model to obtain 39 individualized models. We compared each of the individualized models with the population model in terms of segmentation accuracy. RESULTS The average improvement of the Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD95) of segmentation can be up to 0.04 (5%) and 0.98 mm (25%), respectively, with the individualized models compared to the population model over 17 selected OARs and a target of 39 patients. Although the average improvement may seem mild, we found that the improvement for outlier patients with structures of large anatomical changes is significant. The number of patients with at least 0.05 DSC improvement or 2 mm HD95 improvement by TTO averaged over the 17 selected structures for the state-of-the-art architecture VoxelMorph is 10 out of 39 test patients. By deriving the individualized model using TTO from the pretrained population model, TTO models can be ready in about 1 min. We also generated the adapted fractional models for each of the 39 test patients by progressively refining the individualized models using TTO to CBCT images acquired at later fractions of online ART treatment. When adapting the individualized model to a later fraction of the same patient, the model can be ready in less than a minute with slightly improved accuracy. CONCLUSIONS The proposed TTO method is well suited for online ART and can boost segmentation accuracy for DL-based DIR models, especially for outlier patients where the pretrained models fail.
Collapse
Affiliation(s)
- Xiao Liang
- Medical Artificial Intelligence and Automation Laboratory and Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Jaehee Chun
- Department of Radiation Oncology, Yonsei Cancer Center, Yonsei University College of Medicine, Seoul, South Korea
| | - Howard Morgan
- Medical Artificial Intelligence and Automation Laboratory and Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Ti Bai
- Medical Artificial Intelligence and Automation Laboratory and Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Dan Nguyen
- Medical Artificial Intelligence and Automation Laboratory and Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Justin C. Park
- Medical Artificial Intelligence and Automation Laboratory and Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Steve Jiang
- Medical Artificial Intelligence and Automation Laboratory and Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| |
Collapse
|
7
|
Deng L, Zhang Y, Qi J, Huang S, Yang X, Wang J. Enhancement of cone beam CT image registration by super-resolution pre-processing algorithm. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2023; 20:4403-4420. [PMID: 36896505 DOI: 10.3934/mbe.2023204] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
In order to enhance cone-beam computed tomography (CBCT) image information and improve the registration accuracy for image-guided radiation therapy, we propose a super-resolution (SR) image enhancement method. This method uses super-resolution techniques to pre-process the CBCT prior to registration. Three rigid registration methods (rigid transformation, affine transformation, and similarity transformation) and a deep learning deformed registration (DLDR) method with and without SR were compared. The five evaluation indices, the mean squared error (MSE), mutual information, Pearson correlation coefficient (PCC), structural similarity index (SSIM), and PCC + SSIM, were used to validate the results of registration with SR. Moreover, the proposed method SR-DLDR was also compared with the VoxelMorph (VM) method. In rigid registration with SR, the registration accuracy improved by up to 6% in the PCC metric. In DLDR with SR, the registration accuracy was improved by up to 5% in PCC + SSIM. When taking the MSE as the loss function, the accuracy of SR-DLDR is equivalent to that of the VM method. In addition, when taking the SSIM as the loss function, the registration accuracy of SR-DLDR is 6% higher than that of VM. SR is a feasible method to be used in medical image registration for planning CT (pCT) and CBCT. The experimental results show that the SR algorithm can improve the accuracy and efficiency of CBCT image alignment regardless of which alignment algorithm is used.
Collapse
Affiliation(s)
- Liwei Deng
- Heilongjiang Provincial Key Laboratory of Complex Intelligent System and Integration, School of Automation, Harbin University of Science and Technology, Harbin 150080, China
| | - Yuanzhi Zhang
- Heilongjiang Provincial Key Laboratory of Complex Intelligent System and Integration, School of Automation, Harbin University of Science and Technology, Harbin 150080, China
| | - Jingjing Qi
- Heilongjiang Provincial Key Laboratory of Complex Intelligent System and Integration, School of Automation, Harbin University of Science and Technology, Harbin 150080, China
| | - Sijuan Huang
- Department of Radiation Oncology; Sun Yat-sen University Cancer Center; State Key Laboratory of Oncology in South China; Collaborative Innovation Center for Cancer Medicine; Guangdong Key Laboratory of Nasopharyngeal Carcinoma Diagnosis and Therapy, Guangzhou 510060, China
| | - Xin Yang
- Department of Radiation Oncology; Sun Yat-sen University Cancer Center; State Key Laboratory of Oncology in South China; Collaborative Innovation Center for Cancer Medicine; Guangdong Key Laboratory of Nasopharyngeal Carcinoma Diagnosis and Therapy, Guangzhou 510060, China
| | - Jing Wang
- Faculty of Rehabilitation Medicine, Biofeedback Laboratory, Guangzhou Xinhua University, Guangzhou 510520, China
| |
Collapse
|
8
|
Chen J, Frey EC, He Y, Segars WP, Li Y, Du Y. TransMorph: Transformer for unsupervised medical image registration. Med Image Anal 2022; 82:102615. [PMID: 36156420 PMCID: PMC9999483 DOI: 10.1016/j.media.2022.102615] [Citation(s) in RCA: 65] [Impact Index Per Article: 32.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2021] [Revised: 06/29/2022] [Accepted: 09/02/2022] [Indexed: 01/04/2023]
Abstract
In the last decade, convolutional neural networks (ConvNets) have been a major focus of research in medical image analysis. However, the performances of ConvNets may be limited by a lack of explicit consideration of the long-range spatial relationships in an image. Recently, Vision Transformer architectures have been proposed to address the shortcomings of ConvNets and have produced state-of-the-art performances in many medical imaging applications. Transformers may be a strong candidate for image registration because their substantially larger receptive field enables a more precise comprehension of the spatial correspondence between moving and fixed images. Here, we present TransMorph, a hybrid Transformer-ConvNet model for volumetric medical image registration. This paper also presents diffeomorphic and Bayesian variants of TransMorph: the diffeomorphic variants ensure the topology-preserving deformations, and the Bayesian variant produces a well-calibrated registration uncertainty estimate. We extensively validated the proposed models using 3D medical images from three applications: inter-patient and atlas-to-patient brain MRI registration and phantom-to-CT registration. The proposed models are evaluated in comparison to a variety of existing registration methods and Transformer architectures. Qualitative and quantitative results demonstrate that the proposed Transformer-based model leads to a substantial performance improvement over the baseline methods, confirming the effectiveness of Transformers for medical image registration.
Collapse
Affiliation(s)
- Junyu Chen
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD, USA; Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA.
| | - Eric C Frey
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD, USA; Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA.
| | - Yufan He
- NVIDIA Corporation, Bethesda, MD, USA.
| | - William P Segars
- Carl E. Ravin Advanced Imaging Laboratories, Department of Radiology, Duke University Medical Center, Durham, NC, USA.
| | - Ye Li
- Center for Advanced Medical Computing and Analysis, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA.
| | - Yong Du
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD, USA.
| |
Collapse
|
9
|
Wang D, Pan Y, Durumeric OC, Reinhardt JM, Hoffman EA, Schroeder JD, Christensen GE. PLOSL: Population learning followed by one shot learning pulmonary image registration using tissue volume preserving and vesselness constraints. Med Image Anal 2022; 79:102434. [PMID: 35430476 PMCID: PMC11225793 DOI: 10.1016/j.media.2022.102434] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2021] [Revised: 01/30/2022] [Accepted: 03/21/2022] [Indexed: 01/12/2023]
Abstract
This paper presents the Population Learning followed by One Shot Learning (PLOSL) pulmonary image registration method. PLOSL is a fast unsupervised learning-based framework for 3D-CT pulmonary image registration algorithm based on combining population learning (PL) and one-shot learning (OSL). The PLOSL image registration has the advantages of the PL and OSL approaches while reducing their respective drawbacks. The advantages of PLOSL include improved performance over PL, substantially reducing OSL training time and reducing the likelihood of OSL getting stuck in local minima. PLOSL pulmonary image registration uses tissue volume preserving and vesselness constraints for registration of inspiration-to-expiration and expiration-to-inspiration pulmonary CT images. A coarse-to-fine convolution encoder-decoder CNN architecture is used to register large and small shape features. During training, the sum of squared tissue volume difference (SSTVD) compensates for intensity differences between inspiration and expiration computed tomography (CT) images and the sum of squared vesselness measure difference (SSVMD) helps match the lung vessel tree. Results show that the PLOSL (SSTVD+SSVMD) algorithm achieved subvoxel landmark error while preserving pulmonary topology on the SPIROMICS data set, the public DIR-LAB COPDGene and 4DCT data sets.
Collapse
Affiliation(s)
- Di Wang
- Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA 52242, USA
| | - Yue Pan
- Elekta Inc., St. Charles City, MO 63303, USA
| | - Oguz C Durumeric
- Department of Mathematics, University of Iowa, Iowa City, IA 52242, USA
| | - Joseph M Reinhardt
- Roy J. Carver Department of Biomedical Engineering, University of Iowa, Iowa City, IA 52242, USA; Department of Radiology, University of Iowa, Iowa City, IA 52242, USA
| | - Eric A Hoffman
- Roy J. Carver Department of Biomedical Engineering, University of Iowa, Iowa City, IA 52242, USA; Department of Radiology, University of Iowa, Iowa City, IA 52242, USA
| | - Joyce D Schroeder
- Department of Radiology and Imaging Sciences, University of Utah, Salt Lake City, UT 84132, USA
| | - Gary E Christensen
- Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA 52242, USA; Department of Radiology Oncology, University of Iowa, Iowa City, IA 52242, USA.
| |
Collapse
|
10
|
GPLFR—Global perspective and local flow registration-for forward-looking sonar images. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07113-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
11
|
Akhavanallaf A, Fayad H, Salimi Y, Aly A, Kharita H, Al Naemi H, Zaidi H. An update on computational anthropomorphic anatomical models. Digit Health 2022; 8:20552076221111941. [PMID: 35847523 PMCID: PMC9277432 DOI: 10.1177/20552076221111941] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2022] [Accepted: 06/19/2022] [Indexed: 11/15/2022] Open
Abstract
The prevalent availability of high-performance computing coupled with validated
computerized simulation platforms as open-source packages have motivated
progress in the development of realistic anthropomorphic computational models of
the human anatomy. The main application of these advanced tools focused on
imaging physics and computational internal/external radiation dosimetry
research. This paper provides an updated review of state-of-the-art developments
and recent advances in the design of sophisticated computational models of the
human anatomy with a particular focus on their use in radiation dosimetry
calculations. The consolidation of flexible and realistic computational models
with biological data and accurate radiation transport modeling tools enables the
capability to produce dosimetric data reflecting actual setup in clinical
setting. These simulation methodologies and results are helpful resources for
the medical physics and medical imaging communities and are expected to impact
the fields of medical imaging and dosimetry calculations profoundly.
Collapse
Affiliation(s)
- Azadeh Akhavanallaf
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
| | - Hadi Fayad
- Hamad Medical Corporation, Doha, Qatar
- Weill Cornell Medicine, Doha, Qatar
| | - Yazdan Salimi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
| | - Antar Aly
- Hamad Medical Corporation, Doha, Qatar
- Weill Cornell Medicine, Doha, Qatar
| | | | - Huda Al Naemi
- Hamad Medical Corporation, Doha, Qatar
- Weill Cornell Medicine, Doha, Qatar
| | - Habib Zaidi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
- Geneva University Neurocenter, Geneva University, Geneva, Switzerland
- Department of Nuclear Medicine and Molecular Imaging, University Medical Center Groningen, University of Groningen, Groningen, Netherlands
- Department of Nuclear Medicine, University of Southern Denmark, Odense, Denmark
| |
Collapse
|
12
|
Zhu Q, Lin G, Sun Y, Wu Y, Zhou Y, Feng Q. Functional magnetic resonance imaging progressive deformable registration based on a cascaded convolutional neural network. Quant Imaging Med Surg 2021; 11:3569-3583. [PMID: 34341732 DOI: 10.21037/qims-20-1289] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2020] [Accepted: 03/18/2021] [Indexed: 11/06/2022]
Abstract
Background Intersubject registration of functional magnetic resonance imaging (fMRI) is necessary for group analysis. Accurate image registration can significantly improve the results of statistical analysis. Traditional methods are achieved by using high-resolution structural images or manually extracting functional information. However, structural alignment does not necessarily lead to functional alignment, and manually extracting functional features is complicated and time-consuming. Recent studies have shown that deep learning-based methods can be used for deformable image registration. Methods We proposed a deep learning framework with a three-cascaded multi-resolution network (MR-Net) to achieve deformable image registration. MR-Net separately extracts the features of moving and fixed images via a two-stream path, predicts a sub-deformation field, and is cascaded three times. The moving and fixed images' deformation field is composed of all sub-deformation fields predicted by the MR-Net. We imposed large smoothness constraints on all sub-deformation fields to ensure their smoothness. Our proposed architecture can complete the progressive registration process to ensure the topology of the deformation field. Results We implemented our method on the 1000 Functional Connectomes Project (FCP) and Eyes Open Eyes Closed fMRI datasets. Our method increased the peak t values in six brain functional networks to 19.8, 17.8, 15.0, 16.4, 17.0, and 13.2. Compared with traditional methods [i.e., FMRIB Software Library (FSL) and Statistical Parametric Mapping (SPM)] and deep learning networks [i.e., VoxelMorph (VM) and Volume Tweening Network (VTN)], our method improved 47.58%, 11.88%, 18.60%, and 15.16%, respectively. Conclusions Our three-cascaded MR-Net can achieve statistically significant improvement in functional consistency across subjects.
Collapse
Affiliation(s)
- Qiaoyun Zhu
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, China.,Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, China
| | - Guoye Lin
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, China.,Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, China
| | - Yuhang Sun
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, China.,Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, China
| | - Yi Wu
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China.,Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, China
| | - Yujia Zhou
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, China.,Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, China
| | - Qianjin Feng
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China.,Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, China.,Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, China
| |
Collapse
|
13
|
Chen J, Li Y, Luna LP, Chung HW, Rowe SP, Du Y, Solnes LB, Frey EC. Learning fuzzy clustering for SPECT/CT segmentation via convolutional neural networks. Med Phys 2021; 48:3860-3877. [PMID: 33905560 PMCID: PMC9973404 DOI: 10.1002/mp.14903] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2021] [Revised: 04/01/2021] [Accepted: 04/12/2021] [Indexed: 01/07/2023] Open
Abstract
PURPOSE Quantitative bone single-photon emission computed tomography (QBSPECT) has the potential to provide a better quantitative assessment of bone metastasis than planar bone scintigraphy due to its ability to better quantify activity in overlapping structures. An important element of assessing the response of bone metastasis is accurate image segmentation. However, limited by the properties of QBSPECT images, the segmentation of anatomical regions-of-interests (ROIs) still relies heavily on the manual delineation by experts. This work proposes a fast and robust automated segmentation method for partitioning a QBSPECT image into lesion, bone, and background. METHODS We present a new unsupervised segmentation loss function and its semi- and supervised variants for training a convolutional neural network (ConvNet). The loss functions were developed based on the objective function of the classical Fuzzy C-means (FCM) algorithm. The first proposed loss function can be computed within the input image itself without any ground truth labels, and is thus unsupervised; the proposed supervised loss function follows the traditional paradigm of the deep learning-based segmentation methods and leverages ground truth labels during training. The last loss function is a combination of the first and the second and includes a weighting parameter, which enables semi-supervised segmentation using deep learning neural network. EXPERIMENTS AND RESULTS We conducted a comprehensive study to compare our proposed methods with ConvNets trained using supervised, cross-entropy and Dice loss functions, and conventional clustering methods. The Dice similarity coefficient (DSC) and several other metrics were used as figures of merit as applied to the task of delineating lesion and bone in both simulated and clinical SPECT/CT images. We experimentally demonstrated that the proposed methods yielded good segmentation results on a clinical dataset even though the training was done using realistic simulated images. On simulated SPECT/CT, the proposed unsupervised model's accuracy was greater than the conventional clustering methods while reducing computation time by 200-fold. For the clinical QBSPECT/CT, the proposed semi-supervised ConvNet model, trained using simulated images, produced DSCs of 0.75 and 0.74 for lesion and bone segmentation in SPECT, and a DSC of 0.79 bone segmentation of CT images. These DSCs were larger than that for standard segmentation loss functions by > 0.4 for SPECT segmentation, and > 0.07 for CT segmentation with P-values < 0.001 from a paired t-test. CONCLUSIONS A ConvNet-based image segmentation method that uses novel loss functions was developed and evaluated. The method can operate in unsupervised, semi-supervised, or fully-supervised modes depending on the availability of annotated training data. The results demonstrated that the proposed method provides fast and robust lesion and bone segmentation for QBSPECT/CT. The method can potentially be applied to other medical image segmentation applications.
Collapse
Affiliation(s)
- Junyu Chen
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD,Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD,Corresponding author
| | - Ye Li
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD,Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD
| | - Licia P. Luna
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD
| | - Hyun Woo Chung
- Department of Nuclear Medicine, Konkuk University Medical Center, Konkuk University School of Medicine, Seoul, South Korea
| | - Steven P. Rowe
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD
| | - Yong Du
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD
| | - Lilja B. Solnes
- Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD
| | - Eric C. Frey
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD,Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institutes, Baltimore, MD
| |
Collapse
|
14
|
Yang J, Yang J, Zhao F, Zhang W. An unsupervised multi-scale framework with attention-based network (MANet) for lung 4D-CT registration. Phys Med Biol 2021; 66. [PMID: 34126608 DOI: 10.1088/1361-6560/ac0afc] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2021] [Accepted: 06/14/2021] [Indexed: 01/25/2023]
Abstract
Deformable image registration (DIR) of 4D-CT is very important in many radiotherapeutic applications including tumor target definition, image fusion, dose accumulation and response evaluation. It is a challenging task to performing accurate and fast DIR of lung 4D-CT images due to its large and complicated deformations. In this study, we propose an unsupervised multi-scale DIR framework with attention-based network (MANet). Three cascaded models used for aligning CT images in different resolution levels were involved and trained by minimizing the loss functions, which were defined as the combination of dissimilarity between the fixed image and the deformed image and DVF regularization term. In addition, attention gates were incorporated into the three models to distinguish the moving structures from non-moving or minimal-moving structures during registration. The three models were trained sequentially and separately to minimize the loss function in each scale to initialize the MANet, and then trained jointly to minimize the total loss function which incorporated an additional dissimilarity between fixed image and deformed image. Besides, an adversarial network was integrated into MANet to enforce the DVF regularization by penalizing the unrealistic deformed images. The proposed MANet was evaluated on the public dir-lab dataset, and the target registration errors (TREs) of the model were compared with convention iterative optimization-based methods and three recently published deep learning-based methods. The initial results showed that the MANet with an average of TRE of 1.53 ± 1.02 mm outperformed other registration methods, and its execution time took about 1 s for DVF estimation with no requirement of manual-tuning for parameters, which demonstrating that our proposed method had the ability of performing superior registration for 4D-CT.
Collapse
Affiliation(s)
- Juan Yang
- School of Physics and Electronics, Shandong Normal University, Jinan 250358, People's Republic of China
| | - Jinhui Yang
- School of Physics and Electronics, Shandong Normal University, Jinan 250358, People's Republic of China
| | - Fen Zhao
- Department of Radiation Oncology, Shandong Cancer Hospital and Institute, Jinan 250117, People's Republic of China
| | - Wenjun Zhang
- Department of Human Resources, Shandong Provincial Third Hospital, Jinan 250031, People's Republic of China
| |
Collapse
|