1
|
Wyburd MK, Dinsdale NK, Jenkinson M, Namburete AIL. Anatomically plausible segmentations: Explicitly preserving topology through prior deformations. Med Image Anal 2024; 97:103222. [PMID: 38936222 DOI: 10.1016/j.media.2024.103222] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2022] [Revised: 05/23/2024] [Accepted: 05/25/2024] [Indexed: 06/29/2024]
Abstract
Since the rise of deep learning, new medical segmentation methods have rapidly been proposed with extremely promising results, often reporting marginal improvements on the previous state-of-the-art (SOTA) method. However, on visual inspection errors are often revealed, such as topological mistakes (e.g. holes or folds), that are not detected using traditional evaluation metrics. Incorrect topology can often lead to errors in clinically required downstream image processing tasks. Therefore, there is a need for new methods to focus on ensuring segmentations are topologically correct. In this work, we present TEDS-Net: a segmentation network that preserves anatomical topology whilst maintaining segmentation performance that is competitive with SOTA baselines. Further, we show how current SOTA segmentation methods can introduce problematic topological errors. TEDS-Net achieves anatomically plausible segmentation by using learnt topology-preserving fields to deform a prior. Traditionally, topology-preserving fields are described in the continuous domain and begin to break down when working in the discrete domain. Here, we introduce additional modifications that more strictly enforce topology preservation. We illustrate our method on an open-source medical heart dataset, performing both single and multi-structure segmentation, and show that the generated fields contain no folding voxels, which corresponds to full topology preservation on individual structures whilst vastly outperforming the other baselines on overall scene topology. The code is available at: https://github.com/mwyburd/TEDS-Net.
Collapse
Affiliation(s)
- Madeleine K Wyburd
- Oxford Machine Learning Neuroimaging Lab (OMNI) Computer Science Department, University of Oxford, Oxford, OX1 3QG, United Kingdom.
| | - Nicola K Dinsdale
- Oxford Machine Learning Neuroimaging Lab (OMNI) Computer Science Department, University of Oxford, Oxford, OX1 3QG, United Kingdom
| | - Mark Jenkinson
- Wellcome Centre for Integrative Neuroimaging, Wellcome Centre for Integrative Neuroimaging, Oxford, United Kingdom; Australian Institute for Machine Learning (AIML), Department of Computer Science, University of Adelaide, Adelaide, Australia; South Australian Health and Medical Research Institute (SAHMRI), North Terrace, Australia
| | - Ana I L Namburete
- Oxford Machine Learning Neuroimaging Lab (OMNI) Computer Science Department, University of Oxford, Oxford, OX1 3QG, United Kingdom; Wellcome Centre for Integrative Neuroimaging, Wellcome Centre for Integrative Neuroimaging, Oxford, United Kingdom
| |
Collapse
|
2
|
Ji W, Yang F. Affine medical image registration with fusion feature mapping in local and global. Phys Med Biol 2024; 69:055029. [PMID: 38324893 DOI: 10.1088/1361-6560/ad2717] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2023] [Accepted: 02/07/2024] [Indexed: 02/09/2024]
Abstract
Objective. Medical image affine registration is a crucial basis before using deformable registration. On the one hand, the traditional affine registration methods based on step-by-step optimization are very time-consuming, so these methods are not compatible with most real-time medical applications. On the other hand, convolutional neural networks are limited in modeling long-range spatial relationships of the features due to inductive biases, such as weight sharing and locality. This is not conducive to affine registration tasks. Therefore, the evolution of real-time and high-accuracy affine medical image registration algorithms is necessary for registration applications.Approach. In this paper, we propose a deep learning-based coarse-to-fine global and local feature fusion architecture for fast affine registration, and we use an unsupervised approach for end-to-end training. We use multiscale convolutional kernels as our elemental convolutional blocks to enhance feature extraction. Then, to learn the long-range spatial relationships of the features, we propose a new affine registration framework with weighted global positional attention that fuses global feature mapping and local feature mapping. Moreover, the fusion regressor is designed to generate the affine parameters.Main results. The additive fusion method can be adaptive to global mapping and local mapping, which improves affine registration accuracy without the center of mass initialization. In addition, the max pooling layer and the multiscale convolutional kernel coding module increase the ability of the model in affine registration.Significance. We validate the effectiveness of our method on the OASIS dataset with 414 3D MRI brain maps. Comprehensive results demonstrate that our method achieves state-of-the-art affine registration accuracy and very efficient runtimes.
Collapse
Affiliation(s)
- Wei Ji
- School of Computer and Electronic Information, Guangxi University, Nanning, Guangxi, 530004, People's Republic of China
| | - Feng Yang
- School of Computer and Electronic Information, Guangxi University, Nanning, Guangxi, 530004, People's Republic of China
- Guangxi Key Laboratory of Multimedia Communications Network Technology, Guangxi University, Nanning, Guangxi, 530004, People's Republic of China
- Key Laboratory of Parallel, Distributed and Intelligent Computing of Guangxi Universities and Colleges, Nanning, Guangxi, 530004, People's Republic of China
| |
Collapse
|
3
|
Osman YBM, Li C, Huang W, Wang S. Collaborative Learning for Annotation-Efficient Volumetric MR Image Segmentation. J Magn Reson Imaging 2023. [PMID: 38156427 DOI: 10.1002/jmri.29194] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2023] [Revised: 12/05/2023] [Accepted: 12/05/2023] [Indexed: 12/30/2023] Open
Abstract
BACKGROUND Deep learning has presented great potential in accurate MR image segmentation when enough labeled data are provided for network optimization. However, manually annotating three-dimensional (3D) MR images is tedious and time-consuming, requiring experts with rich domain knowledge and experience. PURPOSE To build a deep learning method exploring sparse annotations, namely only a single two-dimensional slice label for each 3D training MR image. STUDY TYPE Retrospective. POPULATION Three-dimensional MR images of 150 subjects from two publicly available datasets were included. Among them, 50 (1377 image slices) are for prostate segmentation. The other 100 (8800 image slices) are for left atrium segmentation. Five-fold cross-validation experiments were carried out utilizing the first dataset. For the second dataset, 80 subjects were used for training and 20 were used for testing. FIELD STRENGTH/SEQUENCE 1.5 T and 3.0 T; axial T2-weighted and late gadolinium-enhanced, 3D respiratory navigated, inversion recovery prepared gradient echo pulse sequence. ASSESSMENT A collaborative learning method by integrating the strengths of semi-supervised and self-supervised learning schemes was developed. The method was trained using labeled central slices and unlabeled noncentral slices. Segmentation performance on testing set was reported quantitatively and qualitatively. STATISTICAL TESTS Quantitative evaluation metrics including boundary intersection-over-union (B-IoU), Dice similarity coefficient, average symmetric surface distance, and relative absolute volume difference were calculated. Paired t test was performed, and P < 0.05 was considered statistically significant. RESULTS Compared to fully supervised training with only the labeled central slice, mean teacher, uncertainty-aware mean teacher, deep co-training, interpolation consistency training (ICT), and ambiguity-consensus mean teacher, the proposed method achieved a substantial improvement in segmentation accuracy, increasing the mean B-IoU significantly by more than 10.0% for prostate segmentation (proposed method B-IoU: 70.3% ± 7.6% vs. ICT B-IoU: 60.3% ± 11.2%) and by more than 6.0% for left atrium segmentation (proposed method B-IoU: 66.1% ± 6.8% vs. ICT B-IoU: 60.1% ± 7.1%). DATA CONCLUSIONS A collaborative learning method trained using sparse annotations can segment prostate and left atrium with high accuracy. LEVEL OF EVIDENCE 0 TECHNICAL EFFICACY: Stage 1.
Collapse
Affiliation(s)
- Yousuf Babiker M Osman
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Cheng Li
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Weijian Huang
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
- Peng Cheng Laboratory, Shenzhen, China
| | - Shanshan Wang
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- Peng Cheng Laboratory, Shenzhen, China
| |
Collapse
|
4
|
Osman YBM, Li C, Huang W, Wang S. Sparse annotation learning for dense volumetric MR image segmentation with uncertainty estimation. Phys Med Biol 2023; 69:015009. [PMID: 38035374 DOI: 10.1088/1361-6560/ad111b] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Accepted: 11/30/2023] [Indexed: 12/02/2023]
Abstract
Objective.Training neural networks for pixel-wise or voxel-wise image segmentation is a challenging task that requires a considerable amount of training samples with highly accurate and densely delineated ground truth maps. This challenge becomes especially prominent in the medical imaging domain, where obtaining reliable annotations for training samples is a difficult, time-consuming, and expert-dependent process. Therefore, developing models that can perform well under the conditions of limited annotated training data is desirable.Approach.In this study, we propose an innovative framework called the extremely sparse annotation neural network (ESA-Net) that learns with only the single central slice label for 3D volumetric segmentation which explores both intra-slice pixel dependencies and inter-slice image correlations with uncertainty estimation. Specifically, ESA-Net consists of four specially designed distinct components: (1) an intra-slice pixel dependency-guided pseudo-label generation module that exploits uncertainty in network predictions while generating pseudo-labels for unlabeled slices with temporal ensembling; (2) an inter-slice image correlation-constrained pseudo-label propagation module which propagates labels from the labeled central slice to unlabeled slices by self-supervised registration with rotation ensembling; (3) a pseudo-label fusion module that fuses the two sets of generated pseudo-labels with voxel-wise uncertainty guidance; and (4) a final segmentation network optimization module to make final predictions with scoring-based label quantification.Main results.Extensive experimental validations have been performed on two popular yet challenging magnetic resonance image segmentation tasks and compared to five state-of-the-art methods.Significance.Results demonstrate that our proposed ESA-Net can consistently achieve better segmentation performances even under the extremely sparse annotation setting, highlighting its effectiveness in exploiting information from unlabeled data.
Collapse
Affiliation(s)
- Yousuf Babiker M Osman
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
- University of Chinese Academy of Sciences, Beijing 100049, People's Republic of China
| | - Cheng Li
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
- Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou 510080, People's Republic of China
| | - Weijian Huang
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
- University of Chinese Academy of Sciences, Beijing 100049, People's Republic of China
- Peng Cheng Laboratory, Shenzhen 518066, People's Republic of China
| | - Shanshan Wang
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
- Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou 510080, People's Republic of China
- Peng Cheng Laboratory, Shenzhen 518066, People's Republic of China
| |
Collapse
|
5
|
Zeng Q, Yang C, Gan Q, Wang Q, Wang S. EDIRNet: an unsupervised deformable registration model for X-ray and neutron images. APPLIED OPTICS 2023; 62:7611-7620. [PMID: 37855468 DOI: 10.1364/ao.500442] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/12/2023] [Accepted: 09/13/2023] [Indexed: 10/20/2023]
Abstract
For high-precision industrial non-destructive testing, multimodal image registration technology can be employed to register X-ray and neutron images. X-ray and neutron image registration algorithms usually use conventional methods through iterative optimization. These methods will increase the cost of registration time and require more initialization parameters. The imaging results of internal sample structures can suffer from edge blurring due to the influence of a neutron beam collimator aperture, X-ray focal point, and imaging angles. We present an unsupervised learning model, EDIRNet, based on deep learning for deformable registration of X-ray and neutron images. We define the registration process as a function capable of estimating the flow field from input images. By leveraging deep learning techniques, we effectively parameterize this function. Consequently, given a registration image, our optimized network parameters enable rapid and direct estimation of the flow field between the images. We design an attention-based edge enhancement module to enhance the edge features of the image. For evaluating our presented network model, we utilize a dataset including 552 pairs of X-ray and neutron images. The experimental results show that the registration accuracy of EDIRNet reaches 93.09%. Compared with traditional algorithms, the accuracy of EDIRNet is improved by 3.17%, and the registration time is reduced by 28.75 s.
Collapse
|
6
|
Ma X, Cui H, Li S, Yang Y, Xia Y. Deformable medical image registration with global-local transformation network and region similarity constraint. Comput Med Imaging Graph 2023; 108:102263. [PMID: 37487363 DOI: 10.1016/j.compmedimag.2023.102263] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2022] [Revised: 05/04/2023] [Accepted: 06/07/2023] [Indexed: 07/26/2023]
Abstract
Deformable medical image registration can achieve fast and accurate alignment between two images, enabling medical professionals to analyze images of different subjects in a unified anatomical space. As such, it plays an important role in many medical image studies. Current deep learning (DL)-based approaches for image registration directly learn spatial transformation from one image to another, relying on a convolutional neural network and ground truth or similarity metrics. However, these methods only use a global similarity energy function to evaluate the similarity of a pair of images, which ignores the similarity of regions of interest (ROIs) within the images. This can limit the accuracy of the image registration and affect the analysis of specific ROIs. Additionally, DL-based methods often estimate global spatial transformations of images directly, without considering local spatial transformations of ROIs within the images. To address this issue, we propose a novel global-local transformation network with a region similarity constraint that maximizes the similarity of ROIs within the images and estimates both global and local spatial transformations simultaneously. Experiments conducted on four public 3D MRI datasets demonstrate that the proposed method achieves the highest registration performance in terms of accuracy and generalization compared to other state-of-the-art methods.
Collapse
Affiliation(s)
- Xinke Ma
- National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an 710072, China
| | - Hengfei Cui
- National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an 710072, China
| | - Shuoyan Li
- National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an 710072, China
| | - Yibo Yang
- King Abdullah University of Science and Technology (KAUST), Thuwal 23955, Saudi Arabia
| | - Yong Xia
- National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an 710072, China.
| |
Collapse
|
7
|
Wu M, He X, Li F, Zhu J, Wang S, Burstein PD. Weakly supervised volumetric prostate registration for MRI-TRUS image driven by signed distance map. Comput Biol Med 2023; 163:107150. [PMID: 37321103 DOI: 10.1016/j.compbiomed.2023.107150] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Revised: 05/24/2023] [Accepted: 06/07/2023] [Indexed: 06/17/2023]
Abstract
Image registration is a fundamental step for MRI-TRUS fusion targeted biopsy. Due to the inherent representational differences between these two image modalities, though, intensity-based similarity losses for registration tend to result in poor performance. To mitigate this, comparison of organ segmentations, functioning as a weak proxy measure of image similarity, has been proposed. Segmentations, though, are limited in their information encoding capabilities. Signed distance maps (SDMs), on the other hand, encode these segmentations into a higher dimensional space where shape and boundary information are implicitly captured, and which, in addition, yield high gradients even for slight mismatches, thus preventing vanishing gradients during deep-network training. Based on these advantages, this study proposes a weakly-supervised deep learning volumetric registration approach driven by a mixed loss that operates both on segmentations and their corresponding SDMs, and which is not only robust to outliers, but also encourages optimal global alignment. Our experimental results, performed on a public prostate MRI-TRUS biopsy dataset, demonstrate that our method outperforms other weakly-supervised registration approaches with a dice similarity coefficient (DSC), Hausdorff distance (HD) and mean surface distance (MSD) of 87.3 ± 11.3, 4.56 ± 1.95 mm, and 0.053 ± 0.026 mm, respectively. We also show that the proposed method effectively preserves the prostate gland's internal structure.
Collapse
Affiliation(s)
- Menglin Wu
- School of Computer Science and Technology, Nanjing Tech University, Nanjing, China.
| | - Xuchen He
- School of Computer Science and Technology, Nanjing Tech University, Nanjing, China
| | - Fan Li
- School of Computer Science and Technology, Nanjing Tech University, Nanjing, China
| | - Jie Zhu
- Senior Department of Urology, The Third Medical Center of Chinese People's Liberation Army General Hospital, Beijing, China.
| | | | | |
Collapse
|
8
|
Gui P, He F, Ling BWK, Zhang D, Ge Z. Normal vibration distribution search-based differential evolution algorithm for multimodal biomedical image registration. Neural Comput Appl 2023; 35:1-23. [PMID: 37362574 PMCID: PMC10227826 DOI: 10.1007/s00521-023-08649-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2022] [Accepted: 05/02/2023] [Indexed: 06/28/2023]
Abstract
In linear registration, a floating image is spatially aligned with a reference image after performing a series of linear metric transformations. Additionally, linear registration is mainly considered a preprocessing version of nonrigid registration. To better accomplish the task of finding the optimal transformation in pairwise intensity-based medical image registration, in this work, we present an optimization algorithm called the normal vibration distribution search-based differential evolution algorithm (NVSA), which is modified from the Bernstein search-based differential evolution (BSD) algorithm. We redesign the search pattern of the BSD algorithm and import several control parameters as part of the fine-tuning process to reduce the difficulty of the algorithm. In this study, 23 classic optimization functions and 16 real-world patients (resulting in 41 multimodal registration scenarios) are used in experiments performed to statistically investigate the problem solving ability of the NVSA. Nine metaheuristic algorithms are used in the conducted experiments. When compared to the commonly utilized registration methods, such as ANTS, Elastix, and FSL, our method achieves better registration performance on the RIRE dataset. Moreover, we prove that our method can perform well with or without its initial spatial transformation in terms of different evaluation indicators, demonstrating its versatility and robustness for various clinical needs and applications. This study establishes the idea that metaheuristic-based methods can better accomplish linear registration tasks than the frequently used approaches; the proposed method demonstrates promise that it can solve real-world clinical and service problems encountered during nonrigid registration as a preprocessing approach.The source code of the NVSA is publicly available at https://github.com/PengGui-N/NVSA.
Collapse
Affiliation(s)
- Peng Gui
- School of Computer Science, Wuhan University, Wuhan, 430072 People’s Republic of China
- AIM Lab, Faculty of IT, Monash University, Melbourne, VIC 3800 Australia
- Monash-Airdoc Research, Monash University, Melbourne, VIC 3800 Australia
| | - Fazhi He
- School of Computer Science, Wuhan University, Wuhan, 430072 People’s Republic of China
| | - Bingo Wing-Kuen Ling
- School of Information Engineering, Guangdong University of Technology, Guangzhou, 510006 People’s Republic of China
| | - Dengyi Zhang
- School of Computer Science, Wuhan University, Wuhan, 430072 People’s Republic of China
| | - Zongyuan Ge
- AIM Lab, Faculty of IT, Monash University, Melbourne, VIC 3800 Australia
- Monash-Airdoc Research, Monash University, Melbourne, VIC 3800 Australia
| |
Collapse
|
9
|
Iglesias JE. A ready-to-use machine learning tool for symmetric multi-modality registration of brain MRI. Sci Rep 2023; 13:6657. [PMID: 37095168 PMCID: PMC10126156 DOI: 10.1038/s41598-023-33781-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2023] [Accepted: 04/19/2023] [Indexed: 04/26/2023] Open
Abstract
Volumetric registration of brain MRI is routinely used in human neuroimaging, e.g., to align different MRI modalities, to measure change in longitudinal analysis, to map an individual to a template, or in registration-based segmentation. Classical registration techniques based on numerical optimization have been very successful in this domain, and are implemented in widespread software suites like ANTs, Elastix, NiftyReg, or DARTEL. Over the last 7-8 years, learning-based techniques have emerged, which have a number of advantages like high computational efficiency, potential for higher accuracy, easy integration of supervision, and the ability to be part of a meta-architectures. However, their adoption in neuroimaging pipelines has so far been almost inexistent. Reasons include: lack of robustness to changes in MRI modality and resolution; lack of robust affine registration modules; lack of (guaranteed) symmetry; and, at a more practical level, the requirement of deep learning expertise that may be lacking at neuroimaging research sites. Here, we present EasyReg, an open-source, learning-based registration tool that can be easily used from the command line without any deep learning expertise or specific hardware. EasyReg combines the features of classical registration tools, the capabilities of modern deep learning methods, and the robustness to changes in MRI modality and resolution provided by our recent work in domain randomization. As a result, EasyReg is: fast; symmetric; diffeomorphic (and thus invertible); agnostic to MRI modality and resolution; compatible with affine and nonlinear registration; and does not require any preprocessing or parameter tuning. We present results on challenging registration tasks, showing that EasyReg is as accurate as classical methods when registering 1 mm isotropic scans within MRI modality, but much more accurate across modalities and resolutions. EasyReg is publicly available as part of FreeSurfer; see https://surfer.nmr.mgh.harvard.edu/fswiki/EasyReg .
Collapse
Affiliation(s)
- Juan Eugenio Iglesias
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School, Boston, 02129, USA.
- Department of Medical Physics and Biomedical Engineering, University College London, London, WC1V 6LJ, UK.
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Boston, 02139, USA.
| |
Collapse
|
10
|
Liu H, Gong G, Zou W, Hu N, Wang J. Topologically preserved registration of 3D CT images with deep networks. Phys Med Biol 2023; 68. [PMID: 36623316 DOI: 10.1088/1361-6560/acb197] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Accepted: 01/09/2023] [Indexed: 01/11/2023]
Abstract
Objective. Computed Tomography (CT) image registration makes fast and accurate imaging-based disease diagnosis possible. We aim to develop a framework which can perform accurate local registration of organs in 3D CT images while preserving the topology of transformation.Approach. In this framework, the Faster R-CNN method is first used to detect local areas containing organs from fixed and moving images whose results are then registered with a weakly supervised deep neural network. In this network, a novel 3D channel coordinate attention (CA) module is introduced to reduce the loss of position information. The image edge loss and the organ labelling loss are used to weakly supervise the training process of our deep network, which enables the network learning to focus on registering organs and image structures. An intuitive inverse module is also used to reduce the folding of deformation field. More specifically, the folding is suppressed directly by simultaneously maximizing forward and backward registration accuracy in the image domain rather than indirectly by measuring the consistency of forward and inverse deformation fields as usual.Main results. Our method achieves an average dice similarity coefficient (DSC) of 0.954 and an average Similarity (Sim) of 0.914 on publicly available liver datasets (LiTS for training and Sliver07 for testing) and achieves an average DSC of 0.914 and an average Sim of 0.947 on our home-built left ventricular myocardium (LVM) dataset.Significance. Experimental results show that our proposed method can significantly improve the registration accuracy of organs such as the liver and LVM. Moreover, our inverse module can intuitively improve the inherent topological preservation of transformations.
Collapse
Affiliation(s)
- Huaying Liu
- School of Electronic and Information Engineering, Soochow University, Suzhou 215006, People's Republic of China
| | - Guanzhong Gong
- Department of Radiation Oncology Physics and Technology, Shandong Cancer Hospital and Institute, Shandong First Medical University and Shandong Academy of Medical Sciences, Jinan 250000, People's Republic of China
| | - Wei Zou
- School of Electronic and Information Engineering, Soochow University, Suzhou 215006, People's Republic of China
| | - Nan Hu
- School of Electronic and Information Engineering, Soochow University, Suzhou 215006, People's Republic of China
| | - Jiajun Wang
- School of Electronic and Information Engineering, Soochow University, Suzhou 215006, People's Republic of China
| |
Collapse
|
11
|
Development and Application of a Standardized Testset for an Artificial Intelligence Medical Device Intended for the Computer-Aided Diagnosis of Diabetic Retinopathy. JOURNAL OF HEALTHCARE ENGINEERING 2023; 2023:7139560. [PMID: 36818382 PMCID: PMC9931476 DOI: 10.1155/2023/7139560] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/01/2022] [Revised: 05/21/2022] [Accepted: 11/24/2022] [Indexed: 02/10/2023]
Abstract
Objective To explore a centralized approach to build test sets and assess the performance of an artificial intelligence medical device (AIMD) which is intended for computer-aided diagnosis of diabetic retinopathy (DR). Method A framework was proposed to conduct data collection, data curation, and annotation. Deidentified colour fundus photographs were collected from 11 partner hospitals with raw labels. Photographs with sensitive information or authenticity issues were excluded during vetting. A team of annotators was recruited through qualification examinations and trained. The annotation process included three steps: initial annotation, review, and arbitration. The annotated data then composed a standardized test set, which was further imported to algorithms under test (AUT) from different developers. The algorithm outputs were compared with the final annotation results (reference standard). Result The test set consists of 6327 digital colour fundus photographs. The final labels include 5 stages of DR and non-DR, as well as other ocular diseases and photographs with unacceptable quality. The Fleiss Kappa was 0.75 among the annotators. The Cohen's kappa between raw labels and final labels is 0.5. Using this test set, five AUTs were tested and compared quantitatively. The metrics include accuracy, sensitivity, and specificity. The AUTs showed inhomogeneous capabilities to classify different types of fundus photographs. Conclusions This article demonstrated a workflow to build standardized test sets and conduct algorithm testing of the AIMD for computer-aided diagnosis of diabetic retinopathy. It may provide a reference to develop technical standards that promote product verification and quality control, improving the comparability of products.
Collapse
|
12
|
Zhang J, Liu Z, Ma Y, Zhao X, Yang B. Part-and-whole: A novel framework for deformable medical image registration. APPL INTELL 2022. [DOI: 10.1007/s10489-022-04329-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
|
13
|
Zou J, Gao B, Song Y, Qin J. A review of deep learning-based deformable medical image registration. Front Oncol 2022; 12:1047215. [PMID: 36568171 PMCID: PMC9768226 DOI: 10.3389/fonc.2022.1047215] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2022] [Accepted: 11/08/2022] [Indexed: 12/12/2022] Open
Abstract
The alignment of images through deformable image registration is vital to clinical applications (e.g., atlas creation, image fusion, and tumor targeting in image-guided navigation systems) and is still a challenging problem. Recent progress in the field of deep learning has significantly advanced the performance of medical image registration. In this review, we present a comprehensive survey on deep learning-based deformable medical image registration methods. These methods are classified into five categories: Deep Iterative Methods, Supervised Methods, Unsupervised Methods, Weakly Supervised Methods, and Latest Methods. A detailed review of each category is provided with discussions about contributions, tasks, and inadequacies. We also provide statistical analysis for the selected papers from the point of view of image modality, the region of interest (ROI), evaluation metrics, and method categories. In addition, we summarize 33 publicly available datasets that are used for benchmarking the registration algorithms. Finally, the remaining challenges, future directions, and potential trends are discussed in our review.
Collapse
Affiliation(s)
- Jing Zou
- Hong Kong Polytechnic University, Hong Kong, Hong Kong SAR, China
| | | | | | | |
Collapse
|
14
|
Cao Y, Fu T, Duan L, Dai Y, Gong L, Cao W, Liu D, Yang X, Ni X, Zheng J. CDFRegNet: A cross-domain fusion registration network for CT-to-CBCT image registration. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 224:107025. [PMID: 35872383 DOI: 10.1016/j.cmpb.2022.107025] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/08/2022] [Revised: 07/01/2022] [Accepted: 07/13/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND AND OBJECTIVE Computer tomography (CT) to cone-beam computed tomography (CBCT) image registration plays an important role in radiotherapy treatment placement, dose verification, and anatomic changes monitoring during radiotherapy. However, fast and accurate CT-to-CBCT image registration is still very challenging due to the intensity differences, the poor image quality of CBCT images, and inconsistent structure information. METHODS To address these problems, a novel unsupervised network named cross-domain fusion registration network (CDFRegNet) is proposed. First, a novel edge-guided attention module (EGAM) is designed, aiming at capturing edge information based on the gradient prior images and guiding the network to model the spatial correspondence between two image domains. Moreover, a novel cross-domain attention module (CDAM) is proposed to improve the network's ability to guide the network to effectively map and fuse the domain-specific features. RESULTS Extensive experiments on a real clinical dataset were carried out, and the experimental results verify that the proposed CDFRegNet can register CT to CBCT images effectively and obtain the best performance, while compared with other representative methods, with a mean DSC of 80.01±7.16%, a mean TRE of 2.27±0.62 mm, and a mean MHD of 1.50±0.32 mm. The ablation experiments also proved that our EGAM and CDAM can further improve the accuracy of the registration network and they can generalize well to other registration networks. CONCLUSION This paper proposed a novel CT-to-CBCT registration method based on EGAM and CDAM, which has the potential to improve the accuracy of multi-domain image registration.
Collapse
Affiliation(s)
- Yuzhu Cao
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei 230026, China; Medical Imaging Department, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
| | - Tianxiao Fu
- Department of Radiation Oncology, The First Affiliated Hospital of Soochow University, Suzhou 215006, China
| | - Luwen Duan
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei 230026, China
| | - Yakang Dai
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei 230026, China; Medical Imaging Department, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
| | - Lun Gong
- Key Laboratory of Mechanism Theory and Equipment Design of Ministry of Education, Tianjin University, Tianjin 300072, China
| | - Weiwei Cao
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei 230026, China; Medical Imaging Department, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
| | - Desen Liu
- Department of Thoracic Surgery, Suzhou Kowloon Hospital, Shanghai Jiao Tong University School of Medicine, Suzhou 215028, China
| | - Xiaodong Yang
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei 230026, China; Medical Imaging Department, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
| | - Xinye Ni
- The Affiliated Changzhou NO.2 People's Hospital of Nanjing Medical University, Changzhou 213003, China.
| | - Jian Zheng
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei 230026, China; Medical Imaging Department, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China; Jinan Guoke Medical Technology Development Co., Ltd, Jinan, 250101, China.
| |
Collapse
|
15
|
Li C, Li W, Liu C, Zheng H, Cai J, Wang S. Artificial intelligence in multi-parametric magnetic resonance imaging: A review. Med Phys 2022; 49:e1024-e1054. [PMID: 35980348 DOI: 10.1002/mp.15936] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2022] [Revised: 08/01/2022] [Accepted: 08/04/2022] [Indexed: 11/06/2022] Open
Abstract
Multi-parametric magnetic resonance imaging (mpMRI) is an indispensable tool in the clinical workflow for the diagnosis and treatment planning of various diseases. Machine learning-based artificial intelligence (AI) methods, especially those adopting the deep learning technique, have been extensively employed to perform mpMRI image classification, segmentation, registration, detection, reconstruction, and super-resolution. The current availability of increasing computational power and fast-improving AI algorithms have empowered numerous computer-based systems for applying mpMRI to disease diagnosis, imaging-guided radiotherapy, patient risk and overall survival time prediction, and the development of advanced quantitative imaging technology for magnetic resonance fingerprinting. However, the wide application of these developed systems in the clinic is still limited by a number of factors, including robustness, reliability, and interpretability. This survey aims to provide an overview for new researchers in the field as well as radiologists with the hope that they can understand the general concepts, main application scenarios, and remaining challenges of AI in mpMRI. This article is protected by copyright. All rights reserved.
Collapse
Affiliation(s)
- Cheng Li
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055, China
| | - Wen Li
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong SAR, China
| | - Chenyang Liu
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong SAR, China
| | - Hairong Zheng
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055, China
| | - Jing Cai
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong SAR, China
| | - Shanshan Wang
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055, China.,Peng Cheng Laboratory, Shenzhen, 518066, China.,Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou, 510080, China
| |
Collapse
|
16
|
GraformerDIR: Graph convolution transformer for deformable image registration. Comput Biol Med 2022; 147:105799. [DOI: 10.1016/j.compbiomed.2022.105799] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2022] [Revised: 05/06/2022] [Accepted: 06/26/2022] [Indexed: 01/02/2023]
|
17
|
Kang M, Hu X, Huang W, Scott MR, Reyes M. Dual-stream Pyramid Registration Network. Med Image Anal 2022; 78:102379. [DOI: 10.1016/j.media.2022.102379] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2020] [Revised: 01/25/2022] [Accepted: 01/27/2022] [Indexed: 10/19/2022]
|
18
|
Multimodal Remote Sensing Image Registration Methods and Advancements: A Survey. REMOTE SENSING 2021. [DOI: 10.3390/rs13245128] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
With rapid advancements in remote sensing image registration algorithms, comprehensive imaging applications are no longer limited to single-modal remote sensing images. Instead, multi-modal remote sensing (MMRS) image registration has become a research focus in recent years. However, considering multi-source, multi-temporal, and multi-spectrum input introduces significant nonlinear radiation differences in MMRS images for which researchers need to develop novel solutions. At present, comprehensive reviews and analyses of MMRS image registration methods are inadequate in related fields. Thus, this paper introduces three theoretical frameworks: namely, area-based, feature-based and deep learning-based methods. We present a brief review of traditional methods and focus on more advanced methods for MMRS image registration proposed in recent years. Our review or comprehensive analysis is intended to provide researchers in related fields with advanced understanding to achieve further breakthroughs and innovations.
Collapse
|