1
|
Wang C, Guo L, Zhu J, Zhu L, Li C, Zhu H, Song A, Lu L, Teng GJ, Navab N, Jiang Z. Review of robotic systems for thoracoabdominal puncture interventional surgery. APL Bioeng 2024; 8:021501. [PMID: 38572313 PMCID: PMC10987197 DOI: 10.1063/5.0180494] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2023] [Accepted: 03/11/2024] [Indexed: 04/05/2024] Open
Abstract
Cancer, with high morbidity and high mortality, is one of the major burdens threatening human health globally. Intervention procedures via percutaneous puncture have been widely used by physicians due to its minimally invasive surgical approach. However, traditional manual puncture intervention depends on personal experience and faces challenges in terms of precisely puncture, learning-curve, safety and efficacy. The development of puncture interventional surgery robotic (PISR) systems could alleviate the aforementioned problems to a certain extent. This paper attempts to review the current status and prospective of PISR systems for thoracic and abdominal application. In this review, the key technologies related to the robotics, including spatial registration, positioning navigation, puncture guidance feedback, respiratory motion compensation, and motion control, are discussed in detail.
Collapse
Affiliation(s)
- Cheng Wang
- Hanglok-Tech Co. Ltd., Hengqin 519000, People's Republic of China
| | - Li Guo
- Hanglok-Tech Co. Ltd., Hengqin 519000, People's Republic of China
| | | | - Lifeng Zhu
- State Key Laboratory of Digital Medical Engineering, Jiangsu Key Lab of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, People's Republic of China
| | - Chichi Li
- School of Computer Science and Engineering, Macau University of Science and Technology, Macau, 999078, People's Republic of China
| | - Haidong Zhu
- Center of Interventional Radiology and Vascular Surgery, Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing 210009, People's Republic of China
| | - Aiguo Song
- State Key Laboratory of Digital Medical Engineering, Jiangsu Key Lab of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, People's Republic of China
| | | | - Gao-Jun Teng
- Center of Interventional Radiology and Vascular Surgery, Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing 210009, People's Republic of China
| | | | - Zhongliang Jiang
- Computer Aided Medical Procedures, Technical University of Munich, Munich 80333, Germany
| |
Collapse
|
2
|
Hausmann D, Lerch A, Hitziger S, Farkas M, Weiland E, Lemke A, Grimm M, Kubik-Huch RA. AI-Supported Autonomous Uterus Reconstructions: First Application in MRI Using 3D SPACE with Iterative Denoising. Acad Radiol 2024; 31:1400-1409. [PMID: 37925344 DOI: 10.1016/j.acra.2023.09.035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2023] [Revised: 09/19/2023] [Accepted: 09/22/2023] [Indexed: 11/06/2023]
Abstract
RATIONALE AND OBJECTIVES T2-weighted imaging in at least two orthogonal planes is recommended for assessment of the uterus. To determine whether a convolutional neural network-based algorithm could be used for the re-constructions of uterus axes derived from a 3D SPACE with iterative denoising. MATERIALS AND METHODS 50 patients aged 18-81 (mean: 42) years who underwent an MRI examination of the uterus participated voluntarily in this prospective study after informed consent. In addition to a standard MRI pelvis protocol, a 3D SPACE research application sequence was acquired in sagittal orientation. Reconstructions for both the cervix and the cavum in the short and long axes were performed by a research trainee (T), an experienced radiologist (E), and the prototype software (P). In the next step, the reconstructions were evaluated anonymously by two experienced readers according to 5-point-Likert-Scales. In addition, the length of the cervical canal, the length of the cavum and the distance between the tube angles were measured on all reconstructions. Interobserver agreement was assessed for all ratings. RESULTS For all axes, significant differences were found between the scores of the reconstructions by research T, E and P. P received higher scores and was preferred significantly more often with the exception of the comparison of the reconstruction Cervix short of E (Cervix short: P vs. T: p = 0.02; P vs. E: p = 0.26; Cervix long: P vs. T: p = 0.01; P vs. E: p < 0.01; Cavum short: P vs. T: p = 0.01; P vs. E: p = 0.02; Cavum long: P vs. T: p < 0.01; P vs. E: p < 0.01). Regarding the measured diameters, (length of cervical canal/cavum/distance between tube angles) significantly larger diameters were recorded for P compared to E and T (Cervix long (mm): T: 25.43; E: 25.65; P: 26.65; Cavum short (mm): T: 26.24; E: 25.04; P: 27.33; Cavum long (mm): T: 31.98; E: 32.91; P: 34.41; P vs. T: p < 0.01); P vs. E: p = 0.04). Moderate to substantial agreement was found between Reader 1 and Reader 2 (range: 0.39-0.67). CONCLUSION P was able to reconstruct the axes at least as well as or better than E and T. P could thereby lead to workflow facilitation and enable more efficient reporting of uterine MRI.
Collapse
Affiliation(s)
- Daniel Hausmann
- Department of Radiology, Kantonsspital Baden, Im Ergel 1, Baden, 5404, Switzerland (D.H., A.L., M.F., M.G., K.H.); Department of Radiology and Nuclear Medicine, University Medical Center Mannheim, Heidelberg University, Mannheim, Germany (D.H.).
| | - Aline Lerch
- Department of Radiology, Kantonsspital Baden, Im Ergel 1, Baden, 5404, Switzerland (D.H., A.L., M.F., M.G., K.H.); Institute for Translational Medicine, ETH Zurich, Zurich, Switzerland (A.L); ETH, Department of Health Sciences and Technology (A.L.)
| | | | - Monika Farkas
- Department of Radiology, Kantonsspital Baden, Im Ergel 1, Baden, 5404, Switzerland (D.H., A.L., M.F., M.G., K.H.)
| | - Elisabeth Weiland
- MR Application Predevelopment, Siemens Healthcare GmbH, Erlangen, Germany (E.W.)
| | | | - Maximilian Grimm
- Department of Radiology, Kantonsspital Baden, Im Ergel 1, Baden, 5404, Switzerland (D.H., A.L., M.F., M.G., K.H.)
| | - Rahel A Kubik-Huch
- Department of Radiology, Kantonsspital Baden, Im Ergel 1, Baden, 5404, Switzerland (D.H., A.L., M.F., M.G., K.H.)
| |
Collapse
|
3
|
Sabater-Gárriz Á, Gaya-Morey FX, Buades-Rubio JM, Manresa-Yee C, Montoya P, Riquelme I. Automated facial recognition system using deep learning for pain assessment in adults with cerebral palsy. Digit Health 2024; 10:20552076241259664. [PMID: 38846372 PMCID: PMC11155325 DOI: 10.1177/20552076241259664] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Accepted: 05/07/2024] [Indexed: 06/09/2024] Open
Abstract
Objective Assessing pain in individuals with neurological conditions like cerebral palsy is challenging due to limited self-reporting and expression abilities. Current methods lack sensitivity and specificity, underlining the need for a reliable evaluation protocol. An automated facial recognition system could revolutionize pain assessment for such patients.The research focuses on two primary goals: developing a dataset of facial pain expressions for individuals with cerebral palsy and creating a deep learning-based automated system for pain assessment tailored to this group. Methods The study trained ten neural networks using three pain image databases and a newly curated CP-PAIN Dataset of 109 images from cerebral palsy patients, classified by experts using the Facial Action Coding System. Results The InceptionV3 model demonstrated promising results, achieving 62.67% accuracy and a 61.12% F1 score on the CP-PAIN dataset. Explainable AI techniques confirmed the consistency of crucial features for pain identification across models. Conclusion The study underscores the potential of deep learning in developing reliable pain detection systems using facial recognition for individuals with communication impairments due to neurological conditions. A more extensive and diverse dataset could further enhance the models' sensitivity to subtle pain expressions in cerebral palsy patients and possibly extend to other complex neurological disorders. This research marks a significant step toward more empathetic and accurate pain management for vulnerable populations.
Collapse
Affiliation(s)
- Álvaro Sabater-Gárriz
- Department of Research and Training, Balearic ASPACE Foundation, Marratxí, Spain
- Department of Nursing and Physiotherapy, University of the Balearic Islands, Palma de Mallorca, Spain
- Research Institute on Health Sciences (IUNICS), University of the Balearic Islands, Palma de Mallorca, Spain
- Health Research Institute of the Balearic Islands (IdISBa), Palma de Mallorca, Spain
| | - F Xavier Gaya-Morey
- Department of Mathematics and Computer Science, University of the Balearic Islands, Palma de Mallorca, Spain
| | - José María Buades-Rubio
- Research Institute on Health Sciences (IUNICS), University of the Balearic Islands, Palma de Mallorca, Spain
- Department of Mathematics and Computer Science, University of the Balearic Islands, Palma de Mallorca, Spain
| | - Cristina Manresa-Yee
- Research Institute on Health Sciences (IUNICS), University of the Balearic Islands, Palma de Mallorca, Spain
- Department of Mathematics and Computer Science, University of the Balearic Islands, Palma de Mallorca, Spain
| | - Pedro Montoya
- Research Institute on Health Sciences (IUNICS), University of the Balearic Islands, Palma de Mallorca, Spain
- Health Research Institute of the Balearic Islands (IdISBa), Palma de Mallorca, Spain
- Center for Mathematics, Computation and Cognition, Federal University of ABC, São Bernardo do Campo, Brazil
| | - Inmaculada Riquelme
- Department of Nursing and Physiotherapy, University of the Balearic Islands, Palma de Mallorca, Spain
- Research Institute on Health Sciences (IUNICS), University of the Balearic Islands, Palma de Mallorca, Spain
- Health Research Institute of the Balearic Islands (IdISBa), Palma de Mallorca, Spain
| |
Collapse
|
4
|
Wang AQ, Yu EM, Dalca AV, Sabuncu MR. A robust and interpretable deep learning framework for multi-modal registration via keypoints. Med Image Anal 2023; 90:102962. [PMID: 37769550 PMCID: PMC10591968 DOI: 10.1016/j.media.2023.102962] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Revised: 08/24/2023] [Accepted: 09/07/2023] [Indexed: 10/03/2023]
Abstract
We present KeyMorph, a deep learning-based image registration framework that relies on automatically detecting corresponding keypoints. State-of-the-art deep learning methods for registration often are not robust to large misalignments, are not interpretable, and do not incorporate the symmetries of the problem. In addition, most models produce only a single prediction at test-time. Our core insight which addresses these shortcomings is that corresponding keypoints between images can be used to obtain the optimal transformation via a differentiable closed-form expression. We use this observation to drive the end-to-end learning of keypoints tailored for the registration task, and without knowledge of ground-truth keypoints. This framework not only leads to substantially more robust registration but also yields better interpretability, since the keypoints reveal which parts of the image are driving the final alignment. Moreover, KeyMorph can be designed to be equivariant under image translations and/or symmetric with respect to the input image ordering. Finally, we show how multiple deformation fields can be computed efficiently and in closed-form at test time corresponding to different transformation variants. We demonstrate the proposed framework in solving 3D affine and spline-based registration of multi-modal brain MRI scans. In particular, we show registration accuracy that surpasses current state-of-the-art methods, especially in the context of large displacements. Our code is available at https://github.com/alanqrwang/keymorph.
Collapse
Affiliation(s)
- Alan Q Wang
- School of Electrical and Computer Engineering, Cornell University and Cornell Tech, New York, NY 10044, USA; Department of Radiology, Weill Cornell Medical School, New York, NY 10065, USA.
| | - Evan M Yu
- Iterative Scopes, Cambridge, MA 02139, USA
| | - Adrian V Dalca
- Computer Science and Artificial Intelligence Lab at the Massachusetts Institute of Technology, Cambridge, MA 02139, USA; A.A. Martinos Center for Biomedical Imaging at the Massachusetts General Hospital, Charlestown, MA 02129, USA
| | - Mert R Sabuncu
- School of Electrical and Computer Engineering, Cornell University and Cornell Tech, New York, NY 10044, USA; Department of Radiology, Weill Cornell Medical School, New York, NY 10065, USA
| |
Collapse
|
5
|
Xu L, Jiang P, Tsui T, Liu J, Zhang X, Yu L, Niu T. 4D-CT deformable image registration using unsupervised recursive cascaded full-resolution residual networks. Bioeng Transl Med 2023; 8:e10587. [PMID: 38023695 PMCID: PMC10658570 DOI: 10.1002/btm2.10587] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2023] [Revised: 06/30/2023] [Accepted: 07/30/2023] [Indexed: 12/01/2023] Open
Abstract
A novel recursive cascaded full-resolution residual network (RCFRR-Net) for abdominal four-dimensional computed tomography (4D-CT) image registration was proposed. The entire network was end-to-end and trained in the unsupervised approach, which meant that the deformation vector field, which presented the ground truth, was not needed during training. The network was designed by cascading three full-resolution residual subnetworks with different architectures. The training loss consisted of the image similarity loss and the deformation vector field regularization loss, which were calculated based on the final warped image and the fixed image, allowing all cascades to be trained jointly and perform the progressive registration cooperatively. Extensive network testing was conducted using diverse datasets, including an internal 4D-CT dataset, a public DIRLAB 4D-CT dataset, and a 4D cone-beam CT (4D-CBCT) dataset. Compared with the iteration-based demon method and two deep learning-based methods (VoxelMorph and recursive cascaded network), the RCFRR-Net achieved consistent and significant gains, which demonstrated that the proposed method had superior performance and generalization capability in medical image registration. The proposed RCFRR-Net was a promising tool for various clinical applications.
Collapse
Affiliation(s)
- Lei Xu
- Department of Radiation Oncologythe First Affiliated Hospital of Xi'an Jiaotong UniversityXi'anShaanxiChina
- Institute of Biomedical EngineeringShenzhen Bay LaboratoryShenzhenGuangdongChina
| | - Ping Jiang
- Department of Radiation OncologyPeking University 3rd HospitalBeijingChina
| | - Tiffany Tsui
- Loyola University Medical CenterMaywoodIllinoisUSA
| | - Junyan Liu
- Department of Radiation OncologyStanford University School of MedicineStanfordCaliforniaUSA
| | - Xiping Zhang
- Department of Radiation OncologyOzarks HealthcareWest PlainsMissouriUSA
| | - Lequan Yu
- Department of Statistics and Actuarial ScienceThe University of Hong Kong, HKSARHong KongChina
| | - Tianye Niu
- Institute of Biomedical EngineeringShenzhen Bay LaboratoryShenzhenGuangdongChina
- Peking University Aerospace School of Clinical Medicine, Aerospace Center HospitalBeijingChina
| |
Collapse
|
6
|
Gan Z, Sun W, Liao K, Yang X. Probabilistic Modeling for Image Registration Using Radial Basis Functions: Application to Cardiac Motion Estimation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023; 34:7324-7338. [PMID: 35073271 DOI: 10.1109/tnnls.2022.3141119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Cardiovascular diseases (CVDs) are the leading cause of death, affecting the cardiac dynamics over the cardiac cycle. Estimation of cardiac motion plays an essential role in many medical clinical tasks. This article proposes a probabilistic framework for image registration using compact support radial basis functions (CSRBFs) to estimate cardiac motion. A variational inference-based generative model with convolutional neural networks (CNNs) is proposed to learn the probabilistic coefficients of CSRBFs used in image deformation. We designed two networks to estimate the deformation coefficients of CSRBFs: the first one solves the spatial transformation using given control points, and the second one models the transformation using drifting control points. The given-point-based network estimates the probabilistic coefficients of control points. In contrast, the drifting-point-based model predicts the probabilistic coefficients and spatial distribution of control points simultaneously. To regularize these coefficients, we derive the bending energy (BE) in the variational bound by defining the covariance of coefficients. The proposed framework has been evaluated on the cardiac motion estimation and the calculation of the myocardial strain. In the experiments, 1409 slice pairs of end-diastolic (ED) and end-systolic (ES) phase in 4-D cardiac magnetic resonance (MR) images selected from three public datasets are employed to evaluate our networks. The experimental results show that our framework outperforms the state-of-the-art registration methods concerning the deformation smoothness and registration accuracy.
Collapse
|
7
|
Chi Y, Xu Y, Liu H, Wu X, Liu Z, Mao J, Xu G, Huang W. A two-step deep learning method for 3DCT-2DUS kidney registration during breathing. Sci Rep 2023; 13:12846. [PMID: 37553480 PMCID: PMC10409729 DOI: 10.1038/s41598-023-40133-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Accepted: 08/04/2023] [Indexed: 08/10/2023] Open
Abstract
This work proposed KidneyRegNet, a novel deep registration pipeline for 3D CT and 2D U/S kidney scans of free breathing, which comprises a feature network, and a 3D-2D CNN-based registration network. The feature network has handcrafted texture feature layers to reduce the semantic gap. The registration network is an encoder-decoder structure with loss of feature-image-motion (FIM), which enables hierarchical regression at decoder layers and avoids multiple network concatenation. It was first pretrained with a retrospective dataset cum training data generation strategy and then adapted to specific patient data under unsupervised one-cycle transfer learning in onsite applications. The experiment was performed on 132 U/S sequences, 39 multiple-phase CT and 210 public single-phase CT images, and 25 pairs of CT and U/S sequences. This resulted in a mean contour distance (MCD) of 0.94 mm between kidneys on CT and U/S images and MCD of 1.15 mm on CT and reference CT images. Datasets with small transformations resulted in MCDs of 0.82 and 1.02 mm, respectively. Large transformations resulted in MCDs of 1.10 and 1.28 mm, respectively. This work addressed difficulties in 3DCT-2DUS kidney registration during free breathing via novel network structures and training strategies.
Collapse
Affiliation(s)
- Yanling Chi
- Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR), 1 Fusionopolis Way #21-01 Connexis South, Singapore, 138632, Republic of Singapore.
| | - Yuyu Xu
- Key Laboratory of Biological Targeting Diagnosis, Therapy and Rehabilitation of Guangdong Higher Education Institutes, The Fifth Affiliated Hospital of Guangzhou Medical University, Guangzhou, 510700, People's Republic of China
| | - Huiying Liu
- Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR), 1 Fusionopolis Way #21-01 Connexis South, Singapore, 138632, Republic of Singapore
| | - Xiaoxiang Wu
- Key Laboratory of Biological Targeting Diagnosis, Therapy and Rehabilitation of Guangdong Higher Education Institutes, The Fifth Affiliated Hospital of Guangzhou Medical University, Guangzhou, 510700, People's Republic of China
| | - Zhiqiang Liu
- Key Laboratory of Biological Targeting Diagnosis, Therapy and Rehabilitation of Guangdong Higher Education Institutes, The Fifth Affiliated Hospital of Guangzhou Medical University, Guangzhou, 510700, People's Republic of China
| | - Jiawei Mao
- Creative Medtech Solutions Pte Ltd, Singapore, Republic of Singapore
| | - Guibin Xu
- Key Laboratory of Biological Targeting Diagnosis, Therapy and Rehabilitation of Guangdong Higher Education Institutes, The Fifth Affiliated Hospital of Guangzhou Medical University, Guangzhou, 510700, People's Republic of China.
| | - Weimin Huang
- Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR), 1 Fusionopolis Way #21-01 Connexis South, Singapore, 138632, Republic of Singapore.
| |
Collapse
|
8
|
Czolbe S, Pegios P, Krause O, Feragen A. Semantic similarity metrics for image registration. Med Image Anal 2023; 87:102830. [PMID: 37172390 DOI: 10.1016/j.media.2023.102830] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2021] [Revised: 01/19/2023] [Accepted: 04/20/2023] [Indexed: 05/15/2023]
Abstract
Image registration aims to find geometric transformations that align images. Most algorithmic and deep learning-based methods solve the registration problem by minimizing a loss function, consisting of a similarity metric comparing the aligned images, and a regularization term ensuring smoothness of the transformation. Existing similarity metrics like Euclidean Distance or Normalized Cross-Correlation focus on aligning pixel intensity values or correlations, giving difficulties with low intensity contrast, noise, and ambiguous matching. We propose a semantic similarity metric for image registration, focusing on aligning image areas based on semantic correspondence instead. Our approach learns dataset-specific features that drive the optimization of a learning-based registration model. We train both an unsupervised approach extracting features with an auto-encoder, and a semi-supervised approach using supplemental segmentation data. We validate the semantic similarity metric using both deep-learning-based and algorithmic image registration methods. Compared to existing methods across four different image modalities and applications, the method achieves consistently high registration accuracy and smooth transformation fields.
Collapse
Affiliation(s)
- Steffen Czolbe
- Department of Computer Science, University of Copenhagen, Denmark.
| | | | - Oswin Krause
- Department of Computer Science, University of Copenhagen, Denmark
| | - Aasa Feragen
- DTU Compute, Technical University of Denmark, Denmark
| |
Collapse
|
9
|
Salehi M, Vafaei Sadr A, Mahdavi SR, Arabi H, Shiri I, Reiazi R. Deep Learning-based Non-rigid Image Registration for High-dose Rate Brachytherapy in Inter-fraction Cervical Cancer. J Digit Imaging 2023; 36:574-587. [PMID: 36417026 PMCID: PMC10039214 DOI: 10.1007/s10278-022-00732-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2021] [Revised: 07/04/2022] [Accepted: 07/18/2022] [Indexed: 11/25/2022] Open
Abstract
In this study, an inter-fraction organ deformation simulation framework for the locally advanced cervical cancer (LACC), which considers the anatomical flexibility, rigidity, and motion within an image deformation, was proposed. Data included 57 CT scans (7202 2D slices) of patients with LACC randomly divided into the train (n = 42) and test (n = 15) datasets. In addition to CT images and the corresponding RT structure (bladder, cervix, and rectum), the bone was segmented, and the coaches were eliminated. The correlated stochastic field was simulated using the same size as the target image (used for deformation) to produce the general random deformation. The deformation field was optimized to have a maximum amplitude in the rectum region, a moderate amplitude in the bladder region, and an amplitude as minimum as possible within bony structures. The DIRNet is a convolutional neural network that consists of convolutional regressors, spatial transformation, as well as resampling blocks. It was implemented by different parameters. Mean Dice indices of 0.89 ± 0.02, 0.96 ± 0.01, and 0.93 ± 0.02 were obtained for the cervix, bladder, and rectum (defined as at-risk organs), respectively. Furthermore, a mean average symmetric surface distance of 1.61 ± 0.46 mm for the cervix, 1.17 ± 0.15 mm for the bladder, and 1.06 ± 0.42 mm for the rectum were achieved. In addition, a mean Jaccard of 0.86 ± 0.04 for the cervix, 0.93 ± 0.01 for the bladder, and 0.88 ± 0.04 for the rectum were observed on the test dataset (15 subjects). Deep learning-based non-rigid image registration is, therefore, proposed for the high-dose-rate brachytherapy in inter-fraction cervical cancer since it outperformed conventional algorithms.
Collapse
Affiliation(s)
- Mohammad Salehi
- Department of Medical Physics, School of Medicine, Iran University of Medical Sciences, Tehran, Iran
| | - Alireza Vafaei Sadr
- Department of Theoretical Physics and Center for Astroparticle Physics, University of Geneva, Geneva, Switzerland
- Institute of Pathology, RWTH Aachen University Hospital, Aachen, Germany
| | - Seied Rabi Mahdavi
- Department of Medical Physics, School of Medicine, Iran University of Medical Sciences, Tehran, Iran
| | - Hossein Arabi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211, Geneva 4, Switzerland
| | - Isaac Shiri
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211, Geneva 4, Switzerland.
| | - Reza Reiazi
- Department of Medical Physics, School of Medicine, Iran University of Medical Sciences, Tehran, Iran.
- Division of Radiation Oncology, Department of Radiation Physics, University of Texas MD Anderson Cancer Center, Houston, USA.
| |
Collapse
|
10
|
Ho TT, Kim WJ, Lee CH, Jin GY, Chae KJ, Choi S. An unsupervised image registration method employing chest computed tomography images and deep neural networks. Comput Biol Med 2023; 154:106612. [PMID: 36738711 DOI: 10.1016/j.compbiomed.2023.106612] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2022] [Revised: 01/11/2023] [Accepted: 01/28/2023] [Indexed: 02/04/2023]
Abstract
BACKGROUND Deformable image registration is crucial for multiple radiation therapy applications. Fast registration of computed tomography (CT) lung images is challenging because of the large and nonlinear deformation between inspiration and expiration. With advancements in deep learning techniques, learning-based registration methods are considered efficient alternatives to traditional methods in terms of accuracy and computational cost. METHOD In this study, an unsupervised lung registration network (LRN) with cycle-consistent training is proposed to align two acquired CT-derived lung datasets during breath-holds at inspiratory and expiratory levels without utilizing any ground-truth registration results. Generally, the LRN model uses three loss functions: image similarity, regularization, and Jacobian determinant. Here, LRN was trained on the CT datasets of 705 subjects and tested using 10 pairs of public CT DIR-Lab datasets. Furthermore, to evaluate the effectiveness of the registration technique, target registration errors (TREs) of the LRN model were compared with those of the conventional algorithm (sum of squared tissue volume difference; SSTVD) and a state-of-the-art unsupervised registration method (VoxelMorph). RESULTS The results showed that the LRN with an average TRE of 1.78 ± 1.56 mm outperformed VoxelMorph with an average TRE of 2.43 ± 2.43 mm, which is comparable to that of SSTVD with an average TRE of 1.66 ± 1.49 mm. In addition, estimating the displacement vector field without any folding voxel consumed less than 2 s, demonstrating the superiority of the learning-based method with respect to fiducial marker tracking and the overall soft tissue alignment with a nearly real-time speed. CONCLUSIONS Therefore, this proposed method shows significant potential for use in time-sensitive pulmonary studies, such as lung motion tracking and image-guided surgery.
Collapse
Affiliation(s)
- Thao Thi Ho
- School of Mechanical Engineering, Kyungpook National University, Daegu, South Korea
| | - Woo Jin Kim
- Department of Internal Medicine and Environmental Health Center, Kangwon National University Hospital, School of Medicine, Kangwon National University, Chuncheon, South Korea
| | - Chang Hyun Lee
- Department of Radiology, Seoul National University, College of Medicine, Seoul National University Hospital, Seoul, South Korea; Department of Radiology, College of Medicine, The University of Iowa, Iowa City, IA, USA
| | - Gong Yong Jin
- Department of Radiology, Research Institute of Clinical Medicine of Jeonbuk National University, Biomedical Research Institute of Jeonbuk National University Hospital, Jeonju, South Korea
| | - Kum Ju Chae
- Department of Radiology, Research Institute of Clinical Medicine of Jeonbuk National University, Biomedical Research Institute of Jeonbuk National University Hospital, Jeonju, South Korea
| | - Sanghun Choi
- School of Mechanical Engineering, Kyungpook National University, Daegu, South Korea.
| |
Collapse
|
11
|
Awan R, Raza SEA, Lotz J, Weiss N, Rajpoot N. Deep feature based cross-slide registration. Comput Med Imaging Graph 2023; 104:102162. [PMID: 36584537 DOI: 10.1016/j.compmedimag.2022.102162] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2022] [Revised: 11/15/2022] [Accepted: 12/08/2022] [Indexed: 12/23/2022]
Abstract
Registration of multiple sections in a tissue block is an important pre-requisite task before any cross-slide image analysis. Non-rigid registration methods are capable of finding correspondence by locally transforming a moving image. These methods often rely on an initial guess to roughly align an image pair linearly and globally. This is essential to prevent convergence to a non-optimal minimum. We explore a deep feature based registration (DFBR) method which utilises data-driven descriptors to estimate the global transformation. A multi-stage strategy is adopted for improving the quality of registration. A visualisation tool is developed to view registered pairs of WSIs at different magnifications. With the help of this tool, one can apply a transformation on the fly without the need to generate a transformed moving WSI in a pyramidal form. We compare the performance on our dataset of data-driven descriptors with that of hand-crafted descriptors. Our approach can align the images with only small registration errors. The efficacy of our proposed method is evaluated for a subsequent non-rigid registration step. To this end, the first two steps of the ANHIR winner's framework are replaced with DFBR to register image pairs provided by the challenge. The modified framework produce comparable results to those of the challenge winning team.
Collapse
Affiliation(s)
- Ruqayya Awan
- Department of Computer Science, University of Warwick, CV4 7AL Coventry, UK.
| | - Shan E Ahmed Raza
- Department of Computer Science, University of Warwick, CV4 7AL Coventry, UK.
| | - Johannes Lotz
- Fraunhofer Institute for Digital Medicine MEVIS, Lübeck, Germany.
| | - Nick Weiss
- Fraunhofer Institute for Digital Medicine MEVIS, Lübeck, Germany.
| | - Nasir Rajpoot
- Department of Computer Science, University of Warwick, CV4 7AL Coventry, UK; Department of Pathology, University Hospitals Coventry, Warwickshire, UK; The Alan Turing Institute, London, UK.
| |
Collapse
|
12
|
Zhu X, Ding M, Zhang X. Free form deformation and symmetry constraint‐based multi‐modal brain image registration using generative adversarial nets. CAAI TRANSACTIONS ON INTELLIGENCE TECHNOLOGY 2023. [DOI: 10.1049/cit2.12159] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/13/2023] Open
Affiliation(s)
- Xingxing Zhu
- Department of Biomedical Engineering School of Life Science and Technology Ministry of Education Key Laboratory of Molecular Biophysics Huazhong University of Science and Technology Wuhan China
| | - Mingyue Ding
- Department of Biomedical Engineering School of Life Science and Technology Ministry of Education Key Laboratory of Molecular Biophysics Huazhong University of Science and Technology Wuhan China
| | - Xuming Zhang
- Department of Biomedical Engineering School of Life Science and Technology Ministry of Education Key Laboratory of Molecular Biophysics Huazhong University of Science and Technology Wuhan China
| |
Collapse
|
13
|
Zhao T, Cao G, Zhang Y, Zhang H, Xia C. Incremental learning of upper limb action pattern recognition based on mechanomyography. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.103959] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/15/2022]
|
14
|
Simović A, Lutovac-Banduka M, Lekić S, Kuleto V. Smart Visualization of Medical Images as a Tool in the Function of Education in Neuroradiology. Diagnostics (Basel) 2022; 12:diagnostics12123208. [PMID: 36553215 PMCID: PMC9777748 DOI: 10.3390/diagnostics12123208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Revised: 12/09/2022] [Accepted: 12/12/2022] [Indexed: 12/23/2022] Open
Abstract
The smart visualization of medical images (SVMI) model is based on multi-detector computed tomography (MDCT) data sets and can provide a clearer view of changes in the brain, such as tumors (expansive changes), bleeding, and ischemia on native imaging (i.e., a non-contrast MDCT scan). The new SVMI method provides a more precise representation of the brain image by hiding pixels that are not carrying information and rescaling and coloring the range of pixels essential for detecting and visualizing the disease. In addition, SVMI can be used to avoid the additional exposure of patients to ionizing radiation, which can lead to the occurrence of allergic reactions due to the contrast media administration. Results of the SVMI model were compared with the final diagnosis of the disease after additional diagnostics and confirmation by neuroradiologists, who are highly trained physicians with many years of experience. The application of the realized and presented SVMI model can optimize the engagement of material, medical, and human resources and has the potential for general application in medical training, education, and clinical research.
Collapse
Affiliation(s)
- Aleksandar Simović
- Department of Information Technology, Information Technology School ITS, 11000 Belgrade, Serbia
- Correspondence:
| | - Maja Lutovac-Banduka
- Department of RT-RK Institute, RT-RK for Computer Based Systems, 21000 Novi Sad, Serbia
| | - Snežana Lekić
- Department of Emergency Neuroradiology, University Clinical Centre of Serbia UKCS, 11000 Belgrade, Serbia
| | - Valentin Kuleto
- Department of Information Technology, Information Technology School ITS, 11000 Belgrade, Serbia
| |
Collapse
|
15
|
Zou J, Gao B, Song Y, Qin J. A review of deep learning-based deformable medical image registration. Front Oncol 2022; 12:1047215. [PMID: 36568171 PMCID: PMC9768226 DOI: 10.3389/fonc.2022.1047215] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2022] [Accepted: 11/08/2022] [Indexed: 12/12/2022] Open
Abstract
The alignment of images through deformable image registration is vital to clinical applications (e.g., atlas creation, image fusion, and tumor targeting in image-guided navigation systems) and is still a challenging problem. Recent progress in the field of deep learning has significantly advanced the performance of medical image registration. In this review, we present a comprehensive survey on deep learning-based deformable medical image registration methods. These methods are classified into five categories: Deep Iterative Methods, Supervised Methods, Unsupervised Methods, Weakly Supervised Methods, and Latest Methods. A detailed review of each category is provided with discussions about contributions, tasks, and inadequacies. We also provide statistical analysis for the selected papers from the point of view of image modality, the region of interest (ROI), evaluation metrics, and method categories. In addition, we summarize 33 publicly available datasets that are used for benchmarking the registration algorithms. Finally, the remaining challenges, future directions, and potential trends are discussed in our review.
Collapse
Affiliation(s)
- Jing Zou
- Hong Kong Polytechnic University, Hong Kong, Hong Kong SAR, China
| | | | | | | |
Collapse
|
16
|
Fan C, Hu K, Yuan Y, Li Y. A Data-driven Analysis of Global Research Trends in Medical Image: A Survey. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.10.047] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
17
|
Ge L, Wei X, Hao Y, Luo J, Xu Y. Unsupervised Histological Image Registration Using Structural Feature Guided Convolutional Neural Network. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:2414-2431. [PMID: 35363611 DOI: 10.1109/tmi.2022.3164088] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Registration of multiple stained images is a fundamental task in histological image analysis. In supervised methods, obtaining ground-truth data with known correspondences is laborious and time-consuming. Thus, unsupervised methods are expected. Unsupervised methods ease the burden of manual annotation but often at the cost of inferior results. In addition, registration of histological images suffers from appearance variance due to multiple staining, repetitive texture, and section missing during making tissue sections. To deal with these challenges, we propose an unsupervised structural feature guided convolutional neural network (SFG). Structural features are robust to multiple staining. The combination of low-resolution rough structural features and high-resolution fine structural features can overcome repetitive texture and section missing, respectively. SFG consists of two components of structural consistency constraints according to the formations of structural features, i.e., dense structural component and sparse structural component. The dense structural component uses structural feature maps of the whole image as structural consistency constraints, which represent local contextual information. The sparse structural component utilizes the distance of automatically obtained matched key points as structural consistency constraints because the matched key points in an image pair emphasize the matching of significant structures, which imply global information. In addition, a multi-scale strategy is used in both dense and sparse structural components to make full use of the structural information at low resolution and high resolution to overcome repetitive texture and section missing. The proposed method was evaluated on a public histological dataset (ANHIR) and ranked first as of Jan 18th, 2022.
Collapse
|
18
|
Cui K, Lin Y, Liu Y, Li Y. Deep residual-SVD network for brain image registration. Phys Med Biol 2022; 67. [DOI: 10.1088/1361-6560/ac79fa] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2022] [Accepted: 06/17/2022] [Indexed: 11/11/2022]
Abstract
Abstract
Objective. Medical image registration aims to find the deformation field that can align two images in a spatial position. A medical image registration method based on U-Net architecture has been proposed currently. However, U-Net architecture has few training parameters, which leads to weak learning ability, and it ignores the adverse effects of image noise on the registration accuracy. The article aims at addressing the problem of weak network learning ability and the adverse effects of noisy images on registration. Approach. Here we propose a novel unsupervised 3D brain image registration framework, which introduces the residual unit and singular value decomposition (SVD) denoising layer on the U-Net architecture. Residual unit solves the problem of network degradation, that is, registration accuracy becomes saturated and then degrades rapidly with the increase in network depth. SVD denoising layer uses the estimated model order for SVD-based low-rank image reconstruction. we use Akaike information criterion to estimate the appropriate model order, which is used to remove noise components. We use the exponential linear unit (ELU) as the activation function, which is more robust to noise than other peers. Main results. The proposed method is evaluated on the publicly available brain MRI datasets: Mindboggle101 and LPBA40. Experimental results demonstrate our method outperforms several state-of-the-art methods for the metric of Dice Score. The mean number of folding voxels and registration time are comparable to state-of-the-art methods. Significance. This study shows that Deep Residual-SVD Network can improve registration accuracy. This study also demonstrate that the residual unit can enhance the learning ability of the network, the SVD denoising layer can denoise effectively, and the ELU is more robust to noise.
Collapse
|
19
|
Chen X, Wang X, Zhang K, Fung KM, Thai TC, Moore K, Mannel RS, Liu H, Zheng B, Qiu Y. Recent advances and clinical applications of deep learning in medical image analysis. Med Image Anal 2022; 79:102444. [PMID: 35472844 PMCID: PMC9156578 DOI: 10.1016/j.media.2022.102444] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2021] [Revised: 03/09/2022] [Accepted: 04/01/2022] [Indexed: 02/07/2023]
Abstract
Deep learning has received extensive research interest in developing new medical image processing algorithms, and deep learning based models have been remarkably successful in a variety of medical imaging tasks to support disease detection and diagnosis. Despite the success, the further improvement of deep learning models in medical image analysis is majorly bottlenecked by the lack of large-sized and well-annotated datasets. In the past five years, many studies have focused on addressing this challenge. In this paper, we reviewed and summarized these recent studies to provide a comprehensive overview of applying deep learning methods in various medical image analysis tasks. Especially, we emphasize the latest progress and contributions of state-of-the-art unsupervised and semi-supervised deep learning in medical image analysis, which are summarized based on different application scenarios, including classification, segmentation, detection, and image registration. We also discuss major technical challenges and suggest possible solutions in the future research efforts.
Collapse
Affiliation(s)
- Xuxin Chen
- School of Electrical and Computer Engineering, University of Oklahoma, Norman, OK 73019, USA
| | - Ximin Wang
- School of Information Science and Technology, ShanghaiTech University, Shanghai 201210, China
| | - Ke Zhang
- School of Electrical and Computer Engineering, University of Oklahoma, Norman, OK 73019, USA
| | - Kar-Ming Fung
- Department of Pathology, University of Oklahoma Health Sciences Center, Oklahoma City, OK 73104, USA
| | - Theresa C Thai
- Department of Radiology, University of Oklahoma Health Sciences Center, Oklahoma City, OK 73104, USA
| | - Kathleen Moore
- Department of Obstetrics and Gynecology, University of Oklahoma Health Sciences Center, Oklahoma City, OK 73104, USA
| | - Robert S Mannel
- Department of Obstetrics and Gynecology, University of Oklahoma Health Sciences Center, Oklahoma City, OK 73104, USA
| | - Hong Liu
- School of Electrical and Computer Engineering, University of Oklahoma, Norman, OK 73019, USA
| | - Bin Zheng
- School of Electrical and Computer Engineering, University of Oklahoma, Norman, OK 73019, USA
| | - Yuchen Qiu
- School of Electrical and Computer Engineering, University of Oklahoma, Norman, OK 73019, USA.
| |
Collapse
|
20
|
Sengupta D, Gupta P, Biswas A. A survey on mutual information based medical image registration algorithms. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2021.11.023] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
21
|
Li H, Fan Y. MDReg-Net: Multi-resolution diffeomorphic image registration using fully convolutional networks with deep self-supervision. Hum Brain Mapp 2022; 43:2218-2231. [PMID: 35072327 PMCID: PMC8996355 DOI: 10.1002/hbm.25782] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2021] [Revised: 12/15/2021] [Accepted: 01/08/2022] [Indexed: 11/09/2022] Open
Abstract
We present a diffeomorphic image registration algorithm to learn spatial transformations between pairs of images to be registered using fully convolutional networks (FCNs) under a self-supervised learning setting. Particularly, a deep neural network is trained to estimate diffeomorphic spatial transformations between pairs of images by maximizing an image-wise similarity metric between fixed and warped moving images, similar to those adopted in conventional image registration algorithms. The network is implemented in a multi-resolution image registration framework to optimize and learn spatial transformations at different image resolutions jointly and incrementally with deep self-supervision in order to better handle large deformation between images. A spatial Gaussian smoothing kernel is integrated with the FCNs to yield sufficiently smooth deformation fields for diffeomorphic image registration. The spatial transformations learned at coarser resolutions are utilized to warp the moving image, which is subsequently used as input to the network for learning incremental transformations at finer resolutions. This procedure proceeds recursively to the full image resolution and the accumulated transformations serve as the final transformation to warp the moving image at the finest resolution. Experimental results for registering high-resolution 3D structural brain magnetic resonance (MR) images have demonstrated that image registration networks trained by our method obtain robust, diffeomorphic image registration results within seconds with improved accuracy compared with state-of-the-art image registration algorithms.
Collapse
Affiliation(s)
- Hongming Li
- Center for Biomedical Image Computing and Analytics (CBICA), Department of Radiology, Perelman School of MedicineUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Yong Fan
- Center for Biomedical Image Computing and Analytics (CBICA), Department of Radiology, Perelman School of MedicineUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | | |
Collapse
|
22
|
Deep learning-based plane pose regression in obstetric ultrasound. Int J Comput Assist Radiol Surg 2022; 17:833-839. [PMID: 35489005 PMCID: PMC9110476 DOI: 10.1007/s11548-022-02609-z] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2022] [Accepted: 03/10/2022] [Indexed: 01/16/2023]
Abstract
Purpose In obstetric ultrasound (US) scanning, the learner’s ability to mentally build a three-dimensional (3D) map of the fetus from a two-dimensional (2D) US image represents a major challenge in skill acquisition. We aim to build a US plane localisation system for 3D visualisation, training, and guidance without integrating additional sensors. Methods We propose a regression convolutional neural network (CNN) using image features to estimate the six-dimensional pose of arbitrarily oriented US planes relative to the fetal brain centre. The network was trained on synthetic images acquired from phantom 3D US volumes and fine-tuned on real scans. Training data was generated by slicing US volumes into imaging planes in Unity at random coordinates and more densely around the standard transventricular (TV) plane. Results With phantom data, the median errors are 0.90 mm/1.17\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$^\circ $$\end{document}∘ and 0.44 mm/1.21\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$^\circ $$\end{document}∘ for random planes and planes close to the TV one, respectively. With real data, using a different fetus with the same gestational age (GA), these errors are 11.84 mm/25.17\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$^\circ $$\end{document}∘. The average inference time is 2.97 ms per plane. Conclusion The proposed network reliably localises US planes within the fetal brain in phantom data and successfully generalises pose regression for an unseen fetal brain from a similar GA as in training. Future development will expand the prediction to volumes of the whole fetus and assess its potential for vision-based, freehand US-assisted navigation when acquiring standard fetal planes. Supplementary Information The online version contains supplementary material available at 10.1007/s11548-022-02609-z.
Collapse
|
23
|
Tax CMW, Bastiani M, Veraart J, Garyfallidis E, Okan Irfanoglu M. What's new and what's next in diffusion MRI preprocessing. Neuroimage 2022; 249:118830. [PMID: 34965454 PMCID: PMC9379864 DOI: 10.1016/j.neuroimage.2021.118830] [Citation(s) in RCA: 29] [Impact Index Per Article: 14.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2021] [Revised: 10/26/2021] [Accepted: 12/15/2021] [Indexed: 02/07/2023] Open
Abstract
Diffusion MRI (dMRI) provides invaluable information for the study of tissue microstructure and brain connectivity, but suffers from a range of imaging artifacts that greatly challenge the analysis of results and their interpretability if not appropriately accounted for. This review will cover dMRI artifacts and preprocessing steps, some of which have not typically been considered in existing pipelines or reviews, or have only gained attention in recent years: brain/skull extraction, B-matrix incompatibilities w.r.t the imaging data, signal drift, Gibbs ringing, noise distribution bias, denoising, between- and within-volumes motion, eddy currents, outliers, susceptibility distortions, EPI Nyquist ghosts, gradient deviations, B1 bias fields, and spatial normalization. The focus will be on "what's new" since the notable advances prior to and brought by the Human Connectome Project (HCP), as presented in the predecessing issue on "Mapping the Connectome" in 2013. In addition to the development of novel strategies for dMRI preprocessing, exciting progress has been made in the availability of open source tools and reproducible pipelines, databases and simulation tools for the evaluation of preprocessing steps, and automated quality control frameworks, amongst others. Finally, this review will consider practical considerations and our view on "what's next" in dMRI preprocessing.
Collapse
Affiliation(s)
- Chantal M W Tax
- Image Sciences Institute, University Medical Center Utrecht, The Netherlands; Cardiff University Brain Research Imaging Centre, School of Physics and Astronomy, Cardiff University, UK.
| | - Matteo Bastiani
- Sir Peter Mansfield Imaging Centre, School of Medicine, University of Nottingham, UK; Wellcome Centre for Integrative Neuroimaging (WIN), Centre for Functional Magnetic Resonance Imaging of the Brain (FMRIB), University of Oxford, UK
| | - Jelle Veraart
- Center for Biomedical Imaging, New York University Grossman School of Medicine, NY, USA
| | | | - M Okan Irfanoglu
- Quantitative Medical Imaging Section, National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Bethesda, MD, USA
| |
Collapse
|
24
|
Qasim M, Puigjaner D, Herrero J, López JM, Olivé C, Fortuny G, Garcia-Bennett J. Biomechanical modelling of the pelvic system: improving the accuracy of the location of neoplasms in MRI-TRUS fusion prostate biopsy. BMC Cancer 2022; 22:338. [PMID: 35351051 PMCID: PMC8962133 DOI: 10.1186/s12885-022-09432-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2021] [Accepted: 03/21/2022] [Indexed: 08/30/2023] Open
Abstract
Background An accurate knowledge of the relocation of prostate neoplasms during biopsy is of great importance to reduce the number of false negative results. Prostate neoplasms are visible in magnetic resonance images (MRI) but it is difficult for the practitioner to locate them at the time of performing a transrectal ultrasound (TRUS) guided biopsy. In this study, we present a new methodology, based on simulation, that predicts both prostate deformation and lesion migration during the biopsy. Methods A three-dimensional (3-D) anatomy model of the pelvic region, based on medical images, is constructed. A finite element (FE) numerical simulation of the organs motion and deformation as a result of the pressure exerted by the TRUS probe is carried out using the Code-Aster open-source computer software. Initial positions of potential prostate lesions prior to biopsy are taken into consideration and the final location of each lesion is targeted in the FE simulation output. Results Our 3-D FE simulations show that the effect of the pressure exerted by the TRUS probe is twofold as the prostate experiences both a motion and a deformation of its original shape. We targeted the relocation of five small prostate lesions when the TRUS probe exerts a force of 30 N on the rectum inner wall. The distance travelled by these lesions ranged between 5.6 and 13.9 mm. Conclusions Our new methodology can help to predict the location of neoplasms during a prostate biopsy but further studies are needed to validate our results. Moreover, the new methodology is completely developed on open-source software, which means that its implementation would be affordable to all healthcare providers.
Collapse
|
25
|
Abbasi S, Tavakoli M, Boveiri HR, Mosleh Shirazi MA, Khayami R, Khorasani H, Javidan R, Mehdizadeh A. Medical image registration using unsupervised deep neural network: A scoping literature review. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103444] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023]
|
26
|
Hoopes A, Hoffmann M, Greve DN, Fischl B, Guttag J, Dalca AV. Learning the Effect of Registration Hyperparameters with HyperMorph. THE JOURNAL OF MACHINE LEARNING FOR BIOMEDICAL IMAGING 2022; 1:003. [PMID: 36147449 PMCID: PMC9491317] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
We introduce HyperMorph, a framework that facilitates efficient hyperparameter tuning in learning-based deformable image registration. Classical registration algorithms perform an iterative pair-wise optimization to compute a deformation field that aligns two images. Recent learning-based approaches leverage large image datasets to learn a function that rapidly estimates a deformation for a given image pair. In both strategies, the accuracy of the resulting spatial correspondences is strongly influenced by the choice of certain hyperparameter values. However, an effective hyperparameter search consumes substantial time and human effort as it often involves training multiple models for different fixed hyperparameter values and may lead to suboptimal registration. We propose an amortized hyperparameter learning strategy to alleviate this burden by learning the impact of hyperparameters on deformation fields. We design a meta network, or hypernetwork, that predicts the parameters of a registration network for input hyperparameters, thereby comprising a single model that generates the optimal deformation field corresponding to given hyperparameter values. This strategy enables fast, high-resolution hyperparameter search at test-time, reducing the inefficiency of traditional approaches while increasing flexibility. We also demonstrate additional benefits of HyperMorph, including enhanced robustness to model initialization and the ability to rapidly identify optimal hyperparameter values specific to a dataset, image contrast, task, or even anatomical region, all without the need to retrain models. We make our code publicly available at http://hypermorph.voxelmorph.net.
Collapse
Affiliation(s)
- Andrew Hoopes
- Martinos Center for Biomedical Imaging, Massachusetts General Hospital
| | - Malte Hoffmann
- Martinos Center for Biomedical Imaging, Massachusetts General Hospital
- Department of Radiology, Harvard Medical School
| | - Douglas N Greve
- Martinos Center for Biomedical Imaging, Massachusetts General Hospital
- Department of Radiology, Harvard Medical School
| | - Bruce Fischl
- Martinos Center for Biomedical Imaging, Massachusetts General Hospital
- Department of Radiology, Harvard Medical School
- Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology
| | - John Guttag
- Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology
| | - Adrian V Dalca
- Martinos Center for Biomedical Imaging, Massachusetts General Hospital
- Department of Radiology, Harvard Medical School
- Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology
| |
Collapse
|
27
|
Hoffmann M, Billot B, Greve DN, Iglesias JE, Fischl B, Dalca AV. SynthMorph: Learning Contrast-Invariant Registration Without Acquired Images. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:543-558. [PMID: 34587005 PMCID: PMC8891043 DOI: 10.1109/tmi.2021.3116879] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Abstract
We introduce a strategy for learning image registration without acquired imaging data, producing powerful networks agnostic to contrast introduced by magnetic resonance imaging (MRI). While classical registration methods accurately estimate the spatial correspondence between images, they solve an optimization problem for every new image pair. Learning-based techniques are fast at test time but limited to registering images with contrasts and geometric content similar to those seen during training. We propose to remove this dependency on training data by leveraging a generative strategy for diverse synthetic label maps and images that exposes networks to a wide range of variability, forcing them to learn more invariant features. This approach results in powerful networks that accurately generalize to a broad array of MRI contrasts. We present extensive experiments with a focus on 3D neuroimaging, showing that this strategy enables robust and accurate registration of arbitrary MRI contrasts even if the target contrast is not seen by the networks during training. We demonstrate registration accuracy surpassing the state of the art both within and across contrasts, using a single model. Critically, training on arbitrary shapes synthesized from noise distributions results in competitive performance, removing the dependency on acquired data of any kind. Additionally, since anatomical label maps are often available for the anatomy of interest, we show that synthesizing images from these dramatically boosts performance, while still avoiding the need for real intensity images. Our code is available at doic https://w3id.org/synthmorph.
Collapse
|
28
|
Soleimani M, Aghagolzadeh A, Ezoji M. Symmetry-based representation for registration of multimodal images. Med Biol Eng Comput 2022; 60:1015-1032. [PMID: 35171412 DOI: 10.1007/s11517-022-02515-1] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2020] [Accepted: 01/21/2022] [Indexed: 11/24/2022]
Abstract
We propose a new two-dimensional structural representation method for registration of multimodal images by using the local structural symmetry of images, which is similar at different modalities. The symmetry is measured in various orientations and the best is mapped and used for the representation image. The optimum performance is obtained when using only two different orientations, which is called binary dominant symmetry representation (BDSR). This representation is highly robust to noise and intensity non-uniformity. We also propose a new objective function based on L2 distance with low sensitivity to the overlapping region. Then, five different meta-heuristic algorithms are comparatively applied. Two of them have been used for the first time on image registration. BDSR remarkably outperforms the previous successful representations, such as entropy images, self-similarity context, and modality-independent local binary pattern, as well as mutual information-based registration, in terms of success rate, runtime, convergence error, and representation construction.
Collapse
Affiliation(s)
- Mojtaba Soleimani
- Faculty of Electrical and Computer Engineering, Babol Noshirvani University of Technology, Babol, Iran
| | - Ali Aghagolzadeh
- Faculty of Electrical and Computer Engineering, Babol Noshirvani University of Technology, Babol, Iran.
| | - Mehdi Ezoji
- Faculty of Electrical and Computer Engineering, Babol Noshirvani University of Technology, Babol, Iran
| |
Collapse
|
29
|
NaGAN: Nadir-like Generative Adversarial Network for Off-Nadir Object Detection of Multi-View Remote Sensing Imagery. REMOTE SENSING 2022. [DOI: 10.3390/rs14040975] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/10/2022]
Abstract
Detecting off-nadir objects is a well-known challenge in remote sensing due to the distortion and mutable representation. Existing methods mainly focus on a narrow range of view angles, and they ignore broad-view pantoscopic remote sensing imagery. To address the off-nadir object detection problem in remote sensing, a new nadir-like generative adversarial network (NaGAN) is proposed in this paper by narrowing the representation differences between the off-nadir and nadir object. NaGAN consists of a generator and a discriminator, in which the generator learns to transform the off-nadir object to a nadir-like one so that they are difficult to discriminate by the discriminator, and the discriminator competes with the generator to learn more nadir-like features. With the progressive competition between the generator and discriminator, the performances of off-nadir object detection are improved significantly. Extensive evaluations on the challenging SpaceNet benchmark for remote sensing demonstrate the superiority of NaGAN to the well-established state-of-the-art in detecting off-nadir objects.
Collapse
|
30
|
Yousef R, Gupta G, Yousef N, Khari M. A holistic overview of deep learning approach in medical imaging. MULTIMEDIA SYSTEMS 2022; 28:881-914. [PMID: 35079207 PMCID: PMC8776556 DOI: 10.1007/s00530-021-00884-5] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/09/2021] [Accepted: 12/23/2021] [Indexed: 05/07/2023]
Abstract
Medical images are a rich source of invaluable necessary information used by clinicians. Recent technologies have introduced many advancements for exploiting the most of this information and use it to generate better analysis. Deep learning (DL) techniques have been empowered in medical images analysis using computer-assisted imaging contexts and presenting a lot of solutions and improvements while analyzing these images by radiologists and other specialists. In this paper, we present a survey of DL techniques used for variety of tasks along with the different medical image's modalities to provide critical review of the recent developments in this direction. We have organized our paper to provide significant contribution of deep leaning traits and learn its concepts, which is in turn helpful for non-expert in medical society. Then, we present several applications of deep learning (e.g., segmentation, classification, detection, etc.) which are commonly used for clinical purposes for different anatomical site, and we also present the main key terms for DL attributes like basic architecture, data augmentation, transfer learning, and feature selection methods. Medical images as inputs to deep learning architectures will be the mainstream in the coming years, and novel DL techniques are predicted to be the core of medical images analysis. We conclude our paper by addressing some research challenges and the suggested solutions for them found in literature, and also future promises and directions for further developments.
Collapse
Affiliation(s)
- Rammah Yousef
- Yogananda School of AI Computer and Data Sciences, Shoolini University, Solan, 173229 Himachal Pradesh India
| | - Gaurav Gupta
- Yogananda School of AI Computer and Data Sciences, Shoolini University, Solan, 173229 Himachal Pradesh India
| | - Nabhan Yousef
- Electronics and Communication Engineering, Marwadi University, Rajkot, Gujrat India
| | - Manju Khari
- Jawaharlal Nehru University, New Delhi, India
| |
Collapse
|
31
|
|
32
|
Song H, Chen L, Cui Y, Li Q, Wang Q, Fan J, Yang J, Zhang L. Denoising of MR and CT images using cascaded multi-supervision convolutional neural networks with progressive training. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2020.10.118] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/21/2022]
|
33
|
|
34
|
Zhao W, Shen L, Islam MT, Qin W, Zhang Z, Liang X, Zhang G, Xu S, Li X. Artificial intelligence in image-guided radiotherapy: a review of treatment target localization. Quant Imaging Med Surg 2021; 11:4881-4894. [PMID: 34888196 PMCID: PMC8611462 DOI: 10.21037/qims-21-199] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2021] [Accepted: 07/05/2021] [Indexed: 01/06/2023]
Abstract
Modern conformal beam delivery techniques require image-guidance to ensure the prescribed dose to be delivered as planned. Recent advances in artificial intelligence (AI) have greatly augmented our ability to accurately localize the treatment target while sparing the normal tissues. In this paper, we review the applications of AI-based algorithms in image-guided radiotherapy (IGRT), and discuss the indications of these applications to the future of clinical practice of radiotherapy. The benefits, limitations and some important trends in research and development of the AI-based IGRT techniques are also discussed. AI-based IGRT techniques have the potential to monitor tumor motion, reduce treatment uncertainty and improve treatment precision. Particularly, these techniques also allow more healthy tissue to be spared while keeping tumor coverage the same or even better.
Collapse
Affiliation(s)
- Wei Zhao
- School of Physics, Beihang University, Beijing, China
| | - Liyue Shen
- Department of Radiation Oncology, Stanford University, Stanford, USA
| | - Md Tauhidul Islam
- Department of Radiation Oncology, Stanford University, Stanford, USA
| | - Wenjian Qin
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Zhicheng Zhang
- Department of Radiation Oncology, Stanford University, Stanford, USA
| | - Xiaokun Liang
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Gaolong Zhang
- School of Physics, Beihang University, Beijing, China
| | - Shouping Xu
- Department of Radiation Oncology, PLA General Hospital, Beijing, China
| | - Xiaomeng Li
- Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, China
| |
Collapse
|
35
|
Xiao H, Teng X, Liu C, Li T, Ren G, Yang R, Shen D, Cai J. A review of deep learning-based three-dimensional medical image registration methods. Quant Imaging Med Surg 2021; 11:4895-4916. [PMID: 34888197 PMCID: PMC8611468 DOI: 10.21037/qims-21-175] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2021] [Accepted: 07/15/2021] [Indexed: 01/10/2023]
Abstract
Medical image registration is a vital component of many medical procedures, such as image-guided radiotherapy (IGRT), as it allows for more accurate dose-delivery and better management of side effects. Recently, the successful implementation of deep learning (DL) in various fields has prompted many research groups to apply DL to three-dimensional (3D) medical image registration. Several of these efforts have led to promising results. This review summarized the progress made in DL-based 3D image registration over the past 5 years and identify existing challenges and potential avenues for further research. The collected studies were statistically analyzed based on the region of interest (ROI), image modality, supervision method, and registration evaluation metrics. The studies were classified into three categories: deep iterative registration, supervised registration, and unsupervised registration. The studies are thoroughly reviewed and their unique contributions are highlighted. A summary is presented following a review of each category of study, discussing its advantages, challenges, and trends. Finally, the common challenges for all categories are discussed, and potential future research topics are identified.
Collapse
Affiliation(s)
- Haonan Xiao
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong, China
| | - Xinzhi Teng
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong, China
| | - Chenyang Liu
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong, China
| | - Tian Li
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong, China
| | - Ge Ren
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong, China
| | - Ruijie Yang
- Department of Radiation Oncology, Peking University Third Hospital, Beijing, China
| | - Dinggang Shen
- School of Biomedical Engineering, ShanghaiTech University, Shanghai, China
- Shanghai United Imaging Intelligence Co., Ltd., Shanghai, China
- Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea
| | - Jing Cai
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong, China
| |
Collapse
|
36
|
Ha IY, Heinrich MP. Modality-agnostic self-supervised deep feature learning and fast instance optimisation for multimodal fusion in ultrasound-guided interventions. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 211:106374. [PMID: 34601186 DOI: 10.1016/j.cmpb.2021.106374] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/20/2020] [Accepted: 08/22/2021] [Indexed: 06/13/2023]
Abstract
BACKGROUND AND OBJECTIVE Fast and robust alignment of pre-operative MRI planning scans to intra-operative ultrasound is an important aspect for automatically supporting image-guided interventions. Thus far, learning-based approaches have failed to tackle the intertwined objectives of fast inference computation time and robustness to unexpectedly large motion and misalignment. In this work, we propose a novel method that decouples deep feature learning and the computation of long ranging local displacement probability maps from fast and robust global transformation prediction. METHODS In our approach, we firstly train a convolutional neural network (CNN) to extract modality-agnostic features with sub-second computation times for both 3D volumes during inference. Using sparsity-based network weight pruning, the model complexity and computation times can be substantially reduced. Based on these features, a large discretized search range of 3D motion vectors is explored to compute a probabilistic displacement map for each control point. These 3D probability maps are employed in our newly proposed, computationally efficient, instance optimisation that robustly estimates the most likely globally linear transformation that best reflects the local displacement beliefs subject to outlier rejection. RESULTS Our experimental validation demonstrates state-of-the-art accuracy on the challenging CuRIOUS dataset with average target registration errors of 2.50 mm, model size of only 1.2 MByte and run times of approx. 3 seconds for a full 3D multimodal registration. CONCLUSION We show that a significant improvement in accuracy and robustness can be gained with instance optimisation and our fast self-supervised deep learning model can achieve state-of-the-art accuracy on challenging registration task in only 3 seconds.
Collapse
Affiliation(s)
- In Young Ha
- Institute of Medical Informatics, University of Luebeck, Ratzeburger Allee 160, 23564 Luebeck, Germany
| | - Mattias P Heinrich
- Institute of Medical Informatics, University of Luebeck, Ratzeburger Allee 160, 23564 Luebeck, Germany.
| |
Collapse
|
37
|
Wu M, Chai Z, Qian G, Lin H, Wang Q, Wang L, Chen H. Development and Evaluation of a Deep Learning Algorithm for Rib Segmentation and Fracture Detection from Multicenter Chest CT Images. Radiol Artif Intell 2021; 3:e200248. [PMID: 34617026 DOI: 10.1148/ryai.2021200248] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2020] [Revised: 06/07/2021] [Accepted: 06/29/2020] [Indexed: 12/12/2022]
Abstract
Purpose To evaluate the performance of a deep learning-based algorithm for automatic detection and labeling of rib fractures from multicenter chest CT images. Materials and Methods This retrospective study included 10 943 patients (mean age, 55 years; 6418 men) from six hospitals (January 1, 2017 to December 30, 2019), which consisted of patients with and without rib fractures who underwent CT. The patients were separated into one training set (n = 2425), two lesion-level test sets (n = 362 and 105), and one examination-level test set (n = 8051). Free-response receiver operating characteristic (FROC) score (mean sensitivity of seven different false-positive rates), precision, sensitivity, and F1 score were used as metrics to assess rib fracture detection performance. Area under the receiver operating characteristic curve (AUC), sensitivity, and specificity were employed to evaluate the classification accuracy. The mean Dice coefficient and accuracy were used to assess the performance of rib labeling. Results In the detection of rib fractures, the model showed an FROC score of 84.3% on test set 1. For test set 2, the algorithm achieved a detection performance (precision, 82.2%; sensitivity, 84.9%; F1 score, 83.3%) comparable to three radiologists (precision, 81.7%, 98.0%, 92.0%; sensitivity, 91.2%, 78.6%, 69.2%; F1 score, 86.1%, 87.2%, 78.9%). When the radiologists used the algorithm, the mean sensitivity of the three radiologists showed an improvement (from 79.7% to 89.2%), with precision achieving similar performance (from 90.6% to 88.4%). Furthermore, the model achieved an AUC of 0.93 (95% CI: 0.91, 0.94), sensitivity of 87.9% (95% CI: 83.7%, 91.4%), and specificity of 85.3% (95% CI: 74.6%, 89.8%) on test set 3. On a subset of test set 1, the model achieved a Dice score of 0.827 with an accuracy of 96.0% for rib segmentation. Conclusion The developed deep learning algorithm was capable of detecting rib fractures, as well as corresponding anatomic locations on CT images.Keywords CT, Ribs© RSNA, 2021.
Collapse
Affiliation(s)
- Mingxiang Wu
- Department of Radiology, Shenzhen People's Hospital, Luohu, China (M.W.); AI Research Laboratory, Imsight Technology, Nanshan, China (Z.C., H.L.); Peng Cheng Laboratory, Nanshan, China (G.Q.); Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China (Q.W.); Department of Computer Science, School of Informatics, Xiamen University, Xiamen, China (L.W.); and Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong (H.C.)
| | - Zhizhong Chai
- Department of Radiology, Shenzhen People's Hospital, Luohu, China (M.W.); AI Research Laboratory, Imsight Technology, Nanshan, China (Z.C., H.L.); Peng Cheng Laboratory, Nanshan, China (G.Q.); Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China (Q.W.); Department of Computer Science, School of Informatics, Xiamen University, Xiamen, China (L.W.); and Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong (H.C.)
| | - Guangwu Qian
- Department of Radiology, Shenzhen People's Hospital, Luohu, China (M.W.); AI Research Laboratory, Imsight Technology, Nanshan, China (Z.C., H.L.); Peng Cheng Laboratory, Nanshan, China (G.Q.); Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China (Q.W.); Department of Computer Science, School of Informatics, Xiamen University, Xiamen, China (L.W.); and Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong (H.C.)
| | - Huangjing Lin
- Department of Radiology, Shenzhen People's Hospital, Luohu, China (M.W.); AI Research Laboratory, Imsight Technology, Nanshan, China (Z.C., H.L.); Peng Cheng Laboratory, Nanshan, China (G.Q.); Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China (Q.W.); Department of Computer Science, School of Informatics, Xiamen University, Xiamen, China (L.W.); and Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong (H.C.)
| | - Qiong Wang
- Department of Radiology, Shenzhen People's Hospital, Luohu, China (M.W.); AI Research Laboratory, Imsight Technology, Nanshan, China (Z.C., H.L.); Peng Cheng Laboratory, Nanshan, China (G.Q.); Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China (Q.W.); Department of Computer Science, School of Informatics, Xiamen University, Xiamen, China (L.W.); and Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong (H.C.)
| | - Liansheng Wang
- Department of Radiology, Shenzhen People's Hospital, Luohu, China (M.W.); AI Research Laboratory, Imsight Technology, Nanshan, China (Z.C., H.L.); Peng Cheng Laboratory, Nanshan, China (G.Q.); Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China (Q.W.); Department of Computer Science, School of Informatics, Xiamen University, Xiamen, China (L.W.); and Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong (H.C.)
| | - Hao Chen
- Department of Radiology, Shenzhen People's Hospital, Luohu, China (M.W.); AI Research Laboratory, Imsight Technology, Nanshan, China (Z.C., H.L.); Peng Cheng Laboratory, Nanshan, China (G.Q.); Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China (Q.W.); Department of Computer Science, School of Informatics, Xiamen University, Xiamen, China (L.W.); and Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong (H.C.)
| |
Collapse
|
38
|
Brosch-Lenz J, Yousefirizi F, Zukotynski K, Beauregard JM, Gaudet V, Saboury B, Rahmim A, Uribe C. Role of Artificial Intelligence in Theranostics:: Toward Routine Personalized Radiopharmaceutical Therapies. PET Clin 2021; 16:627-641. [PMID: 34537133 DOI: 10.1016/j.cpet.2021.06.002] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Abstract
We highlight emerging uses of artificial intelligence (AI) in the field of theranostics, focusing on its significant potential to enable routine and reliable personalization of radiopharmaceutical therapies (RPTs). Personalized RPTs require patient-specific dosimetry calculations accompanying therapy. Additionally we discuss the potential to exploit biological information from diagnostic and therapeutic molecular images to derive biomarkers for absorbed dose and outcome prediction; toward personalization of therapies. We try to motivate the nuclear medicine community to expand and align efforts into making routine and reliable personalization of RPTs a reality.
Collapse
Affiliation(s)
- Julia Brosch-Lenz
- Department of Integrative Oncology, BC Cancer Research Institute, 675 West 10th Avenue, Vancouver, British Columbia V5Z 1L3, Canada
| | - Fereshteh Yousefirizi
- Department of Integrative Oncology, BC Cancer Research Institute, 675 West 10th Avenue, Vancouver, British Columbia V5Z 1L3, Canada
| | - Katherine Zukotynski
- Department of Medicine and Radiology, McMaster University, 1200 Main Street West, Hamilton, Ontario L9G 4X5, Canada
| | - Jean-Mathieu Beauregard
- Department of Radiology and Nuclear Medicine, Cancer Research Centre, Université Laval, 2325 Rue de l'Université, Québec City, Quebec G1V 0A6, Canada; Department of Medical Imaging, Research Center (Oncology Axis), CHU de Québec - Université Laval, 2325 Rue de l'Université, Québec City, Quebec G1V 0A6, Canada
| | - Vincent Gaudet
- Department of Electrical and Computer Engineering, University of Waterloo, 200 University Avenue West, Waterloo, Ontario N2L 3G1, Canada
| | - Babak Saboury
- Department of Radiology and Imaging Sciences, Clinical Center, National Institutes of Health, 9000 Rockville Pike, Bethesda, MD 20892, USA; Department of Computer Science and Electrical Engineering, University of Maryland Baltimore County, Baltimore, MD, USA; Department of Radiology, Hospital of the University of Pennsylvania, 3400 Spruce Street, Philadelphia, PA 19104, USA
| | - Arman Rahmim
- Department of Integrative Oncology, BC Cancer Research Institute, 675 West 10th Avenue, Vancouver, British Columbia V5Z 1L3, Canada; Department of Radiology, University of British Columbia, 11th Floor, 2775 Laurel St, Vancouver, British Columbia V5Z 1M9, Canada; Department of Physics, University of British Columbia, 325 - 6224 Agricultural Road, Vancouver, British Columbia V6T 1Z1, Canada
| | - Carlos Uribe
- Department of Radiology, University of British Columbia, 11th Floor, 2775 Laurel St, Vancouver, British Columbia V5Z 1M9, Canada; Department of Functional Imaging, BC Cancer, 675 West 10th Avenue, Vancouver, British Columbia V5Z 1L3, Canada.
| |
Collapse
|
39
|
Takeshima H. Deep Learning and Its Application to Function Approximation for MR in Medicine: An Overview. Magn Reson Med Sci 2021; 21:553-568. [PMID: 34544924 DOI: 10.2463/mrms.rev.2021-0040] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022] Open
Abstract
This article presents an overview of deep learning (DL) and its applications to function approximation for MR in medicine. The aim of this article is to help readers develop various applications of DL. DL has made a large impact on the literature of many medical sciences, including MR. However, its technical details are not easily understandable for non-experts of machine learning (ML).The first part of this article presents an overview of DL and its related technologies, such as artificial intelligence (AI) and ML. AI is explained as a function that can receive many inputs and produce many outputs. ML is a process of fitting the function to training data. DL is a kind of ML, which uses a composite of many functions to approximate the function of interest. This composite function is called a deep neural network (DNN), and the functions composited into a DNN are called layers. This first part also covers the underlying technologies required for DL, such as loss functions, optimization, initialization, linear layers, non-linearities, normalization, recurrent neural networks, regularization, data augmentation, residual connections, autoencoders, generative adversarial networks, model and data sizes, and complex-valued neural networks.The second part of this article presents an overview of the applications of DL in MR and explains how functions represented as DNNs are applied to various applications, such as RF pulse, pulse sequence, reconstruction, motion correction, spectroscopy, parameter mapping, image synthesis, and segmentation.
Collapse
Affiliation(s)
- Hidenori Takeshima
- Advanced Technology Research Department, Research and Development Center, Canon Medical Systems Corporation
| |
Collapse
|
40
|
Martin D, Tong E, Kelly B, Yeom K, Yedavalli V. Current Perspectives of Artificial Intelligence in Pediatric Neuroradiology: An Overview. FRONTIERS IN RADIOLOGY 2021; 1:713681. [PMID: 37492174 PMCID: PMC10365125 DOI: 10.3389/fradi.2021.713681] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/23/2021] [Accepted: 07/21/2021] [Indexed: 07/27/2023]
Abstract
Artificial Intelligence, Machine Learning, and myriad related techniques are becoming ever more commonplace throughout industry and society, and radiology is by no means an exception. It is essential for every radiologists of every subspecialty to gain familiarity and confidence with these techniques as they become increasingly incorporated into the routine practice in both academic and private practice settings. In this article, we provide a brief review of several definitions and techniques that are commonly used in AI, and in particular machine vision, and examples of how they are currently being applied to the setting of clinical neuroradiology. We then review the unique challenges that the adoption and application of faces within the subspecialty of pediatric neuroradiology, and how these obstacles may be overcome. We conclude by presenting specific examples of how AI is currently being applied within the field of pediatric neuroradiology and the potential opportunities that are available for future applications.
Collapse
Affiliation(s)
- Dann Martin
- Vanderbilt University, Nashville, TN, United States
| | - Elizabeth Tong
- Department of Neuroradiology, Stanford Health Care, Stanford, CA, United States
| | - Brendan Kelly
- Insight Centre for Data Analytics, University College Dublin, Dublin, Ireland
| | - Kristen Yeom
- Department of Neuroradiology, Stanford Health Care, Stanford, CA, United States
| | | |
Collapse
|
41
|
Guan S, Wang T, Sun K, Meng C. Transfer Learning for Nonrigid 2D/3D Cardiovascular Images Registration. IEEE J Biomed Health Inform 2021; 25:3300-3309. [PMID: 33347417 DOI: 10.1109/jbhi.2020.3045977] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Cardiovascular image registration is an essential approach to combine the advantages of preoperative 3D computed tomography angiograph (CTA) images and intraoperative 2D X-ray/digital subtraction angiography (DSA) images together in minimally invasive vascular interventional surgery (MIVI). Recent studies have shown that convolutional neural network (CNN) regression model can be used to register these two modality vascular images with fast speed and satisfactory accuracy. However, CNN regression model trained by tens of thousands of images of one patient is often unable to be applied to another patient due to the large difference and deformation of vascular structure in different patients. To overcome this challenge, we evaluate the ability of transfer learning (TL) for the registration of 2D/3D deformable cardiovascular images. Frozen weights in the convolutional layers were optimized to find the best common feature extractors for TL. After TL, the training data set size was reduced to 200 for a randomly selected patient to get accurate registration results. We compared the effectiveness of our proposed nonrigid registration model after TL with not only that without TL but also some traditional intensity-based methods to evaluate that our nonrigid model after TL performs better on deformable cardiovascular image registration.
Collapse
|
42
|
|
43
|
van Eijnatten M, Rundo L, Batenburg KJ, Lucka F, Beddowes E, Caldas C, Gallagher FA, Sala E, Schönlieb CB, Woitek R. 3D deformable registration of longitudinal abdominopelvic CT images using unsupervised deep learning. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 208:106261. [PMID: 34289437 DOI: 10.1016/j.cmpb.2021.106261] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/01/2020] [Accepted: 06/24/2021] [Indexed: 06/13/2023]
Abstract
BACKGROUND AND OBJECTIVES Deep learning is being increasingly used for deformable image registration and unsupervised approaches, in particular, have shown great potential. However, the registration of abdominopelvic Computed Tomography (CT) images remains challenging due to the larger displacements compared to those in brain or prostate Magnetic Resonance Imaging datasets that are typically considered as benchmarks. In this study, we investigate the use of the commonly used unsupervised deep learning framework VoxelMorph for the registration of a longitudinal abdominopelvic CT dataset acquired in patients with bone metastases from breast cancer. METHODS As a pre-processing step, the abdominopelvic CT images were refined by automatically removing the CT table and all other extra-corporeal components. To improve the learning capabilities of the VoxelMorph framework when only a limited amount of training data is available, a novel incremental training strategy is proposed based on simulated deformations of consecutive CT images in the longitudinal dataset. This devised training strategy was compared against training on simulated deformations of a single CT volume. A widely used software toolbox for deformable image registration called NiftyReg was used as a benchmark. The evaluations were performed by calculating the Dice Similarity Coefficient (DSC) between manual vertebrae segmentations and the Structural Similarity Index (SSIM). RESULTS The CT table removal procedure allowed both VoxelMorph and NiftyReg to achieve significantly better registration performance. In a 4-fold cross-validation scheme, the incremental training strategy resulted in better registration performance compared to training on a single volume, with a mean DSC of 0.929±0.037 and 0.883±0.033, and a mean SSIM of 0.984±0.009 and 0.969±0.007, respectively. Although our deformable image registration method did not outperform NiftyReg in terms of DSC (0.988±0.003) or SSIM (0.995±0.002), the registrations were approximately 300 times faster. CONCLUSIONS This study showed the feasibility of deep learning based deformable registration of longitudinal abdominopelvic CT images via a novel incremental training strategy based on simulated deformations.
Collapse
Affiliation(s)
- Maureen van Eijnatten
- Centrum Wiskunde & Informatica, 1098 XG Amsterdam, the Netherlands; Medical Image Analysis Group, Department of Biomedical Engineering, Eindhoven University of Technology, 5600 MB Eindhoven, the Netherlands.
| | - Leonardo Rundo
- Department of Radiology, University of Cambridge, CB2 0QQ Cambridge, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, CB2 0RE Cambridge, United Kingdom
| | - K Joost Batenburg
- Centrum Wiskunde & Informatica, 1098 XG Amsterdam, the Netherlands; Mathematical Institute, Leiden University, 2300 RA Leiden, the Netherlands
| | - Felix Lucka
- Centrum Wiskunde & Informatica, 1098 XG Amsterdam, the Netherlands; Centre for Medical Image Computing, University College London, WC1E 6BT London, United Kingdom
| | - Emma Beddowes
- Cancer Research UK Cambridge Centre, University of Cambridge, CB2 0RE Cambridge, United Kingdom; Cancer Research UK Cambridge Institute, University of Cambridge, CB2 0RE Cambridge, United Kingdom; Department of Oncology, Addenbrooke's Hospital, Cambridge University Hospitals National Health Service (NHS) Foundation Trust, CB2 0QQ Cambridge, United Kingdom
| | - Carlos Caldas
- Cancer Research UK Cambridge Centre, University of Cambridge, CB2 0RE Cambridge, United Kingdom; Cancer Research UK Cambridge Institute, University of Cambridge, CB2 0RE Cambridge, United Kingdom; Department of Oncology, Addenbrooke's Hospital, Cambridge University Hospitals National Health Service (NHS) Foundation Trust, CB2 0QQ Cambridge, United Kingdom
| | - Ferdia A Gallagher
- Department of Radiology, University of Cambridge, CB2 0QQ Cambridge, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, CB2 0RE Cambridge, United Kingdom
| | - Evis Sala
- Department of Radiology, University of Cambridge, CB2 0QQ Cambridge, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, CB2 0RE Cambridge, United Kingdom
| | - Carola-Bibiane Schönlieb
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, CB3 0WA Cambridge, United Kingdom
| | - Ramona Woitek
- Department of Radiology, University of Cambridge, CB2 0QQ Cambridge, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, CB2 0RE Cambridge, United Kingdom; Department of Biomedical Imaging and Image-guided Therapy, Medical University Vienna, 1090 Vienna, Austria
| |
Collapse
|
44
|
Sankareswaran SP, Krishnan M. Unsupervised end-to-end Brain Tumor Magnetic Resonance Image Registration using RBCNN: Rigid Transformation, B-Spline Transformation and Convolutional Neural Network. Curr Med Imaging 2021; 18:387-397. [PMID: 34365954 DOI: 10.2174/1573405617666210806125526] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2021] [Revised: 05/20/2021] [Accepted: 06/01/2021] [Indexed: 11/22/2022]
Abstract
BACKGROUND Image registration is the process of aligning two or more images in a single coordinate. Now a days, medical image registration plays a significant role in computer assisted disease diagnosis, treatment, and surgery. The different modalities available in the medical image makes medical image registration as an essential step in Computer Assisted Diagnosis(CAD), Computer-Aided Therapy (CAT) and Computer-Assisted Surgery (CAS). Problem definition: Recently many learning based methods were employed for disease detection and classification but those methods were not suitable for real time due to delayed response and need of pre alignment,labeling. METHOD The proposed research constructed a deep learning model with Rigid transform and B-Spline transform for medical image registration for an automatic brain tumour finding. The proposed research consists of two steps. First steps uses Rigid transformation based Convolutional Neural Network and the second step uses B-Spline transform based Convolutional Neural Network. The model is trained and tested with 3624 MR (Magnetic Resonance) images to assess the performance. The researchers believe that MR images helps in success the treatment of brain tumour people. RESULT The result of the proposed method is compared with the Rigid Convolutional Neural Network (CNN), Rigid CNN + Thin-Plat Spline (TPS), Affine CNN, Voxel morph, ADMIR (Affine and Deformable Medical Image Registration) and ANT(Advanced Normalization Tools) using DICE score, average symmetric surface distance (ASD), and Hausdorff distance. CONCLUSION The RBCNN model will help the physician to automatically detect and classify the brain tumor quickly(18 Sec) and efficiently with out doing any pre-alignment and labeling.
Collapse
Affiliation(s)
- Senthil Pandi Sankareswaran
- Department of Computer Science and Engineering, Mohamed Sathak A. J. College of Engineering, Tamil Nadu. India
| | - Mahadevan Krishnan
- Department of Electrical and Electronics Engineering, PSNA College of Engineering and Technology, Tamil Nadu. India
| |
Collapse
|
45
|
Bayesian Fully Convolutional Networks for Brain Image Registration. JOURNAL OF HEALTHCARE ENGINEERING 2021; 2021:5528160. [PMID: 34354807 PMCID: PMC8331272 DOI: 10.1155/2021/5528160] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/28/2021] [Revised: 06/17/2021] [Accepted: 07/13/2021] [Indexed: 11/30/2022]
Abstract
The purpose of medical image registration is to find geometric transformations that align two medical images so that the corresponding voxels on two images are spatially consistent. Nonrigid medical image registration is a key step in medical image processing, such as image comparison, data fusion, target recognition, and pathological change analysis. Existing registration methods only consider registration accuracy but largely neglect the uncertainty of registration results. In this work, a method based on the Bayesian fully convolutional neural network is proposed for nonrigid medical image registration. The proposed method can generate a geometric uncertainty map to calculate the uncertainty of registration results. This uncertainty can be interpreted as a confidence interval, which is essential for judging whether the source data are abnormal. Moreover, the proposed method introduces group normalization, which is conducive to the network convergence of the Bayesian neural network. Some representative learning-based image registration methods are compared with the proposed method on different image datasets. Experimental results show that the registration accuracy of the proposed method is better than that of the methods, and its antifolding performance is comparable to that of fast image registration and VoxelMorph. Furthermore, the proposed method can evaluate the uncertainty of registration results.
Collapse
|
46
|
Prakash J, Wang V, Quinn RE, Mitchell CS. Unsupervised Machine Learning to Identify Separable Clinical Alzheimer's Disease Sub-Populations. Brain Sci 2021; 11:977. [PMID: 34439596 PMCID: PMC8392842 DOI: 10.3390/brainsci11080977] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2021] [Revised: 07/10/2021] [Accepted: 07/20/2021] [Indexed: 11/20/2022] Open
Abstract
Heterogeneity among Alzheimer's disease (AD) patients confounds clinical trial patient selection and therapeutic efficacy evaluation. This work defines separable AD clinical sub-populations using unsupervised machine learning. Clustering (t-SNE followed by k-means) of patient features and association rule mining (ARM) was performed on the ADNIMERGE dataset from the Alzheimer's Disease Neuroimaging Initiative (ADNI). Patient sociodemographics, brain imaging, biomarkers, cognitive tests, and medication usage were included for analysis. Four AD clinical sub-populations were identified using between-cluster mean fold changes [cognitive performance, brain volume]: cluster-1 represented least severe disease [+17.3, +13.3]; cluster-0 [-4.6, +3.8] and cluster-3 [+10.8, -4.9] represented mid-severity sub-populations; cluster-2 represented most severe disease [-18.4, -8.4]. ARM assessed frequently occurring pharmacologic substances within the 4 sub-populations. No drug class was associated with the least severe AD (cluster-1), likely due to lesser antecedent disease. Anti-hyperlipidemia drugs associated with cluster-0 (mid-severity, higher volume). Interestingly, antioxidants vitamin C and E associated with cluster-3 (mid-severity, higher cognition). Anti-depressants like Zoloft associated with most severe disease (cluster-2). Vitamin D is protective for AD, but ARM identified significant underutilization across all AD sub-populations. Identification and feature characterization of four distinct AD sub-population "clusters" using standard clinical features enhances future clinical trial selection criteria and cross-study comparative analysis.
Collapse
Affiliation(s)
- Jayant Prakash
- Laboratory for Pathology Dynamics, Department of Biomedical Engineering, Georgia Institute of Technology and Emory University School of Medicine, Atlanta, GA 30332, USA; (J.P.); (V.W.); (R.E.Q.III)
- Department of Computer Science, Georgia Institute of Technology, Atlanta, GA 30332, USA
| | - Velda Wang
- Laboratory for Pathology Dynamics, Department of Biomedical Engineering, Georgia Institute of Technology and Emory University School of Medicine, Atlanta, GA 30332, USA; (J.P.); (V.W.); (R.E.Q.III)
| | - Robert E. Quinn
- Laboratory for Pathology Dynamics, Department of Biomedical Engineering, Georgia Institute of Technology and Emory University School of Medicine, Atlanta, GA 30332, USA; (J.P.); (V.W.); (R.E.Q.III)
- Department of Computer Science, Georgia Institute of Technology, Atlanta, GA 30332, USA
| | - Cassie S. Mitchell
- Laboratory for Pathology Dynamics, Department of Biomedical Engineering, Georgia Institute of Technology and Emory University School of Medicine, Atlanta, GA 30332, USA; (J.P.); (V.W.); (R.E.Q.III)
- Center for Machine Learning, Georgia Institute of Technology, Atlanta, GA 30332, USA
| |
Collapse
|
47
|
Aoki T, Kamiya T, Lu H, Terasawa T, Ueno M, Hayashida Y, Murakami S, Korogi Y. CT temporal subtraction: techniques and clinical applications. Quant Imaging Med Surg 2021; 11:2214-2223. [PMID: 34079696 DOI: 10.21037/qims-20-1367] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Affiliation(s)
- Takatoshi Aoki
- Department of Radiology, University of Occupational and Environmental Health School of Medicine, Kitakyushu, Japan
| | - Tohru Kamiya
- Graduate School of Engineering, Kyushu Institute of Technology, Kitakyushu, Japan
| | - Huimin Lu
- Graduate School of Engineering, Kyushu Institute of Technology, Kitakyushu, Japan
| | - Takashi Terasawa
- Department of Radiology, University of Occupational and Environmental Health School of Medicine, Kitakyushu, Japan
| | - Midori Ueno
- Department of Radiology, University of Occupational and Environmental Health School of Medicine, Kitakyushu, Japan
| | - Yoshiko Hayashida
- Department of Radiology, University of Occupational and Environmental Health School of Medicine, Kitakyushu, Japan
| | - Seiichi Murakami
- Department of Radiological Science, Faculty of Health Sciences, Junshin Gakuen University, Fukuoka, Japan
| | - Yukunori Korogi
- Department of Radiology, University of Occupational and Environmental Health School of Medicine, Kitakyushu, Japan
| |
Collapse
|
48
|
Deformable adversarial registration network with multiple loss constraints. Comput Med Imaging Graph 2021; 91:101931. [PMID: 34090262 DOI: 10.1016/j.compmedimag.2021.101931] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2020] [Revised: 03/11/2021] [Accepted: 04/26/2021] [Indexed: 11/23/2022]
Abstract
Deformable medical image registration has the necessary value of theoretical research and clinical application. Traditional methods cannot meet clinical application standards in terms of registration accuracy and efficiency. This article proposes a deformable generate adversarial registration framework, which avoids the dependence on ground-truth deformation. The proposed residual registration network based on Nested U-Net has excellent feature extraction ability and robustness. Multiple constraints that incorporate the potential information of anatomical segmentation extracted by the discriminator can help the model adapt to different modal registration tasks. Through interpatient X-ray chest registration, the deep-supervised training method, and the proposed loss constraint are proved to improve the model's performance and training stability. The experimental results show that our model, compared with state-of-the-art methods, provides a more accurate spatial alignment relationship between different patients' lung organs while ensuring the displacement field's authenticity. Finally, we explored the relationship between the accuracy and validity of the model.
Collapse
|
49
|
Ankenbrand MJ, Lohr D, Schlötelburg W, Reiter T, Wech T, Schreiber LM. Deep learning-based cardiac cine segmentation: Transfer learning application to 7T ultrahigh-field MRI. Magn Reson Med 2021; 86:2179-2191. [PMID: 34002412 DOI: 10.1002/mrm.28822] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2020] [Revised: 03/18/2021] [Accepted: 04/09/2021] [Indexed: 12/24/2022]
Abstract
PURPOSE Artificial neural networks show promising performance in automatic segmentation of cardiac MRI. However, training requires large amounts of annotated data and generalization to different vendors, field strengths, sequence parameters, and pathologies is limited. Transfer learning addresses this challenge, but specific recommendations regarding type and amount of data required is lacking. In this study, we assess data requirements for transfer learning to experimental cardiac MRI at 7T where the segmentation task can be challenging. In addition, we provide guidelines, tools, and annotated data to enable transfer learning approaches by other researchers and clinicians. METHODS A publicly available segmentation model was used to annotate a publicly available data set. This labeled data set was subsequently used to train a neural network for segmentation of left ventricle and myocardium in cardiac cine MRI. The network is used as starting point for transfer learning to 7T cine data of healthy volunteers (n = 22; 7873 images) by updating the pre-trained weights. Structured and random data subsets of different sizes were used to systematically assess data requirements for successful transfer learning. RESULTS Inconsistencies in the publically available data set were corrected, labels created, and a neural network trained. On 7T cardiac cine images the model pre-trained on public imaging data, acquired at 1.5T and 3T, achieved DICELV = 0.835 and DICEMY = 0.670. Transfer learning using 7T cine data and ImageNet weight initialization improved model performance to DICELV = 0.900 and DICEMY = 0.791. Using only end-systolic and end-diastolic images reduced training data by 90%, with no negative impact on segmentation performance (DICELV = 0.908, DICEMY = 0.805). CONCLUSIONS This work demonstrates and quantifies the benefits of transfer learning for cardiac cine image segmentation. We provide practical guidelines for researchers planning transfer learning projects in cardiac MRI and make data, models, and code publicly available.
Collapse
Affiliation(s)
- Markus Johannes Ankenbrand
- Chair of Cellular and Molecular Imaging, Comprehensive Heart Failure Center (CHFC), University Hospital Wuerzburg, Wuerzburg, Germany
| | - David Lohr
- Chair of Cellular and Molecular Imaging, Comprehensive Heart Failure Center (CHFC), University Hospital Wuerzburg, Wuerzburg, Germany
| | - Wiebke Schlötelburg
- Chair of Cellular and Molecular Imaging, Comprehensive Heart Failure Center (CHFC), University Hospital Wuerzburg, Wuerzburg, Germany.,Department of Radiology, University Hospital Wuerzburg, Wuerzburg, Germany
| | - Theresa Reiter
- Chair of Cellular and Molecular Imaging, Comprehensive Heart Failure Center (CHFC), University Hospital Wuerzburg, Wuerzburg, Germany.,Department of Internal Medicine I, University Hospital Wuerzburg, Wuerzburg, Germany
| | - Tobias Wech
- Department of Radiology, University Hospital Wuerzburg, Wuerzburg, Germany
| | - Laura Maria Schreiber
- Chair of Cellular and Molecular Imaging, Comprehensive Heart Failure Center (CHFC), University Hospital Wuerzburg, Wuerzburg, Germany
| |
Collapse
|
50
|
Du B, Liao J, Turkbey B, Yan P. Multi-Task Learning for Registering Images With Large Deformation. IEEE J Biomed Health Inform 2021; 25:1624-1633. [PMID: 32795972 PMCID: PMC8162989 DOI: 10.1109/jbhi.2020.3016699] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Accurate registration of prostate magnetic resonance imaging (MRI) images of the same subject acquired at different time points helps diagnose cancer and monitor the tumor progress. However, it is very challenging especially when one image was acquired with the use of endorectal coil (ERC) but the other was not, which causes significant deformation. Classical iterative image registration methods are also computationally intensive. Deep learning based registration frameworks have recently been developed and demonstrated promising performance. However, the lack of proper constraints often results in unrealistic registration. In this paper, we propose a multi-task learning based registration network with anatomical constraint to address these issues. The proposed approach uses a cycle constraint loss to achieve forward/backward registration and an inverse constraint loss to encourage diffeomorphic registration. In addition, an adaptive anatomical constraint aiming for regularizing the registration network with the use of anatomical labels is introduced through weak supervision. Our experiments on registering prostate MR images of the same subject obtained at different time points with and without ERC show that the proposed method achieves very promising performance under different measures in dealing with the large deformation. Compared with other existing methods, our approach works more efficiently with average running time less than a second and is able to obtain more visually realistic results.
Collapse
Affiliation(s)
- Bo Du
- School of Computer Science, institute of Artificial Intelligence, Wuhan University, Wuhan, China
| | - Jiandong Liao
- School of Computer Science, institute of Artificial Intelligence, Wuhan University, Wuhan, China
| | - Baris Turkbey
- Molecular Imaging Program, National Cancer Institute, National Institutes of Health, Bethesda, MD 20892, USA
| | - Pingkun Yan
- Department of Biomedical Engineering and the Center for Biotechnology and Interdisciplinary Studies at Rensselaer Polytechnic Institute, Troy, NY 12180, USA
| |
Collapse
|